By Celso Villar Torino, Fernando Menezes Silva, and Henrique Gomes Ribeiro
Changing from a conventional operating system to a digital one can be daunting for plant operators. The keys to a successful transition at the 14,000-mw Itaipu hydroelectric plant on the border of Brazil and Paraguay were a thorough plan, a rigorous schedule, and sensitivity to the operators’ needs and perceptions.
In December 2002, Itaipu Binacional completed the installation and commissioning of a new supervisory control and data acquisition (SCADA) system at the Itaipu hydroelectric plant on the Parana River in Brazil and Paraguay. The decision to adopt a digital control system had been a critical one, because the 14,000-mw plant — the largest fully operational hydro project in the world — provides approximately 22 percent of Brazil’s electricity and 97 percent of Paraguay’s. But successful equipment installation and testing was only the beginning. The next step was to equip the plant’s experienced, well-established workforce with the knowledge, confidence, and skills necessary for working in the new operating environment. To meet this challenge, Itaipu’s management developed a two-year program of training and progressive implementation of the SCADA system. As a result of this carefully planned program, the transition from conventional operation to digital operation was accomplished without causing interruptions in power production, equipment damage, or incidents affecting personnel safety.
Making the change to a digital control system
Managers of Itaipu Binacional, a joint agency of the Brazilian and Paraguayan governments, had recognized for some time that industry modernization, the cost and difficulty of acquiring analog parts, and the high value of improving efficiency all pointed toward installation of a SCADA system. After an international public tender of competitive bids, management chose the Ranger SCADA system manufactured by the Bailey division of ABB USA. In addition to supplying the physical system and English language manuals, the manufacturer provided training for a 15-person team of graduate professional engineers from Itaipu. Seven of these trained engineers were assigned to a “SCADA operation expert” group, which would be responsible for training the remaining operation staff.
When the new system was commissioned, a workforce of 80 operators, having on average 17 years of experience, staffed the plant in six-hour shifts. Before the advent of the new system, operators had personally collected 50 items of data from various locations in the plant, completing paper record forms four times per day. The installed SCADA system comprises 18,000 data acquisition points distributed among 750 cubicles within the plant. The entire system is operated from three consoles in the plant’s central control room.
In planning the transition to the new operating system, Itaipu’s managers saw a clear need for a carefully prepared and executed master plan meeting the following objectives:
- To implement a tool affecting 14,000 mw of capacity and more than 90 million megawatt-hours of electricity per year without adversely affecting production;
- To thoroughly exploit all of the new system’s functionality before the expiration of the guarantee period;
- To ensure the safety of personnel and equipment during the transition; and
- To provide an effective procedure for training and certifying professional staff in the use of the new equipment.
Designing an effective training plan
The training plan needed to be carefully balanced between two competing objectives: meeting standards of quality appropriate to the strategic importance of the plant, and completing the transition quickly enough to assess the new equipment’s performance soon after installation.
Henrique Ribeiro, a member of the SCADA operation expert group, instructs an operator in the SCADA A module at the training consoles.
In designing the plan, Itaipu was able to build on lessons learned during a previous transition to a digital control system at a 500-kilovolt substation. Managers felt they knew what to expect, both in terms of the technical aspects of the installation and in terms of human reactions to technological change. In the case of the substation, all of the operators, operation supervisors, and managers had attended a lecture delivered by a psychologist on the subject of managing permanent change in one’s life. The seminar helped the operators to view change as a fact of life and as a positive opportunity, an attitude that they carried into the SCADA transition period.
In light of the volume of information to be transmitted and the relentless time schedule, the training was divided into three modules:
- The overview module, which was delivered in lecture form to all of the operating team and dealt broadly with the elements of the new system;
- The SCADA A module, which covered basic information about operating the system and was delivered in one-on-one training sessions using a console in the training room; and
- The SCADA B module, which addressed the functionality of the system in the same one-on-one format but in greater detail than the A module.
The SCADA system collects data from 18,000 points within the Itaipu hydropower plant, delivering data in real time to operators in this central control room. The entire plant is operated from the central control room.
The training modules were initially delivered to a priority group of 52 operators, selected on the basis of experience and seniority. In addition to conducting the three training modules, the SCADA operation expert group prepared two training manuals. Another important training tool was a simulator package provided by the equipment vendor.
The SCADA operation expert group, consisting of seven members of Itaipu’s professional engineering staff, received training from the equipment manufacturer. Group members were then charged with training operators and preparing manuals for the new operating system.
One other tool proved to be vital for both training and support of operators during normal working conditions. This was the SARTRE (Activation of Rules in Real Time) software, an application that was developed by Itaipu’s engineers in conjunction with the commissioning of the SCADA system. Incontrasttothe SCADAsystem, which records operator errors but does not correct or identify them as errors, SARTRE is designed to detect possible errors and provide real-time guidance to the operator. In Itaipu’s experience, SARTRE and similar applications can make enormous contributions to overcoming natural human resistance to adopting and trusting new tools.
The certification process: measuring success
Each operator was required to obtain certification in the use of the new system in order to continue in his or her position. Certification was contingent upon completing the overview and 30 hours of individual training under each of modules A and B, and also upon successful completion of an exam for each module, covering the material in the training manuals. The SCADA A exam consisted of multiple-choice theoretical questions. A passing score was 70 percent or higher.
The SCADA B exam was more difficult. The theoretical exam included multiple-choice questions, questions requiring a direct answer, and questions requiring action by the trainee at the console. The trainee’s performance at the console was graded as “incorrect,” “correct,” or “correct with difficulty” and entered as part of a weighted exam score. In addition to the theoretical exam, each trainee was presented with three scenarios in the operation simulator. The three scenarios were drawn from a total of nine scenarios that had actually occurred during plant operation. To pass, trainees needed to earn a score of 70 percent or higher in each part of the exam.
The initial results of the training and certification program were a 100 percent approval rate for operators in the priority group taking the SCADA A exam, and a 71 percent approval rate for those taking the SCADA B exam. The trainees who did not initially obtain approval in SCADA B were provided with additional training. In providing the additional training, management took care to maintain discretion about each trainee’s exam scores, and to treat the process as a necessary preparation for technological change — not a selective process that could cost an employee his job. Upon retaking the exam, all of these trainees won approval.
A plan for gradual operation
The SCADA system was equipped with a switch feature that allowed it to be deployed alone or to work in conjunction with some elements of the conventional system. Possible operational modes were as follows:
- Supervision and command exclusively by the conventional system;
- Supervision by both systems, but command only by the conventional system;
- Supervision by both systems but command only by the digital system; and
- Supervision and command exclusively by the digital system.
These options allowed Itaipu’s planners to establish a staged transition to digital operation that was synchronized with the training schedule (see Table 1). In the first stage, digital supervision was available but commands could only be issued through SCADA during commercial hours (6:30 a.m. to 11:00 a.m. and 1:00 p.m. to 3:30 p.m.), and only under the direction of the designated SCADA operation expert group. This stage coincided with the training and SCADA A certification.
After concluding training and SCADA A certification, the schedule progressed to assisted operation in commercial hours and through the lunch hour. Including the lunch hour in the SCADA operation period eliminated two high-risk maneuvers, disconnecting and reconnecting the system for the lunch break.
The third stage, 24-hour assisted operation, involved reassigning the SCADA operation group to the five operating shifts. This stage began after all operators had completed the SCADA B training and certification. In the fourth and last stage before beginning normal operation, operators used round-the-clock SCADA supervision and control without formal assistance by instructors. However, for 30 days one SCADA operation group instructor was present on all shifts to provide support if needed. After this period ended in January 2005, the operation expert group remained among the shifts but were not deliberately assigned to cover all shifts.
The plan included specific guidelines for restoring the plant to normal operation after a disturbance, such as might be caused by an equipment malfunction or outage. In the first and second stages, the operators were not allowed to perform SCADA commands after a disturbance except with the assistance of an instructor. However, after training and certification in SCADA B, the SCADA command was the preferred method of returning to normal operation, unless a shift supervisor determined that the conventional system should be used for reasons of operational safety.
Lessons learned from training and transition
Itaipu’s management attributes the success of the transition process to several specific features of the training plan. Foremost is the focus on professional empowerment of all of the participants, including trainees, instructors, schedulers and logisticians, and programmers. Also essential, especially considering that the operators worked in multiple shift rotations, was the meticulously defined training schedule.
There was also an emphasis on transparency and respect for professional staff who were being subjected to the certification and evaluation process after almost 20 years of experience. Recognizing the operators’ substantial experience with the conventional system, management also sought to use special communication strategies and incentives for change. In addition to these strategies, the SARTRE software, by providing real-time guidance to operators as they worked with the new system, was a key tool for building confidence in the technology.
The transition was completed on schedule with no loss of production attributed to the new operating system, which has been fully operational since January 2005. The system — now controlling two new units commissioned in 2006 and 2007 as well as the original 18 — has resulted in significant quality improvements in many aspects of plant operation, including production, operator training and awareness, safety, and diagnosis of outages or equipment failures.
Mr. Torino, Mr. Menezes, and Mr. Ribeiro may be contacted at ITAIPU Binacional, Operation Power Plant — OPUO.DT, Av. Tancredo Neves, 6731, Caixa Postal: 1577, Foz do Iguaçu, Paraná, CEP 85866-900, Brazil; (55) 45-35203990; E-mail: torino@itaipu. gov.br; email@example.com; and firstname.lastname@example.org.
Celso Villar Torino, manager of the operation division at the Itaipu Binacional hydroelectric power plant, was directly responsible for the transition from the conventional operating system to the digital one. Fernando de Menezes is a senior engineer and operation supervisor at Itaipu. He provided technical support to the operation division’s management during the transition and coordinated the “SCADA expert” group. Henrique Ribeiro, an engineer and operation supervisor at Itaipu, also provided technical support during the transition and was a member of the SCADA expert group.