Save Article Instructions

Upgrading Controls Reduces Unit Downtime

To reduce time needed to troubleshoot operational problems and to get units back on line quickly at seven of its eight hydro plants, Sacramento Municipal Utility District is converting to programmable logic controllers from hard-wired, relay-based logics. The conversion involves upgrading governors and turbine shutoff valve controls. With half the units converted, the utility already sees decreases in unit downtime.

By William G. Pope and Ronald J. Hahn

Sacramento Municipal Utility District (SMUD) is halfway through a project to upgrade the controls at seven of the eight powerhouses that make up the 688-MW Upper American River Project. The goal of this upgrade is to reduce time required to troubleshoot problems with the governors and the turbine shutoff valves. By replacing the hard-wired, relay-based logic controls with a programmable logic controller (PLC) system, SMUD has reduced time needed to address problems, increased operating efficiency of the units, and improved availability of spare parts.

As of September 2008, five of the ten units had been upgraded. The entire project is scheduled to be completed in early 2010.

Reasons behind the upgrade

SMUD’s Upper American River Project consists of eight powerhouses, built between 1960 and 1971, with a total of ten turbine-generating units. The units were equipped with mechanical and MOD II analog governors from Woodward. In 1994, SMUD installed Woodward 501 digital governors on four of the ten units. However, Woodward began phasing out support of the digital governors in 1998. By that time, the analog governors were not supported.

Although SMUD kept all the governors operating by purchasing spare parts from Woodward and other vendors, the district knew it eventually would need to replace them. In 2003, when evaluating a solution to this problem, SMUD determined it needed to install a PLC system.

A PLC system offers a number of advantages over a relay-based control system. First, under the relay-based control system, without support from the manufacturer, spare parts for the governors are not available. Second, the only way to change the function of a relay-based system is by rewiring, which can be time-consuming and expensive. With a PLC-based system, the relay logic is incorporated in the software. Third, troubleshooting a relay-based system is time-consuming. For example, locating a problem involves several hours and requires a current meter. With a PLC-based system, personnel can look at the logic on screen to identify the source of the problem within minutes.

Deciding what to upgrade

In addition to the lack of parts and support for the governors, SMUD faced two other challenges with its controls at the Upper American River Project. First, the turbine shutoff valve control and overspeed trip systems varied widely in their control philosophy, making it difficult for SMUD to standardize its systems and components. Second, components (such as the governor analog control boards, governor power supplies, and turbine shutoff valve control relays) were becoming increasingly problematic, as a result of failures.

The nature of these challenges influenced SMUD’s decisions about what to upgrade. Components being installed include:

Details about the upgrades to governors, turbine shutoff valves, and the SCADA system are provided on pages 11 through 14.

Benefits of the upgrade

SMUD is already reaping several important benefits from this controls upgrade. Benefits include improved generating efficiency, reduced reliance on original equipment manufacturers for spare parts, and standardization of spare parts and troubleshooting. In addition, the utility has improved system monitoring and performance assessment, load rejection response, governor oil hygiene, reliability and repeatability, and system monitoring and trending. The upgrade enables SMUD to reduce oil maintenance, eliminate mercury environmental hazards, and leverage system resources and training.

More details about these benefits are provided on pages 14 and 16.

Work completed, future plans

In the summer of 2005, SMUD contracted with System 3 Inc. in Carmichael, Calif., and North American Phoenix Energy Services (previously North American Hydro, LLC) in Schofield, Wis., to upgrade the governors and turbine shutoff valve control systems for the ten units. North American Phoenix is providing system design, supply, manufacturing, installation supervision, start-up, testing, and training services for the project. System 3 is providing the installation labor services.

SMUD engineers and technicians performed formal design reviews and factory testing of equipment to ensure hassle-free installation and start up. Western Electricity Coordinating Council testing, to ensure the new governor acts appropriately to maintain grid frequency, has been successfully performed.

After the governor and turbine shutoff valve systems are upgraded, another upgrade project is planned to begin in 2012. The next project will include eliminating the relay controls associated with the unit start/stop functions, station air, cooling water, high-pressure lubrication, and annunciator systems. These relay functions will be replaced with PLC control. The existing PLC and SCADA infrastructure will be used, and additional PLC input/output module control cabinets and instrumentation will be installed.

Details of the upgrade

Details of the various components of the upgrade, including work on governors, turbine shutoff valves, and the SCADA system, are described in the following sections.


The original intent of the governor upgrade was to replace the analog and digital governors with a system based on off-the-shelf PLC components and hydraulic pilot control valves, and to interface with the five mechanical governors via PLC components. However, in June 2008, based on the benefits seen with the digital conversions performed, SMUD decided to also upgrade the mechanical governors, complete with PLC components and hydraulic pilot control valves.

The programmable logic controller-based governors associated with the ten turbine-generating units in Sacramento Municipal Utility District’s eight-powerhouse 688-MW Upper American River Project allow better unit control.
Click here to enlarge image

For all governors, SMUD decided to install North American Phoenix digital governor systems, which utilize Allen-Bradley ControlLogix PLCs to control the accumulator and governor oil pumps, speed detection, and unit creep detection.

Upgrades to the governor accumulator and oil pump controls include eliminating the accumulator-mounted level switches and replacing their function with a magnetic level indicator manufactured by Magnetrol. This indicator uses a magnetic float in a stainless steel tube to provide primary level measurement. A transmitter provides a proportional electronic signal to the PLC and is used to start and stop the governor pumps and air compressor. Magnetic switches on the outside of the tube provide for alarming and shutdown functions.

Pressure switches used for lead/lag governor pump control are being replaced with a Siemens pressure transmitter, to allow crews to adjust governor pump lead and lag start pressures without recalibrating pressure switches. In addition, several units have experienced governor pump loading and unloading problems associated with the original mechanisms. All mechanical pump unloaders are being retrofitted with solenoid-driven unloader valves, manufactured by Rexroth, with pump discharge pressure switch feedback.

With regard to speed detection, a new speed detection/speed switch panel will replace mechanical mercury speed switches and Woodward 501/MOD II speed switches.

For the digital governors, closed-loop frequency control, power control, needle sequencing, and wicket gate control are provided through the governor PLC software. The control algorithm, called TEPID (Transient Enhanced Proportional Integral Derivative), is supplied by North American Phoenix Energy Services.

A kidney-loop oil filtration system installed for each of the ten new digital governor systems in seven Sacramento Municipal Utility District powerhouses will reduce annual oil filtration maintenance (from annually to once every five years) and improve reliability.
Click here to enlarge image

For the digital and analog governors, multi-stage pilot valves and shutdown cylinders are being replaced with an integrated manifold consisting of a single-stage proportioning valve and 24-volt direct current (DC) shutdown solenoid. A linear variable differential transformer, which directly measures linear displacement, is installed on each distributing valve to provide feedback of actual valve position for fast response closed-loop position control.

Installation of the new proportioning valves, to replace the Woodward multi-stage governor pilot valves, prompted the installation of improved oil filtration. A 5-micron duplex filtration system is being installed directly upstream of the oil supply to the proportioning valves. To reduce annual oil filtration maintenance and improve reliability, a kidney-loop filtration system is being installed with each new system. SMUD’s governor oil maintenance program includes annual filtration of all governor oil to 5 microns. The upgraded filtration will allow SMUD to reduce its maintenance cycle of complete draining of the accumulator and sump from once a year to every five years.

New magnetostrictive-type, linear position transmitters are being installed on each gate, needle, and deflector servomotor. These transmitters provide position signals for closed-loop gate, needle, and deflector control. These transmitters reduce maintenance because there are no zero and span calibration adjustments on the instruments.

Improved speed sensing, dead stop, and creep detection is being installed for the digital and analog governors. Improved speed sensing consists of installation of three zero-velocity speed pickups and a VT Interface module from North American Phoenix that measures unit frequency from the generator’s voltage transformer. High-resolution digital speed detection is achieved through the PLC. The VT Interface provides for reliable measurements of unit frequency down to about 6 Hertz (Hz), even without excitation on the generator. Before, SMUD had no direct frequency measurement until the unit was excited.

All units are being upgraded with a new creep detection system. Two of the speed pickups installed on the unit speed gear and wired to the PLC provide the primary measurements. The speed pickups are located so that one pickup is on the rising of one gear tooth and one is located on the falling of one gear tooth. Both pickups must show a change of state before the PLC will issue a creep alarm. This feature prevents the new system from reacting to unit vibration or issuing false creep alarms.

Speed switch contacts required to initiate actions such as excitation and initiation of cooling water are being installed, replacing 60 mechanical mercury type switches. The new contacts are being installed in a speed switch panel that consists of redundant speed monitors/ switches, each driven by one of the three zero-velocity pickups.

Turbine shutoff valves

The intent of the turbine shutoff valve control upgrades is to install a common control philosophy for all units and to provide commonality of components and spare parts. A North American Phoenix PLC-based control system is being installed for each turbine shutoff valve system to provide the open and close sequencing, safety interlocking, system status display, and alarming. A color liquid crystal display (LCD) operator interface provides information on the local narrative sequence step status (what step the control system is in as the valve goes through its multi-step opening sequence) and alarming.

The original systems vary widely from plant to plant and include air-operated pilot valves, motor-operated 12-way cam valves, solenoid/mechanical linkage-operated pilot valves, and oil hydraulically controlled pilot valves. The air-operated systems typically are fed from the governor accumulator, and tubing leaks often reduced accumulator air pressure levels, causing unit trips. The solenoid/mechanical linkage systems have had numerous instances of linkage binding and failure. The motor-operated cam valve system has had countless mercury limit switch failures, one of which caused the mercury vial to break, resulting in an extensive hazardous materials cleanup. In addition, normal loosening of the valve cams caused interruptions to the open and close sequencing, resulting in extensive troubleshooting time to reposition the cams.

To provide a more reliable system with adequate instrumentation for troubleshooting, a North American Phoenix system consisting of hydraulic oil-operated shear-seal valves is being installed. Hydraulic pressure to each actuator is controlled with 125-volt DC shear-seal directional valves, which are impervious to buildup of siltation that could cause the valves to stick. Each actuator has open and close limit switches that are integrated with the new PLC. The limit switches provide verification that the actuator valves are in the correct position during open and close sequencing and provide alarm and troubleshooting information. For example, if a valve was commanded to the open or closed position, the limit switches confirm that the valve reached the desired position. Existing upstream and downstream seat limit switches are being integrated with the new PLC.

All units are being provided with pressure switches on the control tubing runs to the upstream and downstream seats to provide status of control pressure. These seats seal the turbine shutoff valve on the upstream and downstream sides when the valve is in the closed position, to provide a leak-free seal. This pressure switch arrangement gives feedback on the actual water pressure being provided to the seats for the two systems where seat limit switches are not feasible to install.

The controls upgrade for the turbine shutoff valves at seven Sacramento Municipal Utility District powerhouses involved installing control line pressure gages and switches. The gages provide a quick and easy status indication of the system and are invaluable in system troubleshooting.
Click here to enlarge image

The prime mover servo and control piping to the turbine shutoff valve are being left in place because they are operating satisfactorily and would be cost-prohibitive to replace. However, the pilot operating and sequencing controls are being completely replaced or upgraded to be consistent with the new control philosophy.

Waterhammer issues in the turbine shutoff valve main servo and seat control lines have been an ongoing issue. To alleviate this problem, new bladder-style accumulators manufactured by Oilair are being installed on all water control lines to the valve main servo, upstream seat, and downstream seat. The accumulators provide a cushion to the piping and tubing when water pressure is applied and released on the line.

Additional troubleshooting aids being installed include pressure gages manufactured by Ashcroft for penstock, scroll case, turbine shutoff valve open and close, bypass valve close, upstream seat apply and retract, downstream seat apply and retract, and turbine shutoff valve trip oil pressures. Previously, personnel had to loosen component fittings to check the presence of water pressure at individual devices. These gages provide a quick and easy status indication of the system and are invaluable in system troubleshooting. All system signals and data are archived locally and at the maintenance headquarters for system trending and troubleshooting using RSBizware Historian and RSView SE software from Rockwell Software and integrated by North American Phoenix.

SCADA system

One major goal while upgrading the governor and turbine shutoff valve control systems was to provide operations, engineering, and maintenance with the ability to view and trend system data from anywhere in the project, including SMUD’s Fresh Pond maintenance headquarters in Pollock Pines, Calif. To ac complish this, SMUD took advantage of an existing infrastructure of high-speed fiber optic and microwave communication between each powerhouse and Fresh Pond.

Work completed includes installation of a PC with short-term (120-day) historical logging capability at each powerhouse, integrated with an HP ProLiant RAID-5 (redundant) historical server at Fresh Pond and an HP 9300 RAID-1 HMI development server at Fresh Pond. The historical server is loaded with RSBizware Historian as a long-term data storage and trending tool and RSView SE as an operator interface tool. An HMI development system (RAID-1 PC grade server) provides an operator interface capable of displaying graphics and trends from any powerhouse, as well as a graphics studio for future graphics development. The historian server continuously collects data from each powerhouse PC for long-term (greater than 120 days) storage. The layout of the HMI/ SCADA network and the configuration of the operator interface software packages provides a robust graphics and historical data logging platform.

Alarm management is also included with the operator interfaces at each powerhouse and at Fresh Pond. The system provides real-time alarming as well as historical alarm data showing times for when an alarm comes in, is acknowledged, and is cleared from the system.

How the upgrade is benefitting SMUD

The controls upgrade is providing several benefits to the utility. These benefits are described in the following paragraphs.

Improved generating efficiency

A new needle sequencing routine resulted in efficiency improvements of 5 to 10 percent for unit loads from 0 to 20 MW at the 82-MW Loon Lake project.

Reduced reliance on OEMs for spare parts

The life cycle and support for original equipment manufacturer (OEM) parts has been short (ten years or less). By using commercially available equivalents, the life of the system before it requires complete replacement is expected to increase to 20 years or more.

Standardization of spare parts and troubleshooting

By standardizing the control strategy and components, SMUD has simplified its spare parts inventory. Previously, a number of incorrect parts had been installed, particularly when replacing turbine shutoff valve air solenoids. This caused several system problems and increased outage time.

Improved system monitoring and performance assessment

Installation of the PLC and HMI systems has improved governor and turbine shutoff valve system monitoring and performance tracking. By reviewing historical trends, engineers and maintenance staff can ensure that each governor is responding as expected. If a problem arises, these trends are used as the main troubleshooting tool to identify the cause.

The valving installed for the governor retrofit project at seven Sacramento Municipal Utility District powerhouses consists of the main distributing valve linear variable differential transformer feedback, hydraulic pilot valve control assembly, and duplex pilot valve filtration.
Click here to enlarge image


Improved load rejection response

The units that have been upgraded to include governing control in the PLC were implemented with North American Phoenix’s TEPID control algorithm, which is designed for enhanced frequency stability control. During start-up load rejection tests performed by SMUD, the units with digital upgrades were found to have significant improvement in load rejection response over units where mechanical governors were retained. For example, at 150-MW Camino, SMUD originally retained the mechanical governor for Unit 1 but upgraded Unit 2 to include governing control in the PLC. SMUD used data from the RSView HMI to generate 100 percent load rejection performance curves for each unit. Peak speed, shown on the curves, indicated a significant reduction in speed increase for Unit 1 during load rejections. SMUD determined that the mechanical overspeed system was tripped during each test of Unit 1. The speed curve was much smoother and less aggressive with the digital governor on Unit 2.

Improved governor oil hygiene and reduced oil maintenance

SMUD’s governor oil maintenance plan included draining and filtering all the governor sump and accumulator oil to 5 microns on an annual basis. Installation of continuous kidney-loop filtration systems has allowed SMUD to update its oil maintenance plan to drain the sump and accumulator for inspection every five years.

Elimination of mercury environmental hazards

By replacing the mechanical mercury speed switches and governor mercury pressure switches, SMUD will be able to eliminate all of the about 60 mercury speed and pressure switches over the course of the upgrade.

Improved reliability and repeatability

Installation of oil hydraulically-operated pilot valve components has improved the reliability and repeatability of the turbine shutoff valve control system. The system is now self-lubricating because all moving parts in the solenoids and actuators are filled with oil. The control actuator’s speed of response has been very repeatable, in contrast to the old system. The bulk of the control relays involved with the open and close sequences are now implemented with PLC logic, removing relay coil failures and dirty contacts from the equation. This improved the robustness of the control logic, and maintenance technicians can easily assess the state of the control system via the PLC logic and HMI graphics.

Improved system monitoring and trending

The original system had no means of real-time monitoring or logging of historical data and alarms. The new system can log all discrete input/output state changes, analog input/output data (based on percent change from the previously logged value), and alarms. The logs can be viewed and changes reviewed to determine where a problem has occurred. All devices are alarmed if feedback does not match the expected state. System performance is easily tracked, as turbine shutoff valve open and close times are logged and reviewed periodically to ensure the valve is operating in a repeatable fashion.

The new control system provides engineering, operation, and maintenance crews with all feedback parameters necessary for timely system recovery. There have been a number of instances of turbine shutoff valve open sequence interruption where the operator has been able to troubleshoot and resolve field device problems (e.g., field valve limit switch misalignment) based on the sequence status information and alarming provided on the local operator display.

Leveraging of system resources and training

Installation of a common control system across all units makes it easier for a technician or engineer to perform maintenance or troubleshoot problems. Once personnel learn to use the system for one unit, they can apply this knowledge to any other unit. This also applies to the PLC system. The PLC logic and operator interface is virtually identical at all sites.

Mr. Pope may be reached at Sacramento Municipal Utility District, 6201 S Street, MS B355, Sacramento, CA 95817; (1) 916-732-5359; E-mail: wpope@smud. org. Mr. Hahn may be reached at North American Phoenix, 8310 Technology Drive, Schofield, WI 54476; (1) 715-359-0209, extension 11; E-mail: ron.

Will Pope, principal controls engineer with the Sacramento Municipal Utility District, is project manager and technical lead on the upgrade project. Ron Hahn, vice president of business development with North American Phoenix Energy Services, provided project technical and commercial management support.

Managing Obsolescence

The new PLC systems being installed have a life expectancy of 20 to 24 years. To mitigate the planned obsolescence of the PLC hardware, SMUD purchased a significant number of spare parts as part of the original installation. Each governor PLC panel includes a spare parts rack that includes one PLC module of each type in the system. The spare parts rack is constantly powered, and the governor PLC monitors the health of each module. This ensures that if a spare module is needed, the spare is functioning properly.

Significant spares of other critical control system components also were purchased. These include panel power supplies, pressure transmitters, level transmitters, speed pickups, Ethernet switches, position transducers, linear variable differential transformers, turbine shutoff valve directional control valves, and governor control valves. As spare parts are exhausted and existing devices phased out, new devices will be purchased and phased in. The panels have been designed so that the PLC racks and modules can easily be replaced while affecting a minimum amount of wiring.

To access this Article, go to: