Seattle City Light spent nine years and nearly $90 million to rehabilitate its 1,050-MW Boundary project on the Pend Oreille River in Washington. By performing much of the work in-house instead of hiring a contractor, the utility estimates it reduced costs by 25 percent, as well as gained valuable staff expertise for use in future rehabs.
By Kevin W. Marshall
Between 1997 and 2006, Seattle City Light spent nearly $90 million to rehabilitate its 1,050-MW Boundary hydroelectric facility. Over the course of this project, most of the major systems and components were modified, overhauled, or replaced. In addition to overhauling all six turbines, Seattle City Light completely reworked the station service electrical system, upgraded the generator protective relays and control systems, improved the water supply and cooling systems, and performed civil improvements to tunnels and roadways.
By performing the engineering, project management, and much of the labor in house, Seattle City Light estimates it saved $30 million to $40 million. As a result of the overhaul, the utility reduced operations and maintenance requirements for the facility and increased overall plant efficiency by about 2 percent.
Background on Boundary
Boundary is the largest of the six hydro projects Seattle City Light owns and operates. This project operates under a 50-year Federal Energy Regulatory Commission (FERC) license issued in 1961. Boundary is in the midst of FERC’s integrated licensing process (ILP) to obtain a new operating license for the facility.
The underground powerhouse was excavated out of a cliff along the bank of the Pend Oreille River. The original powerhouse, completed in 1967, contained four 150-MW turbines. An expansion completed in 1987 added two 210-MW turbines. After rewinds performed over the years, capacity of the first four turbines increased to about 157.5 MW.
During a good water year, the Boundary facility generates 50 percent of Seattle City Light’s electricity, almost 4 million megawatt-hours. Boundary is operated in a load-following mode that shapes available water to deliver power during peak-load hours, with a total plant capacity of about 1,050 MW.
Deciding to rehab the project
Over its 30 years of operation before the rehab, Boundary Dam experienced several challenges. Within a year after commercial operation began in 1967, one turbine was nearly destroyed when a wicket gate rotated into the runner. In 1975, a hanger bar fracture dropped a 110-ton head gate to an operating turbine. Before the hanger bars could be replaced, two more failed. In 1987, when the two new units were added in the expansion, the generators vibrated badly. The vibration resulted in damage to the structural concrete supporting the generators, with the concrete cracking and spalling in several places. Because these problems hindered power production, Seattle City Light dealt with them quickly.
However, other minor problems were not addressed because of a lack of available labor. For example, the cast steel runners on the four older turbines suffered from persistent blade cracking. Then, the wicket gates on the two newer turbines became difficult to operate because of bushing wear. Creativity was required to keep the powerhouse’s 480-volt (V) distribution system operating. The manufacturer no longer supported much of the system’s equipment, which was obsolete. Control and communication systems, vital because power dispatching is accomplished from Seattle (more than 300 miles to the west), were becoming unreliable. Cooling system problems were affecting generator temperatures during summer months when river temperatures are high. As these problems accumulated, they too began to affect power production.
In gearing up for the rehab, Seattle City Light hired a consultant to develop a condition report. The idea was to get an objective view of what needed to be fixed, as well as how much it would cost. The report was comprehensive but not well organized. At first glance, it was an overwhelming list. Hiring the consultant to run the rehab seemed a sensible decision. The total cost estimate was $90 million, not including the turbine overhauls, with $30 million going to the consultant for engineering and administrative charges.
However, when Seattle City Light took a closer look at the condition report, things did not appear so intimidating. The bulk of the projects related to mechanical work on the turbines, the powerhouse electrical and communication system, or civil work on the access roads and tunnels. Seattle City Light decided to investigate the option of performing the work using utility personnel.
Prioritizing the rehab process
Seattle City Light management supported the Boundary rehab but wanted assurance that money would be well spent. A consultant from the Tennessee Valley Authority (TVA) was hired to provide guidance and report to the superintendent, and a Seattle City Light engineer with a talent for budgeting and labor planning was promoted into a project management director position. Then, the utility formed a committee of engineers and project managers from the utility’s mechanical, electrical, and civil groups to assign priorities, estimate budgets, and produce progress reports. This team met weekly during the course of the project. Coordination among the groups was crucial because the value of generation lost due to poor planning could have been considerable.
The project evaluation process included an economic justification step that forced the estimation of the project’s net present value. As a result, many of the individual projects were eliminated or deferred. Work like installation of digital controls and replacement of functioning equipment could wait. The station service work, turbine overhauls, and protective relay upgrades that were critical to making power had to come first.
Station service work
Electrical service to the Boundary powerhouse is backfed from the switchyard, along a 4,000-foot path that includes a 300-foot vertical cliff face. Along this, the two main 13.8-kilovolt (kV) feeders were direct buried in places and were elsewhere routed right next to each other. Their insulation was degrading, and they were susceptible to single point failure. This meant a problem with one conductor could affect the other, leading to complete loss of electrical service. Other problems were:
– The main 480-V switchgear had to be replaced;
– An underground vault, the vital power source for flood control equipment at the dam, was shifting and too small to be improved or to accept the output of an additional emergency generator; and
– New controls were needed for the sluice gates and head gates.
The specification covering the station service work had to describe the sequence of events for decommissioning the old equipment and putting new equipment into service, as well as all the steps in between. Time spent developing this document proved invaluable. It forced Seattle City Light personnel to anticipate all the problems that could arise and decide when equipment should be purchased and installed to avoid affecting other plant operations.
Power City of Spokane, Wash., the electrical contractor for the station service work, was on site for nearly two years, working closely with the Boundary electrical staff. Much of the work requiring equipment outages was performed at night. Seattle City Light handed over control of the project to the on-site electrical supervisor, as logistics problems were more easily solved on site. An attitude of ownership resulted, contributing to the successful completion of the project without any change orders – a rare occurrence on a project of this scale.
Station service work included:
– New feeders from the switchyard to the powerhouse, routed separately to avoid single point failure;
– A new building constructed at the forebay to house the new switchgear and diesel generator;
– Obsolete powerhouse 480-volt distribution switchgear was replaced; and
– Numerous motor and other equipment controls were replaced.
The major station service work lasted two years. However, the final commissioning, along with redesign and installation of the smaller pieces of equipment (such as gate controls and governor work), took plant staff longer to complete. Costs for the station service work amounted to $10 million. About $2 million went to the contractor, and the remainder was used for equipment purchase and Seattle City Light labor.
Significant site improvements were needed for the rehab project to move forward. The Boundary access roads are cut into steep banks and suffered from frequent mud and rock slides. Oil and fuel storage tanks and pumping areas were inadequate and in poor condition. The powerhouse potable water system was being used in applications it was not designed for. A new water source was needed because the creek had become unreliable. Finally, there was not enough covered area for storage of parts and staging of supplies.
The civil work was divided into several phases. The first phase involved construction of a shipping and receiving building for staging parts and components. The design included 1,000 square feet of office space, 2,000 square feet of equipment storage, and an 800-square-foot wash bay with a recycling sump.
The next phase included road work and other site improvements. The access road into Boundary is a 12-mile-long, two-lane road, going steeply downhill for the last mile. Goodfellow Brothers of Wenatchee, Wash., installed a 1,000-foot-long wall of ecology blocks along each side of the road at its steepest points, to reduce the possibility of blockage due to slides. New culverts were installed to more adequately handle storm water drainage.
Rock falls are frequent at Boundary because the powerhouse is located at the base of a near vertical rock wall fronting the river. To protect guests and visitors from falling rock, a 120-foot tunnel-portal extension was built at the entrance to the underground plant.
Spill gates at the Boundary project were refurbished as part of a nine-year rehabilitation to ensure continued reliable operation.
Another trouble area is the sheer cliff face above the transformer bays. Falling rocks could damage the conductors where they exit the main transformers before traveling to the switchyard. A recent event severed three 240-kV conductors and damaged a primary load-interrupter, resulting in a forced outage of a large machine for several days during high flow operation. This area has been mapped and will be monitored as Seattle City Light investigates possible solutions to the exposed face and the risk of more damage to the conductors.
Overall, Seattle City Light spent about $6 million on access road and service area improvements. This work resulted in the savings of many labor hours previously required for digging out after extreme weather events, along with time lost due to mobility restrictions.
By 1995, plans were in the works to upgrade the original four turbines with new runners to address problems with repeated blade end cracking. But while planning was still in progress, inspections of the two newer turbines showed they needed attention as well. Bushing wear was causing wicket gate misalignment and contact with the facing plates, making operation unpredictable. It was imperative that while one unit was being overhauled, the remaining five available units function reliably. Overhauls of the older units would have to wait until the newer units could be repaired.
At Boundary, all six turbines are needed when river flow is high (April through June), leaving nine months to complete an overhaul. Seattle City Light had performed many turbine overhauls over the years, but this was going to be the biggest. Committing virtually all the utility’s maintenance people to overhauls at Boundary was out of the question. And contracting out the entire job would be very expensive. There are a limited number of contractors capable of performing the work, and they all wanted top dollar to commit a crew to Boundary for nine months a year for six years.
Seattle City Light did not have two necessary items for the overhauls: tooling for machine work and a crew. The two tooling fixtures needed were a boring bar for cutting the circular turbine surfaces (seal rings and facing plates) and a large vertical mill, or turntable, for machine work on the removable components (head cover and operating ring).
The boring bar is a big vertical tube, installed in the center of the disassembled turbine, rotated by a hydraulic motor. It has a horizontal arm with a tool head on the end that does the cutting. For the turbines at Boundary, the boring bar had to be designed to cut surfaces up to 23 feet in diameter, and the arms were made from 39-inch-diameter seamless thick wall tubing.
The turntable is a large bed that rotates the head cover and the operating ring while a stationary tool head cuts them. Because the head cover is almost 24 feet in diameter and weighs more than 40,000 pounds, the turntable must be very well built to be capable of the intricate work involved.
Because they cannot easily be adapted to other uses, boring bars and turntables of the size needed for Boundary are not commonly available. Seattle City Light had to build them. The bearings, hydraulic motors, and tooling slides were purchased, but the remainder of the pieces were designed and fabricated by Seattle City Light personnel. The approximate $500,000 cost was justified because the same tooling was used on all six turbines. In comparison, quotes for contracting out the required machine work exceeded $350,000 per turbine.
The next obstacle to overcome was finding a crew. Because of the proposed nine-month schedule, the crew did not have to be very large. However, the utility expected that hiring experienced craftsmen (millwrights and machinists) willing to commit to six years of work would be difficult. Luckily, there were a few retirees, as well as others with previous experience who were in between jobs but wanted to remain in the area. About a third of the crew were personnel with extensive experience, another third were young guys with little experience, and the rest were in between. They worked under a Seattle City Light crew chief and turned out to be a fine crew.
The turbine overhauls were completed one per year, beginning in 1997 and ending in 2003. The two newer turbines were completely disassembled and refurbished but did not need new runners. On the four older units, in addition to the normal refurbishment work, the runners were replaced. This replacement required modifications to the embedded upper draft tube sections. Working on the two newer turbines first was beneficial because when the more complicated overhauls on the older turbines came up, the crew knew the work. Several project managers and engineers from Seattle City Light were involved over the years, but essentially the same 15-person crew worked on all six overhauls at Boundary.
While the two newer units were being rehabilitated, the process of buying new runners for the four older units was going forward. Noell GmbH (a subsidiary of Babcock-Borzig) of Wurzburg, Germany, was the winning bidder, decided by competitive model testing.
During post-overhaul testing performed by Seattle City Light personnel, three of these four units exhibited weighted average efficiencies of 92 percent. Although this was an improvement of 2 percent over the original runners, the contract with Noell guaranteed 94 percent. The situation was complicated by the fact that Noell’s parent company entered bankruptcy in Germany before the overhauls were completed. Under conditions of the original specification, contract penalties could only be assessed after completion of a fully compliant efficiency test administered by a third party.
The dispute was eventually resolved in favor of Seattle City Light when Noell was forced to negotiate for release of the contract bond. In return for this release, Noell paid the full contract penalty. Although the penalty amount will not completely cover the estimated value of lost generation over the projected 20-year life of the units, this is an example of how specific and defensible contract language can be administered when performance metrics are not realized.
Welders installed a modified draft tube section for the units at the 1,050-MW Boundary project to accommodate new runners.
After these negotiations, efficiency testing performed by EPFL (Ecole Polytechnic Federale de Lausanne) in Switzerland confirmed that Seattle City Light tests showing an efficiency of 92 percent were accurate. Seattle City Light is unsure why efficiencies of more than 94 percent attained during model testing could not be replicated with the full-scale runners.
The average cost for overhauls was $2.9 million for the two newer turbines and $4.3 million for the four older turbines. Seattle City Light estimates it achieved at least a 25 percent cost reduction by doing the work with utility personnel versus hiring a contractor.
There were many miscellaneous projects performed on auxiliary plant systems as part of the Boundary rehabilitation. The cooling system plumbing and air coolers on the four older units were replaced to address problems with generator overheating in the summer. Control systems were replaced on the sump pumps, head gates, and sluice gates. The spill gates and head gates were refurbished. In addition, numerous underwater inspections were conducted at the forebay and tailrace to determine the condition of gate sealing surfaces and other underwater structures.
Some work continues at Boundary (such as replacement of the network-based control systems, governor, and elevator), but the rehab project is considered complete. Although a consultant with more resources may have been able to complete the work in a shorter period of time, Seattle City Light saved $30 million to $40 million and gained a wealth of knowledge by maintaining internal control of this project.
As we look forward to years of reliable operation of the Boundary project, Seattle City Light is converting to a maintenance management system using Maximo software from MRO Software. This system is expected to allow more detailed tracking and assignment of maintenance work.
During the Boundary rehab, Seattle City Light learned two valuable lessons:
– Having a well-defined hierarchy and transparent business processes for project justification and evaluation can eliminate possible controversies regarding priorities and the sequence in which projects are performed.
– Hiring the right personnel is key. Both the TVA management consultant hired as operations director and the staff mechanical engineer promoted into a project coordinator position were well qualified. Their competence gave management confidence that things were under control.
Mr. Marshall may be reached at Seattle City Light, 700 5th Avenue, P.O. Box 34023, Seattle, WA 98124-4023; (1) 206-615-0972; E-mail: kevin.marshall@ seattle.gov.
Kevin Marshall, P.E., senior mechanical engineer with Seattle City Light’s generation engineering division, was the project engineer/manager for the last two Boundary turbine overhauls.