To standardize lubrication practices for hundreds of pieces of equipment at its four hydroelectric projects, Grant County Public Utility District established a lubricant maintenance program. Through the program, the PUD lowered the frequency of oil changes, reduced man-hours spent on lubrication, and improved the quality of lubricants in service.
By Thomas W. Brittain
The four hydro projects Grant County Public Utility District (PUD) owns and/or operates contain hundreds of lubricated pieces of equipment. The projects are 1,038-MW Wanapum, 907-MW Priest Rapids, 9.4-MW Quincy Chute, and 6.5-MW Potholes East Canal. At these facilities, the PUD is working to improve the quality of lubricants while reducing costs associated with lubrication.
In 1987, the PUD’s lubrication practices were inconsistent. Nobody was in charge of lubrication practices, so personnel at each plant used different lubricants from different suppliers. Some equipment was lubricated too much, and some not enough. Oil sampling was performed sporadically, and results from samples typically were not used to make decisions about lubrication or equipment maintenance. No kidney loop filtering of lubricants was performed. In addition, most oil changes were performed based on elapsed time, regardless of the oil condition.
In 2002, Grant County PUD undertook a concerted effort to improve lubricant practices at its hydro facilities. As a result, oil changes now are based on results from testing of oil samples or operating hours on the oil. Many lubricant problems are fixed by performing kidney filtering of the oil while it is in the equipment. Finally, improved breather filters keep dirt out of equipment so that we do not need to remove it later, at higher costs.
However, arriving at this point required a significant transition within the PUD. This transition took lots of work but has provided good results.
Grant County PUD’s lubricant history
Developments in the field of tribology have come fast over the past 20 years. The quality of lubrication products and supporting equipment are constantly improving. However, Grant County PUD’s lubrication practices had not kept up with the advances. In 1987, our lube practices were the same as they had been since the plants were built. We performed regular lubricant changes, and we filtered our turbine oil every four or five years. This program seemed to be effective, but a closer look would have shown otherwise.
The Priest Rapids facility contains about 135,000 gallons of turbine oil, and Wanapum about 100,000 gallons. This amount of oil is an investment worth maintaining. The storage and cleaning equipment for this oil was typical for the era. The “dirty” storage tanks held all the oil drained from a single unit, and the “clean” tanks contained a filtered charge of extra oil to put back into a unit after an overhaul. A centrifuge was used to remove water and dirt from the oil. We tried to overhaul each of our ten turbine-generating units at each plant every four years. We had become proficient at draining, centrifuge filtering, and refilling oil.
For other equipment at the facilities, lubricant maintenance primarily involved changing oil. Oil changes for most of the equipment in this category – such as gearboxes, small hydraulic systems, and engines – were scheduled as part of the PUD’s periodic maintenance program, mostly on an annual basis.
One commonality between our two main plants was an elementary oil analysis program. In the 1980s, the turbine oil supplier sent free sampling kits for the oil in the governors and generator thrust bearings. Tests performed included viscosity, oxidation, water content, and spectrographic measurement of contaminants, wear metals, and additive package. However, the free testing did not include particle counting.
We never took action based on these tests. There were several reasons why. First, the results did not provide the information necessary to determine a course of action. Without a particle count, we did not know the level of microscopic debris in the oil. Second, while component wear did show up in the sample results as increasing levels of iron, copper, lead, and tin, these elevated levels were present in all the turbine oil. Because we moved the oil between units, the oil in all units developed a homogeneous level of background wear metals. This masked any sudden rise that would have betrayed a problem. Also, because the contaminant level rose slowly, and a succession of engineers looked at the samples over the years, we never saw this wear as a sign of a problem. Oil sample data was reviewed annually, and we did not compile enough data to establish a solid trend. We had oil sampling without effective analysis.
A few simple things could have helped this situation. Paying for particle counts would have told us the level of particulate contamination in the oil. We needed to set alarm levels based on industry and internal standards to tell us when action was needed. We also needed better filtration equipment for cleaning dirty oil. Our centrifuge equipment was labor-intensive, requiring almost constant monitoring during operation. Finally, trending sample results over the years would have turned data into useful information and revealed slowly degrading conditions.
One other important point: Some hydro plants do not have staff with expertise in lubricants. Decisions often are left to the mechanics, foremen, and supervisors, or engineered on a case-by-case basis. Yet lubricant maintenance is the most frequent and often the most important preventive maintenance activity performed on rotating equipment. Grant County PUD struggled with a lack of expertise and decentralized decision-making when we needed consistent, high-quality decisions and actions.
Why Grant County PUD established a lubricant program
While our lack of a modern lubrication program did not cripple us, unnecessary oil changes are expensive and do not extend equipment life. We had to overhaul our governors every four years. Governor components should have lasted much longer, but we thought this was the best we could do. Fine abrasive material too small to see was wearing out close-tolerance components that should have lasted longer. In addition, we missed chances to prevent failures in equipment. For example, a spillway gate gear reducer had a manufacturing error that caused gear misalignment. Resulting high stresses wore and eventually fatigue-failed a gear. Oil samples would have detected symptoms well before the failure. Air compressor problems showed up as failure of the compressor instead of oil sample action limits being exceeded.
As a result of excessive operational problems at the facilities, starting in 1985 we performed an engineering review of the governor maintenance program. The review by PUD engineers revealed that the strainer filters supplied by the original equipment manufacturer (OEM) were inadequate for protecting the rotating pilot valve and gate limit valve. Along with an overhaul of every governor, the strainers were upgraded at that time to modern Hilco filter equipment from Hilliard Corporation. The dirty oil in the units limited us to using filter elements rated for particles 40 microns or larger. Otherwise, we would have been changing the filter elements weekly. Even with this limitation, performance was much better than the 100+ micron strainers they replaced, reducing wear rates and problems in these critical areas. Pilot valve wear slowed, and control problems diminished but were not eliminated.
The free oil sampling of the generating units also came under scrutiny as we realized that we needed to know how dirty the oil was. About 1992, we discontinued use of the oil company’s sampling service, instead choosing the faster turnaround and more complete testing provided by our local lab.
Today, particle counting is included in the free testing supplied by our turbine oil supplier, and we occasionally send them samples. However, we gladly pay to have our independent lab perform all normal testing.
Chronic equipment problems sometimes prompted us to switch lubricant brands or types. One example is gearboxes for five fish ladder attraction water pumps at Priest Rapids. These connect a 300-horsepower motor to a low-speed, low-head water pump. The gearboxes required twice yearly oil changes and still wore out too many bearings and gears. The plant facility engineer and mechanic foreman worked with a lubricant supplier to try a specialty gear lube, at the same time cutting back to annual oil changes. Wear rates and temperatures dropped to a more acceptable level. However, we still had wear, so more work needed to be done.
Powerhouse bridge crane gearbox oil samples were first taken in 1992 to determine if we could extend the length of time between oil changes, which were then performed annually. The samples came back very dirty, but everything else was normal. This exposed the major problems facing us at that time. First, we did not have alarm and action levels and did not know how to establish them. Second, we did not have the equipment or the expertise to filter gear lubricants.
For the rest of the equipment, there was little oil sampling done. Oil changes were performed based on elapsed time, often annually. Given the conservative recommendations from equipment manufacturers, this was normal practice. In some cases, annual lube changes did not even satisfy the level of maintenance called for by operations and maintenance manuals for some equipment.
Work to establish the program
In the late 1990s, the PUD performed work process reviews of all major business functions. Results of these reviews indicated that our hydro facilities needed a reliability centered maintenance (RCM) program. The goal was to ensure that the right maintenance was being performed on the right equipment at the right time. This RCM review started moving us away from changing lubricants based on time to performing lubricant maintenance based on sample results.
This move to oil sampling could have created problems because we did not have the knowledge or tools to take action based on the sample results. Fortunately, these problems came slowly.
In 2002, we hired an experienced consultant, Rocky Thompson, to assist us in implementing the results of the RCM program. Rocky was especially strong in implementing a lubrication program. While the RCM analysis continued, he started to take action based on bad oil sample results.
Some of the actions included installing better breather-filters on gearboxes and hydraulic systems. This hole in our lubricant practices had never been identified. If a gearbox had an open pipe fitting as a breather when it was installed, that was still the situation 40 years later. Hydraulic tanks and gearboxes have been checked to determine if improving the breather element would keep dirt or water out of the equipment. Much of our outside equipment has been retrofitted with desiccant breathers to filter the dirt while also preventing water vapor from condensing inside. Most inside equipment has been equipped with cheaper air breather filters that capture dirt.
Cleaning equipment internals was the next challenge. Past attempts to clean equipment containing oil with high particle counts had met with inconsistent results. The usual way to deal with this situation had been to change the oil. However, oil samples taken after the change often showed that the new oil had high levels of dirt and wear metals contamination. The oil change was not cleaning the equipment, which then contaminated the new oil.
Rocky recommended two models of portable filters that would help deal with this problem. They are simple to operate and light enough to carry up on a crane. One model, the Mighty Might from Norman Filter Company, pumps 4 gallons per minute of turbine or hydraulic oil through two filter elements in series. It cost about $1,000. The other model, the High Viscosity Mighty Might from Norman, pumps 1 gallon per minute of fluid through the same types of elements, and it works on gear lubes quite well. Cost for this model is about $2,000. The elements are typically a 10- or 25-micron element, followed by a 3-micron element.
For very dirty equipment (such as outdoor gearboxes and hydraulic systems that had been neglected), we started draining, flushing, and refilling the equipment, then filtering the oil. If the equipment contained newer oil that was dirty, the procedure was modified to filter the oil rather than draining it. Whenever possible, we operated the equipment to stir up the oil during the filtering process. This moved the dirt in the equipment into the oil so it could be filtered out. Otherwise, later operation would mix the dirt into the cleaned oil.
We immediately hit a snag. Cleaning dirty equipment often takes more man-hours than a simple oil change, and we have more work than people. When Rocky proposed that a full cleanup be performed on most of the 12 spillway gate gearboxes, both the foremen and planners objected. As a compromise, oil changes were performed on the cleaner gearboxes, and only a couple of the dirtiest gearboxes received the full treatment. As a result, the dirtiest gearboxes became the cleanest gearboxes, and the other gearboxes did not improve much. These results convinced everyone that the best way for us to get the dirt out is to perform the full treatment.
This leads to another major concern we have encountered since starting the oil analysis program: “How clean does the oil need to be?” General industry standards are fine for most equipment that runs constantly. These standards provide oil cleanliness levels based on the equipment in which the oil is operating. For example, high-pressure hydraulic systems require cleaner oil than a slow-speed gearbox. Predictors are available for shortened component life expectancy based on varying levels of water or contaminants in the oil, but these usually assume 24/7 operation. How do we apply these standards to equipment that only runs a few hours per year? How do we apply them to equipment that has very little effect on the production of electricity but may need to operate in blackout situations, or to equipment that is rarely used and exists just to maintain the plant?
Our simple answer to the complex question is: “It depends.” It depends on more than the RCM criticality ranking of the equipment. It also depends on operating regime, environment, system redundancy, experience, and future plans for the equipment. It depends on how much dirty oil will affect the life of the equipment, as well as the replacement cost of the equipment. Cleanliness of the oil directly affects wear rate. But if the oil contamination in a standby gearbox would result in a life of 100 years, filtering is not worthwhile.
We use more than the industry standards or OEM recommendations for setting cleanliness goals. Maintenance is focused on economically keeping the plant operating. Some of the equipment does not need to be in pristine condition. If we can take care of the critical equipment and the equipment that is harmed quickly by bad lubrication, we are on solid ground. For equipment that is less critical and not harmed quickly by dirty oil, we tolerate dirtier oil.
A solution that required patience and persistence was the purchase and installation of kidney loop filters on every turbine-generating unit at Wanapum and Priest Rapids. We are pleased with the final installations that use two series-mounted high-capacity filters rated down to 3 microns. Large element sizes give us the ability to keep oil clean with few filter changes. Installation of piping and valves allows us to also filter thrust bearing oil while the unit is operating. Screw pumps give us quiet operation and long life. This equipment allows us to meet OEM oil cleanliness requirements, extending the life of all oil-lubricated parts in the turbine and governor.
Initial startup of the filters on units with dirty oil required changing elements weekly or more often. We even went to a 25-micron initial element with a 10-micron secondary element to allow filtration of larger particles first. As the oil in the governor, piping, servomotors, and Kaplan turbine hub cleaned up, we have found that the 3-micron elements will last the expected one year between changes. We are reducing planned intrusive maintenance on governors and expect reduced governor wear and problems because of the clean oil.
Getting consistent, representative oil samples is vital so that actions are based on good data. Some equipment can only be sampled via the drain port, which is where the dirtiest oil is found. Sampling oil from this location can result in changing clean oil based on bad samples. Sample port adapters can be installed in the drain connection or the vent port. When properly installed, the sample port’s tube reaches the middle level of the oil reservoir, where the “live” zone has oil that is circulating through the wear surfaces. Oil samples are representative and consistent.
One key program parameter is the frequency of oil samples. Our goal is to sample equipment often enough to catch developing problems, but not more often. We refine sampling rates to fit the equipment and its service. Some equipment has very stable oil quality, while equipment like compressors can develop problems quickly. Problem equipment and critical equipment gets sampled more often than equipment that runs infrequently, has no history of problems, or is not critical to the important plant functions.
Some of the best support has been from mechanics who take the initiative to install sample ports or filter breathers on equipment. Our warehouse stocks the parts to perform this work, and sometimes they are installed while other work is being done. For equipment that will be filtered again in the future, mechanics also have been interested in installing locking quick disconnects. They know this is a cleaner and easier way to connect to the portable kidney loop equipment. Even if we only filter the equipment every year or two, it is worth the effort to install the quick disconnects the first time the equipment is to be filtered.
The best part of our earlier oil sampling endeavors was that we established a good relationship with a local oil analysis laboratory. The lab has always been able to supply good data and help us interpret it. We now know that the lab offers service beyond sending paper reports of samples it has received. Because there is no manual telling us what every oil sample reveals and what to do about it, long conversations with the lab fill in the picture. We know the equipment, and they know oil analysis. Together, we usually can figure out what samples are telling us and how to deal with whatever comes up.
Lab costs are $25 per typical sample, but our costs to take the sample and process the results exceed this. Some samples result in work that cost hundreds or thousands of dollars. However, lack of action can result in failures that cost far more. Good analysis tools are necessary to extract the most meaning from the sample data.
Rocky convinced Grant County PUD management that using an oil analysis database was necessary to reap the most from the investment in the samples. In 2003, we began using the FleetOil database from Dingo Maintenance Systems. This database holds all data from every sample of every piece of equipment. It also provides trend graphs of sample data points for a piece of equipment or a group of like equipment. We can easily determine the worst equipment and observe trends that show the need to take action or the good results of oil changes or filtration. Alarm limits set for every sample parameter automatically alert the analyst. The alarms are set for a grouping of similar equipment, and there is no limit to the number of settings available. If equipment requires strict standards to be enforced, these alarms make it easy to train someone to manage the sampling program.
Future lubricant work planned
In 2006, we took more than 400 oil samples at our four generating facilities. We sampled equipment that varied from 4,000-gallon thrust bearing tubs to 3-gallon gearboxes, from emergency station service diesel engines to mobile crane hydraulic systems. We sampled highly critical equipment and old equipment that will not affect any important functions but is expensive to replace. We used oil samples to initiate work that would not have been performed otherwise, and to skip maintenance that would have wasted money.
To prove the case for on-line filtration, we are sampling the fish attraction water pump gearboxes at Priest Rapids. High iron levels from gear wear, visible evidence of the wear, and the history of premature wear/failure of gears and bearings all led us to request the assistance of the gear manufacturer. The company’s inspection revealed that we lacked proper setup and alignment procedures, and we needed to maintain cleaner oil. We will continue to use oil analysis to monitor the gearboxes as we move forward with new gearing and bearings, try a new lubricant, and install new filtration. Oil analysis will be a key indicator that we are successful, or a warning that we are not.
Because the PUD has a good preventive maintenance program, most of the bankable, provable savings we realize come from not throwing away good oil and the man-hours it takes to perform the oil change. The costs of changing oil in a hydro plant include overhead of the purchasing, warehousing, and supervisory staff and the facilities to house and maintain the maintenance staff. The cost also includes handling and disposal of the used oil.
Our constant goal is to prevent equipment contamination. Keeping the dirt out is cheaper than removing it, but not always completely attainable. Removal of contamination via filtration instead of changing oil is another aim of the program. Overall cost reduction and contamination reduction require a balanced approach using multiple tools and realistic cleanliness standards.
Mr. Brittain may be reached at Grant County Public Utility District, 15655 Wanapum Village Lane S.W., Beverly, WA 99321; (1) 509-754-5088, extension 2508; E-mail: firstname.lastname@example.org.
Thanks to William Stein, product application specialist, Shell Global Solutions (US) Inc., for review and input to this effort.
Tom Brittain, P.E., is a mechanical engineer with Grant County Public Utility District. He is responsible for the lubrication program at the district’s four hydro projects.
Online Lubrication Resources
The Internet has many authoritative sites and resources provided by leaders in the fields of maintenance and lubrication.
– Uptime magazine covers all major predictive maintenance technologies and offers online tutorials, web casts, and subject forums. http://reliabilityweb. com
– The website for Noria, dedicated to lubrication, offers several free publica- tions and training courses, as well as past and present articles. http:// noria.com
– Lubrication Management and Technology is a good free magazine. www.lmtinfo.com
– NAPA sells lots of filters nationwide that cross over with many industrial original equipment manufacturer filters. www.napafilters.com/filterlookup
– The Society of Tribologists and Lubrication Engineers (STLE) appeals to lubricant and additive manufacturers, researchers, and end users. Tribol- ogy and Lubrication Technology magazine is free for STLE members. www.stle.org/ index.cfm
– The U.S. Department of the Interior’s Bureau of Reclamation has excellent lubrication publications aimed at hydroelectric plants that are free on the web. The Bureau has “Facilities, Instructions, Standards, and Techniques Volume 2-4 Lubrication of Powerplant Equipment.” www.usbr.gov/power/ data/fist_pub.html
– The U.S. Army Corps of Engineers has Lubricants and Hydraulic Fluids, EM 1110-2-1424. www.usace.army.mil/publications/eng-manuals/em1110-2- 1424/toc.htm
– The Electric Power Research Institute (EPRI) publishes “Lube Oil Predictive Maintenance Handling, and Quality Assurance Guideline.” This compre- hensive reference is free to members or is available for purchase. www.epri.com
– Lubricant suppliers – such as Shell, Exxon-Mobil, ConocoPhillips, and Chevron/Texaco – have brochures available. www.shell-lubricants.com, www.exxonmobil.com/siteflow/brands/SF_BR_Exxon.asp, http://lubricants. conocophillips.com, or www.chevronlubricants.com/worldwide/ northamerica/na_lubricantsforbiz/default.asp