Web of PdM systems catches problems early

Most top plants have predictive maintenance systems (PdM) systems, because they have proved to boost plant availability and safety, pay for themselves when they prevent even a single unplanned outage, and eliminate middle-of-the-night phone calls. Now one utility has taken the next step: It has not only installed PdM systems at six of its plants but also networked those systems. The network leverages—in real time—the expertise of the maintenance analysts using the systems.

By Ricky Lee Badger, Predictive Maintenance Manager, Westar Energy

A year or two ago, Topcka-based Westar Energy decided to become proactive rather than reactive about maintenance by committing lo install predictive maintenance (PdM) systems at all the generating plants in its portfolio. At the same time, the utility realized that the systems would he even more beneficial to the enterprise if they were networked. A PdM network would create a virtual team of analysts and engineers who could collaborate on a solution to a problem at any of the plants.

To get the blessing of corporate executives, Westar plant engineers and managers had lo define the scope of the networking project involving the six plants at which PdM systems would he installed and configured to share data (Figure 1).

Westar Energy's PdM network locations
Westar's PdM network interconnects six power plants and covers vibration analysis, oil analysis, thermography, and motor current analysis. Engineers and technicians can view data collected at any plant at any time, increasing their collective problem-solving capability. Source: Westar Energy

Next, it had to be determined what kind of data would be shared. In the end the following types of information were selected: monthly monitoring data, trend data, and analytical data related lo equipment performance and degradation. The PdM system developed by Westar uses a wide variety of technologies, including vibration analysis, oil analysis, motor current analysis, and thermography. The result of the definition effort was the architecture for a network of PdM systems depicted in Figure 2. The project to install the systems and network them has been completed and is up and running after about 18 months of concerted effort by the plants. Collectively, the systems in the six plants monitor vibration on 1.030 machines once a month, take thermography readings on 4,882 components annually, and do monthly oil analysis on 260 machines.

Westar's PdM network configurationThe network gives engineers, maintenance workers, and operating personnel read-only access on read-only workstations but gives full access rights to the plant's PdM manager and maintenance analysts. Source: Westar Energy

The three-liered wide-area network (WAN) gives the overall system's internal customers—personnel in the six plants' operations, engineering, and maintenance departments—read-only access to the data in any PdM system. But it gives analysis in the plants full access to the PdM network server, which allows them to maximize both their productivity (by sharing experiences and solving problems with their peers virtually face-to-face) and their skills (by attending virtual training classes on technical topics such as vibration analysis).

Preparing for the systems

To kick off installation of the networking project, Westar created a six-step procedure for preparing the equipment in the six plants lo be monitored by the new PdM systems (Figure 3). Each of the following steps was executed as pan of a PdM pro-gram at each of the six plants.

Step 1: began with compiling a list of all the equipment identified by engineering and plant management as requiring PdM. The list was then sorted into groups of equipment of similar type, location, and data collection-time requirements.

Step 2: was the creation of an engineering database capable of supporting each predictive technology. It was initially populated with each piece of equipment's nomenclature and ID number, data from its initial field walk-down, and its O&M history Also added to the database were fault frequencies for bells, couplings, and bearings; alarm limit sets; analysis parameter sets; rotor and stator data collected from motor vendors; and data needed for thermography and oil analysis in short, any information that might facilitate problem-solving and decision-making.

Step 3: involved another field walk-down of each piece of equipment to be monitored, during which special attention was paid to locational impacts on PdM data collection. Safety and accessibility were major considerations. For example, during the walk-downs, optimal vibration-monitoring points were identified and marked on each piece of rotating machinery.

Step 4: entailed the building of each system's hierarchical database. The hierarchy chosen goes as follows, from top lo bottom: energy center, station, equipment, and data collection point. A station is defined as any typically self-contained, stand-alone unit within a plant (for example a building, floor, or process). The term "equipment" refers either to a mechanical drive like a motor, a working unit such as a pump or fan, or any electrical support component. A data collection point is any point on a piece of equipment at which measurements or samples are taken. At this step in the process, procedures were developed to control how and by whom PdM data were going to be collected and who would be responsible for their analysis.

Step 5: was the selection of both parameter bands and alarm limits. Seven analysis parameter bands were created for analysts to use for monitoring, tracking, and trending purposes and to facilitate comparisons of levels within each band. Each parameter hand forms a boundary limit around specific frequencies of interest. Specific alarm limits for certain types of machines were set as a function of the machine's rotating speed. The alarm limit sets are based on ISO 10816-1 and were placed around each parameter band. If any parameter band exceeds a defined limit, an alarm flag is set for that measurement point and machine.

Step 6: looked ahead lo making each plant's PdM system work with Westar's computerized maintenance monitoring sys-tem from Synergen Inc., Walnut Creek, Calif. At Westar, this system is called the Maintenance Work Control System, and it supports periodic monitoring and analysis. It allows a data technician or analyst to per-form routine monitoring as outlined by the equipment task list generated by the Synergen system. During this step, the data from the Synergen system was uploaded into the PdM software program for analysis. After uploading, the data were run through an exception analysis program, which com-pared the mass data with the parameter bands and alarm limits for the Westar fleet set up in step 5. Each point that the program listed as an exception was determined to be a point that needed further analysis. This process reduced the number of points that the analyst had lo analyze. If the analyst confirmed a problem, Westar's Engineering and Operations Department was notified via a Vibration, Oil, or Thermography Report. A work request was then generated in the Maintenance Work Control System.

Structuring the network
After planning for the six separate PdM systems, it was time to figure out how lo network them into an enterprise wide information system with two levels of access.

Step 1 set the stage for the networking part of the project by clarifying the strategic purpose that sharing PdM data among plants would serve. Essentially, its objective was a fuller understanding and delineation of the benefits that the network would bring to Westar's internal customers. The most pertinent question asked during this step was, "Which network features and characteristics would create the most value?" Among the most important answers to (hat question was, "The overall system should eliminate, as much as possible, any bureaucracy between the customers and the individual plant PdM groups." Giving everyone at Westar read-only access to the PdM data of all six plants served (hat purpose.

Step 2 began the actual organization of the network, working backward from its customers. During this step, the following basic questions were asked;

• What people, skills, information, training, and equipment arc required to meet the needs of each customer?

• What is the minimum number of analysts we need for each front-line group?

• What special expertise do we need on the back-line support group to support the groups on the front line?

After collecting responses, it was decided to move decision-making responsibility to the front line of each PdM group and give customers the equipment and training needed to let them access and trend their own equipment data.
Step 3's objective was to create a working environment for the six individual plant PdM groups in which analysts could be proactive, creative, and accountable as a self-managed part of the network. Among the information, limits, and expectations provided to each analyst group were a list of customers, quantitative and qualitative metrics of accountability and customer satisfaction, and a solid understanding of how productivity would be measured.

Step 4, as you might expect, was the most complicated and important. Its objective was to create the networked information system that would interconnect the six PdM groups. The first part of this task was straightforward. It entailed connecting the six systems to an existing wide-area network and loading the PdM software onto a new, dedicated server.
Next came one of the most important stages of the network definition ---- selling up the two access levels. Analysts in the six plants were given full access rights to the network. Customers across the fleet were provided with read-only access to the PdM server. In essence, this marked the birth of the networked PdM information system as a 24/7 resource. It has five scats for analysts, whose access rights allow them to make changes to the hierarchical database and upload and download data collection rights. The system also has 500 seals for users—work control planners, engineers, managers, operational personnel, and anyone else outside the PdM groups who could con-tribute to the PdM effort with read-only.

Supporting resources
No project can succeed on its own without adequate support in the form or" tools and trained personnel. Once the project was rolled out, Westar1 s top management provided those resources immediately. Two PdM professionals were hired and charged with building an appropriate-size stall. Staffing levels were determined by performing a time study at each plant. The PdM expert walked each plant and recorded the time needed to take vibration, thermography, and oil samples, as well as the time required to analyze them.

The result of this effort is that, today one oil analyst and one thermographer cover all the plants, whereas the number of vibration analysts at each plant depends on its size. The small gas-fired plants have one, but the larger Jeffrey Energy Center has three. The leader of the PdM crew is one of the working managers and handles vibration readings at the smaller Tccumseh Energy Center. At the smaller plants, the vibration analyst does double duly by collecting oil samples and sending them to Westar's central oil lab for analysis.

Expecting people (o do their jobs without the right equipment is an exercise in futility. So Westar Energy outfitted predictive maintenance crews with a state-of-the-art Flir infrared camera ($56,000) and Kmerson Process Management/CSI 2I20A dual-channel vibration analyzers ($26,000 apiece). Data is uploaded from each plant into the RBMware that integrates data from the analyzers and converts vibration, alignment, balance, oil analysis, motor diagnostics, and IR thermography data into a formal acceptable by the WAN. Live updates of machine condition can he viewed by anyone on the WAN

The payoffs
The networked PdM system has done wonders for Westar's bottom line. That's no surprise, because identifying a key piece of equipment that is about to fail and repairing or replacing it saves big bucks these days. Missing a single opportunity lo sell power in a high-demand market can cost a plant tens of thousands of dollars in lost revenue— enough to pay for all the PdM equipment you'll ever need.
Consider, for example, the following case studies of problems at Westar Energy plants that the new PdM system has helped solve.

Vibration analyzer payoff. An auxiliary cooling water pump on Lawrence unit 5, a 500-MW coal plant, had been experiencing periodic motor failures for 18 months. Each trip caused a unit derate, because the plant needs both pumps running to achieve full load during the summer months. Vibration on the offending pump peaked as high as 0.35 in/sec. The PdM crew began by checking alignment and attachment points—the normal cause of such problems. But then they used their new CSI 2120A vibration analyzer to collect cross-channel phase readings. The analyzer quickly pointed the team to an off-center bored coupling that was to blame. Since the coupling was replaced, the cooling water pump has operated at 0.03 in/sec, well within acceptable limits.

Exhaust fan failure averted. At the Tccumseh Energy Center, the coal feed conveyor system feeds the 85-MW unit 7 and the 155-MW unit 8. Routine readings indicated that the crusher house exhaust fan was running slightly high at 0.14 in/sec. The vibration analyzer's waveform capability showed tremendous energy release, as high as 22 Gs. This level of energy is more than enough to rapidly degrade the exhaust fan and cause a catastrophic failure or, at best, shear the shaft. The root cause was rapidly identified as improper clearances in the roller taper bearings. The bearings were replaced, clearances were properly reset and the vibration readings dropped to a comfortable 0.08 in/sec and 3 Gs. The les-son learned here was lhat the problem would not have been detected if only a velocity probe had been used.

Sawy diagnosis. At the Jeffrey Energy Center, a tail pulley bearing in the coal-handling system underscored the difficulty of measuring vibration in slow-turning (<100 rpm) equipment by velocity probes or accelerometers. The alarm readings were a minisculc 0.004 in/sec at the 59-rpm turning speed of the shaft. By using its 500-Hz high-pass filter, the measuring instrument can discard useless data at lower than that frequency, leaving a spectrum of ultrasonic frequencies for analysis. At this point, the PdM system was put into play. By matching bearing response data in its database lo the recorded fault frequencies, the system diagnosed the problem as an outer race defect.

Motor replacement. The Lawrence Energy Center had an ash pump that often tripped its breaker, forcing an operator to make a long walk downstairs to reset the motor breaker. The pump vibration readings were within spec, but a frequency analyzer showed rotor bar pass frequencies. Alarms were set at 0.06 in/sec and trended. When the vibration readings jumped to 0.15 in/sec with side band frequencies of twice the line frequency, the PdM system helped diagnose the problem as a textbook case of a broken or loose rotor bar. The measured energy release also jumped to 17 Gs, more than enough to shear the shaft or cause an internal catastrophic failure. The motor was immediately replaced, after which readings dropped to 0.02 in/sec and 3 Gs—well within normal operating bands.

Oil analysis. The Jeffrey Energy Center collects oil samples every month and looks for ferrous and nonferrous particles and water in the oil. Although the plant lacks a full-blown oil analysis lab, management did have the foresight to spend $.10,000 for a CSI 5200 oil analyzer to do oil screenings. In the past, the solution would have called for operators to collect samples and send them off to an outside lab—a service for which Westar Energy paid about $62,000/year. Doing the work in-house meant that the lab results could be entered into the network database, where they became available on the network almost immediately. When water was found in a lube-oil sample taken from the turbine-driven boiler feed pump during a regular monthly inspection, maintenance crews were able to spring into immediate action. (Figure 4, page 28). It wasn't long until a turbine-side seal was found to be leaking water into the lubrication system. The seal was immediately fixed, and an in-house portable filtration device removed the rest of the water from the lube system (Figure 5, page 28).

Thermographic analysis: Westar Energy's twice-annual thermographic regimen keeps a technician busy year-round. During a routine inspection of unit 5 at the Lawrence Energy Center, the 4,160-V to 480-V step-down transformer, which powers the unit's cooling lower, showed an 85F temperature rise on the enclosure (Figure 6). Removing the enclosure revealed power wiring just beginning to sizzle and on the verge of catastrophic failure that would have shut down the entire 400-MW plant. At this plant, the thermographer's favorite instrument is a Thermocam Reporter 2000 Pro. Using it, he performs a boiler scan at Lawrence and other Westar plants once a year, looking for refractory hot spots that need repair or additional insulation for personnel safety (Figure 7).

Note:

Following article was published in POWER magazine in its May2003 edition: