Software ranges from shrink-wrapped products available “off the shelf” to custom corporate implementations of enterprise systems that require sessions with shrinks to keep everyone sane. Regardless of its complexity, every piece of software a plant uses, or interfaces to, poses critical issues that require life-cycle management. Although functionality has always been the chief specification for software, plants must pay far more attention to long-term quality issues. These two concerns are often at odds with each other.

A recent survey of power plant owner/operators revealed that “version control” is one of the most prevalent issues with software applications—along with having the right team in place to manage the software after the vendor’s implementation team has left. (See “Integrated software platform eludes many owners/operators” in POWER, September 2007.) In fact, these are only two of the many life-cycle software issues facing plant managers. Others include:

  • Cyber security—from mundane patches for the latest viruses to new code and logic to protect against criminal or terrorist activity.
  • Personnel training.
  • Tuning, calibration, and retraining of model-based software, collectively called configuration control.
  • Consistency in software tools.
  • Although software life-cycle management may seem like yet another paper-trailcreation exercise or an otherwise additional nuisance for a plant, the consequences of not thinking in terms of the bigger picture are real. Here’s just one example: Twice in the same year at one utility plant, software systems were responsible for safety and reliability impacts during plant start-up simply because the staff could not find the current version of the system software.

    Like any other component
    One way to think about the software life cycle is to treat a piece of software just as you would a piece of equipment in the physical plant. Software should be purchased, designed, integrated (with the computer hardware and other software), and maintained to satisfy a specific set of functions.

    During the “design” phase, you define the functionality required, write a specification, purchase or design the software, install and test it, and then turn it over to the plant. At this point, the software needs an “owner,” just like any other plant component. The long-term care and feeding of software involves operating and maintenance considerations, as do physical components. Economic decisions about the level of maintenance for software will depend on its expected useful life. As with physical component maintenance, you will need some mix of corrective, preventive, and predictive software maintenance strategies, which will also require reviews, validation, and upkeep on defined, regular schedules.

    At nuclear plants, where change and innovation only invites increased regulatory scrutiny and paperwork, software is generally expected to have a long life cycle. Nuclear facilities also have great difficulty managing things without paper. Even when they create an electronic document, they typically manage the information as electronic paper—as a simple electronic file. But it is simply too cumbersome to manage the life cycle of a digital product like software with paper. The volume and complexity of the information is too great.

    Fossil plants are used to dealing with more frequent intervention in their software systems, frequently released new versions and revisions, enhancements (purportedly for efficiency and productivity), and even wholesale replacement after only a few years. Software suppliers issue service packs and revisions to add functionality to a device, to respond to customer requests, or to fix “bugs” and flaws. Users also add or delete data points, change parameters, and reconfigure systems regularly. Plant safety, reliability, and performance are at stake if revisions and changes are not formally documented and managed.

    Even when life-cycle management processes are implemented, owner/operators invariably underestimate the complexity of digital systems. The processes for dealing with software in the field either tend to be underdesigned or are overwhelmed and out of date shortly after rollout. Maintenance workers carry some tools that are recognized by everyone and others that are recognized by those with only limited mechanical work experience. Likewise, software maintenance and upkeep requires tools, and their use needs to be just as typical and self-explanatory. Ideally, these tools shouldn’t require training before being used by those responsible for the software. Usually, this means that they need to be provided on common platforms, like Microsoft-, or web-based interfaces.

    One common tool for plant systems is a well-defined configuration management capability. Plants must be able to compare a specific software configuration to an “asbuilt,” “as-required,” or “current” configuration. Some software vendors supply these configuration management tools and some don’t. Some tools supplied with an automation system work well; some don’t. Invariably, vendors focus on the initial design and development of their software, not on long-term life-cycle issues.

    The data management toolset also needs to be consistent with the toolsets required of other software packages. For example, plants are typically designed using sophisticated computer-aided design software programs. “Virtual” plants are delivered along with the physical plant. If a virtual plant is going to have any “life-cycle” value to plant management, then the upkeep tools must be familiar and self-explanatory to those responsible for its maintenance. Owner/operators are demanding that

    A foundation for a software life-cycle management policy

    A standardization process for managing the software life cycle was proposed in “Software Life Cycle Management Standard for Micro-Processor Based Instrumentation and Control Systems,” a paper presented at the ISA/EPRI Power Industry Symposium (POWID) in 2005 by Randall L. Miller and H. Leo Staples of OGE Energy Corp. At this year’s POWID Symposium this seminal work was referenced by Richard Marshall of Control Solutions LLC and George Shi of Kansas City Power & Light in their paper and presentation, “Developing Configuration Management Processes for Power Plant Control Systems.”

    Power Plant Control Systems.” Miller and Staples observed that service packs and revisions do not always apply to the configuration at a specific plant, but the revision still needs to be tracked. The first step is to determine the relevance of the revision to a particular plant. For example, the software supplier may issue a revision that enhances a servo card for a motion control device, but the plant system does not have servo cards.

    On the user side, a log of all revisions made at the plant provides a troubleshooting tool in the event of problems. Revisions include changes to the database, scaling changes, control setting adjustments, and loop reconfigurations.

    One recommendation made in the 2005 paper is to treat software as if it were a physical engineering drawing. Two current copies of the current software configuration should be kept— one at the plant and one at corporate—along with the log. Tasks residing at the plant would include assigning an engineer or technical staff person responsibility for all software systems. That person maintains the log, ensures that plant staff are adhering to the documentation procedures, conducts reviews of proposed revisions (with appropriate plant and corporate personnel), and conducts an annual audit of the software storage area. The corporate assignee tracks changes made to plant systems and maintains the corporate logs and copies of software.

    Marshall and Shi’s presentation provided examples of both paper- and electronic-based configuration management (CM) of the software associated with a distributed control system, which is a critical subset of the software life cycle. The electronic version can be built up from common PC software products like Microsoft Sharepoint Server and Infopath applications. Online processes can usually be created without any additional programming, thus making them accessible to a wider user base. Electronic processes designed for software CM also create their own audit trail and are automatically archived.

    For additional background, readers are referred to the technical report “ISA TR84.00.03.2002: Guidance for Testing of Process Sector Safety Instrumented Functions Implemented as or within Safety Instrumented Systems [ISA 2004].” This document provides a foundation for deriving or customizing a standardized life-cycle software management program for owner/operators.

    the design data be provided in forms that plant management can use throughout the life of the plant. This impacts the tools selected by the engineering, procurement, and construction contractors to design the plant and “deliver” a virtual plant.

    The service center model
    One way to manage software life-cycle issues (and, for that matter, current software operations) is through the third-party service or partner model. With this approach, the software vendor essentially becomes the assigned “owner” of the plant’s software. Several power plant software suppliers now offer a monitoring and diagnostic service on a subscription fee basis.

    One of the life-cycle benefits of this approach is that plant systems or components can be consistently benchmarked to those at other facilities of similar vintage, duty, operating life, and model number. Model-based software may be especially amenable to a services approach. Essentially, software design, engineering, revision, upkeep, calibration and retraining, and maintenance are managed by a third party, allowing the staff to focus on the power plant’s physical operation and maintenance, not software life-cycle issues.

    Many owner/operators with large portfolios of plant assets have already converted to a centralized performance monitoring facility model where monitoring and diagnostic (M&D) responsibility either resides or is shared. (See page 66 for a description of Entergy’s Performance Monitoring and Diagnostic Center.) Independent power producers and merchant plant owner/operators, who manage many of the latest combined-cycle facilities, depend on the gas and steam turbine supplier for 24/7 monitoring and diagnostics; related services providers collect, analyze, and manage fleetwide data for such machines. Now, software vendors with even more specialized, and often more complex, M&D capabilities are beginning to deploy the services model to power plants.

    Going forward
    Below are some specific recommendations for managing software life-cycle issues.

  • Measure to manage

  • You can’t manage
    what you don’t measure. Some in the software business talk of software quality assurance (QA). The diagram provides a high-level overview of the metrics being contemplated for QA evaluations of software.

  • Hold implementation post-mortems.

  • Plenty of meetings are held to plan a successful software implementation, culminating in the “go live” date. Post-implementation meetings should be held to determine what went right, what went wrong, who retains ownership, how to meet continuous training requirements, who handles software storage, and the like.

  • Conduct usability evaluations.

  • This is an emerging area of analysis, as indicated by the recent publication of books such as
    Maturing Usability: Quality in Software, Interaction, and Value, by B. Jefferies, et al. (October 2007). Most power plant owner/ operators assess products through the experience
    of their peers in power generation;formal usability evaluations may prove to be excellent supplements.

    Inspect, don’t expect.jpg

    Software quality assurance evaluations often include these metrics. You can’t control what you don’t measure.

  • Outsource and partner

  • If your plant is not the flagship of the fleet, if power generation is not the “core business” of your plant owner, or if you are owned by a small utility, resources may be scarce. Even contemplating the purchase of performance-based software may be daunting because of a lack of expertise. The emerging service center model is ideal for these situations.

    Leave a Reply