We have two MS5001 turbines on our site (GT-601 & GT-602) having Mark-V turbine control system and EX2000 installed. They share a common HMI and have 1 local HMI each. The local HMIs are in the field whereas common HMI is in the control room. Communication with common HMI is established using ModHub coaxial to optical fiber media converters because of the distance. We have been facing communication issues recently which made the excitation system of GT-602 switch to manual mode twice.
This issue was faced 2.5 yrs ago as well and after extensive troubleshooting the ModHub card was found to be faulty. Because of the common HMI and interconnected network, the fault in one turbine affects the communication on the other as well. This being an intermittent fault we always isolate parts of the system, keep them under observation for a while and then narrow down the possible issues with wire, Tees, communication card, end termination resistor, ModHub card, cable's end connectors, optical fibers etc. The whole process takes quite long. Please also note that the HMIs values disappearance has occurred more times than the Ex2000 switchover from auto to manual
I have two questions regarding the case:
1. I've observed that both times this issue occurred, one of the turbines was shutdown. Can there be an actual communication overloading issue because of increased error counts and alarms as happens with some of the DCS?
2. Can you recommend some ARCnet analyzer module for easier faulty node identification? Someone recommended SH ARCALYZER for the case but I haven't received any history of any company ever using it with Mark-V. I need one through which I can monitor all ARCnet nodes including the Ex-2000 nodes.
Do let me know if any further information is required for the case.
Back when I started working on GE Mark V turbine control systems, I was terrified of the ARCnet-based StageLink--worried that if there were a problem it would be extremely difficult to find and resolve in the field. My fears--at that time--were for naught, as the initial implementations of the ARCnet-based StageLink were virtually bullet-proof.
THEN, GE went and changed from the IDOS-based <I> operator interface which was designed with the capabilities of the ARCnet-based StageLink in mind, to the MS-Windows-based HMI operator interface running CIMPLICITY (now affectionately known as PROFICY Machine Edition). That was the beginning of a LOT of problems with communications between Mark Vs, EX2000s, and GE Mark V HMIs.
The majority of problems were due to the way that CIMPLICITY worked--it asked for the status of EVERY signal in the CIMPLICITY project on every data request. Whereas the <I> only asked for the status of signals which were being displayed on whatever display was active (that included any Short-Term Trending running in the background, and any MODBUS or GSM applications running in the background). This change to CIMPLICITY greatly increased the traffic on the ARCnet-based StageLink, and when there were multiple GE Mark V HMIs (running CIMPLICITY) GE "imposed" a limit of two GE Mark V HMI 'servers' per Mark V to try to limit the traffic as best as possible. (A GE Mark V HMI server is one that is receiving and transmitting data directly on the StageLink; a GE Mark V HMI viewer was a GE Mark V HMI that received data for CIMPLICITY over Ethernet from a GE Mark V HMI server, which limited the traffic on the StageLink.)
One site I worked on had eight (8) GE Mark V HMI servers (one in the local control compartment of each of eight gas turbines), and four in the remote control room, along with an OSM (On-Site Monitor) which gathered data over the StageLink for a remoter Monitoring & Diagnostic Center. The StageLink was very taxed, to say the least, and GE switched the local HMIs to viewers to reduce the network traffic.
Another problem which hampers the StageLink is when there are signals in the CIMPLICITY project which are NOT used by the turbines/EXs. For example, if someone mis-spells a signal name in the CIOMPLICITY project (such as L2TVC instead of L2TVX) and leaves that signal in the CIMPLICITY project it will keep asking the Mark V(s) for the status of that signal on every request. Or, as was the case with MANY jobs shipped from the GE factory, signals which weren't used for a particular job were left in the CIMPLICITY project CIMPLICITY would continuously ask for the status of those signals, creating a lot of unnecessary network traffic.
Finally, if there is a MODBUS link to a DCS, or a GSM link (GE Standard Messaging Protocol) to a DCS, and the link is misconfigured (as many are) then the StageLink will be flooded with malformed requests or request for signal data that doesn't exist. And, in some cases MODBUS and/or GSM configurations requested ALL data points at the fastest possible rate--which was unnecessary and also added to the StageLink network traffic.
When there are network traffic issues, there are usually Diagnostic Alarms, either on the HMI or on the LCC/SLCC displays of the <C> processor or the main processor of the EX2000. Messages on the LCC/SLCC display with acronyms like DPRAM (Dual-Ported RAM) and COMM are indications of StageLink traffic problems. Also, the "normal" LCC/SLCC display shows the amount of idle time on the processor (in %) and if that's very low, especially on <C> or <R> (or <D>, if so equippped), that can be a very good indication of high network traffic.
The manufacturer of the MODHUB, Contemporary Controls, has the best ARCnet analyzers--and they have been very helpful to many of us in the field working on GE ARCnet-based StageLinks. They don't know much about CIMPLICITY or MODBUS or GSM, but they know a LOT about ARCnet and how GE implemented ARCnet for the Mark V turbine control systems. I believe their URL is www.ccontrol.com. And, don't hesitate to email or call them with questions about their equipment and its capabilities--they have always been very good with their technical support when using their products.
I suggest you review your site network configuration, including any MODBUS and/or GSM link configurations before you do anything. Because, if this problem started after someone made a change to one or the other, a seemingly innocuous change, then that would be the place to start looking.
Don't overlook the CIMPLICITY projects on each of the HMIs, either. It's entirely possible that one of more of the projects have extraneous signals, or mis-spelled signals. This happens someone tries to add signals to project(s) in the field and don't know the proper procedure (which varies with different versions of CIMPLICITY and TCI and MS-Windows) and haven't created back-ups for recovery....
Using a protocol analyzer can have a steep learning curve, and it may just show there's a LOT of traffic. Without a deep knowledge of how the StageLink work, and how any links (such as MODBUS and GSM) work, the protocol analyzer might not reveal much.
Finally, there were a LOT of improper BNC connectors used on coaxial cables for StageLinks in the field. (There are Ethernet BNC connectors and cables, and ARCnet BNC connectors and cables.... and Ethernet components were always much more available and they looked exactly like their ARCnet counterparts, so they must be the same, right?) Right??? You don't know how many times electricians and electrical contractors told me that--because they couldn't easily get ARCnet components, or the ARCnet components were more expensive.) And, even some improper cabling. Sometimes they worked fine for years, and suddenly quit. And, many BNC connectors also develop corrosion over time, inside the connectors where it's very difficult to see. And, some sites even used "twist-on" male BNC connectors on the ends of coaxial cables--which were notorious for bad installations and connections. Proper grounding of the coaxial cable shield through the BNC connectors is VERY important and often overlooked. Sometimes one or two stands of the braided sheath used for the coxial cable shield would be improperly touching the internal conductor when making-up the cable ends.
So, those are some things to consider when troubleshooting ARCnet-based StageLink problems. There are a LOT of possible contributors to network traffic issues. Hope this helps.
Please write back to let us know how you fare in resolving the problems!