We have two MS5001 turbines on our site (GT-601 & GT-602) having Mark-V turbine control system and EX2000 installed. They share a common HMI and have 1 local HMI each. The local HMIs are in the field whereas common HMI is in the control room. Communication with common HMI is established using ModHub coaxial to optical fiber media converters because of the distance. We have been facing communication issues recently which made the excitation system of GT-602 switch to manual mode twice.
This issue was faced 2.5 yrs ago as well and after extensive troubleshooting the ModHub card was found to be faulty. Because of the common HMI and interconnected network, the fault in one turbine affects the communication on the other as well. This being an intermittent fault we always isolate parts of the system, keep them under observation for a while and then narrow down the possible issues with wire, Tees, communication card, end termination resistor, ModHub card, cable's end connectors, optical fibers etc. The whole process takes quite long. Please also note that the HMIs values disappearance has occurred more times than the Ex2000 switchover from auto to manual
I have two questions regarding the case:
1. I've observed that both times this issue occurred, one of the turbines was shutdown. Can there be an actual communication overloading issue because of increased error counts and alarms as happens with some of the DCS?
2. Can you recommend some ARCnet analyzer module for easier faulty node identification? Someone recommended SH ARCALYZER for the case but I haven't received any history of any company ever using it with Mark-V. I need one through which I can monitor all ARCnet nodes including the Ex-2000 nodes.
Do let me know if any further information is required for the case.
Back when I started working on GE Mark V turbine control systems, I was terrified of the ARCnet-based StageLink--worried that if there were a problem it would be extremely difficult to find and resolve in the field. My fears--at that time--were for naught, as the initial implementations of the ARCnet-based StageLink were virtually bullet-proof.
THEN, GE went and changed from the IDOS-based <I> operator interface which was designed with the capabilities of the ARCnet-based StageLink in mind, to the MS-Windows-based HMI operator interface running CIMPLICITY (now affectionately known as PROFICY Machine Edition). That was the beginning of a LOT of problems with communications between Mark Vs, EX2000s, and GE Mark V HMIs.
The majority of problems were due to the way that CIMPLICITY worked--it asked for the status of EVERY signal in the CIMPLICITY project on every data request. Whereas the <I> only asked for the status of signals which were being displayed on whatever display was active (that included any Short-Term Trending running in the background, and any MODBUS or GSM applications running in the background). This change to CIMPLICITY greatly increased the traffic on the ARCnet-based StageLink, and when there were multiple GE Mark V HMIs (running CIMPLICITY) GE "imposed" a limit of two GE Mark V HMI 'servers' per Mark V to try to limit the traffic as best as possible. (A GE Mark V HMI server is one that is receiving and transmitting data directly on the StageLink; a GE Mark V HMI viewer was a GE Mark V HMI that received data for CIMPLICITY over Ethernet from a GE Mark V HMI server, which limited the traffic on the StageLink.)
One site I worked on had eight (8) GE Mark V HMI servers (one in the local control compartment of each of eight gas turbines), and four in the remote control room, along with an OSM (On-Site Monitor) which gathered data over the StageLink for a remoter Monitoring & Diagnostic Center. The StageLink was very taxed, to say the least, and GE switched the local HMIs to viewers to reduce the network traffic.
Another problem which hampers the StageLink is when there are signals in the CIMPLICITY project which are NOT used by the turbines/EXs. For example, if someone mis-spells a signal name in the CIOMPLICITY project (such as L2TVC instead of L2TVX) and leaves that signal in the CIMPLICITY project it will keep asking the Mark V(s) for the status of that signal on every request. Or, as was the case with MANY jobs shipped from the GE factory, signals which weren't used for a particular job were left in the CIMPLICITY project CIMPLICITY would continuously ask for the status of those signals, creating a lot of unnecessary network traffic.
Finally, if there is a MODBUS link to a DCS, or a GSM link (GE Standard Messaging Protocol) to a DCS, and the link is misconfigured (as many are) then the StageLink will be flooded with malformed requests or request for signal data that doesn't exist. And, in some cases MODBUS and/or GSM configurations requested ALL data points at the fastest possible rate--which was unnecessary and also added to the StageLink network traffic.
When there are network traffic issues, there are usually Diagnostic Alarms, either on the HMI or on the LCC/SLCC displays of the <C> processor or the main processor of the EX2000. Messages on the LCC/SLCC display with acronyms like DPRAM (Dual-Ported RAM) and COMM are indications of StageLink traffic problems. Also, the "normal" LCC/SLCC display shows the amount of idle time on the processor (in %) and if that's very low, especially on <C> or <R> (or <D>, if so equippped), that can be a very good indication of high network traffic.
The manufacturer of the MODHUB, Contemporary Controls, has the best ARCnet analyzers--and they have been very helpful to many of us in the field working on GE ARCnet-based StageLinks. They don't know much about CIMPLICITY or MODBUS or GSM, but they know a LOT about ARCnet and how GE implemented ARCnet for the Mark V turbine control systems. I believe their URL is www.ccontrol.com. And, don't hesitate to email or call them with questions about their equipment and its capabilities--they have always been very good with their technical support when using their products.
I suggest you review your site network configuration, including any MODBUS and/or GSM link configurations before you do anything. Because, if this problem started after someone made a change to one or the other, a seemingly innocuous change, then that would be the place to start looking.
Don't overlook the CIMPLICITY projects on each of the HMIs, either. It's entirely possible that one of more of the projects have extraneous signals, or mis-spelled signals. This happens someone tries to add signals to project(s) in the field and don't know the proper procedure (which varies with different versions of CIMPLICITY and TCI and MS-Windows) and haven't created back-ups for recovery....
Using a protocol analyzer can have a steep learning curve, and it may just show there's a LOT of traffic. Without a deep knowledge of how the StageLink work, and how any links (such as MODBUS and GSM) work, the protocol analyzer might not reveal much.
Finally, there were a LOT of improper BNC connectors used on coaxial cables for StageLinks in the field. (There are Ethernet BNC connectors and cables, and ARCnet BNC connectors and cables.... and Ethernet components were always much more available and they looked exactly like their ARCnet counterparts, so they must be the same, right?) Right??? You don't know how many times electricians and electrical contractors told me that--because they couldn't easily get ARCnet components, or the ARCnet components were more expensive.) And, even some improper cabling. Sometimes they worked fine for years, and suddenly quit. And, many BNC connectors also develop corrosion over time, inside the connectors where it's very difficult to see. And, some sites even used "twist-on" male BNC connectors on the ends of coaxial cables--which were notorious for bad installations and connections. Proper grounding of the coaxial cable shield through the BNC connectors is VERY important and often overlooked. Sometimes one or two stands of the braided sheath used for the coxial cable shield would be improperly touching the internal conductor when making-up the cable ends.
So, those are some things to consider when troubleshooting ARCnet-based StageLink problems. There are a LOT of possible contributors to network traffic issues. Hope this helps.
Please write back to let us know how you fare in resolving the problems!
Thank you for the detailed reply. Really appreciate it. As mentioned by you, the number of contributing factors to these are numerous and issue identification through system isolations takes luck-dependent time. Had the fault not been intermittent it would have been easier to identify.
The ModHub card's health deterioration was identified as the issue. We changed the card and there have been no issues faced since.
For this job, the following things were done:
- Turn-wise each Tee was replaced, issue did not get resolved
- Optical fiber cable health was checked
- Coaxial cable connectors were checked
- All contacts were cleaned using contact cleaner
- For future projects we will ensure never to have the control of excitation systems on ARCnet. There is no alarm configured for EX2000 switching from auto to manual. For the time being we have established a procedure for the operator to confirm the status of EX2k after every communication fault.
- Grounding of all components was ensured and all connectors were cleaned with contact cleaner
- Visual inspections for rust/ dust were made
- ModHub card replacement in the local room of GT-602 resolved the node-reconfiguration leading to ARCnet communication issue.
I'm exploring two options for the matter.
1. To get an ARCnet protocol analyzer. If nothing else, it can make faulty node identification easier. However, the data mapping in all options I've found so far are slightly difficult to interpret (or maybe its because of my current unfamiliarity)
2. The other option I'm exploring is to simply use an ARCnet to ethernet conversion module, assign IP addresses to each node and use the current softwares for node monitoring, data mapping and historization. I can't find anyone who has used this on Mark-V before though
I need a monitoring module since not only will this save troubleshooting time but we can also keep a track of the node healths off and on.
Please do let me know if you know of any other company that has used ARCnet monitoring tools. Also, again, thank you for the detailed insight into Mark-V's limitations related to Windows based HMIs (all our HMIs are windows based).
Thanks VERY MUCH for the detailed feedback! It's so important for others who read these threads now--and in the future--to know what was done and how the problem was ultimately resolved.
I believe there are several companies, including some divisions of GE, that are using ARCnet-to-Ethernet/Ethernet-to-ARCnet converters for cabling. There is still a short bit of ARCnet required, and I'm not really sure what this affords anyone in terms of reliability or availability. Ethernet cabling is just as prone to problems as ARCnet cabling, and the converters add yet more points of failure. In my experience, the ARCnet--when properly installed--has been extremely reliable. Until the introduction of CIMPLICITY, and then, even afterwards if proper care is taken to configure and maintain CIMPLICITY projects & displays, HMI hard drives, and any communication links (GSM and MODBUS).
The lack of documentation by GE, and even their lack of attention when upgrading HMIs, contributes to lots of unnecessary errors.
Again, thanks very much for the detailed reply--it shows a definite logical approach to identifying the problem, instead of the usual "shotgun" approach (try everything at once and hope it solves the problem--which never identifies exactly what the problem was!).
Does the EX2000 have a discrete (contact) output that can be connected to a spare, unused Mark V discrete (contact) input to indicate when the EX2000 has changed from Auto to Manual? This could be a relatively simple solution to this particular problem.