MARKV: Swapping BNC Connectors of EX2000 and MarkV HMI on CTBA Card

We have two GE Frame V gas turbines installed on our site. Each of the gas turbine is communicating with EX2000 and MarkV HMI (computer based and with cimplicity). The arcnet cables are terminated on CTBA card of the C core. Can we swap these cables on CTBA card? Is it possible while maintaining same functionality?

The reason for doing this is that we are experiencing communication problem between EX2000 and MarkV panel. EX2000 is showing the diagnostic alarm 418 on local display. As the communication link is failed between MarkV and EX2000 we can not reset it from HMI. HMI is only showing two process alarms regarding EX2000 i.e. P389 and P426. P389 tells that the communication link is failed and P426 is showing that there is some diagnostic alarm on EX2000.

I have read the other posts regarding the same problem and their solutions. Unfortunately I don't have authorization to implement any of these on EX2000 (Even I can't reset alarm on EX2000). I was instructed to change the CTBA card without any reason. Instead of changing the card I want to swap the Arcnet cables on CTBA card so that I don't have to change this card if after swapping there is no problem in communication between MarkV and HMI and troubleshooting can be done on EX2000 and the stagelink segment between EX2000 and MarkV.
 
Esoteric_Stone,

Hmmm.... Don't think I understand the question or the issue. Let me preface my comments by saying I'm not very familiar with EX2000 excitation systems, but I have troubleshot and resolved lots of StatusS and StageLink communication issues over the years.

There are two BNC connectors on the CTBA card, identified as JAI and JAJ. There are two small relays on the CTBA card that are a part of the communications network (I think they are small rectangular "canned" relays, or they may be small, rectangular yellow plastic relays. I believe one of the relays is associated with JAI and the other with JAJ. A lot of times these relays fail and cause problems--but this requires changing the CTBA to rectify.

Now, there's no difference between what JAI and JAJ do--they are functionally identical. It shouldn't make any difference what is plugged into JAI or JAJ. So, it shouldn't pose any problems. However, if the StageLink cable from the HMI(s) is plugged into, say, JAI, and the EX2K is plugged into JAJ and you swap the cables from JAI to JAJ and JAJ to JAI and the HMIs stop working--then it's a pretty safe bet that the relay for JAJ has failed, and, again, they are soldered on to the CTBA card and the CTBA card must be replaced to resolve the problem.

You could try sourcing a BNC tee connector for use with ARCnet networks (they have different electrical characteristics than BNC connectors for Ethernet networks!) and putting it on either JAI or JAJ and then plugging in the coaxial cable from the EX2K into one side of the BNC tee connector and the HMI cable into the other side of the BNC tee connector. That is also a "legal" termination for the two network connections on the CTBA. So, if you have determined that either of the two BNC terminations on the CTBA is not working and the other one is, then you can use the ARCnet BNC tee connector to prevent having to replace the CTBA until a more convenient time (during a maintenance outage)--but you should replace the CTBA card if one of the BNC ports isn't working because if the last remaining port fails, then the CTBA will have to be replaced.

You can refer to the Mark V Application Manual, GEH-6195, for more information about permissible ("legal") StageLink (and EX2K) connections.

Hope this helps! Write back to let us know how you fare, please.

[Ethernet BNC tee connectors have been used with mixed success on ARCnet-based Mark V networks. But, they have been known to intermittently fail, and to fail outright. Also, corrosion can cause problems with BNC connectors in the right environmnent (or, maybe that should have been in the wrong environment!). Check to make sure the connections are bright and shiny.

MANY BNC connectors were very poorly terminated during initial installation and commissioning and have cause problems over the years since installation and commissioning. Some of them have been very intermittent and VERY frustrating. So, if the terminations don't look very good (visually), you may just want to obtain a proper crimping tool and connectors and cut off the connectors from the original installation and install crimp new connectors on.

Finally, many BNC cables were abused during installation, having been badly snagged when being pulled through conduits or penetrations and severely "kinked". This also happened to factory-provided cables which were pulled through conduits and penetrations using poor techniques which damaged the factory-crimped connectors.]
 
C
I have never seen the CTBA card cause arcnet problems. The CTBA is merely a landing point for the connector. The TCCA card in <c> and <d> core handles arcnet communication. CTBA is always located in the upper right hand corner of a MKV cabinet usually out of reach from casual contact.

The EX2000 however has the arcnet connector out in the open. Usually to the right of the EX2000 core mounting; a terrible location. I have seen that board damaged by technicians.

ARCNET is not something new. I will not go into detail but it is a "bus" network with terminating resistors 93 ohms. Switching the cables on <c> ctba will have no affect. You will have a brief break in communication.

Since; you still have communication throughout your bus network. i.e. C core, HMI are happy. The problem resides on the EX2000.

Utilize ARCWHO command on the HMI to ascertain WHO is listening. Files on the MKV HMI will reveal the addresses of HMI, <c>, EX2000, etc.

Look for the termination resistors to ascertain where the limits of your "bus" network begin and end. Draw a one line of the network, it helps.

I would help more but information regarding EX2000, HMI, C Core diag from LCC were limited.
 
CuriousOne,

Of all the cards I have replaced in Mark V control systems (and I didn't replace that many), the CTBA was one of the most common to be replaced--mostly because of the failure of the communication relays on the CTBA. Those relays, if I recall correctly, drop out when <C> is not powered-up and that allows unbroken StageLink communications through the Mark V (presuming the StageLink didn't terminate in the Mark V but went on to other locations in the plant, such as other Mark Vs or HMIs or even EX2Ks). So, that means they are energized whenever <C> is powered-up, and sometimes the coils of the relays failed (and that failure rate seemed to increase in Mark V panels where the ambient temperature was not properly maintained, either too hot and/or too humid).

Which brings me to another issue. Approximately half-way through the production life of the Mark V GE decided that any StageLink segment that terminated on the CTBA should have a BNC tee connector with a 93-ohm termination resistor on it--instead of just plugging the coaxial cable into the JAI or JAI port. The reasoning was that it improved StageLink communications, and it seemed to work. It would look like this:<pre>
BNC
Tee
Connector
->-------------|-----|T|
Coaxial |
StageLink -
Cable JAI or JAJ
from external
nodes</pre>So, my recommendation would be to either:

1) use a BNC tee connector on either JAI or JAJ and plug the cables from the HMI and the EX2K into the tee connector (that is, NOT to plug the cables directly in to JAI and JAJ),

or,

2) use two BNC tee connectors each with 93-0hm termination resistors, one on JAI and the other on JAJ, and plug the cables from the HMI and EX2K into the other side of each BNC tee connector.

Again, if swapping the cables causes one link to start working and the other to fail then it's a safe bet that there's something amiss with the relays on the CTBA or the CTBA. Remember that the CTBA has ribbon cables to connect the StageLink to the TCCA (if I recall correctly) and those terminations should also be checked--especially based on past problems at this site.

And, most importantly, the integrity of the coaxial cable and connectors on the cables should also be verified--particularly on the cable between <C> and the EX2K.

One more thing I remembered: It was common, if the EX2K was located in the PEECC (usually when the application was for a brushless exciter), for a long coaxial cable to be used to make the connection between the Mark V and the EX2K. This meant that the coaxial cable was coiled somewhere in the PEECC--usually in the EX2K, but sometimes in the Mark V. Large coils of coaxial cable are not good for ARCnet-based communication links, especially if they are located near high voltage/high-current cables/wires. This goes for coaxial cables between "local" Mark V operator interfaces (operator interfaces located in the PEECC very near the Mark V) as well. If the coaxial cable must be coiled, it should be coiled in an area where it is not near other high-voltage and/or high-current conductors (so, not in the bottom of the EX2K or Mark V control panels).

Hope this helps!
 
CSA,

I will try to explain the question again. We were having communication problem between EX2000 and MarkV. The communication between them was failed while the turbine was running. This problem do occur three to four times every year. Sometimes this problem goes away without doing anything special and sometimes it persists. This time this alarm was persisting. HMI was working fine. I don't know what others do on different plants but over here the things are strictly defined. Electrical is one department and Instrument is other department and no one can touch or do anything with each others system. On our plant EX2000 belongs to Electrical department and MarkV belongs to Instrument department. When the problem appears people usually say that this is the problem of MarkV as it is appearing on MarkV without knowing the cause of the problem. Troubleshooting is one art and convincing people is another art. If 40 percent of the energy is used on diagnosing the problem then the remaining 60 percent is wasted on convincing the people. I said wasted because when people don't know they usually don't hear others as well. I was having no doubt on CTBA or TCCA card that's why I was trying to find a way in which I can convince others that this time the communication failure is not from MarkV end. I hope this clears the situation.

You have worked on status S messages or issues. Can you please guide me how can I understand this? Any manual? I was looking at the diagnostic counters today (particularly TCCA card). There was something about Status S and status page etc but I was unable to understand most of the things. Is there any book or reference material where I can read its description. I have searched on this forum about status s and I found some information but I want to know in detail.

Now I will come to what we have done today. The communication between MarkV and EX2000 was restored just before we started the activity but we decided to check the things. First of all we swapped the cables plugged on JAI and JAJ. The communication between HMI and EX2K was fine. Next we opened the coax cable from MarkV and EX2K and checked its resistance (found OK). Connections of coax cable were also checked (Found Ok). In EX2K coax is terminated on LNTB. A cable is going to arcnet card from LNTB. When it was tried to press this cable the alarms started to appear on EX2K. The alarm was 417 and it was not resetting (Tried to reset from EX2K). Based on this it was decided to change the arcnet card and the cable between arcnet card and LNTB. After that the problem was resolved.

The gas turbine was started. After some time diagnostic alarm D1068 (TCCA Status S page xmit failure) appeared on HMI. But it appeared on both turbines with gap of 10 minutes. I will try to explain the arcnet architecture on our site. We have two gas turbines. Their architecture is shown below

EX2K----->MarkV(GT1)----->HMI(for GT1 in MarkV compartment)--->HUB(COax to fiber optic)=======>Hub(Fiber optic to Coax)------------------------> HMI(for GT1 and GT2 in control Room) <----- Hub (coax to fiber optic)<========= Hub(fiber optic to coax)<----------HMI (For GT2 in MarkV compartment) <------- MarkV(GT2)<----EX2k

BNC T connectors are used on HMIs in the MarkV compartments and the HMI (for GT1 and GT2) in control room.

I have read that the diagnostic alarm 1068 shows that there is some problem in communication between EX2K and MarkV. We got this alarm on both GTs. This alarm was not persisting. It came only for brief time may be for a second or so. Before this activity we were not having any problem on GT1. If we are getting this alarm on both GTs the are we having communication problem on both GTs (With EX2K ofcourse). It is worth mentioning here that with TCCA diagnostic alarm 1068 we also got alarm 417 on EX2K of the GT2 (on which we performed activity today) but there was no communication failure and we were able to reset it from HMI. Any suggestion about this?

I liked your suggestion of using BNC T connector in case of failure of relay of JAI or JAJ. I have saved it for future.

In the end I would like to thank you, the founder and moderators of this forum.
 
C
> A cable is going to arcnet card from LNTB. When it was
> tried to press this cable the alarms started to appear on EX2K.

I always keep two of these cards on hand. I have had far more problems with this card than the CTDA. Glad it fixed your problem and Glad you shared the results.
 
Esoteric_Stone,

Yes; we have heard of the strict delineations between departments--and the problems it causes when things like this happen. And, of course, it's always the control system's fault--there are just too many wires and too many LEDs and it's just so mysterious that it MUST be the cause of all problems. And so the Instrument technicians, or the Instrumentation & Control technicians, or the Controls technicians must spend hours and hours proving it's NOT the control system, which no one ever believes anyway. ("He just got lucky THIS time--but it's always been the control system's fault in the past!")

I would say that based on the "loose connections" you're reported in the past that at the very next opportunity the Mark V turbine control panel(s) at your site should be powered-down and each and every ribbon cable and plugged connection should be opened/closed several times, with some conductive grease (remember: too much as as bad as too little!) applied to the pins/sockets to improve conductivity and at least prolong the period until the next intermittent connection problem occurs, at which time the same thing should again be done (if not on a periodic basis, and I usually recommend every two years, or so, depending on the environment where the turbine control panels are located (oil refinery; chemical refinery; paper mill; steel mill; cement plant; etc.)).

As for documentation about Status-S protocol, there's even less written about that than StageLink protocol. Fortunately, it either works, or it doesn't. It's not something someone really has to know anything about, but when it doesn't work it's usually hardware-related, not software. When I first started working on Mark V turbine control systems I felt I had to learn everything I could about ARCnet, and soon found I was wasting my time. It either works, or it doesn't--and if it doesn't work, again, it's usually hardware-related.

Believe me when I tell you that working on Mark V communications is much, Much, MUCH simpler than working on Mark VI or Mark VIe communications (which are Ethernet-based). And, again, when those communication links aren't working it's always hardware-related--it's just that there is SO much hardware, and it's not just "plug-and-play" (it's more like "plug-and-PRAY"). And there even multiple protocols used on the same Ethernet cable, which complicates troubleshooting even more.

I have seen some fiber optic repeaters used in StageLink applications cause a lot of problems--but this doesn't seem to be happening at your site. Is the location where the EX2K located air conditioned, or is it just left to mostly ambient conditions? Heat and humidity and dust are any electrical connection's worst enemies. And depending on the dust, and any other airborne contaminants the problems can even be worse--hence the recommendation for conductive grease, especially on the connectors used in the manufacture of Mark V cards (which is more susceptible to the effects of corrosion and humidity than some other commonly used materials).
 
CSA,

We do check the ribbon cables in every shutdown. The turbines are shutdown once in a year for annual maintenance and checking. We will do it again in coming shutdown.

So whenever a diagnostic alarm related to status-S occurs I should believe that the problem may be in hardware and I should check following things

1. Ribbon cable between TCCA and CTBA
2. Relay failure on CTBA card
3. Problem may be on EX2K

Am I right? and if HMI is working fine then is it safe to say that CTBA or TCCA is not faulty? EX2K and HMI both are on stage link and are communicating with MarkV via CTBA. I have read that EX2K uses Status-S protocol for communication with MarkV. Which protocol is used for communication between MarkV and HMI? I mean what will the diagnostic alarm look like if there is communication problem between MarkV and HMI?

I am quite amazed to know that communication system for MarkV is more reliable or simpler than MarKVI or VIe. I used to think that these problems may not have occurred if we were using MarKVI or VIe. Thanks for providing information in this regard.

Yes the place where EX2K is located is air conditioned. Its located in MarkV compartment provided by GE (The same compartments whose doors can not be opened from inside without punching their locks).
 
Esoteric_Stone,

What does checking the ribbon cables include? Because this is the second thread where it's been told that the ribbon cables were checked, and then later it was told that pushing on the ribbon cables resulted in problems that resulted in replacing the cable and/or the card and that resolved the problem(s). It would seem that if the cables are being checked, either that check does not include the application of electrically conductive grease, or too much grease is being applied, or the cables are not properly re-seated and are loose. That's seems to be about the only possibilities--<b>based on the information provided.</b> (I'm only trying to understand, and to point out, what the problem may actually be--not a cable or manufacturing or design problem, but a human-caused problem, at least for this and the other thread you recently provided feedback on.)

I would say your troubleshooting logic is basically good, with the caveat that you didn't specify how the coaxial cables from the EX2K and the HMI were terminated on the CTBA. Again, if either of the relays on the CTBA fail then they basically short out the connection to the TCCA/<C>/Mark V, and if one cable from either the HMI or the EX2K is plugged into each of JAI and JAJ and either of those segments stop working then it could be CTBA (or the TCCA, though usually if the ARCnet works from either JAI or JAJ then the TCCA is usually okay).

CuriousOne says his experience is that the EX2K end of the coaxial segment between the Mark V and the EX2K is usually the culprit. My experience is that it's usually the cable between the Mark V and the EX2K.

The Mark V and the operator interfaces communicate over an ARCnet-based coaxial network called the StageLink, using a proprietary ARCnet protocol (which isn't very different from standard ARCnet protocol). And, if the "link" between the Mark V and the operator interface is "broken" there will be no Diagnostic Alarm on the operator interface or <C> (or <D>, if so equipped). If the operator is a GE Mark V HMI (running MS-Windows and CIMPLICITY) there will be no data displayed on the graphic displays and no alarms in the Alarm Window(s). If the operator interface is an <I> (running MS-DOS and IDOS) the data in the fields on the displays will freeze (not change) and after a couple of seconds the time/date in corner of displays OTHER THAN THE Main Menu will change, to red, I believe, to indicate a loss of StageLink communications.

My issues with Mark VI/VIe communication (specifically over the UDH (Unit Data Highway) which is what connects the HMIs to the Mark VI/VIe) revolve around poor commissioning practices which result in different configurations on the HMIs on a site. And, the complicated--and undocumented--network configuration which uses managed network switches. And, when redundancy is purchased/supplied, things can get even more complicated and misconfigured. And, if you search some recent threads about Mark VI-to-HMI communications you will see just how complicated it can be. We're still waiting for one poster to provide feedback on how a problem was resolved at his site--and they had to have someone come to site to help resolve the problems (several control.com responders tried to help, but weren't successful). It's just very complicated.

Having said that, I just recently had the opportunity to visit and get some hands-on experience with an Emerson turbine control retrofit installation, and if the GE Mark VI/VIe networking is complicated, the Emerson stuff was at least two or five orders of magnitude more complicated. And, while Siemens SPPA-T3000 stuff has some very nice features, their networking takes several people with phone and email support to get working on a site. It's also EXTREMELY complicated and not for the faint of heart, and I believe it is also lacking in documentation which can be helpful in troubleshooting--though most sites seem to allow them to log in remotely to help with any communications issues. So, there are more complicated systems--and the Mark V system is amazingly simple, comparatively speaking.
 
CSA,

Thanks for your recommendation. If we go for recommendation one and say if we use T connector on JAI then will we leave JAJ open? I mean we don't have to use 93ohm termination resistor for JAJ. Right?
 
CSA,

Thanks for your feedback. Just one more question. If we are using BNC T connector say on JAI, and on one side of connector EX2K is connected and on the other side HMI is connected. Then what will happen if there is communication problem on EX2K? Say arcnet card is faulty or the coax cable segment between EX2K and MarkV is opened or shorted.
 
CSA

This is how we check the ribbon cables and other things in annual shutdown.

1. Cleaning of MarkV cores/panel/cards.
We power down all the cores of the MarkV from <PD> core. Then we use the compressed air to remove dust from the cards/panel (if there is any).

2. Checking of ribbon cables and coax cables
We remove the ribbon cables and while inserting the ribbon cables back we check the male part of the connector and female part on the ribbon cable. If the holes on the female side are slightly bigger (I don't know why sometimes they are) then we apply the conductive grease. I don't know any special procedure for applying the conductive grease but we apply it on the male pins of the connector one by one and take care that it should not be too much. Then we reseat the connector. Yes we don't use conductive grease for every ribbon cable.

3. Arcnet connections are checked on the CTBA card and on EX2K side. Usually the the connections on the HMI are not checked.

If you have any other recommendations for annual maintenance and checking I would be glad to hear them.

Currently we are using both JAI and JAJ on the CTBA card. We are planning to use BNC T connector.

I am happy to know that I am living in much simpler world of automation. I am a newbie in the world of automation and MarkV. Usually we study and learn, make mistakes and learn but in this automation world I have realized that we don't have any space for mistakes. So the only tool available to learn is to study or to ask. I don't know but the later I don't like but we can not learn everything just by studying. We MUST learn from others experience.
 
Regarding the comments about the complicated networks associated with MK-6e, Ovation and T3000, they are all indeed complex (and needlessly so in many if not most cases), but IMO if one invests the time to become familiar with networking, routing and switch/firewall configuration, it is not complicated.

Particularly with MK-6e & T3000, the OEM makes an effort to make it hard for the owner to acquire understanding/documentation of the network, but with persistence it is all easy enough to gain. During purchasing and commissioning efforts the owner must take pains to acquire all passwords so that a knowledgeable person can acquire the configuration information and troubleshoot in the future, not to mention change configurations to correct errors made by the OEM, which are common.

The key with such systems is the owner must have a person involved hands-on during commissioning who has strong knowledge of networking in general as well as switch/firewall configuration, otherwise that organization will likely be doomed to resorting to calling the hotline or traveling an OEM rep to the site frequently to resolve issues, and frequently the OEM reps who end up traveling to site have very weak knowledge levels (turnover is very high in those jobs) making it a science project every time.
 
The same thing you've described was happening when the EX2K is plugged into JAJ and communication between the Mark V and the EX2K is lost.
 
Mike West,

What you say is very true--and it's possible when the owner has multiple turbines and multiple sites and can afford to dedicate resources and deploy them across multiple sites.

But MANY turbine installations consist of a single unit, or two--sometimes three on combined-cycle applications--and most owners can't afford to have a single person just for turbine control networking. Even if the turbine control networking person has other duties and responsibilities it's still costly to train that individual in all the aspects of networking and network management and troubleshooting.

The OEMs need to recognize this, and, I'm sure you've encountered OEM field service who weren't sufficiently trained in all aspects of implementation and design. The systems are just too complicated and too poorly documented.
 
B

Bob Peterson

Not having anything specific to do with GT operators, but it amazes me how many end users accept equipment with all kinds of custom code, passwords, configurations, and documentation of all sorts that need to be available to maintain and repair the equipment but never bother to collect it.

I run into situations pretty where an OEM or integrator never bothered to transfer a software license to the end user and when the OEM or integrator went out of business, the end user was in a world of hurt until they could resolve it.
 
Top