Gas turbine was operating at 23MW load. Suddenly following sequence of alarms appeared and turbine got tripped.

17.03.36.125 1 Q 0062 ALM COMMON IO COMMUNICATION LOSS
17.03.36.125 1 Q 0079 ALM LUBE OIL TANK TEMPERATURE LOW
17.03.36.187 1 L43FG EVT GAS FUEL SELECTED
17.03.36.218 1 Q 0120 ALM EXHASUT TEMP HIGH
17.03.36.218 1 Q 0121 ALM EXHAUST OVER TEMP TRIP
17.03.36.218 1 Q 0391 ALM COOLING WATER TANK LOW
17.03.36.218 1 Q 0392 ALM COOLING WATER TANK LOW LOW
17.03.36.250 1 L83CAPCAN EVT LOAD SELECTION CANCEL LOGIC
17.03.36.312 1 L4T EVT MASTER PROTECTIVE / TURBINE TRIP
17.03.43.750 0 Q 0062 ALM COMMON IO COMMUNICATION LOSS

I tried to find out alarm generation cause for "Communication loss" from Mark V manuals but could not find it. Can you please help to identify the cause of this alarm and why it should trip gas turbine on exhaust temp trip? Exhaust temperature trends were showing no abnormality. Also there are two HMIs connected to Mark V system through C and D core. Data loss was on HMIs was reported for few seconds at the time of trip.

 

Divyang Shah,

TMR or SIMPLEX Mark V?

This alarm usually means there was a loss of IONET communications between <Q> and <C>. If the Mark V is a SIMPLEX, then it's going to trip.

There have been TILs (Technical Information Letters) about PROM upgrades to resolve this problem.

Another possible contributor if the TILs have been done is corrosion on cable connectors. Have you applied conductive grease to the card and cable connectors recently?

Are there any battery grounds on the Mark V? Are they intermittent, or continuous?

Does the Mark V have <DACA> AC-to-DC converters? Are the DACAs powered by an inverter or UPS?

 

Thank you very much Mr. CSA for providing feedback.

This is TMR system. There were no alarms generated related to battery ground recently. No <DACA> is present in this system. I am new to Mark System. Can you please tell me from where to get these TILs?

Thanks in advance.

 

Divyang Shah,

TILs are issued by GE. If the turbine/control system was purchased from GE, they will usually mail (post) them to the site. If the turbine/control system was purchased from a licensee of GE (BHEL, for example) the licensee would usually forward a copy (usually by post) to the site.

Some posters here on control.com have electronic versions of TILs; I do not.

One thing to note about GE TILs: They are sometimes very important to implement very quickly in order to protect the turbine and/or auxiliaries. Sometimes, they are used to get owners/operators to pay for fixing things GE has identified need to be fixed (as in pay GE for fixing problems GE created, either directly or indirectly). Many sites choose not to implement TILs, or only to implement some of them. This is done at their own peril, since most TILs seem to arrive at site after the warranty period has expired.

Lately, TILs have been used almost unabashedly to generate service business for GE by "alerting" users to conditions which are questionable issues regarding unit safety--which was the original intent of TILs, to alert users to potentially serious safety, and/or reliability, issues.

But, for this particular issue, loss of Common (<C>) I/O Communication, there was at least one TIL, if not more (for "bug" fixes identified by GE which owners/operators had to pay for). And, again, corrosion is a simple issue--which was also identified by GE in a TIL--which any technician can investigate and proactively resolve by the judicious application of conductive grease to connectors/pins of Mark V cards and cables. And, it is something which should be done on a periodic basis (say approximately every two years or so, when the control panel is cleaned, while powered-down) to prevent recurring problems as the grease dries.

 

Thanks for the reply.

On checking the logic, I found L3COMM signal used on BBL of exhaust temperature trip. I could not find the usage of this signal. Can you please help me to understand the usage of this signal in mentioned logic?

Thanks.

 

Divyang Shah,

The designers of the Mark V "decided" not to show the origination of many signals used in logic that are "written to" by operating system software. L3COMM is one of those signals; it is written to by operating system software running in each of the Mark V processors.

There really isn't much written about these signals in Mark V. I believe there was some brief notes about these kinds of signals in the CSP "Fix" utility which was informally circulated many years ago, and I seem to recall some signals were also documented (and I use that term very loosely when referring to Mark V documentation) in the Mark V Maintenance Manual (GEH-5980) and the Mark V Application Manual (GEH-6195). Other than that, I can't think of any publicly available documentation.

If I recall correctly, L3COMM is a logic "1" when the processor determines that it is communicating via the DENET (Data Exchange NETwork) with other processors.

Looking at the reported alarms in your original post it would appear that it's possible some "critical" I/O was mistakenly connected to <C> and when communication with <C> was lost for the brief period of time that <Q> tripped the turbine. But that's just an educated guess based solely on the Process Alarm text messages. It was, unfortunately, a common problem that critical inputs and outputs were improperly connected to <C>, usually by field personnel. I find a lot of this; and unfortunately there was also some of this same poor practice done by various packagers of GE-design heavy duty gas turbines that used Mark V turbine control panels, as well.

In defense of this practice, sometimes there just wasn't enough <Q> I/O available to incorporate all the critical I/O and so some had to be assigned to <C>. That just meant that the design philosophy of being able to run a turbine without <C> and <C>'s I/O isn't possible, and as long as the Customer understood that limitation everything is okay. But, many Customers never received that message....

At this writing, I don't have access to any of my old Mark V documents and software, so unfortunately I can't be of much more help on this particular logic signal.

 

As CSA said,

It seems some critical inputs might be configured to <C> and communication lost caused for tip.

Please verify the connectivity of the ribbon cable which is connected between <Q> (if TMR R, S T)and <C>.

Take care
G.Rajesh

 

Finally as indicated by Mr. CSA, the problem seems to be with EPROM versions of Mark V cards. I was able to find the TIL related to this issue.
Thank a lot.

As indicated in this TIL, the IO communication board may take all outputs to fail safe value of "0". This happened with our case and all I/Os were driven to "0" value as indicated by sequence of events. But here one more problem was experienced. GT breaker did not get open as normally "1" is required to open the breaker. GT got tripped on reverse power. Can any one provide input on fail safe configuration of output going to open the breaker? Why "1" is selected to trip the breaker? Normally in Instrumentation terms, fail safe means "0".

Thanks in advance.

 

Divyang Shah, you are leaving me a little confused.

As long as the generator was in no danger, as it appears it was safe in your case, why not let the breaker stay closed until all the load is off the unit?

No need in opening the breaker under full load if nothing is in danger. It seems that waiting till the reverse power relay picks up is a good way of determining when the breaker should be opened.

Thanks
Mark Allen

 

Mr Shah,

We have experienced numerous such loss of IO events in our 7EA fleet (2000-2002 vintage, TMR MK-5 control, 24 units across 4 sites), since initial installation. I have a question and some comments:

1. Which TIL are you referring to? The only related TIL I’m aware of is TIL-1480-2, which does not refer to the behavior of IO points being set to zero. Can you tell me the TIL number you are referring to? I think you may have found one we are unaware of.

2. We have implemented a number of logic changes (in house, not GE) to address operational issues caused by these events. This has eliminated the issues that result in trips or other problems and we are living with the events. Typically the event lasts 7-9 seconds, with most or all of the analog IO in the effected core being set to zero. These are peakers and most of the events occurs with the unit shutdown.

3. The events seem to occur more often in <R> across our fleet. For example, in 2007 we had 59 <R> loss of IO events, and 12 events in <S> & <T> combined in our 24 unit fleet.

4. This issue was identified as a Warrantee issue early on. It took a lot of nudging to get GE to admit that there was an issue and among the corrective action attempts was GE implemented TIL-1480 which did not solve the issue. GE walked away from the issue at the end of the warrantee period, it was dropped as part of a settlement. Within the past year an engineer tried to re-initiate obtaining GE assistance with the problem, the PAC response was “We are unaware of such an issue in the fleet”, and a repeat of a lot of the same boiler plate recommendations we tried in the 2002-2005 time frame.

5. Despite significant troubleshooting and efforts of various OEM and non-OEM personnel, we have not solved the problem.

6. You will find these problems do exist at other utility installations; it is discussed at length on the 7EA user forum in the 2003-2008 time frame - that forum is open to owners only, no vendors or consultants.

7. We have not noticed this issue occur on any of our 6 7FA’s with MK-5 control - 2003 vintage.

 

We are struggling with the trips on one of our 7EA gas turbine with Mark V. Paid 40,000 to GE to send a TA out and install the prom set and they also added so time delays in the logic. One week after the prom set was changed we were dispatched and tripped 4 hours into the run. We tripped on about 15 alarms coming in at the same time. The alarms and trips are not consistent on every trip. We are replacing the Mark V in November with Emerson Ovation. I am interested in what was changed in logic that can stop these trips. I have given up on GE helping with this issue. They have visited the plant three times in the last six months with no results.

 

Kurt Lammrish/Mike West,

Yes, this problem does seem to be on-going for some sites, and a PROM upgrade does seem to solve the problem for some sites but not for others.

The DENET connects <C> to <R>, <S>, and <T> and when this problem occurs it does, for some odd reason, seem to affect <R> more often then <S> or <T>.

There are basically four versions of the Mark V: There's the Mark V "A", the Mark V "Hybrid A", the Mark V "Hybrid B" and the Mark V "B" (if I recall correctly). Some versions of TCI's CARD_ID.EXE can actually report the Mark V version.

I mention this because I've never tried to determine if this is a problem that is more prevalent with one version of the Mark V than another. It's interesting that Mike West says they don't have a problem with Mark Vs used on F-class turbines. The Gas PROMset on the DCC/SDCC cards should be the same. About the only difference I can think of would be that many F-class turbines used TCQF cards, but that's just a SWAG at this point.

I definitely did find at one site recently that a trip was incorrectly configured and connected to <CD>, and moving that to <QD1> has resolved the problem--seemingly. Because I did also go through every connector in the panel and did also find corrosion on some connector pins and applied conductive grease to every connector. (The panel was also pretty dirty, and the air conditioner in the PEECC was not working properly and the environment in the PEECC was also humid as a result--not overly hot, but humid. All of the cards were removed and vacuumed one at a timed, and then re-installed.) I do know that the Customer has not complained once since then of the 'Loss of Common I/O' alarms, which they received on a fairly regular basis, along with intermittent trips. (By the way, the problem was on a 6FA Mark V panel, and I don't recall if it used TCQF cards or not.)

I, too, would be interested to know what time delays were added to the sequencing in the Mark V in an attempt to resolve this problem.

I'm always interested in trends and anomalies, and it would certainly be interesting to track this problem to one or more specific versions of Mark V panels and card sets and/or PROMsets.

I also think that this problem is exacerbated by battery charger issues (failing capacitors) and unresolved and intermittent battery grounds. Several sites I have tried to assist with this problem have had intermittent battery grounds which were toggling in and out just prior to the trip. And, I believe that some of the capacitors used in some of the battery chargers provided with new units were of suspect quality.

Also, there were some Mark V panels that were powered up from the battery charger WITHOUT the battery (poor start-up practices and construction delays and a desire to "establish communications" as quickly as possible during initial commissioning) and I believe this wreaked havoc on some of the components in the Mark V, including the <CPF> (Conditioning Power Filter) installed in the back of the bottom of the panel is a REALLY difficult position to get to. I know of a couple of sites that have bypassed the <CPF> and had really good results in eliminating intermittent alarms (not this 'Loss of Common I/O' alarm issue, but other nuisance alarms, mostly power supply-related and ground-related). While GE frowned on this practice, there were many field service personnel who did this on a regular basis. And, years after the commissioning there is no way to know if it was done or not and what the effects were.

I've seen this particular issue develop over time, as well as beginning during commissioning. I seem to recall that it started occurring on one site (Frame 9E) after a PROM change.... I think it's a strange concurrence of conditions that cause it to exist, and that makes it difficult to find. It's sad that GE has glossed over the issue, and now, of course, that they're on to the Mark VIe, and probably something newer as we write, they will not really do much to support the issue.

However, it would be interesting to try to gather data and try to find the pattern/anomaly that leads to the problem and its resolution. Perhaps one of the Users Groups is the way to do this.

Kurt, sorry to hear about your issue. We would be very interested to follow the project of converting to Ovation to know how it goes.

 

thanks CSA

 

thanks for all