First of all I am thankful to the contributors like CSA and others for sharing their experiences. I am a silent reader of this forum and found it very helpful. I want to express my gratitude for all the contributors and moderators for such a wonderful forum.

Mark V TMR system is installed on our site for Frame V gas turbines. Mark V was initially configured with <I> but in 2008 <I> was upgraded to HMI. Two gas turbines are installed at our site and each gas turbine has its own HMI in the GT compartment and we have one HMI in the control room for both the gas turbines for displaying data. The fuel used for the gas turbine is Natural Gas.

Our gas turbine system was working fine. During shutdown ground fault was observe on one of the gas turbine. The ground fault was troubleshooted. The ground fault did not disappear when we removed the power of the I/O cores. We decided to power down the Mark V panel. <R>, <S>, <T> and <C> cores were turned off from toggle buttons in <PD> core. The wires taking power to I/O cores were disconnected. We disconnected every wire going out of the TCPD card of <PD> core. The problem was still there. It was suspected that the other devices which are powered by 125VDC are the causing the ground fault problem. The breakers of each device (motors) etc were were turned off one by one but the fault still persisted. Our Mark V panel and Generator protection panel are getting the power from one breaker. The fuse of the Generator protection panel was removed but the problem still persisted. We decided to check the 125VDC batteries and its bus. The battery was leaked and its liquid was spilled out and was causing the ground fault problem. The liquid was cleaned and the ground fault problem was rectified.

After maintenance the Mark V panel was powered up. All the cores attained A7 status. After that the gas turbine was started successfully. After a week the S core was found in A6 status. Following alarm log was found
28-JAN-2015 15:54:54.718 T1 1 Q 0144 ALM COMPRESSOR INLET THERMOCOUPLES DISAGREE
28-JAN-2015 15:54:54.718 T1 1 Q 0407 ALM COMP INLET TEMP TC TROUBLE
28-JAN-2015 15:54:54.718 T1 1 Q 0408 ALM GAS FUEL TEMPERATURE HIGH SPREAD
28-JAN-2015 15:54:56.000 T1 1 T 0325 DIA LCC No arcnet communication with <S>
28-JAN-2015 15:54:57.031 T1 1 Q 0000 ALM DIAGNOSTIC ALARM <C><Q>
28-JAN-2015 15:54:58.000 T1 1 R 0325 DIA LCC No arcnet communication with <S>
28-JAN-2015 15:54:58.000 T1 1 R 1698 DIA TCE1 TMR check trouble, PTR1
28-JAN-2015 15:54:58.000 T1 1 T 1698 DIA TCE1 TMR check trouble, PTR1
28-JAN-2015 15:54:59.000 T1 1 C 0325 DIA LCC No arcnet communication with <S>
28-JAN-2015 15:54:59.031 T1 1 Q 0000 ALM DIAGNOSTIC ALARM <C><Q>
28-JAN-2015 15:55:28.000 T1 1 S 1251 DIA TCQA 4-20mA output disagrees w/ref CH.B
28-JAN-2015 15:55:29.031 T1 1 Q 0000 ALM DIAGNOSTIC ALARM <C><Q>
28-JAN-2015 15:55:30.000 T1 1 S 0312 DIA LCC: Processor re-booted independently
28-JAN-2015 15:55:30.000 T1 1 S 0326 DIA LCC No arcnet communication with <T>
28-JAN-2015 15:55:30.093 T1 1 Q 0000 ALM DIAGNOSTIC ALARM <C><Q>
28-JAN-2015 15:55:34.000 T1 0 T 0325 DIA LCC No arcnet communication with <S>
28-JAN-2015 15:55:34.000 T1 0 R 0325 DIA LCC No arcnet communication with <S>
28-JAN-2015 15:55:35.000 T1 0 C 0325 DIA LCC No arcnet communication with <S>
28-JAN-2015 15:55:36.000 T1 0 S 0312 DIA LCC: Processor re-booted independently
28-JAN-2015 15:55:36.000 T1 0 S 0326 DIA LCC No arcnet communication with <T>
28-JAN-2015 15:58:43.625 T1 0 Q 0087 ALM TURBINE INCOMPLETE SEQUENCE
28-JAN-2015 16:00:58.343 T1 1 Q 0099 ALM TURBINE AIR INLET TROUBLE
28-JAN-2015 16:01:01.500 T1 0 Q 0397 ALM EX2K OPERATOR SELECTED EXCITATION OFF
28-JAN-2015 16:04:15.406 T1 1 Q 0117 ALM HYDRAULIC PROTECTIVE TROUBLE
28-JAN-2015 16:04:16.468 T1 1 Q 0167 ALM GAS RATIO VALVE POSITION SERVO TROUBLE
28-JAN-2015 16:04:18.000 T1 1 R 1696 DIA TCE1 TMR check trouble, ETR1
28-JAN-2015 16:04:18.000 T1 1 R 1697 DIA TCE1 TMR check trouble, ETR2
28-JAN-2015 16:04:18.000 T1 1 T 1696 DIA TCE1 TMR check trouble, ETR1
28-JAN-2015 16:04:18.000 T1 1 T 1697 DIA TCE1 TMR check trouble, ETR2
28-JAN-2015 16:04:19.031 T1 1 Q 0000 ALM DIAGNOSTIC ALARM <C><Q>
28-JAN-2015 16:04:23.000 T1 0 T 1698 DIA TCE1 TMR check trouble, PTR1
28-JAN-2015 16:04:25.000 T1 0 R 1698 DIA TCE1 TMR check trouble, PTR1
28-JAN-2015 16:12:25.000 T1 0 R 1696 DIA TCE1 TMR check trouble, ETR1
28-JAN-2015 16:12:25.000 T1 0 R 1697 DIA TCE1 TMR check trouble, ETR2
28-JAN-2015 16:12:25.000 T1 0 T 1696 DIA TCE1 TMR check trouble, ETR1
28-JAN-2015 16:12:25.000 T1 0 T 1697 DIA TCE1 TMR check trouble, ETR2

All the cables and connectors were checked thoroughly and found OK. We decided to re-boot the core by the toggle switch in the <PD> core. The S core was initialized and successfully attained the A7 status.
After two three days again S core was found in A6 status. Alarm log for this period is given below

31-JAN-2015 13:49:40.000 T1 0 C 3785 DIA Voter mismatch, <S> DV_ERR
31-JAN-2015 14:01:31.218 T1 1 Q 0144 ALM COMPRESSOR INLET THERMOCOUPLES DISAGREE
31-JAN-2015 14:01:31.218 T1 1 Q 0407 ALM COMP INLET TEMP TC TROUBLE
31-JAN-2015 14:01:31.218 T1 1 Q 0408 ALM GAS FUEL TEMPERATURE HIGH SPREAD
31-JAN-2015 14:01:32.000 T1 1 C 0325 DIA LCC No arcnet communication with <S>
31-JAN-2015 14:01:32.031 T1 1 Q 0000 ALM DIAGNOSTIC ALARM <C><Q>
31-JAN-2015 14:01:34.000 T1 1 R 0325 DIA LCC No arcnet communication with <S>
31-JAN-2015 14:01:34.000 T1 1 R 1698 DIA TCE1 TMR check trouble, PTR1
31-JAN-2015 14:01:34.000 T1 1 T 0325 DIA LCC No arcnet communication with <S>
31-JAN-2015 14:01:34.000 T1 1 T 1698 DIA TCE1 TMR check trouble, PTR1
31-JAN-2015 14:01:35.031 T1 1 Q 0000 ALM DIAGNOSTIC ALARM <C><Q>
31-JAN-2015 14:01:59.000 T1 1 Q 0093 ALM COMMON IO COMMUNICATION LOSS
31-JAN-2015 14:01:59.750 T1 0 Q 0093 ALM COMMON IO COMMUNICATION LOSS
31-JAN-2015 14:02:04.000 T1 1 S 1251 DIA TCQA 4-20mA output disagrees w/ref CH.B
31-JAN-2015 14:02:05.031 T1 1 Q 0000 ALM DIAGNOSTIC ALARM <C><Q>
31-JAN-2015 14:02:06.000 T1 1 S 0312 DIA LCC: Processor re-booted independently
31-JAN-2015 14:02:06.000 T1 1 S 0324 DIA LCC No arcnet communication with <R>
31-JAN-2015 14:02:07.031 T1 1 Q 0000 ALM DIAGNOSTIC ALARM <C><Q>
31-JAN-2015 14:02:11.000 T1 0 R 0325 DIA LCC No arcnet communication with <S>
31-JAN-2015 14:02:11.000 T1 0 T 0325 DIA LCC No arcnet communication with <S>
31-JAN-2015 14:02:12.000 T1 0 C 0325 DIA LCC No arcnet communication with <S>
31-JAN-2015 14:02:15.000 T1 0 S 0312 DIA LCC: Processor re-booted independently
31-JAN-2015 14:02:15.000 T1 0 S 0324 DIA LCC No arcnet communication with <R>

The core was again re-booted and again it attained the A7 status successfully. Keeping in view 0325 diagnostic alarm we decided to change the LCC card as our first step if the core is found stuck in A6 status again.

Two weeks later again the S core was found stuck in A6 status. The log of this time is given below

14-FEB-2015 17:19:22.593 T1 1 Q 0144 ALM COMPRESSOR INLET THERMOCOUPLES DISAGREE
14-FEB-2015 17:19:22.593 T1 1 Q 0407 ALM COMP INLET TEMP TC TROUBLE
14-FEB-2015 17:19:22.593 T1 1 Q 0408 ALM GAS FUEL TEMPERATURE HIGH SPREAD
14-FEB-2015 17:19:24.000 T1 1 T 0325 DIA LCC No arcnet communication with <S>
14-FEB-2015 17:19:24.000 T1 1 R 0325 DIA LCC No arcnet communication with <S>
14-FEB-2015 17:19:26.000 T1 1 R 1698 DIA TCE1 TMR check trouble, PTR1
14-FEB-2015 17:19:26.000 T1 1 T 1698 DIA TCE1 TMR check trouble, PTR1
14-FEB-2015 17:19:27.000 T1 1 C 0325 DIA LCC No arcnet communication with <S>
14-FEB-2015 17:19:56.000 T1 1 S 1251 DIA TCQA 4-20mA output disagrees w/ref CH.B
14-FEB-2015 17:19:58.000 T1 1 S 0312 DIA LCC: Processor re-booted independently
14-FEB-2015 17:19:58.000 T1 1 S 0326 DIA LCC No arcnet communication with <T>
14-FEB-2015 17:20:02.000 T1 0 C 0325 DIA LCC No arcnet communication with <S>
14-FEB-2015 17:20:03.000 T1 0 R 0325 DIA LCC No arcnet communication with <S>
14-FEB-2015 17:20:05.000 T1 0 T 0325 DIA LCC No arcnet communication with <S>
14-FEB-2015 17:20:05.000 T1 0 S 0312 DIA LCC: Processor re-booted independently
14-FEB-2015 17:20:05.000 T1 0 S 0326 DIA LCC No arcnet communication with <T>

While we were checking the status on the display of LCC we decided to check the I/O status of the cards. As we opened the door of the core the S core attained the A7 status. We were quite amazed. Just to check we again closed the door with force the S core initialized. The we opened the door again and just tapped on the left side of keypad the (S core was in A7 status at this time) S core initialized once again. The S core attained the A7 status but it was giving following errors on the display

• 2 DCC errors
• Missing I/O Card
• I/O CFG errors

We reinitialized the core from the push button in the S core. The S core again attained the A7 status but again these errors were shown. We re-booted the core from the toggle switch in the <PD> core and the S core attained A7 status with no errors.

It is worth mentioning here that the ground fault has again appeared on the Mark V.

By keeping in view the above behavior can anyone suggest what the actual problem is?
We are planning to change the LCC card and check the cables and connectors once again. Are we on right track ?

I want to mention one more thing. This time we are getting two diagnostic alarms of S core which is still persisting.
• D273 IOMA power supply out of limits P15
• D274 IOMA power supply out of limits N15

We have checked the diagnostic counters. P15 and N15 were in their limits (15 +-0.75V). P5 was not in the limit 4.75 to 5.25V. P5 was showing 5.62V. We are also planning to rectify this by adjusting R29 potentiometer on TCPS card.

When P15 and N15 are in their limits but still the alarm is displayed on HMI can we deduce that we have faulty DCC card?

Please let me know if any additional information is required.

 

Thanks for a very comprehensive bit of troubleshooting on what looks like a pretty nasty fault. I reckon you are on the right track changing the LCC card, this troubleshooting task is going to be a process of elimination anyway. The missing I/O card error is for sure a clue, if you get this again, try and look at the controller card status and see if you pin down the failing card. Unfortunately, ground faults on MKV panels can give some really strange faults and, although it is a pain in the ass, I would also try and get the ground fault cleared, there may be a multiple ground, if you get a 125 +- ground plus a 28V ground I've seen it cause loss of I/O or strange controller re-boots.

Good luck and report back

 

Esoteric_Stone,

I think this, or a very similar problem, was recently posted and responded to.

At any rate, I'm going to be brief. The most likely causes are, in order of likelihood from highest to lowest:

--Bad ribbon cable running between the LCC, DCC and TCQA card; this cable has no pull tabs and has been known to be easily damaged when cards are replaced if great care has not been taken when disconnecting the ribbon cable from its receptacles

--Problem with the LCC and/or DCC

--Problem with the TCQA card

--Intermittent problem with PROMset "legs" (pins) in PROM sockets; as with ribbon cables/connectors they can also be compromised by corrosion

--Problem with TCPS card or one of the fuses on the TCPS card

The LCC keypad can be used to reset diagnostic alarm messages on the LCC display; I believe the procedure is described in the Mark V Maintenance Manual, GEH-5980. And, I also believe it's been described in one or more threads on control.com. So, you don't have to reset the processor to clear diagnostic alarm messages on the LCC display. I believe many LCC display diagnostic alarm messages can also be cleared by ussuing a Diagnostic Alarm Reset from the HMI.

Please write back to let us know how you fare.

 

Thanks glenmorangie and CSA for you replies. I will write back as soon as we will perform any activity.

 

We have checked the IO states on the LCC display while the S core was in A7 status. Following IO cards were present (Their IO status is also shown).<pre>
Object ID 1 TCQA A7
Object ID 4 TCDA A7
Object ID 8 LCCq A7
Object ID 12 DCCq A7
Object ID 13 IOMA A7
Object ID 15 TCEA A7</pre>
When the door was opened the cards of core were disturbed by pressing them. The core restarted. This time the core did not attain A7 status (Luckily). It was again stuck on A6 status.

The IO status of cards was again checked. Some cards were showing A6 status and others were not appearing in the IO states (As shown below).<pre>
Object ID 1 TCQA A6
Object ID 4 TCDA Not Appeared in IO states
Object ID 8 LCCq A6
Object ID 12 DCCq A6
Object ID 13 IOMA Not Appeared in IO states
Object ID 15 TCEA Not Appeared in IO states</pre>
By looking in to the drawings we deduced cards not appearing in IO states are the cables of the cards not appeared in IO states on LCC display are going the TCQC card. The TCQC card was powered by TCPS card via JC ribbon cable. When this JC ribbon cable was pressed on TCPS card the core rebooted. We suspected that this ribbon cable is loose. S core was powered down. All the cables from TCPS card were taken out. TCPS card itself was taken out of the core. We inspected the TCPS card for any physical damage. Fuses on TCPS card were also checked. The conductive grease was applied on all the ribbon cable connectors.

TCPS card was installed again. All the ribbon cables were seated carefully. The core was powered up. It attained the A7 status successfully (All the IO cards were showing A7 status on LCC display).
We again tried to disturb the cards by pressing the cards but nothing happened (The core remained in A7 status this time).

We suspected that the JC ribbon cable was not properly seated which was giving power to TCQC card. Due to loss of power on TCQC card, the core was rebooting.

The turbine is running with all the cores in A7 status. If anybody has any other suggestion or idea then please share it.

 

Esoteric_stone,

The TCDA and TCEA cards are connected to the TCQC via the IONET cable. The IOMA (I/O Master) isn't really a "card"; it's a chip on the DCC card.

It could, indeed be that the TCQC card is experiencing problems, or that there is some problem with the IONET cable running from the TCQC card, through the TCEA card in Loc. 3 of <P> to the TCDA card in Loc. 2 of <QD1>.

You had said that you had checked all the cables for tightness....

The first couple of alarms in the Alarm Printouts you provided were for T/Cs connected to the TCQA card; that's why I suspected the cable between the LCC, DCC and TCQA cards--it's known to be a problem.

But, if the TCPS is intermittent or the cable providing power to the various cards in the processors isn't properly seated, that could also be the problem.

Hopefully, you've solved this problem--only time will tell!

 

CSA

The gas turbine is running without any problem. I hope that the problem is solved.

Yes we checked the cables first time, but at that time the core was not rebooting when we were pressing the connectors or cards. It just happened the second time. I hope that the problem is solved but time will tell, just like you said.

I have tried to find more information about IOMA in manuals but could not find much detail. Can you explain what is the purpose of IOMA and what will happen if IOMA is not in A7 state?

And please also explain what will happen if any other card is not in A7 state?

What will happen if LCC display is showing DCC errors when the gas turbine is running and A7 status is also shown?

 

While running with A7 status, error messages on LCC will not harm the machines. These can be easliy cleared from the LCC using the keypads.

from LCC do the following
1. press LCC/DCC
2.Inc / dec to get clear errors (system error something like, may be on s.no 7)
3.enter to find total error
4.clr

it'll show **ABORT**

Now you will see only A7.

take care
g.rajesh

 

CSA

Yes it was only the matter of time. The problem was appeared again just two weeks after the rectification. We changed the TCPS card and the problem was resolved. Now the core is running for last six months without rebooting.

Gragesh
Thanks for your reply.

 

Esoteric_Stone,

Thank you for the feedback!