Rebooting of Mark V control processors during Lightening

V

Thread Starter

VRVR

We are facing a problem of Mark V controllers <R>, <S>, <T> whenever lightening strikes near by electrical switch yard.

It was observed that some times, lightening struck to the near by electrcial switch yard, due to which grid supply trips. At the same time all the processors of T3 Mark V reboots on its own, This causes tripping of T3 machine.

Is there any specific experience elsewhere or is there any specific methods to be taken to protect from lightening for Mark V controls?
 
What's the difference between this post and a previous one which seems extremely similar in many respects:

http://www.control.com/thread/1026246171

No useful feedback was provided then; it was never responded to by the originator.

Have you used any kind of monitoring and recording equipment to monitor the power supply(s) to the Mark V during the lightning storms?

Have you used any kind of monitoring and recording equipment to monitor the potential differences between the earth pits during the lightning storms?

If you have one or more remote operator interfaces which are connected to the Mark Vs with copper coaxial cable (that is, no fiber optic cable between the remote operator interface and the Mark V) *and* that cable is just running in a cable tray between the remote location and the Mark V, then that could be another reason for your problem. Even if it's in a conduit in an elevated pipe tray or cable tray between buildings, it can still problematic. If the cable is copper coax, is it shielded cable, and is the cable shield properly grounded?

If the Mark V is powered by a (nominal) 125 VDC battery AND it also has one or more <DACA>s (AC-to-DC converters) this could be another source of your problem. <DACA>s are *NOT* filters, and if a voltage spike is getting into the AC power source and is making it through the <DACA>(s) then this could be yet another reason for your problem.

If a voltage spike is making it through the battery charger and getting to the Mark V power supplies this could be yet another reason for your problem.

This is just a *partial* list of things which could be causing the problems at your site; none of them are related to the Mark V except that the Mark V is affected by them. It really sounds like you need someone knowledgeable in lightning suppression to visit your site, analyze the situation, and make some recommendations. (Other than creating separate earth pits.)

One previous lightning problem thread had a comment to the effect that it's not possible to control out of spec lightning. Lightning is very difficult to "react to". In that same thread, there was mention that of two more important critera: a low resistance path to ground (thousands of amps through a fraction of an ohm can be a very big voltage "drop"), and that not just low resistance paths were required, but that the path must also be low reactance (though there was no additional info provided on that).

If you experience lots of lightning strikes, then your plant should have some kind of suppression or protection. There are systems and consultants who are knowledgeable and can help; there's just too many things we don't know about your plant to be able to help you here (over and above creating separate earth pits). If you're asking if other sites with Mark Vs have had problems with lightning strikes, yes, though from the sounds of it your site experiences them quite frequently and possibly quite violently, as well. Many have solved their problems by taking action on some of the possible causes listed here or taken steps to reduce the problems, and some just live with it.

I will argue the problem is not with the Mark V, it's with the plant design. Because there are many more plants with Mark Vs that experience lightning storms and lightning strikes that don't have this type of problem than do. So, again, the problem is not likely with the Mark V, but rather with the installation of the Mark V and possibly with the design of the plant.

It just might be though, that lightening is much more disruptive than lightning. (Sorry; couldn't resist that. I tried; I really tried. But, we need some levity once in a while here.)
 
Thank you CSA for your valuable input.

I think the earlier post is by one of my colleagues, however I have yet to confirm with him.

We did not used any monitoring/recording equipment earlier. However during the last one month period an oscilloscope is in line for monitoring the power supply levels.

No we haven't used any kind of monitoring and recording equipment to monitor the potential differences between the earth pits during the lightning storms. We have checked the earth resistances of the earth pits and found to be within acceptable limits, less than 0.25 Ohms.

Our operator stations are within the same building where the Mark V panels are installed, coaxial copper cables are connected.

Power supply to Mark V is fed from a 125 VDC batter charger with a battery bank. Input this system is AC and being galvanically isolated with a battery bank in between. In this scenario how an input AC spike can result in a DC output spike is not very clear. Also we request you to note there are four such machines with identical setup running in the same plant in close proximity to the switch yard where lightning strikes. However we have not noted any such issues of rebooting of processors in other machines. This created a major doubt whether some thing is wrong with the specific control panel earthing.

It was also to be noted that only <R>, <S> and <T> computers rebooted, one after another. Before one processor could come to A7 stage other processor rebooted and hence machine tripped. <C> computer did not reboot.

Your evaluation and further suggestions are requested.
 
I think you have pretty well demonstrated that there is something amiss with the installation of the one unit out of four which is experiencing problems. It may be extremely difficult to track down, but now that you are gathering some data, you should be able to get to the bottom of the dilemma if you are persistent.

You also haven't told us whether or not the problem has persisted from original commissioning or whether it started some time afterwards. If it started sometime afterwards, can you think of anything that was done prior to the re-booting events occurring? Was a card or cards replaced? Was the panel powered down for some maintenance outage?

I don't think I've ever seen galvanic isolation between a battery charger (which has a DC output) and a battery and the load it supplies. I'm no expert in galvanic isolation, but I think it wouldn't allow the DC output from the battery charger to get to the battery or the load it supplies. I have seen the battery chargers provided with most heavy duty gas turbine applications have spikes in their DC output when there is a spike on the AC input, such as from a lightning surge. I've even seen the output filter capacitors fail from the spike on the output. This was caused when the lighting surge went through the station auxiliary transformer to the 440 VAC bus. Actually, other equipment was damaged, too.

I don't think I've ever seen the note that first <R> re-booted, then <S>, and then <T> in this or any previous post. I think they all said that <R>, <S>, and <T> re-booted and the implication was that they re-booted simultaneously. So this is new information. I have never seen such a thing, and if <C> is running, then there should be a printed record of the Diagnostic Alarms annunciated when this is occurring. That should and would most likely be at least somewhat helpful.

If any one of <R> or <S> or <T> is already less than A7 and any other processor re-boots, then the turbine will trip. A7 means the processor is participating in voting with the other processors (it is synch'ed with the other processors) and its outputs are enabled. If its outputs are not enabled and any other processor's outputs are disabled by a re-boot of the processor, then the two-out-of-three voting thing is gonna trip the turbine. Period.

It would also be helpful to know what Diag. Alarms are present before the lightning strike causes the event.

Let us know what the o'scope tells you the next time an event occurs. I also think it would be extremely interesting to know what the difference in the ground potentials is between the unit which is experiencing the problem and any unit which is not when a lightning strike caused this event to occur.
 
I think processor's (R, S, T) toggle switches in <PD> core are not grounded. Please check the same.

Regards,
Tushar
 
This is what I find interesting. C core is not part of the power loss. Although, ALL cores are fed from PD core. The unique difference is that C core is connected to CD core. RST are connected to the QD1/QD2 cores with separate board for each core but with usually one to two power feeds from PD core for the DTBx boards. What are the voltage readings from the diagnostics counters display for R, S, and T? Many installations do not have a alarm for voltage imbalance until breaker closure.

The following statement is to get CSA thinking. 125 VDC input and outputs on DTBA, B, C, D located on the QD core. A single or multiple grounds exists, perhaps not detected by the Mark V, but a megger would find the insulation leakage. The lightning strike reveals the ground, then draws too much power from the QD cores and causes R, S, or T to reboot.

Without diagnostic alarms, I can only speculate!
I believe an input or output on QD core will be found to be source of the problem. Perhaps a digital feedback from an AC motor that was running during the lightning strike.
 
CTTech,

The same 125 VDC source feeds <CD> and <QDn>; in fact, the positive 125 VDC (the (107) "bus") from <CD> is commonly used to supply contact inputs connected to <QDn>, and vice versa. This is what makes troubleshooting grounds so difficult on Speedtronic turbine control systems, and drives people crazy once they finally understand what's being done.
 
CSA

Yes. All cores use the same power source(PD), but a single set of wires feed DTBA and is then daisy chained to DTBB. Another set of wires for DTBC and DTBD from PD core. Wires have current limitations than when reached could lower voltage to 90vdc perhaps causing QD power supply to switch off momentarily. Although the 90VDC I recall probably has something to do with an old MK I invertor, GE might have included that criteria in a MKV power supply board. I will research that.

The point was that QD contains a board for each core RST and a single power supply for all three. If R,S, or T loose connection to their respective counterparts in QD many errors will occur, *perhaps* even a reboot of that core.

I have never witnessed a reboot because of this but I was trying to point out the difference between CD and QD core in respect to C and RST.

I was hoping to draw out some long archived data from your experience in respect to why C core was not affected and RST were affected. I did not even approach the shared IO located on R core or how the triple redundant stuff has one input on R, and input on S, and a input on T.

I will blow the vARs off a few tonight and perhaps that will cloud my thinking enough to recall some of my long archived data.

CTTech
 
CTTech,
Contact inputs are optically isolated from the electronics on the TCDA cards. Even though the wires that feed the DTBA/DTBB on <QD1> are separately fused in <PD>, it's the same 125 VDC source (or should be).

Each of the TCDAs in <QD1> is powered from the TCPS in the core the TCDA is associated with. Power down <R>, and only the TCDA in Loc. ! of <QD1> should power down.

I have no idea why <C> isn't re-booting when <R>, <S>, and <T> are re-booting, nor why <R>, <S>, and <T> are re-booting one at a time. That is truly a mystery. But, even though it's believed all the units are exactly the same on a multi-unit site are exactly the same even if they were all installed at the same time, they almost never are. So, there's some difference that's leading to this anomaly.

The really "interesting" part is that we have learned they don't all re-boot simultaneously (which is what I've always experienced during a lightning strike), but, according to recent information <R> re-boots, and before it returns to I/O Status A7 <S> re-boots, and that's what causes the turbine to trip: two of the processor's outputs are not enabled.

At least that's what we can surmise from the information that we've been provided. Further, it appears the re-booting continues with <T> then re-booting, in some kind of "logical" progression, which is even stranger. And, this is all being associated with a lightning strike, which lasts for a split second. And we all know how long it takes for a Mark V processor to re-boot.

It's really difficult to imagine how this kind of "delayed reaction" to a lightning strike can cause all three processors to re-boot one after another. I'm just completely stumped about this; very curious, but completely stumped.

I would venture that a *complete* list of the Diagnostic Alarms present before, during, and after the event would yield some interesting information. Also, I'm going to venture that there is at least a partial ground on one leg of the 125 VDC supply, but that's just a SWAG (most sites have at least a partial ground at any given time during operation, and sites which experience a lot of rain and moisture and humidity usually have more than their share of grounds).

But, I'll also venture that we'll never see a *complete* list of ALL Diag. Alarms before, during, and after the event because, well, they're Diag. Alarms and most sites ignore them because they're just nuisances. After all the unit doesn't trip because of Diag. Alarms, right? So, why pay any attention to them? And, every time I've asked a site to provide a complete list of Diag. Alarms before during and after an event it just never happens. The printer wasn't working; there wasn't any paper; the ribbon was bad; we lost the printouts; we don't use the printer because it wastes paper; the paper was jammed in the low-tech dot matrix printer; the orangutan ate the printouts; you get the idea. Or, they only provide the list of Diag. Alarms they think might be relevant.

Lastly, in a TMR Mark V control system there are no redundant contact inputs connected <R>, <S>, and <T>. The closest thing to that is the three redundant pressure switches for the gas- and/or liquid fuel trip oil pressure sensing (63HG-n and 63HL-n), which are "fanned" to all three processors. The designers of the Mark V TMR system intended them to be independently connected to <R>, <S>, and <T>, but there was no cost-effective method to do so. Even with the Mark VI and Mark VIe, which could accommodate individual contact inputs to <R>, <S>, and <T>, GE still fans them out to all three processors. (Old habits die hard!)

I sincerely doubt that we'll be able to solve this mystery on control.com. I'd be extremely happy if we could, but there's just a lot of things that, as explained, are extremely odd and illogical.
 
Dear CSA/CTTech,

I am encouraged by the responses and valuable inputs provided by you. I have collected the diagnostic data during the last two trips (pasted below).

I am extremely sorry for the wrong information provided earlier that the processor re-booted one after another. In fact all of them re-booted simultaneously.

**************May 10th Trip**********************

Time Unit S P Drop/Point Typ Description
------------------------ ------ -- ------------------------------- --- ------------------------------
[63AT-3]
10-MAY-2008 23:15:25.000 T3 1 C 0007 DIA DCC Unable to get DPM buffer for xmit
10-MAY-2008 23:15:33.000 T3 0 R 0274 DIA IOMA Power supply out of limits, N15
10-MAY-2008 23:15:33.000 T3 0 R 0273 DIA IOMA Power supply out of limits, P15
10-MAY-2008 23:15:33.000 T3 0 S 0274 DIA IOMA Power supply out of limits, N15
10-MAY-2008 23:15:50.000 T3 1 C 0009 DIA DCC DPM: Invalid destination address
10-MAY-2008 23:15:54.000 T3 1 R 0273 DIA IOMA Power supply out of limits, P15
10-MAY-2008 23:15:54.000 T3 1 R 0274 DIA IOMA Power supply out of limits, N15
10-MAY-2008 23:15:54.000 T3 1 R 0312 DIA LCC: Processor re-booted independently
10-MAY-2008 23:15:54.000 T3 1 R 0963 DIA TCD1 Relays dropped due to IONET failure
10-MAY-2008 23:15:54.000 T3 1 R 1709 DIA TCE1 Loopback, relay, PTR2
10-MAY-2008 23:15:54.000 T3 1 R 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
10-MAY-2008 23:15:54.000 T3 1 S 0274 DIA IOMA Power supply out of limits, N15
10-MAY-2008 23:15:54.000 T3 1 S 0312 DIA LCC: Processor re-booted independently
10-MAY-2008 23:15:54.000 T3 1 S 0963 DIA TCD1 Relays dropped due to IONET failure
10-MAY-2008 23:15:54.000 T3 1 S 1709 DIA TCE1 Loopback, relay, PTR2
10-MAY-2008 23:15:54.000 T3 1 S 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
10-MAY-2008 23:15:54.000 T3 1 T 0312 DIA LCC: Processor re-booted independently
10-MAY-2008 23:15:54.000 T3 1 T 0963 DIA TCD1 Relays dropped due to IONET failure
10-MAY-2008 23:15:54.000 T3 1 T 1709 DIA TCE1 Loopback, relay, PTR2
10-MAY-2008 23:15:54.000 T3 1 T 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
10-MAY-2008 23:15:55.000 T3 1 C 0324 DIA LCC No arcnet communication with <R>
10-MAY-2008 23:15:55.000 T3 1 C 0325 DIA LCC No arcnet communication with <S>
10-MAY-2008 23:15:55.000 T3 1 C 0326 DIA LCC No arcnet communication with <T>
10-MAY-2008 23:16:03.000 T3 0 R 0312 DIA LCC: Processor re-booted independently
10-MAY-2008 23:16:03.000 T3 0 S 0312 DIA LCC: Processor re-booted independently
10-MAY-2008 23:16:03.000 T3 0 T 0312 DIA LCC: Processor re-booted independently
10-MAY-2008 23:16:05.000 T3 0 C 0324 DIA LCC No arcnet communication with <R>
10-MAY-2008 23:16:05.000 T3 0 C 0325 DIA LCC No arcnet communication with <S>
10-MAY-2008 23:16:05.000 T3 0 C 0326 DIA LCC No arcnet communication with <T>
10-MAY-2008 23:16:18.000 T3 1 R 1696 DIA TCE1 TMR check trouble, ETR1
10-MAY-2008 23:16:18.000 T3 1 R 1697 DIA TCE1 TMR check trouble, ETR2
10-MAY-2008 23:16:18.000 T3 1 S 1696 DIA TCE1 TMR check trouble, ETR1
10-MAY-2008 23:16:18.000 T3 1 S 1697 DIA TCE1 TMR check trouble, ETR2
10-MAY-2008 23:16:18.000 T3 1 T 1696 DIA TCE1 TMR check trouble, ETR1
10-MAY-2008 23:16:18.000 T3 1 T 1697 DIA TCE1 TMR check trouble, ETR2
10-MAY-2008 23:16:23.000 T3 0 T 0963 DIA TCD1 Relays dropped due to IONET failure
10-MAY-2008 23:16:23.000 T3 0 T 1709 DIA TCE1 Loopback, relay, PTR2
10-MAY-2008 23:16:23.000 T3 0 T 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
10-MAY-2008 23:16:33.000 T3 0 R 1709 DIA TCE1 Loopback, relay, PTR2
10-MAY-2008 23:16:33.000 T3 0 R 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
10-MAY-2008 23:16:33.000 T3 0 S 0963 DIA TCD1 Relays dropped due to IONET failure
10-MAY-2008 23:16:33.000 T3 0 S 1709 DIA TCE1 Loopback, relay, PTR2
10-MAY-2008 23:16:33.000 T3 0 S 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
10-MAY-2008 23:16:33.000 T3 0 R 1696 DIA TCE1 TMR check trouble, ETR1
10-MAY-2008 23:16:33.000 T3 0 R 1697 DIA TCE1 TMR check trouble, ETR2
10-MAY-2008 23:16:33.000 T3 0 S 1696 DIA TCE1 TMR check trouble, ETR1
10-MAY-2008 23:16:33.000 T3 0 S 1697 DIA TCE1 TMR check trouble, ETR2
10-MAY-2008 23:16:35.000 T3 0 R 0963 DIA TCD1 Relays dropped due to IONET failure
10-MAY-2008 23:16:35.000 T3 0 T 1696 DIA TCE1 TMR check trouble, ETR1
10-MAY-2008 23:16:35.000 T3 0 T 1697 DIA TCE1 TMR check trouble, ETR2
10-MAY-2008 23:17:19.000 T3 0 C 0007 DIA DCC Unable to get DPM buffer for xmit
10-MAY-2008 23:17:19.000 T3 0 C 0009 DIA DCC DPM: Invalid destination address
10-MAY-2008 23:27:54.000 T3 1 R 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:27:54.000 T3 1 S 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:27:54.000 T3 1 T 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:28:03.000 T3 0 R 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:28:03.000 T3 0 S 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:28:03.000 T3 0 T 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:33:20.000 T3 1 T 1744 DIA TCE1 Flame detector #1 out of limits
10-MAY-2008 23:33:29.000 T3 0 T 1744 DIA TCE1 Flame detector #1 out of limits
10-MAY-2008 23:34:38.000 T3 1 R 1744 DIA TCE1 Flame detector #1 out of limits
10-MAY-2008 23:34:38.000 T3 1 S 1744 DIA TCE1 Flame detector #1 out of limits
10-MAY-2008 23:34:38.000 T3 1 T 1744 DIA TCE1 Flame detector #1 out of limits
10-MAY-2008 23:34:45.000 T3 0 T 1744 DIA TCE1 Flame detector #1 out of limits
10-MAY-2008 23:34:47.000 T3 0 R 1744 DIA TCE1 Flame detector #1 out of limits
10-MAY-2008 23:34:47.000 T3 0 S 1744 DIA TCE1 Flame detector #1 out of limits
10-MAY-2008 23:35:04.000 T3 1 T 1744 DIA TCE1 Flame detector #1 out of limits
10-MAY-2008 23:35:04.000 T3 1 R 1744 DIA TCE1 Flame detector #1 out of limits
10-MAY-2008 23:35:04.000 T3 1 S 1744 DIA TCE1 Flame detector #1 out of limits
10-MAY-2008 23:35:11.000 T3 0 S 1744 DIA TCE1 Flame detector #1 out of limits
10-MAY-2008 23:35:13.000 T3 0 T 1744 DIA TCE1 Flame detector #1 out of limits
10-MAY-2008 23:35:13.000 T3 0 R 1744 DIA TCE1 Flame detector #1 out of limits
10-MAY-2008 23:36:19.000 T3 0 S 0274 DIA IOMA Power supply out of limits, N15
10-MAY-2008 23:36:20.000 T3 1 S 0274 DIA IOMA Power supply out of limits, N15
10-MAY-2008 23:39:43.000 T3 0 S 0274 DIA IOMA Power supply out of limits, N15
10-MAY-2008 23:39:48.000 T3 1 S 0274 DIA IOMA Power supply out of limits, N15
10-MAY-2008 23:41:00.000 T3 1 T 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:41:00.000 T3 1 R 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:41:00.000 T3 1 S 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:41:07.000 T3 0 R 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:41:07.000 T3 0 T 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:41:09.000 T3 0 S 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:52:14.000 T3 1 R 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:52:14.000 T3 1 S 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:52:14.000 T3 1 T 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:52:21.000 T3 0 R 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:52:21.000 T3 0 S 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:52:23.000 T3 0 T 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:54:00.000 T3 1 T 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:54:00.000 T3 1 R 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:54:00.000 T3 1 S 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:54:07.000 T3 0 R 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:54:09.000 T3 0 S 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:54:09.000 T3 0 T 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:58:44.000 T3 1 T 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:58:44.000 T3 1 R 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:58:44.000 T3 1 S 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:58:51.000 T3 0 S 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:58:51.000 T3 0 T 1747 DIA TCE1 Flame detector #4 out of limits
10-MAY-2008 23:58:53.000 T3 0 R 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:10:54.000 T3 1 R 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:10:54.000 T3 1 S 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:10:54.000 T3 1 T 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:11:01.000 T3 0 R 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:11:01.000 T3 0 T 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:11:03.000 T3 0 S 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:17:30.000 T3 1 R 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:17:30.000 T3 1 S 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:17:30.000 T3 1 T 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:17:37.000 T3 0 T 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:17:39.000 T3 0 R 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:17:39.000 T3 0 S 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:42:58.000 T3 1 R 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:42:58.000 T3 1 S 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:42:58.000 T3 1 T 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:43:13.000 T3 0 S 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:43:13.000 T3 0 T 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:43:15.000 T3 0 R 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:43:26.000 T3 1 R 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:43:26.000 T3 1 S 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:43:26.000 T3 1 T 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:43:35.000 T3 0 R 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:43:35.000 T3 0 S 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 00:43:35.000 T3 0 T 1747 DIA TCE1 Flame detector #4 out of limits
11-MAY-2008 01:10:10.000 T3 1 R 1709 DIA TCE1 Loopback, relay, PTR2
11-MAY-2008 01:10:10.000 T3 1 R 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
11-MAY-2008 01:10:10.000 T3 1 S 1709 DIA TCE1 Loopback, relay, PTR2
11-MAY-2008 01:10:10.000 T3 1 S 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
11-MAY-2008 01:10:10.000 T3 1 T 1709 DIA TCE1 Loopback, relay, PTR2
11-MAY-2008 01:10:10.000 T3 1 T 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
11-MAY-2008 01:11:07.000 T3 0 R 1709 DIA TCE1 Loopback, relay, PTR2
11-MAY-2008 01:11:07.000 T3 0 R 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
11-MAY-2008 01:11:07.000 T3 0 T 1709 DIA TCE1 Loopback, relay, PTR2
11-MAY-2008 01:11:07.000 T3 0 T 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
11-MAY-2008 01:11:09.000 T3 0 S 1709 DIA TCE1 Loopback, relay, PTR2
11-MAY-2008 01:11:09.000 T3 0 S 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
11-MAY-2008 01:11:58.000 T3 1 R 1709 DIA TCE1 Loopback, relay, PTR2
11-MAY-2008 01:11:58.000 T3 1 R 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
11-MAY-2008 01:11:58.000 T3 1 S 1709 DIA TCE1 Loopback, relay, PTR2
11-MAY-2008 01:11:58.000 T3 1 S 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
11-MAY-2008 01:11:58.000 T3 1 T 1709 DIA TCE1 Loopback, relay, PTR2
11-MAY-2008 01:11:58.000 T3 1 T 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
11-MAY-2008 01:15:57.000 T3 0 R 1709 DIA TCE1 Loopback, relay, PTR2
11-MAY-2008 01:15:57.000 T3 0 R 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
11-MAY-2008 01:15:57.000 T3 0 S 1709 DIA TCE1 Loopback, relay, PTR2
11-MAY-2008 01:15:57.000 T3 0 S 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
11-MAY-2008 01:15:57.000 T3 0 T 1709 DIA TCE1 Loopback, relay, PTR2
11-MAY-2008 01:15:57.000 T3 0 T 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
11-MAY-2008 01:16:46.000 T3 1 R 1709 DIA TCE1 Loopback, relay, PTR2
11-MAY-2008 01:16:46.000 T3 1 R 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
11-MAY-2008 01:16:46.000 T3 1 S 1709 DIA TCE1 Loopback, relay, PTR2
11-MAY-2008 01:16:46.000 T3 1 S 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
11-MAY-2008 01:16:46.000 T3 1 T 1709 DIA TCE1 Loopback, relay, PTR2
11-MAY-2008 01:16:46.000 T3 1 T 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
11-MAY-2008 01:23:23.000 T3 0 R 1709 DIA TCE1 Loopback, relay, PTR2
11-MAY-2008 01:23:23.000 T3 0 R 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
11-MAY-2008 01:23:23.000 T3 0 S 1709 DIA TCE1 Loopback, relay, PTR2
11-MAY-2008 01:23:23.000 T3 0 S 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
11-MAY-2008 01:23:23.000 T3 0 T 1709 DIA TCE1 Loopback, relay, PTR2
11-MAY-2008 01:23:23.000 T3 0 T 1714 DIA TCE1 Loopback, solenoid, ETD2/SOL2/ETSV
11-MAY-2008 02:30:59.000 T3 0 S 0274 DIA IOMA Power supply out of limits, N15
11-MAY-2008 02:31:00.000 T3 1 S 0274 DIA IOMA Power supply out of limits, N15
11-MAY-2008 02:45:19.000 T3 0 S 0274 DIA IOMA Power supply out of limits, N15
11-MAY-2008 02:45:20.000 T3 1 S 0274 DIA IOMA Power supply out of limits, N15
11-MAY-2008 02:46:25.000 T3 0 S 0274 DIA IOMA Power supply out of limits, N15
11-MAY-2008 02:46:30.000 T3 1 S 0274 DIA IOMA Power supply out of limits, N15
11-MAY-2008 02:51:25.000 T3 0 S 0274 DIA IOMA Power supply out of limits, N15
11-MAY-2008 02:51:26.000 T3 1 S 0274 DIA IOMA Power supply out of limits, N15

--------------------end--------------------------
***************June 22nd Trip********************

1. 19:02:24 hrs, T3 C DCC BMS out of memory D2
2. 19:02:24 T3 C DCC UDM no BMS memory buffer avaialable
3. 19:02:38 T3 S TCD1 relays dropped due to I/O Arcnet failure
4. 19:02:38 T3 T TCD1 relays dropped due to I/O Arcnet failure
5. 19:02:38 T3 T TCE1 Loopback Relay PTR2
6. 19:02:38 T3 T TCE1 Loopback Solenoids ETB2/SOL2/ETSU
7. 19:02:38 T3 R TCD1 relays dropped due to I/O Arcnet failure
8. 19:02;38 T3 R TCE1 Loopback Relay PTR2
10. 19:02:39 T3 C LCC No Arcnet communication with R
11. 19:02:39 T3 C LCC No Arcnet communication with S
12. 19:02:39 T3 C LCC No Arcnet communication with T
13. 19:02:40 T3 R Processor rebooted independently
14. 19:02:40 T3 S Processor rebooted independently
15. 19:02:40 T3 T Processor rebooted independently
16. 19:03:02 T3 R TCE1 TMR Check Trouble ETR1
17. 19:03:02 T3 R TCE1 TMR Check Trouble ETR2
18. 19:03:02 T3 S TCE1 TMR Check Trouble ETR1
19. 19:03:02 T3 S TCE1 TMR Check Trouble ETR2

Request you to provide your valuable input to resolve this issue.
 
Has anyone thought about ground potential rise? When a lightening strike happens, it is dispersed throughout the grounding grid in the substation, raising the electric potential of the Ground in the immediate area. This rise in potential is distributed to any electrically connected component, even through pipes and other pieces of metal.

It could be that your power supplies, which are neatly grounded, are somehow connected to the substation's grounding grid. It's not uncommon to have a switchyard located extremely close to the PEECC's that house your RST processors. Could be that the grounding could be so near, or even comingled, that they are experiencing the same problems.

If your power supplies are using a common ground to the switchyard, this definitely would cause their power supplies to go out of limits. This would happen instantaneously, knocking out all three at once.

Mike T.
 
This could also be related to the communications related issues that show up in the logs. A ground potential rise could certainly disrupt your ARCNet communications as well, ground loops are a common problem on RG-62/U based ARCNet.

No communications means no voting, which means <P> takes over, and <P> only knows how to trip.

Mike T.
 
Hi ALL

We could resolve this problem, by relocating the Mark V earth pits. Earth pits were relocated to a place which is diagonally opposite to the existing place, solved this mysterious problem. Suspect that the least resistance path for the lightning discharge could be the location of earth pits.

Thanks for your support and response!!!
 
Top