Persistant voter mismatch alarms

Hello,
We have eight 2000 vintage frame 7EA with DLN1 and MarkV control system and EX2000 exciter.

When we came onsite this morning, 7 of the 8 units were displaying the following alarms (these are from our Unit 6);
Description - Variable Name
Voter mismatch, <R> SSDIFF1 - T6.D_3449_C
Voter mismatch, <R> SSDIFF1 - T6.D_3450_C
Voter mismatch, <S> SSDIFF1 - T6.D_3690_C
Voter mismatch, <S> SSDIFF1 - T6.D_3691_C
Voter mismatch, <T> SSDIFF1 - T6.D_3931_C
Voter mismatch, <T> SSDIFF1 - T6.D_3932_C
Voter mismatch, <R> L52B_SEL - T6.D_1834_C
Voter mismatch, <S> L52B_SEL - T6.D_2330_C
Voter mismatch, <T> L52B_SEL - T6.D_2826_C

Affected Units
Demand Display of SSDIFF1(phase) shows value toggling sporadically between 1800deg and 0deg.
Prevote Data display of SSDIFF1(phase) shows value toggling sporadically between 1800deg and 0deg.
The cores do not change states simultaneously, resulting in the mismatch alarms.

Demand Display of SSDIFF1(slip) shows value toggling sporadically between 128% and 0%.
Prevote Data display of SSDIFF1(slip) shows value toggling sporadically between 128% and 0%.
The cores do not change states simultaneously, resulting in the mismatch alarms.

Demand Display of L52B_ALM shows value toggling sporadically between 1 and 0.
Prevote Data display of L52B_SEL shows value toggling sporadically between 1 and 0.
The cores do not change states simultaneously, resulting in the mismatch alarms.

Unaffected Unit
Demand Display of SSDIFF1(phase) shows stable value of 1800deg.
Prevote Data display of SSDIFF1(phase) shows stable value of 1800deg.

Demand Display of SSDIFF1(slip) shows stable value of 128%.
Prevote Data display of SSDIFF1(slip) shows shows stable value of 128%.

Demand Display of L52B_ALM has a stable 0.
Prevote Data display L52B_ALM has a stable 0.

We ran all 8 units the night before, but shutdown in the same manner as we always do.
The alarms were not present for ~1 hour after the shutdown.

Today, I rebooted <R>, <S>, <T>, <C>, <X>, <Y>, <Z> cores, as well as the HMI (which restarts the project) on one of the affected units.
Afterwards, it was still behaving the same.

What could cause this?
What risk does this pose?
Can this interfere with unit synchronization?
Will the condition(s) clear up once we generate again?
 
I made a couple of typos.

SSDIFF1(phase) values toggle between 180deg and 0deg.

SSDIFF1(slip) should have been SFDIFF1(slip) on Variable Names T6.D_3450_C, T6.D_3691_C, and T6.D_3932_C. These are toggling between 128% and 0%, and ARE causing voter mismatches.
 
Hello,

These values are calculated on TCEA firmware in Autosynchro block.
It is definitely same signal "SSDIFF1" found mismatch by SIFT VOTING according to OEM manual..

L52B_sel is 52G breaker status coming from P core (PTBA card ) Did you investigate on that side...

How about synchronization phase , is that ok ...to get online...

You may have to upgrade/update TCEA firmware

Will have a look on OEM manual see what we got there..

Any time!

James
 
Thanks for helping James.

All 8 units have open 52G breakers. The 128VDC between pins 36 and 37 on PTBA board is stable. Of course, there are multiple breaker status switches, but a continuity test on the OTBA 36 & 37 wires show an open condition.

We have not been able to test synchronization/breaker close functionality since these anomalies. They were all working fine numerous times before these symptoms appeared.

What has me so puzzled is that it happened on 7 independent units (21 TCEA boards) overnight. The weather here was good, so i don't believe we may have experienced lightning damage.

This has never happened before. Firmware failure or issues on so many units seems very unlikely.

I really appreciate any and all of your inputs/suggestions.
 
Thanks for helping James.

All 8 units have open 52G breakers. The 128VDC between pins 36 and 37 on PTBA board is stable. Of course, there are multiple breaker status switches, but a continuity test on the OTBA 36 & 37 wires show an open condition.

We have not been able to test synchronization/breaker close functionality since these anomalies. They were all working fine numerous times before these symptoms appeared.

What has me so puzzled is that it happened on 7 independent units (21 TCEA boards) overnight. The weather here was good, so i don't believe we may have experienced lightning damage.

This has never happened before. Firmware failure or issues on so many units seems very unlikely.

I really appreciate any and all of your inputs/suggestions.
Thanks for your reply and inputs MarklaneV

For L52B i Notice that you have a "dry contact" and you did not perform Breaker closure /opening as you stated..

There is a signal name L86MR_TCEA is called TCEA Master reset can you monitor it through your site CSP files ..

These "voting mismatchs " seems to have same origin as mentionned on my previous post so best way would be to track that L86MR_TCEA signal and see if it operating properly as per you site CSP Files..

Will have a closer look on a CSP file for a frame 7E and get back asap

Regards,
James
 
Just a couple of things. L52B_SEL is usually the "breaker select" contact that tells the Mark V which of a possible two breakers is to be closed by the Mark V. (The Mark V can be used to close a generator breaker, as well as a tie breaker, or other breaker which connects the turbine-generator to a load or grid or plant, etc.) That's from memory, and I don't have access to my old Mark V Manuals and documents at this writing.

It seems VERY odd that this started at roughly the same time for seven of eight units. I have seen something like this occur on one or two units which were connected to a grid that experienced a voltage surge (doesn't have to be a lightning strike).

I would be extremely surprised, shocked and amazed if some operator hadn't already tried a MASTER RESET (and another also tried it, as well as an Operations Supervisor). It's the first--and worst--thing operators do, but no one will EVER convince them of that. (That's coming from nearly 40 years of painful, personal experience.) But, stranger things have happened--and this would be truly strange.

I sincerely doubt the firmware (PROMs need upgrading--but it could be that it's overdue for an upgrade AND conditions were perfect for some trigger event to make this occur. (This has happened once or twice in my career. PROM upgrades can be avoided for many problems, but sometimes things conspire to "require" them to be updated. Problem is, when you call the OEM--they're going to do their damnedest to try to convince you to upgrade the Mark V to a Mark VIe.)

I suspect you're going to find that something happened in the dead of night, or when one or more generator breakers opened, or when some switching was going on in a nearby substation. This is just really, really odd to occur without some kind of stimulus/trigger on seven of eight units. (The one unit that didn't experience this issue--was it's generator breaker opened first or last as the eight units were being shut down? AND, is it possible that more than one generator breaker opened at roughly the same instant in time (within a few cycles (of 60 Hz) of each other?)

Personally, I'd be looking at PT drawers, disconnects/stabs, wiring, PT fuses, etc., as part of the investigation. I'd be asking the utility/grid/nearby substation operators what was happening last night just before or during or just after the units were taken off line.

I also don't get the T6.D_3nnn_C references.... Doesn't look the least familiar. Is this Delta V signal naming? Or PI? And, what's the significance of the numerals and the _C reference?

I might be wrong about the L52B_SEL; it's been a VERY LONG time.... But I do seem to recall that there was a way to change the breaker parameters when closing two different types of breakers with different closing times (the self-adapt correction feature that causes so much consternation and confusion and speculation).

Anyway, hope this helps!

(Is this site anywhere near Terre Haute, IN, or Ohio? I seem to remember hearing about several sites with eight 7EAs being built in the lower Midwest during "The Bubble"....)
 
After printing out all alarms several Master Resets were done. This includes the L86MR_TCEA. Same symptoms persist.
All Mark V cores were rebooted (R,S,T,X,Y,Z,C) simultaneously. While they were powered off the HMI computer was rebooted. Same symptoms persist.

The unaffected unit was the 1st breaker to open. I observed that the mismatch alarms were not happening for more than an hour after all unit breakers were open. Because the time stamps are refreshed, I cannot tell exactly when they started but they are coming every 5-10 minutes now.

We have simple cycle units. L52B_SEL is the only output when the TCEA receives input from a dry contact (breaker limit switch) on PTBA 36 & 37. In other words, AFTER the breaker is already closed. Therefore, I don't understand what it does.

I have reviewed the grid voltage on PI and do not see any surge anomalies.

We are running all units today. All of them synchronized and closed their breakers normally. None of the mismatches are occurring while a unit is running. L52B_SEL is 1, SSDIFF1 is 0.00, & SFDIFF1 is 0.00.

I will report whether symptoms return when units are shut down.

I have spare TCEA boards with PROM's. Are these PROMS written to when "eeprom down USER" is performed?
Would you recommend just putting new PROM's in? Will they need to be downloaded?
 
After printing out all alarms several Master Resets were done. This includes the L86MR_TCEA. Same symptoms persist.
All Mark V cores were rebooted (R,S,T,X,Y,Z,C) simultaneously. While they were powered off the HMI computer was rebooted. Same symptoms persist.

The unaffected unit was the 1st breaker to open. I observed that the mismatch alarms were not happening for more than an hour after all unit breakers were open. Because the time stamps are refreshed, I cannot tell exactly when they started but they are coming every 5-10 minutes now.

We have simple cycle units. L52B_SEL is the only output when the TCEA receives input from a dry contact (breaker limit switch) on PTBA 36 & 37. In other words, AFTER the breaker is already closed. Therefore, I don't understand what it does.

I have reviewed the grid voltage on PI and do not see any surge anomalies.

We are running all units today. All of them synchronized and closed their breakers normally. None of the mismatches are occurring while a unit is running. L52B_SEL is 1, SSDIFF1 is 0.00, & SFDIFF1 is 0.00.

I will report whether symptoms return when units are shut down.

I have spare TCEA boards with PROM's. Are these PROMS written to when "eeprom down USER" is performed?
Would you recommend just putting new PROM's in? Will they need to be downloaded?
Thanks for the impurs...
P core (TCEA card) is "standard" got is own firmware and cannot be modified application whitout chnaging EPROMs as per OEM manual...

Did you ever did a soft reset on SDCC ...as you seems to have done "Hard reset "...We know that some are not conviced by "soft reset " but you can try it ...( see OEM manual "MarkV application manual "..

Ah Chicago ...I did commissioning on 8*Frame6B with MarkIV was equipped with ELIN Excitation AVR /Protection GCP at Peco Site which is Owned by EXELON...

Great time over there..Calumet city you may know it ..

Any time!
James
 
#1. I really appreciate you all taking the time to puzzle over this with me and give ideas.

Update:

We shut down 4 of our units. All were ones that had recurring mismatches.
3 of them resumed the mismatch symptoms. 1 of them did not!!

Of the 4 units that are still running now, 3 of them were having recurring mismatches.
We have high hopes that they will also "self heal".

I will resume troubleshooting tomorrow.

I am familiar with the little white button on the SDCC for soft reboot. I don't see any risk in trying that tomorrow.
I may also try spare PROM's in one of the units.

We are very near Dixon, IL and yes I know Calumet.
 
MARKlaneV,

When a EEPROM Download is performed, the information downloaded ONLY goes to a EEPROM chip in the U9 socket on the DCC/SDCC card of the processor being downloaded to. The downloaded information ONLY gets to the intended card's RAM from the EEPROM chip in socket U9 is what is transferred when the card is re-booted (the card where the intended changes are supposed to end up--which could be the TCQA card, or the TCDA card in Slot 1 or Slot 2 or Slot 3 of <QD1>, or the TCEA in slot 1 or -3 or -5 of <P>). But it ONLY gets into the RAM of the intended card when the card's power is cycled. THE EXCEPTION to this is the DCC/SDCC (and probably the LCC/SLCC)--which gets rebooted (RAM erased) with the little white button on the DDC/SDCC.

When a soft reboot is done (using the little white button on the DCC/SDCC card) all that happens is the DCC/SDCC card is cycled--meaning the RAM on the DCC/SDSCC (and probably on the LCC/SLCC) has to go to the EEPROM chip in socket U9 on the DCC/SDCC to get the information needed to boot up. THAT MEANS NONE OF THE OTHER I/O CARDS ASSOCIATED WITH THAT PROCESSOR HAVE TO GO TO THE EEPROM CHIP IN SOCKET U9 TO GET ANY INFORMATION, meaning any changes intended for those other I/O cards NEVER get to those I/O cards. That's why it's almost always best to do a hard reboot--because every I/O card has to get its required information from the EEPROM chip in socket U9 on the DCC/SDCC card. (In this particular case, if ControlsGuy25 is advocating getting EEPROM I/O Configuration information from the EEPROM chip in socket U9 to the TCEA card by performing a soft reboot, that ain't gonna have the desired effect.)

The test is going to be what happens when the units shut down this evening.... Will they, or won't they? (Go back to as before, or continue with the Voter Mismatch Alarms....)

So, I think you've answered at least a couple of the questions in your original post, MARKlaneV. Something has changed, but darned if I know (based on the information provided) what that is.

If the Diag. Alarms are still present after shutdown today, you could try swapping out one or two TCEA cards to see what happens.

Sorry; I just can't get to my old Mark V notes about L52B_SEL.... The B_SEL stood for "Breaker Select" so my recollection tells me that isn't breaker status, unless it means the selected breaker status.?.?.? And, if there's only one breaker being closed by the Mark V, then it would be the "Selected Breaker" for sure. Sorry; I just can't remember without being able to see my notes in my old Mark V Application Manual (Rev. B). There's a LOT of notes and flags and highlighting in that version.... And, too many pages to scan.
 
1 Appreciate to talk here and trying to support as best as I can !

There is also a switch for rebooting TCEA card on PD core.. did you tried that one ...

One suggestion would be, re-booting that TCEA card by using the switch in the <P> core.

Thanks for your inputs much appreciated..


Stay safe & healthy

James
 
Csa

I was telling to original poster to be aware of precautions to have before doing a " soft reboot"...
I did not "advocated" him to do it..but suggesting this solution ( and beware him about instructions to follow prior to do it by reading teh OEM right technical manual for sure!;)
 
Gents:

I think you are on the verge of a main GSU short....

The ground reference at the transformer seem to have changed or the resistor may have opened!
 
Top