Persistant voter mismatch alarms

Depending on how the ground reference is monitored for grid operations ; the wiring scheme for a BLOCK or multi-block will change as seen by a unit control and logic.

Not sure if this will help or not. I looked a bit at our logic and some information I have and I'm wonder if there is something that stops the TCEA's looking for sync when the unit is not running. Figure 7-36 and Figure 7-35 in the application manual shows a sync check Description for Auto and Manual Sync, L52B_sel is not on the Manual sync Description. Are the affected units stuck on Autosync for some reason?

Page 9 of mk5-nl-Sec4-07 show a line diagram of the TCEA Autosync. It shows a bypass bit of L25_bypass. In the logic it has something to do with Deadbus and L52G_Close.

I see there is info for TCEA in the IO config. as well as info for sync and phase limits. Maybe a download and reboot all?

I'm no expert on this but hopefully I can provide some help



WOW!!! Those are some old documents; they bring back a lot of memories....

They do represent how much the Mark V has changed over the years, and it changed a LOT very quickly in the early years (as I'm sure you're aware if you have these documents).

It's apparent my recollection of the purpose of L52B_SEL is incorrect; I did say I might be wrong about that, and I am.

So, here's something which has always caused a lot of consternation and concern over the years--leaving the SYNCH SELection in AUTO or OFF. MOST--but NOT all--Mark* turbine control logic was set up to require the operator to select AUTO or MAN synchronization when the MASTER CONTROL (MODE) selector was in AUTO. Many sites just always leave the synch selection in AUTO if they operate with the MASTER CONTROL (MODE) selector in AUTO, and that can cause some funny things to happen to the synch circuits when starting up and shutting down (including some of the Diag. Alarms listed).

HOWEVER, if the MASTER CONTROL (MODE) selector in REMOTE (and even in SERIAL REMOTE or CABLE REMOTE) the unit will AUTOMATICALLY begin the synchronization process when the unit reaches FSNL (Full Speed-No Load)--whether or not the SYNCH SELector was in AUTO or OFF. (Older GE-design heavy duty gas turbine generator control panels with bat-handle switches for synchronization selection had REM/OFF as the name for the "off" (middle) position. And the REM meant that when the SYNCH SELector switch was in the middle (REM/OFF) position and the MASTER CONTROL (MODE) selector was in REMOTE that synchronization would automatically start without anyone having to put the SYNCH SELector switch in AUTO. And, that would catch some people off guard--especially Operations Supervisors and more especially Plant Managers (who would always say, "That DAMNED Mark* is CRAZY and closed the breaker without permission!!! FIX IT!!! It's NOT SUPPOSED to do that!!!"

The purpose of REM/OFF on the SYNCH SELector switch, and REMOTE on the MASTER CONTROL (MODE) selector was initially for unmanned sites (peakers, mostly). By leaving the MASTER CONTROL (MODE) selector in REMOTE and the SYNCH SELector switch is REM/OFF, an unmanned unit could be started (from a remote location) and would automatically synchronize and close the breaker without any "extra" steps involved. Convenient--for unmanned units. Not so "convenient" for manned units (such as most sites are these days, since many are combined cycle plants). This also prevents the nuisance Diag. Alarms associated with auto synch functionality working when the unit was nowhere near synch speed.... (the synch scope would hum and dither; the synch lights would flicker; the synch check relay disk would also dither sometimes).

SOME units, but NOT all, had another "synchronization permissive" signal which the site/operators could use to "delay" synchronization when the MASTER CONTROL (MODE) selector was in REMOTE until they set the permissive to allow synchronization. BUT, most sites were unaware that when in REMOTE

SO, I'm wondering how this site in the Illinois corn field has their MASTER CONTROL (MODE) selector switch and the SYNCH SELector switch selection set--and if this has possible changed recently....???

Also, I do recall that the NEXUS "boxes" were sometimes glitchy, and sometimes had to be updated (firmware). It still seems to defy reason how most of the boxes would all exhibit the same behaviour at the same time.... Was there any work done in the high yards or protective relaying/metering recently (just prior to the strange appearance of these Diag. Alarms)???

I'm really keen to follow this and learn something (new). (Other than my recollection of the function of L52B_SEL was incorrect...!)

Hopefully we will get some answers and more information after the other units were shut down last night.
Curious One & HamsterKing,

Thanks for the ideas. It will take me some time to digest and then research them. Unfortunately, I am stuck as Control Room operator today (all 8 are running this morning).

On shutdown last night, the voter matches resumed. I was not here for the start-up to try to observe just when they stop happening, since they do not happen while the unit breaker is closed. I was going to create a view2 to capture, but don't know how or if voter mismatches can be captured that way.

I will attempt to find the right drawings to determine if PT's and CT's connect directly to EX2000 and MarkV, or if there is an interposing device.

I have had thoughts to;
1st - download and reboot all cores in this order <R>,<S>,<T>,>X>,<Y>,<Z>,<C>.
2nd - replace TCEA proms with spares and reboot in this order <X>,<Y>,<Z>.
You can at least try to get trend view of SSDIFF1 ..& L52B_SEL...see if you get any interference/inappropriate/incorrect behaviour /operation during auto synch or Manual synch...

Any time!

Haven't you already re-booted the processors and the TCEAs?

At least on one of the units; and was it the unit that had the Diag. Alarms mysteriously disappear?

When you're troubleshooting a problem like this, with multiple units, I STRONGLY recommend you write down each thing you do or observe--and the results of what you do--for each unit, on a separate sheet of paper for each unit. It can get out of control pretty quickly (and we are communicating via, after all!).

VIEW2 can't capture Diag. Alarms. If you have GE Mark V HMIs (running MS-Windows and CIMPLICITY or PROFICY)--AND they were properly configured, the alarms--all of them, should be automatically saved to one of the HMI hard drives. And, this usually happens for some period of time, in a "circular" fashion (say it was set up for 7 days; on the eighth day, the data in the oldest folder would be overwritten with the alarms from the present day, and so on, so that there should always be 7 days of alarms on the hard drive--if it was set up for 7 days, AND if it's working properly). I've seen it set up for 30 days, and 14 days, and 15 days. "The standard is: There is no standard." And, even if the time period is properly specified, if the folders specified weren't created, or they were deleted, or they don't have the correct names, the alarms won't be recorded.

AND, on a multi-unit site such as you are on, there can be one or two HMIs dedicated as Alarm Servers (depending on the vintage of the GE Mark V HMIs)--so finding the correct HMI and the correct folders/directories can be difficult. Hopefully, you know this already, and can find the alarm files on the HMI. (Don't ask me how to check the configuration, or how to set it up, or how to change it. The process was NEVER documented (imagine that!!!)--and it changed multiple times, and continues to change from what I see. Sites which are getting GE Mark V HMIs running ControlST and WorkstationST are a REAL jumble--and there's no checklist for the installers/commissioning personnel, or for any kind of quality check before it leaves "the factory", or for the owner/operator to use to verify all the features are working properly. It's quite messy, and getting messier--again, because so little of this is documented. The "tools" are there and available--but few people know about them; the configuration changes; and nothing is ever documented. So, how to keep up? (I don't; I gave up long ago.)

VIEW tools can only be used to gather data values; they don't record alarms. I have been told that some of the newer GE Mark V HMIs running ControlST/WorkstationST and ToolboxST can actually use Trender (or Trend Recorder--the name of that function has changed over time as well!) to capture data from Mark V turbine control panels. I haven't seen t myself, and I don't know if it will capture alarms (Process and/or Diagnostic) like it does for Mark VI and Mark VIe. If it's true, it's something to be investigated, and it's a damn sight easier to use than the VIEW tools, too.

Hope this helps!!!
Thanks CSA,

MODE SELECT is OFF, but switched to AUTO for starting/running. CONTROL LOCATION is in REMOTE. CABLE REMOTE is ENABLED. This for a sister plant to dial in and start the units in our absence, and for AGC control via our dispatchers. Almost all of the commands that they can send are configured in CLBR_Q_SRC as "momentary". SPEED/LOAD Raise & Lower and Volts Raise & Lower commands are "maintained" so that AGC can function, as well as base load or pre-select targets can be achieved.

I did reboots on 1 unit, only. It was not the unit where the alarms disappeared. The alarms came back on that unit though on last nights shutdown.

If the Q core eeprom is somehow corrupted, then the TCEA may pick up the corruption. How so many got corrupted at the same time mystifies me.

I like view2 because I can open the *.OUT file as a CSV and save it as an excel spreadsheet for easy manipulation and/or trending.

Each unit has it's own server where there are files written for Alarms, SOE, Event, and Diagnostics. These files are written on the hour (but not every hour) and some are as old as January of this year. It very tedious and time consuming to look through, but would be a way to find mismatch occurrences. I don't know how well the time stamps would synch up.

You all have asked many questions, and I will research and try to answer them all tomorrow.

I'm curious if L25PX is a 1 or 0 when your unit is shutdown. In our logic L25PX would open up contacts so SSDIFF1 is not written to anymore. Of course our units are 7FA's so it may be different but it is all in a big block SYNC1.

Also, It looks like there is a ribbon cable to report 52G from from PTBA to TCEA. Hard to believe it would happen on so many units all at once though. It also looks like 25p is in that circuit as well. (See TCEA.PNG)


Problem resolved!!!

I want to thank all of you for your ideas / questions / suggestions.

It was external to the Mark V. There was a hard ground fault on the positive side of our high yard 125VDC system. The Mark V alarms were a symptom of stray and "un-seeable" dithering.

We found the that the 345KV 86BF (breaker failure) lockouts were chattering (buzzing) frequently. These cascade back to the GCB 86's and other protective relays. Apparently the noise on 125VDC circuitry confused the TCEA's about breaker status resulting in the mismatches. I not smart enough to determine the exact path from 345 breaker protection to the Mark V, but all mismatches ceased upon isolating this ground.

I will remember this experience for at least a few weeks, maybe even months.