Mark V Voter Mismatch

R

Thread Starter

Ron

We have had Voter Mismatch alarms on our system for a few months now. Some of the voter mismatch alarm points are going to the QD2 core and some are going to the CD core (we have over 25 voter mismatch alarms and only a handful are going to the CD core). I checked the Pre Vote data and the T core does not agree with R and S. If all of the alarms were tied to the QD2 core it would make it easier to troubleshoot, but since they are on the C and QD2 core I am confused. Any advice?
 
Ron,

There are 100s of Voter Mismatch Diagnostic Alarms possible on any Mark V turbine control system. They can be related to logic signals or analog signals that are not directly related to inputs or outputs. They can be related to logic or analog signals directly related to inputs or outputs.

From the information provided it would seem you are referring to Voter Mismatch Diagnostic Alarms related to discreet (contact) inputs to more than one Discreet Input core (in this case <QD2> and <CD>).

When there are multiple Voter Mismatch Diagnostic Alarms related to discreet inputs connected to discreet input cores that usually suggests a high amount of AC ripple riding on top of the 125 VDC being used as the wetting/interrogation voltage for the discreet input circuits. You can usually take a quality digital VOM and set it to AC mode and connect one lead to a good ground and use the other to read the AC voltage present on the positive leg of the 125 VDC voltage and then the negative leg of the 125 VDC voltage. Usually when the AC voltage exceeds approximately 40 VAC with respect to ground on either DC leg then these kinds of intermittent Voter Mismatch Diagnostic Alarms.

AC usually gets induced on contact input wiring in the field when it's incorrectly run in the same conduit as motor power wiring or in the same cable tray as motor power wiring. Somewhere, there is a high-current AC source run in close proximity to 125 VDC discreet input wiring and is inducing a high AC voltage on to DC wiring.

Another possible cause is that the output filter capacitors of the 125 VDC battery charger are failing and need replacing. This is more difficult to diagnose, but does happen.

Also, if the unit experiences frequent intermittent 125 VDC Battery Ground Alarms, this can make the problem worse.

So, you have several possibilities. None are particularly easy to find.

Lastly, there was a GE TIL (Technical Information Letter) out about a PROM change to help reduce the sensitivity of the TCDA PROMsets to AC ripple; sorry I don't recall the TIL number. But, if your site hasn't kept up with Mark V TILs this could be a possible problem/solution, though an expensive one that would require getting GE involved--and they are going to <b>HARD</b>sell a Mark V LE (Life Extension) or Mark V Migration controls retrofit.

Unless you need to add I/O capacity and/or you require additional memory and/or computational horsepower it's not necessary to replace or upgrade your Mark V using either of the recommended GE solutions. Both of those "solutions" will leave you with the same I/O cores, with the same I/O card carriers (with cards mounted upside down on the backs of the card carriers), lots of ribbon cables, and the same high density I/O terminal boards and wiring mess.

If you are going to upgrade your turbine control system, go for a full Mark VIe upgrade, removing and replacing the Mark V control panel altogether. That way you will get rid of all of the Mark V hardware and get a chance to clean up the field wiring terminations.

A well-planned upgrade outage should take no more time than one of those Mark V LEs or -Migrations, and in the end you will have a much cleaner installation with easier to service cards and field wiring. You will also have I/O expansion capability and more memory and computational horsepower with a full Mark VIe, instead of Mark VIe components made to fit a Mark V form factor and using ribbon cables and twenty year-old input terminal boards in a very compact 54"x20" panel in the lovable Mark V I/O cores.

At any rate, please write back to let us know how you fare with solving the problems. I would suspect that unless some wiring modifications have been made in the field that you will be battling several problems at the same time: battery charger output capacitors failing, poor construction wiring practices, and TCDA PROM version. Also, if the panel experiences frequent, intermittent 125 VDC battery ground alarms that can make other problems worse.

Hope this helps!
 
Thank you for the reply!!!

We are only getting 2 volts AC on the DC feed and our battery voltage is balanced.

I mentioned in the first post that the Voter Mismatch was coming from the CD and T cores but apparently they are isolated to the T core on DTBA and DTBB boards.

I also have a BATT REF alarm in and in Diag C I am showing 17 volts for P15 and N15 for the T core. I am not sure if this is related but it does seem suspect.
 
Ron,

When you say "...DC feed..." are you referring to the two-conductor cable that feeds the DTBA, and is jumpered down to the DTBB? They come from the <PD> core; I think it's from the J12C (for <QD2>) connector/fuses in the <PD> core. I think there are LEDs on the fuses (at least one of them) for the power supply (125 VDC to the discrete I/O cores) on the printed circuit card in the <PD> core, next to the J12C connector, which if I recall correctly is in the lower left corner of the printed circuit card in the <PD> cor;. (It's been a long time since I've peered into a Mark V, and I may be wrong about the designation of the power connectors, too. It's either J12x or J8x, where 'x' is A, B, or C.)

Probably once you solve the 125 VDC power supply problem, the BATREF alarms will go away.

I'm afraid I have some bad news about the DIAGC information. There was only one person in the world who could tell you if the DIAGC.DAT file matched the card and PROMs in your Mark V panel, and he's retired from GE. Worse, a LOT of Mark V panels were shipped with the wrong DIAGC.DAT file, and so there are a LOT of Mark Vs in existence with the wrong DIAGC.DAT file.

That means the data made available via DIAGC is questionable.

Sorry; but that's the truth.

Anyway, hope this information helps. Again, the power for the DTBA comes from the <PD> core, via a fused output. And, it's jumpered from the DTBA to the DTBB card.

Please write back to let us know how you fare!
 
CSA,

After reading some of the other topics on control.com, we have concluded that this is not an issue with the 125VDC power supply to the system. No AC ripple is present, no serious grounds are present, and the voter mismatches have been (and are) persistent.

In addition to the information contained in the previous post, we would like to add the following history:
1. After a lightning strike, there was damage to the MkV system. We were compelled to replace the <PD> board and PTBA in <P> core.

2. After said lighting strike, a slew of voter mismatches have been in. These mismatches are what the original post was referring to.

Today, we did the following tests, with the <T> core showing a voter mismatch:

1. Confirmed all fuses on <PD> are good.
2. Measured 125VDC across J12/JY on DTBA/DTBB
3. Confirmed that all 4 fuses on TCPS card in T core are good.
4. Confirmed readings on test points on TCPS in T core card
5. I cannot confirm, but it is believed that at this point the LED's on R and S were flashing in synch, and T was the "odd man out".

No IOnet errors are present. All cables are firmly secured and no damage apparent. We do have a tube of the conductive grease and have been inspecting connectors carefully as we work with them.

Once we were convinced that the issue wasn't a lack of power, we swapped TCDA boards between S and T.

This had the following results, and is our current state:

1. A slew of voter mismatches came in on the <R> core.
2. The LED patterns on S and T now agree. <R> is a pattern on its own, but matches R, S, and T on our other MkV unit at the site.
3. The voltages on jumper JP match between S and T. R matches RST on our other unit.

Swapping the cards back made no difference.

We haven't been able to actually toggle an input on <R> yet and see the pre-vote data, to confirm that it is reading correctly. However, all of our other checks indicate that <R> is correct.

For reference, the LED pattern on <R> is 11000000 to 11110000. S and T are both going from 11000000 to 11101101.

Our conclusion at the moment is that the TCPS card has some kind of issue which is damaging the TCDA cards and that currently both <S> and <T> are dead.

Does this sound at all reasonable? What else can we do to troubleshoot this?

Thanks!
 
B

bicycle mechanic

It sounds like an EEPROM on a board was affected by that lighting strike.

I would do, a "down load all" to the cores and reboot.

If that doesn't work, start looking to replace EEPROM or proms to solve the problem.

I would exhaust those possibility first.

Also, the dual ported ram on the SDCC. Some of these were soldered and some socketed.






 
We have been instructed to download to "user" and NEVER download to "all" as this would erase the counters for fired hours, starts etc. We did perform a EEprom download to "user" which did not resolve our issue. If downloading to "all" is the solution, then we will have no choice but I would like to be sure before we erase the counters. Any thoughts? Thank you for the reply!
 
B

bicycle mechanic

I am a bit fuzzy as to lost counter info by doing a "down load all".

If the current running programs were never backed up somewhere, and you reverted to as shipped settings then lost of counter info possible among other things. (control constants)

Try uploading first, save in a file and quarantined.

IF you have A7 status on all cores and still voter mismatch, with newly downloaded files, it's indication of SDCC problem.

Or better yet! before doing all that, use the terminal interface monitor (TIM) and see what cores have programs loaded and running, and status of memory segments.

Start with TIM!
 
The ALL parameter includes FORMAT (which effectively erases the EEPROM) and TOTD (Totalizer Data). If the Totalizer Data <b>ISN'T</b> uploaded before it is downloaded then old, or even blank Totalizer Data will be downloaded with ALL.

I've been ruminating on this, and while it seems like the power supply to the <T> TCDA seems to be causing the TCDA in Loc. 3 to fail (as indicated by the "failure" of the card swapped in to Loc. 3) it's not conclusive (at least to my thought process). It's difficult to conceive how the power supply cable could be damaged or be causing a card failure.

What does the SLCC display say the I/O States of all of <T>'s complement of cards is? Is <T> (and <S>) at A7--but the TCDA card(s) is (are) at something other than A7? This would mean that at some point the TCDA was at A6/A7 then it went to some other level.

I don't know what TIM<b>N</b> would tell us, and it would be necessary to be able to see the file to analyze it, and GE, in their consistently inconsistent way, doesn't document all of the information available in a TIMM session. I believe what Bicycle Mechanic is suggesting is to set up a computer using a serial cable connected to <T>'s TIMN port on the QTBA with the power to <T> off, then apply power to <T> and capture the boot-up sequence.

The LED bar-graphs on the three TCDAs should be flashing in the same sequence; the fact that they aren't indicates there are Diagnostic Alarms on the card--which you have indicated.

<Q> SDCCs communicate with TCDAs via the IONET, which connects to the respective TCEA in <P>. This revelation about a damaging lightning strike is causing me to have some questions.

Actually I'm at loss because, for me, everything isn't adding up to a likely scenario. So, it seems the TCDA in Loc. 3 is "failing" for some reason. There IS 125 VDC at the DTBA/DTBB, but the TCDA in Loc. 3 didn't see it. And Voter Mismatch Diagnostic Alarms are toggling at a periodic rate.

I would be looking at the power supply to the TCDA in Loc. 3 as well as the IONET from <T> to the TCEA in Loc. 5 of <P> to the TCDA in Loc. 3. When you swap TCDAs in a digital I/O core it's NOT necessary to download anything to either card--they have the same info (they are redundant).

I wonder if some other cards (the TCEA Loc. 5?) weren't "disturbed" during the lightning troubleshooting. But that doesn't explain why TCDAs are "failing"....

Again, something isn't adding up here. I'll keep thinking about what might be the problem, but I would suggest the IONET from <T> as well as the power supply cable from <T>'s TCPS be checked along their entire lengths to be sure something isn't amiss.

Please write back to let us know how you progress.
 
Hi Ron,

I recently came across a case which is similar to yours.

We first started by measuring the panel power supply. For our case, the AC ripple were found on all voltage levels, this may be the root cause of problem but why was it only affecting a core? so we continued our investigation.

Our customers were just like you, swapped a card and had another core corrupted (i would say), both now failed to boot to A7.

Long story cut short, our head engineer decided to reprogram the EPROMs of TCQA & TCDA (Data in EPROMs might have been corrupted) and that solved the former part, but not the latter part- voter mismatch. In the end,we found out the problem was the power supply from TCPS to <QD>TCDA. So we replaced TCPS card, and everything works like a charm.

I would suggest checking TCPS card, and TCDA & TCQA EPROMS. Happy to share, thank you.
 
With laptop connected to <T> core via serial cable, we turned <T> on. No errors during bootup.

In addition, TIMN shows that <T> and all of its IO cards are all at A7. No system errors present. Same for S, R.

The only indicators we have found thus far are the persistent voter mismatch alarms and BATREF alarm in DiagC.

If you are familiar with the DTBx boards, there is a small bank of 3 relays. Our understanding is that these are energized to provide the 125VDC wetting voltage to the TCDA cards. Currently, KS and KT are de-energized. So we are trying to figure out what would prevent these relays from being picked up.

This issue must be at a very low level, since everything is at A7. It almost seems like a 24VDC power cable is broken somewhere for picking up the relays on DTBx boards.


<b>-----update-----</b>

After wondering if some cap/inductive filter on the TCPS was bad, sending a startup surge to the TCDA, we pulled the TCPS board to inspect. We found that FU4 is actually blown.

Previously, when we checked the fuses, we measured voltage on each side and continuity with the ohm meter (with core powered down). Due to the difficulty of seeing FU4, we didn't notice its condition until the board was out.

However, <b>with the fuse pulled</b>, there is still continuity across the fuse clip (!).

-----/update-----

Here is the log for <T> from TIMN. Note that we had <S> powered down for the moment.

<<< SDCq GHD Version 1.3 186 restart >>>
DCC/TCQC stall test started

<<< SDCq GHD Version 1.3 186 restart >>>
DCC/TCQC stall test passed
DCC-RAM passed
1K DPRAM found at: 5000 : 0000
2K DPRAM found at: 5200 : 0000
2K DPRAM found at: 5400 : 0000
DPRAM unpopulated at: 5480 : 0000
DPRAM unpopulated at: 5500 : 0000
DPRAM unpopulated at: 5580 : 0000

<<< DCC-186 initialized >>>


----- SDCq GHD Version 1.3 Mark 5 Monitor <T> -----
<Q> DCC 186: Oct 30 1997 at 15:18:57
BBL library: Oct 30 1997 at 15:17:58
Current time: Sep 30 2013 - 7:30:54

#S

----- Agent Status -----<pre>
Status> I

obj card maj min I/O cfg qst cfg cfg eep rsp msg msg
ID name rev rev stat stat ID flgs tics ofst type seq stat
-----------------------------------------------------------------
1 TCQA 02 06 A7 D0 22 01 0001 4200 05 0007 00
4 TCD1 03 09 A7 D0 21 01 0060 4280 05 0011 00
5 TCD2 03 09 A7 D0 21 01 0060 42AC 05 0013 00
8 LCCq 04 04 A7 D0 5 01 5320 42FB 05 0016 00
12 DCCq 01 03 A7 D0 27 01 0001 4322 05 0003 00
13 IOMA 04 05 A7 D0 21 01 0020 437E 05 000B 00
15 TCE1 05 03 A7 D0 21 01 0040 43C1 05 000D 00

Status> E

---- System error counters ----
:- No system errors detected


Status> V

Agent: T I/O state: A7 Conforming: 03 Voter busy: 53%
I/O ready: yes Agent rdy: yes Voting Agents: CR-T
Frame Rate: 16 Hz Voting Mode: TMR

DCC voter: State: 0400 Status: 0 Active: 0
LCC voter: State: 0430 Status: 1 BMS OK: 0

VDP: R TDP: C Present: R:2F S:00 T:0F - C:CF D:00
Time stats: Status: 00 OK cntr: 1406 missed cntr: 2
Vote stats: Status: 03 OK cntr: 1215 missed cntr: 0
Voter buffs: Current: 5 Total: 6480 Bad type: 0

Misc counters:
DCC OK: 1202
DCC aborted: 0
LCC degenerate votes: 1202
DCC stand alone passes: 0

LCC sequence error: 0
LCC lost time sync: 27
DCC frame_overrun: 0

Status></pre>
 
B

bicyclemechanic

I am looking at the checks thus far.

A thought, have you disconnected 125V supply to the DTBA boards.

What if, you have a condition that is bringing that entire 125V core circuit down upon boot up, which is not there when program is not running.

I would disconnect the 125vdc plug on the terminal boards and see if all processors become stable.

I recall those boards having a resistor bridge network(I could be wrong),but if main into that net work was lost expected signals would be different from one core to another.

If system permits, disconnect across cores and compare.
 
sd123,

Thanks for the information.

I, too, have measured low resistance ("continuity") between fuse clips on TCPS cards when the fuse was not installed. Without having any elementary (as GE calls schematic drawings) for the cards I have to say there must be parallel paths which cause this. I do know that some circuits on the <P> core have leakage currents that make it appear as though circuits are energized (contacts closed) when they are not, which lead to a lot of head-scratching once.

I don't have access to any TCPS drawings at this writing, but there is one small fuse (glass ferrule type) on the TCPS that is well hidden in the lower left corner, if I recall correctly, and until the card is removed it is very difficult to see if the fuse if blown or not. And, it's also very difficult to measure the presence or absence of voltage across the fuse clips when the card is installed in Loc. 5 of any of the processor cores. Unfortunately, with this particlar fuse, it's really necessary to remove the card and remove the fuse and measure the fuse resistance to see if it's intact or blown as the fuse element is almost micro-thin and the glass ferrule is very short and the open portion of he fuse can be hidden behind the metal cover on either end of the fuse (another painful lesson--"It doesn't LOOK like it's blown, so it must NOT be blown, and there is "continuity" [low resistance but not 0 ohms] between the fuse clips!").

Also, unfortunately when a lightning strike occurs all of the collateral damage can be difficult to find quickly.

Since the DTBx cards are common to all three cores (for a <QDn> core), I don't think the wetting voltage can be segregated for particular cores--it's always applied to the contacts which are then "fanned out" to the individual TCDA cards. It might be that the wetting voltage isn't applied to the TCDA (or sensed by the TCDA) unless the respective relay on the DTBx card is energized, and if there isn't voltage from the TCPS card to do that then the TCDA can't see the wetting voltage and hence the BATREF low voltage alarm(s) get annunciated.

However, it's also hard to imagine--and disconcerting, too--how the absence of power from one fuse can "fail" a TCDA and then substituting a good TCDA into that Loc. can cause another TCDA to fail, or how moving a TCDA from a Loc. which isn't getting power from a fuse on a TCPS into a Loc. which IS getting power from a fuse on a TCPS can "fail." I guess its possible the problem is the TCDA card which was in Loc. 3 caused the fuse in <T> to blow, and when swapped into Loc. 2 has also caused the same fuse on the TCPS in <S> to blow.

And it's hard to believe this would allow a TCDA to get to A6/A7 if it can't see the wetting voltage. A6 is the I/O State that says the card is synchronized to the DCC/SDCC is ready to enable it's outputs. A7 means the outputs are enabled. A processor won't go to A7 until all of it's cards get to A6 and then they will all go to A7, and one or more cards can then "drop out" of A7 but the processor can still remain at A7....

Please let us know how the troubleshooting goes. It would certainly seem the TCPS(s), and or the fuse(s), are suspect--at a minimum. But, the TCPS cable from <T> to Loc. 3 of the <QDn> core, or the TCDA that was originally in Loc. 3 of that digital I/O core could also be suspect.
 
All,

Just a quick reply to bring this matter to a conclusion:

1. FU4 was definitely blown. We replaced it and it blew again.

The misleading readings with a volt meter can all be easily explained by drawing out a power supply circuit.

The bottom line is that one must measure voltage across the fuse clip, not to ground with AC systems.

In this case, that test was not feasible, so, the best method is to perform a visual inspection or check the impedance of the fuse OUT of the circuit. (in circuit most likely backfeeds through transformer(s))

2. Once we confirmed that FU4 was blowing, we unplugged things and looked for shorts.

3. With an ohmmeter, we found that the TCDA in <T> core had a dead short from positive to negative. The other TCDA boards did not have a dead short.

4. We pulled the board, replaced it, and all our voter mismatch errors went away.

Out of curiosity, we proceeded to:

5. With an infrared camera, we used a current control power supply to power up the board and limited it to about 500ma. We quickly identified that a surface mount capacitor (installed from positive to negative, as some kind of ripple filter) was taking the heat. We removed the capacitor, and the board powered on normally.

6. We ordered a new capacitor (50 cents?), and the board works now. We put it on the shelf for emergency testing only, not to be put back into production.

Thanks for your help!
Stephen
 
Top