Mark V DCCC Alarms

R

Thread Starter

RVSSSRAO

We have many DCC errors increasing continously in <S> Core of Mark V Control System. How to proceed with below alarms? please let me know step by step procedure. We have checked JL Cables, Relays, Replacement of DCC Card etc. Further let us know on how to solve.

D10-<s> DCC DPM: no BMS memory for isr
D7- <s> DCC unable to get DPM Buffer for Xmit
D2- <s> DCC-BMS out of memory
D9- <s> DCC DPM Invalid destination address
DCC errors found 20028 and was increasing

1708 <S>TCE1 Loop back relay,PTR1 1698<S> TCE1 TMR check trouble,PTR1 1354 <S> TCQA PTR1relay driver RD2 failure 1698 <T> TCE1 TMR check trouble,PTR1 1698 <R> TCE1 TMR check trouble,PTR1 2344 <S> Voter mismatch ,<S> L4PTR1 FB

thanks in advance.
rvsssrao
 
Did this problem start occurring after some incident or after some other card in <S> was replaced (like the LCC/SLCC or the TCQC card)?

>From the second group of Diag. Alarms, it appears that <S> has decided NOT to participate in running the turbine, so effectively you have only the designated voter (usually <R>) controlling the unit when it is running....

It is presumed that <S> is at I/O Status A7 at all times while this occurring. Have you used the LCC/SLCC display keypad to check the I/O Status of the cards in <S> and associated with <S>? (Search control.com for explanations of how to use the LCC/SLCC keypad to check I/O Status of individual cards.)

The first group of Diagnostic Alarms usually occurs when either there are too many requests being made of a processor OR when there is a problem with the configuration of the requests OR when there is a problem with the communication cabling OR if there is a problem with the hardware ("Berg") jumper settings on either the LCC/SLCC and/or the DCC/SDCC cards.

Unless there is some VIEWn routine which is asking for a LOT of information from <S> processor ALL the time, the first doesn't seem too likely--unless there's something you're not telling us. Like, is this on an LM Mark V panel???

Since it would most likely be only that one of the other processors (<R>, <T>, <C>) that would be configuring the requests of <S>, so this doesn't seem like it's the cause of the problem, either.

If the requests for data which didn't exist at a particular memory location were being made, this could also cause the problem--but it would seem that it would occur on every processor.

The DENET (Data Exchange NETwork) is how the control processors and <C> communicate. If there was a problem with the DENET or the cabling or the cards through which the DENET passes, this may cause problems similar to the one you are experiencing. The routing of the DENET is kind of vague, but includes the TCQC card (the VARC cable) and the LCC/SLCC card.

It's conceivable that there is a hardware ("Berg") jumper setting issue--have you confirmed that all the hardware jumpers on the LCC/SLCC, DCC/SDCC cards in <R>, and <S>, and <T> match each other? You should also check the VARC and ARCPL cable connectors on the TCQC card of <S> and the LCC/SLCC card of <S> (refer to the Signal Flow Diagrams in Appendix D of the Mk V Application Manual, GEH-6195, the drawing titled "<Q> Core - DENET and BUNet Connections on TCQC Terminal Board."

You say you've replaced the DCC card, so that shouldn't be the problem. Try replacing the LCC card, then the TCQC card.

Many times these nuisance Diag. Alarms don't result in problems with running the unit, but if your post is understood correctly and you have all the Diag. Alarms you listed in the post at the same time, then there is definitely something which is causing <S> to not participate in running (and protecting) the unit, and you are right to want to get it corrected.

It's really difficult to tell you exactly how to proceed, since we don't know exactly what PREceded this condition and what all has been done. And when someone says they.ve done this or that, ETC., it's difficult to know exactly what was done and how it was done.

There is a TIL (Technical Information Letter) out about using conductive grease on the connectors of the cards in Mk V turbine control panels. It's conceivable that there is a high-resistance connection in one of the connectors which is causing a problem. It would be necessary to shut down to do this, but have you did you use the conductive grease when you replaced the DCC card? Have you tried unseating and reseating the VARC and ARCPL connectors (again--do this only with the unit shut down and <S> powered down!)?

You can always consult with the OEM/packager for more help/assistance. If you have a GE-packaged unit you may be able to convince your local Customer Service representative to open a PAC (Power Answer Center) case for you.

Let us know what you find out!

markvguy
 
> Did this problem start occurring after some incident or after some other card in <S> was replaced (like the LCC/SLCC or the TCQC card)?

ANS : NO .STARTED SUDDENLY.SOME QUIET SOME MONTHS NO DCC ALARMS APPEARED.
>
> >From the second group of Diag. Alarms, it appears that <S> has decided NOT to participate in running the turbine, so effectively you have only the designated voter (usually <R>) controlling the unit when it is running....
>
> It is presumed that <S> is at I/O Status A7 at all times while this occurring. Have you used the LCC/SLCC display keypad to check the I/O Status of the cards in <S> and associated with <S>? (Search control.com for explanations of how to use the LCC/SLCC keypad to check I/O Status of individual cards.)

ANS : CHECKED ALL CARDS STATUS. SHOWING A7.

> The first group of Diagnostic Alarms usually occurs when either there are too many requests being made of a processor OR when there is a problem with the configuration of the requests OR when there is a problem with the communication cabling OR if there is a problem with the hardware ("Berg") jumper settings on either the LCC/SLCC and/or the DCC/SDCC cards.

ANS : SO HOW TO CHECK FOR TRAFFIC IN COMMUNICATION NODES.

> Unless there is some VIEWn routine which is asking for a LOT of information from <S> processor ALL the time, the first doesn't seem too likely--unless there's something you're not telling us. Like, is this on an LM Mark V panel???

ANS : NORMALLY WE ARE COLLECTING VIEWn DATA FROM <S> OF ALL GT PROCESSORS.PRESENTLY NOT INITIATED WHEN MACHINE IS RUNNING.
>
> Since it would most likely be only that one of the other processors (<R>, <T>, <C>) that would be configuring the requests of <S>, so this doesn't seem like it's the cause of the problem, either.

ANS : SO HOW TO CHECK. WHERE TO CHECK.

> If the requests for data which didn't exist at a particular memory location were being made, this could also cause the problem--but it would seem that it would occur on every processor.
>
> The DENET (Data Exchange NETwork) is how the control processors and <C> communicate. If there was a problem with the DENET or the cabling or the cards through which the DENET passes, this may cause problems similar to the one you are experiencing. The routing of the DENET is kind of vague, but includes the TCQC card (the VARC cable) and the LCC/SLCC card.
>
> It's conceivable that there is a hardware ("Berg") jumper setting issue--have you confirmed that all the hardware jumpers on the LCC/SLCC, DCC/SDCC cards in <R>, and <S>, and <T> match each other? You should also check the VARC and ARCPL cable connectors on the TCQC card of <S> and the LCC/SLCC card of <S> (refer to the Signal Flow Diagrams in Appendix D of the Mk V Application Manual, GEH-6195, the drawing titled "<Q> Core - DENET and BUNet Connections on TCQC Terminal Board."
>
> You say you've replaced the DCC card, so that shouldn't be the problem. Try replacing the LCC card, then the TCQC card.
>
> Many times these nuisance Diag. Alarms don't result in problems with running the unit, but if your post is understood correctly and you have all the Diag. Alarms you listed in the post at the same time, then there is definitely something which is causing <S> to not participate in running (and protecting) the unit, and you are right to want to get it corrected.

ANS : BUT WE 1 SHORT OF NUISANCE TRIP.

> It's really difficult to tell you exactly how to proceed, since we don't know exactly what PREceded this condition and what all has been done. And when someone says they.ve done this or that, ETC., it's difficult to know exactly what was done and how it was done.
>
> There is a TIL (Technical Information Letter) out about using conductive grease on the connectors of the cards in Mk V turbine control panels. It's conceivable that there is a high-resistance connection in one of the connectors which is causing a problem. It would be necessary to shut down to do this, but have you did you use the conductive grease when you replaced the DCC card? Have you tried unseating and reseating the VARC and ARCPL connectors (again--do this only with the unit shut down and <S> powered down!)?

ANS : GREASE WAS NOT USED.

> You can always consult with the OEM/packager for more help/assistance. If you have a GE-packaged unit you may be able to convince your local Customer Service representative to open a PAC (Power Answer Center) case for you.
>
> Let us know what you find out!
> markvguy
 
Much of the first reply was kind of "thinking on html", kind of going through a mental check-list of what could be causing the problems. Unfortunately, the designers of the Mk V did not provide much in the way of internal troubleshooting capabilities, so easily checking for DENET traffic is not possible. In fact, troubleshooting internal operations is difficult even for GE factory personnel, since there are not many avenues to get at internal operating data and there aren't many engineers left that familiar with Mk V turbine control internal operations.

One of the first questions to ask here is: Do you have <I>s or GE Mark V HMIs? If you have HMIs, have you always had HMIs or were they recently purchased/installed--along with new PROMset(s)???

Have you implemented TIL-1480 R2?

It's VERY interesting that the problems are only being experienced on <S>, and that <S> is being "interrogated" using one of the VIEWn tools.!.?.!

Which VIEWn tool(s) are you using to gather data?

Are you using a file to list the points to be gathered by the VIEWn tool? If so, can you copy the pointlist file to a reply?

Are you using a batch file to run the VIEWn tool? If so, does the VIEWn utility generate any "warning" when it is started by the batch file?

At what rate is the data being gathered with the VIEWn tool? Please list the command line entry which is used to start the VIEWn tool?

When did you start using VIEWn to gather data from the <S> processors? Did you make any changes to the VIEW2 data file/routine around the time the DCC errors began?

What happens when you clear the system errors from the LCC Display keypad when the VIEWn utility is NOT running? (To clear system errors using the LCC Display keypad, press DCC on the LCC display keypad; '186 MONITOR' should appear in the display. Press INC until 'CLEAR SYS ERRORS' is shown in the display, then press ENTER. Press INC again until 'CLEAR SYS ERRORS' is shown in the display and press ENTER again. If the number shown is 0, then errors have been cleared and reset; if the number is small or is incrementing while you are watching it, the errors have not been cleared. Press NORM to return to the default display, or the display will return to the default display after a brief period of time.)

You seem to be saying that there are periods of time when the alarms do not appear, then they suddenly start appearing again. Is that correct?

Have you tried replacing the LCC card?

Again, it's very difficult to say how to proceed because it's very difficult to know what's been done, what's being done, and how things were done. Don't take offense at that; it is sensed that you are somewhat frustrated with the responses you are getting, but you should know that there are LOTS of sites which are experiencing the SAME problems you are and they run just fine day in and day out--and even GE doesn't know exactly what to do to resolve the problem(s). You are getting assistance from a free forum--if you were paying for this advice, you might have a legitimate gripe.

You're comment about being "...1 short of nuisance trip..." is completely baffling.

Unfortunately, because of the lack of troubleshooting tools for problems such as this, it quite often boils down to the (repugnant) task of swapping cards...

It is recommended that you shut the unit down, power down the <S> processor, unseat (unplug) all the cables one at a time and before plugging them back in applying the recommended conductive grease to the terminals (you can usually find conductive grease packed in the boxes with spare printed circuit cards; GE started shipping the grease in the late 1990's with all cards sold as spares or repair-and-return replacements), and then reseating the cables. This should be done on all the cables of the LCC card, the DCC card, and the TCQC card--especially the cables labelled VARC and/or ARCPL.

Be sure to pay attention to the orientation of the cables on the LCC card, especially the cables attached to connectors at the bottom of the card. Also, be especially careful with the 3PL cable which runs between the LCC card, the DCC card, and the TCQA (and TCQB, if so equipped). The 3PL cables does not have pull tabs and it can be very easily damaged if not pulled loose very carefully and reseated by pressing very firmly all along the top rib of the connector!

If that doesn't resolve the problem, then try replacing the LCC card.

This author believes there is some "feature" of certain Mk V PROM revisions which, when activated by an incorrectly formed request for data or for data which which doesn't exists or for data at an "excessive" rate causes the DPM (Dual-Ported Memory) and associated memory buffers and memory handling routines to be corrupted, sometimes beyond "repair". In most cases, these alarms are nuisance alarms and don't adversely affect the operation of the unit (again, if a poll of Mk V operators/technicians was taken, it would probably reveal that somewhere between 30- and 40% of panels exhibit this behavior either intermittently or continually...sad, but true!).

What's troubling in your case is the PTR1 Feedback trouble and relay driver failure alarms, which indicate more serious problems--possibly (likely) not even related to the LCC errors. That's one of the difficulties of dealing with issues like this that have existed for some period of time before something else happens which causes people to take notice and begin to "relate" unrelated things. Understanding and troubleshooting Mk V internal operations and Diagnostic Alarms takes quite a bit of patience and on-the-job experience. And, has been said many times previously--ANY process- or diagnostic alarm should be troubleshot and resolved as quickly as it is annunciated. Failure to do so can cause huge amounts of time and effort to be wasted; take it from experience.

markvguy
 
ANS(RVSSSRAO) : WE HAVE HMI'S SINCE MACHINES WERE COMMISSIONED IN 2003.
>
> Have you implemented TIL-1480 R2?

ANS(RVSSSRAO) : TIL 1480 R2 WAS IMPLEMENTED BUT SAME DIAGNOSTIC ALARMS CONTINUED.

> It's VERY interesting that the problems are only being experienced on <S>, and that <S> is being "interrogated" using one of the VIEWn tools.!.?.!
>
ANS(RVSSSRAO) : WE COLLECT VIEW 2 DATA FOR REFERENCE PURPOSE & DATA IS COLLECTED FROM <S> CORE ONLY. WE HAVE 6 GT'S & FOR ALL GT'S WE COLLECT DATA FROM <S> CORE. BUT WE HAVE DIAGNOSTIC ALARMS FROM <S> OF ONE OF OUR GT'S.

> Which VIEWn tool(s) are you using to gather data?
>
ANS(RVSSSRAO) WE USE VIEW 2 FILE FOR DATA COLLECTION.

> Are you using a file to list the points to be gathered by the VIEWn tool? If so, can you copy the pointlist file to a reply?
>
ANS(RVSSSRAO) : WE CREATE A FILE WITH LIST OF POINTS TO BE COLLECTED.

> Are you using a batch file to run the VIEWn tool? If so, does the VIEWn utility generate any "warning" when it is started by the batch file?
>
ANS(RVSSSRAO) : WE RUN BATCH FILE FOR COLLECTING DATA.

> At what rate is the data being gathered with the VIEWn tool? Please list the command line entry which is used to start the VIEWn tool?

ANS(RVSSSRAO) : 1 SAMPLE PER MINUTE FOR 50 POINTSMAXIMUM.

> When did you start using VIEWn to gather data from the <S> processors? Did you make any changes to the VIEW2 data file/routine around the time the DCC errors began?
>

ANS(RVSSSRAO) : WE USE VIEW 2 FILE WHEN WE DO FUEL CHANGEOVER ONLY OR DURING PERFORMANCE TESTS ETC.NO ACTIVITY WAS DONE RECENTLY ON VIEW2 FILE.

> What happens when you clear the system errors from the LCC Display keypad when the VIEWn utility is NOT running? (To clear system errors using the LCC Display keypad, press DCC on the LCC display keypad; '186 MONITOR' should appear in the display. Press INC until 'CLEAR SYS ERRORS' is shown in the display, then press ENTER. Press INC again until 'CLEAR SYS ERRORS' is shown in the display and press ENTER again. If the number shown is 0, then errors have been cleared and reset; if the number is small or is incrementing while you are watching it, the errors have not been cleared. Press NORM to return to the default display, or the display will return to the default display after a brief period of time.)

ANS(RVSSSRAO) : EVEN AFTER RESETTING ALARMS ARE INCREASING CONTINUOSLY WHEN VIEW 2 FILE IS NOT RUNNING.WE DO THE SAME PROCEDURE FOR CLEARING OF ALARMS FROM LCC.

> You seem to be saying that there are periods of time when the alarms do not appear, then they suddenly start appearing again. Is that correct?

> ANS(RVSSSRAO) : FOR 3-4 MONTHS FROM NOW NO ALARMS WERE EXISTING .SUDDENLY DURING MAINTENCE WE SWITCHED OFF ENTIRE MARK V PANEL POWER SUPPLY & SICE THEN IT STARTED.

> Have you tried replacing the LCC card?

ANS(RVSSSRAO) : YES

> Again, it's very difficult to say how to proceed because it's very difficult to know what's been done, what's being done, and how things were done. Don't take offense at that; it is sensed that you are somewhat frustrated with the responses you are getting, but you should know that there are LOTS of sites which are experiencing the SAME problems you are and they run just fine day in and day out--and even GE doesn't know exactly what to do to resolve the problem(s). You are getting assistance from a free forum--if you were paying for this advice, you might have a legitimate gripe.
>
> You're comment about being "...1 short of nuisance trip..." is completely baffling.
>
> Unfortunately, because of the lack of troubleshooting tools for problems such as this, it quite often boils down to the (repugnant) task of swapping cards...
>
> It is recommended that you shut the unit down, power down the <S> processor, unseat (unplug) all the cables one at a time and before plugging them back in applying the recommended conductive grease to the terminals (you can usually find conductive grease packed in the boxes with spare printed circuit cards; GE started shipping the grease in the late 1990's with all cards sold as spares or repair-and-return replacements), and then reseating the cables. This should be done on all the cables of the LCC card, the DCC card, and the TCQC card--especially the cables labelled VARC and/or ARCPL.

ANS(RVSSSRAO) : WE WILL BE DOING THIS BUT HOW DIFFICULT TO REMOVE THE CONNECTOR ONCE GREASE HAD BEEN APPLIED.

> Be sure to pay attention to the orientation of the cables on the LCC card, especially the cables attached to connectors at the bottom of the card. Also, be especially careful with the 3PL cable which runs between the LCC card, the DCC card, and the TCQA (and TCQB, if so equipped). The 3PL cables does not have pull tabs and it can be very easily damaged if not pulled loose very carefully and reseated by pressing very firmly all along the top rib of the connector!
>
> If that doesn't resolve the problem, then try replacing the LCC card.
>
> This author believes there is some "feature" of certain Mk V PROM revisions which, when activated by an incorrectly formed request for data or for data which which doesn't exists or for data at an "excessive" rate causes the DPM (Dual-Ported Memory) and associated memory buffers and memory handling routines to be corrupted, sometimes beyond "repair". In most cases, these alarms are nuisance alarms and don't adversely affect the operation of the unit (again, if a poll of Mk V operators/technicians was taken, it would probably reveal that somewhere between 30- and 40% of panels exhibit this behavior either intermittently or continually...sad, but true!).
>
> What's troubling in your case is the PTR1 Feedback trouble and relay driver failure alarms, which indicate more serious problems--possibly (likely) not even related to the LCC errors. That's one of the difficulties of dealing with issues like this that have existed for some period of time before something else happens which causes people to take notice and begin to "relate" unrelated things. Understanding and troubleshooting Mk V internal operations and Diagnostic Alarms takes quite a bit of patience and on-the-job experience. And, has been said many times previously--ANY process- or diagnostic alarm should be troubleshot and resolved as quickly as it is annunciated. Failure to do so can cause huge amounts of time and effort to be wasted; take it from experience.

ANS(RVSSSRAO) : WHAT NEED TO BE CHECKED FOR PTR1 PROBLEM.

THANKS FOR ADVICE & SUGGESTIONS.
 
Hi,

Very very interesting and troubleshooting tips, hope this will help others also.

In this context there is one particular sentence I am not able to understand: "Since it would most likely be only that one of the other processors (<R>, <T>, <C>) that would be configuring the requests of <S>, so this doesn't seem like it's the cause of the problem, either."

Would you please elaborate the meaning of this sentence?

Regards,
RAM
 
One thing you should know about GE Mark V HMI Servers versus <I>s: They generate a LOT more traffic on the StageLink than <I>s--significantly more! And that's even BEFORE one starts running VIEWn programs for data-gathering. (An HMI Server is an HMI with an ARCnet card that is communicating with Mk V turbine control panel(s) on the StageLink.)

An <I> only asks for data for points which are configured to appear on the display which is currently being viewed on the monitor--as well as any MODBUS or GSM (GE Drive Systems Standard Message Protocol) points, if the <I> is configured for either of those communication options.

A GE Mark V HMI Server is ALWAYS asking for EVERY POINT configured in the CIMPLICITY Point Database which is every point on every display--and if it's a multi-unit GE Mark V HMI Server, well it's asking for that information for every unit it communicates with! (It is thought that's why GE has set a "limit" of no more than two GE Mark V HMI Servers asking for data from a single Mk V turbine control panel).

By the way--how many GE Mark V HMI Servers are asking for data from the Mk V turbine control panel which has the problems with the <S> processor? (An GE Mark V HMI Server

There should be no increase in removing the connectors when conductive grease is applied; it is non-hardening and "greasy" (i.e., slippery) and should actually make removing the connectors easier. There was usually an information sheet packaged with the tubes of conductive grease about how to apply the grease.

Again, many times resolving an issue such as this comes down to replacing cards; this author abhors "troubleshooting" that way, but sometimes there is no other way. Again, this author knows of many sites which experience this same problem, have never even tried to troubleshoot the problem, and have experienced no problems trying to start or run the units.!.?.!.? It's one of the "mysteries of Mk V."

One interesting thing that can be done is to swap cards between processors and see if the problem follows the card. For example, take the DCC from <S> and exchange it with the one from <R>, do the appropriate configurations and downloads after powering-up, re-boot and if the problem starts occurring in <R> and not in <S>, then the problem is the DCC card. You can do the same with the LCC card. If the problem stays in <S>, then it may be a VARC or ARCPL cable/connector problem.

markvguy
 
It is presumed that <R>, <S>, <T> and <C> request information from each other at the beginning of the frame. In thinking about, it seemed that if the requests of <S> were malformed by one processor, they would be malformed to the other processors, who would be complaining similarly.

That's pure speculation, but it doesn't seem illogical. The internal operating system of the Mk V isn't documented and there isn't any convenient way to observe the requests on DENET (Data Exchange NETwork).

Again, most of this is just "thinking on html" trying to logically derive what the problem might be.

This author has seen several cases of individual processors being powered-down and then misbehaving inexplicably when powered-up--especially when the main breaker feeding the entire panel is used to power down the processors and power them back up all at the same time. Or, when the switch in the <PD> core is cycled very quickly. Sometimes, just powering down for a minute or so with the switch in the <PD> core will resolve the issue; sometimes it won't.

This is a practice which is quite common in Mk V training courses--when there's NO inputs or outputs connected to the panel, and when they are being powered by AC-DC converters with limited current output capability. In the field, Mk Vs are generally powered by nominal 125 VDC batteries with very high current output typically through a 50A breaker, and chargers with high current outputs connected to the battery. This author has never condoned the practice of re-booting an entire panel with the main breaker, even in training classes. And what people see in class, they usually carry back to the field....

It will be interesting to learn how the Mk V panel was powered down.

markvguy
 
We have 6 GT's with Individual HMI & control system at local & 3 nos. of multi user licensed HMI at main control room ( 2 for <D> Core & 1 for <C> Core)

all are connected in daisy chain mode with Fibre optic cable via MOD HUB.for all GT's we are collecting data from <S> Computer.

What is this it VARC or ARCPL cable/connector.

Thanks for reply in advance.
 
>From the 9th paragraph of the first reply, "...refer to the Signal Flow Diagrams in Appendix D of the Mk V Application Manual, GEH-6195, the drawing titled "<Q> Core - DENET and BUNet Connections on TCQC Terminal Board...."

markvguy
 
Dear Mark V Guy,

We got the opportunity to go one step ahead in carrying out the further diagnosis in solving the <S> Core DCC Errors.

We forced some signals like L4_XTP, L20 FG1X ETC to check the Relays status at TCTG Card. We found k14 relay not getting activated. So we checked the cable from TCQA Card of <S> core to TCTG Card. We repalced the cable & found no change in status. So find concluded that TCQA Card is faulty. So we replaced the Card. Presently no diagnostic alarms appeared till now. Machine was taken back into service 2 days back. It's kept under observation. Any further action will be intimated to you.

Thanks for your action & continued support.

rvsssrao
 
Congratulations on finding the problem, and thanks for the feedback! It helps everyone who reads the posts and have or may have the same or similar problems. Thanks also for the details of the troubleshooting; they are also very helpful to this author and others, as well.

markvguy
 
Top