Dear all

Our plant is using MARK-V system.

Few days ago, one alarm, "Forced Logic Signal Detected", occurred.

It is problem, because we did not perform the logic forcing process.

We found some point in TGM system(_L3232_SP, _L3241_SP etc.) are on Logic forcing display now.
Most of the points are set on "0" and one point, _L3241_SP, is set on "1". We guess that the original signals are "0".

There was no event, alarm and operation on MARK-V when alarm occurred.

I am not sure that it is a software problem or a hardware problem.

I just write down this post thread to get any information or experience.

If you have any experience or opinion about this problem, please reply for me.

Thanks in advance

 

You need to speak with TGM about this problem.

In the past, it has been noted that poorly configured communications requests--trying to force signals that don't exist; making LOTS of data requests for HMI display; and poorly and incorrectly configured MODBUS data requests--have caused the problem.

As with most plants that purchase a third-party HMI (non-GE) they almost ALWAYS leave at least one GE operator interface (<I> or GE Mark* V HMI) still connected to the StageLink, and running "in the background." (Someone in management just isn't fully convinced the third-party HMI is reliable and trustworthy.... But they bought it because it was cheaper than GE's offerings.) If that operator interface is a GE Mark* V HMI, particularly if it's a multi-unit HMI, it can be adding significantly to the StageLink network traffic. AND, worse, MANY GE Mark* V CIMPLICITY projects were configured with signals that didn't actually exist in the CDB (Control Signal Database) of the Mark* V, and that leads to unexplained phenomena. Add to that the bandwidth required for MODBUS (particulary MODBUS TCP)--requesting LOTS of signal values (just because someone thought it should be done) and doing so at a very fast rate, including signal names that don't actually exist or were improperly typed or copied-and-pasted, and then throw in one or more third-party HMIs, and that there is a real recipe for quite a mess.

You, probably with the help of the TGM provider, needs to review communications for proper configuration, reasonable communication request rates, and eliminate any unnecessary or improper signal name values. And seriously consider shutting down and GE operator interfaces (<I>s or HMIs) if there are any. Grow some hair; dive into the pool, instead of just dipping your pinky toe in the water.

There may be other issues, too, especially if the TGM is a MODBUS slave (and if there are multiple TGM MODBUS slaves), and if the GE Mark* V HMI is still being used as a MODBUS slave. People think they can ask for the value of just about every signal in the Mark* V CDB at very fast rates, and end up not even storing the values of 10% of the signals for which values are being requested! (AND, they get upset when the Mark* V can't keep up with the data requests! Not recognizing the MODBUS communications go through an operator interface/HMI and then to the Mark* V and back to the operator interface/HMI and to whatever is making the data requests (yes; it's a lot of bandwidth, and the StageLink is robust and reliable--up to a point. My memory seems to think it's 2.5 MB system, or something like that. And there's the DENET (Data Exchange Network) inside the Mark* V which lets all the processors talk to each other and share data, and <C> (and possibly <D>) all have to pass that through the StageLink. There really is a limit to this stuff, and while the technology (ARCnet) is good, it's getting older every day, month, year, decade.

Also, if any of the Control Processors are experiencing LOTS of error messages (as displayed on the LCC/SLCC keypad display) and the number of errors continuously increases, every few seconds, that's a sure sign of communication issues. That needs to be investigated and resolve because that's going to lead to problems almost every time, and usually--they are self-inflicted problems (incorrect data request configurations; excessively fast data requests; huge number of signal value data requests).

 

@NOCU,

I wanted to add that it would appear the signals you are listing are spare logic points, not used for any purpose in the CSP (Control Sequence Program). [I am presuming this is a Mark* V--and not one of the Mark* V-to-Mark* VIe perversions.]

In the Mark* V any logic signal that IS NOT written to by some function (logic; algorithm; etc.)--such as spare logic points--will, if forced, remain at the forced value if simply unforced, because there is nothing telling the logic signal to be either a logic "0" or a logic "1". I have seen several Mark* Vs that, when rebooted in an unrecommended manner, will set one or three or six or seven logic signals to some value other than logic "0" (the default state). These would appear in the Logic Forcing Display and would cause some consternation and concern at the site. I attributed this to DENET and RAM communication issues between processors.

I NEVER saw any logic signal what was written to in the CSP get inadvertently forced to any value during boot-up/initialization. EVER.

I also know that some people leave the Logic Forcing Display up and visible on a GE Mark* HMI, usually with one or more signals populating the display as they monitor their state/value (because one can put real (analog) values on a Logic Forcing Display--but they ARE NOT forceable). People see this as an easy way to monitor logic and/or real (analog) values for troubleshooting purposes. HOWEVER, there is a flaw in this practice--the Logic Forcing Display is continually (multiple times per second!) polling all of the processors for any forced signals. This GREATLY increases the network traffic on the StageLink as well as on the DENET between processors in the Mark* V. THIS is not desirable, even with GE Mark* V HMIs as the CIMPLICITY project is also polling the Mark* V for the states and values of all signals in the CSP--even ones not currently being shown on any CIMPLICITY display. Add the multiple requests per second to the CIMPLICITY requests, along with alarm and event traffic on the StageLink and, if there are multiple Mark* V panels (and even EX2000 exciters) and the network traffic can be, well, busy like the I-405 on a Friday afternoon where cars are literally crawling between stops (for hours).

I don't know how the TGM works--you'd need to ask them--but I suspect it is similar to the GE Mark* HMI in that is continually requesting the states and values of CSP signals even for signals that are not currently visible on any display/screen. (This is also certainly true of any MODBUS functionality in the TGM, as well as the GE Mark* V HMI....)

And, if you have even one GE Mark* V HMI still working on the StageLink (as most sites do because they just can't "cut the cord" when they spend thousands of dollars/Euros on a third-party HMI...) that already has a pretty high traffic "presence" on the StageLink and the DENET.

All this being said, poorly configured HMI software (CIMPLICITY and possibly TGM), and poorly configured TGM software), and MODBUS (if being used) can all confuse the Mark* V processors. This usually results in continuously incrementing error messages on LCC/SLCC keypad displays--which most sites ignore because they don't trip the turbine....

That's all I got.

Please let us know how you resolve this problem.

 

Thanks for your reply.

As you said, the forced points are spare points, we guess. Because we can not find them in CSP, Prevote, unitdata.dat and io.asg. We can just try to do UNFORCE ALL, but we do not. Because we do not know the reason why the points are forced yet.

On our cabling drawing, MARK-V and TGM network is connected with each "C" core and "D" core with STAGELINK. And MARK-V C core communicates with HMI#1, #2 and Historian <H>. MARK-V D core communicates with HMI#3 and Historian <H>.
In case of MODBUS, HMI#1, #2, #3 and <H> communicates with our DCS(called PMS). PMS requests lots of information from MARK-V and TGM. HMIs are GE MARK V HMI.

Few days ago, in TGM panel we heard some noise like "pi----------", high frequency noise or dolphin's sound. Based on your opinion, we assumes "C" or "D" CORE in TGM has some problem. But on display each core, "A7" and "40%" messages only. It looks like normal status so we are not sure if the situations are related or not.

I think if we do something, just try the UNFORCING ALL function or we can also power off the "C" core or "D" core to find the reason of noise in TGM and replace it.

The problem, you said, like traffic, poorly configured HMI software and MODBUS, are can not solved now.

I will try to solve this problem with replacement or reset the process and if we find the reason i will inform to you.

Anyway I really appreciate with your kindness.

 

Would you please define TGM and where the mnemonic came from? The only TGM I know of with regard to Mark* V is a third-party HMI system, so I’m confused. Is the Mark* V not being used to control a combustion turbine or steam turbine?

The high-pitched whine might be the result of failing components (capacitors, if I recall correctly) on the TCPS card in Loc. 5 of the processor cores (<C>, <D>, <R>, <S>, <T>).

I never recommend using UNFORCE ALL. Just select one signal at a time and click on UNFORCE. If, as we suspect, the signal(s) are spare when unforced the logic state will most likely not change because there is nothing telling the signal to be a logic “1” or logic “0”. If that’s what happens when you unforce one individual signal of this group of signals ending in _SP then do the same with the remaining signals ending in _SP.

If management is frightened of this recommendation and the equipment is running without any other problems with this group of _SP signals then just leave them be as they are and wait until management deems it safe to take some action (unforcing the signals one at a time, and/or replacing TCPS card(s)).

But with that many HMIs and a Historian and <C> and <D> cores and all those MODBUS connections it’s actually surprising (to me, at least) this situation hasn’t happened before. I would be very surprised if there wasn’t Diagnostic Alarms related to this problem and/or error messages on at least one of the processor LCL/SLCC displays related to this problem.

There are third-party companies that refurbish Mark* V cards, and a couple of them claim to have developed and/or sourced reliable components for the known TCPS issues.

Finally, several years ago there was a serious problem with a Mark* V that was very poorly written up but was ultimately found to be the result of extreme overloading of the StageLink that had been ignored for a very long time with no effort to fix or resolve the issue. The processors were just being repeatedly hammered by the HMIs and MODBUS requests (badly configured requests, if I recall correctly). It was described (incompletely!) in a post to Control.com; it should be still available in the archives of Control.com but I don’t know how to narrow a search to find that one thread. If you find it, please don’t be alarmed about how the situation was originally reported—again the description was vague and incomplete and I believe one contributor did attempt to get the original poster to clarify the claims with correct information.

Anyway, if you would help to clear my confusion over TGM I would be very grateful. Thank you!