bpsemd,

If the problem stays in <S> all the time, and you've swapped or replaced the TCQA, SDCC and SLCC, and the T/C termination board on <R> and the problem goes away for a short time then reappears it would seem it's poor contact of pins/cables and/or bad cables. It's not difficult to swap 3PL cables between processors, and it might even be possible to swap the JAS with JAR or JAT at the TBQx (the T/C terminal board on <R>) just for a test to see if the problem is the 3PL or the JAS. that would allow you to concentrate your efforts on cabling/contact integrity--instead of card failure(s), which appear to have been ruled out.

Troubleshooting is, many times, a process of elimination--focus your efforts in one area, and that proves unfruitful then move to another area. You seem to have eliminated cards, so now the focus can move to cables and/or contact integrity. When troubleshooting a problem you've never experienced before, try to break it down into discrete areas and then work logically and methodically through each area until you've eliminated those that haven't proved to resolve the problem until you find the area or component that does.

"Be smarter than the equipment," is often my mantra. It's difficult to remember but we are. (Smarter than the equipment.)

The Fortune at the bottom of this page read, "Patience is a form of despair, disguised as virtue." Sometimes that is very true. But don't give up! You've eliminated a lot of things; so it's a safe bet you're close to the resolution!

 

What happened to me recently is not an answer for what is wrong in your situation, but it shows how what appears to be a bad thermocouple or a bad thermocouple card is neither and the situation requires more effort to figure out what's wrong.

I had a thermocouple problem last week on an 8 channel AI card that is probably 6-7 years old. The real problem wasn't the thermocouple or the AI card even though the symptoms screamed "open thermocouple" and when that wasn't the case, "bad AI card" and then finally "bad terminal block", but it wasn't that either.

The PAC is a modular 4 slot rack in a portable monitoring panel and it had worked OK as recently as the day before but at that moment was back in the shop for check-out for the next job.

It was configured for Type J, 0-1200 Deg F. The displayed value was 1200 Deg F (upscale burnout, for this channel) with the thermocouple sitting at ambient temperature, probably 65 Deg F. The AI card status LED was green, not red or blinking red.

I shorted the AI input with a jumper cable so it should have read 65 Deg F plus/minus whatever. The reading stayed at 1200 Deg F. "Upscale burnout" was not a true indication, because there was a dead short across the input. But the dead short did not read the CJ temp like it should have, leading me to believe that it was an AI card problem, probably a CJ problem.

There was an unused channel on the AI card, ch 8. This model uses two CJ thermistors, one at the bottom, one at the top. I could not recall whether the card faults with only one bad CJ or two, but I figured I'd try the unused channel at the other end of the card because it's easy to quickly enable, configure and check. Putting a T/C on channel 8 had exactly the same results: indicated 1200 Deg F at a real temp of 65 deg F.

So, it looked like the AI card was the problem, in spite of its green status LED. The cards are hot swappable, so I pulled the AI card just to have a look at it. It looked OK, there was no apparent overheating damage or smoke residue. I plugged in a known good replacement AI card into the slot and saw the same problem: 1200 Deg F with a jumper across the terminals.

What? I pull the terminal block off wondering if it was a bad terminal block. I measured 0 ohms across the backside teminals with the jumper across the front side screw terminals. The block tested good but I got a spare terminal block just in case.

This time, as I push the replacement terminal block onto the AI card, the guy I'm working with yells that the display momentarily showed 65 Deg F. Huh? Maybe the problem is the terminal block. I pull the terminal block off, push it back on, but there's no change in temp; temp is still at 1200 Deg. I wiggle the terminal block, no change.

Time to think. Where did the momentary good reading come from? It's known good AI card, known good input, but a bad reading. Back plane? CPU? Backplane problems are really rare. But so are CPU problems.

The AI card is in the first slot, next to the CPU. I'm sitting there, thinking this through it and absent mindedly gazing at the AI card. It's then that I notice that the bottom CPU rack screw is loose, in fact neither CPU screw is even started into its rack nut. Both are captive machine screws 'hanging loose'. The CPU is not tight in its slot! I push the CPU in and it slides back a little bit and bottoms into the rack. The guy at the display yells, "Hey, it's working. 64 Degrees".

The CPU was pushed in just far enough for its backplane connector to get power and process some logic, but not far enough to get a valid T/C reading on the slot 1 AI card. Go figure. When the CPU was screwed down, everything worked fine. Although it worked the day before, the CPU had apparently worked loose during shipment for this particular problem to occur. I probably couldn't recreate that exact 'marginal connector contact' situation again, if I wanted to. It was a one time situational "flakey".

But the point is, what looked like a thermocouple problem (upscale burnout) and then an AI card problem (upscale burnout with a dead short across the input) or maybe even a terminal block problem (momentary good reading when pushing the terminal block) can turn out to be something else all together.

Keep your eyes open and WRITE DOWN EVERY TEST you make and its results. All this stuff gets blurry over time, particularly over an extended period of time. Clear, written notes help enormously in reviewing what's been done and what else might need to be looked at.

 

I don't know anything at all about the machine you guys are talking about but one thing puzzles me.

Why is every third one in error?

Have you tried checking out the Cold Junction Compensation?
Remove all the thermocouples or thermocouple compensation cable and short the input terminals together.

If you do this the display should show the ambient temperature at the input terminals. This is coming from the cold junction device. This is a fairly standard troubleshooting method for thermocouple input devices.

 

Roy Matson,

The Speedtronic Mark V in this case is a TMR (Triple Modular Redundant) gas turbine control panel, and Speedtronic turbine controls are purpose-built control systems--similar to but sometimes very different from other industrial control systems. The exhaust T/Cs are divided into three groups--with No's 1, 4, 7, 10, 13, 16, 19, etc, going to one processor (<R>); No's. 2, 5, 8, 11, 14, 17, 20, etc,going to the second control processor (<S>); and No's 3, 6, 9, 12, 15, 18, 21, etc. going to the third processor (<T>.).

Any critical T/Cs--and exhaust T/Cs are critical T/Cs--are triple redundant--even though they all terminate on one T/C input terminal board mounted on the <R> control processor. Critical T/Cs are "fanned out" to the TCQA analog I/O cards in each of the three control processors. TCQA I/O cards handle T/c inputs, mA inputs and outputs, Speed inputs, servo-valve output so, LVDT inputs, etc.

Each TCQA does its own cold junction compensation and the signal name is TCQA_CJ_0. The cold junction compensation for <S> appears to be working fine--it's just that even though the TCQA cards have been replaced and swapped between control processors AND the T/C input terminal board has been replaced the T/C values of <S> processor all revert to -18 deg C from normal values after some time.

All of the terminal boards and most of the I/O cards in the Mark V are connected with ribbon cables--which are known to be susceptible to corrosion. Conductive grease can prevent corrosion for a couple of years but should be reapplied to ribbon cable connectors every two years or so.

The original poster can try the trouble-shooting methods you have suggested but I suspect that in time the value of any T/C input to <S> is going to revert to -18 deg C. That's why it's been recommended to swap ribbon cables to see if the problem follows a particular ribbon cable--but conductive grease should also alleviate the problem if applied properly.

This purpose-built control system is an odd duck and doesn't always lend itself to normal troubleshooting methods. There is no backplane the cards are plugged into--they're all connected using cables, mostly ribbon cables--they're the only components which haven't been replaced yet. Even the microprocessor card in <S> Has been replaced--to no avail--which is another reason why it's been suggested to try swapping cables AND applying conductive grease to all ribbon cables and -connectors.

Hope this helps to explain the odd duck that is Mark V--at least in a very general way. Thanks for the inputs; it's good to know how other industries and industrial controllers operate and are troubleshot. Too often the simple things are overlooked when working with Speedtronic turbine control panels since they are similar to--but not exactly like--most industrial control systems.

 

Hello Everyone,

Thanks lot again for all of your valuable suggestions. At last, the problem is over after replacing the SDCC card of <S> core 3 days back. I've taken this time to inform you to be sure whether this problem come back again or not. To CSA: According to your advice, I'll grease the connector A.S.A.P.

I've another issue. An alarm (NO. 377) is there. AVR EXCITER FIELD BKR OFF. Actually this alarm comes few days ago and I didn't mentioned it as there was a big problem going on. This alarm comes during our diagnosis about Exhaust TC issue. We decided to Format and Download programs to <S> core. When we did that, we found there was no Sequence file (SEQ*.src). So, we copied the sequence file from a backup and again try to download. This time everything goes fine but TC issue did not resolve. Moreover, above mentioned alarm comes.

Now, is this an alarm? Coz, when our machine was running this alarm or even AVR EXCITER FIELD BKR ON never comes. So, how can we remove this? or is it a notification only?

 

bpsemd,

Congratulations, and thanks very much for the feedback! "Feedback is the most important contribution!"(c) here at control.com--it's what sets this forum apart from most others on the World Wide Web.

I'm sure you had your reasons for downloading; I'm not going to comment on that. What's very surprising is that--if I understand correctly according to the information provided--you only downloaded to <S> and then this alarm came up. That's pretty odd because one would think the limit switch of the breaker (or the UV relay--we don't know what drives the alarm at your site) would be terminated on the <QD1> core, and that it would be fanned out to all of the control processors. If it's terminated on the <CD> core and then fanned out to the three control processors, ..., well, even then it's still hard to understand how just downloading to <S> would cause this problem--unless there was a Voting Mismatch alarm prior to downloading to <S>, AND there is some kind of difference, either in the CSP or the I/O Configuration of this particular discrete input, that resulted in two out of three control processors acknowledging that the breaker is open.

The likely purpose of this alarm is to warn the operator prior to starting the unit that there is no AC power to the exciter regulator, so therefore AC terminal voltage will likely not build up to the required level. And, also, if something caused the breaker to trip while the unit was running this alarm would alert the operator to the fact that AC power to the exciter regulator was lost--hence the reason for all the other alarms!

Personally, I'm not a fan of using a Process Alarm to alert operators of something that is normal--like the AC breaker being closed, which it should be for most normal operations. It just breeds contempt for alarms and causes most operators to just ignore any alarm unless they hear the sound of the compressor bleed valves opening when the unit trips, or the sound of the generator breaker opening when the unit trips. And, then they can't distinguish which of the multitude of alarms--many of which they've been "overlooking" (the politically correct way of saying IGNORING which is what most operators do with most alarms) is the cause of the trip. Operators just aren't held accountable for Alarm Management like they should be, and they aren't trained and don't take the initiative to learn and understand every alarm coming from the turbine control panel--which is THEIR job. Their job is not to call someone every time an alarm is annunciated and just continue to eat biryani and read the newspaper and let someone else deal with the alarm(s).

Anyway, you would need to provide a LOT more information, such as the name and type of device used to sense the exciter power AC breaker position, where that signal wire is terminated, what related Diagnostic Alarms are associated with that input, what the Pre-Vote display value of that input is (presuming it's terminated on <QD1>), the actual state of the sensing device when the breaker is open--and when it's closed; the value of the inversion mask for that input, and the alarm rung for the alarm from the CSP.

Best of luck with the re-start; hope all goes well! And thanks again for the feedback!

 

CSA,

Thank-you for the very clear explanation.

I still believe if you short out the thermocouple input terminals the system should see ambient temperature, that's been true of any thermocouple system I have ever worked on. Shorting the terminals will tell you if its inside the unit or outside that's all it should read the terminals temperature.

I wondered if -18°C is the cold junction reversed somehow.

The bad connection theory certainly sounds plausible, I have often seen a little cell set up by some corrosion.

Conductive grease, that sounds like something I have never seen, usually an organic compound like copper coat is non conductive, electrically at any rate. You aren't talking about di-electric grease are you, that's the worst stuff for an electrical contact.

Sorry, there's no need to respond, as I say I have no knowledge of your system.

Cheers
Roy

 

>But if we open any thermocouple connection in MarkV, then
>value shows -83 0c. Please give your valuable suggestion.

To add to my last post I would also be looking to find a thermocouple grounded in the field.

Shouldn't it failsafe upscale or is it a low temperature trip?

 

Roy Matson,

You are correct that if the T/C input terminals are shorted at the T/C terminal board the Mark V will display ambient temperature.

But, again--being a purpose-built control system the Mark V goes downscale when the input circuit is open. There are lots of reasons for this, but for turbine control systems this is considered failsafe.

And, the Mark V can accept grounded or ungrounded T/C inputs without any issues.

Again, Speedtronic turbine control systems are really something of an oddity in the industrial control world, but all of these kinds of decisions were all made with reliability in mind. Reliability is key for GE in their turbine control systems. They work very hard to prevent single-point failures and to have as much redundancy as possible--though there are always people who argue the Speedtronic turbine control systems don't have enough redundancy, that they have too much redundancy, and that they too complicated. There's a little bit of truth to all of these things, but as I'm sure you've come to realize in your career: Engineering is a series of compromises. And it all begins with philosophy and design intent, and a lot of the compromises fall out from there.

While there are many programmable action/logic controllers used for combustion- and steam turbine control my personal preference is still a purpose-built turbine control system that doesn't require a lot of converters (since a lot of turbine control field devices and instruments are atypical of many other industrial processes and control systems). To me, that reduces complexity and single-point failures--both of which can be very problematic.

Of course, these purpose-built systems are not cheap, though they have come down in price in recent years (but--so has the quality of the components with the decision to manufacture many of them in Asia without a rigorous quality control system, relying on the manufacturer to meet quality targets and letting them decide if the targets have been met or not--something a lot of companies are doing more and more of these days in the race to CHEAP).

Anyway, thanks for your help--it would seem the problem is solved. And, "conductive grease" can be the same grease used on PC microprocessors when clamped to a heatsink. I forget the formula and the manufacturer's brand name, but it's not anything special as I recall. I've purchased heat sink grease from PC retailers and used it with good success--at least I've never heard the places I've used it complain about any problems.