R Controller Failed to Come Online After GT Shutdown

Recently our MARK VIe is shutdown for carrying some welding jobs by mechanical in GT shutdown. after completion of jobs, MARK Vie was powered on. after power on, R controller didn't come online. where as the S, T are in online & designated controller LED is off in all the three controllers. no data coming in cimplicty as DC is not determined.

finally with BGGTS help, we have downloaded the firmware, application code from EWS. finally R controller came in line & R became DC.

why it happened? as a usual practice, we are switching off the MARK VIe for doing the welding jobs in GT & this is the first time controller got failed after power off.

please explain, why it could happen?

I experienced a very similar occurrence just a couple of months ago. A Mark VIe with UCSB processors that had been shut down for several months was re-started and <S> and <T> came back up just fine, but <R> had lost all of it's "memory" and had to have the firmware and application code re-downloaded to it in order to re-establish UDH communications.

For some odd reason, <S> did not take over the duties as Designated Controller, so there was no information available on the HMI displays. I think this can be explained because <R> had NOT been a Designated Controller after power-up, and so <S> could not take over Designated Controller Duties.

But, I was at a loss at to explain why the non-volatile memory of <R> was lost. I suspect it wasn't totally lost, but something had caused the boot-up/initialization process to complete properly. I have always powered-up Mark* processors of Mark V and newer beginning with <T>, then <S>, then <R>. But, I have a feeling that there's something about newer Mark VIe firmware that doesn't like this boot-up process. I have not had a chance to test this theory yet.

I also suspect that many welders are NOT following good practices when welding around a turbine control system--AND, that proper earthing (grounding) practices were not followed during initial construction of the turbine, control system and field devices. Many welders just do not seem to understand that when welding around a plant with a digital control system it is necessary to attach the welding ground (earth) lead as close to the are where the welding is taking place as possible. They just believe they can attach the ground to ANY grounded area anywhere in the plant and the welder will work just fine. While that's essentially true, the longer the current has to travel from the welding stinger/rod to the welder ground (earth) the more chance there is for trouble.

The whole "functional" (instrument) earthing scheme combined with a separate "protective" (safety) earthing scheme makes a LOT of sense. BUT, unless construction personnel are trained in how to properly terminate field devices and instruments during construction, and plant personnel are trained in connecting devices to "earth" as plant instrumentation is added and maintained, problems can be inadvertently caused. Worse, intermittent problems can be cause--which can be the hardest problems to find and resolve.

Combine less-than-ideal welding practices and poor earthing practices and there is a recipe for problems.

I want to ask--where is the welding taking place around the turbine and auxiliaries, and why is it necessary for so much welding to be done after the unit has been commissioned and placed in service? It just seems odd that so much welding is necessary after a unit has been commissioned, and that it is necessary to power down the Mark VIe.

And, powering down the Mark VIe is also something which should be done in an orderly fashion. Just using the 125 VDC mains circuit breaker to remove and apply power is NOT a good method. The processors should be powered-down in an orderly fashion one at a time, and then the 125 VDC mains power can be removed. When powering-up the Mark VIe, the mains should be applied, then each processor should be powered-up one at a time, waiting a couple of minutes before applying power to the next processor. As mentioned, I have always powered-up beginning with <T>, then <S>, then <R>. I suspect newer versions of Mark VIe firmware might do better by powering-up <R> first, letting it become the Designated Controller (I think it will, or it might when <S> gets powered-up). Then, power-up <S>, and then finally <T>. (And I usually apply power to the PPROs prior to powering-up the control processors.)

That's about all I can say about what happened. I have yet to have a chance to talk with former GE colleagues to see if they have any ideas about what might have happened, but having this experience (yours and mine) makes me thing something has changed in the firmware world--and maybe GE hasn't heard of the issues yet. OR, more likely, there has been no formal documentation of a change in power-up procedures, because in the past there has never been a formal power-up procedure.

If you can provide more details about how the panel was powered-down, the types of UCSx processors in the panel, and how the panel was powered-up it would be great for when I talk with someone from GE. And, if you can tell us how the welder(s) are placing the welding grounds (close or far) when welding around the unit, that would be good information for all who read this thread.

Hope this helps--and we look forward to hearing details and information about your experience and plant practices.

Hello,

Just to add, me too I have experienced the same last april, and during a plant outage to permit work on insulation issue in one 400kv line on the substation GIS.

Black start weren't available!! We used just the EGD 400V to keep some working equipments in the commun service...etc, and during these maneuvers we lost the 230 UPS that supplies MKVIE. After restoring the 230UPS, We have got the problem, HMI ST showing nothing!! mechanical BOP, and ESD too.

By formating flash memory and downloading firmware/code app/boot..etc on ST MKVIE R/T OK, but not on "S", because no UDH communication, couldn't do the download. Setting up a new controller has solved the problem, and MKVIE finally dertemined the DC.

Same think with the Mech BOP controller "R", changing by new one...On ESD, just changing flash memory on 'S' and downloading has solved the problem.

Conclustion: lose of the PS caused the problem, maybe by shutting down manually controllers one by one we can avoid such problem.

Chemsouhd,

Thank you for the information. I, too, believe that an orderly shutdown and power-up of the processors is important, and without that it seems that some version of UCSB is susceptible to loss of memory.