GE Mark VI TMR controller failure

S

Thread Starter

Sha

In GE Mark VI TMR System, we noticed the controller status R & S is boot and T in Failure. Can someone tell me how to rectify this problem? We tried restarting R, S, T controllers but no improvement.
 
You have not provided nearly enough information for anyone to make an educated response to your post.

<b>When did the problem start?</b> After an electrical storm? After a loss of DC power? After a loss of AC power? After an AC power surge/spike? After an attempt at making application code (sequencing) or Control Constant changes?

<b>What have you done to try to troubleshoot the problem?</b>

<b>What Diagnostic Alarms are being annunciated?</b>

Are there any faults indicated on the processor rack power supply(s)?

What is the power supply for the Mark VI: 125 VDC or an AC source (110 VAC or 220 VAC)? Or, is there a primary DC source and one or more AC back-up sources?

What is the positive and negative voltage of the 125 VDC supply with respect to ground? (Presuming the panel uses 125 VDC voltage for discrete input wetting voltage and DC solenoid supply voltage. It would also be helpful to know what the application is: steam turbine, gas turbine, aero-derivative gas turbine, heavy duty gas turbine, generator drive or mechanical (compressor) drive, boiler feed pump turbine, BOP (Balance of Plant)/DCS control, ?)
 
GE Mark VI TMR is used in GT and it is running with 120VDC power supply, The system was running normally except one indication ie) MVAR. Even though after disconnecting its field cable, still TMR gave -35MVAR reading. Hence we restarted R, S, T and end up with problems. Today we recycle the power of R,S,T and the state of the controller now is
R - Always boot --> Missing state data packet
S- Failure and T-Failure --> Expander load failure

Physically there is no failure in R,S,T, I/O Modules, power supplies in panel, all the LED’s are normal, because we compared with other system which have the same configuration of other GT.

I would like to know the following:

1) What is the order sequence of loading application code,product code.

2) Whether .m6b file (Project Application database) needs to be downloaded to controllers?

3) Is their any I/O loading is required.

Please reply.
 
> What did you did to resolve the problem? <

I was working to solve this problem too, we downloaded pcode or runtime to all three processors then reboot. (soft reboot is not enough, must be hard reboot - power off and on).

Then, we downloaded application code to all three processors, again we did hard reboot.

Finally, we downloaded firmware to VCMI cards.

Then all processors back to green control state.

Note: make sure your Pcode is correct. If wrong Pcode file is used, it won't work.

Thanks for all.
 
Ali,

I never experience problem similar to yours. But I wonder what cause the problem to occur in the first place. Is there any manuals on downloading the product and application code? If no appreciate if you could share some knowledge on it. I hope that I will never face the same problem but I think better be safe than sorry later ;)


 
Our site just experienced the same problem. We had to reload Pcode/runtime and all that good stuff. GE believes it has to due with the backup battery UCVE card. Unfortunately you can not replace the battery in house, for it's soldered/ and sometimes glued. GE is issuing a TIL for this issue.
 
Hello Guys,

I just had exactly the same problem, R & S are in control, T is offline (failed), gas turbine, turbo-generator.

help please, where to start investigating.....

any will be highly appreciated

sam
 
Hello again

SD, I am not aware of such manuals.

Sam, That is how it started in our case also. R was fail and S and T were in control. So avoid touching the good ones. Try to hard reboot T only and see if it come back. If not, download pcode then choose NO when asked to reboot (soft) and make On/Off reboot. Then, validate,build and download application code then reboot again. Download firmware to VCMI card (T only) then reboot. Good luck
 
I want to caution people about the practice of building and validating a good project (applicaton code) before downloading when working with a TMR Mark VI Speedtronic turbine control panel.

Toolbox dynamically maps memory addresses, and each time a project is built Toolbox can reassign signal names (one or more) to new memory locations. Now, if this newly (re-)built project (application code) is downloaded to a single processor it can result in Diagnostic Alarms, and worse depending on other conditions.

If you have a project (.m6b file application code) which is showing 'Equal' for the two processors which are still on-line and working properly, (re-)building and validating and downloading to a single processor can cause new problems.

So, <b>EITHER</b> don't (re-)build an Equal project which has had no changes made to it <b>OR</b> download a (re-)built and validated project's application code to <b>all three processors</b> to avoid new, unintended consequences. If you have a project that is 'Equal' to on-line processor(s) there should be no need to build it again. Building is necessary when application code changes like sequencing or some Control Constant modifications or some I/O assignments or reassignments are made.

If you build (re-build) a project for any reason, then you should download the application code to <b>all three processors</b> in a TMR control panel. And then reboot each processor one at a time.
 
Hello Ali & everybody,

Thank you for your quick reply. my question can I download pcode, and VMCI firmware and hard reboot the <T> core while the unit is running?

any help will be highly appreciated.

Sam
 
9. Open up Toolbox 10 and the correct .m6b file and set privilege to Level 2

10. Download Runtime by highlighting the header MK VI IO and selecting Device pulldown menu> Download> Runtime...
* Perform Runtime first
* Perform Application Code second

11. Build the code and save the file, then select the Download icon. Be sure to un-select the “Download to Memory” box for this first download.

12. Reboot all the cores.
* When rebooting start with Rack 1, then followed by Rack 0 (powering up and powering down)

13. Right-click on the MK VI IO header and select download. Then check only “Download Configuration” tab and proceed.

14. Reboot all controllers (don’t forget the VPROs) and see if the program initializes (heartbeat)

15. If unsuccessful, repeat step 11 but be sure to download to memory as well as permanent storage.

***********Select Firmware option during download---if download doesn’t work after that de-select firmware option***************************************

***Always pay close attention to the information window at the bottom of the Toolbox screen. It is helpful to make this window larger and any problems may be more apparent.
 
F

Fran McMenamin

> my question can I download pcode, and VMCI firmware and hard reboot the <T> core while the unit is running? <

Probably not recommended.
 
Dear Sam,

Before you try download anything on your T controller, please check first power supply to this controller and try what's diagnostics appear on T controller. You can refer to manual for each error code and there is action to do.

You can try any activity to T controller like download or hardboot, but please check other diagnostics for Controller., VCMI, and IO's and it shall be good for R and S. If you have fault in any cards, check the diagnostics and try to clear/reset the error using toolbox. I have experience in this TMR system, if T is error, and one of R/S IO have error in same channel/tag, it can be shut down/trip your turbine. This happen due to voting detect 2 of 3 signal goes bad (after T restart/download). Please be careful and check your IO diagnostics especially.

Thanks
 
Thank you CSA for your comment
In my case the project was showing minor difference, so i choose to rebuild.

Sam
Your T processor is out anyhow so i think it is ok to reboot T while your unit is running.

Thank you Fran, which manual is that?

BR,
 
The partial list of actions (beginning at Step 9) is incomplete, and we don't know the context or the origin of the document.

For the best operation of a TMR control system all three processors need to be using the same code with signal names assigned to the same memory locations to avoid Diagnostic Alarms and other unintended consequences.

Toolbox dynamically assigns signal point names to memory locations.

Building a project file is like compiling a computer program, and in that process signal names are assigned to memory locations. In older Speedtronic turbine control systems, there was a very formal and rigid memory "mapping" scheme, with certain memory areas reserved for particular functions and types of signals. And the assignments within these memory areas were on a "first come, first served" basis based on locations in configuration files. So, as long as the configuration files weren't edited the signal name-to-memory location assignments would be identical each time the "job" was complied (to use Mark V terminology).

In the Mark VI, there are very few memory areas which are reserved for particular functions and signal types. And, when the Build function is executed there is no "rule" on which signals get assigned first, or third, or ninety-second, or last. The Validate function just checks to make sure that signal assignments were completed and that any reserved signal types were assigned to the proper memory areas/locations. Validate doesn't check to see if the signal name assignments in the current build are the same as in any previous build.

So, it's possible to have the exact same project file (sequencing, Control Constants, macros, blocks, pins, EGD assignments, etc.) with slightly different, or even radically different, signal name-to-memory location assignments. And, both would Validate correctly! And both would download and execute properly.

It's just that when the Mark VI processors are checking for voting and execution problems if the signal names are not assigned to the same memory locations <b>in all three processors</b> then problems can occur, from simple Voting Mismatch Diagnostic Alarms to more problems with outputs.

So, any time a project file is Built and Validated, it's a wise idea to download that build to all three processors, and re-boot if necessary.

Just think about it. If you have a project file which is already Built and Validated and has been downloaded to the Mark VI processors (let's presume the same project file was downloaded to all three processors), and all the processors were indicating Equal while on-line after the initial download, and then something happens to one of the three processors and it's felt that the project file needs to be downloaded again, does it make sense to Build and Validate a "perfectly good" project file which was previously running in all three processors with no problems when that Build could reassign any number of signal names to new memory locations?

I've seen a <b>lot</b> of "unexplainable problems" and Diagnostic Alarms resolved by simply downloading the same project file (application code) to all three processors and then re-booting them one at a time. Nothing more. No Build & Validate. Just a download of the current project file to all three processors, and an individual re-boot of the three processors.

So, consider how the application code running in all three processors needs to have memory assignments in the same locations for trouble-free and error-free operation. Consider what happens during a Build. Consider what could happen during a re-Build of the same project file.

And, when looking at any instructions written or provided by GE or anyone, consider the context of that instruction. Was it written for an initial download of software? Was it written for an upgrade of Toolbox software and pcode (product code)?

When considering whether or not to Build and Validate a project file, make that decision based on whether or not any changes have been made to the application code and if the project file has been previously downloaded to all the processors and was deemed Equal and worked properly. Because a re-Build of an unchanged project file <b>may not be</b> (isn't this fun!) identical to the previous Build when it comes to signal assignments and the most trouble-free operation occurs when the same Build of a project file is running in all three processors. Validating a Build does not check the current Build against the previous Build for memory assignments.
 
I came across this thread yesterday while searching for solutions to a similar problem on my site.

Mine is also a Mark VI TMR system on a MS5001PA turbine. The sequence of events is as follows:

1. Momentary loss of AC power in control room, coupled with UPS failure causes HMI to go off while unit is running.

2. Upon restoration of power, HMI is booted up and cimplicity project is started but display is blank.

3. Troubleshooting reveals there is no communication with controller (ping test of controller IP address)

4. Ethernet switch (which is AC powered) is observed to have an led illuminating a fault.

5. In the process of trying to reset the ethernet switch by powering it off, the DC supply to the controllers is erroneously powered off (GT shuts down :( in the process)

6. Switch is eventually reset by toggling power supply on-and-off.

7. Ping test now reveals communication is established with controller but when toolbox is opened and taken online, R, S and T controllers take turns to appear in "fail" status, depending on which is reset.

8. I continue to manually reset the controllers, one after the other but problem remains.

9. I resist the urge to try out some of the suggestions in this thread in order to gather more information and power-off the entire system instead.

10. This morning (approximately 24hrs after powering it off), I power-on the controllers and everything is normal, just like it never happened.

Naturally I should be happy about this but I am not. I'll feel a lot better if i knew what exactly caused the problem and how powering it off for several hours resolved the issue.

Hoping someone out there can help me out in this regard :)
 
Just verify the sequence of powering up the R, S & T cores plus the Protection core.

If you restarted the R, S & T cores prior to starting up the protection core; then you will face the problem you mentioned. Doing the same in inverse order will lead to a normal reboot.
 
J
It appears that you have received sufficient support on this issue but I wanted to make you aware of another support option.

If you are having troubleshooting or performance issues on your GE Mark series control, HMIs, Excitation or DCS, you can receive free support at GE's Controls Connect internet portal.

If you are a turbine operator Controls Connect gives you access to our knowledge database which has an advanced search engine for finding knowledge articles, manuals, application guides, TILs, white papers, etc. We also provide phone/email support from our experienced technical support team. This is all a free service.

In addition, we are launching a free LIFECYCLE WEBINAR series that is available to Controls Connect users. I have attached a link to the webinar invitation if you are interested. The series will cover multiple topics addressing Mark V, VI and VIe and include multiple options for support beyond 2015. We will also have several sessions on Mark I, II and IV. The title of the Webinar Series is “Discover the Cure for Obsolescence”. Hopefully, you will find it valuable. http://geglobal.informz.net/geglobal/archives/archive_2468882.html

CONTROLS CONNECT NOTE: If you have a GE SSO#/Login and are a GE turbine operator, business associate or EPC firm, please send me an email at [email protected] with the subject "Controls Connect Access" and include your SSO#, Name and Company. Note that we do not accept generic email addresses (e.g. @yahoo, @hotmail, @gmail, etc). Company based email addresses are required for security reasons. Your registration will take approximately 3 days. If you do not have a GE SSO# you can register at http://site.ge-energy.com/online_tools/controls_connect.htm or just perform a Google search on "Controls Connect". Just click on the register button to start the process. Registration forms should require less than 5 minutes to complete. If you have any questions feel free to contact me at [email protected]
I am the Product Line Leader at GE Measurement and Controls.
 
Top