We are having a very odd problem with our Mark VI PLC, where all tests we did are pointing to the core R UCVE failure. May you please see our tests descriptions and give your opinion about the diagnostics?
First impressions: the gas turbine (F6B, Mark VI PLC) was stopped for a 10 day maintenance with 100% good diagnostics of the PLC. As usual, 3 days before start-up we went to do a full check-up, where we found out the core R VSVO card indicating some failures, with its redundant signals (tripled) showing frozen values: VSVO S and T cores was showing live values changing (very slightly), but the R core was completely frozen and showing negative values.
At first we thought it was a VSVO problem, so we got a spare card and went to substitute. Before doing that, GE suggested a reboot of rack R (power off and back on), to see if the failure would persist. When we did that, the R core did not came back. We tried a full reboot (rack R, S and T), and the result is:
SCENARIO "A" [INIT]
- When connected to S or T: Core S in 0xc3 / Core T in 0xc3 / Core R in UNKNOWN state.
- When connected to R: Core R in FAIL (red) / Core S in UNKNOWN state / Core T in UNKNOWN state.
Then, we tried a full offline download (for the 3 cores), getting the error: "-Error D165 - SYSERR sys loader (165): cannot online download an unconfigured UC2000". When download to S and T only, the download completes fine.
SCENARIO "A" [END]
The next step was doing a complete restart of core R UCVE:
- Switch off rack R; removed UCVE; removed CF card;
- Using a CF USB reader, we downloaded the compact flash using Control Toolbox (configuring IP);
- Put UCVE CF carf back; put UCVE back to rack R; disconnected VCMI card (as requested by the GE manual);
- Ping to core R UCVE: successful
- Downloaded the product code (runtime) to core R only, using Control Toolbox (select.dnl file), selecting only "download to permanent storage" and NOT selecting "download to memory". The options "download symbol table" and "download compressed controller file" did not even appeared.
- Download was successful.
- Power off rack R; re-inserted VCMI; power on rack R;
- SCENARIO "A" was maintained.
After that, we tried a full power cycle of the Mark VI (turn off all racks R, S, T, X, Y, Z; then turn on in the order X, Y, Z, R, S, T). SCENARIO "A" was maintained.
To try to verify/confirm if the problem was in fact the UCVE, we did the following:
- Switched off core R and S;
- Removed UCVE R and S and swapped the CF card between them;
- Inserted the R_UCVE in the S rack and the S_UCVE in the R rack; did a full power cycle;
- we noticed that the failure followed the R_UCVE card, that is:
* When connected to R or T: Core R in 0xc3 / Core T in 0xc3 / now Core S in UNKNOWN state.
* When connected to S: now Core S in FAIL (red) / Core R in UNKNOWN state / Core T in UNKNOWN state.
Some questions (please think out of the box):
1) Can we say with 100% certainty that the issue is related to the R_UCVE card?
2) What is the "D165 - SYSERR sys loader (165)" error and what is the UC2000?
3) We have a PC MIP board (RJ 45 connector) installed in all 3 UCVE controllers, and the one in the R_UCVE is not working (LED off). What is the function of this board? Is this only for redundancy with the LAN RJ45 UDH port? Can we disconnect this board from the R_UCVE and try a download without it? We disconnected the network cables from PC MIP of S_UCVE and T_UCVE and did a download to them, and it went just fine.
Unfortunately we don't have a UCVE spare (we tried to buy, but GE informed they don't supply this card anymore). We are now working with two possible solutions:
- Migrate UCVE to UCVG or UCVH (if GE have them for sale);
- We are looking for GE clients to buy UCVE cards for replacement;
What do you think, any other ideas?
We are very thankful for any help you may provide. Very best regards.
Sorry to hear about your problems. I, too, have fought valiantly with UCVEs--and lost.
I don't have access at this writing to a Mark VI System Guide, GEH-6421, to try to understand what a PC MIP is or does. And, I'm not clear how you could communicate with <S> or <T> if you unplugged the UDH Etnernet cables.... I guess, technically, it might be possible through the IONET but that doesn't fit with my understanding of what the IONET does. I've never been able to communicate with a control processor unless a UDH Ethernet cable was connected to the UCVx card. (In thinking about this, I have sometimes wondered how it's possible to communicate with the VPROs when the only communication link to the control processors is the IONET.... Hmmm.... )
Anyway, I would suggest you try eliminating the PC MIP to see if that has any effect--it certainly could, if the UDH Ethernet cable is plugged into it. (Does the Mark VI have redundant UDH Ethernet cables between the control processors and the Ethernet switch? Is that what the PC MIP is being used for? (Just thinking digitally....))
If that doesn't work (eliminating the PC MIP), I would agree--the UCVE is probably failed or failing. Have you tried any of the advertisers that support control.com for availability of UCVEs? I'm referring specifically to Gas Turbine Controls Corp (www.gasturbinecontrols.com)
You can use your preferred World Wide Web search engine to search for Speedtronic (Mark) spare cards; there are several vendors of such cards. One that's not listed is MD&A Turbines, in Latham, NY. I've come to learn they have a parts division that has a limited inventory of Mark VI cards/parts, and they might be worth contacting. Their website is www.mdaturbines.com, and the applicable webpage is:
The person to contact and his telephone number (in the USA) is listed near the bottom of the page.
Hope this helps! Please write back to let us know how you fare.
Hi, CSA. Thanks for the reply.
We did solve the problem, thankfully. But the root cause is a mistery. I'll explain:
By the end, there was no hardware in fail AT ALL!! No UCVE, flash memory, VCMI card. Not even the original VSVO card that started all this process.
After several trials and swaps between UCVEs, we realized that when the UCVE controller was in UNRECOGNIZED mode, their VCMI card became in IONET communication fail. However, we were sure since the beginning that there was no fail in the VCMI card.
GE support was terrible. Did not help at all, and I would say they made our problem even worse this time, since they kept distracting us asking to do several other things and saying to us that the problem was with the VCMI card.
In our case, we didn't had much option other than trying, since a new card would take days to arrive. What fixed the problem by the end, was a full re-flash of all of the UCVE controllers individually, even the ones the were presenting no errors. Then we downloaded the runtime and application code to each controller and all failures were simply gone!
Remember that before we couldn't even download to R_UCVE, and when we swapped it with the S_UCVE or T_UCVE, the failure followed the R_UCVE. Crazy, right?
We still do not understand how a simple rack turn-off/turn-on task would demand this procedure to re-flash all the controllers (that was what triggered all of this issue, check the first posted message). And knowing GE, we don't expect them to have any Technical Report or article explaining this. Probably we will never know what really happened.
It is a very disappointing that one PLC gives us this much trouble just to know where the issue is. The lack of good and clear diagnostics of the Mark VI is really a pain in the ...
Thanks so much for the update!
One thing I'm coming to understand about the Mark VI is: It seems the order and timing of power-up of the three control processors is more important than for other TMR Mark* turbine control systems.
My experience with TMR Mark IV and -V turbine control systems Was that usually I would power-up a single processor and wait for it to get to the "all good" I/O state, then power-up another processor and wait for it to get to the"all good" I/O state, and then the last processor. And that pretty much always worked. I followed the same steps with early Mark VI panels and early versions of Toolbox with good results.
That doesn't seem to be the case with more recent versions of Toolbox and firmware. I'm still trying to to find a consistent method of power-up of the panel processors that always works--but that's proving very elusive. I think much of the process of a successful power-up revolves around the establishment of IONET communications--BUT that's just a wild-arsed guess; I have no actionable data to support that.
Again, thanks for the update!
If you may allow me to share my experience!
VCMI stands for VME Communication Master Interface. There are four or six tiny memory chips resides on VCMI. (I cannot recall exactly) Those chips stores IDs for all the boards in the VME rack.
Each individual memory chip has few of boards IDs store. For example, chip A store IDs for board in position 1-4, chip B store IDs for board in position 5-8 and so on....
Two years ago, one of GE customer had an HMI upgraded from Toolbox Classic to ControlST. During an upgrade, the First TA had to cycle power-on/off Mark VI rack, flashed and re-flashed some of VCMI and UCVE a few times. Not sure, how the problem happened but he was struggling to return Mark V rack back to normal after an HMI upgrade.
As far as I know from customer, he struggled for a few days. At the end, everything are back to normal. No any cards have been replaced. This sounds similar to MarkUser's case as far as I can recalled.
Two months later, the same Mark VI rack was powered down for one week for a maintenance activity. After power-up again, they found VSVO failed and cannot download to UCVE.
Customer was struggling troubleshooting problem for almost a week before they decided to get help from GE. On this note, this customer has two Ex-Control TA employed by them.
I was working on the new GE unit next door and been instructed to leave everything and attain the Mark VI problem. When I arrived at site, not the whole story were communicated to me. However, I found one UCVE, and four or five I/O boards failed.
The root cause (???) of this problem was one chip that store ID for I/O boards on VCMI has failed. The VCMI correlates position ID on VME rack to the each individual I/O board for UCVE.
For example position 7 is VSVO position 8 is VRTD, position 9 is VTCC and so on. I believe someone, made this problem worse by attempted flashed the I/O board. Because of VCMI correlates wrongly correlated board type to the board position on VME rack. The I/O board was flashed with wrong application.
Also due to the chip failure on VCMI, the board from location 7 was miss-read and determined by UCVE as a fatal error. That is the reason why it appears as UCVE has failed. However, when UCVE was replaced, problem remains.
I was able to nailed down this problem by connecting the Hyper-teminal session to serial port on VCMI. From VCMI boot-up messages, one can see how its correlates I/O board position and board type. Accompanied by some logged message from TSM session to UCVE, the problem is nailed down to a failed VCMI. One this note, there is no fault LEDs on VCMI.
Visually VCMI is looked healthy from LED and Diagnostic alarm point of view.
After I replaced two VCMIs (One because of failed chip, the other I guess was due to cycled-power on/off too many times), re-flashed I/O board to correct board type, replaced one VRTD (it was flashed with wrong application). Proceed with first time panel power-up procedure, everything is OK. It is also good to point out that, there is no VSVO failure.
A close look to both failed VCMIs, there are black dust accumulated on the board.
Looked around the PEECC, the only source of black dust is a HVAC's filtered. Customer confirmed that filter life is is expired and it suppose to be replaced several months ago, anytime operator come inside the PEECC, the worn a paper mask to avoid that dust.
My wild guess is, that filter released black dust and dust being sucking by VME rack cooling fan. When the Mark VI rack is power-on, everything seems OK because rack always warm. No moisture can accumulate.
However dust absorbed moisture when Mark VI rack is power off for several days and at some point HVAC turned off for other maintenance activity. This is a root cause of VCMI card failure.
Reading your post, I think 'GE May be right about problem on VCMI'. I just share this because of I came across some thing similar two years ago.
Sorry to gear about your issue and glad to know you fixed it. Just because you asked about the companies that sell GE SPEEDTORNIC parts i would like to point out your attention to WORLD OF CONTROLS. They have an extensive inventory of all GE SPEEDTRONIC parts. I have personally purchased from them and been very satisfied. You could approach them.