GE MarkVIe (Migration) - IONet Timeout \ Watchdog Protection Trip

Good Evening,

I am curious if anyone has seen anything similar to the attached trip log? We have a simplex migration system (Mark V boards) that was recently installed and had the unit trip with the diagnostic log below. We lost all indication on the unit for ~ 90s. We also checked the heartbeat on the controller and it seems to have restarted. I am not sure if that count increases resets after a reboot, but when we checked the counts it matched with restarting in this same data period. Since this event we have not seen the issue repeat itself, did the controller reboot itself? If so, how did it capture diagnostics for the lost IONet connections?


As an update, we found from advanced diagnostics that indicated "FPGA CRC - reset detected." Anyone dealt with that before? We are working with support in tandem.
Thanks for the update! You probably already know, FPGA stands for Field-Programmable Gate Array, and I think they are used in several I/O Packs. Do you know which one(s)?
This was from the advanced diagnostics on “R”, I'm not sure if that FPGA CRC error would propagate up from the IO packs or not?

We would have UCSC, PIOA, PMVD, PMVE(R), PMVE(C) & PMVP.
I successfully avoided the Mark V-to-Mark VIe migration. Don’t know about that at all. I’ve never even had a glance at the manuals for it; it’s such a kludge. And a lot of people bought it thinking it was something entirely different, and were displeased when they saw what they got. (Same wiring; still using ribbon cabl; same card carriers; etc.)

Anyway, good to hear you’re in touch with the supplier. Am interested in the resolution (since I do work on UCSx processors).
Thanks for the reminder, I know how important it is to follow up on questions like these for future reference.

Preliminary thoughts from the support group point to an FPGA phenomenon called a single event upset (SEU) which may cause a controller or IO pack to reboot. Unfortunately there isn’t much else to diagnose, but will continue to monitor.

We will look into implementing a redundant controller (R,S) configuration to mitigate this issue.

(We should have already as most other site DCS controls have redundancy.)

Thanks for the information!

SIMPLEX does not have any redundancy. Is the turbine a gasser or a steamer?

SEU, huh? I’m going to add that TLA (Three-Letter Acronym) to my repertoire. Might prove handy one of these days. (As is said, “If you can’t dazzle them with brilliance, baffle them with bullshite!” Tech support is really good at it, too!)
This is a small 20MW steam unit that had simplex Mark V. I’ll look to transition to redundant controllers (R,S & X,Y,Z). As long as the IOPacks we have can support the communication to R and S.

That was our thought as well on the SEU…but I guess we just don’t have additional diagnostics to think otherwise.