GE Speedtronic MkIV & MkV

Hello all,

Can I please have some assistance in the methodology for undertaking a complete back up of all controller parameters and storage of such back up's?

Our Asset Management Plan and Risk Register has highlighted as HIGH RISK the fact that back up's of controllers have not been undertaken for several years, and the likes of reading EPROMS is a lost skill, especially with 'older' technologies.

As well as EPROM back up's, how else are MkIV & MkV backed up in event of 'catastrophic failure' for a system rebuild?

Many thanks in advance.

KWS
 
KWS,

I've been thinking about this.... And it's not a simple task--for either control system.

In my mind, this is two tasks, for each control system:

1) Backing-up PROMs/EEPROMs
2) Backing-up configuration software and operating parameters

PROMs/EEPROMs
There are companies who offer EEPROM back-up services, though some have had more success than others. At least one seems to have amassed a stock of Mark V EEPROMs which they can use to produce more without having to send someone to site with a PROM burner to burn new PROMs. So, that's one option for Mark V. I am reluctant to name any providers or provide any recommendations/cautions for any providers. You're on your own to find them and vet them--and I suggest vetting any you seriously consider by asking for references and checking those references (ALWAYS the hardest part for most people--cold-calling others to ask for their experiences, but it can save you a LOT of headaches and troubles which is harder to get past than cold-calling and asking for information/experiences).

The Mark V EEPROM failures I have personally witnessed were likely caused by lightning strikes (a really serious, close strike which took out other equipment at the site) or improper handling of the EEPROMs when changing I/O cards--by far, the biggest cause of EEPROM failures. There are still EEPROM burners available in the market, though they are getting harder to find (and more expensive. The prices did fall a few years back as people realized those chips were being phased out of industry in general, but they seem to have risen a little since then. And, they're not difficult to use--the biggest problem is handling the existing EEPROMs properly when using them to burn spares. (I think some of the newer EEPROM burners can even digitally store the information on USB drives or hard-drives for future use--but I haven't personally seen any of those, just anecdotal information.)

I'm not aware of anything similar for Mark IV, but I've also never seen a Mark IV PROM failure. RAM back-up battery failures (for those Mark IV panels with battery-backed RAM, sometimes called BRAM), but not a PROM failure.

Configuration Software/Operating Parameters
Backing-up Mark IV information is definitely not for the faint of heart. It requires patience and a bit of fabrication skills (making serial connectors/adapters) and some disappearing software (LapLink used to be the preferred comm software, but sometimes MS-Windows programs will do in a pinch--thought that takes some patience as well). Finally, most laptops these days no longer have serial ports (and configuring the ones that do also requires patience and little bit of trial-and-error), so one is forced to acquire and configure USB-to-serial adapters. And while there's no shortage of them available, MANY of them are very cheap Asian-made pieces of junk, and I have even personally experienced the required driver software which had viruses infect the computer it was installed on!!! (Requiring even MORE patience and time and money and software to remove--and I'm not sure it was ever fully removed...) If you're going to go down this rabbit hole, spend the money to buy QUALITY name-brand USB-to-serial adapters/driver software!!!! So, this is not one I'm going to attempt to detail on this forum, it's simply too complicated and fraught with problems.

One can print (on paper) or capture the printer output (another patience-requiring exercise, also requiring serial ports (or USB-to-serial adapters/software and cables) all of the unit-specific software and parameters. Other than printing them (or capturing them digitally from the Mark IV printer port) this is by far the easiest, if "old school"--but then the Mark IV is old school (good, old school!).

The operator interfaces (<I> or GE Mark V HMI) used with Mark V can be tricky, and if an <I> is still being used it can be difficult. For an <I>, I suggest finding a hard drive and removing the hard drive from the <I> and using it to make a mirror image ("ghosting") to the new hard drive. That way, you have everything just as it exists on the current hard drive. Tough to find hard drives that small these days (small enough and old enough for the CPUs used with <I>s to recognize). And, really--that's not the weak link with an <I>--it's the ARCnet card, and with the age of the CPU it's the platform/motherboard and hard drive.

For GE Mark V HMI the best thing to do is to get a quality archival program such as Acronis sells. We don't know what vintage OS is being used on the operator interfaces at the site, so you're on your own there to find software that is compatible with the OS. Then, you need to stop the CIMPLICITY project (of course, you will do this when the unit is NOT running...) AND stop the proprietary GE background service, TCI (the latter you can do from the OS's Control Panel applet or from the prompt of a command line window with the command 'NET STOP TCI' (without the quotes)). You need to stop this program and service because BOTH can open files when they are running which some archive software can't deal with--and worse, don't tell you it can't back them up leaving you with a false sense of security. So, just stop them both (the CIMPLICITY project first, then TCI) for the duration of the archive/back-up and then when finished just re-start the HMI from the MS-Windows START icon (they will both automatically re-start).

Do a complete back-up, including the OS--in other words, make a bootable copy of the entire hard drive--a "ghost" image as it sometimes referred to. That way, you have EVERYTHING you need to rather quickly recover. And, I also recommend acquiring a suitable hard drive, and transferring that image to the hard drive which will speed things up in the event of a serious problem.


That's about all I can do from here. I would estimate if you want to do a complete digital back-up of the Mark IV, it's going to take about a month--including the time to acquire the components to make and assemble the cables, reading the Mark IV Maintenance Manual, GEK-83883 (if I recall the GE publication designation correctly), section on communicating with the Mark IV, acquiring/configuring the software on a laptop, getting everything to work properly to get a prompt to appear from the Mark IV, and actually capturing the information. But, it's still going to take a LOT of effort for the uninitiated and it's not for the faint of heart. An older IT person might have the fortitude to try it, but that person would have to have worked on command-line OS's and be prepared to read (what is a short section) in the Mark IV Maintenance Manual several times as well as do some trial-and-error comm attempts. The Mark V is a little simpler, but the EEPROM stuff is complicated.

There are still a couple of people around working for various service providers who know Mark IV and you might be able to get one of them to come to site to help. But, again--you need to ask for and check references because some companies are just sending their "brightest" and most enthusiastic people to do a job like this, and you may end up paying for several weeks of effort or even be left with a non-working system and a big bill and searching for someone else to help recover. "Pay me now, or pay me later!" (The least expensive option is NOT ALWAYS the least expensive in the long run.)

Best of luck--please write back to let us know how you fare!
 
Ken Symmons,

After re-reading your post I have a little more information to offer.

Mark IV
While there is a way to download programming and configuration (from a laptop) to a Mark IV, that's not the normal way it's done. There are several versions of Mark IV, and depending on the version the way the programming and configuration information is stored in the Mark IV the way it is backed-up would be different, if it's even possible.

Since the Mark IV is a 'stand-alone' controller and doesn't use a PC for programming and configuration it's not really subject to the type of "loss of mind" other control systems are. The factory programmed and configured the Mark IV using tools they had at the time. It was possible to use a 'dumb terminal' to make modifications to the program and configuration (not so much the early versions, but certainly the later versions were more field-configurable, to an extent).

When PCs (at first, lunchbox PCs or portable PCs (which were so heavy they could throw your shoulder out!)) with serial ports became more common in the field and they were capable of being used as 'dumb terminals' they were. But, they still required the same cables as dumb terminals (null modem cables, with an odd 25-pin connector on one end, and usually a 9-pin connector on the other end). Since the printers used for Mark IV were also serially connected to the Mark IV, it was also possible to use some adapters to use the same cables to capture the Mark IV printer output to the hard drive (or floppy disk) of a PC.

Early versions of the Mark IV used batter-backed RAM (BRAM), and those chips are hard to duplicate, if at all (I never heard of anyone who was successful doing so--but I imagine there were some resourceful people who were). Later versions of the Mark IV used EPROMs, and dual EPROMs which could be duplicated and used in a catastrophic failure situation. But there was risk in doing so, bending chip pins/legs, static electricity, etc.

But, having paper copies of the programming and configuration is really about the safest thing around for Mark IV (in my personal opinion). Because finding someone with the cables and know-how to recover from serious problem using chips is still going to require comparing what is on those chips with what should be in the Mark IV--and that's going to require another skillset which is also disappearing. It's not gone; it's just disappearing.

Mark V
Mark V is very different. It's a blank page, and the configuration necessary to control a particular turbine, driven device and its auxiliaries is downloaded to the various control and communication processors (and protective processors) from a PC-based operator interface. So, really, as long as the operator interface is viable and has very recent programming and configuration information, it's pretty likely the Mark V can be restored.

HOWEVER, there are EPROMs installed on many of the I/O cards in the Mark V which can also lose their mind (it's difficult to accomplish, but it has been known to happen). And one of the ways it happens is that when replacing I/O cards in the Mark V it's often necessary to remove EPROMs from the card being replaced and installing them in sockets on the new I/O card being installed. And, this is where the problems usually occur--mishandling, and static electricity. So, many sites have opted to have someone duplicate these socketed EPROMs to have copies available just in case.

Without an operator interface, it's next to impossible to operate a Mark V. So, it's really critical to be prepared for problems with the Mark V operator interface. And that preparation is also very crucial to being able to restore serious problems with a Mark V panel (which would, in reality, be kind of difficult to envision under normal operating circumstances).

The mechanics of all of this for these two very different control systems is pretty involved and detailed. And, GE really didn't properly document any of this....

Hope this helps!
 
CSA

Many thanks for your reply. I was hoping you'd see the post!! I've found your comments on the forum over the years thorough and extremely helpful. My best regards from Western Australia.

KWS
 
Ken,

CSA is very on point with all of his discussion here. I would like to add that there are companies who still employee engineers from the days when MkIV and MkV were being developed in Salem, VA. who worked either in Salem full time or in the field installing or commissioning these control systems. They are out there, however, not all ex-field engineers will have the necessary tools and/or knowledge to perform what is discussed here. Nothing against those engineers, it is just the way GE taught field engineers and what Salem was willing to teach those field engineers who were not a part of manufacturing.
 
Top