GE 90-30's faulting during online edits

J

Thread Starter

John McCoy

We have recently had a few incidents where GE 90-30's fault when an online edit is saved. At least two sources at GE have said that it is the nature of the beast... as online edits are made, the PLC memory becomes fragmented. Eventually, memory is exhausted and the processor crashes. Once, we have received a warning message which prevented us from saving the change, but usually we crash without warning. Even the serial and ethernet ports shut down and we must cycle power to the rack to get them back.

We asked GE if we had overlooked this in the documentation. Amazingly, they say it's not in the documentation!

GE's only solution is to occasionally stop the processor and do a "stop mode store," which will load the logic without fragmentation. This is probably acceptable in many applications, but not ours.

We use these PLC's for utilities and HVAC at a rapidly growing hospital campus. Due to the growth, new features or logic changes are required frequently, while keeping the processes online. We can switch most of the systems to manual to do the stop-mode store, but our client strongly objects to doing this very often.

We are already using 364 CPUs, which are the most capable available in terms of memory and horsepower, so we can't just throw more hardware at the problem.

The main solution we are currently considering is to use GE's hot standby redundancy scheme to allow us to stop-mode store one processor while the other is in control. Obviously, this is an expensive and complicated fix, but less so than porting all the logic to a new platform.

I'd be interested in hearing others' experience with GE or other equipment.

Thanks,
John McCoy
 
E

Eric J. Feight

Nature of the beast is probably the most accurate description. I have been using VersaPro for 90-30 programming lately. To be honest I haven’t looked to see specifically what the issues are with an on-the-fly download. I do know that with the old LogicMonster software you had the option to halt the processor in the "Outputs Hold Last State" mode. This solved all the "Hot" download issues I ran into. It has been my experience that on-line edits are good for quick fixes during start-up. However, any major changes are best done offline and tested if possible then downloaded when the system can be safely taken offline.

As for HBR, it’s a good application that has it’s place. You do need to put some forethought into the hardware before you jump into it. I don’t know the nature of your application so I cannot speak to your specific situation. The best approach I have found is to put together two five slot racks with the following configuration: Power Supply, CPU (Use one with an Ethernet Port!), Genius Bus Controller. Use Field Control for all your I/O even local stuff. Assign addresses 30 and 31 to your bus controllers. Configure your GBIs for Hot-Backup. The GBIs will then take care of the redundancy, here’s how: each GBI in the system will take it’s output status from the controller at address 31. If it fails to get a response for three consecutive bus scans it will then take output status info from the controller at bus address 30 until 31 comes back. HBR software is cool in mission critical applications where 3 scans is too long, for example high speed positioning or precise speed regulation.

My approach requires a little programming but doesn’t add to the software cost. The program in each controller should be identical with the exception of data synchronization. I use a separate subroutine, that way when I make a program change I can simply duplicate the whole program and then copy the old subroutine and I’m done. I use Ethernet Datagrams for data synchronization. Don’t let them tell you that you need to install another set of GBCs to do this. Unless you have a ton of data it’s a waste. If you’re connecting to a SCADA system you may want to hard wire an output to an input. The "Primary" program always turns the output on and the "Back-Up" program turns it off. Monitor the input from your SCADA software to determine who’s in control.

I have implemented this successfully several times. I don’t know if this is worthwhile just to prevent crashes from affecting your system while doing edits or not. You will have a more robust system overall. That might benefit your customer. You’ll still have the cost of the extra rack, PS, CPU, and GBC, but at least you won’t have to add the cost of the HBR software (It’s not called HBR anymore. I cannot remember the new name, but it still costs as much!)

Hope this helps………
 
S

Steve Bailey

My understanding of the way the 90-30 handles run-mode stores is this: The modified block(s) are downloaded to available memory in the CPU. When complete, the pointer for the start of block memory location is changed. If the modified block is larger than the amount of free memory available, the PLC will crash. This is the first time I've heard of memory fragmentation being a problem, but I guess it could happen.

As an alternative to a redundant system, you could consider a more distributed approach, with multiple CPUs, each controlling a smaller portion of the system and sharing pertinent data over a network connection. With an architecture like this, you could periodically shut down an individual CPU for a stop-mode store to do a defrag without taking the entire system down.
 
K
John,
I have experienced identical problems. Below is a copy of the answer I got from GE technical support regarding this:
"We have a known issue with internal heap memory in the 364. In certain systems, it may run out, resulting in the symptom you see. The new release
which fixes the issue will be coming out very soon."
Kurt
 
J
Kurt-

Thanks to you and all the others for your comments. After making a big stink with our local distributor, we also got word last week that a fix for the 364 is on the way. Supposedly a firmware revision is coming out the week of 6/11.

John
 
Top