Has anybody seen a BRC 100 display a flashing green LED during normal operation (i.e. not during or after a configuration change)? My primary (LED 7&8) module has been showing up like this for a few weeks. An inspect from Composer tools shows normal execute primary, execute FTX backup. Byte 2 on a status report shows code 34. A format/initialize fixes the problem for a day or two, sometimes more, sometimes less.

brianoustad at boisepaper. com

 

Brian,

I've seen the blinking green light on BRC100s numerous times on primary/redundant pairs and stand alone BRC100s as well. When the green LED blinks while the BRC is in execute mode, it means that NVRAM has failed. The controller will continue to operate because, during execution, a copy of NVRAM is sent to SRAM where the program actually runs.

I've seen them run for months in this condition without problems, for instance when we had no choice on stand alone BRCs running steam turbines that could not be taken down for reinitialization of NVRAM.

However, if the module is put in configure mode it will red light because there is nothing in NVRAM. In that case it has to be reinitialized. The headache for us is for stand alone BRC's because the NVRAM failure prevents even tunable spec changes. You have to reinitialize to restore that functionality.

I'm surprised that your NVRAM failure is reoccurring so soon after initialization as they usually take more time to do that. There are technical bulletins put out by ABB that attribute these NVRAM failures to two common causes: High frequency power supply noise on the DC buses of PSII power supplies and voltage drops (even small drops) on the buses due to loose or corroded connections. We have found loose cable crimps as well.

ABB says to check the DC buses for noise using a minimum 200 megahertz scope. I don't have the bulletin in front of me, but I believe the noise limit is 200mV peak to peak at 5 volts, and the tolerance increases at the higher voltages.

Every power supply bus I've checked shows around 2-3 volts pk-pk at about 50 megahertz, and many of our BRCs experience periodic NVRAM failures--some 4 per month, which is not supposed to happen.

ABB says the cause of this noise lies within the power factor correction circuitry of the PSIIs. There are 3 fixes: Purchase and install an external filter capacitor kit that evidently drains the noise from the buses to ground, send the supplies back to ABB for on board upgrade, or upgrade to the PSIII supplies.

We plan to upgrade to the PSIIIs because the PSIIs are beyond their rated life expectancy of 5 years.

In your case, I'd check for noise and voltage drops, correct any problems, and if all else fails, change out the NVRAM chip. It's the big Watchdog chip with the battery on its back. It's supposed to last for 10 years, but ABB recommends changing them out sooner, like around 7-8 years. We have had at least one Watchdog go bad.

We've also had the firmware chips go bad, although that usually shows up as a ROM checksum error and the controller red lights.

By the way, what firmware are your BRCs running? Most of ours are at E0, and there have been many improvements (i.e. problems fixed) since then. F7 is the current rev.

 

Thanks for the reply.

Here is what ABB sent me. BTW we are rev F5.

How to recover form an NVRAM error: (Primary Status, flashing green with red LED 7 and 8 on. Redundant Status solid green ‘with no’ red LED on.)

1. Redundant Status solid green with no red LED on means the redundant module has no configuration in it. You would not want to fail the primary module because the redundant module will not take over the process and the system would come down.

2. You need to make a decision when you can bring the system down and do a ‘hard initialization’ on both the Primary and Redundant modules. On the ‘Special Operations Switch’, put poles 1 and 7 to ‘open = 1 = off ‘position. Plug in, when it red lights do a 1 and 6, plug it in and wait for it to red light. Now put the switch settings on the Special Operations Switch back to its original settings and plug the modules back into their original slot. They should come up in configure mode, ready for the configuration to be down loaded.

How to recover form an NVRAM error: (Primary Status, flashing green with red LED 7 and 8 on. Redundant Status solid green ‘with’ red LED 8 on.)

1. Fail the primary module. The Redundant should take over as the Primary.

2. Pull the failed module out and do a ‘hard initialization on it. Special Operations Switch, put poles 1 and 7 to ‘open = 1 = off ‘position. Plug in, when it red lights do a 1 and 6, plug it in and wait for it to red light. Now put the switch settings on the Special Operations Switch back to its original settings and plug the module back into its original slot. It should come up as a hot back up to the new Primary module.

Important:
If you have a spare pair of adequate controllers, you may want to initialize, down load and put them into Execute, first, to put in place of the controller pair that is in there. This will minimize down time.

As far as the FCs that write to the NVRAM, the only one you can see is the REMSET (FC68), you can check S4 to see if the output has changed since the failure occurred.

If your Checkpointing to the backup is within a cycle or two (FC 80, S2, S4 and N+4) you should have very little (if any) bump when failing over.

Do a Problem Report on the Back up module to make sure everything is looking good before failing over. (first two byte should be 75 and 00 )

If you have a Batch, C or any other type of program in these controllers, we do not know what will happen when the controller is failed to the back up.

We are confident the fail over will be without indecent, but we cannot guarantee anything. Please take precautions before failing to the back up in case there is a large bump or the back up red lights.

When changing NVRAM chips that are purchased from ABB there is a 56 year shelf life on them at 20 degrees C. This is because the chips have not been powered up yet.

When they are put in a module and power it up, they will last at least 10 years weather they are in a powered up state or not.

When you buy a module from ABB, look at the Assembly Date stamped on the module. That date is the starting point for that NVRAM chip. That is the date the module was powered up and tested.

Remember, there is a portion of memory in the NVRAM chip that is set aside for the bios or operating system that will start up the module. When new NVRAM chips are put in for the first time, the module must be formatted to copy the bios from the firmware to this memory location in the module.

If the module is not formatted it may red light, not fail over or display erratic behavior.

Best regards,

William M. Kline
Technical Support Specialist
Automation Services
ABB Inc.