We have a bank of 12 blowers in our industrial wastewater treatment plant and they have all been shutting down Tuesday mornings. Looking back it happened last year where they shut down at about 7:45 AM give or take about 5 minutes 8 times over about 6 months (with one time at 6AM) always on Tuesday morning. Then the problem went away and didn't happen for about 9 months. Now it is happening again only every other Tuesday morning at about 7AM since mid-June. As soon as the operators notice they are down they can re-start them from the HMI without having to clear any alarms or reset them in the field.

Control wise there is no common interlock to shut down all the blowers each shut down is per blower. I have added extra points to our historian to try to see if it is the PLC shutting it down but haven't found anything. All I have found is that we lose the run command and the run status at the same time so I am going to try to see if I can add traps to the logic to find which happens first because either will trigger the other.

Incoming power wise they added some relays to the incoming power (2 separate transformers each feeding half the blowers) and when the shutdown happened the relays were still energized meaning they didn't lose power.

I have sent an e-mail request to the facility to see if anybody does work in that area on Tuesday mornings or does anything routinely on Tuesday mornings and nobody could think of anything.

We are starting to look at signal power inside the PLC cabinet but the fact that it is so routinely on Tuesday mornings make me think it is something that is programmed instead of just an equipment glitch.
The blowers are not on a VFD it is a simple starter in the MCC for each.

Anybody experience something like this before and can guide me on where to look or what to do next? I would greatly appreciate any help.

 

Is there any chance of network storm?

Alternatively are IEDs connected to network or Peer-to-peer communications to other PLCs?

We have experienced these issues with similar results.

 

Sounds like you have the right approach with looking for command interruptions etc. That makes this a real puzzler!

Just some random experiences:

I once had something similar with a bank of air compressors at a glass plant. After a month of doing what you are doing, it turned out that there was a safety monitoring board on the compressors themselves, not part of the plant wide control system. That board looked at motor CURRENT, as opposed to contactor status, to determine if the compressor was working or not, based on the idea that if the unloader valve failed to close 10 seconds after the motor was started, the current would be too low. We had added Soft Starters, so depending on numerous conditions, some days the compressors ramped to speed slower than other days and on the slower days, the timer timed out before the unloader valve had a chance to close and load up the compressor, so it was shut down after 10 seconds with no "fault" indication anywhere.

Another one I suffered through turned out to be CAUSED by a big air compressor that was using a 480V coil on the motor starter. There was no surge suppression on that coil, so randomly, the kick-back spike on the 480V line caused by that starter coil would falsely trigger a control circuit going to a PLC (using unshielded wire), which the PLC interpreted as a shutdown command. But in that case we DID find it by looking at the command histogram, as you have done.

Yet another one was caused by line spikes that came off of an ancient centrifuge in a meat packing plant that was used to separate fats from the collected water used to hose the place down every night. In only ran once per week, but it was a complicated Y-Delta 4-speed starter setup built in Germany, with upward of a dozen contactors and interlocks. When they stared that thing up, it threw out 3 or 4 HUGE voltage spikes on the line that wrecked havoc on everything, including command/control signals, and everything was 120V controls. But because it only happened once per week, nobody saw it happen and most of the guys troubleshooting it had no idea that centrifuge was even there. I actually saw it and asked what it was for when I saw all of those contactors, they had to dig through a lot of maintenance department records to find the ONE GUY who was aware of it's purpose and how it was used.

I feel for you because I've been there, done that, got the T-shirt and the hat...

Good luck!

 

Two things come to mind. The first is that you are seeing a power dip (below the motor starter's dropout voltage; the relays you installed may have different dropouts or be on different power) due to something switching in your power grid or the utility's power grid (put a Drantez power monitor (or equal) on your pump power and see if it shows anything) (a look as the pump power commonalities on your electrical drawings wouldn't hurt either). The second is a cybersecurity problem where someone is accessing your system remotely via an internet connection, your enterprise network, a remote SCADA connection, or other remote access connection, inadvertently by an employee via a thumb drive/USB flash drive (for games, favorite program, etc.), or internally (or ex-internally) via disgruntled employee or former employee.

William (Bill) L. Mostia, Jr. PE
ISA Fellow, SIS-TECH Fellow,
FS Eng. (TUV Rheinland)
SIS-TECH Solutions, LP

"No trees were killed to send this message, but a large number of electrons were terribly inconvenienced." Neil deGrasse Tyson

Any information is provided on a Caveat Emptor basis.