Although not exactly in line with the request, this does fulfill the "lighter side" part: Story as related by a former co-worker......

This particular individual was employed by a certain large robotics company. A certain automobile manufacturer had a 50% stake in this robot company. He was involved with installing a rather long painting line at the automobile manufacturer that involved a few dozen robots, and several hundred sensors, actuators, etc.

After installation was over, the line supervisor called my friend back up and told him that one of the robots was having power troubles. Apparently,
they would occaisionally find the robot powered down, with no apparent explanation. My friend was dispatched back to the site.

After 2 days of watching power meters, scope traces, installing line filters, etc, he was no closer to a solution. It seemed that the problem
was extremely intermittent, and he was unable to be there at the precise moment of failure.

The third day, he pulled a chair up to the line, and refuse to move from his spot. His persistence was rewarded a couple hours later when he saw one of the adjacent robots make a high-speed move over to the controller of the problematic robot, and then make a quick jab at the "Power Off" button.

A review of the program in the "aggressor" revealed a subroutine that would only be invoked by a complex set of unrelated sensor states and
actuator positions, whose sole intent was to reach over and power down the neighboring controller.

I never did find out what happened to the programmer, but the customer, of course, was not very amused.

--Joe Jansen

 

I swear to God that this is a true story. Many years ago we had a very ambitious engineer try and automate a portion of one of our food processing lines. This particular section of the line involved the indexing of stacks of cheese slices. Indexing cheese slices is a very difficult operation as the stack must be well supported as they have no rigidity to them. This
engineers solution was to build a conveyor that consisted of 12 smaller o-ring conveyors that butted up end to end with each other. Each of the
smaller o-ring conveyors was powered by a mega buck hydraulic motor. Photocells at the inlet and outlet of each of the smaller segments would
scan the stacks of cheese going in and out of each of the "cells". Kind of like a shift register... when a station had no stack it requested the previous station to feed its stack along to it. It was a nightmare that was never mean to work and it was not long before it went to that great idea graveyard in the sky...

However... During the 6 months or so that we experimented with this contraption, we noticed that it would run not bad with Operator A present
but three times worse with Operator B. After many hours of head scratching we determined that the only difference between Operator A & B was their sexes. Op A was male while Op B was a rather largely endowed women. The ladies endowments were triggering the photocells and causing the shift register to go nuts. Changing the focal length of the photocells cured the
problem. It was however a much harder thing to keep a straight face when near the young women.

Best Regards...

Rick Kelly
Chief Electrical Technician
Natural Cuts, Ingleside, Ont.
Kraft Canada Inc.

V (613) 537-8069
F (613) 537-8044

 

Here is one that that illustrates the "don't get lost by concentrating on the details without looking at the overall process" principle and probably still applies to many Wastewater Treatment Plants even today.

One of the most common WW treatment processes is aeration of sludge. Basically, you inject air through a tank of wastewater and try to maintain
the dissolved oxygen level at a desired level. Typically, the air is provided by large blowers, with big electric appetites, and the air flow is
controller by modulating the number of blowers on at one time, blower suction control valves, blower discharge control valves or combinations of all three. The trick is to minimize the electric energy used by the blowers, but produce enough air to keep DO where you want it. The method I have seen most often used (probably because it is given in an EPA publication and few designers are willing to take responsibility for a design that differs) is to try and maintain a pressure in the blower discharge header that is slightly higher than what is needed to for the air flow required. Seems reasonalbe, but I have never seen one of these work properly. The reason is that the discharge pressure depends on a lot more than the depth of the injectors in the wastewater and the air flow is extremely sensitive to the pressure. What normally happens is that, in order to have enough air for reliable DO control, the pressure setpoint is so high that very little energy is saved. A setup that does work for cases where there is a control valve on the discharge line/s is to vary blower output based on the discharge control valve position. I have had good results with adjusting blower output to keep valve (usually butterfly) position in the 50% - 60% open range.

Al Pawlowski, PE
dba ALMONT Engineering
Baton Rouge, LA USA

 

The first automation company I worked for (a Swedish Company while living in Singapore) had the early story of a cookie factory that installed a simple PID controller on the automated oven - and the cookies always started coming out burned. So they had to send a technician out a few times & finally he caught old Hans (not his real name) who had been in charge of the manual setting before tweaking the local manual gas valve when he though the oven was getting too cold or hot. So the PID controller was fighting him & the cookies were losing. Of course old Hans was the grandpa no one wanted to upset & in his old age he was still important to the bakery.

Solution: They came in over a night and removed the ball in the manual valve - there was already another shut-off & and emergency shutoff up-stream in the gas line. So now Hans could play with this value all he wanted, the cookies started coming out fine. The company was happy the PID was working - and Old Hans was happy he could "help" the PID by tweaking his old manual valve.

 

I had a customer using an HMI where the operator could not quite reach it. They used a metal handle on the touch screen and whoops broke the screen.

Another time an engineer I knew pressed an unmarked red button wondering what it did. It shut down an entire galvanising line.

One unfortunate colleague when he was a rookie turned off a panel to open the door so that he could go on-line to the PLC. Whoops.

One customer had an electrician who wanted grounding straps on all fibre optic photo-eyes because the electrician believed that there was a 50V or so potential on any fibre optic cable to carry the signal.

The same customer had another electrician who wired up a new room with 208 VAC instead of 120 VAC and then wondered why the lights were so bright. He also was seen trying to change an output card (he was also observed to be trying to change the software) because some bulbs on a Christmas Tree light stand was not working. He neglected to try and change the burned out light
bulbs.

Another of their electricians was with the maintenance manager investigating a problem on a sterile process that required 8 hours of downtime if sterility was lost. The maintenance manager realised that the problem was with a specialty high-speed counter card. He told the electrician not to touch anything while he went to the spares bin. Halfway back to the panel with the replacement card he met the electrician who had decided to shut down the remote rack and remove the faulty card. An easy well-timed hot swap would have avoided any unpleasantries.

I have seen every type of "cheater" there is trying to by-pass and circumvent safety systems. I even saw a coin (lucky farthing I suppose)
taped to a touch screen over the alarm acknowledge button rather than fix the cause of the alarm. I have seen code that does not do anything (machine builder OEM ladder is always good for a laugh) and seen customers who swear
their previous system always did something when the code base had no such feature whatsoever.

RA

 

I was on a navy ship where a 2 of 3 voting scheme for boiler drum-level consisted of three sailors watching the same level guage. When more than one saw high level, then... !

Regards,
Phil Corso, PE
(Boca Raton, FL)

 

I have seen a fair number of these home made boards around. One OEM I worked with had a line of machines that used these boards to control braking. They only made a few machines a year of this type, and their production run of 50 eventually ran out. When they ran out, they found they had no drawings of the board and the engineer that had designed it was long gone.

They ended up having to borrow one of these boards from a customer whose machine was down for rebuilding, and get it to a local PCB maker to have it reverse engineered so they would have boards available for new machines and for spare parts.

 

I installed a SCADA system in a large copper mine, to control the distribution of liquid mine tailings. Late one night I received a telephone call from the irate Mine Manager who informed me that he'd disconnected the Remote Paging System because his wife couldn't sleep because it had been beeping for two hours.

The following morning they discovered that a $50,000 electric motor had fried itself because the well had gone dry about 2 hours after the
manager shut off the annunciator. This was the second time he'd done this.

Two years later, the mine manager doesn't work there anymore, and the Remote Pager is NEVER turned off !!

Mark Hill

 

I am not sure this is relevant but I like telling this story anyway. Back in 1984 I worked for a PLC manufacturer. We received an urgent call from the developer of amusement park rides that was trying to get their new zero gravity drop ride operational prior to Cedar Point opening for the Memorial Day weekend. Cedar Point had spent millions on advertising this ride but the ride wasn't working. I headed to the site with an engineer on the Friday before the big opening to work this miracle. When we got there we observed that everytime the car was dropped the PLC I/O would flicker triggering the braking/safety
systems that brought the car to an immediate halt. From top speed to zero in about 2 feet. So we opened the cabinets and hooked our equipment to monitor things and had them cycle the ride a few times. After one or two cycles we noticed the smell of ozone and could see a flash of light coming from the I/O panel. It turns out that they had run a 24V signal a wire from the solid state output on the PLC I/O card several hundred feet from the panel to the top of the ride's tower where there were several DC solenoids that were used to release the car. When they de-energized these relays the voltage spike caused an arc from the 600V rated terminals (that were tested to over 2000V) to the ground that put a voltage spike too large to be seen completely on our oscilliscope on the PLC ground causing the system to reset. Amazingly, none of the components were damaged as the arc must have acted as a kind of voltage limiter that kept the on-board protection from frying. A freewheeling diode fixed the problem long enough to work out a better system (it wasn't my design). The electric field around that wire probably could have stopped a
pacemaker.

Now here is the funny part: while all this was going on and the car was screeching to a halt repeatedly in such a manner that would have
surely caused anybody severe discomfort, if not physical injury, if they were in the car when it braked that hard there were at least 50 people standing in line waiting for the ride to open. When we finally got it running a few hours later (after I took a ride in it...what a blast) these people nearly knocked each other over trying to be the first person to ride this thing (actually the engineer and myself were the first to ride it as it turns out). Talk about blind trust. I knew what was happening but they didn't. The ride engineers did not have an interest in taking a ride. hmmm.

Regards,
Ralph Mackiewicz
SISCO, Inc.

 

Alright, here is one of many "Disasters" I have investigated:

A client had a TMR system on a reactor, but to save money opted to use only one set of three temp xmitters to serve both the shutdown and
control subsystems.

Over a weekend one of the 16-channel A/D cards failed to function properly. It was changed with a card from maintenance inventory. The three temp signals used in the TMR system were derived from the same card (yes, you could call it common-mode-failure). Thus the reactor was subject to an extreme over temp, because the settings on the replacement were incorrect for the function.

The error was eventually detected before the reactor was damage. But, not soon enough. Some $5,000,000 of catalyst was destroyed. And of
course you know who was blamed.

Remember, you opened pandora's box... and it may never close again!

Regards,
Phil Corso, PE
(Boca Raton, FL)

 

Walt reminded me of some statistics of the past. I remember a dozen or so years ago when the common percentage of control system loops in manual override in process plants was 50%. There was even an article in InTech about the issue. I asked users and vendors in my safety classes back then what they thought the number was. 50%. <Sheesh!> I know the number is lower now, but it sure isn't zero!!

 

When I was in China a few years ago working on some textile equipment, we found that the Chinese did not appreciate the safety that was placed on the machine to discourage anyone from getting to close to the spinning beam. They solved this problem by positioning a small girl BETWEEN the spinning beam and the safety bar, as it was much easier to repair broken ends there then to reach over the safety bar. I think I still have some photos of this somewhere. I wish I had video of it.

 

Responding to Walt's Fri, Aug 30, 11:02 am:

Walt, I strongly disagree. Sometimes it's the operator! Sometimes the tech! But, most of the time, it's the engineer!

Regards,
Phil Corso, PE
(Boca Raton, FL)

 

Ralph's tale reminds me of one that is directly relevant to the original post.- passed on by a colleague - ever noticed how all these dodgy things seem to happen to someone else?

The original Star Wars ride at D********* apparently caused a lot of concern as to how the facility could absolutely ensure that there were no customers standing on the access ramps when they were retracted prior to starting the simulator.

A lot of hi-tech solutions were tried, including photo detectors and eventually a video monitor. However, all had their problems. In the end,
the solution adopted was to make one of the "ground crew" making sure everyone was strapped in etc go out the entrance door, another out the exit door, and ensure that the ramps were clear before giving the word to go.

Bruce.

 

One of the worst things I ever saw was at an insulation factory down in ____. Safety and employee well-being were not important (as were good salaries).

The insulation moved by at a few metres per minute and it was sectioned off into the batts by a large guillotine. Sometimes the product would buckle on the line. Instead of stopping the line, one lucky employee would get to crawl on the conveyor and stretch out the insulation, just ahead of the guillotine. He had to remember to crawl against the line so that his legs would not get chopped off.

I was horrified. The locals and a few machine reps. who travelled around told me that this was quite normal. Refusal would mean dismissal and there was lots of cannon fodder.

 

Well, Phil, I think we are in "violent agreement." In the specific case in my story, however, the engineer had argued with the owner until he was purple, even enlisting my assistance, and they insisted on a completely
automatic control system and an automated plant. At the commissioning, we found out why.

The engineers who allowed the EPA regulations to buffalo them into building vastly oversized plants and inappropriate controls are surely responsible. But so are the owners and regulators who "enabled" them.

Walt Boyes

---------SPITZER AND BOYES, LLC-------------
"Consulting from the engineer
to the distribution channel"
www.spitzerandboyes.com
[email protected]
21118 SE 278th Place
Maple Valley, WA 98038
253-709-5046 cell 425-432-8262 home office
fax:801-749-7142
--------------------------------------------

 

How about this for a story. I was ordered by my supervisor to place the step down transformer for a positioner in the same 4x4 box that the 4-20mA signal wire goes thru. Apparently they didn't cover that in T.V. repair school.

Frustrated AAS Instrumentation tech.

 

Several thousand chickens were lost because a changeover contactor in the generator switch panel opened, cutting main power to the facility, without starting the standby generator. The failure was caused by the control fuse blowing because the diesel generator jacket heater happended to have also been connected to it. The heater had developed an intermittent faultafter some years (and not for the first time). To be
fair the OEM changeover panel had no provision for any other auxiliary AC supply, although that wasn=92t insurmountable.
No one had appreciated that the integrity of the control fuse was crucial to keeping the power on the plant. It might have been a good idea to use mechanically latched contactors instead. Normally this would have been a glitch but not a catastrophe. The plant was fully alarmed, with an autodialler to the manager's house, and usually he could respond immediately. Only problem this time was the plant had recently changed hands and no one knew the autodialler had a rechargeable battery which had also failed. So it was curtains for the chickens. The testing had all been done by simulating a mains failure at the control panel, so the faulty battery in the autodialler didn't show up because the autodialler was still mains powered. The old adage about failures always coming in threes, and latent failures coming to the fore at the worst possible time applied.

Gerry Coates
ISP Consulting Engineers Ltd
New Zealand