Does anyone have any good stories/anecdotes about bypassed control and protection systems or ways in which control has been achieved in innovative or unusual ways?

I am doing a short presentation as part of our graduate training scheme (me being the graduate) and I was planning to use some stories on the lighter side of control engineering.

I work in the oil & gas industry but I think some of the best stories come from farmers - electricians frequently comment on how often they find blown motors etc. and when they check the fuse they find no.8 (about 2.5mm thick) wire put in fuses to stop them blowing! there must be all sorts of stories about fancy control system set-ups that didn't work which were replaced by something very basic that did the job better. anything that seems to relate post it here.

Thanks

Daryn

 

Good Evening, I'll give you the quick version of my own experience. I am a Project Engineer (Electrical Engineer) for a Coal Company. I currently take care of our PLC 5's and other AB PLC's at the Mine Site. Our Preperation Plant (where we wash the Coal) uses massive amounts of water for the cleaning process and we are always trying diffrent methods of streamlining and trying to improve effency. Well I have been going through the program for an elabrorate water injection system for one of our "circuits" that would inject fresh water into a closed circuit for the purpose of maintaining a specific gravity. Well the former gentleman in my position had managed to spend in the neighborhood of 30k dollars on valves, pumps and a multitude of other sensors just for this one purpose, but to no avail. The system never did work, and as one of the projects I managed to inherit I came to the conclusion that I could make the system work, minus all the valves, pumps and extraneous sensors. All I needed was a single PID instruction in the program to automate the use of a "bleeder valve" so that fresh water contamination was avoided by managing the fluid that was in the circuit. This allows us to contain the fluid in the circuit, and instead of constantly pumping more chemicals in it to maintain the gravity we are able to just continously recycle the fluid. This one small improvment managed to save thousands of dollars per week and countless dollars over the course of the year. Not to mentioin all those spare valves and other toys I managed to eleminate. A happy ending for all. Hope this is what you were looking for.

glen

 

I don't know if this is what you're looking for but it's one of my favorites.

Years ago a friend of mind did a control system for a small milking facility at a local Kibbutz. System completed, handshaking over, my friend drove back to the office. Arriving at the office he found a message that the cow gate (the gate at the head of the milk queue) opened unexpectedly allowing the cows to make an unscheduled exit from the miking station.

My friend drove back to the kibbutz, checked the logic, found nothing wrong, and drove back to the office.

The next day he recieved another call. The cow gate had once again opened unexpectedly. Once again he drove back to the kibbutz, checked the logic, and found nothing wrong.

While discussing the problem with one of the operators they noticed that the last cow in line was hiting the control panel with her tail.

In order to increase reliability in the system the designers had used peizo push buttons instead of regular mechanical buttons. The cows tail was enough to activate the peizo buttons.

2 minutes and timer later, the problem was corrected.

=The end=

(8{)} ( .)

 

I was once in an Ethylene cracker in Turkey where the alarm annuciators (old pre DCS style) had paper clips forced into a number of the switches.
They said that whenever there was a thunder storm a number of trips would fire off wrongly and they could not afford the cracker to shut down every time there was a storm!

 

About a year ago I got a call from a friend who's neighbor owned a company that was having problems measuring RPM on a turbine. I dropped by and found that they had hired an out of work engineer to put in some expensive control and data acquisition hardware but the workmanship quality was poor, and the turbine RPM readings weren't working. It took only seconds to deduce the problem. When I looked in the panel, I found a home-made circuit board that had a mouse pad glued to the back of it to keep it from shorting out on the enclosure. They had been billed several thousand dollars for that board. I told them where to get an off the shelf din rail mountable module for less than $200.00. They did so, it worked, and they returned the non functioning home-made board to the guy who made it and refused to pay the bill.

 

Back in the good old days of computerized process control, when big central computers and black & white CRTs were state-of-the-art, I received a call from a customer about 2:00AM saying that his CRT stopped working. After spending a few minutes on the phone with him I went to the office and got a spare unit. When I got to the plant I decided to check things out before I lugged the new CRT up the 2 flights of stairs. I was really glad I did when I discovered that the janitor had decided to pull the power plug to the CRT so he could wax the control room floor. From then on that was the first question out of my mouth when a customer called with a problem.

 

Heres another one:
A shop supervisor once asked me to remove the safety doors and disable the safety interlocks on several high speed CNC lathes and install light curtians instead so the operators could gain precious seconds when changing parts. I enquired "What about metal chips and ejected parts?" With a serious look on his face he said that was what the light curtian was for. I told him it was not a force field, to turn off Star Trek TV, and come back to reality. I refused to make the modification, but he pestered me about it for several months before giving up.

 

One of the guys here told me about a problem he had to attend to on a boiler control system many years ago.

The boiler steam pressure and set-point were recorded using a rotary chart recorder. After a change in set-point the process variable would gradually rise up to the setpoint. One of the boiler operators had noticed that on the weekend previous, the pressure had been following the set-point very rapidly but with a bit more offset than usual and was concerned that the control system had not been working properly.

They followed the problem through for a while until eventually they found out what had happened when they got the truth out of the weekend operator. Turns out (in order to improve the control) he had unlatched the process variable pen and tied it directly to the setpoint arm, leaving no record (or indication) of the pressure.

You can have the best control systems in the world but i guess they still need protection from those who THINK they can improve it. I remember from university, that some of the automation and control students used to like breaking into the traffic light control boxes and 'improving' the control so that the green light lasted much longer for the direction they frequently travelled. i was a witness only of course.

 

A year or so ago I was doing field service work for a major injection molding machine manufacturer. While this didn't happen to me it did happen to one of my co-workers. He had gone to a customer's site to do some hydraulic troubleshooting. While out in the shop he watched a young fella kneel on the tie bars and reach into the die area (2000 ton press). Suddenly the dies started to close up and the guy fell...backwards, out of the die area. No one could move they were so surprised. Upon inspection our man found that the door safties had been wired around as well as a few other customer 'modifications'. Luckily the young fella got to go home that night and tell about it.

Bob

 

A number of years ago I was working as an electrician in a small factory. Being the senior man I was always on call. About 2am in the morning one of the junior electricians woke me with a problem that he couldn't get a tach feedback voltage at the proper point, thus the machine speed could not be controlled. He had spent about 3 hours troubleshooting the machine. Two calls later I went on in to the plant. Having him show me what he had checked I proceeded to pick my best screw driver and sharply tap the proper relay back into it's socket. With 10 minutes travel time to and from the shop and 10 minutes fixing the problem, I was properly cussed out by the 'kid' and earned my call in pay and went back home to bed. The really good mechanics get paid for knowing where to hit the engine, don't they?
Bob

 

Check Archives for ENGR: Bites or ENGR: Believe It, True Event

Regards,
Phil Corso, PE
(Boca Raton, FL)

 

Working with a wastewater treatment facility, a lift pump was burnt out three consecutive months. The electrician claimed it could only be caused by the PLC causing the pump to come on while the gates were closed. The logic indicated that the PLC would not call for the pump under these circumstances. Questioning staff, one of the operators indicated that it must be the PLC, because they would have to reload the program after every failure. After asking them to notify us immediately upon failure, we remained on-call and awaited the next catastrophy. The call came on a Sunday morning. We raced to the site and found that the PLC was wiped out - no data, no program. It closer inspection revealed the control relays had jumpers on them. A quick conversation with the weekend operator revealed that when the PLC would invoke an "interlock" failsafe, he would open the PLC and press what he thought was the "RESET" - which in reality was the cold-boot. Then to get the pump started, he'd simply jumper the relay. Since the operator the next morning would know how to reset the condition, he'd remove the jumper before going home. When confronted at the next meeting, the operator exclaimed that "no one told me not to press the reset button" - to which his boss replied, "I didn't tell you not to stick peanuts up your nose either, did you do that?" No problems with the system reoccurred.

 

Hi
I used to work for a metal processing company and on one occasion I remember getting a call to come and look at a large horizontal bandsaw which was having problems with its chip conveyor tripping out it's thermal overload. When I got to the machine the operator was standing in front of the open control cabinet trying to reset the thermal overload. He was using a small stick, as he did not want to get an electric shock. The problem was he was standing in a puddle of cutting fluid from which the same stick had previously been floating in!

In the same company was an extremely old and tired machine that unrolled and cut to length sheet metal from large rolls. When I first started in the job the operators had to manually guide the unrolling sheet at the same time as operate the feed controls (due to excess wear in the feeder). Unfortunately the controls were mounted away from the point that they needed to stand. To solve this problem the operators simply fixed funnel shaped attachments over the top of the two push buttons and used a broom handle to operate the two push buttons as needed. When I asked how well this worked they said that it only took a few weeks practice to be able to push these buttons without looking at them! Needless to say not long after this I fixed the mechanical wear that they were trying to overcome and fitted an extra set of buttons in a remote pendant for them...

 

oh ya.... listen to this,

We have a machine that uses x-rays to focus in on targets on the inner layers of a pc board before drilling reference holes. Now the loader for this machine broke down one day and while waiting for parts we asked the operators to manually load the product by opening the front door and inserting the panel. I came in to work one morning and saw that the operator had taped up the door switches so he wouldn't have to open and close this lead filled door. After explaining to him the dangers of xray radiation he was still not convinced that there was a problem. I then threatened to have
him fired if he did it again.

 

Something a bit different - just because you specialise in controls, do you get treated as though you don't understand basic mechanics?

Fifteen years ago I commissioned a new HVAC control system for a large theatre complex. As I was pre-commissioning one air handling unit, I noticed that the main supply fan was making a horrible noise - like a bearing running dry? I reported it to the resident maintenance engineers.

I went back a couple of weeks later, and found the side panel of the fan section removed. Shrapnel, looking like it had been heat-treated, was embedded in the sides of the cubicle. The bearing had seized, and the fan had shattered!

 

------------ Forwarded Message ------------
From: "Walt Boyes" <[email protected]>
To: <[email protected]>
Subject: RE: ENGR: bypassed controls - any good stories?

In fact, it isn't just farmers!

In the late 1980s, the EPA did a study of automatically controlled
disinfection systems at POTWs (publicly-owned wastewater treatment works)
and found that almost 85% of them (!!!) were left in manual control. The
reasons for this were interesting, but mostly boiled down to two: the plant
had been designed for 25 year expansion, so it was operating at its lowest
level of capacity, and the flowmeter's signal was in the mud and didn't
provide enough control span, and the contact chamber was providing hours of
retention instead of 30 minutes, and the residual analyzer loop couldn't
even begin to operate.

And during a visit to a large chemical/pharmaceutical company in the 1980s
(who at the time owned their own control system company) the plant
management confessed to me that the only reason they still were using the
company's control system was that the corporate parents insisted on it...but
they kept it in manual because it was easier and cheaper than trying to make
it work.

But farmers do take the entire dessert tray, not just the cake. I installed
in the late 1970s a complex chlorination dechlorination system for a very
small sewer district in extreme Northern California. It was so highly
overdesigned that I was truly worried, but the operator (part-time,
moonlighting from another city) was good, so I was hopeful.

At the installation and commissioning meeting, the President of the Board of
the District (a dairy farmer by trade) came up to me and took me aside.
"Tell me," he said, "now that we have this wonderful all-automatic system,
that means we can fire the operator and it will work by itself. Right?"

It took me a minute to see that he was dead serious.

In a year, there were no automatic control devices in that plant. The
operator had removed the entire system and replaced it with all manual
control. Worked better, gave less grief.

Sometimes it isn't the process, it's the control system.

Sometimes it isn't the operator.

Walt Boyes

---------SPITZER AND BOYES, LLC-------------
"Consulting from the engineer
to the distribution channel"
www.spitzerandboyes.com
[email protected]
21118 SE 278th Place
Maple Valley, WA 98038
253-709-5046 cell 425-432-8262 home office
fax:801-749-7142
--------------------------------------------

 

Daryn,

Check out Trevor Kletz's story about the snake pit - how to connect any one of abut 6 outlet pipes to any one of about another 6 storage tanks, with no chance of cross contamination and mixing products.

Hard answer - multiple valve manifold with check and double check to make sure one and only one inlet and ditto outlet, can be opened at the same
time.

Easy answer - terminate all lines in appropriate fittings and connect via hoses.

Another incident - traffic lights in an aluminium smelter were modified to allow the lights to be controlled from a general-purpose PLC to a dedicated system. All that was involved was the transfer of an existing operational program to a new PLC(with relevant alterations to I/O addressing).

New system was installed and comissioned succesfully on all pot-lines. Tech went home for the night.

Next morning - tech returns - chaos. One of the pot-lines had gone into spasm with all lights flashing on and off. Installation team kicks into high gear, checks out the program, finding all 4 lines are still running the same software. Photosensors detecting the movemnent of overhead cranes are all checked - still no joy. etc.
3 hours later, after makig yet another check on the most remote pohotocell, a tech notices that someone has leant a broom against the Lamp Test
button placed out in then open so anyone can easily check that the lights are working. Result -lamps have gone into test routine which results in them all flashing.

Bruce.

 

Please remember that there are also plenty of examples of absolute DISASTERS that resulted from control and protection systems being placed in bypass, such as Chernobyl. This is NOT an issue to take 'lightly'!

 

Some years back, I was asked to replace a stop pushbutton which was defective. The switch was supposed to have been bypassed. So I went out to
the field and pressed the pushbutton a couple of times before opening the enclosure. Almost immediately I recieved a message from the control room telling me to cease and desist as some interlocks were getting operated. The problem was that the same switch was wired to the DCS as well as a relay panel. The chap who bypassed the switch had only looked at the terminals in the DCS marshalling cabinet. Nobody seemed to even know that the switch was wired anywhere else at all.

 

I am not so sure I would refer to this situation as "working better". After all, the fuses are there for a number of good reasons, and removing them so you can overload the motor and burn it up is not a good practice.

I do have an interesting anecdote along these lines though.

A project I worked on a number of years ago had limit switches on the valves. I got a call one day from a service guy who told me that instead of adjusting the limit switches properly when changing limit stops on some valves, the operators had used wooden shims. Seems like the electricians were the only ones allowed to adjust limit switches, and they were generally slow to get around to such things, so the operators found a workaround.