SCADA Control Room Management

T

Thread Starter

Tony D.

I have recently been hired by a large utility to "organize" the Com room, provide for documentation, and generally organize the place for a future manager.

I'm no Scada expert. I'm an IT guy.

There is a wide disagreement in the organization about who is responsible for this function. Actually this does to seem to be causing any trouble now--but it will.

I would appreciate hearing your experiences on where Scada control fits and who is responsible for it. Thanks.
 
As per SCADA control room building is concern, construction of it is done by civil dept... But location of the control room will be decided by instrumentation or electrical people.
 
J

Jasmin Ouellet

I worked for an A-B distributor for years, and the same problem came quite often:

IT guy pretends it's theirs, because they handle computer, network, security and update.

Instruments/electronics pretends it's theirs because of the nature of the function performed.

what I noted most often, the real choice will depend on the knowledge of the instruments/electronics guy. if they are familiar enough with computers and network, they shall keep it. (you find that most departments have 2-3 "computer whiz kids".)

Keep in mind that most of the time, security is not an issue as most of industrial network are limitated to the PLCs, HMI and programming station. All of which, that should NOT be connected to the company's network, therefore greatly reducing the chances of being hacked!

Update: Often, it is NOT something you want to do as soon as a new update is availlable. (IT guys do!) Take for example, Allen-Bradley had a lot of problems with XP SP2... SP1 works fine? Keep it! I ran into a lot of places where they updated to SP2 as soon as it was out (on IT's recommendations) and ended up re-formatting the computer, so it worked "as before".

NETWORK: Most industrial networks are quite simple. A few basic switches, a few "192.168.1.xxx" addresses, and that's about it! If they need to connect to the company network, let's say to bring data to the SCADA, use a computer with 2 NIC cards and get the IT guy to handle that! You could also use an industrial bridge/gateway, such as A-B's Controllogix architecture with 2 (or more) Ethernet cards...

Regards,

Jasmin Ouellet
 
D

Daniel Chartier

Hello Tony;

From my experience this should be done by the team that does the programming/commissioning of the plant control system. In my organisation this would be the Instrumentation and Controls team. They are the ones that know how to make the system "dance", once mechanical and electrical have ensured the equipment is functional, and once IT have provided adequate communication services to interface the PLC and the SCADA.

No offence to the IT department of which you are part, but this is our turf, our specialty, even up to the ergonomics of the control room, and the power of the computers installed.

Anyone who interferes with this philosophy in our organization will be held reponsible for the adequate functioning of the plant.

For example, we had a run-in recently with our project manager who wanted to save a few bucks and reduced the size and number of work tables we had requested for the control room at a plant we were setting up this summer. A small thing, right? Who needs all that working area for 2 or 3 3 SCADA screens, wasted space anyway.

Then the CCTV control screens arrived, one had to be setup on a "temporary shelf" (2 cardboard boxes on top of each other) with their screen controller/recorder, the other one was kicked in by a worker as it was left on the floor, the vendors removed the one that was left and refused to re-install it until the broken one was replaced (do you know how much a non-interlaced colour screen costs, when the vendor is mad at you? not counting 8 weeks delay in delivery...) and adequate work space provided for their equipment; then the computers for Building services were brought in... The issue has just recently been resolved, and a furious project manager had to disburse from a closed project budget (Ouch!) to have our original furniture recommendations followed and finally installed with 3 months delay; only now will the client accept the plant as finished (not the only snag, but a major one). Just a small thing really.

If you have to do this yourself (generally that happens if you have nothing to do the day the manager sees you and needs a body to get rid of a problem) then consider getting help and collaboration from the people who do know what is needed for this. And a couple of rounds of beer will go a long way to making friends at that level. Still my experience talking... ;)

Hope this helps,
Daniel Chartier
 
N

Nathan Boeger

I wasn't going to touch this one, but I can't resist after hearing a few responses. Your innocent question ends up being a really sticky issue in a lot of organizations.

Typically for really small operations Jasmine is dead on. Integrators/PLC programmers can easily handle setting up a dedicated switch on a single non-routable class C subnet - it's a no brainer. No IT department required! Most don't understand much about the setup or what's going on especially if you add 'net access, but what difference does that make? Problems/poor implementations typically arise due to lack of IT experience. Let me give a few examples:
-Management wants to view data from the "business network", but the data lies on the "control network". Integrators often don't handle this very elegantly.
-Remote access: Integrators will often leave an insecure modem connection running PCAnywhere instead of considering a VPN.
-Lack of Hardware knowledge: Integrators will often leave a consumer switch in a production environment. While this works fine, it misses the benefit of what IT can provide. You'll sometimes see numerous separate switches where there could be VLANs, "random problems" with communications due to not dealing well with broadcast packets, RJ45 ends not done to spec, etc.
-Backup/maintenance issues: Typically IT has the infrastructure to provide consistent backups and updates whereas it's a tough issue for the plant guys.

IT on the other hand, from an integrators perspective, can be a pain to deal with. They're not typically familiar with what it takes to communicate with field devices or support HMI/SCADA applications. They often forget that production is king and that their mission in a control room isn't to lock out designers of the system. When left unchecked, IT will often over complicate requirements and policy, perhaps to feel like their doing something.

The real key is to work together to plan what makes sense given everybody's strengths. Clearly outline what IT should implement/support to make everybody happy. For example, IT should provide physical network infrastructure. All parties involved can participate in determining how it makes sense to subnet the network. IT could then provide reliable connections where traffic and security can be monitored. They can securely provide remote access, etc. IT can also do backups and updates per integrators recommendations (ie be scared to update service packs with some industrial software). IT will really be helpful as everyone moves to SQL databases and remote access. Security may not seem like a concern, but it's coming...In general control guys need to respect that IT does have a useful specialty that they don't, and IT needs to get the hint that it's unacceptable to mess with control guys access or any controls hardware/software that they don't understand - "oops, I left the network down for the afternoon" holds a different level of significance when it comes to production. I strongly support cross training between the controls guys and IT. Petty bickering or a "one or the other do all" approach is ridiculous.

I've had a lot of success doing projects with IT when I included them from the beginning. I've also been involved in heated battles with "them" when both sides tried to sneak around the other for convenience.

just my $.02...

----
Nathan Boeger
http://www.inductiveautomation.com
"Design Simplicity Cures Engineered Complexity"
 
M

Michael Batchelor

/* Rant Mode On

Quite honestly, I don't think *EITHER* of those answers is good.

/* Rant Off, engage brain I hope */

This struggle has been going on for a couple of decades now. I'll outline what I see quickly here, and try to follow up with a more complete essay next week. Yea, this will take a whole essay to get right.

A couple of hundred years ago mechanical guys were putting in steam and water power to replace muscles, and they were somehow "separated" from
the ordinary people who sewed, or chopped, or whatever the labor of the plant did to produce the product. As time went by, plants developed a
"maintenance staff" that had mechanics and an "engineering staff" that had guys who planned new things. (OK, I realize that in most companies this was all one guy. Think in the abstract here.)

Then, a hundred years ago these newfangled electric motors started showing up on the scene, and electrical specialists from outside came in
the plant and motorized things that had been run by the overhead power shaft. Again, as time went by the "electrical gurus" segregated into
electricians in a maintenance department and electrical engineers in a planning group.

Now, here's where we break down. When the computers first started showing up, they were giant things that were owned by the finance
department to keep the books. Unlike the steam engines and electric motors, these things came into the business and got comfortable without
ever being involved in the actual "production" of the plant. So the "computer guys" never became integrated into the planners and
maintainers groups. That isn't to say that the IT departments didn't have planners and maintainers, but unlike the production plant, those planners and maintainers worked in the same group and under the same boss usually.

In a regular manufacturing plant, the mechanical and electrical engineers work for an engineering boss, and the mechanics and electricians work for the maintenance supervisor. However the
informatics engineers work for the CIO or CFO, and informatics technicians work for the same CIO or CFO. Except that now it's not just the general ledger they're dealing with. It's the production equipment. The General Ledger requirements and the Production Floor requirements are *DIFFERENT* and sometimes at odds with one another.

The fact that they have competing needs isn't anyone's fault, nor is it particularly bad. The two environments are just different, and that's
just a fact.

What I predicted 20 years ago - 1987 actually - that has obviously failed to materialize was that the information systems discipline would
split along the same lines as the mechanical and electrical disciplines. A group of informatics planners would fall under the engineering
manager, and a group of informatics technicians would fall under the maintenance umbrella. So a new project might have p mechanical engineers, q electrical engineers, and r informatics engineers working on it. But after it's rolled out the maintenance department might send a
mechanic, an electrician or an informatics tech out depending on what's wrong with the thing when the operator writes a trouble ticket.

Instead, what seems to be happening in some plants is that trench warfare is beginning to break out about who owns the box that looks like
a users desktop computer but in reality runs the plant. At least it seems that in the plant Tony D. has been tasked to organize someone is
looking at it instead of fighting about it. I hope it turns out well.

I have yet to ever see any plant organized along the line I've suggested with the information systems design/planning guys working for the
engineering boss and the information systems technicians working for the maintenance boss. Not a single one. But I'll bet it would work better if it *WAS* organized that way.

MB
--
Michael R. Batchelor

www.ind-info.com/schedule.html

GUERRILLA MAINTENANCE [TM] PLC Training
5 Day Hands on PLC Boot Camp for Allen Bradley
PLC-5, SLC-500, and ControlLogix

If you aren't satisfied, don't pay for. Guaranteed. Period.

[email protected]

Industrial Informatics, Inc.
1013 Bankton Cir., Suite C
Charleston, SC 29406

843-329-0342 x111 Voice
843-412-2692 Cell
843-329-0343 FAX
 
I've read Michael's article before but can't remember where. I agree with his proficy and I'm pushing to see it through. The only difference is that I see two groups, one for office networks and one for production networks. The biggest issue I see is money. Upper management can't see spending money for IT in the Maint/Eng group when there is already a group to cover Network maintenance.

The problem with upper managment's theory is that comparing an Office network to a production network is like comparing a 4-door sedan to a MAC truck. Yes, they each drive the same roads but each has a unique and completely different purpose. Most IT personnel don’t understand this. They try to Copy/Paste the office network into the production area and it WILL fail. As Nathan said “"oops, I left the network down for the afternoon" holds a different level of significance when it comes to production.”

Referring to Jasmin’s comment “(you find that most departments have 2-3 "computer whiz kids".)”. I cut my teeth on a 10Base2 network in production because our IT group said, “Go play with your little toys and leave us alone.” Looking back it was the best thing he could have said. I played with my toys and I learned. Trial and error, back then you couldn’t find experts in this area that were reliable. Now our production network dwarfs the office network in our plant and is still under control of Maint/Eng. Today I can’t “play” anymore; production is too reliant on the network. This means that people hired in production networks have to be trained in PRODUCTION networks.

Now I look at IT with contempt at their ignorance of what it takes to make a production network reliable. Most of them think finance makes the money in a plant.

As for a SCADA system, I would use industrial equipment, not office equipment and build a stone wall around the production network with only a pinhole for information to travel through that can be plugged in a moments notice. In my plant “NIMBA” was the warning shot across the bow and “SASSER” was the justification for our diligence in a rock wall of security.
 
M

Michael Batchelor

I really hope that you've seen this somewhere besides somewhere that I've posted it. I really have been advocating such a move since the late 80s when we all used Usenet news, but I've never heard anyone else except me singing this song. I don't even care if I don't get credit for the idea; I just want to see the problem solved. (Who was it? Reagan? That said it's amazing what you can get done when you don't care who gets the credit.)

I disagree, however, that there should be two groups. In the same way that there are not two groups of electricians - one to handle the bathroom light switch and one to handle the motor change out on the conveyor line - there shouldn't be two groups of data technicians. (Obviously gigantic plants have multiple groups of all disciplines with different areas of responsibilities - let's talk about average size plants here.) That isn't to say that the average plant maintenance staff doesn't have a couple of guys who are the ones you send to fix outlets and light switches and a couple of other guys you send to do the heavy lifting. Every good electrical supervisor has a grasp of his technician's capabilities and a good handle on the differences of requirements between the bathroom light switch and the conveyor motor. Likewise for the mechanical guys. Different requirements for different areas, but one supervisor.

I also won't deny that the requirements for the "plant desktop data network" and the "central accounting system" and the "production network" are very different. But the Data Supervisor should have a handle on the differences just like the Electrical Supervisor and the Mechanical Supervisor, and he should report to the Maintenance Manager just like they do. The problem is that it would require *GIANT* organizational changes, with winners and losers. The problem is political, not technical, and the CIO has a corner glass office next to the CEO, while the Maintenance Managers and the Engineering Managers have cubes out on the floor. I *WILL* deny that any CIO wants to move his desk down to the dirty production environment. Witness that this discussion is going on in a "controls newsgroup" and not in an "IT newsgroup."

--
Michael R. Batchelor

www.ind-info.com
 
Tom,

I have the problem where IT has *improved the security on the Intranet* which drops any attachments (i.e. AutoCAD drawings) to protect the system and in result reduce our productivity *who knows where this will end*.

Maybe we should revert back to snail mail?

Dennis
 
W
It is happening. Peter Zornio, when he was at Honeywell, predicted it to me years ago. He said that it was far easier to take a plant networks and information person and make them understand enterprise IT than to take an enterprise IT manager and get him or her to understand how a plant works, and why it is different.

At this past week's Invensys Customer Conference, Marty Edwards (who recently moved from Georgia Pacific to Idaho National Laboratory) showed a chart with the differences between IT upgrade and patching practice and what is allowable on the plant floor-- and there were few similarities. IT simply doesn't understand the fundamental difference.

The biggest difference is that in the Enterprise world, the fundamental security practice is to protect the server--- save the data! In the plant floor environment, the fundamental security practice is to protect the controller and the plant floor loops and controls. These are diametrically opposed objectives.

That's why, as Eric Byres said to me the other day, Cisco never came up with the Tofino device. If you haven't seen that yet, you will. It is a revolutionary edge peripheral firewall that was designed to protect controllers and field devices. MTL will be selling it in 2007, and they are demonstrating it periodically. I saw it at the Honeywell EMEA User Group meeting in Spain a couple of weeks ago. Byres was there, and had a demo set up with a networked PLC operating a pump and level controller loop to fill and drain a storage tank. There was an HMI so we could see what was going on.

Byres used some basic blackhat penetration software and was able to spoof the HMI so that everything continued to look fine, and then cause the pump to force on, and overflow the tank. He could do anything he wanted to the PLC and nobody would know.

Then he put the Tofino firewall in the line between the network and the PLC, and the PLC absolutely disappeared to the blackhat software.

As we look at the coming proliferation of networked wireless devices in plants, this kind of device (Joanne Byres assures me the patents are good, strong and unbreakable) is going to be a necessity right at the field device (whether sensor, controller or final control).

Walt Boyes
Editor in Chief CONTROL magazine
Putman Media Inc.
555 W. Pierce Rd. Suite 301 Itasca, IL 60143 www.controlglobal.com
blog: Sound Off!! at controlglobal.com
630-467-1301 x 368 [email protected]
 
M

Michael Batchelor

I'll still stand by my assertion below that this is going to take giant organizational changes. Walt is dead on the money in all of his assertions, but still, as I quote myself below, witness that this discussion is going on in a controls newsgroup, not an IT newsgroup. Until this starts getting exposure in the IT world, it's DOA.

Anyone got the guts to throw this thread on the desk of their CIO?

Michael
 
D

Dennis Patterson

The question is not difficult to answer. Just because i can drive, doesnt make me a proffessional courier. Just because its in a PC doesnt make it IT responsibility. If you are to use that approach, then you could argue that the lady at the front desk using the word processor should look after the SCADA!

Dennis Patterson
 
> I'm no Scada expert. I'm an IT guy.

You just answered your own question! Leave it for the SCADA Expert

Michael
 
Of course I agree with you.

However, these changes are beginning to happen.
I am aware of more than a dozen people who head both plant automation and IT in their plants.

IT is being downsized because it isn't cost effective the way it is currently organized.

Walt Boyes
Editor in Chief
Control magazine
www.controlglobal.com
blog:Sound OFF!! http://waltboyes.livejournal.com
_________________

Putman Media Inc.
555 W. Pierce Rd. Suite 301
Itasca, IL 60143
630-467-1301 x368
[email protected]
 
Nathan,

Thank you for touching this one.

My original post complained about IT mucking with the Intranet to their comfort or pleasure and our regret -- I've since got over that and realized that the 1,000s of peaple in the company have the same problem which will result in an early resolution. I still do not believe they should go about installing patches untill all of the ramifications are understood for IT and Production. I do believe cooperation would be a great help in both cases.

Dennis
 
N

Nathan Boeger

> It is happening. Peter Zornio, when he was at Honeywell, predicted it to me years ago. He said that it was far easier to take a plant networks and information person and make them understand enterprise IT than to take an enterprise IT manager and get him or her to understand how a plant works, and why it is different. <

In fundamental concept perhaps - but there is a distinction between "enterprise IT" and desktop support. Your typical plant network person doesn't know a thing about properly managing servers, networking beyond fundamentals, etc. They have to start at square 1 in their IT curriculum, just as the IT manager needs to start at the beginning with controls technology. Both sides need to learn to look beyond the physical hardware with which they may assume to be familiar. This may not address your point directly, but far too many controls people bash IT for running desktop patches that screw up shoddy HMI programs. Just because that's a typical IT department responsibility doesn't mean that it's the bulk of what they specialize in. IT needs to be informed of how to support specialized devices (PCs running controls apps). This is especially true since HMI vendors are so irresponsible about supporting the platform on which they run.

> IT simply doesn't understand the fundamental difference. <

Inform them - IT's job is to support and maintain computers and the network. If an IT department works for a manufacturing plant, they need to know what production priorities are.

> Then he put the Tofino firewall in the line between the network and the PLC, and the PLC absolutely disappeared to the blackhat software. <

That's what a firewall does. I think the Tofino device is a cool thing - it allows a PLC type who has no idea what he's doing with networking or security to provide some amount of protection to the plant floor - an innovative idea. However, a good IT department would do better. Worst case they could work with you on configuring that device, and actually understand what's going on.

I feel weird standing on the side of IT after doing project after project arguing these points from the other side. We all recognize the convergence of technologies. Now it's important to recognize each others skill sets in the area. SCADA security tends to be pathetic. Most industrial organizations will wish they'd worked with their IT department if they ever get attacked.

----
Nathan Boeger
boeger AT inductive automation DOT com
Inductive Automation
"Design Simplicity Cures Engineered Complexity"
 
M

Michael Batchelor

My point exactly of why there needs to be one group, not two separate groups. It is true that the physical appearance fools a lot of people. It looks like a computer, so it belongs to us. The PLC doesnt' look like a computer, so you can play with the software on it willy-nilly without any procedures, despite the fact that some of those PLCs are now collecting data that's regulated by Sarbanes Oxley.

This isn't a simple issue. There are really competing needs here, some of which are mutually exclusive, or at least requires creative solutions. If one manager has responsibility, and understands both sets of requirements, then it's more likely to get solved.

Michael
 
I have to take exception to something Nathan said after he quoted me.

It is not true, in my experience, that vendors produce either shoddy programs or support them badly. Nor does Microsoft produce bad software.

Before everybody from the "Church of Kill Bill" leaps on me again, let me point out that CERT shows a large number of vulnerabilities in every distro of xNIX from OS X to Linux. If you look at the size of the distribution of MS apps vs the size of the xNIX apps, if there were as many Linux boxes as there are Windows boxes, we'd all be complaining about how shoddy Linux is, or how virus and trojan vulnerable OS X is...

The issue, as Joe Weiss from Kema and I were talking about on the phone last night, is not really one of coding. It is about policies and procedures, auditing and training.

I read somewhere that on the order of 40% of all control systems still have "password" as the root administrator password years after delivery, and the "guest" signon is still enabled in more than half.

You can't legitimately thwack vendors for that.

Walt Boyes
Editor in Chief
Control magazine
www.controlglobal.com
blog:Sound OFF!! http://waltboyes.livejournal.com
_________________

Putman Media Inc.
555 W. Pierce Rd. Suite 301
Itasca, IL 60143
630-467-1301 x368
[email protected]
 
M

Michael Batchelor

> The issue, as Joe Weiss from Kema and I were talking about on the phone last
> night, is not really one of coding. It is about policies and procedures,
> auditing and training. <

Exactly. And I'll assert again, there should be *ONE* set of policies and procedures, not two or a dozen. That's not to say that there won't be two paragraphs or sections dealing with differences between two types of systems. In truth, there are probably a dozen different "domains" of how data systems get used, and all of them need to be addressed within the policies. The mainframes that are still around, and don't think they aren't, need - and should have - a very different set of rules from the desktops, and from the automated equipment, and from the Time and Attendance clocks, and from the security cameras that now write to a HDD instead of a VHS tape.

Some people have told me that it's insane to lump all of them together, but I'll stick to my example that electricity has now moved from the plant floor into the office, and there's 480VAC turning lathes as well as an outlet in the restroom with a GFI and another outlet under the desk that has your monitor plugged into it. "Computers" will, and must necessarily, become as pervasive as electricity.

> I read somewhere that on the order of 40% of all control systems still have
> "password" as the root administrator password years after delivery, and the
> "guest" signon is still enabled in more than half. <

I can attest to this first hand as well. I was called to a plant that I hadn't been into for at least 8 years to see if I could get a daily production report that had stopped printing to work again. No one knew the password, so I tried the default distribution password I have used for years. It worked first time.

Michael

--
Michael R. Batchelor

www.ind-info.com
 
M

Michael Griffin

In repy to Walt Boyes - There are two points which you brought up which I think need addressing.

The first is (to paraphrase it) the claim that "MS-Windows gets cracked much more often than Unix/BSD/Linux because more people use it". That theory has been thoroughly debunked many times already in the general computer press so I won't address it here. The security problems with MS-Windows are due to deep seated design problems and a reluctance by Microsoft to admit to problems they don't want to fix.

The relevance of this first point to automation applications is that anyone who is counting on "security by obscurity" for their automation application is kidding himself. The fact that MMI and SCADA systems are used less than e-mail and word processors does not inherently make them less vulnerable.

The second point that needs addressing is the claim that vendors cannot take responsibility for default passwords not being changed. There is a solution for that and it is to not have any default passwords or "guest" accounts. When you install the software, you should get no functionality until you create your own login and password.

The reason why so many systems install with default passwords is because having an installed default password is less work than writing code which handles the special case of starting up the first time with no passwords configured. It is also seen as being more "user friendly", because you can install and start up the software without having to create any logins (under the theory that someone will get around to changing the passwords "later"). The problem with that approach is that if the software looks as if it is working people are inclined to just leave it alone rather than digging into it to find out how it works.

So yes, we can "legitimately thwack vendors for that".
 
Top