This message is a follow-up to a previous one I posted a month ago (19/06/06) titled: "HMI: COMM: An interesting article on a SCADA security vulnerability." There have been further developments which some people may find interesting.

There is a group in the USA which seems to be organised by the "US Department of Energy and the US Department of Homeland Security" called the "SCADA and Control Systems Procurement Project". They are evidently working on standards for SCADA security for "critical infrastructure" (generating plants, traffic
signals, dams, etc.).

These standards are for specification language for security which customers will be expected to use in their contracts for SCADA systems. The language covers system specifications and acceptance tests. I gather that it is expected that there will eventually be legislation requiring the system operators to implement these guidelines as minimum security standards for all new "critical" SCADA systems.

This requirement will be passed down to integrators as part of the procurement contract. It thus appears that integrators in the USA will be required to become more familiar with security if they wish to stay in the business. It is also likely that other countries will be adopting similar measures (if they have not already done so). The participants so far appear to be mainly large American electric utilities (although I am not familiar enough with the
USA to be sure who all of them are).

There is at present a draft document (PDF) available from the "SCADA and Control Systems Procurement Project" web site (see below for the link). The document is still being worked on and is at present incomplete. The group invites comment and participation, so anyone who deals with these sorts of things as part of their business may want to look at this before they find themselves having to implement something they feel is impractical.

I have skimmed over the document (48 pages), and most of it appears to concern general computer system security (including access control, network patitioning, configuration, etc.), rather than measures which are specific to
SCADA systems. In other words, it is more about the operating system and network configuration the SCADA system is hosted on than about SCADA itself.

However, the intent is very clearly that the integrators who install the SCADA systems will be required (as part of the contract and acceptance tests) to design and implement the security measures, so this isn't something that
can be fobbed off on the customer.

One thing which is quite clear at this point however is that simply buying a computer with the latest version of MS-Windows and stuffing the SCADA software CD in the drive will not even come close to meeting the minimum
requirements. The system must be stripped down to remove all unused software, operating system file access permissions must be tightend up, default accounts must be removed, account logging and auditing implemented, known
flaws patched, etc. In other words, there is a lot of system preparation work which must be done as well as ongoing maintenance.

The history of this security specification process started out with concerns about the security of critical infrastructure. However, there is no clear lower limit to what can be considered "critical". I can see this standard
being applied to water and sewage systems, process industries, food processing, etc. I rather strongly suspect that many integrators (especially the smaller ones) will be unprepared for this, and will get burned on a future contract when they find they have gotten into something over their heads.

I would be interested in any comments on the above.

Article in The Register (re-print of Security Focus version):
http://www.theregister.co.uk/2006/07/28/scada_security_push/

Original version in Security Focus:
http://www.securityfocus.com/news/11402

Web site for "SCADA and Control Systems Procurement Project"
http://www.cscic.state.ny.us/msisac/scada/

Link to draft specification (PDF, 447k):
http://www.cscic.state.ny.us/msisac/scada/documents/25-jul-06-SCADA-procurement-draft.pdf

 

In my opinion, integrators don't pay nearly enough attention to security. Critical infrastructure cannot have its fly wide open to the very real threat of hackers and terrorists. All the time I happen upon private FTP servers with anonymous access to fairly sensitive industrial data, because there's only one IT person for an entire company and they're either incompetent or their boss is too lazy to remember passwords for things.

 

That's why I see exposing automation networks as indefensible unless you have the knowledge and vigilance to adequately secure them. And very few people have these. And as a practical matter, it may well be impossible to secure Windows and other shrinkwrap software because it's impossible to audit and may contain any number of yet to be discovered vulnerabilities. It's just a bad idea to do critical systems on other than a private isolated network.

But that hasn't stopped anybody.

Regards
cww

 

In reply to both Curt Wuollet and Publius - The existing SCADA systems that the working group is concerned about were typically designed to be operated on an isolated network. It has turned out however that these networks are not *staying* isolated.

Companies want to connect the "operating" (plant) network to the "business" network to provide real time transfer of orders and production results. The business network in turn is directly or indirectly connected to the rest of the world (or at least to the rest of the company).

I can give you a real world example of this from a couple of years ago (I have mentioned this incident previously). In the aftermath of the Canada-USA black-out of 2003, information about various power plant problems surfaced which would normally not have been made public. One such incident was a problem at a nuclear power plant in Ohio USA (which happened to be owned by the same company where the black-out originated).

In early 2003, they had a serious problem with their control system caused by a computer virus. Fortunately, the plant was already shut down due to unrelated problems. The MS "worm of the week" (SLQ Slammer at that time) entered the control network via the business network and found at least one MS-Windows server to infect. The worm's scanning activities then crashed another computer running the "Safety Parameter Display System", which monitored things like coolant systems, core temperatures, radiation levels, etc. Later, another computer also crashed due to the scanning activities. (The articles which reported on this used the term "crash" without providing more details).

The worm got into the control network because the control network was connected to the business network in order to provide data on things like
operating efficiencies. People on the business network want this data because they believe it is worth millions of dollars to them. The business network was theoretically protected by a firewall, but there were in fact multiple connections to outside business partners which bypassed the firewall (the third parties likely needed more access than the firewall would allow).

The SQL Slammer worm was a very agressive "scanner" (probing for more systems to infect). Even if it couldn't infect a system, it would use so much network bandwidth scanning that the network would be clogged up so the legitimate
applications couldn't transfer data in a timely manner (causing communication time-outs).

Links:
http://www.securityfocus.com/news/6767
http://www.theregister.co.uk/2003/08/20/slammer_worm_crashed_ohio_nuke/


To bring us back to the current day and the report which I pointed out, the "SCADA and Control Systems Procurement Project" working group apparently does intend to deal with these sorts of problems. They however intend to create secure, reliable SCADA systems by writing this in as a requirement in contracts with their suppliers (apparently the integrators). The scope of the
"SCADA and Control Systems Procurement Project" is to come up with common contract language saying that they want it, not to figure out in detail how
to do it (i.e. it isn't a "howto" on actually creating secure systems).

The above example was for a nuclear power plant. However, as I said in a previous message, there is really no obvious lower limit as to what constitutes "critical infrastructure". Municipal water systems could also be considered equally "critical". What I see the problem as being is:

1) The "critical infrastructure" owners intend to push the responsibility down onto their integrators. They apparently have no intention of dealing with this using their own internal resources.

2) Most SCADA or MMI packages are not designed to provide this level of security on their own. This is not necessarily due to shortcomings in these
packages themselves (although some do have their own security vulnerabilities). Rather, a complete "system" typically requires the integrator to separately purchase a third party operating system (usually MS-Windows) and often a database (e.g. the MS-SQL Server database which in the example was attacked by the SQL Slammer worm). The most vulnerable portions are not being provided by the SCADA or MMI software vendors, so they can't address the security problems.

3) The third party operating system and database is supposed to be tailored to close off unused ports, strip out all unused software and drivers, etc. This requires detailed knowledge of each product, plus hours of work which must be repeated for each installation. You can't simply order a computer from Dell and expect it to meet the contract requirements (see the report referenced in my previous message).

4) Integrators are in many instances completely unprepared to provide this level of security themselves. They often don't have the knowledge to dig into the OS to this depth. Their expertise often lies in the production process, not in operating systems and databases.

5) Even if the OS were to be adequately tailored to meet the requirement, who is going to maintain it in its special tailored version? What happens if you download the latest MS-Windows patches - do you end up with MS Media Player again after spending so much time to remove it? Do you end up with vulnerable ports being turned back on again?

If this "SCADA and Control Systems Procurement Project" is really going to accomplish something, I can really only see it as being feasible if the
SCADA/MMI software vendor supplies all the major components - SCADA/MMI development software, operating system, and database (if used). The software vendor in this case would be responsible for making sure the default install of the entire system is secure. They could also maintain the software update and patch system (via customer software maintenance contracts). The integrator could then concentrate on tailoring the application to the production process. A secure data exchange "box" could provide a secure connection between the production and business systems.

The above may perhaps already be the case for some specialist SCADA/MMI systems, but it is not so for the "mass market" ones we typically hear about on this list. I can think of ways by which these software vendors could provide complete packages, but it would require some new thinking on their
part, and changes in the way they conduct their business.

 

If I may comment on some points on SCADA (or any other application for that matter) security:

1. As you pointed out, the application can be no more secure than the underlying operating system. Of course, the people implementing the system have to set it up to maximize the security and that job is hardest on Windows just because of the flexibility that delights most users. The system has to be stripped down to the bones to make it secure so that no code comes over the network to be executed in the browser, no files are opened with office products that execute code in the file and no email comes in with attachments to be opened.

With other operating systems, setting security is much simpler as the design is more robust and none more so than OpenVMS. Happily, the critical rod control in most nuclear plants is based on VMS. For those who doubt, check the reported vulnerabilities and I know that OpenVMS has a long-term future with HP.

2. Regardless of how secure the SCADA system is set up, over time the security policies have to be maintained so that insecurities do not creep in over time. The more users that are able to make changes to the system, the higher the probability that this can occur.

Peter

 

It all comes back to the choice of the world's least secure and least reliable software as the basis. At that point the only hope is isolation. There are much better choices readily available, choices where, for example you really can remove everything but what you need and still have a functional system. Choices that are secure enough for the router companies and the NSA. But even there, by the current model, you would plop closed, secret, gobs of binary only drek on top, written by God knows who and totally inauditable. And then by popular demand you would let anyone load anything so they can access the system from home, etc. The only reason that current offerings exposed to public networks aren't 100% compromised is that there aren't that many people interested in what's on them. People who make these kinds of choices and hope for security are delusional. The kinds of things people want and need can be done and done securely, just not with the crud they insist on using. When it's a struggle to keep your desktop cleaned out and running, it's irrational to see that same software as the basis for a secure, critical system of any sort. Even if you hire a team of experts to administer it. Right now, the industry is chained to an albatross and all the wishing and feel-good gatherings won't change that. It's merely a state of denial brought on by popularity.

Regards

cww

 

Let me apologize in advance for the length... but it's all relavent, promise!

Your points regarding (in)security of Microsoft Windows and SQL server are point-on, minus one thing: for a knowledgeable Windows System Administrator (I am one), the security is not difficult to apply repeatedly with minimal effort.

Yes, you read that correctly. Windows security is easy to replicate on each server sold and to reapply after each patch update.

How? Microsoft included the capability to use security templates in every Windows OS version since NT. Once I do the research to confirm my security configuration, I create the template. For each new system, I apply the template in about 5 minutes. And that's necessary only if the vendors are building the base servers from scratch, which is very hard to believe they would do. Much easier and more reliable to clone a disk with the core system load on it.

For production systems, one can reanalyze the computer's security against the baseline security configuration (called auditing) and look for changes. In the MS built-in tool, it shows up as red "X's" for each mismatch.

Is it perfect? Of course not. The only truly secure system is one with the cable cut (and wireless link and thumb drives and... well, you know!) But you are correct. Folks always find ways in, often as simple as through thumb drives with mp3's on it.

Using this security approach, all but the most dedicated, disciplined, and skilled "black hat" hackers will give up before breaking in. They can get in if they really want to, but it would be a major pain to do so.

Here's MS's HOWTO for the analysis tool:
http://support.microsoft.com/default.aspx?scid=kb;en-us;Q313203&sd=tech

Here's a 1-page simple summary of how it works:
http://esj.com/Security/article.aspx?EditorialsID=1141

Here's a 1-page summary of how to audit a computer's existing security against the "security baseline":
http://esj.com/Security/article.aspx?EditorialsID=1164

As for the vendors? Train their System Administrators to implment industry standard best practices... here are the tools:
www.CISecurity.org (Center for Internet Security).

As a comparison, I also harden security for Solaris and Linux systems. When we finish, all three system OS's are all nearly equal in penetration tests -- so long as they have current patches. Yes, that means phase out systems BEFORE the OS vendor stops providing security patches.

If you like, you may email me at [email protected]. Sorry this is so long.

Thx, Jim M.

 

For the sake of argument let's agree that you can secure Windows systems for more than a few moments. At least 90% of what I've actually seen running in the real world is pretty close to as distributed with any "service packs" absolutely necessary to get the software to load and run. That is the problem. Even those few establishments that happen to have any Windows talent available keep them so busy rebooting, reloading, and tending the other infelicities associated with living with Windows that keeping machines updated somehow fares badly in comparison with keeping them at least marginally running. Many automation products won't run on a secure machine as they depend on kewl features that _are_ the security holes. And the software itself ranges from crude globs of VB to some fairly well crafted stuff. Most of which tends to be well past it's planned obsolete date, and ridiculously old from a security standpoint, that's if the vendor has time to worry about security, which they don't. And if they provide security support for old products, which will get you gales of laughter. So if, in an industry that tends to be so glad just to get a system running that they don't and can't afford to touch it, you can get them to expend resources and keep their systems unstable forever, yes, you might be able to use Windows more securely. I simply don't see that happening any time soon.

I'm fairly sure even the Windows crowd can agree with me on at least a few of those points. If they aren't still in denial. For my own part, when I'm forced to use it, an 8 year old version that still runs the software I need for the machines that I have to work on, I thank God when it finishes booting and I have a genuine dread of mucking with it. For $10k or so, and a week or two with luck, I could upgrade some of it to run on an OS from this century. Much of it will never be ported to anything you could secure. This is the state of the art in Automation. And the legacy of the proprietary model. I submit that it makes security as we know it pretty much impossible. I await your solution.

Regards
cww

 

In reply to Curt Wuollet, Jim Magnant and Peter Clout- This reply is a bit late, but as the subject is quite interesting I wish to continue it.

I won't argue as to whether it is theoretically possible to make any particular operating system "secure". "Security" is a subjective concept rather like "safety". The question should not be whether an OS is "secure" in some abstract sense, but rather whether it meets the security requirements specified for the application with little or no change to the default configuration.

I think that Mr. Magnant has made a salient point, although perhaps not necessarily the one he intended. It may indeed be possible to make MS-Windows secure and to maintain the security - if you are an expert. Mr. Magnant has stated that he is indeed an expert in computer systems security in multiple operating systems (MS-Windows, Solaris, Linux). Very, very few people working in the automation field today though can claim to be experts with any operating system.

I doubt that even one in a thousand people creating WinCC / Wonderware / Citech / etc. applications can claim more than a passing familiarity with the internals of the operating system they are hosting the application on. They tend to be people with expertise in the problem domain who are simply using the software as yet another tool. The whole point of MMI/SCADA software using MS-Windows was that these people were not *supposed* to have to know anything about operating systems. They just had to purchase a PC with MS-Windows pre-loaded and to install the SCADA software on it using the automatic install script. Now however this model no longer works, and they are suddenly expected to have a level of expertise that even most full time MS-Windows administrators do not have.

To meet the requirements of the "SCADA and Control Systems Procurement Project" (see the previous messages for a description of this) will require a lot more than simply ordering a computer from Dell and sticking the SCADA CD in the drive. The operating system must be stripped of all unnecessary components and all unused ports closed. This must be done not just once, but each time it is updated. Every system will be different as each integrator (or subcontractor) will have a different idea as to the "right" way to do it.

The degree of security for the current system will depend upon the depth of knowledge of the original integrator, as well as the knowledge of the personnel maintaining the system after it is installed. The most knowledgable party should be the original software publisher (SCADA or MMI vendor). At present however, they simply wash their hands of the whole affair as they don't supply the most problematic components. I don't see this as continuing to be technically or economically viable from the customer's perspective.

What I do see as being viable is where the original software publisher would take responsibility for supplying a complete system in a "secure" configuration. This would require maintaining a customised repository of *all* the software on a typical system, including the operating system, database, web server, etc. Since it is the software publisher who is maintaining and taking responsibility for the repository, they can ensure that the system as a whole meets the security requirements. If they offer a broad enough range of software besides the actual MMI/SCADA components most applications could be constructed solely from their repository.

This model isn't unprecedented, as many of the large scale financial and business systems where security is paramount do in fact operate this way. A single vendor is responsible for supplying and supporting *all* the software on a system. The vendor does not actually write all the software, but they do take responsibility for the third party software they supply and support. The end customers maintain a support contract with them if they wish to keep their systems up to date.

So, to summarise the proposed system would work more or less like the following:

- The end customer specifies the SCADA / MMI system they want. This would typically be the one they have an existing service contract for.

- The software and PC would be acquired through arrangements which are similar to those used today.

- The integrator would completely wipe the hard drive on the PC, and install over it the complete development or run-time SCADA / MMI package on the PC from a single CD or DVD. This would include the SCADA software, operating system, database, web server, etc. Additional components could be downloaded from the vendor's on-line repository. The "as installed" configuration would meet recognised security standards.

- The integrator would develop the SCADA / MMI screens, scripts, etc., and deliver it to the customer as usual.

- All future security updates for all software on the system would come from a single source; the original software publisher.

I don't see any reasonable alternative to this model, unless you consider burying your head in the sand an alternative.