Computer Virus in Electric Utility in Australia

M

Thread Starter

M Griffin

There were reports in the news recently that a computer virus caused serious problems earlier this month for an electric utility in NSW and Queensland in Australia. The news reports that I have seen about it were vague and conflicting. I was wondering if anyone had any solid news on what went on (as opposed to the rumours that are floating around).

The story as I understand it is:

1) An electric utility in NSW and Queensland in Australia was severely affected by a computer virus at the beginning of October.

2) The problem apparently originated in their business systems.

3) There was no effective separation between the business systems and the control systems.

4) The operating system which the control system runs on used Solaris. However, the operator stations which were used to interact with the controls ran X terminal software on PCs using MS Windows. These MS Windows workstations were directly vulnerable to the virus.

5) As a result of the above, the virus affected the operator control stations and prevented the operators from monitoring or interacting with the control system (which itself was still functioning).

6) Outside help had to be brought in to deal with the problem, and dealing with it was a long and expensive process.

The news reports that I have seen so far on this have been vague and confused. I was hoping that something more solid would turn up later, but so far nothing has.

I was wondering if anyone had any substantial information on:

a) What took place, particularly with regards to how it started,

b) What was affected,

c) How long it took to get under control, and

d) what temporary and permanent measures were taken to deal with it.

It would also be interesting to hear what measures anyone else has taken to prevent a similar or more serious problem from occurring in their own SCADA, HMI, or other system. I suspect that the typical SCADA, HMI, or plant historian system would be much more vulnerable than was the case in this instance.
 
I am also interested in knowing the details about this case. In our installation, the SCADA/PLC Network is connected to business servers through a client station. In other words, one Control Client of SCADA/PLC network (this is the development station) has two Fiber Optic Network interfaces (NIC Modules) and one each is connected to control and business network respectively. We use a lot of VB based programs (Old DDE technique) that generate production reports for higher management and puts them on the Business Network. This station is working as a network bridge between the two networks. All our software is based on Windows based Operating System and hence we are worried that viruses may creep in to our control network through business network. We are using Norton Anti-virus in all our networks and it is regularly updated. Ours is 24X7 manufacturing unit and hence it is necessary that we do not have any plant outages due to infected stations/servers. Please advise and comment.
 
C

curt wuollet

The scanty information I read said that the idiots had their operation dependent on Windows even though they didn't think it was. It was just a matter of time.

Regards,
cww
 
In reply to RK Sastry: I can imagine a number of ways to improve your system. However the fact that the bridge and report generation functions are combined in one system adds an unknown complicating factor.

I think the "bridge" system needs to be something that doesn't have the same vulnerabilities as what you are attempting to protect. As it stands now, the bridge can get infected and then in turn pass on the problem to all the systems on the control side of the network. In other words, so far as viruses, worms, and other similar problems are concerned, your control network is connected to your business network.

I think the most straightforward solution to this is to simply replace the current "bridge" system with one running a different operating system (Linux, BSD, OpenSolaris, etc.). The report generator could be a normal web application.

A program running somewhere on the control network would stuff data into the (we'll call it "report bridge"). The "report bridge" would store the information in a database. Reports would be generated using standard web application techniques (the report gets generated on the fly when you request a web page).

This means replacing your existing report system because the VB programs won't run on the new system. I considered the possibility of keeping the existing report scripts by adding another computer to the system, but that adds a lot of complication. This isn't something that people would appreciate very much later if they have to troubleshoot problems or make changes.

If you want to discuss the above further let me know. The details would depend on what protocols you can use to communicate with the SCADA system.

As for your anti-virus systems, I'm not a big fan of depending on anti-virus. When a new virus (or worm, or whatever) is going around, the anti-virus won't help you until an update for it is available. Viruses spread the most rapidly in the first 24 to 36 hours of their existence. Since your control network is managed separately, I assume you are updating the virus signatures manually, if at all.

Also, sometimes the anti-virus programs seem to be worse than the viruses in causing disruption. You probably shouldn't deploy an anti-virus update until you have had a chance to test how it affects your SCADA system.

All this means there is probably a gap in your coverage just at the time when you need it the most. I'm not saying to not use anti-virus at all, but I am saying that I think the protection it offers is very limited.
 
A

Andrey Romanenko

Hello,

I agree with Michael's ideas on the replacement of the bridge with a more robust OS and eventually moving away the reporting to another station.

Ciengis has an open-source solution called Plantstreamer that is a plant historian that runs on Linux (or any other java enabled platform). It gets the data via OPC and stores it in a PostgreSQL database. A separate web based system, Plantbeat, is a front-end for the database with many features including online reporting, performance monitoring, etc. We also use Openreports for on-the-fly and offline reporting. The latter creates periodic reports and either stores them or sends them via email to the interested authorized users.

Currently the systems are used at a large petrochemical site performing very well.

I suggest you take a look at Michael's open-source project Mblogic, too, because it provides an interesting web-based HMI and automation platform.

In either case windows viruses stand no chance in these environments.

Please contact me if you are interested in Plantstreamer and Plantbeat.


Best Regards,
Andrey Romanenko
andrey at ciengis dot com
 
We are also a 24x7 manufacturing unit having three standalone SCADA running on Windows platform. I meanwhile explored to "network" these SCADA with our Business network (i.e. LAN). But the vendor who supplied the system was not very happy with the idea. He suggested me to install a Router and Firewall in between SCADA systems and Business Network to bridge two networks. Ultimately judging the risk of viruses, I dropped the project.
 
And here
http://www.integral.com.au/wps/wcm/...ERES&CACHEID=2dc9fa804fc929beb9c3bd1b70e88c03

You will find:
2 October 2009

The facts: IT virus contained and controlled
Integral Energy has successfully contained the W32/Virut virus.

It only infects executable files (.exe).

This incident has not affected electricity supplies to our customers, nor has it threatened the power grid.

Service levels to customers have been maintained.

This incident has not compromised Integral Energy’s data.

Data transfers to and from Integral Energy via a website or file transfer protocol (FTP) do not pose a threat to other parties.

Integral Energy has put in place recovery plans to eliminate the virus from its business systems. This involves rebuilding personal computers.

As part of these plans, an investigation is underway into the source and cause of the virus.

The majority of issues resulting from this virus have now been resolved.

Integral Energy is reviewing strategies to minimize this risk in the future
 
Hello Michael,

Thank you for posting this as I am trying to keep some kind of log on incidents like this as I have had some exposure to this before.

I looked up the incident and found the following perhaps you did the same but I thought I'd include the links anyway

"http://www.smh.com.au/technology/se...eak-a-threat-to-power-grid-20091001-gdrx.html"

"http://www.zdnet.com.au/news/securi...ttp://news.zdnet.com/2100-9595_22-348060.html"

Regarding measures to be taken, the best publicly available guidance I've seen is the NIST doc SP800-82. Be prepared for some reading!

"http://csrc.nist.gov/publications/PubsDrafts.html"

There is also an ISA standard SP99 but you have to pay for that.

Having read both NIST, ISA and more generally as well as having to put some of these principles into practice I think they have a reasonable stance i.e. don't put all your eggs in one basket (as my mother used to say). Practically they recognise some of the difficulties.

There are a significant number of Windows machines on operator consoles and operating as control system servers in my experience and there would be no Linux fallback as you suspect. And there's no hope of changing this in any short timeframe despite those they may desire otherwise.

Hope this helps
DaveMH
 
C
Yes, nothing to see here, move along! For every one of these that goes public there are at least 10 that are kept secret for security reasons. If their concept of security wasn't a joke, we wouldn't be reading about them. Windows security is an oxymoron.

Regards,
cww
 
In reply to Dilip: Standard routers and firewalls can help somewhat with older types of worms, or at least mitigate the consequences. Their effectiveness is limited against modern worms, which often operate by exploiting holes in permitted actions (e.g. buffer overflows and SQL injection). They are generally useless against viruses.

I don't think that normal routers and firewalls are the answer to your situation. You would need something that examines the actual data to decide what to let through.
 
In response to various posts: I was aware of the Sydney Morning Herald report, but that was one of the vague and conflicting reports I was referring to. The Inquirer link had a bit more information, but that was based on a Slashdot post, which is not exactly one of the most reliable sources.

There was a more serious incident several years ago at a nuclear power plant in the USA.
http://www.securityfocus.com/news/6767

The article starts: "The Slammer worm penetrated a private computer network at Ohio's Davis-Besse nuclear power plant in January and disabled a safety monitoring system for nearly five hours, despite a belief by plant personnel that the network was protected by a firewall, (...)"

As for security "solutions", here's a couple of examples:

The following adds Modbus/TCP related features to the standard Linux Netfilter packet filtering:
http://modbusfw.sourceforge.net/

The following is a commercial product which I believe is also based on Linux Netfilter:
http://www.tofinosecurity.com/

For complex systems, a Netfilter type solution might be the only solution. The problem I see with this though is you have to decide what to let through and what to block. If you do that wrong, then you don't have the protection that you thought you did.

Furthermore, a lot of application vulnerabilities consist of sending "legitimate" data that just happens to exploit a bug in a program. That means that you can have the filtering set up perfectly as to specification, but still be vulnerable in practise.

Most people though seem to have relatively simple requirements where they simply want to make report and status data available to the business network. I think that a computer that sits on both the business and control networks is a more straight forward solution, provided it is a computer which doesn't have the same problems that the business and control computers do.

Even with that isolation though, the control system is still not fully protected. At the nuclear plant mentioned above, most of the control system was immune to the worm itself. However, the worm was scanning (for other targets to attack) so actively that it clogged up the network so that the otherwise unaffected systems couldn't communicate reliably. In other words, if you depend on the network for anything important it only takes *one* vulnerable computer to compromise the whole system.

All it takes is for someone to plug a USB stick into one of the computers (the default configuration for MS Windows is to automatically run programs on it) or to plug a laptop into the network and you have a big problem.

The answer the SCADA (and related) industry to all this seems to be to wash their hands of the whole affair and say that security is the customer's problem. I don't think that is a reasonable approach. However, I suspect that the vendors of the most popular SCADA systems don't have the knowledge or experience to deal with the real problem effectively.
 
Michael

I don't think it is entirely fair to say that all vendors wash their hands and say it is the customers problem (I don't work for a vendor before anybody asks). Some vendors do have some expertise in this area. But, nobody know it all. There is work going on to address some of the frailties that are particular to control systems both within vendors and as a collaborative body e.g.

"http://www.wurldtech.com/services/certified_devices.html"

Admittedly some of this is driven by commercial imperatives and this should not be the only solution. But, responsible people (from all types of companies) are thinking about this and trying to do something. I expect it will be a slow process and mistakes will be made and I don't believe total security will ever be possible.

For the reasons you highlighted defense in depth is important. Don't rely upon filters and firewalls alone or single box solutions of any type. There will be errors or ways round (e.g. USB sticks, mobile devices). Use simple solutions for simple situations and small degrees of risk.

Ultimately the responsibility for security lies with those on the operating site as they are the ones left after the project team and vendors walk away. And also those that provide the link to business systems. Responsible ones are developing their own polices and still learning themselves. If you have a customer who doesn't, help them come to an informed, rational design decision without resorting to alarmist language. Security nazis are as unhelpful as any other type, as are cynics.

We are, hopefully, engineers seeking practical solutions, in the real world.

DaveMH
 
M

Matt Warshawsky

In response to the various posts:

A firewall with no open ports (i.e. doesn't let any traffic in), would stop any worm.

If the internal SCADA system alone behind a closed firewall does not allow anyone to open email, plug in USB drives, or generally execute anything other than the SCADA system, then a virus cannot run either and there are plenty of kiosk tools that limit the user to a single application.

The problems then occur when you either open up some ports, create a VPN connection or the like, or when you allow users of your SCADA system to do things other than use the SCADA system (and often OS user security controls can help with this: i.e. no OS administrator login to run the system)

If you then want to make your data available from outside, you can push the data out using many different ways and then ignore any response. Although it is possible for the data to become corrupted once it is "outside", there is no way for a virus, worm, or hacker to get back in through the data push.

This is the idea behind DAQConnect (www.daqconnect.com). Data is pushed out to the DAQConnect servers over HTTPS and nothing comes back. You access the data on the DAQConnect servers, and thus don't have to open any ports or expose any of the normal vulnerabilities. The biggest concern in this system is actually someone stealing your DAQConnect password and somehow altering your data (or more likely editing your screens to make your data look altered). But even with your login information they can never get back to the main SCADA system to do harm.
 
M

Matt Warshawsky

I guess I should add:

A firewall is actually just a piece of software running on an embedded system, so technically, although it would stop worms, it can be hacked. So really the only way to fully secure a SCADA system is to put the SCADA computers in locked boxes (so nothing external could be plugged in) running kiosk software (to prevent running any other apps), and unplugging it from any other network.

Alas, like all of life, there is the risk vs reward question. Running your SCADA system in such an environment would be secure, but then you can't take advantage of all the things integration with other systems would get you, most of which increase efficiency and profit.

So, in my opinion its best to simply reduce the risk as much as is feasible and perform two steps to ensure that if you do get hacked, get a virus or otherwise, you can quickly recover:

1) ensure that you have hardwire safeties (i.e. not running through complex logic like a PLC or PC) on everything so that the SCADA system can not possibly control your system in a way that could cause injury or property damage. This is process control 101 and applies always.

2) perform proper, rotating system backups. If recovery time is also an issue, create images of each system so they can be quickly restored.
 
In reply to Matt Warshawsky: It's just as well that you added those remarks in your second reply, because the first thing that came to my mind when reading your first response was that a firewall is just another piece of software, and it can have security problems as well.

Also, a lot of the newer security vulnerabilities operate by exploiting bugs in the actions that are permitted, not by doing things that can simply be blocked. Firewalls are so common these days that modern viruses and worms have to work around them all the time anyway.

Yes it's true that if you don't connect the control network to the business network, then these problems don't arise (although you can still have problems from other vectors). But you've got to remember that what we are talking about is connecting the business network to the control network so that they can share data on things like schedules, quantities, costs, results, etc. So, what we are talking about is how to deal with the risks this causes.

As for DAQConnect, I don't see that as addressing the sort of questions that we are talking about. What we are talking about here is close two-way data sharing between the control system and the business systems (ERP, managers, etc.). I can think of several pretty obvious markets for what DAQConnect can do, but I don't think this one is a good fit.

As for hardwired safeties and backups, those are mitigation strategies, not prevention. In the types of applications we are talking about here, unscheduled downtime is something to be avoided at all costs. In those cases the emphasis should be on means to avoid the problems in the first place.
 
In reply to DaveMH: You said: "Ultimately the responsibility for security lies with those on the operating site as they are the ones left after the project team and vendors walk away."

That's the problem that I'm talking about. I'm not in the SCADA business, and it's quite possible that I don't know what I am talking about. However, it appears to me that many SCADA systems bear a lot of resemblance to important business systems.

In those systems, the vendors don't "walk away" unless you want them to. They stand behind their systems completely for as long as you have it. Now they do expect to get paid for that. But if a large business's directors can approve a support contract for a bean counting system, they can approve one for the system that is responsible for actually making the money.

I can get a disk that has an operating system, database, web server, development software, and everything else I need from a single vendor. If the system matters enough to me, I can buy an annual contract where they will give me 24/7 support to stand behind every piece of software that was on that disk. There's no finger pointing at other vendors. They take responsibility for everything. There are even multiple vendors selling the same thing, so I can shop around for the best price or service. Now, why isn't the SCADA business more like that?

To be fair, I should point out that the control system in Australia that started this discussion wasn't itself affected by the virus. It was running Solaris, which is immune to MS Windows viruses. It was the operator terminals were using MS Windows to run X terminal software which had the problem.

However, I think my main point stands. I think that to seriously address security problems requires the SCADA vendors to take more responsibility for it. And for that to work requires them to supply the complete platform, not just "their" piece of it. This model works quite successfully elsewhere, so I don't see why it couldn't work here as well.
 
M

Matt Warshawsky

Yes, you are correct on everything you said. In my opinion, the reason there are so many security vulnerabilities is that protocols are often designed to do too much, perhaps to make them more marketable, perhaps because they aren't designed for control systems. In the end, the KISS mantra should really be utilized. Passing simple data between two different machines shouldn't require a complex protocol, and therefore should be easy to code without creating security holes. But alas, most protocols try and do everything and thus the problems.

You are correct, DAQConnect is not a good fit for connection to business systems. It was just an example of how data can get out of a system without creating any security issues in the system itself.

As for hardwire safeties and backups, my point wasn't that you shouldn't do whatever you can to avoid the problem, but simply that with these safeties in place, you can safely balance the risk vs reward/cost of implementation. Yes downtime can be expensive, but that has to be compared to the chance of a problem happening and the cost to prevent it. Many security strategies are inexpensive to implement and should always be in place, but there is no reason to implement a million dollar security strategy if a virus can only cause $50,000 of downtime costs because you had good backups. Some of these articles quoted really seem to try and invoke fear, but really its just standard business risk analysis.
 
Top