Today is...
Thursday, December 14, 2017
Welcome to Control.com, the global online
community of automation professionals.
Featured Video...
Featured Video
Wiring and programming your servos and I/O just got a lot easier...
Our Advertisers
Help keep our servers running...
Patronize our advertisers!
Visit our Post Archive
New Windows Virus Targeted at Industrial Controls
Security researches have revealed new virus which has been specifically designed to take over industrial controls and has been detected in a number of plants.

There has been a news announcement that there is an MS Windows virus which is specifically designed to take over industrial controls, and this virus has been found in at least 14 plants in several parts of the world. This is not the same virus which we discussed earlier this year, but rather another one altogether. The full story is here:

http://www.itworld.com/security/120550/siemens-stuxnet-worm-hit-industrial-systems

The following is a quote from the story:

The software operates in two stages following infection, according to Symantec Security Response Supervisor Liam O'Murchu. First it uploads configuration information about the Siemens system to a command-and-control server. Then the attackers are able to pick a target and actually reprogram the way it works. "They decide how they want the PLCs to work for them, and then they send code to the infected machines that will change how the PLCs work," O'Murchu said.

End of quote.

There is now a security patch from Microsoft to deal with this. The virus was first detected over a year ago. However, it is normal for information of this type to be held back from the public until Microsoft writes a fix for Windows and is ready to release it. In other words, anyone using this software could have had this virus on their systems for over a year.

The article also states that due to the nature of this virus, simply running a virus removal program will not remove all parts of it. You need to wipe everything and "restore from a secure backup". The article doesn't give details, but that usually means wipe the hard drive and re-install everything (including Windows) from CD. Get some good advice from a qualified IT person before just running a virus scanner and assuming that you're covered.

This particular virus has been targeted at Siemens WinCC and PCS-7, but users of other programs should not feel complacent. There could easily be other variants of the same virus targeted at other packages. The virus is using Windows security holes, so any software using Windows is potentially vulnerable to similar attacks.

You have to wonder if it is really worth it to hook your machines up to the network. I think if you can it is worth it to have your network physically on its own wiring/switches and IP subnet that is not connected to office computers and/or the internet router. I don't know if you could then have a "secure" router/firewall linking your industrial network to your database and file servers, etc. In the end someone has to set this up and maintain it and if your not an almost fulltime IT guy or a security expert what do you do? The problem becomes more tricky when you consider anyone with an infected laptop can plug in behind the firewall (maybe to do some innocent programming) and infect everything anyway.

I guess one question we must ask ourselves is this: "Is it really worth it to connect your office computers [or other non secure servers/PCs] with your critical manufacturing machines?" Does a power plant or a chemical plant really need to expose itself to the internet or office PCs?

So much for Antivirus scanners.

~Ken

By curt wuollet on 15 September, 2010 - 3:45 pm

It's not a good idea, never has been, probably never will be, as long as Windows is the target. Someone did the research and came up with the number that Windows was the target for 99.4% of the malware written to date. It would seem irrational to expose it to the wider world unnecessarily. I think that makes it irrational to consider for Line of Business automation work, period. But, if you are going to use it anyway, you should at least isolate it, especially since automation systems tend not to be updated or brought up and down for virus scans etc. It's not hard to see why security is an iceberg up ahead. The least maintained systems with the most attacked software doesn't seem like a recipe for sound automation. But, I must not be thinking clearly :^) After all, _everybody_ does it.

Regards
cww

Generic malware isn't the problem; viruses like Stuxnet are specifically targeting a specific industrial site, and can be tailored to attack zero-day holes even in linux. No OS platform is 100% secure. Worse, the configuration and apps running on the OS are not necessarily secure; as I pointed out on a previous thread on this subject, the root account password on some industrial controllers is hardcoded to "password"!

Sites need to have procedures and policies in place that reduce their exposure--isolation of the networks, limited physical access to the PCs (ie it is hard to plug a USB drive into a machine locked in a cabinet), managed switches that do not allow unauthorized devices.

Stuxnet also shows that the supply chain is a risk--don't trust your vendors to ship clean PCs. :) That, certainly is a vote for open source, as it is hard to verify that nothing has been tampered with in a proprietary system--especially one with a huge number of files and processes like Windows.

By curt wuollet on 21 September, 2010 - 7:35 pm

No, nothing is immune from a determined attack by someone who is smart enough and has the resources. My question is: Why do we have to make it as easy as possible and use the worst case combination of the world's favorite virus target, insecure apps, and the worst case for virus propagation, having the whole organization using these? And conveniently networked together! If you wanted to make a Honeypot to deliberately collect viruses, you couldn't do much better than that. Seriously, how could you make it any easier? Traditionally, PLCs have been a hard target for mischief. Not because they are particularly secure by design, but because they were isolated and you would need to have physical access and the right tools to sabotage anything. Now, many are tightly tied to, and systems dependent on, extremely vulnerable PCs, that for many reasons, are probably the least frequently maintained and updated in the company. That's why it's not very surprising that this sort of thing is happening, what's surprising is what took them so long? And, of course, that people will continue to follow this worst case model.

Regards
cww

Yes, no OS is 100% secure. However, some are a good deal worse than others. You need to be able to spend most of your time concentrating on running your operations rather than babysitting the OS. That's just common business sense. Somehow banks and stock brokerages manage to be on the Internet without this being an insuperable problem. I would think that with all the money they have someone would have taken a crack at them if it were just a matter of making the effort.

As for the Stuxnet virus itself, there's really nothing that exceptional about it from a technical standpoint. You can buy zero day MS Windows virus exploits on the black market. There are businesses (mainly based in eastern Europe) that write them and then sell them to third parties. The third parties then add whatever payload they want. That's usually to set up a bot-net, but there can be other reasons as well.

As for targeting a specific site, again that isn't new. There's actually a name for it - spear-phishing. Up until now they have been targeted at business or communications systems rather than industrial controls.

There are really only two things that are new in this case. One is that it appears to have been used for political purposes, likely by a country who is trying to hinder Iran's nuclear industry and has no qualms about harming third parties while they are at it. I don't think this is the forum in which to debate on who that country likely is.

The other new thing is that this is targeted at an industrial control system. A lot of people who read this forum have been crossing their fingers and hoping this would never happen. It has happened though, and it has no doubt opened the eyes of a number of people as to the potential money-making opportunities available.

This time, it was likely a political attack. The next incident could simply be a more routine "business" effort. Many legitimate Internet businesses do pay protection money to organised crime to avoid being hacked or knocked off the Internet. You don't have to create a virus in order to use one. You can buy them off the shelf from black market specialists. After that, you just need a payload, and that probably isn't too difficult in itself.

I can think several people whom I believe have the technical ability to create such a thing (although I don't think they would) given an off the shelf virus. It is also relatively easy to think of ways of making large sums of money by knowing that a large plant will be disrupted at a certain time.

What I expect to see happen over the next few years is companies trying to sell useless "security scanners" and "detectors" that make customers feel better without actually doing anything for security. Most of that will contribute more to downtime than it will to security, and we'll have lots of questions here from people who just want to turn it all off.

I do reply to Griffin
You wrote: There are really only two things that are new in this case. One is that it appears to have been used for political purposes, likely by a country who is trying to hinder Iran's nuclear industry and has no qualms about harming third parties while they are at it. I don't think this is the forum in which to debate on who that country likely is.

Its not the forum to debate but: Why nobody blames Siemens for selling technology to Iran, that helps their nuclear industry.

I don't understand about Viruses and not about nuclear industry but it seems to me that the second is much more dangerous.

I think there's two lessons to learn from this. One is that in the past a lot of people knew that this was possible, but they stuck their heads in the sand and said it won't happen because industrial control is just too obscure. Now, there have been two separate incidents where specific industrial control systems have been targeted. We can't ignore the issue anymore.

The other lesson is that we can't expect automation engineers or plant operators to be security experts. I can guaranty that the majority of the people who are or become affected by this virus won't know how to deal with it properly.

The security problem in this instance is a complex one with multiple aspects to it, but neither vendor (Microsoft or Siemens) is taking responsibility for fixing the problem as a whole. What we have in this announcement is that Microsoft is plugging one hole (out of several), but washing their hands of the matter so far as the other aspects are concerned. If you apply their patch, you won't get the virus the same way again if you don't already have it, but it does nothing for systems that are already affected. None of the reports say anything about what Siemens is doing (or if they are doing anything at all).

I believe the automation vendors need to take responsibility for handling this type of problem by offering a complete managed package from top to bottom just like the enterprise Linux distros do. By that I don't mean the solution is to stick a Linux OS under their products and call it done. I mean they need to take the same operating approach where the vendor takes responsibility for *all software* in the system, including the OS (whatever they happen to use), database, and SCADA. In this case as long as the customer pays their maintenance contract, a single source (the vendor) takes care of all these issues from top to bottom and the user just has to worry about how to apply the software to their business operations.

I don't think that's a one size fits all solution. For smaller operations it probably doesn't make sense. For very large complex systems however, I think it's the only feasible solution.

By Nathan Boeger on 22 December, 2010 - 8:23 pm

It doesn't matter if your network is isolated. Stuxnet used USB drives to infect the first nodes, then peer-to-peer attacks to spread.

----
Nathan Boeger
http://notanotherindustrialblog.blogspot.com

In reply to Nathan Boeger: Anyone who remembers the early days of MS-DOS in business will remember the numerous viruses that were transmitted via floppy disks. Networks were rare and expensive so everyone passed information back and forth on floppy disks. Anti-virus software would scan floppy disks whenever you inserted them in a drive, but that didn't prevent viruses from spreading rapidly everywhere. So, anyone who remembers those days shouldn't be too surprised at seeing Stuxnet spreading without networks. It's a throwback to the early days of viruses rather than something new. Viruses only started using networks when networks became common.

On an unrelated note, those were also the days when Apple used MacOS in the MacIntosh (this was an OS that they had designed themselves). Mac viruses mostly disappeared since Apple switched to using unix (OS/X), but in MacOS days they got lots of them. I'm not entirely sure what lesson to draw from that, but it's interesting to think about.

While isolating the network is not by itself protection against a virus, if the I/O network is separated from the supervisory network it does prevent a direct attack on the I/O by a straightforward MS Windows virus located on a SCADA or HMI PC. It also prevents a DDOS (overloading the network with messages) of the I/O.

A few years ago a nuclear power plant in the USA had it's control network knocked out by the SQL Slammer worm which attacked the MS SQL Server database. The actual *controllers* weren't infected by the virus. They just couldn't communicate with anything because the single MS Windows PC on the network hogged all the bandwidth. It took hours to get things under control, and the plant was very fortunate that the reactor had already been shut down for other reasons.

By curt wuollet on 15 September, 2010 - 7:59 pm

Here's the best blog I've seen on where the problem lies.

http://blog.linuxtoday.com/blog/2010/04/junk-cyber-crim.html

Regards
cww

The worst that can happen to Linux is to succeed.
As soon as the number of Linux installations will be considerable, the malware will get there.

In reply to MiguelS: Linux is already widely used in critical applications, including banking, stock markets, e-commerce, etc. It is in fact much more widely used than Windows in applications where real money is at stake. Large automation systems could be viewed as similar to those.

So, how does your argument that security is simply inversely related to market share fit with that? It certainly doesn't fit with what Microsoft themselves have said about such things. They claim it most definitely is possible to make a more secure OS than whatever they sold you the last time, and they always claim that each new version is more secure than the last one. As to where their OS product line sits with respect to others on security, I will leave that argument to others.

As I said before however, I don't think the solution is to just stick a different OS underneath the SCADA and call the job "done". I think there needs to be a different business model where the SCADA vendor takes responsibility for the full software stack from top to bottom rather than the endless finger pointing which we have now.

If a SCADA vendor decides their solution to this includes using Linux or BSD, then that's fine. It's not however the goal. The goal is to have a professionally managed secure platform. The present situation where the vendors wash their hands of all responsibility once they've cashed your cheque simply isn't working.

By Chris Jennings on 16 September, 2010 - 9:38 pm

> As I said before however, I don't think the solution is to just stick a different OS underneath the SCADA and call the job "done". I think there needs to be a different business model where the SCADA vendor takes responsibility for the full software stack from top to bottom rather than the endless finger pointing which we have now. <

> If a SCADA vendor decides their solution to this includes using Linux or BSD, then that's fine. It's not however the goal. The goal is to have a professionally managed secure platform. The present situation where the vendors wash their hands of all responsibility once they've cashed your cheque simply isn't working. <

I totally agree!

Remember the good old days when control system vendors used to make their entire product, from I/O and controllers to the HMI. They had their own operating systems and software that only worked with their hardware. Customers complained that it was closed off and didn't allow for competition, customers wanted "Open Control Systems". Well now we have it, but we still aren't happy.

Is it another example of globalisation gone wrong, one product that does a lot of different things but very badly? Or are we ourselves to blame by asking vendors to make their products standardised, open, compatible we are now bearing the fruit of our requests.

As technology gets more and more complex it gets harder and harder for one company to make an entire product themselves without getting something "off the shelf". I can't imagine that we can return back to the good old days, that perhaps weren't so good. But we need to put more effort in specifying our needs as customers and what we view as important. This will drive the market to make the changes necessary to meet that need.

Be it Windows, *nix, Linux, iOS or whatever they are each complex, and because of that are just as likely as each other to have errors, faults and mistakes that could cause security concerns. Market share does play a major part in the amount of malware available for a software platform. Just look at the recent issues with "Jailbreaking" iPhones and iPods, initially Apple was insisting that Viruses and security concerns don't exist for their products. Suddenly they are the market leader in something and hacks, cracks and vulnerabilities are uncovered regularly. Linux has the advantage of being open source so security through obscurity is no longer an option, if that option wasn't available to the closed source companies I imagine that we would have a much better outcome. However this would also require end users to be more capable. The dumbing down of computer software is a major concern, by making software accessible and very flexible they open to many more vulnerabilities than locked down, specific applications.

Sorry for the rant, but I lament at what has become of IT.

Chris Jennings

In reply to Chris Jennings: The newer SCADA and HMI systems are still (mainly) proprietary and are (mainly) using proprietary operating systems and databases. They're just running on cheaper commodity hardware and you know the brand of the third party OS and other components that they use.

What has happened more recently is that operating systems, databases and other infrastructure have now themselves become commodities. If switching to commodity hardware makes sense, then it makes equal sense to continue the same logic up the stack. Automation vendors could be taking advantage of this to not just cut their costs, but to change how they operate and offer their customers a managed product.

There's no reason why a SCADA vendor shouldn't be able to take commodity software (OS, database, language tools, etc.), add their own product (which would still be proprietary), and offer it to customers all on a single disk. They could then promise to support everything on that disk 100%, including security issues. This is a very successful model in critical financial systems, so there's no reason why it couldn't work here as well.

I think that someone *will* do this eventually, and the other vendors will have to scramble to match it.

By curt wuollet on 19 September, 2010 - 2:09 am

I'm sorry Michael, I just posted my reservations as disagreement with you. Now I see we are, as usual in agreement that we shouldn't throw standards and commonality away in hopes of security. Giving the vendor license to roll their own OS, for example, would be moving in the wrong direction.

Regards
cww

By curt wuollet on 19 September, 2010 - 2:04 am

This is one of the few areas where I respectfully disagree with Michael. Not because it wouldn't improve the situation for security, but because it would provoke a era of proprietary zeal such as even the automation world has never seen. Everything possible would be perverted for absolute lock-in and wrapped in the flag of security. I think that the vendors should be responsible for their solutions end to end, that's good, but that they should do it by adopting the standards and commonality that would make it manageable. So, if they need a secure OS, they should all use the same one and spend the time they would spend developing N totally incompatible ones improving the security of the one standard. And maintaining it, so that it remains secure. This could be easily done with Linux, for example, an "automationix" distribution could be tailored specifically for automation and could be made very bulletproof by exclusion of unnecessary cruft and concentrating on the real problems in industry. I feel one arrow with all that wood behind it, would be infinitely more secure than the 50 independent parallel implementations we would see each with one fiftieth of the possible effort to secure them.

Just my opinion

Regards
cww

In reply to Curt Wuollet: If you are using a proprietary SCADA or HMI, you're already locked-in to the vendor. If you want an open SCADA system, then you need to be using open SCADA software. What I'm talking about is the "appliance" model of software, which can be proprietary, open, or somewhere in between. I don't doubt that an open system is better, but that's another discussion.

As for creating an "Automationix" distro, you might want to look at something like this:
http://susestudio.com/

That's from the number 2 enterprise Linux distro. People are also doing "spins" (derivatives) based on Fedora and Debian, but those take a bit more work. EMC2 (the open CNC project) do a derivative based on Ubuntu. If you're looking for an interesting project, try making an Automationix spin and see how it turns out.

From a practical point of view, I would imagine that a SCADA vendor who wanted to make a SCADA appliance would use something similar. They definitely wouldn't write their own OS (it wouldn't be economically feasible). They could outsource platform technical support to the parent distro, but still retain the option of changing platform suppliers later if there was a business justification for it. There are companies outside of the automation field which are doing this for a while now. The automation business is simply (as usual) several years behind the curve.

and a long comes WiFi and other wireless networks ..
and device pairing...

the most secure system is one that no one can use.

I'd be interested in any justification for that statement. It comes straight from Redmond, but even they don't _believe_ it. Check out what the NSA, people who know, did for Linux as a platform for various govt. agencies when security is an issue.

Regards
cww

An analysis of this virus has been published by a security consulting firm in Germany, and the results are very interesting.

http://www.langner.com/en/index.htm

The summary is the virus appears to inject S7 code into OB 35 (timed interrupt), presumably to disrupt a process.

The article's conclusions are even more interesting. The virus appears to be targeted at the Iranian nuclear industry, and is the product of a country who has a strong interest in monitoring and disrupting that program.

The virus infection probably entered the system via an integrator (who was not aware that his own computer had this virus) who also does a lot of business in other the other named countries which were also affected. These secondary sites were not the actual targets however, and tertiary infections spread from there.


The following is my own conclusion. If I had to guess, I would suspect that the integrator was targeted by sending him an e-mail with any one of numerous viruses that target Outlook. The virus would then remain dormant on his laptop until he was present at a customer site and connected to their control network, at which point the virus would then cross to the (MS Windows part of the) control system.

Someone who wants to target a control system does not have to develop a new virus. You can purchase these off the shelf from the thriving MS Windows virus industry. If you are willing to pay a premium, you can get the first use of a new virus.

Now what you do is you pick your target. You find an engineering employee in that target company our you find an integrator who does a lot of business with that company. Next, you send an e-mail to them with an attachment (e.g. "see the attachment for the new project spec"). If you are using a new virus, your virus scanners probably won't block it. The virus then runs and installs itself in his computer.

Now, at some point that employee or integrator will need to do some work related to the target control system and the virus can do whatever it is that it is intended to do. If you have a SCADA or HMI system that uses MS Windows, it can install itself on that system. If you are just using PLCs, it can read the PLC memory or inject code into the PLC program. As to why someone might want to do that, well I suppose there could be lots of reasons.

This is an interesting example of how security is no stronger than the weakest link. You can have all the wonderful firewalls, virus scanners, and network isolation you want, but you still have to connect that laptop (or USB key) into the system if you hope to do anything useful with it.

Thank you Michael for continuing to feed us examples of security incidents and further information. I have stopped actively contributing to these discussions (I am bored of repeating myself) but appreciate the intent.

David

By the other David on 22 September, 2010 - 8:46 pm

For those interested in Siemens' response to StuxNet here's the Siemens page

http://tinyurl.com/23zb62k

By curt wuollet on 23 September, 2010 - 4:21 pm

That is amazing, and it sounds like Siemens is full on scrambling. If it does what they say it does, they _should_ be. They are extremely lucky that this was apparently a targeted attack. If the virus can load blocks into a running program, it's not hard to imagine making a general purpose Simatic killer if they opened up the trigger conditions. I'm sure someone could come up with a block, that when loaded, could hose almost any running program or do other malicious mischief. Even done blindly, that could cause tremendous mayhem and damage and maybe even kill people. At least they shared the blame.

Regards
cww

Siemens is saying that they know of 14 control systems that are affected. Other sources are saying 45,000 computers are affected. Although this virus seems to be looking for a very specific target, it is quite capable when it comes to actually getting on computers.

It is possible to imagine someone creating a variant of this that went after a popular brand of gas turbine control. Someone could get the viruses in position and then sell short electricity prices on the spot market for a specific block of time. They could then trip several power plants and watch the money roll in as the price of electricity rises. If they were careful enough not to trip too many plants at once, the plant operators would probably just write it off as an unexplained trip and not investigate it further.

There are probably other ways to manipulate spot prices for oil, natural gas, and other commodities. There is a *lot* of money to be made doing things like that then there is operating spam bot-nets. I will be surprised if someone doesn't take advantage of it.

By Rahul P Sharma on 25 September, 2010 - 9:21 am

Does the virus remain confined to the PC running windows or it can also install itself in the PLC Program and affect the sequence logics....??

The virus installs itself on the PC using security holes in MS Windows to get in. I believe it actually exploits 4 different known MS Windows security holes so it has alternate means of spreading if some methods are blocked.

Once the virus is present on the PC, it looks for the PLC and then downloads new program logic into it (in OB35). We don't know what that new logic does to the production process, but this is the reason why people are saying that it is targeted at a specific plant (because it is modifying some specific logic).

If you follow this link you will see what the actual PLC program changes are:
http://www.langner.com/en/index.htm

It's quite easy to imagine new viruses following the same pattern in future. If all someone wanted to do was to fault the processor, there are many ways to do that without needing to know anything about the equipment being controlled. Debugging a situation like that would be extremely difficult for a plant operator. They would reload the PLC program, replace hardware, replace IO, and do practically everything else before suspecting the PC was injecting code into the PLC.

Now all we need is a very smart highly trained individual to carry out the scheme below. Oops, you would also have to be hungry and immoral.

But if you are the first you are not hungry, and probably much too busy making money to consider immoral methods.

I know I am, hope you are too.

Rob

In reply to Honders: There are plenty of people around who do this sort of thing already; they just haven't been doing it to PLCs. Up until now people were saying "it can't be done". Well, it's been done, so that excuse is now gone. There's a thriving market in MS Windows viruses right now. Just bring a few of those guys together with people who know something about PLCs and creating this would be easy.

Modern viruses are written to make money. They're used for fraud, extortion, spam, and no doubt many other things as well. As soon as someone can figure how to make money by temporarily knocking out a power plant, pipeline, or anything else, it will be as common as spam e-mail (it is estimated that more than 85% of all e-mail is spam, most of which is sent by computers that have been taken over by viruses).

There's more news on the MS Windows virus which is attacking Siemens control systems. The BBC is quoting Iranian news sources as saying that they have detected the virus at 30,000 IP addresses. Since more than one computer can be at one IP address (due to network address translation), there may be many more affected computers than this.
http://www.bbc.co.uk/news/world-middle-east-11414483

The virus is also affecting India, Pakistan, and Indonesia, and has also been found at a number of other places around the world.

AFP is quoting American sources as saying the virus has been found affecting water treatment plants and chemical plants which also use similar Siemens controls.
http://www.google.com/hostednews/afp/article/ALeqM5h4PIBLHvZzzlyUTBOP7U_evSK2wQ

The virus is apparently still quite active. Microsoft has patched 2 of the 4 vulnerabilities which it can use, but there is no word yet on when the remaining vulnerabilities will be fixed.

Iranian sources have said they have seen 5 different versions of the virus so far.

I'm trying to get a handle on the software technology that would be used to communicate with a PLC behind a firewall. I know zilch about Siemens PLCs, so I'll muse about what would be needed to access a Modbus/TCP speaking PLC (Schnieder, Control Microsystems, Sixnet, e.g.)

I know three ways to do this:

1) allow access to a Modbus/TCP PLC intentionally by opening a port on the router and directing Modbus/TCP traffic (default port 502) to that one PLC or one Modbus gateway.

2) use a VPN.

3) install Hamachi (or the like) on one of the computers inside the plant network.

If anyone has a 4,5,6, etc., let me know. Since I don't, I'll bet the Siemens attacks are based on a Hamachi-like technology, where all TCP connections are outbound. Firewalls filter URLs but don't generally stop outbound TCP connections. There is usually nothing that prevents a pair processes, over the Internet, from making a network based on outbound TCP connections only.

I'm not complaining about Hamachi. I like it. It is an open source program, so the cat's out the bag. Logmein is managing Hamachi, primarily for Windows platforms. Ready-to-run Linux versions of Hamachi are also available.

Someone builds a fence, someone else figures out how to climb over it.

By David Ferguson on 27 September, 2010 - 7:27 am

How about a giant wooden horse as a gift, Engineer takes laptop home, it gets infected with auto-running code. Goes back to work connects behind the firewall to the PLC, virus loads and code finds PLC and modifies an OB code (OB35 ?) program and machine crashes. Bets on that market (as has been said) and makes money from stock changes or his companies code............

In other words, who says it needs to be connected to the internet ? We are very protective of the whole firewall internet ting, but in reality I have played that cat and mouse with Interns who know more than we can keep up with each summer. As has been written over and over, there are many ways to attack Troy.

Dave Ferguson
Control Systems Engineer

Tofino have a firewall which has deep packet inspection for Modbus TCP see URL below. Might be worth a look

http://www.tofinosecurity.com

Dave MH

So what? Did you realy mean all that virus/pc-stuff will never affect the plc side? It was IMHO just a question of time. We all wanted the overall network stuff, the everything connected to everyone stuff. Go and face it that this (...) will be the future. Wake up! Learn both sides (pc & plc) to be ahead. Kick the pc-network-admin and dont take every gadget on the plc-side for good. Less could be more in the future :)

The problem with giving generic advice on this sort of question is that there isn't a simple answer to give. Network security varies from place to place, and what works in one application won't work in another. The other thing about it is that security that isn't monitored and kept up to date isn't much use.

You said the Modbus/TCP network is "behind a firewall". Is that a firewall for the entire site (including the business offices) and you want to connect to the PLC from any place on the Internet? Or is that an internal firewall, and you just want to isolate the control network from the business operations? Or are you trying to do both?

For accessing a site from somewhere else, you really need to talk to and work with the people responsible for the IT network. They are (hopefully) the professionals who know how things are already set up and can advise you on how to work with their system. It won't do you any good to find a solution and then have them make a change which negates it a week later.

For an internal firewall (separating the plant from the business network), someone has already mentioned Tofino. I haven't bought their product, but the overall design sounds good. You would need to talk to them however to see if it does what you need for your application.

From what I understand about it, Tofino runs Linux and uses the packet filtering which is built into the OS (iptables). Most professional grade security appliances (firewall boxes) work the same way, so this part of it sounds pretty standard. They then add some software to make configuration easier (you can configure iptables with just a text editor, but it's not easy). They also add some industrial network specific packet filtering to make sure that for example the Modbus/TCP messages are really valid Modbus/TCP messages and not garbage that is meant to trigger bugs in your PLC's firmware.

If you want a do-it-yourself solution, you can set up more or less the same thing using Modbus/TCP netfilter extensions (you can download these from Sourceforge). Be prepared to do a fair bit of work if you plan to set all this up yourself however.

As for the Siemens attacks, the virus gets onto the network via USB sticks or by being transported via someone's laptop. At some point you need to program or configure your equipment, so there really isn't a good way to avoid this risk. Once the virus has access to the network it spreads through the network itself. It is using multiple holes in MS Windows, not all of which are patched yet. Once it is on the same network as the PLC it can re-program the PLC by using the normal PLC programming commands.

When this virus first started spreading Siemens was taking a lot of flack for their use of hard coded passwords to access the database they were using MS SQL Server. However that flack was coming from the traditional MS Windows security industry, which tends to think in terms of databases being the target. Instead, the main target of the virus was the PLC. The virus can do that by simply using normal programming commands. I don't think there is much that Siemens can really do to improve security so long as they are using an OS that is so notorious for having security holes.

By bob peterson on 27 September, 2010 - 6:06 pm

It seems to me that a lot of this could have been avoided if people left the cpu keyswitch in a position where program changes are disabled.

To me that is an inexcusable, and has absolutely nothing to do with any "gaping" holes in MS products.

It's like blaming the burglar because you are too lazy to lock your front door. Yea, he is still responsible for his actions, but so are you.

I would not be getting too much pleasure out of this for the MS haters. If the "free' OS's ever start to replace the ones you have to pay for, it is almost a certainty that the hackers will start to attack them, and they will have every bit as much success with them. It's just about volume. There is just no reason for the hackers to go after the other OS's.

By David Ferguson on 28 September, 2010 - 7:27 am

While I agree with you about the keyswitch, that will not solve anything, for as soon as you move the keyswitch to program.............the virus which has been laying in wait, incubating, sees the switch change when you actually need to make a ligitimate change and infection happens.

Avoiding the illness is the issue, not getting infected when everyone around you is sick is another.

Also the OS in this case is just the delivery system, the virus is actually changing the program of the PLC, this is a little different for it is a targeted virus, it is not attacking a vulnerability of the OS itself (directly) but using it as a trojan horse.

And oh by the way, not all plc's have a "keyswitch".

Dave Ferguson
Control Systems Engineer

By David Ferguson on 28 September, 2010 - 7:30 am

I also am not going to talk about this on here anymore, for the very writers of this virus are probably monitoring our answers to make the next one better.....

Dave Ferguson

Hello All,

If you want to see what the future holds for control system security take a look at...

http://iase.disa.mil/stigs/index.html

The Department of Defense has already done extensive security studies on everything from basic workstations, networking, servers and put out Security Technical Implementation Guides (STIGs). These guides show you where to plug potential security threats and are extensive. Following is the STIG for the Windows Operating System...

http://iase.disa.mil/stigs/content_pages/windows_os_security.html

For example, in the Network STIG, it is a requirement to disable unused ports on all Ethernet switches, hubs, routers. So these STIGs are more than just software based but also implement physical security policies.

Having done a control system project with a government agency these STIGs are required to be implemented and took quite a bit of time to review and see how they would impact the functionally of the control system.

Based on last weeks news about a control system being infected, this is going to force us control engineers to start looking hard at security when developing these systems. The security standards that will be mandated are most likely coming from the STIGs put out by the DoD.

In reply to ChiefGeek63: Those are probably useful guidelines for anyone who has the US military as a customer, and many companies will have similar policies on how they want their computers set up. However, those are simply basic security guidelines intended to make sure that normal users don't have more access authorisation than they should. The MS Windows permissions system is very convoluted and inconsistent, so this can be difficult to get right if you have a large and complex network.

However useful those documents might be to someone setting up a new computer (and I'm not saying they're not useful for some people), that is orthogonal to the virus problem. A virus simply goes around all the security and in through any one of multiple holes in the security fence. In other words, you can set whatever security settings you want, but that has no effect on the virus.

The latest news where a virus can use a Step7 PLC program to spread itself is particularly serious. A Step7 project apparently isn't just a simple data file. What makes is so serious is that PLC programs are very valuable (they run your plant), and you need to keep them for the life of the equipment. They also by their nature tend to be something that gets passed between different parties from different companies. What's more, integrators deal with a *lot* of customers, so the risk of virus infection for them is very high. Once a virus gets established it will keep coming back from old PLC programs from various sources.

What is more, even if you can run a virus "cleaner" program on the PLC program, how do you check if the PLC program was corrupted (unintentionally) by either the virus or the virus cleaner program? The only way to be sure would be to actually check each and every program against what is known to work in the machine (and hope there are no logic time bombs already present). That sounds like a lot of work for a large plant.

There are a lot of difficult questions being raised by this situation, and I will be interested to see if anyone comes up with any real answers for them.

By curt wuollet on 29 September, 2010 - 3:24 am

That is quite interesting reading. The Linux section implies that it's pretty secure out of the box with most actions: "Ensure that xxx is xxx". I think that it is extremely important that the defaults be set for maximum security as real world installation tends towards, run the CD and go. I know they tried this with Vista and people hated it, but shipping wide open, then blaming the user is unrealistic and inexcusable. If you ship secure and people have to open all the security holes to get their favorite features (like auto executing files on a USB drive), then you can blame the user. It would also require application vendors to state which relaxations are needed to run their blob and provide some incentive to minimize these. I don't know if Microsoft could be persuaded to do this after Vista, but it would be no problem for an "Automationix" distribution of Linux to be very secure (full SELinux) out of the box. The vast majority of security breaches could be avoided with just this simple and logical measure. Even this breach seems to exploit old and well known weaknesses that could be closed by default.

Regards
cww

By curt wuollet on 28 September, 2010 - 1:58 pm

Hi Bob

I don't think anyone is getting any pleasure out of this. I'm with you on the keyswitch thing, but people do want to be able to edit remotely and they probably will leave the PLC set up for that, regardless of whether they should. As I said before, I, the US Army, the NSA, and many others would be interested if there were the slightest shred of evidence that Linux is as permeable as Windows. For the time being, I think I'll go with their risk assessment. And a better analogy might be blaming the burglar because the safe wasn't locked and the front door lock has been broken for years.

Regards
cww

In reply to bob peterson: If you look at the rest of the paragraph in context you will see that the point I was trying to make was the story as originally reported in the press was focused on the wrong aspect. In the original news reports Siemens was taking a lot of flack over using hard coded database passwords. While that may be a problem, the database wasn't in fact the primary target for the virus. It was only much later that the press starting reporting the PLC re-programming angle.

Most industrial control systems are based on the premise that all the elements that make up the control can "trust" each other. That is, they assume that the PLCs, servo drives, I/O modules, SCADA and HMI systems, etc. won't deliberately do anything malicious. That assumption applies to all the normal vendors, not just Siemens. I can't in fact think of a vendor which does *not* assume this. So, an important point to make here is that while Siemens may have been the target in this case, users of systems from every other vendor face exactly the same problem. This is why I haven't emphasised the "Siemens" aspect of this story. It's an industry wide problem.

The problem is that in such a situation, once a PC has been taken over by a virus, the virus author can do virtually anything that a "legitimate" user can do. There are *many* things other than reprogramming the PLC that such a virus could do to cause a serious problem.

Give those circumstance, then as I said there is really not much that Siemens (or any other vendor) can do to prevent this so long as they are relying on a third party OS (MS Windows) to provide a degree of security which historically it has never been able to. The point here is not to bash Microsoft (however much they may deserve it in this case). The point is that given that industrial control systems are based on trust, the security of the system as a whole depends on MS Windows PCs not getting viruses.

Up until now people have been ignoring this problem by saying that virus writers would never do this because PLCs are just too obscure for them to notice. Well, now they *have* done it, so that excuse no longer holds. It's something that we all have to think about.

There's more developments in this story, and this looks like the worst news yet. According to Symantec (an anti-virus vendor) the virus will directly infect Step7 project files (backups of the PLC program).

http://www.symantec.com/connect/blogs/stuxnet-infection-step-7-projects

What happens is the virus scans your disks for copies of Step7 projects (PLC programs). When it finds them it puts a copy of the virus inside. When you attempt to open the project file again later, your computer gets re-infected with the virus. There is no fix for this at this time.

This means you can't trust any Step7 backups you may have. You can't trust any Step7 PLC programs that you get from anyone else (e.g. equipment builders, third party integrators, etc.). Opening any of them can spread the virus.

I think this explains why the virus has spread so widely and so quickly. I don't think we've seen the last of this.

> This means you can't trust any Step7 backups you may have. You can't trust any Step7 PLC programs that you get from anyone else (e.g. equipment builders, third party integrators, etc.). Opening any of them can spread the virus. <

That's the point. In the past we had PGs, just for that reason, ugly, heavy and slow. No shiny PC with browsers, mail, entertainment and internet.

Roll back!

Keep the Step7-box just for programming and all is fine. Close projects and let MD5 seal it.

Use the keyswitch at the PLC.

Use MD5 always.

The "virus" was successful because of laziness. You maybe care about becoming infected by using stuff from usb-sticks but none of you thinking the same way by using the cable for the network.

Safety is a permanent process and it is hard work, anytime and anyplace.

Numsi said:

> That's the point. In the past we had PGs, just for that
> reason, ugly, heavy and slow. No shiny PC with browsers,
> mail, entertainment and internet.
>
> Roll back!

While I mostly agree with this, some of us (maybe even many of is ??) are in situations where we have to store or backup our files to a shared fileserver. Most of the time these shared resources are open to office computer access.

Somehow spending my whole day on a computer editing PLC files in a "black box" programming unit and then having a regular computer next to it that is hooked up to the mothership is unreasonable. Do you have all of your "programming units" backed up regularly and store copes of these backups off site in case of a fire?

Even if you could pull this off your network administrator and fellow employees would think you are crazy, even though you would be mostly right to work in this way!

I think the automation, business, and "office" all need to be on separate secure physical connections and bridged only by tightly managed secure switches/firewalls. Unfortunately many small and medium sized companies don't have IT staff that is trained well enough to analyze and support these cobbled together manufacturing situations. For years companies have been busy trying to tie their automation into the business backbone and now we are suffering.

~Ken

"While I mostly agree with this, some of us (maybe even many of is??) are in situations where we have to store or backup our files to a shared fileserver. Most of the time these shared resources are open to office computer access."

I see no reason they couldn't be archived or encapsulated in some way the virus isn't prepared to deal with. Simply ZIPping them might be adequate. ZIPping with a password or encrypting the entire application would be other steps that should be increasingly confounding to the virus.

"Even if you could pull this off your network administrator and fellow employees would think you are crazy, even though you would be mostly right to work in this way!"

That's an odd statement. So if you're doing the right thing and everything thinks you're nuts, you should stop doing it? Sounds like THEY'RE the ones who need to adjust.

In reply to Steve Myres: You said: "I see no reason they couldn't be archived or encapsulated in some way the virus isn't prepared to deal with. Simply ZIPping them might be adequate."

This virus already looks for PLC program files in ZIP archives and infects them there. The answer's not that simple.

Once a virus is established in your computer, it can do literally *anything* the computer is capable of doing. It can do things like insert itself into the operating system so that if you attempt to look at a file to see if it has been changed, it can just intercept that operating system call and show you innocuous looking data. Some viruses do exactly that.

There are literally tens of thousands of different MS Windows viruses. If there was an easy answer to them, the people running office IT systems which use MS Windows would be doing it now. They're not, so there probably isn't any answer that doesn't involve a complete re-design of MS Windows from the ground up, and if Microsoft did that, then probably none of our existing MS Windows software would run.

If there's an answer here, it isn't going to be quick or simple.

This virus already looks for PLC program files in ZIP archives and infects them there. The answer's not that simple.

OK, but that was the LEAST aggressive option I mentioned. If the archive is stored with decent encryption, I think either the extension or file format (if the virus directly examines the file) would be likely to cause the virus to overlook them, and if not, the encryption should prevent it from infecting the archive.

That still leaves open the question of how to secure against the virus when it's converted back to native form when you have to use it, but that can be viewed as a separate issue, and an easier one than if its solution had to deal with security during backup storage.

Your Virus could just corrupt your super secure archived project on a file server.... In other words it couldn't infect your project file, but destroy it.

KEJR

Ken,

When you figure out exactly what you think the hazard vector is, state it and we can discuss it.

In the meantime, you're correct of course that the virus could destroy the archive rather than subtly corrupt it, but a destroyed archive is orders of magnitude less dangerous than an altered one. Besides, if the virus is incapable of recognizing the archive and damages it anyway, that means it's attacking files randomly whether automation-related or not, which is not a new or unique hazard. Generic virii have always had that capability.

Look, I'm not defending Windows per se, in fact would probably prefer Linux-based programming tools, but if you're going to raise a hypothetical, think it through well enough that if I respond to it, you're first response isn't "But that doesn't address this other issue!" (It wasn't intended to. It was a response to what was posted)

By Ken Emmons Jr. on 1 October, 2010 - 8:06 am

Steve,
I know there have been viruses taking out data for many years. If you are the guy who is in charge of getting your backup restored it doesn't matter if the archive is destroyed or altered, you still don't have a backup. Up until know I've only heard of viruses taking out the common file names like .doc, .jpg, or trying to currupt your entire hard drive.

Curt: I agree with your assessment of the Linux versus Windows debate in terms of security. I have also seen a trend in Linux lately where some newer distributions are trying to be too user friendly and sacrificing on security. Things like asking you if you want save the root password for package managers, etc. Hopefully Linux doesn't become too "friendly".

Having said that I just got a brand new Windows7 laptop from Dell the other night (Getting an UBUNTU laptop from them costs more apparently...) and I was unimpressed with how they went through all the great care to set up a user account and didn't even care to set you up with a separate administrator account. With all the alleged security updates for windows 7 they still set you up by default to be an administrator (without asking you). Sure you can go back and change it, but most people don't know this or care.

~KEJR

By curt wuollet on 1 October, 2010 - 4:12 pm

Let's pretend for a moment that the industry had finally had enough of the Windows insanity and wanted to actually pursue bringing forth an automation oriented OS. They could take an existing high quality Linux distribution (it's OK, everyone does this) although there would be vigorous debate about which, and retain a few good used Linux folks who know automation, (I'm available:^)) and before we're to service pack 3 on W7, they could have a really good target to port to.

If they were respectful of the Linux community, there would be an outpouring of help for such an important (to the community) development.

The up front cost would be minimal, insignificant in corporate terms. There would be costs for the extra security to head off the flood of MS reps offering discounts, bribes, gifts, threats, etc. once the word got out. But, since Linux is pretty close to optimal for customization, there wouldn't be any extensive code work or rewrites needed. They could then port their apps once.

This would not be trivial, but would be well justified by the vastly decreased cost down the line, since _they_ could control the upgrade pace. That would be a direct benefit. The indirect benefits would be stellar. If they want a protocol supported, a bounty that that wouldn't buy a customer golf outing would cause it to appear. A security issue like the subject matter, could be solved _fast_ by application of funds that wouldn't even rate a call back from Redmond. Instead of being surprised by new interfaces and features and having to write to them, they could request new features that make sense for automation.

Or they could continue using whatever version makes sense for them. It would very likely have Modbus and Modbus/TCP built in, rational support for all the serial variants, support for any number of serial ports. and they could very likely use the same distribution for embedded panels and network gadgets of all sorts.

At this point it would make sense for any vendor who sells boards r devices into the automation market, to develop drivers, and if they insisted, they would be Open drivers so they could be well understood. There would be extremely easy choices in the UI to differentiate and apply branding and having the whole *nix array of tools for scripting, programming, glue, etc., would preclude enormous amounts of one shot integration and programming.

After about 2.0, it would be an unstoppable force and nearly bulletproof and we could all get back to automation solutions. It could easily meet Michael's criteria where the whole ball of wax comes on one CD and configured for _real_ security. I'm surprised they haven't done this simply for the cost/benefit of knowing what environment you will be running on and not supporting the hundreds, if not thousands of significant Windows variations they have to worry about now. It would be a much nicer world, once you learned Linux.

Regards
cww

In reply to David Ferguson: To correct something that you said, the PLC was not being "hacked". It was being re-programmed using the normal PLC commands which are present for that purpose. The computer however was being "hacked" by the virus, via security holes in MS Windows.

There is a very important difference between the two situations. There are many tens of thousands (hundreds of thousands?) of viruses which affect MS Windows. Viruses which affect anything else are extremely rare. While this particular virus is quite clever, the same sort of attack on PLCs (any brand of PLC) could be repeated using any run of the mill MS Windows virus. New MS Windows viruses come out all the time, so anyone who felt motivated could repeat this feat quite easily.

And to address another point that you stated:

> switching the OS will not change a thing, if
> they want in, they will get in, even if that
> means walking in and accessing the machine with
> the Linux software...........IT IS NOT THE
> SOFTWARE, it is the fact that there is software
> at all, instead of hard coded HARDWARE.......

Well, if as you said you're granting physical access, then they can indeed reprogram the hard coded hardware just as easily as they could the software. Of course they could just smash the machine up with a sledge hammer in that case too.

You can talk about all sorts of hypothetical situations, but I like to look at things from a practical point of view. Let's look at what is happening in the real world here. We don't have the Mission Impossible team rappelling through the skylights to plant a bug in our PLC programs. We have an MS Windows virus just like thousands of other MS Windows viruses except this one re-programs PLCs instead of stealing information or sending spam. It's just taking advantage of bugs and holes in MS Windows, just like all the other viruses.

From a practical point of view, we don't need to solve the global MS Windows virus problem. We just need to find a way to avoid being directly affected by it.

By David Ferguson on 2 October, 2010 - 4:46 pm

That is my point, a trojan horse is being used to get in but once in, it is the PLC that is being modified "hacked" (using the tools to modify the PLC program) if that makes you feel any better. I agree potato or potato........

As to Curt's point, you can lock down Windows just as secure if you want to and you should.......we do. If you install it straight out of the box then you get what you deserve, but then again everything said above is my point exactly.

As to the age old WINDOWS rant ................. you get out what you put in, if you leave things wide open and plug them into the internet and let them play solitaire on your control boxes..........you deserve to get HACKED.

Dave Ferguson
Control Systems Engineer

By curt wuollet on 3 October, 2010 - 1:48 am

I'd like to be a fly on the wall when you tell your customer they deserved to get hacked. And I'd like a show of hands of all our gentle readers as to how many of them could install Windows tonight, and get it locked down properly, up to the latest threats, betting their job upon it?

I think distributing a secured system is a much better bet. A purpose built OS without thousands of published exploits would seem much less likely to be compromised by any rational analysis, but as I've mentioned before, the defense of that decision is anything but rational. And simply _not_ leaving security up to chance or the luck of the draw as to who installs the stuff seems like a more reasonable choice if you want any degree of confidence. But, in the real world, the way it is actually being done, is obviously not securing anything.

Controlling a large portion of that process and eliminating the variability would seem to be hard to argue against objectively. But then, sticking with the most penetrated OS in existence must make sense to someone.

Regards
cww

By David Ferguson on 2 October, 2010 - 7:50 am

Curt:

Nor will they know or care about Linux security, we are a product of our own demise...........if you make it complicated or if it requires them to RTFM (read the F***ing Manual) then this is what happens. Our jobs get hard as we are expected to know all of this (including Windows security, or Linux). As we have less and less people doing more and more, as I like to say the memory bank only holds so much and there are fewer and fewer good control people out there let alone good ones with IT security background.

But lets not kid ourselves, THEY ARE ATTACKING THE PLC they are getting there via a wooden horse (Windows) but they are getting there because the guards are sleeping and the very same people you say will configure Linux to be secure etc will be a short in demand as the ones who know how to truly lock down windows. In other words there are not enough good guards out there anymore. There are not 2 or 3 for every company in America (let alone the world).

It would be just a matter of time until the same determined person would get to the PLC. It is also just a matter of time before someone with a little knowledge, opens up all your security holes because they are tired of getting phone calls (just like Windows).

The only thing your plan would do is to DELAY this happening again, but the huge outlay of effort to switch over would cost the country billions. I say train people to secure what we already own. We won't do it because we are lazy but it takes issues like this to bring it to a head.

The issue is if we go out and switch every company and every software vendor to Linux based OS with much pain and cost (yes it is more than the cost of the OS, it is the time to rebuild every machine and the downtime and resources which do not exist), then someone will hack it because it is now a bigger target. We would have SP's to fix the patches. We would complain that it was not accessible and we would have to open it up for accessability and after 10 - 15 years (if you could pull it off in that time), we would be on this list complaining and calling to use Blunix or whatever was the latest patch.

But we would be too stupid to put 2 and 2 together (goldfish memory) and realize we had not addressed the real issue. Do not hook your control system to the Internet without DMZ (and I say do not at all), do not allow access to your control system boxes (KVM's extenders and locked control rooms). Hire back some of the people you laid off to stay on top of securing your system and generally do the right things..........

Like they say the only secure way to prevent access to a computer is to shut it off.

The answer to 10 problems is not 11.

Dave Ferguson

In reply to David Ferguson: You said:

>the very same people you say will
>configure Linux to be secure
>etc will be a short in demand as the
>ones who know how to truly lock down
>windows. In other words there are not
>enough good guards out there anymore.

I think you're missing the point. Of *course* the average *user* won't and can't know how to secure a computer properly. That's what people have been trying to *tell* you. However the *vendor* can hire the necessary experts to figure out how to do this *provided* the vendor is supplying the complete OS and application stack on the same disk as their SCADA/HMI package.

You would stick the disk in the slot and press the go button and 15 minutes later it would have installed everything you need for a secure SCADA or HMI system. You wouldn't configure anything except your SCADA/HMI screens and tags. That's how high security computer systems work. If the vendor has to re-compile the OS with some special compiler setting to get "super security" mode, that's *their* problem. And if the resulting system won't run iTunes and 15 year old accounting packages, nobody cares. It's an appliance. It has a particular purpose in mind so the vendor doesn't have to make compromises to keep the video game and Facebook fans happy.

The SCADA and HMI vendors can't do this under their existing model because they aren't supplying the complete software stack (OS, libraries, development tool chain, databases, etc.). Their customers are buying PCs at stores or from on-line retailers with MS Windows already installed and configured any of hundreds of different ways with different MS Windows versions, different libraries, different Dot Net frameworks, etc.

There's no standard system they can rely on so they can't make any assumptions about the way things are configured and they know that if they start meddling too much with the configuration settings in MS Windows the whole thing might stop working. The result is that the responsibility for securing the system gets left up to the end user, which usually means it doesn't get done at all.

The only way to reliably do a proper job of computer security is to completely wipe the hard drive and install a system that is closely tailored to the application and carefully tested. Trying to tweak random video game / Facebook computers into high security systems is a waste of time.

By curt wuollet on 3 October, 2010 - 2:00 am

That is pretty much the point, yes. And even if you can't go all the way
to the appliance level, simply using an OS with all the consumer crap deleted and all unnecessary services disabled would be orders of magnitude better than depending on the average user, who is often the problem.

Regards
cww

By curt wuollet on 3 October, 2010 - 3:40 am

Hi Dave

DF: >Curt:
Nor will they know or care about Linux security, we are a product of our own demise...........if you make it complicated or if it requires them to RTFM (read the F***ing Manual) then this is what happens. Our jobs get hard as we are expected to know all of this (including Windows security, or Linux). As we have less and less people doing more and more, as I like to say the memory bank only holds so much and there are fewer and fewer good control people out there let alone good ones with IT security background.<

The idea is that they don't have to care about security, it's out
of their hands. And as for complexity, the application would be all
they have to worry about. With a dedicated automation OS, you can
set it up so they never even know what OS it's running on. All they
see is the application, you can't get much simpler than that. And I
can probably personally suggest enough people with the required expertise to cover the automation vendors. That's the whole advantage of securing it at the source. You would never know that your Android phone is running Linux, and it doesn't seem to confuse anyone.

DF: > But lets not kid ourselves, THEY ARE ATTACKING THE PLC they are getting there via a wooden horse (Windows) but they are getting there because the guards are sleeping and the very same people you say will configure Linux to be secure etc will be a short in demand as the ones who know how to truly lock down windows. In other words there are not enough good guards out there anymore. There are not 2 or 3 for every company in America (let alone the world).<

As I've said before, I think any "always on" connection between your control systems and the world is simply not worth the risk. But, since people will do this anyway, a secured system is even more important.

DF: >It would be just a matter of time until the same determined person would get to the PLC. It is also just a matter of time before someone with a little knowledge, opens up all your security holes because they are tired of getting phone calls (just like Windows). The only thing your plan would do is to DELAY this happening again, but the huge outlay of effort to switch over would cost the country billions. I say train people to secure what we already own. We won't do it because we are lazy but it takes issues like this to bring it to a head.<

A more reasoned argument, but I think the delay has already occurred. It's time to do something different and it will not happen given the direction Windows is taking. It's simply the wrong direction for automation use. I am reminded of Deming's definition of insanity. That's where you keep doing things the same way and expect the result to be different.

If past history is any indication, that change is not coming from Redmond. And users, by and large, don't seem to be changing much either.

DF: >The issue is if we go out and switch every company and every software vendor to Linux based OS with much pain and cost (yes it is more than the cost of the OS, it is the time to rebuild every machine and the downtime and resources which do not exist), then someone will hack it because it is now a bigger target. We would have SP's to fix the patches. We would complain that it was not accessible and we would have to open it up for accessability and after 10 - 15 years (if you could pull it off in that time), we would be on this list complaining and calling to use Blunix or whatever was the latest patch.<

I suspect the cost would not be much more that the cost of all the antivirus measures and labor to try to plug the leaky boat when offset from savings on the software and elimination of the upgrade mill. For the vendors to have control of the OS would save vast amounts of time and effort riding the MS merry go round. It wouldn't
take many "upgrades" eliminated to pay for the whole process. And everybody saves that way.

DF: >But we would be too stupid to put 2 and 2 together (goldfish memory) and realize we had not addressed the real issue. Do not hook your control system to the Internet without DMZ (and I say do not at all), do not allow access to your control system boxes (KVM's extenders and locked control rooms). Hire back some of the people you laid off to stay on top of securing your system and generally do the right things.......... Like they say the only secure way to prevent access to a computer is to shut it off. The answer to 10 problems is not 11.<

The problem with that is that, while we aren't looking the PLC is becoming the computer, That will continue and speaks even more strongly for a uniform OS that can run on the PCs and PACs across architectures and is tailored for automation. Switching now could make that happen seamlessly. I would be surprised if someone, somewhere, isn't already designing a PAC using Android or Embedded Linux. It's simply the easiest way.

Even I can do that :^)

Regards
cww

By David Ferguson on 3 October, 2010 - 6:55 pm

Well I guess we will just go into year 12 or 13 or 15 and see who is right. I am guessing that we will not do anything about this issue. There are rumors out there that this looks to be too good a virus to be other than being done by a "nationality" state sponsored event. My guess is if that is the case...........no system would be safe.

I think you oversimplify the solution and all of the issues with hardware etc. but then again we are sitting at this babble for over a decade and here we still sit......the loud minority still has not convinced the silent majority to change.......

Remember the huge expense you say is coming from the Redmond extortion upgrade mill is a very, very small part of the projects that I do. I am lucky if the control system is 1% of the project............so getting someone to take on a risk or a change from "acceptable" risk won't be happening..................after all how is the acceptance coming for the "open" plc, many posts, no open PLC, no open HMI yet?

Not going to happen in my career time, doesn't even hit the radar of the people I work for.

But before we argue this we should know what is really happening with Stuxnet, not what is being thrown out in the media and on this list as FUD........

Dave Ferguson
Still making a living solving problems with the currently accepted methods !!!!!

By Ken Emmons Jr. on 4 October, 2010 - 8:36 am

Curt,
I kind of know where you are going with this secure linux thing, but to go around saying it will be cheaper to replace all of our PLC systems is a bit odd. Perhaps starting from scratch (where it is appropriate) with a new SCADA/HMI type system similar to Michael's MBLogic platform would be good for *new* projects, but we still have existing apps to support.

Plus there are applications where standard linux is not appropriate (RTOS and motion).

CWW said:
> I suspect the cost would not be much more that the cost of
> all the antivirus measures and labor to try to plug the leaky
> boat when offset from savings on the software and elimination
> of the upgrade mill.

A lot of my machine programs take man months to complete/debug due to their complexity and I'm sure I am not alone. It would be someone's full time job to retrofit my machines and it would take years and a lot of down time to accomplish.

Changing to a secure linux out of the box might be a choice for some that have one system doing some simple sequencing and monitoring but what if you have dozens of existing systems across your plant (or customers' plants)? If things really got that bad (and they might) I would ask our IT guy to give me exclusive access to one programming laptop with a dedicated fileserver share. I would then unplug the machines from any Ethernet connections except where needed for programming. This would be a lot less expensive than reprogramming
all of my machines!

I really feel for people having to connect all their machines to windows computers connected to non secure database systems and windows SCADA machines. These might be good candidates for a systematic redesign.

~KEJR

By curt wuollet on 5 October, 2010 - 12:25 am

It certainly would be cheaper if big automation switched to an automation oriented version of Linux. There would be porting costs, but then _they_ would control the upgrade cycle. Imagine for a moment, what it cost to rewrite for Vista, and it's already on the trash heap. Wouldn't it be cheaper if the current version of your software simply ran on the next version of the OS? If you could plan a life cycle? If they and their customers weren't driven to upgrade every time MS needs a revenue bump? And yes, you would have to support the old apps. You have to support the old apps either way. And it's not like Windows has meant that you could write once and be done.

In the typical lifetime of an automated machine, lets say 15 years, you have had W3.1, NT, W95, WNT4, W98, W2K, WME, WXP, Vista, and W7 and there's probably at least one I forgot. I don't pretend to be expert on how much commonality there is, but that's a _lot_ of churn that could be avoided simply by having an automation oriented OS with a lifetime of at least 7 years. That churn has cost a _lot_ of money, which indeed is the major reason for it. And if you look at how many of those "advances" were a good thing for automation, the cost/benefit looks pretty dismal.

In fact, I'm fairly sure I've programmed the same PLCs in all or most of those, and I haven't noticed things getting much better. I have accumulated a vast number of versions of various automation apps which combined with all the versions of Windows represents a pretty big mess with _high_ attendant costs and very little relevant or necessary "innovation". Cost, large - benefit, very modest at best. That's because it wasn't at all driven by, or even considering the needs of, automation.

Now suppose that span had been spent using an OS managed for the long life cycles of automation, with only those changes needed or at least, desirable for automation. With attention paid to backwards compatibility and the other criteria important to automation and control. I think the cost would have been very much lower. And not just for the vendors, but multiplied by the users. And that's not even counting all the anti malware crap. And just wait until MS goes careening off into Cloud Computing, or the next buzzword after that, and obsoletes whatever you hold dear.

Can you see, that for control people, controlling their environment holds major advantages, and _planning_ change would save big bucks? Getting off the crazy train would make a _lot_ of things better, not the least of which is security. In summary, as we know better than most folks, a process that you can control is far more likely to produce favorable results than a process that is completely beyond your control.

Regards
cww

Speaking as a developer of Windows based SCADA/HMI and engineering software (our controllers are running a RTOS, of course), I'm very sick of the constant Windows version churn. Years ago I put together a PanelPC based HMI product based on Embedded NT 4.0 that fits in 20MB of flash and still works perfectly well today--but Microsoft will no longer sell licenses for NT 4.0 embedded, so I was forced to move the whole thing to XP. XP is now obsolete, and we basically have to obsolete the product because porting it again is just too much of a pain.

On the other hand, we chose Windows for our products for good reasons: at the time the project started (about 1996), the linux platform was nowhere near mature enough. Microsoft's development tools are still the best of any platform (IMO, not going to start an argument about that :) ), and in 1996 it wasn't even close. Later, even better tools were available on Windows--the .NET platform is far easier to develop on than C++, lowering development costs and allowing us to have a better product than we could have delivered on linux or Unix with the same development resources.

Now that linux is a more viable option, we have a code base of 5 million lines of Win32 heavy C++ code and C# code--changing platforms isn't going to happen. Plus, too many of our users have had bad experiences with *nix based engineering tools and want no part of it--even if the modern linux world is totally different.

We also get major productivity benefits for being on Windows: easy networking, ability to run our engineering tools on any PC (including a lot of home PCs for our users), ability to integrate tons of 3rd party tools (ie OPC clients and servers, DTMs, etc), and easy integration with enterprise level applications (ie copying to and from Excel, Matlab, etc). This matters a lot to the huge number of users that are not actually running on the ICS, but are developing standard libraries, doing customer FATs, etc.

Certainly we've seen viruses and malware attack customer systems; usually it is a result of a lack of procedures or discipline to follow them on their part. We ship our systems properly configured to restrict access to USB drives, installation of software, etc. When the customer does something bad anyway, we kind of feel like a dentist does when they tell patients to floss. We could enforce the rules by having a closed architecture--it really wouldn't even require linux, just some super glue as Ken E suggested. However, most of our customers have told us they prefer an open system so they can make changes themselves. Though we do still get the occasional demand for Solaris support. :)

Going forward, one way a lot of vendors are dealing with the constant platform churn (including us) is to write lots of HTTP/HTML based tools. Embedding a small HTTP server into the controller allows us to drop a cheap interchangeable web browser in for our UI. It isn't appropriate for our main engineering tools or plant HMI, but works quite well for configuration and automation of small devices.

In reply to Demigrog: I think you have a good point about legacy code. A lot of existing automation software originates from the mid 1990s era, when MS Windows NT started to see serious commercial use. A lot of current also started out on MS Windows 95/98, and then jumped from there to MS Windows NT. GUI code is what tends to be the least portable unless you use a cross platform GUI toolkit. While cross platform toolkits are quit common today they weren't common back then.

I want to make a point about the "gluing USB ports" solution that often comes up. That was a trick that came from the IT industry a few years ago, but it's not really that realistic today. It is pretty hard to find new hardware that comes with PS/2 mouse ports, and it won't be long before PS/2 keyboard ports disappear as well. You can't glue all of them shut. People already do things like unplug their mouse so they can plug in a USB stick (I've seen it done!) to copy some files. Putting glue into the USB ports is just giving you a false sense of security while solving nothing. USB ports are essential and here to stay, and quite frankly much less of a risk than the Ethernet ports (and I can't imagine the utility of a SCADA or HMI system that can't communicate with anything else).

In reply to Ken Emmons Jr: If an existing vendor decided to do a Linux or BSD version of one of their existing products, I imagine they would base the design on their existing MS Windows version. In that case a user should be able to port a machine project over to that platform with minimal (if any) changes.

I'm not going to claim to be an expert on SCADA systems, but I have written a soft logic system so I might have a few ideas about what the issues are. I can say that with a soft logic system that unless you've made some very poor design decisions there really wouldn't be a lot of platform specific features in software of that nature. It's basically just a lot of network I/O and logic processing. With my own program I ship the exact same code for both the Linux and MS Windows packages. People run the Linux version on Apple Macs, and I've never even tested it on that platform (I don't have a Mac). The user programs, configurations, and screens are absolutely identical no matter which platform you run it on.

I won't comment on what tool chain or library problems a SCADA/HMI vendor might have porting their software over, since that isn't visible to the end users. What might affect an end user is if the vendor is using third party code for certain features but can't find a cross platform equivalent that is at least 95% compatible. I was going to list some examples of that, but after doing a bit of research I can't actually think if any likely ones that I'm sure of. The user program, graphics, etc. could be completely independent of the underlying OS platform.

In the end, it will all come down to whether vendors think there is really an upgrade market for products that offer better security. After recent events, I wouldn't want to say that there *isn't* a market for it, even if companies with critical operations have to be pushed into it by legislation. Anyone who did get into that market however would probably include software tools to help convert existing applications to their system since customers wouldn't want to re-write everything from scratch. There's nothing unusual about that in the software industry however.

I think that creating a "secure system" involves a lot more than just a port to a different OS (although that might be part of the process). I've hashed over that idea a few times before however, so I won't go over it again in this message.

By curt wuollet on 30 September, 2010 - 5:08 pm

I find it amazing the extremes that people will go to in ignoring the real issue and taking the most complex way out. We're already talking about vastly more screwing around to continue using Windows than is expended in solving automation problems. Yet, when I conclude that it simply is not suitable for automation, there is a great hew and cry and extensive rationalization for using, arguably, the worst possible choice. For example, they are called PC viruses and PC problems when there is absolutely nothing wrong with the PCs. And some say that a change of operating systems would do no good when well over 95% of the known successful virii live on Windows, and the rest you've never heard of, because they were not successful. And the virus problem is only a minor part of why Windows is a poor choice. Speaking of Windows generically, it's huge, it includes vast amounts of cruft of no conceivable use in industrial applications, it has been notably unreliable and insecure and is aimed at gaming, multimedia, and entertainment. There is no way to customize it, you take it all or none at all. The authors have been busy trying to do away with serial and parallel ports and the simple busses that work well for control. The licensing excesses are legendary and although they've finally been unsuccessful at forcing upgrades and had to concede on XP, they will surely get back in stride if at all possible. There are technical issues with control and the design of needed features is a secret and there is little hope of seeing or modifying the source to tailor scheduling, latency or IO to best meet automation needs. Their upgrade treadmill costs millions to support on the user side so that they can make millions on the supplier side and ISVs no sooner complete an upgrade than that version is thrown away and you start over. Except that you have that legacy to support for 20 years. With all that, and there's much more, I cannot see how it's so wonderful that people will all but lay down their lives defending it and reject any suggestion that an OS that can be tailored to automation and revved in a rational manner and has well established security and reliability might, be a better idea. What's wrong with this picture?

Regards
cww

By David Ferguson on 1 October, 2010 - 11:15 am

Cmon Curt:

Lets get real, if the programming software was Linux based, and all systems running power plants etc was Linux based, how long do you actually think it would be before a determined hacker, accessed that control system if they really wanted to.

The internal processor in the PLC is being attacked using the programming software for that chip. That software is running on windows and they are getting in using social networking, bad practices etc. So how long before they access that chip through Linux using similar methods.

The real issue involves connecting control systems of any kind other than hard coded chip based systems, in other words anything that can be programmed, can be hacked if it is connected or accessed. If the machine is programmed and then we lose the flexibility to change it by burning the program to roms (and this has holes) is the only way to stop it and we lose the gains made in the past 30 years.

The other day I saw an ODB connector for a car, that is wireless and you connect to the wifi in your car with an ipod or iphone and access motor data. How long do you think it is before someone reverse engineers the Diablo preditor programmer or some such thing and hacks the Hemo motor via wifi and ruins or gathers data from the competition at the racetrack or just for fun going down the road if these systems are out there in numbers etc.

If it is accessable which gives us flexibility and the ability to modify (anything) these are the issues that ensue.

A gun can be used for target practice or to kill people, we implement laws to discourage the latter use. In the same way this is a new area and maybe it needs the same treatment, as death can come from this sort of a hack. It is one thing to be stealing corporate info and making a machine do things it was not intended to do.

Once again (12 years and running) switching the OS will not change a thing, if they want in, they will get in, even if that means walking in and accessing the machine with the Linux software...........IT IS NOT THE SOFTWARE, it is the fact that there is software at all, instead of hard coded HARDWARE.............

Dave Ferguson
Control Systems Engineer

By curt wuollet on 2 October, 2010 - 3:02 am

It could be a very long time before anybody broke in. I don't think you're getting how dramatically different that environment could be. When you use Windows, you get the same system as everyone else with any thing that any app might need included, likely enabled, and commonly exploited. We've seen how that model works. You can do things to secure it, if you know about them, and you are allowed to spend the time and effort. Obviously this doesn't happen.

Now let's say I was selling into a critical application like, say a power plant. There would be the application. The system would boot into the application and shut down if the application was exited. No browser, no FTP, no MP3 player, no games, no unnecessary services at all. No outside access, no USB port drivers. SELinux set to high security, tripwire running, the integral firewall tight. For extreme security the necessary services could use non well known private socket addresses.

The system could even be run from CD which would make it pretty much hopeless trying to modify or corrupt files and the file count could be kept down to where it would be practical to checksum them on a cron job. If intrusion was detected or SELinux violations occur, you can simply reboot the CD and you're back to square one. The operator can't screw it up, even physical access wouldn't help much, it just won't do anything it's not intended to do.

They are already doing this for critical systems such as routers and the like. Practical systems for less critical applications could allow development and reasonable versatility and still eliminate any real practical route for takeover. When you can control the OS, you can tailor it to the threat and make it extremely difficult to mess with. Any level between consumer friendly Ubuntu and single purpose appliance is achievable and readily distributed locked down and about as hard as you can get. It would not be impossible to do mischief, but it would be pretty close to impossible to take over the machine and you would be detected doing it. That's a far cry from anything achievable with Windows where you take what they give you.

Regards
cww

@Ken
Yepp, I know the probs.

A couple of years ago I stated that there is no one who knows both sites, the PLC and PC. And joining them together in a safe way is a big challenge. Today this is still true, unfortunately.

Security is a hard job and it is full-time.

I remember a line in Australia where a secretary printed to a printer which ip owned by a frequency drive. That was fun... Took a couple of days with a lot of manpower to find out what went wrong including no production.

As I sayed, this everyone network with anything stuff. Its like heaven and hell. Heaven is if you get what you like from your boss. Hell is if you have to make it work what your boss is expecting from it. Try to get your boss at least half way on your side.

Thanks for reading :)

In reply to Ken E: I think you've hit the nail on the head on several points. There is risk in doing certain things, but there is also risk in not doing them. Consider the following dilemmas:

1) Your new production line has just started up, and as the machine builder is on his way out the door he holds out a USB stick to you and says "here's a copy of all the PLC programs, do you want them?". If you take them, might they have a virus (after all, who knows where his laptop has been)? If you refuse them, how do you get the line running again when a PLC loses its program, or has to be replaced?

2) Your backup system consists of storing copies of the PLC programs on the network where everyone can get them. Unfortunately, the virus can get them as well. If you *don't* store them there however, then what are the chances of ever finding an up to date and working copy of a PLC program again when you do need it again in an emergency (probably about zero)?

3) If you connect the production system to the business, you risk having a virus spread from one to the other. On the other hand, if you don't connect them together how do you move accurate and timely production information between the two so that your company can remain competitive?

In each of these cases we are balancing two risks. If we do one thing we *might* have a virus problem. If we do the other we will almost certainly suffer other problems (lost programs, uncompetitive operations, your company going broke, etc.). I expect to see happen is for companies to sell expensive and useless placebo software "solutions" that do nothing to address the root of the problem.

Here's the root of the problem. Businesses have been getting viruses on their office PCs for years (decades even), with no sign that this will ever change. If we do things exactly the same way in the plant, why would we expect to see different results? After all, if it were just a matter of following a few simple security check lists, then wouldn't the office IT types be doing that already?

On the other hand all banks, insurance companies, and large businesses of many description use very different back office system designs than what gets used in front office applications. They have their "production systems" hooked up to the Internet, but I don't see too many of them getting taken down by viruses, despite the attractive target they must make. When companies like Google get hacked, it's generally through things like people in their sales departments getting a virus on their office PC and getting their account passwords stolen (which is the same thing as mentioned above). Genuinely critical back office stuff rarely gets hacked directly.

We have two types of system to consider. Type "A" is the front office desktop PC. Type "B" is the "critical" back office business systems. Type "A" gets viruses regularly (enough that selling virus scanning software is a multi-billion dollar business). We just live with the problem because let's face it, it doesn't really matter if the latest policy memo from HR on the allocation of parking spots gets e-mailed this week or next week. For type "B" these sorts of problems are very rare (almost unheard of). When type "B" systems go down, the business loses money; sometimes a *lot* of money.

So, are automation applications type "A" (non-critical) or type "B" (critical to the business)? If they're type "B" (critical), then why are they designed to be exactly like type "A" (non-critical) systems?

Someone once said that the definition of insanity is doing the same thing over and over again and somehow expecting different results. I look at what is going on in the automation industry and it looks pretty crazy to me.

In reply to M. Griffin:

Yeah, I agree with your points too. We are now in a complete catch 22 that has allowed a culture that accepts someone [in a business setting] to install flash player to play facebook games at lunch time and letting grandma in accounting click on MrCutieCat.jpg.exe from an email attachment, all with administrator rights!!!! Some of this is dying down, but very slowly and many people don't have a choice at the present (There may be a critical application requiring administrator rights, etc, and changing it would nearly bankrupt the company...).

I have to be nice to my IT guy because he might be monitoring this (wink, wink) :o) but in reality the automation guys are going to have to be their own security experts in order to understand the unique requirements of networking and security as it applies to manufacturing and equipment. Lets face it, it doesn't follow the typical model of typical databases and servers.

Its one thing to say that "we should know better", but in reality many companies in this economy have extremely limited resources, to the point where day to day business operations are barely getting done. When you are always putting out fires you don't have time to put up the fireproofing and sprinklers! :o)

KEJR

By Rahul P Sharma on 29 September, 2010 - 11:35 pm

Is it time to go back to the good old days when PLCs ladders were programmed not thru a PC terminal but directly by using the Keypad on the PLC and assigning the memory addresses to the contacts.....!!!! Something that we used to do with the 8085 Microprocessor kits in college labs..... Instead of using a cross compiler and PC based system and programming the kit, it used to be directly programmed from the attached key pad.....!!!

Even then the system cant be declared foolproof..... Cos the PCs would be used at at least some level.... May be the PLC manufacturing process is some PC based system..... If a person is genuinely interested in letting a virus loose, he may do it at the PLC manufacturing end..... some virus code may be made memory resident at the time of manufacturing the PLC hardware...... And when the PLC is programmed at the site even using a direct keyboard attached to it without a PC, the virus would still be there.......!! and I guess there is no limit to going backwards in stages...... If PLC manufacturing process can be made manual and PC independent, then the virus writer can target the memory chips making companies and make their chips virus resident at some stage before they are shipped to the PLC manufacturer.....

The dangerous aspect of stuxnet is the idea, that PLCs can be infected with a virus and the possibility of an industrial disaster can happen..... I think there's no good way to stop the viruses from spreading itself..... Today we have stuxnet..... And am sure, there would be a hacker smiling somewhere thinking to target other PLC makes too..... Siemens is just a symbolic case, may be cos the fathers of Stuxnet had a certain country in mind which might be running a Step 7 system for their nuclear plant...... But this certainly raises the possibility of similar viruses for other systems too......

In reply to Rahul Sharma:

Again, someone had already mentioned this. If you take a programming PC (any OS, doesn't matter) and just load the PLC software and super glue all removable media and network ports/wifi you will essentially have a extremely secure system (and lock it in a vault with security guards for extra measure...). For something like a nuke plant or a high risk power plant/refinery, whatever, this sounds like a good idea. You need to do manual backups to a device that can be removed and not shared with any other computers.

Some of us fall in the middle, however, where we need to collaborate with others, put our files on shared resources and don't always have a choice of how that information is protected (it is guarded by IT and you are at their mercy).

Others are in situations where the manufacturing floor has to be linked into the business end of the company for critical run time data exchange and control and the automation guy doesn't always have a choice of how the network switches and routers are configured in the server/network room. The automation guy doesn't always know how to do this or has time to do this either.

I've always tried to design my machines where you pull the network plug and the machine still runs. Maybe I've been lucky in that most of my machines are standalone and don't need a lot of network connectivity to the corporate LAN other than non critical data such as reporting, etc. Others are not that lucky.

The fact that a virus is attacking PLC backup files on a filesystem is very disturbing and blows the whole theory of "running your machine unplugged". One consolation I have is that most of the office computer accounts don't have access to the machine backup files, but there are enough people that do have access that it worries me.

~KEJR

There was an article, page 1 of Wall Street Journal, April 8, 2009, "Electricity Grid in U.S. Penetrated by Spies", which says "Cyberspies have penetrated the US electrical grid and left behind software programs that could be used to disrupt the system, according to current and former national security officials...The espionage appeared pervuasive across the US". This reminds me of the movie Tora Tora Tora, about Pearl Harbor.

I almost never have customers ask me about security when coosing a product. I was pleasantly surprised today to see that one of our customers using the WinPAC contrller http://www.icpdas-usa.com/products.php?PID=3380 (Win CE OS) did shut down ftp access (which is the user manual recommended procedure).

In new developments this virus is now reported to be spreading in China. The following link has a good summary of that as well as the current situation elsewhere.

http://www.theregister.co.uk/2010/10/01/stuxnet_china_analysis/

Is it possible that Simocode motor protections will be infected too?

In reply to Robert: Reports indicate that so far the virus seem to be targeting its payload for the combination of Siemens WinCC with Siemens S7 PLCs. Hypothetically any PLC, servo drive, robot, or anything else that can be programmed or configured via a PC could be affected by some future version of this or some other virus. However, right now this particular virus is designed to re-program S7 PLCs.

The virus does take up residence in any MS Windows computer it can find. The virus is being sprayed out in all directions in the hope that it will find a PLC somewhere. Most of the people who are having problems with this virus have nothing to do with PLCs. Some of the people reading this may have the virus on their computers at work or at home but don't realise it yet.

This is the most complete technical description of the Stuxnet virus I have found:

http://www.symantec.com/connect/blogs/w32stuxnet-dossier

There is a link to a pdf titled W32.Stuxnet Dossier at the bottom of the page.

In reply to KirkC: That is an interesting analysis from Symantec (an anti-virus vendor). Some of what they have said however doesn't agree with other analysts, such as whether the most heavily affected country is Iran or India. That may however be do to those vendors having more customers (and so more reporting points) in those countries.

What I found particularly interesting can be summarized as follows:

1) The virus will directly infect Step-7 and WinCC project files. If you open one of these infected files, your computer will become infected with the virus. Apparently the project files are not simply data, but opening of the files can trigger loading of executable code. This is leveraging the design of Step-7 and WinCC project files with a known design flaw in MS Windows (other viruses have also used the same technique in the past).

2) It directly modifies programs in running PLCs. I will come back to this point later.

3) The virus injects itself into the Step-7 programming software and takes over some of its functions. One of the things it does is hide the changes in the PLC programs from you. If you attempt to view a program on line in a PLC to see if it has been affected by the virus, it will strip out the modifications and show you the "cleaned" version. It also intercepts attempts to erase certain blocks to prevent you from removing them. What this means is that if your computer has the virus, you *cannot* tell if your PLCs have been modified by examining the PLC programs.

4) One of the things the modified PLC programs do is to modify Profibus messages "on the fly". It replaces a standard library function block with a modified version, which can alter the Profibus messages and then pass them on to the "real" function block. This is one of the ways in which it can hide what is doing in your program.

5) If you connect the the modified PLC on line, it will temporarily suppress the modified behaviour. That may be intended to try to hide the changes from someone attempting to trouble shoot the machine.

6) The press reports so far have been emphasizing how the effects were very narrowly targeted at a specific configuration. From what I can see in the report, that is based on a misconception they have about PLCs. The "narrow targeting" is simply that the virus is targeted at two specific models of S7 PLC. However, compatibility between different models of PLC is usually limited, so it shouldn't be surprising that the virus was written with specific models in mind. The actual PLCs are the S7-315 and the S7-417, but those are two very popular Siemens PLCs, so the targeting isn't really all *that* narrow!

7) What seems to be impressing the anti-virus vendors and the "security industry" about this virus is that the authors have gone through the effort to combine multiple virus techniques in a single virus. Generally, virus authors just rely on one technique at a time. I read the whole report, and while I'm not an anti virus specialist, I didn't see anything that on it's own was exceptional about this virus (other than the target). Some of the things about it that people had thought were new turned out to in fact have been seen before, but were just never fixed in MS Windows (and several of the known security holes that it uses are still not patched). As I said, what is apparently new is that someone went through the trouble of combining multiple techniques in a single virus.

8) It is apparent from reading the report that the anti-virus industry doesn't understand the industrial market or the conditions under which which people working in it operate. The anti-virus specialists apparently had a great deal of difficulty understanding how the virus operated because they simply didn't know what a PLC was or anything about how they are used. I think they *still* don't grasp all the implications. There are lots of very simple things that a virus could do to a PLC that would be disruptive to an operation but very difficult to trace.


I think what has kept this sort of thing from being more common up to now has been lack of motivation on the part of people who might be able to do it. Now that it has received so much attention people may start to think about ways in which they could benefit if they could temporarily shut down or disrupt things like electric power plants, pipelines, oil terminals, large mines, semi-conductor manufacturing facilities, and other operations.

By burt johnson on 5 October, 2010 - 4:58 pm

just to add: a lot of talk about which os to use windows? linux? old school siemens T2000 dcs runs on a stable platform of unix. makes me happy that we have not upgraded to T3000 on windows os. will be happy to ride out the storm before that move is made!

Reply to M Griffin: I have a few general impressions, a somewhat different take, from reading Symantec's W32.Stuxnet dossier.

1) The virus's authors went to great lengths in researching a small subset of automation platforms that have to be working together and developed the virus to hit that type of target: WinCC AND specific Siemens controllers AND Profibus. The virus might be present but ineffective without this combination.

2) Even if the conditions above are satisfied, knowledge of the target network's topology are still required to get the virus to fully propagate and be effective.

3) If the virus's target was a small number of industrial systems in Iran (the Bushehr nuclear power plant has been reported as a likely primary target) it was, as the kids say, 'an epic fail'. The virus was detected and it propagated way outside its target domain, setting off the burglar alarm for everyone to hear.

Security on industrial networks is important and there is a valid argument for beefing it up. But here's what will happen in the wake of Stuxnet virus: the operating funds in the budgets of industrial facilities will be directed toward security and away from maintenance and core development. Just like after 911 in the US: an expensive wizz-bang, ultra-encrypted IP camera will be pointed at a broken motor operated valve, just to make sure no one sneaks up to it with a wrench to fix it.

I just sent the head of IT an email stating that although we don't use siemens products that this virus can infect some other systems if it were to have variants written to target other PLCs and/or automation devices.

I also mentioned that we have windows NT and windows 2000 running on some systems that are vulnerable to stuxnet and its variants and there are no patches available for these OS's.

I had asked if it would be wise to change user rights on these machine "users" and/or put them on their own subnet to keep them less vulnerable.

The short answer I got is that we have tape backup for our fileserver, and that we don't want to have automation systems on their own subnet right now.

I am quite concerned with this, but again, I am not a network or security expert as I've mentioned previously. It does smell funny to me and I'm not sure what to do about it. Our machines are, for most purposes, mission critical and we can lose serious money if they go down for extended periods.

How does an automation guy who is computer saavy but not a security expert deal with IT on these issues. I really want to work together on tightening security.

~KEJR

In reply to KenE: What I would do in your shoes is to bring the issue of security up via the engineering and production management channels. I would keep it at the level of "showing general concern" and not point any fingers at anyone. Put it in writing in e-mails however that there are viruses going around right now which are intended to shut down production lines and possibly cause damage, and this isn't just a hypothetical situation anymore.

To automation professionals, this is a serious concern. To an IT manager, it's just another virus. IT people have been living with them for years and don't worry about them much anymore. And if a virus knocks out the e-mail system for a few days, people are inconvenienced but the company doesn't suffer that much.

If a virus knocks out the plant however, that comes right out of the company's bottom line. In that situation though, the IT manager knows he isn't going to be the first one in line for blame, because the production equipment really isn't his department.

As for what you can do, well the first thing is to get people to agree that something should be done. The IT manager should be in a position where he is offering advice on how to solve it, but it's the engineering and production managers who should be deciding whether anything should be done. You might be using PCs in the plant, but they are production assets, not IT assets.

In more news, IDG Computerworld reports: "Dutch multinationals under attack from Stuxnet worm".
http://news.idg.no/cw/art.cfm?id=7948A378-1A64-67EA-E473A18859381224
They state: "A major supplier of industrial sorting systems based in the Netherlands has repelled two attacks by the dangerous Stuxnet worm ...". The problems here were in airport luggage sorting systems.

What is of particular interest is the response (or lack of response) from uses so far. The news report states:

"The commercial sector is mostly still oblivious, however. Communication efforts from Microsoft and Siemens about Stuxnet are missed, ignored or not taken seriously by IT companies and administrators. Companies are either not aware at all, or see no need for updates. Van Asseldonk, of service provider TASK24, is not aware of any company that has implemented the important patch from Siemens."

By Rahul P Sharma on 6 October, 2010 - 1:57 am

I agree that people in the industry are still not willing to take this seriously or are lazy to even talk about this virus...... The feeling that 'we are not at risk' is all too pervasive..... I spoke about this virus with a couple of friends in industry and they just couldn't appreciate the potential implications of such a threat..... And what surprised me the most was that our Siemens' person, who routinely comes for installation and commissioning and Maintenance purposes to the Plant was not even aware of this thing.....!!! And he just took this information with a mild sense of surprise.....

I am just wondering if during his next visit, should we even allow him to connect his Field PG (A normal Laptop used to communicate with Siemens' PLCs) to our Step 7 PLCs....!!! If he doesnt even know that such a threat exists and India is being reported in news as one of the country with highest number of Stuxnet Infections, then it is impossible to expect them to have taken any precautionary measures, like disinfecting their own Field PGs which they often use at various sites for PLC Maintenance..... I guess, this lack of awareness amongst those responsible is a bigger threat than the virus itself...... These guys will go on using the same Field PGs at all the sites oblivious of the threat of virus lurking in their own backyard.....!!

au revoir
Rahul

In reply to Rahul P Sharma: I would suspect that the sort of person that you are talking about is exactly the sort of person who is (unknowingly) spreading the virus in India. All he has to do is visit one customer who has the virus, pick it up on his laptop, and he will end up spreading it to every plant that he visits.

You may wish to contact Siemens and discuss your concerns with them. You may be able to work out a procedure to at least reduce the risks. The same problem applies to independent integrators and machine builders by the way. They are in exactly the same situation of visiting multiple sites and being exposed to the virus.

By curt wuollet on 7 October, 2010 - 1:21 am

Yes, these posts about lack of concern or complete ignorance point out that the current situation is bad, and what is most likely to happen, (very little) isn't going to improve it. That's why we need inherently more secure configurations from the get-go, so they are reasonably secure without depending on what _should_ happen.

Regards
cww

By curt wuollet on 8 October, 2010 - 12:08 am

Here's what Microsoft says we should do about their security/virus/malware problem. And no, I don't make this stuff up:^)

http://www.bbc.co.uk/news/technology-11483008

Lets all blame the user.

Regards
cww

In reply to curt wuollet: That idea (the Microsoft certificate) won't work for the simple reason that they are essentially asking the computer "do you have a virus"? A virus of course would simply answer "no, everything's fine here". The idea has been floated before, and it was pretty conclusively shown to be full of holes.

Microsoft already uses digitally signed certificates for drivers and several other purposes in an attempt to solve the virus problem. The particular virus we are talking about just used genuine certificates from two different sources to masquerade as legitimate software. While PKI is useful for some things, it isn't the security panacea that Microsoft was hoping for when they started using it. In fact, it's turned out to be a bit of a farce.

By curt wuollet on 8 October, 2010 - 10:49 pm

Well, maybe they should do it. I can't think of anything that would expand the Linux ranks faster than if the average Windows user got kicked off the web a couple times...Well, it might take a dozen times or so, they never blame Microsoft for Windows problems. I agree that it is a bit naive coming from a high ranking researcher at MS. Makes you suspect his credibility. But, I'm sure there's money in it, maybe you get to pay them to be allowed back on the net?

Regards
cww

By David Ferguson on 8 October, 2010 - 10:10 pm

For one thing, how can you say that this is the opinion of Microsoft any more than you are speaking for the entire control community on this site. Second the lack of people who are locking down their MS machines is no different than the Mechanic who screws up your car by not knowing his job.

The fact is the ART of control systems engineering is a conglomeration of many skill sets and no college out there or vocational school teaches it. You have to be an expert in multiple skillsets equally. Lets say there are 10 skillsets (arguable) they may include, Electrical skills, Instrumentation skills, computer skills, programming skills, mechanical knowledge skills, business skills, process skills, NETWORK SECURITY SKILLS, craftsman skills, maintenance skills, etc.

No one coming out of US colleges today is qualified to do this job any more. I laugh at the people who hold their nose high and say that "He is an EE so he is more qualified than that guy who went to votech". Bullsh*t, it is a conglomerate of 10-15 skillsets and to be good you need them all.,......I personally know like 4-5 guys who really have this skillset and they are very very rare and in great demand.

The fact that the web surfing set who can barely find their e-mail is setting up your control system is just a sign of how DUMB they are and we are for letting our systems get set up this way.

My personal opinion is that if you set your systems up this way. you DESERVE to get hacked. read that again YOU DESERVE TO GET HACKED. Evolution has a way of taking care of this sort of thing.......

Quit blaming the misuse of a tool for being the problem and lets start addressing the real issue which is the lack of people out there mentoring up the next generation to "get it". I take great personal pride in having had a number of Interns who are now out there in society and are very , very good at the skills needed to do things right. And they all came in with one degree or another and quickly were taught that they knew little or nothing of this SKILL / CRAFT/ TRADE?PROFESSION.

Quit blaming the tools and start fixing the users..........we are getting nowhere with this kind of babble.....

Dave Ferguson
Tool User

In reply to David Ferguson: I think we can take an official corporate post on an official corporate web site by a "Corporate Vice President" (that's his job title) acting in his official capacity as speaking on behalf of his company.

Here's the two points from the web site which I suspect that Mr. Wuollet particularly finds objectionable (or perhaps risible):

"Voluntary behavior and market forces are the preferred means to drive action but if those means fail, then governments should ensure these concepts are advanced."

and also:

(Microsoft) "will also advocate for legislation and policies worldwide that help advance" (the proposals).

Yes, that is an official statement by a Microsoft vice president saying that they intend to lobby for laws to be passed ensuring you keep your Windows patches up to date and have anti-virus installed.

And to repeat, I think it's a stupid idea and it has no hope of achieving its stated objectives. In addition, it would certainly do nothing to have prevented the situation related to the virus we are discussing here.

By David Ferguson on 9 October, 2010 - 8:21 am

Michael:

From his Blog post "In my speech today at the International Security Solutions Europe (ISSE) Conference in Berlin, Germany, I PROPOSE ONE{POSSIBLE approach to addressing botnets and other malware impacting consumer machines."

This has nothing to do with security, there is a move afoot by all of the big players, including MS, Comcast and Murdock, the RAA and Intel and others to enforce "net neutrality" and this scare is just another way to get to a different means. ATT already implemented a piece with bandwidth metering, tiered access metering. They want to control access to the pipes and be able to turn on and off people for a number of reasons. There is also movement by RAA to shut off your access after downloading ilegal movies and music forever without trial, so if someone highjacks your wifi (because you did not secure it, Linux based) and botnets you for music downloads etc. shut you off. Sort of like, if you want to go to war for Iraqs oil or to avenge your dad's name or because you own shares in a certain cleanup company, divert their attention and go in after the nukes that do not exist or after OSBAMA. Or cheat on your e-mails about global warming reports to implement carbon credits by scaring the public (still to come, notice they have backed off). Well by "scaring" us into the fear of cyber terror, we can take back control of this damn media thing run amok........ie China and Google..............there is more going on here than meets the eye...........focus on the ball, my point is this is diversive tactics.

Agreed, it has NOTHING TO DO WITH THE ISSUE ...................... the reason this stuff gets opened up to the lowest denominator is that you have non professionals using computers and in order for them to use them and not bug the rest of us, they have been DUMMIFIED so they can get their e-mail and play Zinga games. When your OS runs from these people to controlling your corporation and plant it gets slightly harder to find the middle ground. When Linux is preconfiged by the gurus to lock down the machine and my mom cannot get to play farmville, they start whining and someone posts a hole to get through, then someone writes a way to infect them. When the guy who should not be programming your control system is blocked, same thing. These same people will demand that the vendors DUMMIFY them also in Linux and we will be right back in the same boat.

Be a professional and lock your machines down and learn the IT portion of YOUR JOB. I do not agree with legislation to lock them down (WE AGREE, less government). But I also do not agree that some other OS will solve our problems.

Curt is quite capable of explaining his view......trust me we all already know that Michael.

PS: Intel just paid $xx Billion (yes billion) for McAfee....................we better get new chipsets also, because why would you pay billions for something that does like $2m a month........................

Dave Ferguson
Control Systems Engineer

By curt wuollet on 10 October, 2010 - 1:21 am

What irritates me it that MS skates and shirks all responsibility while everyone and everything else in the world is left holding the bag and making all the gyrations and hassle and expense to clean up their mess so they can continue leaving security holes open to make their floobydust features work. For example, yes, autorun is kewl, but executing any old binary someone puts in the slot is just plain stupid. Especially when it execs in the space of a privileged user because they don't want to make logins too hard and separate accounts by default. And they can't possibly deny that it's a problem, so they simply declare that it's your problem. And it really irritates me that everybody just bends over and lets them do it. That's what irritates me about that particular article. That and they can bend heaven and earth to avoid _their_ having to do anything serious about it. They sell on the basis that any idiot can run it, but blame them when they do, without having a security expert close all the intentional holes. And somehow that is perfectly acceptable. Your problem.

Regards
cww

By David Ferguson on 10 October, 2010 - 2:31 pm

and as I have said over and over......when Linux is in the hands of the anyone can do it set, the software (in order to be accepted) will be opened up or people will abandon it (right or wrong). When they cannot just start it and run, they will go away.

Now I will give you that something "free" is a lot harder to complain or quit on as you have nothing invested. But as you invest time (our most precious commodity), you quit when you use too much. But what is the difference between your pre-configured CD load and my knowledgeable MS configured HMI config "locked down".

I heard on a podcast the other day that the Internet is dead......I laughed and then listened, they said the future is in apps....again I laughed....then I watched my wife who hates computers interoperate with her ipod and she just wants the finger tap that does what SHE wants. She does not care that it is at WWW dot whatever or how the underlying structure works, it just does.

I am in the paper business, magazines and catalog paper.....I completely realize watching her who reads 5 papers on the web via apps that I am history.....luckily I am not in the paper business, but the automation business and do not care what it is I automate.

I also have a house that was full of college students and constantly watched how they work and communicate............they operate in a compartmentalized world and basically function through apps and any kind of underlying knowledge and having to type addresses etc stinks to them.

What's my point, my point is that such things are what we are getting to , less and less knowledge of the underlying technology nor caring what it is and eventually it will all go away. This has always been the case and the end user of my stuff just uses the "HMI App" whatever piece of equipment it controls. The responsibility to keep that underlying technology "safe" lies with me the user of that technology.

Problem is a lot of those doing controls do not take that part of their job serious because they come from this app mentality and do not see the importance and when it fails they blame their users....just like you accuse MS of.

Now MS has had to open things up to the lowest denominator because that is what customers demanded. Apple and the iPhone, iPod, iPad have introduced a new model which APPLE controls (as much as they can)....they approve apps and make sure that these issues cannot happen (yet). But as it gets bigger and bigger it is harder and harder to control. I personally think MS is in trouble because the cheese moved to a new model and they have not adapted, but I do not think the new model is a PC or a server running an OS (as far as the end user is concerned) it is to compartmentalized apps that are approved by someone who does know what is important to protect the end users..........

I think putting Linux as a replacement for MS is like me introducing a new type of paper for magazines and catalogs to be printed on ........it is not realizing the media is not the issue, it is the delivery system that is changing. Newspapers have been slow to realize this and therefore have lost 30% of their business this year. As I said I am in the "Solutions business" (personally), not the automation or paper business. My company is in the advertising business......although they may think they are in the paper business.....

Lots of babbling and probably lost my point......but I feel better.....

Thanks Curt

Dave Ferguson
Technology User

By curt wuollet on 11 October, 2010 - 2:06 am

> and as I have said over and over......when Linux is in the hands of the anyone can do it set, the software (in order to be accepted) will be opened up or people will abandon it (right or wrong). When they cannot just start it and run, they will go away. <
Yes, but saying it over and over again doesn't make it true. Linux _is_ in the hands of whoever wants it, with comparable ease of installation and it is, by default, more secure. At the very least, it won't exec random crap you stick in the drives. Or 99.99% of the existing viruses, etc.

> Now I will give you that something "free" is a lot harder to complain or quit on as you have nothing invested. But as you invest time (our most precious commodity), you quit when you use too much. But what is the difference between your pre-configured CD load and my knowledgeable MS configured HMI config "locked down". <

About $500. Counting the precious time you spend locking it down and the time you waste with antivirus scans and updates and all the other malarkey that goes with running Windows, probably _much_ more. That is a particularly unfavorable point to argue as the lost time protecting and putzing with Windows costs business billions. And I'm sure it's even worse on the ISV side, adding the churn. I've used both extensively, and I can assure you that I waste a lot more time with Windows than Linux. Even the most shallow analysis will prove that. And that's with "consumer" distributions.

> I heard on a podcast the other day that the Internet is dead......I laughed and then listened, they said the future is in apps....again I laughed....then I watched my wife who hates computers interoperate with her ipod and she just wants the finger tap that does what SHE wants. She does not care that it is at WWW dot whatever or how the underlying structure works, it just does. <

Exactly, a setup that boots to the application and simply does the task at hand would save vast amounts of screwing around. And like your wife, whatever is behind that really fades in importance.

> I also have a house that was full of college students and constantly watched how they work and communicate............they operate in a compartmentalized world and basically function through apps and any kind of underlying knowledge and having to type addresses etc stinks to them. <

Yes, we are approaching computers as appliances.

> What's my point, my point is that such things are what we are getting to , less and less knowledge of the underlying technology nor caring what it is and eventually it will all go away. This has always been the case and the end user of my stuff just uses the "HMI App" whatever piece of equipment it controls. The responsibility to keep that underlying technology "safe" lies with me the user of that technology. <

Exactly, that's why an OS that can be configured to provide only the needed services makes much more sense than one that stresses the platform over the application.

> Problem is a lot of those doing controls do not take that part of their job serious because they come from this app mentality and do not see the importance and when it fails they blame their users....just like you accuse MS of. <

Exactly why shipping the application with a secured OS as one unit would be a tremendous advantage over simply "Install this on your Windows PC" which may already contain a full load of virii, malware, might even be part of a botnet or owned by a hacker.

> Now MS has had to open things up to the lowest denominator because that is what customers demanded. Apple and the iPhone, iPod, iPad have introduced a new model which APPLE controls (as much as they can)....they approve apps and make sure that these issues cannot happen (yet). But as it gets bigger and bigger it is harder and harder to control. I personally think MS is in trouble because the cheese moved to a new model and they have not adapted, but I do not think the new model is a PC or a server running an OS (as far as the end user is concerned) it is to compartmentalized apps that are approved by someone who does know what is important to protect the end users.......... <

Yep, and what OS supports that model now? And more importantly, can let you, as the content provider support that model? Android is an example. Many providers.

> I think putting Linux as a replacement for MS is like me introducing a new type of paper for magazines and catalogs to be printed on ........it is not realizing the media is not the issue, it is the delivery system that is changing. Newspapers have been slow to realize this and therefore have lost 30% of their business this year. As I said I am in the "Solutions business" (personally), not the automation or paper business. My company is in the advertising business......although they may think they are in the paper business.....<

That's a silly analogy seeing how, at the moment Linux is beating the snot out of Windows in that exact market and we're only seeing the tip of the iceberg. But more important is, why? It's because of all the things I have been pointing out. It can be tailored into Android and the amazing array of embedded gadgets because of it's modularity, ease in porting to new hardware and suitability for the "post PC" class of applications as well as all the other sizes and classes of platform needed by automation. You could say that the success is vindication of my point of view. But it didn't need vindication, it's obvious from watching the market. And it doesn't take Columbo to figure out that using one OS on all the platforms in your vertical is going to be a lot cheaper and easier than managing five penguins and a whale. Even if the whale is really kewl.

> Lots of babbling and probably lost my point......but I feel better..... <

I'm glad you feel better, change is going to come, I'd just like it to be sooner rather than later. But, for example, PACs are already powerful enough to host their own programming environment, if it isn't Windows. Someone's going to figure that out soon. Look at a PLC, then look at a cell phone, the opportunity for convergence is hard to miss. That's why when I designed a PLC, I left the processor slot open to change.

Regards
cww

By David Ferguson on 11 October, 2010 - 7:38 am

I give up Curt......you win ......and yet for over 10 years.......you haven't.

Here I will reply for you

But Dave, I am not about winning or losing only trying to make this a better world.

Dave Ferguson

By curt wuollet on 11 October, 2010 - 8:01 pm

I think it really a worthwhile discussion at this inflection point where security shortcomings are just popping up on the (1950's Bendix) automation radar. I really appreciate the comments of our real live Windows programmer as they make real the lock-in that the leaders would face if they dared change their relationship with MS. It would be close to a declaration of war. This brings up the likelihood that the change may come not from the leaders but by replacement, someone who isn't in bed with MS and doesn't have a huge legacy and can use the efficiencies of a more suitable platform to carve out a place with features and pricing. Maybe someone who wants to be positioned for the smart power transmission and general infrastructure replacement market on the horizon. Maybe a well known name that wants to get back in the market, or someone who holds a mediocre market share and would like a change in the status quo. Lots of household names could do something like this and have a good chance of pulling it off.

Regards
cww

In reply to David Ferguson:

I agree with a lot of what you said. I am a user of windows and linux and not one of these systems is the utmost secure in the hands of any old user. Why? Because the system can be altered on either Windows, or Linux. The user can let down their guard at will.

I do believe Microsoft is to blame for the shell "shortcut" bug and many other subtleties. I think we can all agree that Microsoft puts their marketing before security, which is disapointing. Having Grandma run as Administrator is asking for it. On a personal note my wife thinks I'm a control freak for not giving her account administrator rights on our windows machine even though I explained both her and I were set up as "regular" users and the administrator account is just for that, Administration. (Hey wait, I guess I am a control(s) freak being on this site...)

I just received a couple of proprietary automation CPUs running Linux and it was shipped with a default root password from the vendor. Sure, I'm going to change it, but do you think a majority of users will? Probably not. So is this linux system more secure than windows, or less? (as a side note this system does have a filesystem mounted read only, but that can be changed with a remount as root, so no real security there).

KEJR

By David Ferguson on 12 October, 2010 - 1:48 pm

KEJR

And in the end.....that is all I am trying to point out, nothing more.

Do I think MS has a halo, no way. I just think that the real issue lies in the computer becoming a means to an end. The user when he /she chooses to use that tool, better know how to use it no matter if from Stanley or Craftsman. Switching hammers will not change this IMHO.

Now could it ship far more secure than it does.....absolutely, but those who do not know how to use a hammer demand that they do not and it is one of the skills of our job that we need to do, regardless of who made the hammer. And if everyone is using one hammer, the evildoers (always wanted to do that) will look at a way to exploit your hammer.

I have received many e-mails etc and know there are a lot of us out there, some of us just cannot resist a good debate....or are that stupid.

Dave Ferguson

In reply to Ken E: Ubuntu Linux avoids the problem by not even having a root (administrator) account while still requiring a password for things like installing software or changing system level configurations. You can create a root account if you know what you are doing (and have the password) but there is normally no real need for it. It's actually quite a convenient way of doing things once you get used to it. I think that Apple does something similar, although I'm not familiar enough with OS/X to say for sure how they do it.

As for security on MS Windows versus Linux (or BSD), the big security advantage that Linux has is less to do with the technical details and more to do with a Linux distro being an integrated solution from a single supplier. If there is a problem they can do whatever needs to be done to fix it properly. If the fix causes problems with any application software, they can just deal with that rather than trying to avoid side effects in software that depends on the bug being present.

On another note, I just saw a news report that Microsoft has just released a new set of security patches to fix a record number of security holes. One of the patches supposedly covers the Stuxnet virus (this is the one we have been talking about here). A fourth security hole which this virus can use is still not fixed yet however.

By curt wuollet on 13 October, 2010 - 3:01 am

It is more secure. There is virtually no way to secure a normal PC from someone with physical access regardless of what OS you are running. So the root login on the console is really not an issue. After all, they can just steal the hard drive and be done with it. And unless you are installing the OS, they need to assign _some_ root password as root is always a separate entity. Otherwise you could not become root if they did not tell you what it is. (there are ways around this which I won't go into here.) And you need to be root to add users, etc.

But, usually, with this as a consideration, the only place you can login as root is at the console, which means that any remote login is running as a normal user. But, of course, you can change this to allow virtually anything. And I'm not saying you can't make a Linux system insecure. It truly is your system and belongs to you so you can do nearly anything. What I am saying is that a Linux system distributed by non-idiots will normally be more secure. The point is, that if you endeavor to distribute Linux, you can distribute it configured very securely and most do. The fact that you can also open everything up is a privilege where you bear the responsibility. Contrast this to Windows where you pretty much take what you get and have to take action to make it secure. Preloads can only be as secure as the people loading the OS make it. It is my contention that making this the responsibility of the distributor makes it far more likely to actually happen than depending on the user. We could very well have distributors who, rather than deal with security questions, take the Windows way out and disable security. But I haven't seen any distribution that does this.

Regards
cww

By David Ferguson on 13 October, 2010 - 8:05 am

Curt:

For EXAMPLE, Emerson supplies PC's for DeltaV that they approve, they then ship the PC to Austin and set the WINDOWS box up they way they want it secured and set up etc. Do they go far enough in my opinion ..... no because none is secure enough, but this is exactly what you are saying. If you choose to go a different route and supply your own PC then their tech support will not support you. There is about a 50 page document of the things they change from normal in their configuration.

Why do the other vendors not do the same.........because Emerson takes a lot of heat for the expense added to set these up with humans who know what they are doing. If I am going to take on that responsibility then I should know what I am doing.

Dave Ferguson

By curt wuollet on 13 October, 2010 - 8:09 pm

That does indeed accomplish the same thing, as much as Windows can be secured and still run popular software. But what you mention is exactly the problem with doing it that way. It takes time for each box. Now if you did it once and copied it to each machine then you could achieve both a measure of security and efficiency. But problems come in because MS doesn't allow just anybody to modify Windows and redistribute due to licensing concerns. They were sued about this by vendors that did want to boot to some other look for example. Very high volume sellers like Dell, might negotiate some some slack here, but it requires a Windows source license and a Windows license for every box sold, even Linux boxes, which I suspect is why a Linux box costs more than the same box with Windows. There may be other secret concessions. Linux doesn't have this type of licensing, so people at say, our level, can legally harden and redistribute and bundle with an application. I _am_ glad to hear that _someone_ is attempting to seriously address the problem. Maybe it will start a trend as the legal types figure out who the customer is likely to try to sue. Kudos to Emerson. Now, they should be able to simply Ghost the first machine and save lots of time and trouble. Many IT shops do this with a site license, but I don't think it is allowed for resellers. Imagine the problem they would have if someone offered "Secure Windows" for sale :^)

Regard
cww

> That does indeed accomplish the same thing, as much as Windows can be secured and still run popular software. But what you mention is <

---- snip ----

> There may be other secret concessions. Linux doesn't have this type of licensing, so people at say, our level, can legally harden and redistribute and bundle with an application. I _am_ glad to hear that _someone_ is attempting to seriously address the problem. <

We solved that problem by creating a CRC based checksum for all critical software parts ... that means the code of the softPLC itself and the code of the control application. So every modification of theses parts will be recognized.

Regards

--Armin

http://www.steinhoff-automation.com

Curt,

I log into this particular embedded system over plain old not-very-secure Telnet with the vendor supplied default root password. I plan to change the password and use ssh in the near future...

My point is that just because it is Linux doesn't mean some company supplying it to you can't configure it, blow the doors off of security, and then ship it to you as a product.

KEJR

By curt wuollet on 13 October, 2010 - 8:15 pm

I agree completely, it's yours to do as you like. But at least in your case, it's clear who is liable.

Regards
cww

There are two previous Control.com discussions that are relevant to this topic. I thought I would point them out to everyone so anyone interested can go back and review them.

The first was "Alleged Internet Attacks on SCADA Systems".
http://www.control.com/thread/1026243185
This was a discussion about news reports that the American CIA was warning electric utilities in the US that someone was attacking or was about to to attack power plant SCADA systems, and they wanted American utilities to take precautions. This seems strangely prescient given current news.

The other was "Computer Virus in Electric Utility in Australia".
http://www.control.com/thread/1255317497
This wasn't a virus which was targeted at control systems. It was a conventional virus which was affecting the control systems indirectly by causing problems for the MS Windows workstations.

As well as the usual opinions, the discussions contain links to relevant news stories. It's good background for anyone who is interested in this subject.

Dear All

Well,What about other PLC's....Is this virus will attack only PLC's or even Motion controllers??... Is this virus can attack Baldor controller's...In my plant all the machines are running on Baldor Motion controller and drives.....and the software used is Mint workbenchV5.5.....Please do let me know so that I can take some action...

With thanks and regards
Edwin

In reply to Edwin: This particular version of this particular virus does two things. One, is it installs a more or less conventional virus in a computer running MS Windows (and it hooks itself into any copies of Siemens Step-7 and WinCC which maybe present). The other thing it does is that when an infected computer connects to a Siemens S7 PLC, it modifies the program in that PLC.

So, what you have is basically two problems. One is a more or less conventional virus problem that affects everyone who uses MS Windows. The majority of reported virus infections are on PCs that don't have any Siemens software installed. It is also important to note that not all the holes in MS Windows that this virus exploits have been plugged yet.

The other problem is that if you are using Siemens PLCs, the virus may have modified your PLC programs via an infected PC.

So far there have been no reports of any versions of this virus that attack Baldor drives. However, if someone wished to write one, they could certainly do so using similar techniques.

By Chris Jennings on 14 October, 2010 - 11:33 pm

It is interesting that just recently I have been working on a Windows based system that is used for recording video of events (paper breaks) on a paper machine. The computer all run WindowsXP embedded and are pretty much bullet proof. There are hardly any drivers installed on the machine, the USB ports don't work (much to my disgust because I can't run diagnostic software) and the network only has basic functionality.

I was trying to see if I could use a "generic" PC as a replacement for the old and obsolete P4 machines with IDE hard disks. The short answer is no, everything is tied to the hardware and Windows is pretty much there just as a basic environment for the software.

So you can make a bullet proof PC running Windows, but not generic Windows. The embedded versions have everything removed that you don't need. The disadvantage is that you tie the software directly to the hardware which could be a problem. Once again I think this puts the ball in the control system vendors court. Microsoft provide the tools and the software to make it possible to distribute a secure Windows based system, but people choose not to do that. Probably because clients want the flexibility and don't like vendor lock in. Perhaps the client is to blame?

Chris Jennings

By David Ferguson on 15 October, 2010 - 9:11 am

BINGO.........this is exactly my point, it can be done today but vendors and users choose not to do it and again it will be the same with Linux IMHO.

Dave Ferguson
Control Systems Engineer

In reply to Chris Jennings: MS Windows XP taken in original condition straight out of the box will not install or run on modern PCs (there are ways of working around this involving loading additional components via floppy disks during installation, but it's not for the faint of heart or the impatient). That may be the reason why you can't install it on a new PC.

In addition, Microsoft does not support an version of MS Windows XP prior to SP3. That means for example that there are no security patches or updates available for the virus we are discussing here. SP3 will be supported for another short while, but after that it's strictly MS Windows Vista and MS Windows 7 (until the replacement for MS Windows 7 comes out, at which point support for Vista gets dropped).

MS Windows XP Embedded is just MS Windows XP with some utilities to allow an OEM to repackage it to use less RAM and disk space and to add some custom branding. It has nothing to do with making the OS more secure or robust. If your copy doesn't have SATA support, then I rather doubt it has any security updates available, even assuming the repackaged version has the ability to install them. MS Windows XP Embedded is an obsolete product. Microsoft now has something intended for that market based on MS Windows 7, but it isn't as customizable to the same degree.

I don't think that MS Windows XP Embedded was really suitable for a "SCADA appliance" application. A SCADA system is probably going to have lots of RAM and hard disk space anyway as the SCADA packages and databases themselves are usually not very light weight. If you were going to do something like that with MS Windows you might be better off starting with one of their server versions, although most people would probably cringe at the cost. It would at least give you an install that didn't have rubbish such as trial software and toolbars installed.

What you would really want to be able to do however is to give a customer a disk that had the OS and application software together. Everything would get installed together and the application vendor could guaranty that everything was compatible and configured correctly. The vendor would also handle all updates (including OS updates), so they could test for any compatibility problems. All support problems would be handled by a single source with no finger-pointing between vendors.

As to whether Microsoft would allow third parties to do this with MS Windows, I can't say for sure, but I rather doubt it. They have become quite restrictive lately as to what you can do with their product. For example it used to be common for PC vendors to put their own splash screens on the boot process, but Microsoft doesn't allow that anymore as they want to elevate their own brand above that of the PC vendors.

I can think of quite a few companies who sell software-only "appliances" using Linux or BSD, but I can't think of a single one that does the same with MS Windows. A vendor can distribute MS Windows as part of a PC, but not as a stand alone customized package. Given the potential market for this in business applications, I imagine that Microsoft simply doesn't allow it.

Actually, I would bet that you could walk into the majority of businesses and find at least one MS Windows installation that was technically "pirated" even if paid for because one or more of the terms and conditions was transgressed in some manner. There are consultants who make a business out of figuring out Microsoft's license terms and explaining them to their customers.

In reply to M. Griffin:

> MS Windows XP Embedded is just MS Windows XP with some
> utilities to allow an OEM to repackage it to use less RAM and
> disk space and to add some custom branding. It has nothing to

I just wanted to add one minor point in that the other use for embedded windows is to lock down the main filesystem for the equivalent of linux "read only root" with AUFS. In theory a simple reboot will get you back in business if some software went bonkers and rewrote some critical feature of the OS or application (could even be a simple virus...)

I'm pretty sure you can do this in regular windows but you have to download the right drivers and utilities to set it up.

I will say, however that in practice this is a bit difficult in that the way windows software is installed and configured does not lend itself to the read only filesystem very well. The windows embedded system lets you use a utility program on another PC to set up an image which can be transferred to the target machine.

I do like the linux command line approach better, once the filesystem is set read only with aufs you simply remount the root filesystem at the console (and with Debian) do apt-get and then remount as readonly. I've not set one of these up myself but I am currently using an embedded linux system that we purchased this way and it is slick.

KEJR

In reply to Ken E: "Locking down" a file system be marking it as read-only may prevent a legitimate program from accidentally writing to that volume, but it won't stop a virus. The virus would simply ignore the read-only marker, or else re-mark it as read-write, write to it, and re-mark it as read-only. A number of copy protection systems use the same techniques as viruses and subvert Windows so they can write to areas of the disk they would not normally be allowed to (e.g. right outside the normal Windows partition altogether).

On the other hand, read-only mounts are very handy in embedded applications, They're used in embedded Linux systems (as you described), and they're also used in live CD and live USB flash systems as well.

They're not really a security feature however, unless the actual media is read-only, either inherently as CD is, or otherwise (e.g. some flash drives used to have hardware write-protect switches).

By Ken Emmons Jr. on 19 October, 2010 - 8:49 am

In reply to M. Griffin:
> "Locking down" a file system be marking it
> as read-only may prevent a legitimate program from
> accidentally writing to that volume, but it won't stop a
> virus.

I know, that's why I mentioned a "simple" virus in my post. If they are writing to the boot sector of the drive or directly to the disk somehow, all bets are off! :o)

KEJR

By curt wuollet on 16 October, 2010 - 1:53 am

I don't think so, because leaving security up to the client obviously doesn't work. And assuming they are competent to secure the PC is wishful thinking. And leaving the power plant PCs wide open because the night operator wants you to, is irresponsible.

Regards
cww

By Chris Jennings on 16 October, 2010 - 7:45 am

Check
https://www.microsoft.com/windowsembedded/en-us/products/weenterprise/default.mspx

Make of it what you will, I won't be editorialising.

Chris Jennings

In reply to Chris Jennings: I had a look at the URL you posted, and I see I was wrong and they actually do have a version of MS Windows Embedded for Vista. I don't know where I got the impression that they didn't. Perhaps it hasn't been too popular.

Rather interestingly, their list of customer applications includes "SCADA devices". They are distinguishing a "device" from a "PC", so I don't think this would have been for a typical SCADA work station. Possibly it might be a panel PC with a "lite" version of a SCADA product? I haven't seen one however.

The products they listed by the way were not for a "software appliance". They assumed you are bundling hardware. What they are saying the "embedded" version gives you is "the flexibility to deploy a custom user interface". What that means more or less is that you can put your own brand name prominently on the screen.

I had a look at their support expiration dates for MS Windows XP Embedded, and they listed the following:

XP: 22-Oct-2004
XP SP1: 10-Apr-2007
XP SP2: 11-Jan-2011
XP SP3: No fixed date.

However, although this policy states that SP2 support is still in effect until next year, their actual support announcements for the Stuxnet virus (the one we are talking about here) don't include patches for SP2, only SP3. So regardless of what their general policy states, they don't seem to be supporting it. They have however been telling customers that they need to upgrade if they want support.

For XP SP3, they don't give a fixed support date and what information they do have just sends you around in circles. It looks like they are playing this one by ear. Since the embedded version is just a derivative of the normal desktop version that will probably depend on what happens in the general market, and right now the majority of their customers are still running MS Windows XP.

The length of their support lifetime however should really only be an issue for someone who doesn't plan on upgrading their SCADA system on a regular basis.

There are new developments in the story about the MS Windows virus which attacks Siemens control systems. There is an AP news article that states the US government is taking an interest in introducing regulations regarding security for control systems. Here's a link to the story:

http://www.ctv.ca/CTVNews/SciTech/20101118/stuxnet-computer-virus-warning-101118/

The same story is also available from multiple other sources if you search for "Stuxnet virus could target many industries, experts warn".

The interesting points in the story are the following:

"They warned that industries are becoming increasingly vulnerable to the so-called Stuxnet worm as they merge networks and computer systems to increase efficiency. The growing danger, said lawmakers, makes it imperative that Congress move on legislation that would expand government controls and set requirements to make systems safer."

and also:

"(Michael Assante) ... encouraged senators to beef up government authorities and consider placing performance requirements and other standards on the industry to curtail unsafe practices and make systems more secure.

The panel chairman, Sen. Joe Lieberman, I-Conn., said legislation on the matter will be a top priority after lawmakers return in January."

From this it looks like there will be big changes coming up in some parts of the controls industry in the US. At this point however, I would predict the result will be lots of pointless paperwork, with suppliers and customers trying to pass off the responsibility to each other, but with no actual discernible benefits.

Real improvement will require significant changes in the way that the problem is approached, but that will probably require another major incident before people are sufficiently stirred to action.

By curt wuollet on 18 November, 2010 - 4:05 pm

I have to say I was intrigued and amused by the hilarious notion that this problem will be solved by legislation. Unless they move to prohibit Windows in critical places, I can't begin to think that legislation will cause people to do what common sense and basic engineering have failed to bring about. Like "Don't include a virus magnet in your design". Millions, perhaps billions, will be spent to avoid the simple solution. And it will include the best legislation that money can buy. Again to avoid the simple solution. And the problem will continue. We've seen this before. MS wins, everyone else loses. But almost everyone likes that. Strange.....

Regards
cww

I would like to steer this discussion back to the Stuxnet topic. Interestingly, this has been described as the first attack in a cyberwar. Many feel that the complexity of the virus is such that it must have been state sponsored and that Israel, America, China, [Insert nation here] must be responsible. It had to take several man-months to code and it took an in-depth knowledge of processes, PLC programming, etc. so it must be assumed that a team was involved. Any thoughts?

http://www.networkworld.com/community/blog/stuxnet-worm-and-cyberwar-what-happens-next

http://news.cnet.com/8301-27080_3-20023124-245.html

To do exactly what Stuxnet did took quite a bit of effort. However, to do something equally effective against most typical targets would not be any more difficult than what many virus authors already do today. Yes it was a targeted attack, but that isn't new. Google was subjected to a targeted attack not long ago that was successful at getting confidential business information. The attackers in that case new that Google used MS Windows XP with IE 6 in their Chinese ad sales office, and they simply took one of the many commercially (black market) available off the shelf MS Windows and IE6 viruses that are out there and added their own custom payload to scoop up Google's sales data. This type of targeted attack happens all the time. There is nothing unusual about that and there is nothing to stop any interested party from repeating this with an industrial target.

There is an entire industry dedicated to creating and selling MS Windows viruses to people who want them. If you want a zero-day MS Windows exploit (one that has never been used before), you can buy those too, they just cost more money (I think the going rate is around $10,000). Some virus authors offer support contracts to their customers. Getting an MS Window virus is not a problem.

As for creating the virus payload (the part that would affect the control system), that doesn't look to be that difficult either. You could use the vendor's own programming software to create your PLC changes, record a download to the PLC, and then figure out how to "re-play" that back again later. If you target an MS Windows SCADA or HMI system instead of a PLC, the job would be even easier as the virus could do the job directly via the OPC server. Since a lot of the more lucrative potential targets have SCADA systems, that's a good target to look for.

There's really only two things which have prevented this problem from occurring so far. One is that the automation control industry is so technologically backwards that it has been difficult to apply current common virus techniques to it. For example, modern viruses normally depend on a network to spread, and networks have only recently become common on the plant floor. Indeed, some of the techniques used by Stuxnet should be quite familiar to anyone who recalls the days before PC networks became common and viruses were usually spread by infected floppy disks.

The other thing which had prevented this from happening is that the people who make money from viruses simply weren't aware of the potential market. Now that Stuxnet has happened, their thoughts will be turning to the automation market and what can be done there. Now someone just has to use their imagination to figure out how to make money by knowing that a generating plant, pipeline, or mine will be knocked out during a certain time window. I'm sure that a commodities speculator could handle that end of things.

If you want a good example of a scenario, consider the following. There are many, many gas turbine generating plants in the world that all have very similar control systems. Find out who the engineering or maintenance staff are at one or more of these plants, and send them an e-mail with a virus labeled "click here for deals on re-built Moog servo valves". Now, if any of those people have their PC come in contact with the GTG control system (perhaps they're already networked directly to the HMI so they can keep track of it?) you have your "in". If your virus can cause difficult to troubleshoot trips, you can create a shortage of electric power in that area. If that plant supplies a de-regulated electricity market, a speculator who knew this was going to happen could have bet on the price of electric power going up. Somebody could make a *lot* of money this way. The same concept can be applied to oil or gas pipelines, large refineries, certain metals mines, etc.

What I find surprising is that this isn't happening all the time right now. Focusing on what was special about Stuxnet is a red herring. What people should be focusing on is how easily much less sophisticated efforts could cause disruption done for profit.

By curt wuollet on 22 December, 2010 - 1:12 pm

The thing that scares me is that even though stuxnet shows a fairly sophisticated and focused effort, it's really way overkill if you just want to take down a competitor or cause mayhem. Many times a day we see someone with a comms issue and the universal answer is "Just use OPC", or some other Windows based method to make things talk. So you don't have to even know anything about the PLCs or process, they've just made their system vulnerable to the simplest of techniques. Just look at the registry and down the box. You can easily get a day or two by simply deleting the registry or if you are feeling nasty, you can edit it and see how many people can figure that out. Or scramble the BIOS. Or corrupt packets, which would be really ugly to figure out, or.......... The system is only as strong as the weakest link and many are dependent on a link known to be weak. With a crafty OPC killer, you could create a generic plant bomb. If it didn't do anything else, it could get wide distribution before it was detected. None of this would require anything new or particularly clever. And you need no knowledge of PLCs or their programming software.

Regards
cww

By Kevin Cooper on 23 December, 2010 - 7:08 pm

There is a sense that this is the beginning of a technological war that attacks physical assets via computer software, which is rather a first. I think that this is a legitimate threat and it also shows the power of industrial controls to the world, who has hitherto taken it all for granted. It is rather astonishing to the general public that physical things (processes, actions, etc.) can be, not only affected, but rendered useless by the same technology that enables them to use Friend Face (insert social site here)

In reply to Kevin Cooper: Whether Stuxnet actually had any effect on its intended target is unproven at this time. However, people shouldn't get too fixated on the details of Stuxnet in particular. Much less sophisticated viruses could have much greater effects on industrial systems with much less effort than what went into Stuxnet.

It is much more reasonable to expect to see virus attacks which are intended to make money for someone. Chronic simple virus attacks are more likely than elaborate ones and are much more likely to achieve results as long as the originator is willing to be flexible in his choice of targets. This is simply an extension of virus attacks from sales and financial systems to other areas of a company, so it shouldn't come as a surprise to anyone.

That's a good point and it seems to be echoed in a new article in ISAs Intech- http://www.isa.org/InTechTemplate.cfm?template=/ContentManagement/ContentDisplay.cfm&ContentID=84318

Please let me interrupt this fascinating debate by offering a sincere thank you to all who have contributed.

As someone about to spend several USD millions on new automation next month, how we deal with the threat of malware invasion is very high on my agenda.

Solving this whole malware issue in a secure way, is surely our industry's top challenge for 2011.

Qarsaan, it will be interesting to see what you do to mitigate things like this. Please post if you have suggestions throughout your project.

I would also be interested in hearing about your project as it progresses. It would at least be interesting to hear what problems you had to take into consideration and the areas where you think the industry in general needs to improve the most.

By Chris Jennings on 30 December, 2010 - 9:12 pm

When you look at the information provided in this thread is almost enough to put together a white paper on best practice. It is something that the industry needs, and we aren't getting much guidance from the automation vendors (perhaps with the exception of Emerson).

Is it something we should continue to collaborate on this mail list or perhaps move to a Wiki/Google Wave or similar device?

Chris Jennings

In reply to Chris Jennings: If you want to discuss security practices, the best place to do that is right here on Control.com. If you go off with a small group of people you lose visibility. Pick a topic with some focus and start a new thread. Give us your opinion, and if anyone has a different one they can join in. Once that topic has been beaten to death, start another one on a new thread.

There isn't going to be a single answer to all applications. So, pick something like "security for PC based HMIs", or whatever takes your fancy. We can flog that question to death and then you can come up with a new one. When there seems to be some general conclusion, perhaps you could summarize it and put it in the Control Wiki.
http://www.controlwiki.com/index.php/Main_Page

By curt wuollet on 31 December, 2010 - 12:07 am

It's simple, don't use Windows.
Problem solved.

Regards
cww

By curt wuollet on 31 December, 2010 - 1:17 pm

Actually, that was a bit short, I apologize. I just find it fascinating the extreme lengths the world + dog will go through to avoid a sound and relatively simple solution to the problem. Yes there is the legacy issue, but it's like Deming's definition of insanity. People keep doing things the same way and expect the results to be different.

It's kinda like the lower Mississippi problem. A hypothetical discussion :^)

Well, we got wiped out again, time to start rebuilding. We'll get the earthmoving equipment out and start making levees.

Hey wait a minute, the earth levees are why we got wiped out.

What are you getting at? I don't understand. Levees are made of dirt.

Yes, but you keep making them and they keep failing.

What do you mean? Earth levees are easy to understand, they look good, they're easy to build and everybody believes they will hold next time, we're _using new dirt_. And we can tear them all out again and put gravel under them, and, and, we'll grow grass on top! It'll cost millions, but we've got dirt.

I've got an alternative, lots of soggy people have joined together and they'll contribute all the concrete and tools you'll need to make stronger levees. You can start out new and put concrete in the places where the dirt doesn't hold.

Get the h*ll outa here with that crazy talk, you don't know nothin about building levees. And we hire half the town pushing wheelbarrows full of dirt. They don't know nothing about pushing wheelbarrows full of concrete. And there's all the contractors and the excavators, and what are we going to do with all that dirt?

I think they could learn to work with concrete, and you could put the dirt behind it.

You got a hearing problem, old son? There ain't anything wrong with dirt levees, we've been building and rebuilding them for years. And the only time we _ever_ have a problem with them is when there's a flood. And even then it's not the levees that are at fault, it's the d*mn water, if we didn't have high water, we'd never have problem with the levees. Jimbob, Buadreux, get the dogs and show this guy the highway....... Hello, Bud?, we're gonna need a whole bunch of that dirt 7.0, yeah, the waterproof stuff... Oh, it's 7.1 now and it's self patching?

Sigh.......
cww

In reply to curt wuollet: I don't think your answer was a bit short. I think however that most people would want to ask you "how"? When someone has a problem, the best answer to give them is "here is the solution to *your* problem". If there is an open source answer to give them, then that's the time to bring it up.

I don't think the answer is to say "persuade the existing big vendors to change". If they want to change, they'll change; if they don't, then they won't. All the successful open source projects that I can think of started as just someone or some group of people who just went ahead and did it, and the big established companies either came in *after* it was already a success or they went out of business. If you take the attitude that nothing can change until the big vendors decide that change is in their interest, then nothing will ever happen. You just have to make things happen on your own.

There's a project that needs doing, and I happen think that you're just the guy to do it. There's a package on Sourceforge called Modbus Firewall, or something like that and it's Linux netfilter extension for Modbus/TCP. Unfortunately, the guy who wrote it is a networking expert from Cisco, so using it as is probably beyond the average automation programmer (I suppose the same could be said about most advanced networking systems).

What it needs is for some sufficiently motivated individual who knows something about Linux and something about C to work with one of the existing popular Linux firewall distros (IPCop, Untangle, ClearOS, etc.) and get it integrated as a standard feature. Once that is done, then when someone asks about security or another Stuxnet happens you can point to "X" as the completely off the shelf, packaged, and ready to use solution which can be used by anyone who can stick a CD into a disk drive and click on a menu.

There you go. It's Linux in automation, people need it, and all it's waiting for is for someone (you?) to go ahead and do it. And it's all about security, so it's right on topic here.

By curt wuollet on 3 January, 2011 - 4:58 pm

Hi Michael

Yes, I agree that would be a useful thing to do. Unfortunately, I haven't been
coding much or advancing my automation projects or anything non-essential lately. I've been living near third world conditions and the Minnesota climate has been keeping me sufficiently busy wrangling firewood to make the necessary concentration unlikely. And, I agree the "how?" is at this point the most crucial question. But, I willingly admit that I am no longer sufficiently engaged to write world class secure networking code. It's well beyond the Bonehead C tm. class of endeavor. But, I appreciate your vote of confidence:^)

It is beyond our power to convince all the majors to switch, but Stuxnet and the need to continue in business after the first real disasters Windows Virii cause will probably take care of that. Our military did an abrupt about face on using Windows in theater after it was demonstrated how idiotic that was, and I credit the automation majors with more acumen than the Army. I hope that trust is not misplaced. In the real world, the extreme CYA desire on the subject of security is not going to favor any startup or unknown.

Real world factors considered, it could happen this way: Security becomes a for real, contractual _requirement_ on important big dollar projects, infrastructure comes to mind. One vendor makes a serious attempt at becoming the "go to" for those contracts, which precludes the use of Windows, and provides the financial incentive to create an end to end product that automation people can grok and can risk using, with the (dubious) comfort of big name backing for the CYA factor. It works much like the familiar. The others move at the usual glacial pace, providing a window to garner market share. Game Over.

The reason that I see this as a possibility is not because I am a Linux advocate or glimpse any vision of world dominance, but because the only remaining way to acquire market share may be to actually do something different, and solve this problem. And no perfectly good disaster should go unleveraged. Being the solution would be a far more powerful position than being part of the problem. I think some entity is close to the tipping point already, and they could change the world in their favor. Have to close, it's getting cold in here again.

Regards
cww

As far as something like stuxnet is concerned, using linux would not have helped. There are zero-day exploits in linux as well, and with the resources behind stuxnet I doubt any platform is secure enough to be safe.

On the other hand, generic malware is a problem that could be solved by moving away from windows. However, in my industry (power generation DCS/HMI) we are not seeing infections in the vast majority of our customers, as they are following good procedures and best practices. We have to balance the pros of a non-Windows platform with the cons (development costs, lack of vendor support, customer familiarity, etc).

I expect to see lots of new cybersecurity product offerings in the near future, starting with catching up control networks to at least the level of corporate networks (managed switches rejecting unauthorized devices, firewalls, more restrictive policies on HMI boxes, disabling unneeded services on embedded devices, etc).

At the very minimum, a competent expert in cybersecurity should review any control network design.

In reply to Demigrog: As far as Stuxnet itself goes, it would be a mistake to get fixated on that in particular. If some of the popular theories are correct, the originator of Stuxnet also had the option of dropping a bomb on the building. You're not going to fix that with a firewall.

However, what was interesting to me about Stuxnet was not the virus itself, but what was done with it - creating the appearance of difficult to find transient bugs in the actual control system. The virus was just a way of delivering the payload. You could do the same thing with any of the many off the shelf MS Windows viruses, and there is an active black market selling those to anyone who wants one. A virus like would be cheap and easy to create by an individual, and there are plenty of commercial reasons for disrupting power plants, pipelines, water systems, and other infrastructure, as well as manufacturing plants.

What I think part of the answer is for SCADA and HMI vendors to offer a managed platform service where they take responsibility for the entire system, including OS and database as well as the SCADA or HMI software itself. To make that practical, then yes I think that they would need to use and OS like Linux or BSD, and a database like Postgres or something similar. That's due to the need for managing the business relationships however, not just whatever technical arguments can be made. A look at the security appliance vendors is instructive; not a single one that I am aware of uses MS Windows.

As for the cons you state, the only one that is really relevant in this situation is that the vendor's existing product lines are closely tied to MS Windows. There are some smaller vendors who don't have that problem, and it doesn't stop anyone from coming out with a new product designed around this idea.

By Chris Jennings on 3 January, 2011 - 4:57 pm

In reply to M Griffin,

I didn't know about the Control.com wiki. However spammers obviously do, is anyone using it aside from spammers at the moment? The recent changes list reads like a who's who of spam.

I would be happy to help moderate the wiki if that helps.

Chris

By curt wuollet on 3 January, 2011 - 5:28 pm

A secure, purpose built, automation version of Linux might not be impenetrable, but it could come close to the best possible case and it would be unlikely to generate any interest from the vast majority of virus authors. And since nearly all of the known exploits involve features totally unnecessary for an _automation_ distribution, it could be far, far, more secure than OTS Windows. Or even what is practically achievable with Windows. And it's doable today, with an investment truly trivial compared to _actually_ securing the status quo. But, I do expect unbelievable extremes and gyrations rather than the simpler solution.

Regards
cww