Hello all,

Currently on our oil & gas platforms, we are going through a phase of a large number of smaller brownfield jobs that require constant modifications to our control system software.

Although there is a procedure on how software changes should be made on the SCADA system (PLCs, HMI, etc.) we currently do not keep any backups of these software for disaster recovery. Our SCADA system providers (ABB & Siemens) hold all master PLC files with them and in the event of an unforseen failure of the components, we would be relying on them mobilizing someone to fly in from half-way around the world in order to restore the software.

I think it would be a good idea to keep a copy of all the software that exists offshore on any equipment such as firmware, logic, etc. in the event of a disaster. This would involve the SCADA, ESD, F&G, and Compressor Contro, and UPS systems just off the top of my head.

I just wanted to ask if anyone could point me to or forward me any sample standards or policies for backup and version control. Are there standards for something like this?

Thanks,
Harish

 

Not only is it a "good idea", it is essential that this is done. Without backups you are running a MASSIVE risk that a small fault can destroy your control system losing months of work & costing huge amounts of money.

Backup strategies are something that we have put in place for several of our customers.

Your individual requirements will vary, but our general approach is to plan so that we can recover from a total loss of the systems we are protecting.

We typically backup on a regular schedule (eg weekly) and keep at least 3 to 4 versions of these backups. This enables you to rollback to a known condition in the event of something like a virus or progressive breakdown. Backups must be both available onsite for immediate use, and stored securely offsite in case of the loss of your storage facility. Finally, you need to test your backups occasionaly to ensure you don't have terrabytes of useless data.

We use "Acronis" backup software to create image copies of computer disks as these can be made online without shutting down the computer being backed up (Unlike Norton Ghost). Also, you need to become familiar with the backup tools built into your particular control systems.

If the customer already has backup procedures used by their IT systems then we often re-use and adapt these.

Rob
www[.]lymac.co.nz

 

Hi,

I work on a hospital with approximetly 200 PLC's and HMI panels and the PLC's are connected to a SCADA.
My job as a "system integrator" is to watch over the hospital so that whenever an entrepreneur makes any changes in a PLC or a new one is installed it is my job to implement the changes in the SCADA and place the latest PLC backup's on our file server.
This is a local standard we have developed and it works fine.
I don't have an answer to your question but maybe it is sufficient for your company to develop a local standard.

 

Backup and version control is "standard" practice. That doesn't mean that most people do a good job of it. It is just something that most people know they should be doing.

There is no "one size fits all" solution to this type of problem however. It needs to fit into how your organisation works. Backup and version control involves management of people as much as it does management of equipment. Running around behind people trying to clean up after them when they change something is futile. The people who are authorised to make changes need to be made responsible for seeing that the correct copies are made and properly stored. This means that it is something that your company's management are convinced is necessary.

Backup and version control isn't just for your programs. You also need it for your drawings, operating instructions, manuals, calibration procedures, and everything else you need to operate your equipment. If you look at a technical solution, you need to look at one that addresses the whole picture, not just a small piece of it.

From what you have said, it doesn't appear that you are making frequent changes to the equipment. This is the situation that most people are in. The most straight forward way to get started on dealing with the problem can be to just set up a series of directories on a network and store the appropriate files in them. Set up a separate directory for each machine or system, and have subdirectories for PLC programs, drawings, configurations, calibration instructions, etc. When you make a new change, don't erase the old files. Store them in an archive (e.g. ZIP) file.

What you should *not* be doing is simply automatically taking regular copies of whatever is in the machines and calling them "good". You can really dig yourself into a hole doing this. If a program has changed, then there should be a good reason for the change, and you can decide then whether to make that version the new backup.

You can get version control software (e.g. GIT, BZR, CVS) which automates some of the work flow. However, you will still need to hunt down and organise all the information before you can install something like that anyway. You will find that making an inventory of your equipment and hunting down all the missing programs and drawings is actually the most difficult part of the project. I would suggest that you don't worry about any sort of version control software until you have a good idea how many files you have and how often they are changing. You may find that an automatic system isn't worth the effort.

You may however still want to automatically backup things like databases if you are logging production data. However, that is a separate issue and you may be doing this already.

 

If I'm understanding you correctly, you're concerned about more than just backing up the current state of your system, and are asking about configuration management more like what source code control providers like CVS, source safe, and team server provide. (ie the ability to look at changes made over time, have traceability to who made the change and why, and the ability to restore a prior version of part or all of the configuration).

A lot is going to depend on what your software provides for you; I'm not familiar enough with ABB/Bailey systems to know about them. For the Mark VIe systems I use, ToolboxST has integrated configuration management that is built on CVS. I can look at the history of a particular controller and even do compares between versions to see exactly what changed. We automatically backup the CMS repository using conventional utilities like Symantec Ghost.

Even without an integrated CMS, anything that uses files for configuration can be stored in something like CVS with some degree of success. With text files you'll even get some basic compare capability.

 

VersionWorks® is an Enterprise Industrial Automation Software Source Configuration Management solution, providing a comprehensive suite of tools to protect, save, restore, discover, and track changes for industrial programmable devices and documentation.

http://versionworks.us

VersionWorks centrally archives and intelligently compares software and document versions using a compare feature explicitly for RSLogix, Simatic S7, Simatic S5, PCS7, InTouch, WinCC, Intellution iFix, Microsoft Word, Excel, Schneider/Modicon, Text, and Binary Files, among many others.

 

Hi Michel,

Apart from this topic, I would be interesting to know what you are doing with 200 PLCs in a Hospital? Is it controlling HVAC? What is the application? which product?

which Scada system you are using?