SCADA Reliability

D

Thread Starter

Dave Millett

Our field is contract operation and maintenance for steam boilers, generators and similar plant. We use various DCS for controlling the plant, and for complex sites we use PLC's and SCADA (RS View).

The PC's running the SCADA's are industrial rack-mounted units, running NT4. We are suffering regular failures where the PC's lock up and crash, often at least once a week. The worst sites are those which are either power generation (I am already investigating the electrical feeds to the PC's), or are not purely Allen Bradley PLC's. On one site our SCADA communicates with Mitsubishi PLC's, on another it's Siemens.

What is everybody else's experience? Are PC crashes common with SCADA's? Has anyone found any typical causes, and what were the solutions? Are third-party drivers not reliable?
 
J

Jake Brodsky

> What is everybody else's experience? Are PC crashes common with SCADA's? Has anyone found any typical causes, and what were the solutions? Are third-party drivers not reliable?

My experience with NT is that people treat NT systems as if they were office computers. This is an invitation for disaster, particularly when you use things such as ActiveX drop-ins or try to run other applications which may not be well behaved. When treated like a dedicated box with a minimal GUI, NT generally behaves itself; though I have doubts that it will ever match dedicated applications running on better understood platforms such as BSD, Linux, HP-UX, or OpenVMS.

I believe the GUI is too tightly coupled to the kernel for NT to ever be considered "stable" across the entire range of applications one could run on it. In contrast to most other OSs, a GUI crash will not take the kernel with it, will not stop a service from running, and usually will not leave the file system in tatters.

On a seperate note: writing stable, well featured drivers for ANY operating system is no picnic. Windows NT is not the easiest operating system to write a driver for, either. It's not so much a question of whether an OEM or a third party can write a better driver, but whether either of them can take the time and care required to craft a stable driver.

However, despite these problems, we are moving ahead with deploying a distributed system of SCADA polling servers based on Windows NT. Believe me, it's not my preferred choice. However, it's what the rest of the staff feels most comfortable with and frankly, when I'm done with it, I want to be able to turn it over to them and do other things...

Good Luck!

Jake Brodsky
[email protected]
 
J
Good points made here. I would like to add a point of clarification on
drivers, particularly non Microsoft 3rd party drivers. One must separate
drivers for NT into 2 camps:

a) Low level drivers that dive deep into the operating system, making low
level calls
to the operating system, or perhaps even (blasphemy) bypass things like the
NT Hardware Abstraction Layer.

b) Higher level drivers that make standard calls to the Windows API to
access the RS-232 COM ports, access the standard TCP/IP Ethernet
connections in the operating system -- -this is where the majority (50% or
more) of SCADA drivers sit.

In the first case, those drivers are very difficult to write, much more
risky for anyone in terms of the side effects of what happens when
something goes wrong. There probably is some truth to an OEM or 3rd party
being hard pressed to write a driver. All the same though, I think it
bears noting that "Microsoft ain't perfect either" -- there's a reason 3rd
parties exist and often sell things that replace/augment/add to things in
the OS at a low level -- it's because Microsoft has not addressed the need
for whatever reason. In some cases, I'd say a dedicated 3rd party who is
smart about his business is going to put MORE attention to detail into the
driver than a large beast like Microsoft because they have to -- if they
don't, they don't eat - it's that simple. The achilles heel of the low
level 3rd party driver writer is when Microsoft changes something in a
service pack at the low level and violates common software interface rules
and removes interfaces that used to exist and were documented. It
sometimes happens. Undocumented interfaces, it's Microsoft's right to
remove them -- they say so -- "if it's undocumented you can't count on it
being here in the future".

Now having said that, I think it also bears noting that most OPC servers
and ActiveX controls for talking to PLCs from SCADA systems are going out
ethernet or serial. There is no need to go to the low level. The
point: Most OPC Servers and ActiveX controls for PLC communications are of
the 2nd type of driver - ones that make calls to the standard windows
APIs. The end point i'm getting at is that one should make blanket
statements that a 3rd party will not necessarily take the time to craft a
stable driver. I think in the case of drivers that are written to the
standard Win32 APIs for serial and ethernet communications one would find
that reputable providers who have been around for many years and who make
the majority of their living from drivers do craft stable products, care
very much about quality, and go to great lengths in testing to insure
stability and to do what they can to mitigate the "microsoft factor" of
when things change in their OS. By using the standard API's that are well
documented and commonly known, risk is minimized for the customer because
those APIs are the least likely to be tinkered with by Microsoft in a way
that breaks applications.

The moral to the end user is - be careful when someone is making very low
level calls or requiring special low level serial or ethernet drivers, for
then higher degree of due diligence might apply than if standard Win32 API
calls are made. If the developer doesn't understand the question, then you
draw your own conclusions.

Hope this helps --

John Weber
Software Toolbox Inc.
 
Hi Dave:
Wow!! If by crashing you mean the OS is gone blue screen on you than I'd have to say No, that's not common. Don't get me wrong, I've seen more than my share of stop codes, but this shouldn't be happening on a commissioned system.
I've used RSLinx, KEP and Wonderwares drivers in the past and never had any kind of major trouble.
I've seen systems stop polling when update times were set too fast on slow network (ie DH485!!!), and the graphics can update slowly when alot of tasks occure all at once (ie report generation & printing, database queries ...) but I haven't seen it take down the OS.
I'd be interested in what the NT event log reported. May want to take a quick look around at things like virtual memory settings, drive usage, services that maybe can't start for one reason or another.
Bottom Line though, RSView shouldn't be crashing your system.

Mike
 
A

Alok Khatlawala

My experience with RS View32 in its client server config and connections to AB PLC 5/80E has been disastorous.
We have been trying to get things right since quite a long time, but still are not confident that the plant would be running safely.
In order to maximise the safety issues, we have undertaken a few precautions like ...
a. Do not do ON-LINE modifications on anything in RS View. I mean "nothing at all". No graphic change, no tag database chane, no NODE switching etc.
b. Do not work on the server at all
c. Ensure that no other appiication is running on the main server
d. Verify the task manager (if running Win NT) and check the memory and CPU usage periodically. Note whether any process in process control is continuously increasing the memory usage OR CPU usage and note this for future reference

These in general has made the RS View system more reliable than failing once every week. The failures happen after longer durations in time, but it helps!!
 
Top