I'm involved in a project that must connect several wastewater treatment plants to a control center. The plant are automatized with different SCADA and PLC's, whose programs can not be modified, and I want to use free software whenever it's possible.

My initial idea is adding to every plant (around 50) a subnetbook (Dell Inspiron mini, probably) with an opc server and openopc gateway running on a virtualbox windows xp. The OS of the netbook will be Debian Linux using python and openopc to access the data. The idea is to add the same infraestructure to all the plants, changing only the opc server depending on the plant current PLC.
Then, I will connect to a main server in an office using GPRS or 3G, depending on the location, must of the times it will be GPRS.

My idea is using MBlogic to make a modbus link between every plant and the control center and MBLogic HMI to show the plant when it is required. The connection with the plants will be done throught a ssh tunnel for safety reasons.

The control center must be able to access in real time to a plant to see how it's working and change parameters, and must store in a database the collected data from all the plants with stadistical purposes. I know MBlogic doesn't provide the latest feature, but I plan to add some python programming to send the data from the plants to the center every hour and use MBLogic only when realtime is needed.

To present the collected data and make reports I plan to use a standard LAMP application, and use apache rewrite and proxy directives to show the MBLogic HMI screens when required, integrating all the user interface under an only web site.

I just like to know if this architecture makes sense to you, specially to Michael Grifin, author of MBlogic who uses to write in this forum.

Regards

 

To deal with your points in no particular order:

For the hardware, I would suggest a regular desktop style computer or a small form factor computer rather than using a netbook. It will be easier to replace if there is a hardware problem and less likely to disappear on you. If you want a solid state hard drive, you can buy 8 - 12 GB ones now for a reasonable price.

As for running MS Windows XP plus OPC plus OpenOPC, in VirtualPC, I think you will want to benchmark this on whatever hardware it is you plan on using. If you plan on using a CPU like the Atom, there is no hardware virtualisation support, so it might be a bit slow.

I can't comment on the GPRS or 3G angle, as that is not my field.

As for automation protocols, which ones do you have to deal with? If you need a special hardware interface for the protocol then you may wish to consider using a protocol converter box which converts the proprietary protocol to Modbus/TCP for you. This would eliminate OPC (which can be quite expensive), which in turn would eliminate the need for OpenOPC and Virtualbox. This simplifies your in plant terminal software quite a bit.

MBLogic doesn't log data to a database at this time. However MBLogic is based on the Twisted communications framework, which *does* have a database interface API. You might want to write the database interface straight into MBLogic instead of writing a separate program. Basically, MBLogic is just a Twisted application with automation related modules integrated into it so you can make it do whatever Twisted can do.

Alternatively, there is a package called Plantstreamer which is a GPLv3 data historian. I haven't used it but one of the authors (Andrey Romanenko) posts on here regularly and could probably tell you more about it. This might be an off the shelf solution that gives you what you want so far as data logging is concerned. It has an OPC DA interface built in.

I believe I understand your reason for running Debian on the in plant PC. It is intended to give you a secure communications link via ssh. I haven't tried this configuration, but I would suggest benchmarking it for performance to see how much overhead the encryption adds. That will help you decide what hardware to use. Whether or not this is a problem will depend on your polling rate.

You also need to deal with making the connection and reconnecting it whenever there is a problem. Twisted handles ssh natively (its one of the built in protocols), but I haven't investigated what would be involved in using the Twisted ssh class rather than using the ordinary Twisted TCP class to send and receive the Modbus messages.

There is a new version of MBLogic which is nearly ready for release. The system data table has been completely re-written and is much easier to interface with now (which was the reason for that change). Basically, if you want to add your own software module to the system, you can now deal directly with the system data table as ordinary integer and boolean lists and don't have to convert the data formats anymore.

As for the HMI, the new (unreleased) version also has an all new and much improved alarm and event system. Alarms are now summarised to their current state instead of just being a historical list. Alarms and events support zones, so a particular client can request just the alarms or events it is interested in.

I can release this version as is if it would be of any help to you in deciding how to do things. The features are done, and I am just waiting for some feedback on the HMI protocol documentation from some partners.

If at any point all you want from MBLogic is just the HMI, you can get the identical HMI via HMIServer (in the MBTools package). This converts the HMI web page data requests to Modbus/TCP requests and directs them to a separate Modbus/TCP server instead of using a built in data table. The new version of this will be released at the same time as the new version of MBLogic (the reason for the delay is the same in both cases).

The LAMP stack for ordinary reporting from the database is probably the easiest approach.

You can also do that using Twisted (instead of Apache) and Nevow (XML page templating). MBLogic has several web servers built into it, and it uses Nevow XML templates to do the on line live status reporting. However, if you're not familiar with them the learning curve can be a bit steep.

So, to summarise the most important points:

- GPRS or 3G - I can't comment on this.

- MBLogic as a data concentrator - This is one of the applications it is meant for.

- OPC in a VM - This sounds complicated. What protocols do you have to deal with? Can you do it without OPC?

- ssh - Should work. What about managing (starting, reconnecting, etc.) the connections though?

- LAMP reporting - Good plan.

- Cost - The most expensive part is probably going to be making the connection from the PLCs. Look at alternative protocol translators.

 

Hi,

I am working in the same segment as you, and the concensus over here is that radio communications on a fixed frequency is simpler to deal with compared to GPRS (even though GPRS is evolving greatly and I haven't tried it myself). In our country Radio is a fair amount cheaper as well.

The radios we use support Modbus directly (as well as many other protocols). My advice would be to not install any PC on your stations - the environment is also very corrosive in there (metal is generally more rusty inside than outside).

I would use a protocol like modbus (supported by many PLC vendors) and concentrate the logic at the center. This is to reduce bandwidth in the 'happy day' scenario.

If you can find a way to update the PLC code from the center this is also extremely useful (if you have long distances I would say almost a must ;-) ).

I would agree with you if you said that TCP/IP would be a big advantage. My current radios can do this as well but the bandwisth is not too great. In any case the installed PLCs currently dont support ethernet here.

Good luck,
Tallak Tveide

 

Thanks for your answers, I'll try to explain my reasons inline:

>As for running MS Windows XP plus OPC plus OpenOPC, in VirtualPC, I think you will want to benchmark this on whatever hardware it is you plan on using. If you plan on using a CPU like the Atom, there is no hardware virtualisation support, so it might be a bit slow. <

I've already tested it, and it works ok for this application requirements. In the water treatment plants , signals timing to the control center are not needed very fast, getting data every 5 or 10 seconds is more than enough. The vbox machine perfomance is pretty good in this notebook. It's like running XP in a 4 or 5 years old computer, with 500 mb of RAM (I could add more if I needed as the linux mem footprint is not so big).

>I can't comment on the GPRS or 3G angle, as that is not my field. <

Just think of 3G as an adsl connection and GPRS as an old modem with bad speeds and noise.

>As for automation protocols, which ones do you have to deal with? <

That's my main problem: the plants have been built in the last 15 years and there are different architectures, brands, models and protocols. And I can not touch anything in the plant as they should continue working with their PLC programs and computers as they are now. That's why I thought of OPC, as I can use an opc server depending on the protocol to connect, and the rest of my application would be the same in every plant, running in the independent subnetbook.

> If you need a special hardware interface for the protocol then you may wish to consider using a protocol converter box which converts the proprietary protocol to Modbus/TCP for you. This would eliminate OPC (which can be quite expensive), which in turn would eliminate the need for OpenOPC and Virtualbox. This simplifies your in plant terminal software quite a bit. <

I'm also thinking of this solution but, as an example, "Omron Host link -> Modbus/TCP" hardware converter is more expensive than a subnotebook + the required opc server (in Spain's dealers prices). And in the subnotebook I can log data when the network link is lost. That's something I can not do if I plug the net to the converter.

>MBLogic doesn't log data to a database at this time. However MBLogic is based on the Twisted communications framework, which *does* have a database interface API. You might want to write the database interface straight into MBLogic instead of writing a separate program. <

right, that was my initial thought.

>Basically, MBLogic is just a Twisted application with automation related modules integrated into it so you can make it do whatever Twisted can do.
>
>Alternatively, there is a package called Plantstreamer which is a GPLv3 data historian. I haven't used it but one of the authors (Andrey Romanenko) posts on here regularly and could probably tell you more about it. This might be an off the shelf solution that gives you what you want so far as data logging is concerned. It has an OPC DA interface built in. <

I know it , but it's java based, with a lot of requirements that will make the network and control center server price and load go up quickly. I'd like to avoid java if it's possible.

>I believe I understand your reason for running Debian on the in plant PC. It is intended to give you a secure communications link via ssh. <

Right, and thinking of securing a windows machine that's going to be connected to Internet is a real headache for me. I feel much more confortable and safer doing it with Debian.

> I haven't tried this configuration, but I would suggest benchmarking it for performance to see how much overhead the encryption adds. That will help you decide what hardware to use. Whether or not this is a problem will depend on your polling rate.<

As I said above, that should't be a problem if I just poll a hundred of bytes every few seconds.

>You also need to deal with making the connection and reconnecting it whenever there is a problem.<

Good to know it.

>There is a new version of MBLogic which is nearly ready for release.<

That'd be perfect

>As for the HMI, the new (unreleased) version also has an all new and much improved alarm and event system.<

After contacting you, today my client has told me that internet explorer must be able to see the HMI pages. That has broken all my expectatives of using your HMI interface, as MS boys don't want to follow the standards and ie doesn't support xhtml nor svg. I'm now replanning the architecture (again), as what I'd like to do is adding the opc module to MBLogic and use all the excelent framework you've done. Now I'm checking the Google Api Ajax to see if I can use them in order to simplify the project as much as possible:
opc server->linux with sqlite and a small webservice for logging and another for real time ---GPRS---> control center with LAMP and Google Api Ajax for real time presentation and reports.

Some time I feel stupid when dealing with some MS-dependant clients ...

Thanks for your suggestions and ideas.

 

I have a problem with the distances: these are the plants that my client maintain, and they are all distributed in Spain, with hundred (and in some case one thousand) of kilometers between the plants and the control center. So a internet mobile phone connection seems to be the only solution (excepting the expensive satellite links)

>The radios we use support Modbus directly (as well as many other protocols). My advice would be to not install any PC on your stations - the environment is also very corrosive in there (metal is generally more rusty inside than outside).<

The pc is going to be stored inside a ruggered box, and the box is going to be inside the office of the plant, as far as possible of the corrosives environment.

>I would use a protocol like modbus (supported by many PLC vendors) and concentrate the logic at the center. This is to reduce bandwidth in the 'happy day' scenario.<

I'd love to use modbus,as it would remove the need I have now of using a Windows box, but I can not touch the protocols the plants are using, and a multiprotocol to modbus/tcp converter is more expensive that the opc server + a subnotebook.

>If you can find a way to update the PLC code from the center this is also extremely useful (if you have long distances I would say almost a must ;-)<

Nope, I'm not allowed to touch inside the PLC's that are running now.

>I would agree with you if you said that TCP/IP would be a big advantage. My current radios can do this as well but the bandwisth is not too great. In any case the installed PLCs currently don't support ethernet here.
Good luck,<

Thanks, I see I'll need it

 

Hello,

>> Alternatively, there is a package called Plantstreamer which is a GPLv3 data historian. <<
[...]
>I know it , but it's java based, with a lot of requirements that will make the network and control center server price and load go up quickly. I'd like to avoid java if it's possible. <

Could you please elaborate on the requirements? We are currently using Plantstreamer in a vbox fetching a few thousand of tags at sampling rates that you mentioned and saving the data in a local PostgreSQL database (open-source) on an average desktop. The only requirement is that you need JRE (downloadable from Sun) in order to run Plantstreamer. You can use it in Linux or Windows as easy as "java -jar Plantstreamer.jar". And its is open-source (GPLv3), you can download it for free.

Regarding reporting, we have a solution based on PHP+Apache+Tomcat for online reporting and trending. For offline reporting we use openreports.

Please contact me (andrey at ciengis.com) if you need further info. Thank you.

Regards,
Andrey Romanenko
Ciengis - Advanced Process Control and Optimization

 

As for your customer wanting to use the MS IE web browser, if they want to spec that then they need to spec exactly which version they mean. If they want it to work with *any* version of IE, then the amount of work you will have to do is that much greater.

The big problem with MS IE is not only is it not compatible with any other web browser, but every version of MS IE is incompatible with every other version of MS IE. Web developers tend to dislike it because it multiplies their work load. Even the simple static page that Control.com is using needs a special fix for MS IE 6.0. Microsoft can't even get their latest version to work properly with their own web site.

On the other hand, it didn't take much effort to get the same demo page that I included with MBLogic and HMIServer to work with virtually every other major browser available (Firefox, Epiphany, Opera, Chrome, Safari, Midori) on two different operating systems. Compatibility between different web browsers is actually very good. It's mainly MS IE that has problems.

In-line SVG is the easiest way to have dynamic graphics as part of a web page. You could use static png images and swap them around with scripting, but that isn't as flexible. You would also have to write a bunch of your own JavaScript to handle this instead of just using the libraries provided with MBLogic.

There are a number of other automation oriented tool kits that call themselves "web based", but they are actually just Java (or MS ActiveX) programs that get downloaded by the web browser and run inside the web browser's process. However, you are limited to whatever they happen to provide and they tend to be quite expensive (which might be a problem with 50 sites).

As for using Google's (or Yahoo's) Ajax toolbox, that will get you a lot of general Ajax features, but how do you integrate that into an automation system without a lot of extra work?

With regards to the requirement for PLC "programs can not be modified", do the PLCs *all* have usable communications ports? If not, then some will need a hardware change. Are *all* of the PLCs communicating with something now? If not, then the programs may have to be modified to get them to communicate with anything. Does someone have a list of what data addresses are to be monitored for each PLC? Sometimes the logic state that someone wants to see isn't represented by an actual address. The rung may have to be modified to put that logic state into a usable form.

I suspect that the customer is specifying things like "no changes to PLC hardware" and "use MS IE web browser" but doesn't realise what effect this has on the system design or the final cost. I think you need to lay out several budget options and discuss them with your customer.

Option A: Doing it with a traditional SCADA system. OPC servers and/or data converters at each site. VPN hardware for security.

Option B: Giving you a free hand to do it your way with your choice of hardware changes (add communication cards to PLCs if necessary), software, web browser, etc.

If you can say something like "option A will cost roughly $100,000 more than option B" then that tends to put the results of their choices in a form they can relate to. I don't however think you want to be running around in circles like you seem to be now.

Oh, and you mentioned Omron. Their own protocol is fully documented and looks fairly straight forward. If the customer has enough Omron PLCs you might want to look into working with it directly instead of using OPC.

 

For GPRS with Modbus i can suggest these devices. http://www.violasystems.com/products/products_arctic_modbus_gateway.htm

I wouldn't use OPC at all. Use Modbus directly instead.

For the software part i would use. http://pvbrowser.org

"pvbrowser is an application framework. It provides a specialized browser for the client computer and an integrated development environment for creating servers that implement your visualization. It also provides data aquisition programs (daemons) for a lot of protocols that connect the real world with your server.

You can surf these visualizations as you do with an ordinary webbrowser. Many users from different places can use the visualizations at the same time. This can be limited to your local area network or across the internet."

pvbrowser is Open Source and runs on
Linux/Unix/OS-X/Windows

 

Oh... OPC... this could become a pain if the machines are not on the same domain (perhaps as a result of your distributed GPRS arcitecture). I would avoid OPC at a great cost.

Using OPC internally in a server though is acceptable, even though in general I try to shy away from it if possible. My (unfounded) reasons from not using OPC would be:

- Does not communicate on a port on TCPIP like normal services (that is it uses several ports and also requires windows security)

- Difficult to fault seek

- Is awfully slow in many use cases

- Uses a lot of bandwidth compared to for example Modbus

- Is tied to M$ Windows, in particular the non-realtime, non deterministic, heavy handed COM/Active-X architecture

- is not an open standard - membership of the OPC foundation has a substantial annual fee

- OPC servers for your distributed station have expensive licenses (imho for the limited functionality they give)

I would consider protocols like Modbus ethernet or for me personally look into something REST (http web) based. Perhaps even SQL to a remote database is viable.

 

After discussing some of the requirements with the client and re-thinking all the project, with the kind help of those of you who answered my questions, I've finally done this plan:

- For the connection to the plant PLC: OPC must still be used, I know of the advantages of using Modbus, but as the plants PLC's are different, and the prices of the different PLC protocols to modbus hardware converters are more expensive than an opcserver, I'm going to work with it.

I've been testing for the last two days this setup:

- a toshiba subnotebook running Debian Linux without starting X.org, with a VirtualBox running a windows XP OS in headless mode.

- The xp has been running a Iconics opc simulator with an OpenOPC Gateway.

- Using python, I've been asking a hundred of different data every 5 seconds for 36 hours from the Debian box to the virtualized XP box. The memory of the subnotebook is 1 GB RAM and there's not swapping at all, even if I give 512 Mb to the windows machine (I think that the opcserver + open opc gateway + windows stuff won't need more than 256 MB in worst cases). The cpu use has never been more than 2,4 % in the worst case.

So it seems it might work, and I can do all the programming in python in the linux machine, using XP only to setup the needed opc server.

- To communicate with the plant, I'll use non-safe internet connections using 3G or GPRS in most cases, I've done this from the subnotebook:

ssh -nNT -R 1100:windows_ip:7766 remote_control_center_ip

This has allowed me to connect in the control center machine to the opc running in windows just doing:

./opc.py -H 127.0.0.1 -P 1100 -o csv Numeric._BSTR Numeric._CY Numeric._I2

So, for real time I can connect from the remote linux server machine to the opc server, keeping the connection encrypted and not needing to open any incoming port in the plant machine connected to Internet.

So, my final plans would be:

- Use in the plant pc a small application that will log in a sqlite all the data from the plant. Periodically it will send this file gzipped to the control center via a ssh tunnel.

- Periodically the plant will query the control center if real time if needed, and in that case, it will open the above tunnel.

- When real time is needed the control center will connect to the plant to get the data and draw it in a web browser. A different local TCP port will be used for every plant.

- Whenever an alarm is raised, it will be sent to the control center.

Finally, I've convinced the client to use firefox to see the data (I'll try anyway to check if there is a way to be able to see part of it using the SVG Renesis Player).

So my plans are:

-adding a OpenOPC driver to MBLogic

-adding database logging capabilities to MBLogic using the twisted framework

- Drawing the plant and real time options using HMIServer

- Use CakePHP for the reports and all the database related stuff. I'll probably use some jQuery plugin or, maybe Chronoscope. http://timepedia.org/, to draw the charts.

Obviously, I won't use the modbus and softPLC capabilities of MBILogic

So, for M Griffin: I'd be very thankful if you can provide me the new system data table you've designed. In order to begin to work in the mapping of the OpenOPC readings to the modbus way you use, I don't think I'll need the new versions of MBLogic in a few days, but knowing the new model for signals, events and alarms would be a great help.

On the other hand, I'll send you the OPC and database logging file sources when I finish them. So you can include it in your project if you think they fit in it. I think that, even if Modbus is much more confortable to work, having an OPC connection will broad the possible uses of MBLogic. At least for those projects that don't need fast signals refreshs.

Regards.

 

> Periodically it will send this file gzipped
> to the control center via a ssh tunnel.

You might also look into what you can do with rsync.

> adding database logging capabilities to
> MBLogic using the twisted framework

Twisted offers a database interface through twisted.enterprise.adbapi

> So, for M Griffin: I'd be very thankful if
> you can provide me the new system data
> table you've designed.

I will release the latest version of MBLogic on Monday after I run the release tests. In short, in the old version the data table stored data in raw binary format and each module had to convert the data before using it. The new version stores data as lists of integers and booleans so no data conversion is required to use the data.

The module MBDataTable instantiates the data table. The class is defined in mbprotocols.ModbusMemTable. The doc strings in ModbusMemTable describe the data access functions.

If you want to talk about implementation details, there is a contact email address in the header of each source file.

> On the other hand, I'll send you the OPC and
> database logging file sources when I finish
> them.

That will be welcomed.

 

Considering that:

1. You need to use OPC Servers to communicate to the variety of devices you're faced with.
and
2. DCOM/OPC doesn't work so hot over GPRS/3G

have you considered using OPC Tunnelling software?

I've deployed solutions using MatrikonOPC's OPC Tunneller over GPRS, I presume it would work just as well over 3G.

Adriel Michaud
MatrikonOPC

 

Read this blog: http://controltoolbox.com/blog/2009/03/post.html

It discusses a similar situation. Using OPC or Modbus on GPRS consumes a great deal of bandwidth without that much of return.

 

As you requested, the new version of MBLogic was released late on Monday. You can download it from SourceForge. The project web site will be updated with new documentation, but the updates are already included in the on-line help system in MBLogic.

I will write developer documentation now that integrating new modules is easier, but if you have any questions in the mean time, please feel free to contact me.