I am looking for a solution for this case, I have been searching but maybe the keywords used are not the best.

Basically, there will be a few RTU's with backup battery and logging capacity, that will be monitoring temperature and/or digital inputs mostly. A gateway will poll the data from the RTU's and a HMI should display that data and store it to a database. The RTU's and gateway device are proprietary, but if needed, it can be re-programmed. Right now it offers Modbus TCP connectivity. The main issue is that when the network is down, the RTU's will continue log their data locally, and when the network is back up, that data should be stored in the database too. One of the specific needs is to know how the temperature behaves during a power outage. Is there a standard approach on how to deal with this case? Any commercial HMI+Database already has this built in?

Here is how I think it could be done, any recommendation/suggestion is welcome:

-As the data is not changing rapidly, the HMI can be doing normal polls at 30s/60s intervals, for example. When the network is back up, the HMI will continue it's normal scan to update the current values, and start in parallel a faster scanning process to poll the RTU's for their logged data, until all that data is in the database.

-Another way could be having the gateway getting the data from the RTU's and send it (CSV via mail or ftp, for example) to the server where the database is hosted, and a helper application will then import to the database.

I am looking at MBLogic and PVBrowser, I think that with both, solutions like above could be implemented. I haven't yet installed latest MBLogic that can log to database. If I use MBLogic should I use the same database or create a separate database for Historical Data? In PVBrowser I've seen mails in the mailing list about connecting to databases. A web interface is needed, and some special plots too (2D or 3D plots for temperature view). In PVBrowser I think I can use GNUPlot for the plots, and I'll have to check how it behaves with standard browser. MBLogic is web HMI, so that part should be fine. In both cases, with the data in a database, the plots could be generated with PHP and some plot library.

FWIW, my background is ANSI (plain) C for embedded controllers. I think I have the skills to modify demos or example code of C++, Python or PHP, but if what I need to code has less OOP, would be better. I don't know SQL, but will be learning the basics to have working demo for this project.

Nelson

 

Normal ANSI-C knowledge should be enough to do visualizations with pvbrowser.

But for database access we use the Qt classes which are in fact C++. There is an example how to do that within the archive
http://pvbrowser.de/pvbrowser/tar/pvbaddon.tar.gz
directory pvbaddon/templates/qtdatabase

There the files qtdatabase.h/qtdatabase.cpp are intended as templates for you own database functions.
With the given functions you can issue any SQL command to the database, where nearly all popular databases are supported.
Also you can populate complete tables within the pvbrowser window with one function call.

For graphics you may use gnuplot as you said but also other integrated functions for xy-plots are available.

 

You asked about MBLogic. I'm the author of that program. The logging currently in MBLogic is for alarm history and events. These are the same alarm and event messages that you see in the HMI. It logs these to a database both so it can restore the alarm and event history in the HMI after a restart, and so you can query the database via web client application (which is also provided). The database is integrated into MBLogic so there is no need to install a separate database for this.

However, it doesn't currently log arbitrary data values, it just logs alarm and event history. From your description, you also want to deal with pulling buffered data stored in RTUs. Those RTUs may buffer the data in different ways and you'll need to work with the RTUs in the manner dictated by the hardware you buy. Or another way to look at is you may need to pick your RTU depending on what you want it to be able to do.

What this means is that MBLogic doesn't at present do what you want off the shelf. If you want to modify it to provide the features you want (you could base this on the current alarm and event code) I can help you with that. There is also an extension mechanism called a "generic client" which lets you add features without changing MBLogic itself. It's currently used for additional protocols, but it is also intended to be used for things like adding custom data loggers. I can also help you with that if you wish.

If I recall correctly, the way that at least some RTUs work is that you can poll for live data using one method (e.g. via Modbus/TCP), and download historical data another way (e.g. via FTP). The way that some people deal with this is that they treat live monitoring and historical data logging as two different tasks. The live monitoring just deals with current data, while the historical data gets downloaded once per hour (or once per day, etc.). The reason for this is that many people are connecting over dial-up lines or mobile phone networks and don't want to pay the connection charges that would be required to maintain a continuous connection to pick up data points by polling for them one by one when in most cases nobody will ever look at them unless something goes wrong. They just connect once per hour, check the current status, and download the accumulated historical data (possibly as a CSV file). If the live status indicates that something has gone wrong, it sends a message (e-mail, SMS, etc.) to someone and also displays it on a summary screen.

I helped someone with a project that involved a large regional water system, but I didn't have any direct contact with his customer so I'm not familiar with a lot of the details. His complete application however involved a combination of a conventional web site plus the MBLogic HMI embedded in a frame in the web page. In this case he was dealing with a very large system spread over many sites, so a lot of the application involved consolidating and summarising information with the MBLogic HMI providing the graphical live component.

Part of your application design is going to depend on taking a step back and looking at things from an operator's perspective. If "a few RTUs" is half a dozen, then things don't have to be very elaborate. If you are talking about dozens of RTUs that have to be monitored and managed by one operator, then a lot of the effort will be in summarizing information in a way that fits this specific project.

As for knowing SQL, you probably won't need to know more than the basics. You will INSERT data to store it, and SELECT to query it. It's actually pretty easy.

For MBLogic I evaluated different types of databases based on what I thought the typical use cases were. I ended up using SQLite, which is an embedded database. It runs as part of the MBLogic program, so there was no need to install, start up, and manage a separate database process. The overall system ends up being a lot simpler and easier to use, and is more than adequate for the volumes of data (and queries) involved. The program can log directly to the database, and query access is provided through a simple web service. The database web service is used by the historical query client (which is just an AJAX web page), but it could also be queried by any other application, including a conventional PHP program that is able to do a simple HTTP GET.

What I have just described in the previous paragraph isn't directly usable in your application, but you could possibly use similar principles. That will however depend on how much data you are logging. If you are logging a dozen bytes of data every minute, that amounts to less than 20 kilobytes per day, or 6 to 7 megabytes per year. If you are logging kilobytes of data per minute, then that amounts to hundreds of megabytes (or even several terabytes) per year, and you probably have to look at something like Postgres. You probably also want to look at winnowing down the data so that you only log changes.

If you want to talk about this further, let me know and I will be happy to discuss it with you whether you use MBLogic or not. I can also give you a direct e-mail contact if there are confidential details that you don't want to discuss here.

 

"If I recall correctly, the way that at least some RTUs work is that you can poll for live data using one method (e.g. via Modbus/TCP), and download historical data another way (e.g. via FTP)."

I've done something similar myself, and in another case I have created separate Modbus tables in the RTU, a smaller one (40001-40099) for current data and configuration info, and 40100-40999 for circular buffer with logged data. This way everything can be done with only Modbus, but it may not be effective with a larger number of RTU's. For the demo it will be 5 RTU's with less than 10 points per RTU, the final application would be more or less like you describe with the water system.

Regarding the database, I've also thought of SQLite to start with. I'll have to think about how to handle the final solution. I know of a case where a colleague created a single database where all the data of various customers were stored, until the db started to give errors.

I might go with PVBrowser now as I may feel more "comfortable" with the C programming, but I think I'll use MBLogic to simulate the Modbus Slave, if I the data table can be modified dynamically during runtime.

Thanks to both MGriffin and pvbrowser for the answers, I'll post specific questions to both on the respective forum/mailinglist