Real Time Sensor Data Acquisition Over Web


Thread Starter



I am planning to launch a new product in the market which acquires mV signals from sensors, convert it to digital data, and sent to a Web server via a GSM or Wifi Module. The web server receives the data in real time and do necessary processes including SMS alert.

I have my own web server operating on CentOS 6.7 Linux Operating system and a dedicated IPV4 Address associated with it.

Please advise me what protocol is to be used for my above requirement, and is there any open source linux applications for real time data acquisition. .


Control Observer

Send the data to the web server in an http or https POST message, with the data encoded as JSON in the body of the message. It's the most common method of data communications over the web so you should be able to find lots of examples of how to do that.

If there is some very strong reason to not use http/https, then use some other very well established and widely used web protocol (e.g. XMPP). However, you will have the fewest problems with firewalls and other such infrastructure if you use http/https.

Do NOT use an "industrial" or "roll your own from bare sockets" protocol for this. You will end up going through a lot of useless effort of re-inventing the wheel for no reason, while only getting it wrong in the end.

Use a well established web protocol and you can use standard web technologies on the web server to solve all the problems of designing the security and authentication components.

On the server, use something like Python, NodeJS, Ruby, etc. There are frameworks such as Django, Web2py, etc. which can provide a lot of the boilerplate, but it's not possible to recommend one without knowing what the overall application needs to do.

If you are POSTing http/https messages from the sensors to the web server, then there's really nothing special required to receive the data. It's just POST data just like dealing with a web browser. Most systems do things this way, including things like mobile phone apps, "Internet of Things" hardware, remote monitoring hardware, etc.

If the sensors need to read information from the server, have them do it via http/https GET messages, again with the data payload encoded as JSON.

I'm recommending JSON data as that is the most widely used, but you can use XML or just separate ordered data by lines, commas, spaces, or whatever suits your fancy. JSON format would be the most flexible and the least work though.

The data reporting and system administration parts of the server will be most of the work, not getting the data.

In the data acquisition hardware, the messages can be constructed using string templates, since the basic message will remain the same with just the values changing.

You will need some sort of authentication/security for the communications, so you will want to look at username/password or some sort of cryptographic keys. Again though, this is a solved problem so look at something comparable and see how they do it. It does mean though that your acquisition field hardware will need to be configurable to allow the user to store the username/password combination (or whatever you use).

The reason why you won't find much information about how to do "industrial" data communications over the web is because there is no reason to do it any differently than how 99.9% of the rest of the world does things. It's simple, secure, reliable, flexible, and it scales up to Google-like sizes.