real-time database

  • Thread starter Zheng YiBing, ABB Bailey Beijing
  • Start date

Thread Starter

Zheng YiBing, ABB Bailey Beijing

Hi, list,
Is there anybody who know what real-time database product can support 52,000 points ? Maybe PI, or something else?

ABB Bailey,
Zheng YiBing

Sage, Pete (IndSys, GEFanuc, Albany)

Certainly PI is a well-respected high-end historian.

Not understanding your requirements, its not clear if it is an overkill solution.

Are you logging 52,000 points every 100ms, once a second, minute, hour or day?

How much data do you plan on leaving in the database - an hour, week, month, year, decade?

What sort of reporting / trending / analysis tools do you require.


"Lurker John" G. Boland

Hi, Zheng,

You ask the automation maillist about a:

<< real-time database product can support 52,000 points ? Maybe PI >>

Yes, I would check with OSI Software (PI).

A word of caution:

Do NOT be impressed with how many points anyone's database can insert or update per second. A lot of these companies have focused on this issue and
have missed a REALLY fundamental issue:

Databases are useful because they provide single-insert / multiple use of data. If data is inserted at, say, 5000 points per second, then it MUST be possible to query and extract data from the database at several times that rate, say, 15000 points per second. That is logical, right?

ASK your potential vendors about that! They may not even be able to ESTIMATE the query response!

Also, even with PI, data retrieved from a compressed historian archive is evenly-spaced interpolated values, not the original raw data. That is necessary to achieve the compression, but it means that your queries will not be "standard" SQL, since they must specify things like the desired spacing interval.

Oh, and another thing...

Many databases do not permit "null" or "blank" entries, so if your SCADA or DCS does not have data for some variables during an update interval, what will the database store? The last value? A pre-set value? Your queries must
filter the returned data, since the database WILL return evenly spaced data, even though there might not be any "real data" in the database. Your
application may then average (for example) these phantom interpolated datapoints...


Good luck. Please let me know of your progress!

"Lurker John" G. Boland, president
VisiBit Corporation
One Parker Square Suite 408
2525 Kell Boulevard
Wichita Falls, Texas 76308

Raesemann, Robert C.

PI is my favorite.

What are you trying to connect to? How many interfaces are you going to have? What type of interface are you planning to use. 52,000 points is a lot of data and the architecture is going to be very important also. What are your requirements for update speed and such. What type of data types will you be looking at? Is it all analog or are some of them digital?

Robert Raesemann
Sea Coast Diversities, Inc.
[email protected]
office: (904)825-0383
cel: (904)613-5988

Romero Espinosa Fernando (Teniente)

Dear Zheng YiBing:

PI System (OSI Software Inc.) can support 52,000 real-time points...and more!, in the most efficient way I have ever seen.

Fernando I. Romero
Control System Engineer
Codelco Chile - Division El Teniente

Todd Johnson

take a look at the National Instruments Lookout SCADA software. It sports a fast historical database, Citadel.

Zheng YiBing

1. 52,000 tags is required, and refresh rate is less than 10s.
2. All tags will be logged every 1 minute.
3. Decade historical data will be stored in the database.

Do you think PI ( or other real-time database) can do this ?

Thank you,

ABB Bailey,
Zheng YiBing.

Keith P. Bouvier

Hello Zheng,

I agree with many of the points which Mr. Boland has raised about the use of PI. I also believe it is a very good and efficient tool for the long term storage of data. My company has utilized the PI product for our clients long term data needs and used Agilent's (formally a part of Hewlett Packard) RTAP product (Real-time Application Package) to couple in real time capabilities (Input and Output control) with PI's long term capabilities. If you are looking for real time control with long term storage you may wish to visit these products.

As I have stated earlier, I agree with much of what Mr. Boland wrote. You must determine your real time requirements verses your long term storage requirements and find a solution which best fits these requirements.

Here is a little information on RTAP. The RTAP product is a tool kit which allows you to construct points based on your needs. In other
words, the database capacity for RTAP is 65,000 points. Each point has the ability to scan in 256 data elements from end devices. The product also has a large suite of communications
protocols available which can poll at sub-second rates. The nice thing about the product is its ability to be easily tailored to your needs both in acquiring data and the presentation of the data.

Good Luck

Keith P. Bouvier | Voice: (504) 889-2784
Computerized Processes Unlimited, Inc. | Fax: (504) 889-2799
4200 S. I-10 Service Rd., Suite 205 | E-mail:
[email protected]
Metairie, LA 70001 | Web:

Steve O'Donnell


Just a minor correction re compressed data within the PI Archive. It is possible to access the compressed or interpolated (evenly spaced) data from PI, it is just dependent on which data access call is used.

Steve O'Donnell
Managing Director
OSI Software Limited
PO Box 8256, Auckland, New Zealand
Ph. 64(9)307-1417 Fax 64(9)307-1418
Email: [email protected]
Another consideration for some users is the capability of the historian to accept data out of time sequence and deliver it in time order when
requested. Why should this be necessary? Well, systems can have delays that are significant to the data rate. If one is reading data at 100Hz and up and some is coming over a network optimized for throughput, then the network buffers can result in remote data (time stamped of course) being out of time order with local data.

For other types of systems, the data rates are 1Hz or less, but because they are widely distributed, the communications can be interrupted and data stored remotely until the communications are restored and the data
downloaded. It can even be that the remote sites are polled every once in a while for accumulated data.

Some historians simple can not cope with this, attaching the time stamp of the historian computer to all data. Other historians can accept time stamps with the data but there is a considerable overheard in accepting such data,
greatly reducing the throughput of the system. Others take such data in their stride.

Peter Clout

Peter Clout
Vista Control Systems, Inc.
176 Central Park Square
Los Alamos, NM 87544-3012
(505) 662-2484
FAX (505) 662-3956
[email protected]

Robert Raesemann

PI in a well designed system can definitely handle the amount of data and the storage requirements. You will need to take a lot of
factors into your design.

I have had problems in the past pumping enough data through interfaces to get it into PI. If, for example, you are using a Bailey CIU at 19.2k then you are limiting the amount of that you can
export from the system. An older system probably couldn't scan all 52,000 tags every 10s. You'll have to make sure that you are starting with an architecture that can pump out what you are
planning to archive.

Once you can get the data into the system fast enough then you have to make sure that you have a server to handle the load. Memory, processor, and especially disk architecture become crucial. If you are going to have a lot of users hitting the server then network performance will also be an issue. As far as disks go, I would definitely spin the $$$ and go with a SCSI RAID array of
10k RPM drives. The OSI folks can help you with the sizing of the RAM.

A decade of storage for 52,000 points is a lot of data. The amount will greatly depend on your tuning of the points. If you require very
accurate high resolution data then you will obviously need a great deal more storage. Drives are cheap these days but if you are going to get much beyond about 200-300GB then it starts to get a little more expensive because you will probably need to move to some sort of external array. If you can handle moving some of the older data offline then it makes it cheaper. Keep the archives under 650MB so that you can store them on a CD. Of course if you start collecting data today, drives will only get bigger and cheaper over the next ten years. I wouldn't worry about having more than 2 years of capacity lined up to start.

Robert Raesemann
Sea Coast Diversities, Inc.
[email protected]
office: (904)825-0383
cel: (904)613-5988

Blanco, Felix

I have experience in both historian data base. see my comments:

I.- Infoplus.21 (my favorite one).

- 52K tag: it is designed to do that.

Schaminee, Bart (IndSys, GEFanuc, Nether

Hello Zeng,

Do you have a demand on how often your process requires an update? I really think (re)compressing is not required here. Most of the times our customers are satisfied with a database like SQL.
SER= Sequent of Event Recording is more a Process Controller issue nowadays, since processors can handle it. In case of trouble the controller send it via TCP/IP or Web to the

The question to you is: how fast does your sampling need to be? It might be that PI gives you an ad-on. Let us find out!

Kind regards,

Bart Schaminee
It is "no fun" when the programs use averaging or evenly spaced data to get back the values. It can really mess systems up.

I've used Wonderware IndustrialSQL Server for some time and it gives you back ALL the values you have logged. It doesn't lose a single value. But as Mr. Boland stated you need to use "non-standard SQL" to do it. Typically design a SQL statement such as:

SELECT DateTime, TagName, Value
FROM v_AnalogHistory
WHERE TagName = 'MyTankTemp'
AND wwRetrievalMode = 'Delta'

The "retreival" part of the query makes InSQL bring ALL values back. If you want it to return the data in evenly spaced time interval you have to specify 'Cyclic' instead of 'Delta'.

In the applications I've designed sofar it has been easy. InSQL gives back NULLS if there has been no logging at all.

If it can handle 52000 points per 10 seconds, I do not know. The largest system I've designed logged 3500 values every 10 seconds and that was a "no sweat" application with moderate hardware.

Lars Ericson