Signal Trending

B

Thread Starter

Bill

I am trying to monitor and trend a signal that has a scale of 1 to 1000. Typically, the signal will trend from about 40 to 280 over a 10 minute period. Periodically, spikes 2 to 4 times the normal signal occur. I need to filter out this signal spikes from the input signal so that the trend is more uniform. I know there has to be a mathematical formula that will allow a range of values to pass and filter those out of the range. But, I also need the range to shift so that the trend is accurate. Any mathematical ideas?
 
S

Steve Myres, PE

Log the last 10 data values in a shift register (FFL+FFU in AB) and trend on the average of the values. I've done this many, many times and it works well.

Simpler but a little less accurate: multiply previous trend value by 9, add current sample, and divide by 10 (or some other smoothing factor).

In either approach, the trends will take a little longer to show, and will be understated, but will be much smoother. This is be the equivalent of looking at a 50 or 200 day trend on a stock chart. Much less responsive, and therefore smoother but also slower to react to changes in the real time data.

Or, you could just define a spike as any value more than 2% (or some factor) different from the trend and ignore it. After you return to normal data, you can interpolate, if needed, and fill in the empty data point(s).
 
R
You could use a running average to do this, and throw in some limiting. If these values of 2 to 4 times the normal signal aren't important or are just spurious signals, and you really don't need to worry about them (I hope nobody's life depends on it) you can do the following:

- Set up a circular buffer big enough to hold the number of samples you want in your average.
- As each value comes in compare it to the last value of the running average you calculated. If it's grossly out of range, throw it out and wait
for the next sample. If it's reasonable, divide it by the number of samples in the buffer and write the result in the next location of the buffer. Sum the values in the buffer, and this will be your running average. When you reach the end of the buffer, go back to the beginning again and overwrite the oldest value in the buffer, without disturbing the other samples in there, and repeat the above.

hope this helps

Rick Daniel
www.instrument.com
 
C

Curt Wuollet

If these readings are truly impossible and don't reflect reality, I have simply checked for more than a reasonable change between readings and
replaced the flyers with the average of the X preceding and X following readings. If you do this to every sample, it's smoothing. if you remove just the flyers it's interpolation. You'll need a FIFO or ring buffer for continuous data so you have access to this window before and after.
I usually put the window and delta limit in a header file so they are easily checked and tuned.

Regards

cww
--
Free Tools!
Machine Automation Tools (LinuxPLC) Free, Truly Open & Publicly Owned
Industrial Automation Software For Linux. mat.sourceforge.net.
Day Job: Heartland Engineering, Automation & ATE for Automotive
Rebuilders.
Consultancy: Wide Open Technologies: Moving Business & Automation to
Linux.
 
F
Bill,

The simplest formula for this is the moving average, where for every data point you compute the average of the last N data points. The greater N is, the more smoothing you get.

A slightly more complex method is to use a FIR (finite impulse response) filter, where instead of a simple average of the last N points, weighting factors are applied to each. In chromatography, filter coefficients to fit M-degree polynomials are popular. For more info, search the net for "Savitsky-Golay".

A lot of software, including Labtech, has built-in functionality to do this kind of filtering. You can play with this by downloading Labtech Notebook or Control at "labtech.com":http://www.labtech.com . Run the "Tutor 2" example to see a moving average example. For the FIR filter, use the FILTER(X) calculated function block with the Savitsky-Golay coefficients supplied in the help info.

Regards,::

Fred 3/27
--
Fred A. Putnam
LABTECH, Suite B9, 2 Dundee Park, Andover, MA, 01810
tel: 978-470-0099 x231, cell: 978-885-8080, FAX: 978-470-3338
 
Rick:
> You could use a running average to do this, and throw in some limiting.
...
> - Set up a circular buffer big enough to hold the number of samples you
> want in your average.
...

The simpler way to do a running average is to do an exponential-decay one. As each value comes in, do the calculation:

avg := g*value + (1-g)*avg

g is between 0 and 1 - the traditional value is 0.1 (10%), because then it's really easy to do by hand, but you can use anything. There are
formulas for effective length of window, but it's probably better to just put it in a spreadsheet and see what you like.

Like Rick said, if you want to exclude the spikes from the average, just don't do the calculation for them; or you can count them with a much
smaller g.

One last suggestion - somehow keep track of how many values are ``spike'', and how many in a row. There's a story about NASA missing that there's a
hole in the ozone layer because of just such a calculation - the values were too small and got thrown away as errors...

You can do that by the same formula as above, only instead of value you put in 0 if it's OK and 1 if it's a spike. Probably use a smaller g, though. Then you can alarm it, or display it, or use it to widen the band. The meaning is "percentage of spike values (scale 0-1)".

Jiri
--
Jiri Baum <[email protected]> http://www.csse.monash.edu.au/~jirib
MAT LinuxPLC project --- http://mat.sf.net --- Machine Automation Tools
 
Well guys, I am thankful for all of the ideas that you gave me on this problem. Just to let you know what I did, here it is.

I created 2 variables, one a loop factor and the other a range factor. The signal that I am trying to trend is monitored every 200 msec. The loop factor determines how many of the 200 msec. scans I want to capture data in. If I capture 10 scans then I divide the sum of the signal by 10. This basically gives me a running average during 2 seconds. During each scan, I use the range factor to filter out the spikes. If the range factor is set for 5 then any signal + / - 5 of the current average signal is simply ignored for that scan. I've simulated this and it looks good. However, if the signal experiences a sudden change that is outside the range for long periods time then the signal trending is flat. I may have to accept a
value that is outside the range if no good signal occurs within a specific time.

Thanks for everyone's ideas.
Bill
 
Top