Filter algorithm for fast, noisy signal

T

Thread Starter

Townsley, Bill

Hi all, I have two signals that are very noisy and quite fast. One is viscosity and the other is density. These signals are currently sent to
strip charts where the Operators get a 'feel' for what the signal is doing and adjust accordingly.
We are going to be moving these two signals into a DCS and would like to apply a filter to them to give a more useful trend. I could build a simple
time-averaging routine but I was wondering if there are any better (maybe faster) methods of achieving this.

Any help would be appreciated.

Thanks,
Bill.
 
N

Nilesh Pradhan

A sound approach would be the following:
1) Identify the maximum freuency of the
input signals: signal obtained from the density and viscosity transducers. This could be theoritically analyzed by the process engineer.
Make sure that you are sampling at least
twice that frequency.
2) Perform a frequency analysis of the signal
and observe for any specific peaks that may
be arising due to the environment.
3)Since you are using a DCS, you may
well use a digital filter: decide upon an IIR
or FIR depending upon the requirement. For this
application, it seems that an IIR is beneficial.
4) Your time averaging rotuine is nothing but
an FIR filter, with all weights equal to 1 and
length dependent upon how many samples you store.
An obvious choice is an IIR design.


Hope this helps.

Nilesh Pradhan
Graduate Student.
 
J

Johan Bengtsson P&L Automatik AB

Hmm, filtering of fast signals is a really interesting topic....

Here are some alternatives that I know of:
1. Average, take a lot of fast samples and use the average of them as one sample.
2. Average, take a lot of fast samples and use the average of the last n samples as the current sample, replacing one sample at a time.
3. Mean value, similar to (2) but you sort all the values and take the one in the middle (or the average of the two in the middle if you have an even number).
4. High order lowpass filters, can be made by using several 1st order filters.

I would suggest using one of (1) - (3), I would prefer (3) but that might depend on the signal. Followed by several normal lowpass filters each set on a short time (same for all filters).

for a sudden change the filters will do approximately the following:
1. The signal will change in two steps during the two following samples out from the filter, the output from the filter is a signal sampled with
a low frequency with all that mean.
2. The signal will change with almost constant speed to the new value
The total time will match how long one sample remains in the filter.
The response starts immediately but lasts for the averaging time.
3. The signal will jump after half the lifetime of one sample in the filter. Ie you insert a dead-time but individual high and low values will be completely removed from the signal.
4. One lowpass filter gives you the classical first order response where 63% will be reached after the given filtertime. When combining several
filters you will get a delay (more and more like dead-time) and a steeper response. The sum of the filtertimes will roughly give you the "delay"
if you accept the delay as the time when about 60-70% on the new value is reached.

Two lowpass filters will kill more noice and less signal than one if the two filters have half the filtertime, but you insert a dead-time.


It would of course be possible to combine (2) and (3) in one filter like this:
- Save a number of samples, for example the last 9 samples.
- Sort them.
- Output the average of for example the tree samples in the middle.
This will give you a filter combining the strengths and weaknesses of (2) and (3) and might be better in your case.


Do not put any of (1)-(3) after a filter of type (2) or (3) (4), it won't improve the signal. Exception: a small low-pass filter built in hardware before or in the input card. After the signal is first sampled use one of the filters (1) to (3) but only one of them (or one using the above mix of (2) and (3))

As I said, this is an interesting topic and I will inverstigate it further when I can find the time....
I am definitely interested in studying other peoples resonses and in what you finally decide to do with the signal.


/Johan Bengtsson

----------------------------------------
P&L, the Academy of Automation
Box 252, S-281 23 H{ssleholm SWEDEN
Tel: +46 451 49 460, Fax: +46 451 89 833
E-mail: [email protected]
Internet: http://www.pol.se/
----------------------------------------
 
R

Robert McDonald

I have similar needs in projects we develop. So I too will be interested in the responses you get.

Our situation is slightly diffrent. We measure force and displacement as we cycle a motor vehicle shock absorber through test strokes and various velocities of linear motion.
We currently us a simple floating average filter to clean up the signal somewhat before analysis however I expect there are far better
solutions.
One problem we experience is the different time lags or phase lags we experience between (a) the two signals and (b) at various velocities.
These lags cause problems in aligning the force data to the displacement data. Any fixed offset means of aligning the data has proven unsatisfactory given the velocity changes taking place. But presently it is the best we have found. So any clues would be appreciated.

Thanks,
Robert.
 
W
Bill,

There are plenty of resources on the web about filtering. Here is a page with an interacive demo that lets you try different coefficients for a
windowed FIR filter:

http://web.mit.edu/6.555/www/fir.html

Analog Devices had an app note (I don't know if it's still available) called "Making the FIR fly"; although this is for DSP applications, it may have some useful info.

For what it's worth....

Cheers,

Willy Smith
Numatico SA
Costa Rica
 
J
Dear Bill,

We have applied the following technique to process such "difficult" signals:

Fill an array of 9 floating point values (say NF[8]) with raw datas.
Put the most recent data in position 0 and then shift the array (NF[1] = NF[0]...NF[8] = NF[7])
Wait till you have 9 samples available.
After the 9th is received your array is full
then, and only then, apply the following formula

Filtered Value = (-21 * NF[0] + 14 * NF[1] + 39 * NF[2] + 54 * NF[3] + 59 * NF[4] + 54 * NF[5] + 39 * NF[6] + 14 * NF[7] - 21 * NF[8}) / 231

(231 is the sum of the coeficients)

Then shift again to eject the oldest data value from the array.

Based on "Savitsky-Golay" filtering technique described in "Numerical Recipes in C" (Cambridge University Press) http://www.nr.com

For this to work, you have to wait till 9 samples are in, which is maybe too long for your application.

Anyway, I hope this help.

Best regards
 
J

Johan Bengtsson

Hmmm, I just got a thought..... No I have not tested this - yet....

Do like this, but before you apply the coefficients and sum it up you sort the values.

(Note: when you shift in a new value you should still shift out the oldest.) This will be a very improved variant of the mean value filter.

I'll test it when I can find the time...


/Johan Bengtsson

----------------------------------------
P&L, the Academy of Automation
Box 252, S-281 23 H{ssleholm SWEDEN
Tel: +46 451 49 460, Fax: +46 451 89 833
E-mail: [email protected]
Internet: http://www.pol.se/
----------------------------------------
 
R

Rodney Stevens

Robert

All filters or averagers will give a lag to the signal.

If you use the same filter for both signals then the lags for a given frequency will be the same. If you use different filters for each signal then they will be different.

Some filters have better phase characteristics than others, but not being an expert on these I would not like to make any rash statements as to the best filter to use in your situation.

You can use correlation techniques that will identify the lag between two signals and use this to dynamically remove the lag.

Rodney Stevens
CSIRO Minerals
http://minerals.csiro.au

Ph. 61 2 97106701
Fax 61 2 97106789

Elan SS s/e 45/7616
http://sites.netscape.net/rodjohnstevens/homepage
 
> Hmmm, I just got a thought..... No I have not tested this - yet....
>
> Do like this, but before you apply the coefficients and sum it up you sort the values.
>
> (Note: when you shift in a new value you should still shift out the oldest.) This will be a very improved variant of the mean value filter.
>
> I'll test it when I can find the time...
>
>
> /Johan Bengtsson

Do NOT sort the array to implement this filter, positions within the array are important for the underlying theory (kind of polynomial fitting, 3rd order here). You need to maintain the "history" of the signal to apply the proper weighted coeficient (for instance the oldest and newest have a negative weight. If you sort, this will be messed).

Some extras:
If you put the filtered values into another 9 points array (say F[8]) and shifting to fill it up you can get almost for free the 1st and 2nd derivatives of your signal.
This will tell you when a "peak" begins (1st derivative) and when a "turn" happens (2nd derivative).
Compared againts appropriate limits you can then activate say an alarm or whatever.

Here it is:
1st = (86 * F[0] - 86 * F[8] - 142 * F[1] + 142 * F[7] - 193 * F[2] + 193 * F[6] - 126 * F[3] + 126 * F[5]) / 1188
NB: F[4] is unused.

2nd = (28 * F[0] + 28 * F[8] + 7 * F[1] + 7 * F[7] - 8 * F[2] -8 * F[6] - 17 * F[3] -17 * F[5] - 20 * F[4]) / 462

Cheers.
 
P

Preston Todd Johnson

While you certainly can use a average (mean) to filter your data, the noisy (and error) will affect your average. You might consider taking the median of say 10 data points at a time. This has worked well for me in the past.

Todd Johnson
 
R
> While you certainly can use a average (mean) to filter your data, the noisy
> (and error) will affect your average. You might consider taking the median
> of say 10 data points at a time. This has worked well for me in the past.
>

I have had pretty good luck just doing a weighted, moving average of the last ten signals. I usually select the last ten values and then weight them 1-10.

I.e. - the oldest signal has a weight of one and the newest signal has a weight of ten.

In one case, I found that this just didn't work because the averaging needed to be done over a longer period of time so I averaged 10 values, saved that as the newest value, and then did a weighted, moving average of the last ten of the averaged values.
 
D

Darold Woodward

The rolling average described below is a good approach.

You can also use a PID controller in the PLC as a "filter" by using the signal from the field as the setpoint. You can adjust the tuning to achieve the desired damping. The gas gauge on your car uses something similar to achieve "anti-slosh." Some PLCs also have an analog function for this. Some instruments also have functions for this, so you should look at the instrument also.

Darold Woodward PE
SEL Inc.
[email protected]
 
J

Johan Bengtsson

Of course the positions are important for the underlying theory, and of course this will be messed up if the array is sorted before applying the coeficients, and of course it will be a nightmare to evaluate this theoretically, but the big question here is still: will it be a better filter or not?
I am going to test it and compare it someday just because I am curious and because it might be better even if all the underlaying filter theories in the world fall apart.


/Johan Bengtsson

----------------------------------------
P&L, the Academy of Automation
Box 252, S-281 23 H{ssleholm SWEDEN
Tel: +46 451 49 460, Fax: +46 451 89 833
E-mail: [email protected]
Internet: http://www.pol.se/
----------------------------------------
 
S

Simone Stefani

Try to use the follow formula:

This function provide to smooth a analog input.
The function calculate the smooth value by add S/16 of previous value with (16-S)/16 of actual value

the formula used is

Out =
([Smooth_factor]/16)*[Prev_value]+((16-
[Smooth_factor])/16)*[Input_val
ue= ]

S = smoothing factor
X1 = initial value/previous output
X2 = new input from I/O table

Valid entries for S smooth factor

If you want to..... then enter.....

Prevent the smoothing process 0
Provide minimal smoothing 1
provide maximum smoothing 15

---oOo---

You can download the routine (in PDF format)from the PLC programming archive. Search for FC5.PDF

http://www.control.com/control_com/PLCArchive

Simone Stefani
Automation Dep.
Romaco Zanchetta
Tel. +39-(0)583-2171
Fax +39-(0)583-217317
e-mail: [email protected]
 
S
Bill, there have been several good posts to you request and I am not necessarily disagreeing with any of them.

The book "Numerical Recipes in C" was already referenced; it is excellent. The book "Discrete-Time Signal Processing" by Oppenheim & Schafer is another I have used with success.

A few thoughts:
1) First, you MUST limit the high frequencies coming into the software with some form of hardware filter. Your software samples MUST be taken faster than twice the highest frequency in the signal (from the Nyquist sampling theorem). *NO* software algorithm will be stable if there are significant signal levels above the sampling frequency - you will get beat frequencies and many low pass digital filters will pass or even amplify high frequencies (relative to the sample rate). I strongly urge that you take some data
points using your DCS, move them to MathCad or Mathmatica and do the suggested FFTs on the signals. You may have to add the suggested
resister/capacitor hardware filter before this data will be useable.

2) You must have a (relatively) stable and even in time sample rate for any of this to work. Signal processing of variable-in-time-data-samples is *VERY* difficult to impossible and may yield results that are ok sometimes and really bad at others.

3) Averages of averages are the same as longer averages. A digital filter of filter data is the same as some digital filter. Unless there is some
specific implementation detail that demands, don't mess with "take an average of ten samples and then average ten of these samples" types of
algorithms. They can be needlessly complex and confusing.

4) How much CPU time do you have in your DCS? The floating point calculations in some weighted averaging schemes can be significant. The below suggested method can be VERY efficient.

5) As noted by someone else, all filtering will induce a phase lag, or delay to the signal. Since the two signals are (I assume) measuring the same stuff, make sure the two signals have the same phase lag (i.e., same filters) or you will be attempting to control based upon density and viscosity at two different points in time -- I doubt you will have much success with that!

6) If you come from the old "Keep It Simple Stupid" school, don't do more than needed. A widely and successfully used digital filtering technique called exponential smoothing is easy, cheap, and often gets the job done. You take a sample and call it Value(n-1). And use it from the very beginning as the value for the first time step if you need it. Then, at each time step (sample interval), take another sample Sample(n) and do the following computation:

Value(n) = (Sample(n)) * (A) + Value(n-1) *(1-A)

where Value(n) is the newly computed value to use in your control algorithm,
Sample(n) is the new measured sample value at this time step,
Value(n-1) is the value of the last time step, and
A is a magic value between 0 and 1.

The value of A (often called "alpha" in descriptions of this filter) determines the response time of the filter. The closer it is to 1, the more the latest sample affects Value(n). The closer it is to 0, the more the old value affects Value(n). Values of 1/10 to 1/30 (1/8, 1/16, 1/32) will usually "quiet" very noisy signals. Larger values are ok when needed.

Note that someone else proposed this with an A of 1/16 in a previous post.

Note that this is a weighted average with (theoretically) infinite terms whose weights are exponentially smaller (as the (1-A) are repeatedly
multiplied together). The smaller A, the more old samples are included in the reading and the more they influence the result, the "quieter" the output signal.

Note you only need to store 1 value, the value used during the last time period.

Note that only one add and two multiplies are needed; further, simple algebra can sometimes manipulate this into forms that require even less
computations.

Note that if you are doing integer arithmetic (in an embedded microprocessor) the selection of A to allow shifting (a fractional power of 2) makes the computations even less.

Note this filter does have a habit of passing (but not amplifying) frequencies that are above its Nyquist sample rate, so be sure to use a
hardware front end filter.

Note this filter introduces a linear phase lag. In at one system I was able to determine this phase lag and effectively remove it from the system aligning the Value(n) with the current point in time and improved the system control considerably, but that is beyond the scope of this already too long post.

Simple. Cheap. Stable. Reliable. Not the answer to all digital filtering needs, but exponential smoothing is effective in many, many situations.

Cheers! and happy filtering!
Steve

----------------------------------------------------------------
Steven B. Cliff title: VP, Research & Development
Control Technology, Inc. email: [email protected]
5734 Middlebrook Pike web: http://www.controltechnology.com
PO Box 59003 voice: (865) 584-0440
Knoxville, TN 37950 fax: (865) 584-5720
 
T

Townsley, Bill

Thanks Steve for a very informative tutorial on filtering... it's been a few years since university...

I don't know exactly what I am up against yet since this application will not be implemented for another 4 or 5 months. When I heard the site guys explain that they wanted the new DCS to incorporate these two signals (possible as control variables!!!), I started to get a little concerned. Hence the posting - I figured that I would need all the help I could get.

It sounds like the exponential method might be a very good one to try first. Currently, the operators look at the 'trend' on a pen chart.

Thanks all for the many postings. I think I have some pretty good information here. At least enough to tackle this one with some confidence.

Thanks again.

Bill
 
Top