ADC-improving stability by math

[KTarke]

This is an idea / a project.
Haven't yet even thought if it is possible with Picaxe, but still I tell what I've been thinking today...

Very often, when reading a sensitive sensor with ADC, one gets quite many "wrong readings", too.

Normal way to smoothen things is to take average of, lets say, of 50 readings.

Anyhow, if electrical disturbance is great, one gets (only) SOME readings, which differ A LOT from average readings. Therefore result from "average of 50" is not "good enough".

There IS a little better way, called "median averaging". Procedure is done like this:
1. Take some readings an stuff them to a STRING
2. Use "bubble sort" to arrange readings to ascending order
3. Take normal average of "most" readings in the middle (= drop the lowest and highest readings...)


This is a effective way of "losing" those "certainly wrong" readings, caused by electrical disturbance.

I've done this a few times in the past, but with a pc-program. Pc-Basics can handle strings... Picaxe-basic does not?
Ofcourse, sorting can be done many ways, and bubble-sort is ineffective, but the code is quite straightforward and simple... If someone knows a way in Picaxe, maybe using peek and poke to memory, to "simulate" strings, this could work.

Things to consider:
-does ineffective sort-code take "too long" (depends on application, ofcourse)
-If string-handling is done by peek/poke, it should be done into RAM, in the long run EEPROM does not like it...?

 

[hippy]

Unoptimised.

 

[LizzieB]

There IS a little better way, called "median averaging". Procedure is done like this:
1. Take some readings an stuff them to a STRING
2. Use "bubble sort" to arrange readings to ascending order
3. Take normal average of "most" readings in the middle (= drop the lowest and highest readings...)

That calculation is usually called a "trimmed mean" if anyone wants to look up the math, and it is good for this type of application.

 

[Dippy]

Yes, I use a similar procedure in some PIC ADC work, but for whipping off highest and lowest you don't need to bubble sort. You can do Max and Min in a loop. Iterate it if you want to.

Note: Many people will understand it more clearly as 'arrays' rather than 'strings'.

 

[BeanieBots]

Ignoring MIN and MAX is certainly one of the most common methods and as described by Dippy, it does not need the use of any sorting.
Another approach often used in instrumentation is to ignore any readings which are outside a predicated range. How this is done depends largely on what your sensor is measuring. If for example it is temperature, then a little knowledge of the system would let you know by how much it is possible to change in a given period. Any readings which have moved by more than that amount are ignored.

 

[Dippy]

Absolutely.
Rejecting 'sillies' or things outside x Sigma are common methods. Obv sample size is important with the latter which can be a little inconvenient without arrays or large space for variables.
Plenty of ways to skin the proverbial cat without need for bubbling. All of these methods and many more have been used for many years.
But it is good to discuss and ponder over them.
Ultimately it is Horses For Courses wrt necessity, processor 'size' and, of course, speed.

 

[KTarke]

In this case, sensor gives readings according to rapidly changing values in vacuum (very low, area is 0-10mmH2O).
One can not EXPECT any result, before taking an amount of measurements...
I can't see, how to use max and min in case, where "valid" adc-readings could be for example between 0 and 300. If set max =300, and getting 299 when ~25 is expected, the 299 "wastes" the "normal" averaging.
That is why, I was thinking a method, where you drop values that differ much from the MEDIAN (= "range, where greatest deal of the values are")

I don't know the right terms... Median comes from statistical mathematics, and I learned to talk about strings in 1980's, when there was no PC yet...

 

[BeanieBots]

For a set of readings you can determine the highest and lowest WHILE THEY ARE READ IN.
You can then discard the highest and lowest BEFORE calculating the average.
This is much quicker (and a lot less code) than getting all the readings then sorting then discarding.

Measuring vacuum in the region of a few mmH2O is not very high vaccum. Why are your readings so "erratic". Maybe this is where you should put more effort. What type of sensor are you using?
If your readings where for high-vac (eg 10e-9 torr) then I could understand because you would be using something like an ION gauge which can be noisy.

 

[alband]

I can't say I understand the application (at all) but for calculating MAX's and MIN's what about this:

Take a first sample of readings (sample 1). Calculate their mean and standard deviation.
Then take a sample of readings (sample 2) and calculate the same statistics.

Then use the calculated mean and standard deviation from sample 1 to calculate acceptable MAX's and MIN's (say... mean +/- {standard deviation x constant}).
Then feed the readings from sample 2 through this MIN/MAX filter to acquire a new set of readings from which a mean can be taken and then used.

This cycle would continue by taking a 3rd sample, calculating the mean and SD, then using the mean and SD from the previous sample to filter out sporadic results and use the remaining results to draw a conclusion.

Basically the cycle uses unfiltered data to calculate the filtering limits of the next set of data. So if a large jump in the mean occurs that is not random, it would be picked up on the 2nd cycle, but if a large jump in the mean occurs randomly, it will be filtered out.

 

[KTarke]

For a set of readings you can determine the highest and lowest WHILE THEY ARE READ IN.
You can then discard the highest and lowest BEFORE calculating the average.
This is much quicker (and a lot less code) than getting all the readings then sorting then discarding.

That method sounds optimal! But I don't know how to code it
Could Yo give some advice?

Measuring vacuum in the region of a few mmH2O is not very high vaccum. Why are your readings so "erratic". Maybe this is where you should put more effort. What type of sensor are you using?
If your readings where for high-vac (eg 10e-9 torr) then I could understand because you would be using something like an ION gauge which can be noisy.

It is very difficult to find a very-low-pressure sensor, that is accurate... I found one, that is really accurate (compared with a expensive calibration meter) AND CHEAP! (<20USD). Because it's cheap, it is therefore badly shielded. Seems to be quite an antenna, collects rf- and magnetic disturbance from surroundings (clearly seen as a spike in readings, when one turns on any electrical equipment, and it does NOT come through Vcc, same with batteries...)
Next as accurate sensor, which is better shielded, costs five times more.
Have to stick with the cheap one, and try to rf-shield it somehow...

Here is the datasheet: http://www.smartec.nl/pdf/DSSPD102DAhyb.pdf
If You look at specs, there is a lot of "NC" -pins. Actually, they are far from NC... there are different voltages on them (first thought was to ground unused pins, can't do even that)

 

[fernando_g]

Although I applaud your idea of "digital filtering", you may still add some good old-fashioned analog filtering prior to A/D conversion.

The simplest solution could be an RC filter, along with some ferrite beads. A more complex solution could be a high-order low pass filter implemented with opamps. Coaxial cable for the signal runs may also be helpful.

How far is the sensor from the actual PICAXE? If your run is more than a few feet in an industrial environment, you may want to convert the sensor's output to a differential signal or perhaps a current loop.

 

[fernando_g]

Unoptimised.

Eeprom( "Computers in the future may weigh no more than 1.5 tons" )
Eeprom( "Popular Mechanics, 1949" )
Oh Boy! Those comments made my smile. You made my morning, even before coffee.

Now seriously, your mention of @ptrInc = b0, made my go back and re-read manual 2. I had missed that very important paragraph on indirect addressing! Very powerful...thanks for bringing it up.

 

[retepsnikrep]

I must admit I generally use the 10X or 100X ADC oversampling idea in a loop and then work out the average. But I like the idea of adding the MAX & MIN to the loop to get rid of the obvious glitches before they throw the average to far out.

 

[Dippy]

It works well. And, yes, where practical a good old fashioned filter helps too.

In one noisy app I did I added a PIC to the sensor end and , in effect, converted the sensor into a 'digital' sensor. Made life so much easier for host.
There are so many methods, each with advantages and disadvantages.
We're going to develop a nice list I can see

 

[fernando_g]

Yes, I use a similar procedure in some PIC ADC work, but for whipping off highest and lowest you don't need to bubble sort. You can do Max and Min in a loop. Iterate it if you want to.

.

I'm intrigued.......
I've given this trick a couple of minutes of thought.... do the MIN and MAX commands support variables?
What I mean is that in Manual 2; the MIN/MAX example is provided:
let b1 = b2 * 10 MAX 50

But could this be written as:
let b1 = b2 * 10 MAX b3

such that the maximum value can be adjusted?
I'm not anywhere near my PICAXE stuff, and won't be for another week, otherwise I would just test it.

 

[KTarke]

Although I applaud your idea of "digital filtering", you may still add some good old-fashioned analog filtering prior to A/D conversion.

The simplest solution could be an RC filter, along with some ferrite beads. A more complex solution could be a high-order low pass filter implemented with opamps. Coaxial cable for the signal runs may also be helpful.

How far is the sensor from the actual PICAXE? If your run is more than a few feet in an industrial environment, you may want to convert the sensor's output to a differential signal or perhaps a current loop.

Unfortunately, sensor "sits" close to Picaxe, no cables (vacuum to the sensor can be lead through a rubber hose)

If using RC-filtering, it has to be a really heavy one... and that may affect linearity (I will experiment with that too)

Problem is the sensor itself. It has a built-in circuit (amplifier, temperature regulation, some sort of linearity conditioner), Seems, that temp-reg and linearity works fine, but when the measure unit is VERY tiny , even small disturbance makes the readout shift 2-5 "units".
Sensor itself seems to be "floating" (touching different parts with finger changes reading a lot), which makes the output vary all the time at least 6 mV. Sounds minor, but when one "unit" is only 2,5mV at output...
I DON'T MIND ACCURACY less than 5 "units" (=12,5mV) , but must get stable readings at 2,5mV... Meaning, that stability is more important than the "real" accuracy.

That is why, I thought that "mathematical" conditining is a better solution (at least, does not affect linearity, as rc-circuits might do)

 

[InvaderZim]

I would definately use some hardware filtering in addition to whatever digital processing you do. Getting rid of high frequency electrical noise is essential, and cheap to do in hardware.

You may have an issue with mechanical noise. Is a vaccuum pump running while you are taking measurements? Are any valves opening or closing? Is anything vibrating? I'd guess one or two sources of mechanical movement is causing your sensor to deflect. Pressure sensors are finicky. If possible, try to sample only when no pumps, etc. are running. If that isn't possible, then a heavier analog or digital filter will be necessary.

 

[Dippy]

fernando, I don't know about PICAXE min max I was referring to calculating a running min or max in another language.
You know, the clunky method.

KTarke, if this thing is SO susceptible to ambient conditions shouln't you really be spending time devising some kind of shielding?
Any amount of ADC averaging/filtering won't correct for an ambient condition that causes an offset or slow oscillation.

Anyway, you use the method you find most appropriate. I'm sure it'll be fine.

 

[MFB]

I think it is unlikely that you will be able to depend on digital filtering alone to make more stable measurements. A low-pass analog filter will be needed to reduce the possibility of noise spikes overloading the ADC. The task of software processing is made much more difficult when trying to deal with inputs that are outside the range of the ADC.

Even an active (opamp) filter can of course be overloaded and may need to be proceeded by a passive RC filter. Regarding your concerns about introducing non-linearity, it is unlikely that rail-rail opamp circuitry will introduce significant errors in a 10-bit instrumentation system.

 

[KTarke]

I would definately use some hardware filtering in addition to whatever digital processing you do. Getting rid of high frequency electrical noise is essential, and cheap to do in hardware.

You may have an issue with mechanical noise. Is a vaccuum pump running while you are taking measurements? Are any valves opening or closing? Is anything vibrating? I'd guess one or two sources of mechanical movement is causing your sensor to deflect. Pressure sensors are finicky. If possible, try to sample only when no pumps, etc. are running. If that isn't possible, then a heavier analog or digital filter will be necessary.

I am experimenting with hw filtering too... haven't yet found the best solution.

Unfortunately, in this application I am measuring vacuum in a chimeney... that is why it is so tiny. No valves,vibration or pumps, and in test conditions the exhaustgasblower (and the triac-regulator) are very well shielded.

AND it is the same ~8mV variance all the time, EVEN if no hose is connected (should show steady Vcc/2 = "0"). The error is the same, when measuring pressure, does not grow. (tested with batteries, only the sensor on a breadboard, Fluke digitar meter)
That is the "ground"-error plus all those spikes that come from rf-disturbance.

 

[MFB]

Assuming a 5 volt supply, 8mV of jitter is pretty much as good as you can expect form an onboard 10-bit ADC. Maybe you need an external 12-bit ADC?

 

[KTarke]

Assuming a 5 volt supply, 8mV of jitter is pretty much as good as you can expect form an onboard 10-bit ADC. Maybe you need an external 12-bit ADC?

I think so too... (that it is best I can get) This jitter comes straight from the sensor, and when the "built-in" adc has it's own, there we are...

THAT is why, I am very ready to deal with the problem by mathematic averaging (and was searching better algorithm than the usual averaging)

Now I think, that all is done, that could be done at hw side: I cut off all "NC" legs from the sensor, welded a cap straight to Vcc and ground-legs, shortened cables between sensor and Picaxe board to only a few centimetres, and wrapped foil over the sensor. Now I get "only" 4, max 5mV jitter measured straight from the sensor.

Still, I try my best to use as few external components as possible,always.
(a learned habit: every extra component is a potential troublemaker...)
That is why, I am not very keen to throw in an external adc or low-pass filter.
I already tried some op-amp circuits. And later managed to drop them off by improving the resolution of Picaxe's own adc (by changing reference voltages,another thread here)

 

[MFB]

Even if you ground the ADC input pin you will see 5 mV worth of LSB jitter. I'm not sure what information you will be able to extract from this noise using software filtering.

 

[BeanieBots]

I try my best to use as few external components as possible,always.
(a learned habit: every extra component is a potential troublemaker...)
That is why, I am not very keen to throw in an external adc or low-pass filter.

For relaible accurate analogue readings, that is NOT the correct approach.
Decoupling, pre-processing and good clean power rails are essential and that objective can only be met by adding in the required components.

There is extra information to be found even in what appears to be random noise on an LSB of an ADC. In fact, sometimes noise is deliberatley introduced to INCREASE the resolution of the A2D systems. It then becomes a statistics issue to determine how close the incomming level actually is to the ADC LSB trip point.

Then there is the bandwidth rule.
Never have a higher bandwidth on the ADC then there is in whole system. All the extra bandwidth can do is introduce errors that need to be averaged out.

 

[KTarke]

Even if you ground the ADC input pin you will see 5 mV worth of LSB jitter. I'm not sure what information you will be able to extract from this noise using software filtering.

No, that was not what I ment.
Adc is steady, the jitter comes out from the sensor, with a STEADY vacuum (or no vacuum at all), output shifts all the time at least 5mV (which is about 2,5Pa)
With normal averaging I can get very reliable readings (compared with a expensive, just calibrated meter)

WHAT I WISH is that the reading ON DISPLAY would not shift all the time a few Pascals, but be more stable.

And THAT can be done by math, and this topic was about to discuss ONLY MATH SOLUTIONS, NOT TO TALK ABOUT SIGNAL CONDITIONING WITH EXTERNAL COMPONENTS.

 

[BeanieBots]

Ok, point taken.
There are many digital filtering methods which can give the same response as an RC filter which will give better results than basic averaging.

This link explains the maths behind it which is PICAXE'able once the coeficients have been determined. (basic multiplication and additiion)
http://www.bores.com/courses/intro/filters/4_eq.htm

Removing the maximum and minimum values from a summation is simple enough (do not confuse with MAX and MIN functions) by testing the data set as it is read in.

EDIT:
If it just the displayed value you wish to keep steady, then you can "slew rate limit" just the display.
If the read in value is X and the display is Y, then if Y<X then Y=Y+A, if Y>X then Y=Y-A
A is chosen to limit the rate at which Y can change.

 

[boriz]

...If it just the displayed value you wish to keep steady, then you can "slew rate limit" just the display.
If the read in value is X and the display is Y, then if Y<X then Y=Y+A, if Y>X then Y=Y-A
A is chosen to limit the rate at which Y can change.

How about something like this:

Do
Read new
Diff=new-curr
curr=curr+(diff/2)
Loop

Essentially the same as BB suggests, but the ‘A’ factor is based upon the difference between the current and the last reading. In this case, A is the difference/2.

EG:

If the current reading is 100 and the new reading is 120, then the difference is 20. So current gets 20/2 added and becomes 110. Next time around, if the new reading remains steady at 120, then the current reading becomes 115. Then 117.5, then 118.75 etc. Getting closer, but never actually getting there. However using Picaxe integer maths where the minimum value of A is 1, then the progression is like this: 100,110,115,118,119,120.

By changing the divisor from 2 to 3, you get a longer settling time but better noise immunity and so on for 4, 5, etc.

Just an idea.

 

[Tyro]

The idea from Boriz is good but may give problems if the signal is reducing. This will generate negative numbers. This can be fixed but requires even more code.

The following is part of some code I am using successfully. I measure a 10-bit signal, sig, 64 times, each time adding it to ‘avgsig’. I leave it as a 16-bit number so that I do not lose the resolution.

Divide ‘avgsig’ by 8.
Multiply ‘base’ (the very well averaged number that can now be used as a displayed number) by 7/8 and then add the 1/8 ‘avgsig’. The 7/8 and the 1/8 could be any other complementing fractions you like. The smaller the fraction, the slower the response with lower noise.

All the numbers are left in 16-bit format until the final number is truncated for a display. One of the differences between scientific calculators and regular ones

for cycles = 0 to 63

gosub measure

avgsig = avgsig + sig 'update avgsig

next

avgsig = avgsig / 8 'these 2 lines remove 1/8 from base
base = base / 8 * 7 + avgsig 'and add 1/8 of avgsig

measure:
'run ADC, manipulate reading and return as sig

 

[MFB]

Not sure if this ( http://en.wikipedia.org/wiki/Stochastic_resonance ) is the technique that BeanieBots had in mind for improving ADC 'resolution' but I don't think it would help using just LSB jitter.

 

[BeanieBots]

Nope, dither rather than stochastic.
http://en.wikipedia.org/wiki/Dither

 

[KTarke]

Thank Yoy BeanieBots,MFB,Tyro and Boriz! These last posts were, what I was was looking for. It is good to learn, what can be done with Picaxes basic...

Picaxe is a very nice piece, easy to get started with (for me, who does not understand anything about Pic assembler...) Basic is a powerfull tool, though often underestimated...
After "finding" Picaxe, I already have a long list of projects, which now I can make real, having a "tool" which I can (fast enough, time is a luxury, because I run my own business) learn to program.

I think that this thread is useful to others than me ,too and a good reference for the topic.


My current project has now evolved without "math averaging" ,to a approvable state. There was a strange solution to the jitter...
As I told before, an average of 50 readings contains a few (too many) "bad" readings, that can "waste" the average.
I had a hunch... and "recorded" to pc a big list of readings (a few times). Looking at the tables gave me an expression, that there is some pattern.
From my experience in industrial electronics... patterns always mean something.
In this case, solution was a surprise: even the sensors datasheet says, that sensors REACTION TIME is 25mS. I ignored that, thought that question was ONLY of reacting for changes... but not! if taking continuos samples, the result is about 70% worse, than if i add 25mS pause between samples...
Can't reason that... Picaxe's input impedance is high enough, as is my Fluke voltmeters... still, timing helps.

Now, average of 50 (without more filtering (math or hw), sensor connected straight to Picaxe adc) gives quite steady reading (jitters less than my calibration meter, 300$)