DIY EC sensor
- Thread starter Sral
- Start date
- Tagged users None
Sral
Valuable Member
@Ranjib : since we already noticed that we both use the Arduino IDE for the ESP extension and the Pico EC Meter we could try my source code on an ESP32 platform. Should work fairly easily.
The ADC has 2 bit less resolution, but my algorithm can improve that fairly easily when there is enough noise and it might have less Nonlinearity issues issues than the Pico ^^
The ADC has 2 bit less resolution, but my algorithm can improve that fairly easily when there is enough noise and it might have less Nonlinearity issues issues than the Pico ^^
Last edited:
Absolutely. Not just you, any who , created an microcontroller based solution should be able to integrate with the esp32 driver in reef-pi, as long as it implements the http interface, since the driver is network based. If WiFi is not an option then we need some other mechanism (Like an i2c equivalent ) which is some work on reef-pi end. But anything that has WiFi, should be able to seamlessly integrate with reef-pi as long as they implement the http api. I can detail that in a doc, you can see the specs in the driver in reef-pi (Consumer ) and firmware for the esp32 repo (producer ) . Let me know what specific help you need
Sral
Valuable Member
Thanks ! The ESP32 code is a good start and I'll look through the Reef-Pi code and docs. If I need anything, I'll let you know. Good thing is that the ESP32 ADC seems to be 12bit as well and might be more linear than the Pico.
I also found a nice way of calibrating the ADC in both chips, "simply" feed a PWM through an RC filter and read that with the ADC. The voltage will not be precise, but I only care about linearity anyway and with the 16bit PWM counter that should be very linear indeed. I can even fix that as a constant calibration source on the board and let the chip do that automatically ^^
I also found a nice way of calibrating the ADC in both chips, "simply" feed a PWM through an RC filter and read that with the ADC. The voltage will not be precise, but I only care about linearity anyway and with the 16bit PWM counter that should be very linear indeed. I can even fix that as a constant calibration source on the board and let the chip do that automatically ^^
Sral
Valuable Member
I smoothed a 15kHz PWM signal, which still gives me 12 bits of PWM resolution on the Raspberry Pi Pico, with a 10k resistor and a 1 µF capacitor. I'm sampling this voltage Using the Arduino IDE the maximum sampling rate is something like 50kHz.
When I look at this smoothed voltage with my ADC every 0.1s I get this :
(values from 10 to 4096 for ~GND to ~3.3V)
Good enough. I probably wont get it lower than that, since at some point the voltage is smoother than the ADC itself ^^
When I smooth this out with a digital filter I get it down to sub-bit resolution:
So I can measure the voltage with 0.1 bit resolution, which should be enough to roughly characterize the chip's ADC.
When I increase the duty cycle the voltage rises, which takes about 0.5s to settle on the capacitor and my digital filter, which I can measure nicely:
As I increase my PWM duty cycle in 12-bit steps (increments of 1 over 4095) my measured voltage increases in 12-bit steps as well (increments of roughly 1 over 4095). GND registers as ~10, this is where the offset at 0 comes from.
This gets interesting at a PWM value of 512, which is where my nonlinearity is:
And tadaa ! There it is
Beautifully resolved with a width corresponding to the expected value of around 10. Nice visualization what this does as well, the voltage rises, but the ADC value is stuck at 512. This means that you can't effectively measure certain voltages from 504 to 514, equaling to roughly:
(504-10)/4085*3.3V = 0.39907 V
to
(514-10)/4085*3.3V = 0.40715 V
Which will all register as the same voltage of 0.39907. At each of these nonlinearities there are 8mV that one can't resolve.
This method sadly currently takes about 2048 seconds, e.g 35 minutes to complete ...
When I look at this smoothed voltage with my ADC every 0.1s I get this :
(values from 10 to 4096 for ~GND to ~3.3V)
Good enough. I probably wont get it lower than that, since at some point the voltage is smoother than the ADC itself ^^
When I smooth this out with a digital filter I get it down to sub-bit resolution:
So I can measure the voltage with 0.1 bit resolution, which should be enough to roughly characterize the chip's ADC.
When I increase the duty cycle the voltage rises, which takes about 0.5s to settle on the capacitor and my digital filter, which I can measure nicely:
As I increase my PWM duty cycle in 12-bit steps (increments of 1 over 4095) my measured voltage increases in 12-bit steps as well (increments of roughly 1 over 4095). GND registers as ~10, this is where the offset at 0 comes from.
This gets interesting at a PWM value of 512, which is where my nonlinearity is:
And tadaa ! There it is
Beautifully resolved with a width corresponding to the expected value of around 10. Nice visualization what this does as well, the voltage rises, but the ADC value is stuck at 512. This means that you can't effectively measure certain voltages from 504 to 514, equaling to roughly:
(504-10)/4085*3.3V = 0.39907 V
to
(514-10)/4085*3.3V = 0.40715 V
Which will all register as the same voltage of 0.39907. At each of these nonlinearities there are 8mV that one can't resolve.
This method sadly currently takes about 2048 seconds, e.g 35 minutes to complete ...
Sral
Valuable Member
I measured a full trace and extracted the data, I'm surprised how well this actually worked:
Blue is the curve measured by my first Source,
orange is a slightly modified fit to this data,
green is the curve from the Raspberry PI's datasheet
red is my own measurement as described one post above.
I'll test which one of these works best, but for now I have a working method that I can slap on an ESP32. I'll expect that the ESP32 works much better, since it doesn't have these nonlinearities and more noise on its ADC, which is actually quite good for my filtering method.
If the ADC had no noise and would stay at a single number I wouldn't be able to smooth this noise out to sub-bit resolution
Blue is the curve measured by my first Source,
orange is a slightly modified fit to this data,
green is the curve from the Raspberry PI's datasheet
red is my own measurement as described one post above.
I'll test which one of these works best, but for now I have a working method that I can slap on an ESP32. I'll expect that the ESP32 works much better, since it doesn't have these nonlinearities and more noise on its ADC, which is actually quite good for my filtering method.
If the ADC had no noise and would stay at a single number I wouldn't be able to smooth this noise out to sub-bit resolution
Sral
Valuable Member
Okay, I tested the ADC of the ESP32 and I'm not convinced that buying 3 of them was a good idea XD
Here you see the curves that the Pi PIco and the ESP32 ADC give:
Idealy, the ADC would give more or less a perfect diagonal, which is the blue curve.
The Pico is pretty close with the green curve on this scale being visualy almost the same.
The ESP32 is rather horrible when you want a measurement range of 0-3.3 V.
You can't measure voltages from 0.000 V to 0.138 V as they will all read as zero.
You can measure voltages between 0.138 V to 2.660 V will be measureable farily well
You can measure voltages over 2.660 V with increasing difficulty.
You can't measure voltages over 3.190 V as they will all read the same value of 4095.
If you rather plot the difference from the ideal curve vs the value the ADC gives you, you get this:
(To get voltage you roughly have to calculate value/4095*3.3V, e.g.:
error of ~10 means 10/4095*3.3V ~ 8 mV
Value of 502 means 502/4095*3.3V ~ 0.405 V)
So the Pico is closer to the ideal, BUT it has these jumps, which regularly leads to voltage "gaps" of around 8 mV each:
You can't measure 0.405 V to 0.413 V as it will all read as 0.405V
You can't measure 1.230 V to 1.238 V as it will all read as 1.230V
You can't measure 2.055 V to 2.063 V as it will all read as 2.055V
You can't measure 2.880 V to 2.888 V as it will all read as 2.880V
The ESP32 has jumps as well, but these are smaller ... The rest of the slope, deviating from the ideal 0 is not as bad as it looks, since I can mostly correct this. I had to offset the ESP32 curve by -180, so it would end up around 0. The ESP32's ADC value is mostly too small by around 0.145V, as I noted above. But if you know that it isn't a problem.
I'll have to test this in a practical circuit. The noise will for sure be worse, but I can simpy increase the filtering and wait a bit longer. In a Reef-Pi scenario you also don't expect a value every 10s ^^
Here you see the curves that the Pi PIco and the ESP32 ADC give:
Idealy, the ADC would give more or less a perfect diagonal, which is the blue curve.
The Pico is pretty close with the green curve on this scale being visualy almost the same.
The ESP32 is rather horrible when you want a measurement range of 0-3.3 V.
You can't measure voltages from 0.000 V to 0.138 V as they will all read as zero.
You can measure voltages between 0.138 V to 2.660 V will be measureable farily well
You can measure voltages over 2.660 V with increasing difficulty.
You can't measure voltages over 3.190 V as they will all read the same value of 4095.
If you rather plot the difference from the ideal curve vs the value the ADC gives you, you get this:
(To get voltage you roughly have to calculate value/4095*3.3V, e.g.:
error of ~10 means 10/4095*3.3V ~ 8 mV
Value of 502 means 502/4095*3.3V ~ 0.405 V)
So the Pico is closer to the ideal, BUT it has these jumps, which regularly leads to voltage "gaps" of around 8 mV each:
You can't measure 0.405 V to 0.413 V as it will all read as 0.405V
You can't measure 1.230 V to 1.238 V as it will all read as 1.230V
You can't measure 2.055 V to 2.063 V as it will all read as 2.055V
You can't measure 2.880 V to 2.888 V as it will all read as 2.880V
The ESP32 has jumps as well, but these are smaller ... The rest of the slope, deviating from the ideal 0 is not as bad as it looks, since I can mostly correct this. I had to offset the ESP32 curve by -180, so it would end up around 0. The ESP32's ADC value is mostly too small by around 0.145V, as I noted above. But if you know that it isn't a problem.
I'll have to test this in a practical circuit. The noise will for sure be worse, but I can simpy increase the filtering and wait a bit longer. In a Reef-Pi scenario you also don't expect a value every 10s ^^
Sral
Valuable Member
Regarding this ADC Error limiting the available voltage range on the ESP32, the documentation of the ESP32 arduino core mentions that this was addressed in different versions of the ESP32. So depending on which chip version you have, this will also be different
Another thing I have noticed: the ESP32 seems to be a lot slower with reading the ADC and applying the digital filter with double precision float operations than the Pi Pico.
With the Pi Pico, reading and operating the double precision floating points in the digital filter took about 10 µs,
whereas with the ESP32 the same code takes about 100 µs !
This will of course severely limit my sampling frequency on the ESP32 ... with the Pico I could sample my 1kHz PWM on two channels with about 20 kHz. With the ESP32 I will probably only manage 2kHz, e.g. 1 sample per peak instead of 10.
Another thing I have noticed: the ESP32 seems to be a lot slower with reading the ADC and applying the digital filter with double precision float operations than the Pi Pico.
With the Pi Pico, reading and operating the double precision floating points in the digital filter took about 10 µs,
whereas with the ESP32 the same code takes about 100 µs !
This will of course severely limit my sampling frequency on the ESP32 ... with the Pico I could sample my 1kHz PWM on two channels with about 20 kHz. With the ESP32 I will probably only manage 2kHz, e.g. 1 sample per peak instead of 10.
Sral
Valuable Member
Alright, I measured a resistance cascade with both the ESP32 and the PiPico and compared them to the values measured by a multimeter (since the multimeter is more accurate than the tabulated value):
On the PiPico I have measured a few more values around the Non-Linear steps, e.g. ~ 0 Ohm, ~480 Ohm and ~4800 Ohm, equaling about 0.0 , 0.34 and 3.4 times the reference resistor of 1400 Ohm.
So if not for the nonlinearities, the PiPico seems to be better in both noise level (with a factor of 10) and accuracy. Both are fine though, if you don't want pinpoint accuracy:
I do however have a plan ... the problem on the Pico only arises, when the measurement is exactly at the nonlinear voltage. If I can manipulate the excitation voltage or measurement offset at these well known positions to be just a few percent different, I can completely avoid this. Better still: I can use my always loved MOSFETs to do this
Let's see if I can make this work !
On the PiPico I have measured a few more values around the Non-Linear steps, e.g. ~ 0 Ohm, ~480 Ohm and ~4800 Ohm, equaling about 0.0 , 0.34 and 3.4 times the reference resistor of 1400 Ohm.
So if not for the nonlinearities, the PiPico seems to be better in both noise level (with a factor of 10) and accuracy. Both are fine though, if you don't want pinpoint accuracy:
I do however have a plan ... the problem on the Pico only arises, when the measurement is exactly at the nonlinear voltage. If I can manipulate the excitation voltage or measurement offset at these well known positions to be just a few percent different, I can completely avoid this. Better still: I can use my always loved MOSFETs to do this
Let's see if I can make this work !
Sral
Valuable Member
Onwards !
I was able to change my offset voltage using a MOSFET that toggled a parallel resistor to my resistance series that created the offset to the middle of my 3.3V rail. This was 1.638V (~1.65 V +/- 1% from the resistors). It wasn't quite perfect, because this connection provided enough parasitic resistance that my offset voltage without the MOSFET being active dropped from 1.638V to 1.620V. I had to compensate this by adding another "parasitic" resistance to 3.3V to offset this and I'm back to 1.649 V.
I'm hoping that this will be less of an issue on a PCB ^^
Activating the MOSFET this way changes my offset voltage from 1.649V to 1.595V, as I calculated.
Now I'll have to measure the position of the nonlinearity with and without the shift
I was able to change my offset voltage using a MOSFET that toggled a parallel resistor to my resistance series that created the offset to the middle of my 3.3V rail. This was 1.638V (~1.65 V +/- 1% from the resistors). It wasn't quite perfect, because this connection provided enough parasitic resistance that my offset voltage without the MOSFET being active dropped from 1.638V to 1.620V. I had to compensate this by adding another "parasitic" resistance to 3.3V to offset this and I'm back to 1.649 V.
I'm hoping that this will be less of an issue on a PCB ^^
Activating the MOSFET this way changes my offset voltage from 1.649V to 1.595V, as I calculated.
Now I'll have to measure the position of the nonlinearity with and without the shift
Sral
Valuable Member
Just for explanation, here is a circuit diagram:
The Pi Pico Outputs a PWM that is AC coupled with capacitor C1.
This AC rectangle signal runs through the R_Ref and R_TEST resistance series.
The voltages are AC coupled with capacitors C2 and C3 to an offset voltage of ~1.65 V so the Pico ADC can measure the full 1.65 rectangular waveform.
When I now activate the MOSFET, I decrease the resistance to GND and pull my offset voltage about 50mV lower.
The Pi Pico Outputs a PWM that is AC coupled with capacitor C1.
This AC rectangle signal runs through the R_Ref and R_TEST resistance series.
The voltages are AC coupled with capacitors C2 and C3 to an offset voltage of ~1.65 V so the Pico ADC can measure the full 1.65 rectangular waveform.
When I now activate the MOSFET, I decrease the resistance to GND and pull my offset voltage about 50mV lower.
Sral
Valuable Member
It worked ! I measured again the measured resistance vs my "known" resistances in the cascade:
Red is without offset
Black is with the offset
As you can see, the red curve still has the nonlinearity at the previously recorded position of 500 Ohm.
However, when I activate my offset splits into two nonlinearities at different values off to the side at about 380 Ohm and 620 Ohm. Everywhere else the values are pretty much the same.
I still have a step in the middle, but I wager that's from the residual uncertainty in my knowledge of the resistance cascade. My multimeter has only 4 digits and simply says "0.502 kOhm" and the 0.1% step means just 0.5 Ohm, so just outside my reach ^^
I therefore call this method a success !
Now I only need to get rid of the -0.4% deviation at 100 Ohms
Red is without offset
Black is with the offset
As you can see, the red curve still has the nonlinearity at the previously recorded position of 500 Ohm.
However, when I activate my offset splits into two nonlinearities at different values off to the side at about 380 Ohm and 620 Ohm. Everywhere else the values are pretty much the same.
I still have a step in the middle, but I wager that's from the residual uncertainty in my knowledge of the resistance cascade. My multimeter has only 4 digits and simply says "0.502 kOhm" and the 0.1% step means just 0.5 Ohm, so just outside my reach ^^
I therefore call this method a success !
Now I only need to get rid of the -0.4% deviation at 100 Ohms
Looking forward to this. Another great thread from you @Sral . im humbled by the sheer volume of high quality insights, effort you put in for this community, across led, ec, dosing and many aspects of control systems. Thank you so much
Sral
Valuable Member
Thanks, I'm having a lot of fun trying to apply my theoretical knowledge in this area. Not so sure about the "high quality" of insights most of the time, but I'm happy to help with what I have !
I'll also have to thank you again for providing my current hobby and building up this platform and community in the first place !
Regarding this ADC Error limiting the available voltage range on the ESP32, the documentation of the ESP32 arduino core mentions that this was addressed in different versions of the ESP32. So depending on which chip version you have, this will also be different
Another thing I have noticed: the ESP32 seems to be a lot slower with reading the ADC and applying the digital filter with double precision float operations than the Pi Pico.
With the Pi Pico, reading and operating the double precision floating points in the digital filter took about 10 µs,
whereas with the ESP32 the same code takes about 100 µs !
This will of course severely limit my sampling frequency on the ESP32 ... with the Pico I could sample my 1kHz PWM on two channels with about 20 kHz. With the ESP32 I will probably only manage 2kHz, e.g. 1 sample per peak instead of 10.
I think as expected. the ESP32 is neither great with FPU (especially doubles), nor even caching execution from flash, nor is its ADC in any particular way great. Most run of the mill tiny MCUs have better ADC units.
Sral
Valuable Member
Thanks for the info ! Although I have found somebody mention that this is only the case if one uses the highest attenuation setting. If you step the voltage down yourself with two cheap resistors this seems to become a lot better. Although GND is still not measureable without some more resistor work creating an offset.
The reality is building a good ADC is hard, and Espressif doesn't use a process where its really feasible or really control for it. Its fine for probably its intended usage (battery level management, some simple sensors), but the spec of +/- 7 LSB is absurd (using the S2 as an example). There is also no typical presented, which somewhat shows they're not even characterizing it.
Using the <$1 SAMD11 as an example (Min, Typ, Max columns), you can expect a much better baseline (this is in single ended mode, it can do differential too, and has an internal PGA if you want it).
Espressif doesn't even characterize the other components (SNR, offset error).
As with any analog system, you can use calibration to remove a lot of the inherent error, as long as the measurement is repeatable. I don't have enough experience to know how consistent a single sample is, temperature coeffficients, etc. For a lot of the measurements we do need, we don't care about speed, so using a very slow sigma-delta external ADC is actually probably ideal, just knowing we won't somehow actually get enough noise free operating to get 16-24b from them.
So, not useless, but its honestly one of the worst ADCs available
(Some background on INL/DNL https://www.analog.com/en/technical-articles/inldnl-measurements-for-types-of-highspeed-adcs.html)
Using the <$1 SAMD11 as an example (Min, Typ, Max columns), you can expect a much better baseline (this is in single ended mode, it can do differential too, and has an internal PGA if you want it).
Espressif doesn't even characterize the other components (SNR, offset error).
As with any analog system, you can use calibration to remove a lot of the inherent error, as long as the measurement is repeatable. I don't have enough experience to know how consistent a single sample is, temperature coeffficients, etc. For a lot of the measurements we do need, we don't care about speed, so using a very slow sigma-delta external ADC is actually probably ideal, just knowing we won't somehow actually get enough noise free operating to get 16-24b from them.
So, not useless, but its honestly one of the worst ADCs available
(Some background on INL/DNL https://www.analog.com/en/technical-articles/inldnl-measurements-for-types-of-highspeed-adcs.html)
Sral
Valuable Member
Yeah, if you scroll back a bit, I was measuring and using the INL myself, pretty horrible at times for some of these microcontrollers ...The reality is building a good ADC is hard, and Espressif doesn't use a process where its really feasible or really control for it. Its fine for probably its intended usage (battery level management, some simple sensors), but the spec of +/- 7 LSB is absurd (using the S2 as an example). There is also no typical presented, which somewhat shows they're not even characterizing it.
Using the <$1 SAMD11 as an example (Min, Typ, Max columns), you can expect a much better baseline (this is in single ended mode, it can do differential too, and has an internal PGA if you want it).
Espressif doesn't even characterize the other components (SNR, offset error).
As with any analog system, you can use calibration to remove a lot of the inherent error, as long as the measurement is repeatable. I don't have enough experience to know how consistent a single sample is, temperature coeffficients, etc. For a lot of the measurements we do need, we don't care about speed, so using a very slow sigma-delta external ADC is actually probably ideal, just knowing we won't somehow actually get enough noise free operating to get 16-24b from them.
So, not useless, but its honestly one of the worst ADCs available
(Some background on INL/DNL https://www.analog.com/en/technical-articles/inldnl-measurements-for-types-of-highspeed-adcs.html)
Raspberry Pi got their's ALMOST to a good state on the Pico. And then messed up some of the capacitor sizes and got 10 LSB peaks in their DNL
curious then, is there any adc that we can interface with pi or esp32 that will be suitable? the existing ads115 is good ? ph board is already resolved, but its with surface mount components. for diy, new works that require circuit construction, it will be nice to have a known good IC that we can use with through hole components and perma proto boards.
Sral
Valuable Member
Yes, I believe both ADS1015 and ADS1115 are very reliable in this regard. They are much slower (~1kHz compared to 200-500 kHz for microcontrollers) but more accurate. The ADS1015 has 12 bits, e.g. about 1mV resolution in the largest range, and only about 1-2mV error, whereas the Pico has up to 8mV in certain places and the ESP32 has 140mV.
I have read that the ESP32‘s problem mainly arises in the full range, so if one uses a lower range and simply steps down the voltage oneself with a few resistors, this could become much better and can replace an ADS1015 for our purposes I guess.
