Smart Cities - Reading data from Environmental Sensors in an Aquarium
Overview
- The Python language is easy to learn and is one of the foremost languages supported by the Raspberry Pi (Pi is short for Python).
- A smaller version of Python (named microPython) is also widely used on Internet of Things (IoT) devices.
- In this lesson we will use Python to access sensor data from an environmental monitoring sensor that will be used to monitor the water conditions in an aquarium at the Tech School.
- The sensor will give us data on water temperature, salt levels (measured by the conductivity of water), dissolved oxygen (available oxygen for fish gills) and other management parameters associated with the sensor such as the battery voltage (normal range is 3.7 to 4.20 volts) and humidity and temperature conditions within the sensor housing.
- Once we have the data on our computer we can save and graph the data.
- All this data allows us to monitor our chosen environment more closely.
Learning Objectives
- Learn how to create and save files in the Python3 Integrated Development and Learning Environment (IDLE3).
- Run python programs in the Python Shell window.
- Run python programs in the Python Editor Window.
Install Python3
Open the Terminal on the Raspberry Pi. Enter the command sudo apt-get update to update the software libraries on the Raspberry Pi.
The update will take approximately one minute depending on the speed of the internet connection.
Now we will do a search for the software we need to install. The Integrated Development Environment (IDE) for Python3 is named IDLE3. Python3 is the current version of Python. We can search for this software using the command sudo apt-cache search idle3.
In this example two results are returned, but we will only use the first listing which is the IDE for Python3.
Now we can install the IDLE3 software (app) using the commmand sudo apt-get install idle3.
You may be asked to confirm the installation. Answer Y for yes. The installation will take 20 seconds.
Using python to collect sensor data
Requesting new data from a sensor URL in Python
- Sensors are now commonly connected to the internet.
- This has given rise to the term – The Internet of Things.
- If a sensor is connected to the internet then it will often have a URL (Universal Resource Locator) to allow us to interact with the sensor.
- One common interaction is to be able to see or download data collected by the sensor.
- To get data from a sensor we need to import the requests library in python.
- In this lesson we will download some data from a water quality sensor for the Tech School aquarium (aquaponics).
- The sensor data includes temperature, conductivity, dissolved oxygen.
- In this example we write a python3 program to download sensor data using the requests library.
- The Python program will be written in the Python Editor window.
- The data will be presented in the Python Shell window.
Create a new file in python named aquarium.py
- Open the Python3 application from the Raspberry Pi > Programming drop down menu (top left of Desktop).
- Select Python3 (IDLE).
- From the File drop down menu select New File.
- This will open up a new file Editor Window.
- We are going to name the file aquarium.py
Create a directory named techschool
- Follow this instruction just in case you don't already have a directory named techschool in your /home/pi directory.
- Open the Terminal and make a directory named techschool in the pi home directory using the command mkdir techschool.
aquarium.py python code
- Enter the following code in Python Editor Window.
# aquarium.py
import requests
r = requests.get('https://api.particle.io/v1/devices/330035000d47373336373936/internalTemp?access_token=1c476acb47bd0b944a031e2859ef7160e4b72a66')
print(r)
- Here is a screen capture of the same code in the Python Editor window in IDLE3.
- Save the file in the techschool directory as aquarium.py
Adding Comments within files
- We can add notes within our program called Comments.
- Comments always start with a # (hash symbol) at the start of the line.
- Computers ignore comments because comments are only read by humans.
- To signal that the text has been commented out, the text colour will change to red.
- In this example the comment is the file name.
Request data from sensor using python
- Enter the following lines of request to request data from the atmospheric sensor.
- The key items in the code are described:
- import requests – this statement imports a python library so that the python program can get data from the internet.
- r = requests.get(URL) – this statement gets the data from a specified URL and stores the result in the variable object named r. URL is short for Uniform Resource Location, also known as a web address.
- print(r) – this statement prints a code that determines if the request was successful or not.
- Save and the Run the program.
Run the Python program
- Save any changes to the file, then select Run > Run module.
- Note - this program may not work outside class hours because the sensor will be turned off.
- This will open up the Python Shell which will print out data from our program.
- The response in the Python Shell should be [200].
- A value of [200] means that the request was successful.
- Other numbers or errors will indicate that there is a problem either with the Python code or with communication with the sensor.
- For example, error code [408] is a timeout error. It means that a response was not received from the environmental sensor after a given period of time. You may get this error if the sensor is turned off.
View sensor data in JSON format
- We can now add some more code so that we can look at the actual data.
- Add the statement print(r.json()) and run the code.
- The method json() means that the data will be presented in JSON format.
- JSON stands for JavaScript Object Notation.
- It is an open standard to help format data for data exchange between computers and it uses human-readable text.
- The data is stored in attribute-value pairs' similar to the format of a dictionary.
# aquarium.py
import requests
r = requests.get('https://api.particle.io/v1/devices/330035000d47373336373936/internalTemp?access_token=1c476acb47bd0b944a031e2859ef7160e4b72a66')
print(r)
print(r.json())
- If you have a careful look through the JSON data you will see the pair 'result': '23.4'
- result - is the attribute
- 23.4 - is the value, in this case the temperature
Other variables to explore
This program requested data from the internalTemp variable. Other variables you can explore include:
- temperature - water temperature from Atlas probe
- conductivity - salt concentration in water using Atlas probe
- disOxygen - dissolved oxygen levels in water. 8 mg/L to 12 mg/L healthy for fish.
- turbidity - measure of how dirty the water is. The higher the turbidity the more dirty the water.
- pressure - measure of water depth. The higher the pressure the greater the depth.
- internalTemp - temperature within sensor casing
- internalHumid - humidity within sensor casing
Data from Peter Hopper lake
Battery data from Peter Hopper lake sensor
# peterHopperBat.py
import datetime
import requests
import json
peter_hopper_bat = '60dbb5b653b63e0cd9b353ca' # battery voltage
api_key = "Ti74NO5dur5Zs8WiVXPEo7VP5VA0PRrvIFmURTzK"
req_uri = 'https://api.eagle.io/api/v1/nodes/'
req_headers = {'Content-Type': 'application/json','X-Api-Key': api_key}
r_bat = requests.get(url = req_uri + peter_hopper_bat, headers = req_headers)
print(r_bat)
print(r_bat(json())
bat = str(r_bat.json()['currentValue'])
print("The battery voltage is " + bat)
More data from Peter Hopper lake sensor
# peterHopperMore.py
import datetime
import requests
import json
peter_hopper_int_temp = '60dbb5b653b63e0cd9b353c6' # internal temp
peter_hopper_bat = '60dbb5b653b63e0cd9b353ca' # battery voltage
api_key = "Ti74NO5dur5Zs8WiVXPEo7VP5VA0PRrvIFmURTzK"
req_uri = 'https://api.eagle.io/api/v1/nodes/'
req_headers = {'Content-Type': 'application/json','X-Api-Key': api_key}
r_bat = requests.get(url = req_uri + peter_hopper_bat, headers = req_headers)
r_int_temp = requests.get(url = req_uri + peter_hopper_int_temp, headers = req_headers)
print(r_bat)
print(r_int_temp)
bat = str(r_bat.json()['currentValue'])
print("The battery voltage is " + bat)
int_temp = str(r_int_temp.json()['currentValue'])
print("The internal temp is " + int_temp)
All data from Peter Hopper lake sensor
Appendix - Setup Particle electron
Particle Builder
Follow the set up instructions for the Particle electron. Particle electron setup using a web browser
Use the Particle Builder in a web browser to update the firmware to version 1.5.0
Create a new program under the code tab. In this case the program was named aquarium-tech and the extension .ino was automatically added.
Firmware upgrade and program upload
Then Flash the new program to the Particle electron.
After flashing the code, the Particle electron will undergo a series of firmware upgrade steps. In this screen capture the firmware is up to version 1.2.1. The firmware upgrade process takes approximately 3-5 minutes.
When the firmware upgrade process is complete version 1.5.0 will be displayed. Note that this is not the latest firmware, however we know that this firmware works with out proposed environmental sensor hardware. Normally the latest (most recent) firmware version is installed.
Particle Console
Once the firmware update is complete you can use the Particle Console in a web browser to monitor some of the key operational parameters on the Particle electron. You can monitor data traffic, 3G signal strength and battery charge status.
In the Particle Build load up relevant app and update any libraries.
In the Particle Build click on Save, Verify and finally Flash to upload the app to the Particle electron.
Particle code for Water quality sensor
Full Particle electron code for this activity.
Particle electron code has been collapsed by default. Click here to expand:
// This #include statement was automatically added by the Particle IDE.
#include <JsonParserGeneratorRK.h>
#include <RunningMedianST.h>
#include <Adafruit_DHT_Particle.h>
// Example script for field deployed particle device - monitoring water quality using Atlas
// Scientific sensors mounted on a Whitebox Labs Tentacle
// Ensure all jumpers on the tentacle board are set to I2C and all ezo boards are also set to I2C
// Consult the tentacle and atlas documentation for further details
#include <Wire.h> //enable I2C.
#define DHTTYPE DHT22 // DHT 22 (AM2302)
#define DHTPIN D3 // what pin we're connected to
#define PRESSPIN A0 // A0 will be used for pressure sensor
#define TURBPIN A1 // Pin for reading turbidity (changed from A0 by Edmond 11 Jan 2022)
#define PWR5V D2 // Pin to turn on power to DHT, Atlas sensors, turbidity and pressure sensor
//#define PWR12V D7 // Pin to turn on power to turbidity sensor - in my case this is a 12V power supply - yours may be something different
SYSTEM_MODE(AUTOMATIC);
DHT dht(DHTPIN, DHTTYPE);
FuelGauge fuel;
double powerSource;
double batteryState;
double BatteryVoltage;
double BatterySOC;
byte code = 0; //used to hold the I2C response code.
byte in_char = 0; //used as a 1 byte buffer to store in bound bytes from the EZO Circuit.
byte i = 0; //counter used for IC2 data
int delay_time = 1800; //used to change the delay needed depending on the command sent to the EZO circuit.
int address;
char ezo_data[48]; //we make a 20 byte character array to hold incoming data from the EZO circuit.
String T_string;
String EC_string;
//String pH_string;
String DO_string;
String T_status;
String EC_status;
//String pH_status;
String DO_status;
String T_int_string;
String H_int_string;
double DateTime;
double Tvalue;
double ECvalue;
//double pHvalue;
double DOvalue;
double NTUV;
double Pvalue;
float internal_Tvalue;
float internal_Hvalue;
//Sampling time
int SampleTime = 30; //Sample time in seconds
int SleepTime = 60; //minutes to sleep before waking again / target is 15 minutes in the field / 60 minutes during testing
int StartTime;
int elapsedTime;
RunningMedianFloat floatSamples = RunningMedianFloat(10); // number of samples to take for each composite sample - water quality
RunningMedianFloat floatSamples_int_T = RunningMedianFloat(4); // number of samples to take for each composite sample - internal temperature
RunningMedianFloat floatSamples_int_H = RunningMedianFloat(4); // number of samples to take for each composite sample - internal humidity
RunningMedianFloat floatSamples_NTUV = RunningMedianFloat(10);
RunningMedianFloat floatSamples_int_P = RunningMedianFloat(10);
//SystemSleepConfiguration config;
////////////////////////////////////////////////////////////////////////////////////////////////
void setup() //hardware initialization.
{
Serial.begin(9600); //enable serial port.
//SerialLogHandler logHandler(LOG_LEVEL_ALL);
//ApplicationWatchdog wd(660000, System.reset); //This Watchdog code will reset the processor if the dog is not kicked every 11 mins which gives time for 2 modem reset's.
//set system power configuration
SystemPowerConfiguration conf;
conf.powerSourceMaxCurrent(550)
.powerSourceMinVoltage(4840)
.batteryChargeCurrent(1024)
.batteryChargeVoltage(4210);
System.setPowerConfiguration(conf);
//Log.info("setPowerConfiguration=%d", res);
Particle.variable("BatteryVoltage", BatteryVoltage);
Particle.variable("BatterySOC", BatterySOC);
Particle.variable("powerSource", powerSource);
Particle.variable("batteryState", batteryState);
Particle.variable("temperature", Tvalue);
Particle.variable("conductivity", ECvalue);
Particle.variable("disOxygen", DOvalue);
Particle.variable("turbidity", NTUV);
Particle.variable("pressure", Pvalue);
Particle.variable("internalTemp", T_int_string);
Particle.variable("internalHumid", H_int_string);
Wire.begin(); //enable I2C port.
pinMode(PWR5V, OUTPUT);
}
//Main Loop
////////////////////////////////////////////////////////////////////////////////////////////////
void loop()
{
//check at startup there is enough power
/////////////////////////////////////////////////////////////////////////
Serial.println("Checking power source");
int powerSource = System.powerSource();
Serial.printlnf("Power source = %i", powerSource);
Serial.println("Checking battery state");
int batteryState = System.batteryState();
Serial.printlnf("Battery state = %i", batteryState);
Serial.println("Checking battery level");
BatteryVoltage = fuel.getVCell();
BatterySOC = fuel.getNormalizedSoC();
Serial.print("Battery Voltage: ");
Serial.println(BatteryVoltage);
Serial.print("Battery State of Charge: ");
Serial.println(BatterySOC);
// More sophisticated battery control for field use - commented out for testing purposes
if (BatterySOC < double(20))
{
Serial.println("Battery below 20% : will sleep for 12 hours to charge battery");
SleepTime = (12*60);
}
else if (BatterySOC < double(40))
{
Serial.println("Battery below 40% : will sleep for 6 hours to charge battery");
SleepTime = (6*60);
}
else if (BatterySOC < double(60))
{
Serial.println("Battery below 60% : will sleep for 2 hours to charge battery");
SleepTime = (2*60); // 2 hours
}
else if (BatterySOC < double(80))
{
Serial.println("Battery below 80% : will sleep for 1 hour to charge battery");
SleepTime = (1*60); // 2 hours
}
else
{
Serial.printlnf("Battery level OK - above 80% - will sleep for %i minutes", SleepTime);
delay(2000);
}
/////////////////////////////////////////////////////////////////////////
Serial.println(("Turn on power to sensors")); //turn on power to sensors
digitalWrite(PWR5V, HIGH);
//Initialise DHT temperature adn humidity sensor
Serial.println("initialising temp and humidity sensor DHT22");
dht.begin();
delay(2000);
//Temp and Humidity
////////////////////////////////////////////////////////////////////////////////////////////////////////////
StartTime = Time.now();
Serial.print("Begin temperature and humidity sampling at: ");
Serial.println(StartTime);
floatSamples_int_T.clear();
floatSamples_int_H.clear();
while (Time.now() - StartTime < 6)
{
internal_Tvalue = dht.getTempCelcius();
internal_Hvalue = dht.getHumidity();
// Check if any reads failed and exit early (to try again).
if (isnan(internal_Hvalue) || isnan(internal_Tvalue) )
{
Serial.println("Failed to read from DHT sensor!");
//T_int_string = "no read";
}
else
{
Serial.printlnf("Spot temperature = %f, Spot humidity = %f", internal_Tvalue, internal_Hvalue);
floatSamples_int_T.add(internal_Tvalue);
floatSamples_int_H.add(internal_Hvalue);
//T_int_string = "yes read";
}
delay(2000);
}
Serial.print("Finished temperature and humidity sampling at: ");
Serial.println(Time.now());
internal_Tvalue = floatSamples_int_T.getMedian();
internal_Hvalue = floatSamples_int_H.getMedian();
T_int_string = String(internal_Tvalue);
H_int_string = String(internal_Hvalue);
Serial.printlnf("Median temperature = %f, Median humidity = %f", internal_Tvalue, internal_Hvalue);
//Water Quality - Temperature
////////////////////////////////////////////////////////////////////////////////////////////////////////////
ezoCommand("W", 102);
Serial.println(("checking T board status: "));
T_status = ezoCommand("Status", 102); //Check the status
Serial.print("T status: ");
Serial.println(T_status);
StartTime = Time.now();
Serial.print("Begin T sampling at: ");
Serial.println(StartTime);
floatSamples.clear();
while (Time.now() - StartTime < SampleTime/3)
{
T_string = ezoCommand("R",102);
Tvalue = atof(T_string);
Serial.print("Spot T = ");
Serial.println(Tvalue);
floatSamples.add(Tvalue);
delay(1000);
}
Serial.print("Finished T sampling at: ");
Serial.println(Time.now());
delay(200);
Tvalue = floatSamples.getMedian();
Serial.print("Median T = ");
Serial.println(Tvalue);
T_string = String(Tvalue);
//Water Quality - Conductivity
////////////////////////////////////////////////////////////////////////////////////////////////////////////
Serial.println(("checking EC board status: "));
EC_status = ezoCommand("STATUS",100); //Check the status
Serial.print(("EC status: "));
Serial.println(EC_status);
Serial.print("Setting EC temperature compensation at: ");
Serial.println(Tvalue);
ezoCommand(String("T,"+T_string), 100);
delay(500);
StartTime = Time.now();
Serial.print("Begin EC sampling at: ");
Serial.println(StartTime);
floatSamples.clear();
while (Time.now() - StartTime < SampleTime)
{
EC_string = ezoCommand("R",100);
ECvalue = atof(EC_string);
Serial.print("Spot EC = ");
Serial.println(ECvalue);
floatSamples.add(ECvalue);
delay(1000);
}
Serial.print("Finished EC sampling at: ");
Serial.println(Time.now());
delay(200);
ECvalue = floatSamples.getMedian();
Serial.print("Median EC = ");
Serial.println(ECvalue);
/*
//Water Quality - pH
////////////////////////////////////////////////////////////////////////////////////////////////////////////
Serial.println(("checking pH board status: "));
pH_status = ezoCommand("STATUS",99); //Check the status
Serial.print(("pH status: "));
Serial.println(pH_status);
Serial.print("Setting pH temperature compensation at: ");
Serial.println(Tvalue);
ezoCommand(String("T,"+T_string), 99);
delay(500);
StartTime = Time.now();
Serial.print("Begin pH sampling at: ");
Serial.println(StartTime);
floatSamples.clear();
while (Time.now() - StartTime < SampleTime)
{
pH_string = ezoCommand("R",99);
pHvalue = atof(pH_string);
Serial.print("Spot pH = ");
Serial.println(pHvalue);
floatSamples.add(pHvalue);
delay(1000);
}
Serial.print("Finished pH sampling at: ");
Serial.println(Time.now());
delay(200);
pHvalue = floatSamples.getMedian();
Serial.print("Median pH = ");
Serial.println(pHvalue);
*/
//Water Quality - Dissolved Oxygen
////////////////////////////////////////////////////////////////////////////////////////////////////////////
Serial.println(("checking DO board status: "));
DO_status = ezoCommand("STATUS",97); //Check the status
Serial.print(("DO status: "));
Serial.println(DO_status);
Serial.print("Setting DO temperature compensation at: ");
Serial.println(Tvalue);
ezoCommand(String("T,"+T_string), 97);
delay(500);
StartTime = Time.now();
Serial.print("Begin DO sampling at: ");
Serial.println(StartTime);
floatSamples.clear();
while (Time.now() - StartTime < SampleTime)
{
DO_string = ezoCommand("R",97);
DOvalue = atof(DO_string);
Serial.print("Spot DO = ");
Serial.println(DOvalue);
floatSamples.add(DOvalue);
delay(1000);
}
Serial.print("Finished DO sampling at: ");
Serial.println(Time.now());
delay(200);
DOvalue = floatSamples.getMedian();
Serial.print("Median DO = ");
Serial.println(DOvalue);
// Turbidity
////////////////////////////////////////////////////////////////////////////////////////////////////////////
StartTime = Time.now();
Serial.print("Begin turbidity sampling at: ");
Serial.println(StartTime);
floatSamples_NTUV.clear();
while (Time.now() - StartTime < SampleTime)
{
NTUV = analogRead(TURBPIN);
Serial.print("NTU: ");
Serial.println(NTUV);
floatSamples_NTUV.add(NTUV);
delay(500);
}
Serial.print("Finished turbidity sampling at: ");
Serial.println(Time.now());
delay(500);
NTUV = floatSamples_NTUV.getMedian();
Serial.print("Median turbidity: ");
Serial.println(NTUV);
delay(1000);
// Pressure transducer - Water level sensor
////////////////////////////////////////////////////////////////////////////////////////////////////////////
StartTime = Time.now();
floatSamples_int_P.clear();
while (Time.now() - StartTime < SampleTime)
{
Pvalue = analogRead(PRESSPIN);
floatSamples_int_P.add(Pvalue);
delay(500);
}
delay(500);
Pvalue = floatSamples_int_P.getMedian();
Serial.print("Median pressure: ");
Serial.println(Pvalue);
delay(20000);
Serial.println("Turning off power to all sensors");
digitalWrite(PWR5V, LOW);
//Sens data to particle cloud
////////////////////////////////////////////////////////////////////////////////////////////////////////////
Serial.println("Preparing sensor data");
delay (1000);
Serial.printlnf("Sensor data - Temp: %f EC: %f DO: %f NTUV: %f Internal Temp: %f Internal Hum: %f Battery Voltage: %f Battery Charge: %f", Tvalue, ECvalue, DOvalue, NTUV, internal_Tvalue, internal_Hvalue, BatteryVoltage, BatterySOC);
Serial.printlnf("Sensor status - T Status: %s EC Status: %s", T_status.c_str(), EC_status.c_str());
delay (1000);
Serial.println("Uploading sensor data");
// *** Comment out following line because don't need to publish data ***
//createPayload(DateTime, Tvalue, ECvalue, DOvalue, NTUV, internal_Tvalue, internal_Hvalue, BatteryVoltage, BatterySOC);
delay (3000);
//Serial.println("Turning off cellular modem");
//Cellular.off();
Serial.printlnf("Going to sleep for %i minutes" , SleepTime);
//System.sleep(config);
// *** Comment out sleep (following line) during testing (Edmond 11 Jan 2022) ***
//System.sleep(SLEEP_MODE_DEEP, (SleepTime * 60)); // wake up every 15 minutes
// 60 second delay between readings
// *** Comment out following line when publishing data and using deep sleep
delay(60000);
}
// Function Definitions
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Read from sensors
////////////////////////////////////////////////////////////////////////////////////////
String ezoCommand(String command, int address)
{
Wire.beginTransmission(address); //call the circuit by its ID number.
Wire.write(command); //transmit the command that was sent through the serial port.
Wire.endTransmission(); //end the I2C data transmission.
delay(delay_time);
Wire.requestFrom(address,48,1); //call the circuit and request 20 bytes (this may be more than we need)
code=Wire.read();
while(Wire.available())
{ //are there bytes to receive.
in_char = Wire.read(); //receive a byte.
ezo_data[i]= in_char; //load this byte into our array.
i+=1; //incur the counter for the array element.
if(in_char==0)
{ //if we see that we have been sent a null command.
i=0; //reset the counter i to 0.
Wire.endTransmission(); //end the I2C data transmission.
break; //exit the while loop.
}
}
switch (code){ //switch case based on what the response code is.
case 1: //decimal 1.
Serial.print("Success - "); //means the command was successful.
break; //exits the switch case.
case 2: //decimal 2.
Serial.print("Failed"); //means the command has failed.
break; //exits the switch case.
case 254: //decimal 254.
Serial.print("Pending"); //means the command has not yet been finished calculating.
break; //exits the switch case.
case 255: //decimal 255.
Serial.print("No Data"); //means there is no further data to send.
break; //exits the switch case.
}
if (code == 1){
return ezo_data ;
}
else {
return "";
}
}
//JSON generator function for sensor data
///////////////////////////////////////////////////////
void createPayload(double DateTime, double Tvalue, double ECvalue, double DOvalue, double NTUV, double internal_Tvalue, double internal_Hvalue, double BatteryVoltage, double BatterySOC)
{
JsonWriterStatic<256> jw;
{
JsonWriterAutoObject obj(&jw);
jw.insertKeyValue("DateTime", Time.now());
jw.insertKeyValue("T", Tvalue);
jw.insertKeyValue("EC", ECvalue);
jw.insertKeyValue("DO", DOvalue);
jw.insertKeyValue("NTUV", NTUV);
jw.insertKeyValue("internal_T", internal_Tvalue);
jw.insertKeyValue("internal_H", internal_Hvalue);
jw.insertKeyValue("BatteryVoltage", BatteryVoltage);
jw.insertKeyValue("BatterySOC", BatterySOC);
}
Particle.publish("PayloadJSON", jw.getBuffer(), PRIVATE);
}