Learning Objectives

Learn how to use the requests library in python to get data from a URL
Learn how to interpret data presented in the JSON format
Learn how to process data in python so that it can be saved in variables and in files
Dweeting data
Creating Dashboards using Node-RED and Plotly.

Requesting new data from a sensor URL in Python

Sensors are now commonly connected to the internet.
This has given rise to the term – The Internet of Things.
If a sensor is connected to the internet then it will often have a URL (Universal Resource Locator) to allow us to interact with the sensor.
One common interaction is to be able to see or download data collected by the sensor.
To get data from a sensor we need to import the requests library in python.
In this lesson we will download some data from an atmospheric sensor.
The sensor data includes temperature, air pressure, sensor battery voltage and signal strength.
In this example we write a python3 program to download sensor data using the requests library.
The data will be presented in the python Shell.

Open IDLE3 Python Editor

From the Raspberry Pi main menu select Programming > Python3 (IDLE)
If IDLE is not installed open the Terminal and enter the commands:
- sudo apt-get update
- sudo apt-get install idle3

Create a folder named water_level_sensor using the File Manager

Click on the File Manager
Select File and New Folder

Enter the new folder name water_level_sensor
This is where we will store our files related to the water sensor.

Create a folder named water_level_sensor using the Terminal

Alternatively, you can make a folder using the Terminal.
Open the Terminal and enter the command ls to list the contents in the default user home directory (/home/pi/)

To create a new folder enter the command mkdir water_level_sensor

Create a new file in python named water_level_sensor.py

Go back to the IDLE3 program
From the File drop down menu select New File.
This will open up a new file Editor Window.

To save the new file, select File > Save As

Navigate to the water_level_sensor directory and then save the file as atm_sensor_get.py
In Linux we never use names that include spaces.
Replace spaces with either the _ (underline) symbol or the – (hyphen) symbol.

Request data from sensor using python

Enter the following lines of code to request data from the water level sensor.
Note that each sensor will have a different URL.
The API (URL) for this activity will be https://app.alphax.cloud/conduitv6?id=F008D1D0968C&token=5ecf2e35d99a404a81408792&limit=1&ch=1

The key items in the code are described:
- import requests – this statement imports a python library so that the python program can get data from the internet.
- r = requests.get(URL) – this statement gets the data from a specified URL and stores the result in the variable object named r. URL is short for Uniform Resource Location, also known as a web address.
- print(r) – this statement prints a code that determines if the request was successful or not.
Save and the Run the program.

Change to the Python Shell screen.
The response in the Python Shell should be [200].
A value of [200] means that the request was successful.
Other numbers or errors will indicate that there is a problem either with the Python code or with communication with the sensor.

View sensor data in JSON format

We can now add some more code so that we can look at the actual data.
Add the following statements to the code:
- import json
- json_level_data = json.dumps(r_level.json())
- print(json_level_data)

The command json.dumps() converts a Python object String Python object (dictionary) into a JSON string by using the json.dumps() method [1].
JSON stands for JavaScript Object Notation.
It is an open standard to help format data for data exchange between computers and it uses human-readable text.
The data is stored in attribute-value pairs similar to the format of a dictionary.

You can see the data output in the python Shell Window.

Unfortunately, presenting the data this way is not very readable.
In the next section of this lesson we will learn how to view and interpret data stored in the JSON format.

JSON Viewer

Copy the Shell output and paste the text into the https://jsonviewer.com/ using the Web Browser on the Pi.
You may need to copy and paste the text in a TextEditor on the Raspberry Pi and then copy it to a Browser.
Paste the copied text in the jsonviewer left window.
Click on the Viewer button to see the same text in a more human friendly format.
The data we are most interested in are:
- val_date - time stamp when sensor data was last updated
- dev_tag - the name of the sensor. In this case Whittlesea Water sensor no. 12
- val_calibrated - the actual water level measured by the sensor in metres.

Learning how to extract Water level data from a Python List

To extract the relevant data from a Python list we enter the following code print(r.json[0][“val_calibrated”])
- [0] – represents the first data set in the list
- [“val_calibrated”] – is the attribute in the attribute-value pair that pairs with the water level value
- print() – will print out the water level data to the Python Shell

The result can be seen on the last line of the Python Shell.
In this case the water level is 0.946 (metres).

Assigning water level data into a variable

Now that we know how to find and extract water level data we can also pass on this value to a variable.
Putting the water level data in a variable allows us to save or manipulate the data more easily, just like in algebra.
Add the following code:
- water_level = r.json()[0][“val_calibrated”] – assign water level data to variable named water_level
- print(f'The water level is {water_level}') – display the value held in water_level variable with some descriptors
Save and then Run this program.

The output of the program is shown below in the Python Shell window.
The very last line of output shows the output of the print() statement.

900px

The output of the program has a lot of information we no longer need.
To reduce the clutter from the output we can comment out some of the statements in our program using the # (hash) symbol.
Note that when we comment out statement the text will change to a red colour.
If we comment out statements the computer will no longer execute (run) them.
Follow the example below and comment out three of the print() statements.
Save and Run the program.

You can see that the output of the program has been simplified.
We can always uncomment these print() statements in the future.

Collecting battery voltage data from the water level sensor

The water level sensor also make battery voltage data available.
The URL request uses channel 255 to request the battery voltage data.
Battery voltage should read between 3.6 and 4.2 Volts.
If the battery voltage drops to less than 3.6 Volts it needs to be charged.
Update the Python code to the following.

The following output will be displayed in the Python Shell.

Adding Time and Date data

Sometime we need to capture the actual date and time on a computer running a program.
To do this we import the datetime library in Python.
The output of the datetime is not human readable and needs to be converted using the strftime() method.
To format the datetime object we need to use some special characters called Directives.
Each Directive has a percentage sign (%) and a single upper or lower-case letter (a-z, A-Z).
- %Y – full version of year – e.g. 2021
- %b – month name short version – e.g. Jan, Feb, Mar
- %d – day of the month 01 to 31 – e.g. 28
- %H – hour 00 to 24 – e.g. 08
- %M – minute 00 to 59 – e.g. 48
To use these Directives to format the date object add the following two lines of code.
- date_stamp = now.strftime("%Y-%b-%d %H:%M")
- print(f'This python program was run at this time {date_stamp}')
The full code is included below.

Save and Run the program.
The date and time output has been highlighted.

Last Sensor Transmission Time and Date stamp

Saving data to a file

Data can be collated and saved into a text file.
Firstly, organise the print statements and outputs so they are in a logical order.
The statement data = str(sensor_date) + "," + str(water_level) + "," + sensor_datetime + "," + str(battery) + "\n" assigns all the data to the variable named data
The individual data is comma delimited.
This will make the data easy to import into a Spreadsheet application.
The data can also be interpreted by graphing packages such as plotly

The output of the program is shown below.

To show that the program is outputting the correct data, open File Manager
Open the file water_level_sensor_data.txt to see the entries in the text file.

Note that the water level data is shown to multiple significant figures.
We can reduce the number of digits by editing the data assignment statement to the following.
data = str(sensor_date) + "," + str(water_level)[:5] + "," + sensor_datetime + "," + str(battery) + "\n"
This will now only save 5 digits (including the decimal point).

Python code

import requests
import json
import datetime

now = datetime.datetime.now()
date_stamp = now.strftime("%Y-%b-%d %H:%M:%S")
print(f'This python program was run at this time {date_stamp}')

URL_level = 'https://app.alphax.cloud/conduitv6?id=F008D1D0968C&token=5ecf2e35d99a404a81408792&limit=1&ch=1'
URL_battery = 'https://app.alphax.cloud/conduitv6?id=F008D1D0968C&token=5ecf2e35d99a404a81408792&limit=1&ch=255'

r_level = requests.get(URL_level)
r_battery = requests.get(URL_battery)

sensor_date = r_level.json()[0]['val_date']
water_level = r_level.json()[0]['val_calibrated']
sensor_datetime = datetime.datetime.fromtimestamp(sensor_date).strftime("%Y-%b-%d %H:%M:%S")
battery = r_battery.json()[0]['val_calibrated']

print(f'The Epoch time (timestamp) for the last sensor reading is {sensor_date}')
print(f'The water level is {water_level}')
print(f'The last sensor recording was made on {sensor_datetime}')
print(f'The battery voltage of the sensor is {battery}')

data = str(sensor_date) + "," + str(water_level)[:5] + "," + sensor_datetime + "," + str(battery) + "\n"
print(f'The data is =  {data}')

f = open('/home/pi/water_level_sensor/water_level_sensor_data.txt','a')
f.write(data)
f.close()

Plotting data to a html file

* In this example we are going to install the plotly python package and then write a program to create some simple graphs.

Installation of the plotly python package

From the Raspberry Pi top bar menu click on the Terminal icon.
Open the Terminal and enter the command pip3 install plotly
pip is short for Pip Installs Python.
pip3 installs packages for python3
The version of plotly that was installed is plotly-5.13.1
Other python packages may also be installed to support plotly.

To test that plotly has been installed correctly open the Python Shell and enter - import plotly

Installing pandas python package to read data from text files

To read data from our data.txt file we need to install another python package named pandas.
To install the pandas package open the Terminal and enter the following commands.
- pip3 install pandas – will take up to 5-10 minutes to install. On an old laptop (2017) it took 30 minutes for pandas to install.
- sudo apt-get update – update all libraries on the Raspberry Pi
Once pandas has been installed check to see if pandas can be imported in Thonny.
Use the command import pandas
If there is no error pandas has been installed correctly.
If however there is an error try installing these additional libraries.
- sudo apt-get install libatlas-base-dev – extra missing software package to support pandas
To test that pandas has been installed correctly open the Python Shell and enter - import pandas
If the pandas python package has installed correctly you should see no errors.
There are errors try installing this additional package
- sudo apt-get update – update all libraries on the Raspberry Pi
- sudo apt-get install libatlas-base-dev – extra missing software package to support pandas

Importing and plotting data from text files

Plotly is good at plotting graphs, but it cannot important data from text files.
To prepare data for plotly we need another python package named pandas.
In the following lesson we will install pandas and demonstrate how to import data from a text file.
Before we can start using pandas we also need to clean and label the data in our text file to make sure it is ready to import.

Cleaning and adding labels to sensor data

To import data from our data.txt file need to clean the data and give labels to the data.
Click on the File Manager.
Navigate to your project’s directory and open the data.txt file.
By default, it should open in the Mousepad text editor.

The first line of the data file needs to include a label for each data set
Data labels for our file are: Date,Temp,Pres,Volt
Make sure there are no spaces between the labels. Spelling must be exactly as shown.
Delete or correct any incomplete data.
Prepare your water_level_sensor_data.txt file so it looks like the following.
The labels use the same naming convention as the variables in our python code.
- sensor_date - Epoch time for the last sensor reading
- water_level - water level
- sensor_datetime = human readable date time for the last sensor reading
- battery - battery voltage
Save and close the file.

Graphing data using plotly and pandas

The panda library will work with plotly to make graphs that we can insert into web pages.
Plotly plots the data, and pandas reads data and processes data from the data.txt file.
- df = pandas.read_csv('/home/pi/water_level_sensor/water_level_sensor_data.txt') - reads the data.txt file and puts the data in a df Object.
- x=df.sensor_datetime - assigns the datetime string as the X-axis on our graph
- y=df.water_level - assigns the water_level to the Y-axis
- title="Smart Tank Water Level" - is the Title of the graph
- xaxis_title = 'Time' - is the label for the X-axis
- yaxis_title = 'Water Level' - is the label for the Y-axis
- plotly.offline.plot(fig,filename="/home/pi/water_level_sensor/water_level.html",auto_open=False) - saves the graph to the file named water_level.html

This is the output of the water level data in the Web Browser.
Some additional data points have been added to make the data points on the graph easier to read.

Python code

import requests
import json
import datetime
import plotly
import plotly.graph_objects as go
import pandas

now = datetime.datetime.now()
date_stamp = now.strftime("%Y-%b-%d %H:%M:%S")
print(f'This python program was run at this time {date_stamp}')

URL_level = 'https://app.alphax.cloud/conduitv6?id=F008D1D0968C&token=5ecf2e35d99a404a81408792&limit=1&ch=1'
URL_battery = 'https://app.alphax.cloud/conduitv6?id=F008D1D0968C&token=5ecf2e35d99a404a81408792&limit=1&ch=255'

r_level = requests.get(URL_level)
r_battery = requests.get(URL_battery)

sensor_date = r_level.json()[0]['val_date']
water_level = r_level.json()[0]['val_calibrated']
sensor_datetime = datetime.datetime.fromtimestamp(sensor_date).strftime("%Y-%b-%d %H:%M:%S")
battery = r_battery.json()[0]['val_calibrated']

print(f'The Epoch time (timestamp) for the last sensor reading is {sensor_date}')
print(f'The water level is {water_level}')
print(f'The last sensor recording was made on {sensor_datetime}')
print(f'The battery voltage of the sensor is {battery}')

data = str(sensor_date) + "," + str(water_level)[:5] + "," + sensor_datetime + "," + str(battery) + "\n"
print(f'The data is =  {data}')

f = open('/home/pi/water_level_sensor/water_level_sensor_data.txt','a')
f.write(data)
f.close()

# import and plot data text file
df = pandas.read_csv('/home/pi/water_level_sensor/water_level_sensor_data.txt')

fig = go.Figure(data = go.Scatter(mode='markers',
                                    x=df.sensor_datetime,
                                    y=df.water_level,
                                    marker=dict(
                                        color="LightSkyBlue",
                                        size=20)
                                    )
                )
fig.update_layout(title="Smart Tank Water Level",
                    xaxis_title = 'Time',
                    yaxis_title = 'Water Level',
                    font=dict(
                        size=20,
                        color="RebeccaPurple"
                        )
                    )

plotly.offline.plot(fig,
                    filename="/home/pi/water_level_sensor/water_level.html",
                    auto_open=False)

print("Smart tank water level data plotted")

Smart Tanks for Schools project

Contents

Learning Objectives

Requesting new data from a sensor URL in Python

Open IDLE3 Python Editor

Create a folder named water_level_sensor using the File Manager

Create a folder named water_level_sensor using the Terminal

Create a new file in python named water_level_sensor.py

Request data from sensor using python

View sensor data in JSON format

JSON Viewer

Learning how to extract Water level data from a Python List

Assigning water level data into a variable

Collecting battery voltage data from the water level sensor

Adding Time and Date data

Last Sensor Transmission Time and Date stamp

Saving data to a file

Python code

Plotting data to a html file

Installation of the plotly python package

Installing pandas python package to read data from text files

Importing and plotting data from text files

Cleaning and adding labels to sensor data

Graphing data using plotly and pandas

Python code

Navigation menu

Smart Tanks for Schools project

Learning Objectives

Requesting new data from a sensor URL in Python

Open IDLE3 Python Editor

Create a folder named water_level_sensor using the File Manager

Create a folder named water_level_sensor using the Terminal

Create a new file in python named water_level_sensor.py

Request data from sensor using python

View sensor data in JSON format

JSON Viewer

Learning how to extract Water level data from a Python List

Assigning water level data into a variable

Collecting battery voltage data from the water level sensor

Adding Time and Date data

Last Sensor Transmission Time and Date stamp

Saving data to a file

Python code

Plotting data to a html file

Installation of the plotly python package

Installing pandas python package to read data from text files

Importing and plotting data from text files

Cleaning and adding labels to sensor data

Graphing data using plotly and pandas

Python code

Navigation menu

Search