Requesting data from a Water Quality Sensor in Peter Hopper lake

Overview

• The aim of this lesson is to learn how to request data from a sensor using Python.
• Every Internet of Things (IoT) device can now be accessed using the internet.
• To retrieve data from a device you simply ask for the data using a special URL (Universal Resource Locator).
• We will learn how to construct this URL to get the specific data that we need.
• IoT Devices that make their data available support these requests with an Application Programming Interface (API).
• What this really means is that when they receive a query from the internet that know what reply to give.

Learning Objectives

• Learn how construct a URL request in Python using the requests.get() method.
• Learn how to extract relevant information from JSON data received from an IoT device

Constructing a URL request in python

Opening the peter_hopper_get_data.py file in Geany

• From the main Raspberry Pi menu navigate to Programming and open Geany.
• Within Geany, go to the File menu and select Open.
• Navigate to the peter-hopper-t3 directory and open peter_hopper_get_data.py
• The initial setup in Geany should look like the following.

Adding static variables – Data Address and API Key

• In order to access data from an IoT device we need to know where the data is stored (by supplying the Data Address) and we need to supply an API Key.

• Data Address
  • data addresses are represented using Hexadecimal notation – 0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f
  • An address can be very long – 60dbb5b653b63e0cd9b353ca
  • Each data item has its own unique address. Can you find the differences below?
    • battery voltage - 60dbb5b653b63e0cd9b353ca
    • temperature - 60dbb5b653b63e0cd9b353c0
    • humidity - 60dbb5b653b63e0cd9b353c8

• API Keys
  • API Keys gives your computer permission to access the data on the sensor.
  • If you don’t have an API Key, you will be denied access.
  • Anyone can steal an API Key, so you should not share it with anyone or publish it.
  • When entering Keys and Addresses be very careful about 0 (zero), o (lower case letter o), O (capital letter O), symbols

• Because Data Addresses and API Keys are long its best to write them up as pairs with spaces in between.
• The computer doesn’t like spaces, so we need to copy the line and create it again – without all the spaces.
• This will help to make your code more human readable and easier to debug.
• We comment out the human readable line with the hash (#) symbol.
• In the code below we created two variables:
  • peter_hopper_bat – holds the Battery Data Address
  • api_key – holds the API Key

• When variables don’t change in a program, we call them Static.
• Static variables are normally written in capitals, but they can be hard to read – e.g. PETER_HOPPER_BAT, API_KEY
• Update your code using the example below, then Save your file.

Building a Request using a Uniform Resource Identifier (URI) and Headers

• To build our Request to the IoT sensor we need to provide a few more details.
• A Uniform Resource Identifier (URI) – which is an internet address
• A Header - to give the server more contextual information

• This URI is easy to understand:
  • https://api.eagle.io/api/v1/nodes/
  • Note that HTTPS is the secure version of HTTP.
  • It is encrypted during transmission.

• This Header(s) contains:
  • data transfer format requirements– e.g. JSON format. JSON looks like this {‘Super hero’:”Spiderman”} – we call it a property and value pair.
  • API Key information – to give us permission to access the data on the server

Python Request get() Method

• We can put all our prepared variable together to create a request.
• The format is REQUESTS.GET (URL, HEADERS)
  • URL is formed by joining the URI and the Data Address together
  • HEADERS – no change

• To see if our program works correctly, we can run it and examine the output in the Python Shell.
• If the Response number is 200, then everything in your code is correct.
• This is the OK response.
• Other numbers normally give a clue to why the request has failed.
• Remember that the error generally will be somewhere in your Python code, unless the sensor or server is not working.

How to extract relevant data from requests

• Now that we know that we can request sensor data we need to make sense of it and just extract the relevant data we need.

JSON sensor data in the Python Shell

• Update your code by adding two additional lines:
  • bat = str(r_bat.json()) – takes the data retrieved and converts it to JSON format then to a String and saves the ** result in the new variable bat
  • print(bat) – prints the contents of variable bat

• If we Save and Run the code example the output should look like the following.
• Although it looks very confusing it is a combination of property and value pairs.
  • {‘property’:’value’}

Visualising JSON data in Firefox

• We can visualise the data returned more easily if we visualise it in Firefox.
• Copy the JSON data output in the Shell and Paste in Raspberry Pi > Accessories > TextEditor
• Select All the text in the TextEditor and right mouse button click Copy the text.
• On you laptop open NotePad and Past the code.
• You need to use NotePad to replace some of the text:
  • replace single quotes (‘) with double quotes (“) – Firefox doesn’t like single quotes
  • replace None with “None” – because it is text
  • replace True with “True” – because it is text
  • replace False with “False” – because it is text

• Save the changes and name the file data1.json

• Download the Firefox Web Browser and install it on your computer.
• Now open up the data1.json file in Firefox to visualise the data.
• If everything goes to plan you should see the following displayed.
• This is how computers see json data.

• If you still get errors, you can put each new set of data on a new line – by pressing the Enter Key after each comma.
• This will help you find errors more easily.
• An error will normally give you a line number and column number.

• In this example, on the third line the word alarm is incorrectly formatted with incorrect quotation marks.

• As this code is processed in Firefox the error message tells us that the error is in line 3 column 11.

Locating the relevant sensor data

• Scroll down the JSON data page displayed in Firefox until you find the property labelled currentValue.
• The value associated with currentValue is 3.8125.
• This is the battery voltage of our sensor

• Return to your Python code and edit the second last line to include [‘currentValue’]
• This will instruct the json() method to only return the Value associated with the Property currentValue

• The result will be displayed in the Python Shell.
• In this case the battery voltage is 3.82, similar to the value displayed in the Firefox Browser.

Requesting more data from sensors

• Now that we know how to request one sensor parameter (battery voltage), we can easily expand this to other parameters.

Adding more sensor parameter addresses

• To access a sensor parameter we need to supply a specific address.
• Update your file with the following addresses to prepare for measuring water temperature, dissolved oxygen and the internal temperature within the sensor.

• Add additional code in the area where the request.get() commands are located.

Checking requests are error free

• And update the code to print out the error code for each returned request.
• If the code is run now it should return a value of [200] in the Python Shell if the request was successful.

• Here is an example of a successful print output in the Python Shell.

Using JSON to extract relevant information from requests

• If the request is successful, the data from the sensor will be stored in the respective request variable.
• For example, dissolved oxygen data will be stored in the r_oxy variable (Really an object type).
• An object is a collection of data and functions that operate on those data.
• To extract just the data we use the json function.
• The json function will find the ‘currentValue’ attribute and extract the associated data value.

Using print() to make it easy to read data values

• To print out the data values for each variable we use the print() function.
• Notice that these print() functions also include some descriptive text to help explain the data value.

• The result will appear in the Python Shell when the program is run.

Saving the additional data and automating your program

• Now that we can extract all the data, we need to save the data in a file and also learn how to automate the running of the python program.

Using the datetime library in python to get the current date and time

• Verify that your code has three import statements.
• Also update your code so that the current time stored in the now object variable type, is processed using the strftime() function so that the year (%Y), month(%b), day(%d), hour(%H) and minute(%M) are correctly formatted.
• The current time and date will be stored in the variable date_stamp.

Create an empty text file for storing data

• Open the Terminal and list all contents by entering the command ls

• Navigate to the peter-hopper-t3 directory with the command cd peter-hopper-t3

• List all the files and directories in the ~/peter-hopper-t3 directory with the command ls

• To create an empty file with the name data.txt enter the command touch data.txt
• Check that the new file has been created with the command ls

• To list the directory contents in long format enter the command ls -l

Using the inbuilt manual (man) in Linux

• If your forget some of the command in Linux you can use the inbuilt manual (man) command.
• In this example we will examine the manual for list (ls) by entering the command man ls

• Press the space bar to navigate down the pages in the manual, or the arrow keys to move up and down.
• Enter q to quit the manual.

• If you keep on scrolling down, you will eventually see the -l option and details on what is does (use a long listing format)

Writing data to a file

• Return to your python file and at the end of the file enter the following code.
• This code will:
  • open a file named data.txt – the ‘a’ term means that new data will be added to the end of the file (appended). If we had used the ‘w’ term, the old data in the file would be written over by the new data. The ‘r’ term is used when we only need to read the file contents, not to write to the file.
  • f.write() – writes our data to the file as one long String
  • f.close() – closes the file
  • print() – gives the computer user some feedback that the file update is complete.

• If you save and run the program the output should be similar to the one below.

Checking the contents of a text file

• We can check the contents of data.txt to see if the data was saved correctly.
• In the Terminal enter cat data.txt
• Notice that the data is in the file, but all the data is mashed together and is unreadable.

• Go back to the python program and modify the f.write() line of code to the following.
• This code now adds a comma “,” between each data item.
• We call this comma delimited or Comma Separated Values (CSV files).

• If we save and run this file again the contents will be more readable.
• But all the data is still on one line.
• We need each new set of data on a separate line.

• Modify the f.write() function to the following.
• Note the addition of the new line “\n” special character at the end. This will ensure that new data is always added on a new line.

• Run the program three times and then check the data.txt file using the command cat data.txt
• You can see that the new data added is on separate lines.
  • CHALLENGE – modify your file so that the f.write function also stores the date_stamp variable

Fixing up a text file

• We can clean up our data.txt file now that we know our program works.
• Open the File Manager, navigate to the peter-hopper-t3 directory.
• Find the file data.txt and right mouse button click on the file.
• Open the file using the Text Editor.

• Your file should look like the one below.
• Notice that the date_stamp variable contents have been added to the data.txt file.

• Clean up your file so that it looks like the one below – five sets of data separated by commas.

Automating your program using Crontab

• We will use Cron or Crontab program to automate the running of our python program.
• In the Terminal enter crontab -e
• The -e option allows us to edit the crontab program.
• To list the contents of crontab use the -l option (letter L)

• Scroll to the end of the crontab file and enter a single statement.
  • 5 * * * * * - runs the program at 5 minutes past the hour, every hour
  • python3 will be used to run the program peter_hopper_get_data.py

• Save the file by entering Ctrl-X to exit.
• Enter Y to save changes

• Press the Enter key to accept the default file name.

• Congratulations. Your file will keep on running so long as your Raspberry Pi is on.
• Don’t forget to check that new data is being added to your data.txt file.