Saving sensor data to a file
Overview
- In this lesson we are going to learn how to save our sensor data to a file.
- Saving data to a file will keep it safe.
- Once data is in a file, we can read the data again and make a graph.
- Reading, saving a graphing data is very important for all environmental and Citizen Science activities.
- Sensors often deliver data a numerical data. Some simple examples are temperature, time, speed, counts, etc.
- In this example we are going to save our data as a String (or text).
- Most files store data as text. In this example we also need to separate the data using commas (,).
- If data is separated by commas it makes each data element human readable and other programs, such as Spreadsheets, can also read the file more easily.
Learning Objectives
- Learn how convert numerical data into a String (or text)
- Learn how to save data to a new empty file
- Learn how to read a saved file and check for data integrity
Converting numerical data to a String
- In this example we will build on the python program from the previous lesson.
- Our python program was used to download data from a sensor (temperature, atmospheric pressure, battery voltage).
- We will now convert this data into a String and put all the data into a single variable named data.
- Each piece of data will be separated by a comma.
Open up the Python file named atm_sensor_get.py
- From the Raspberry Pi main menu drop down select Programming > Python3 (IDLE).
- This will open the Python Shell.
- From the File drop down menu select Recent Files.
- Choose the file named atm_sensor_get.py
- Edit the file so that it appears like the example below.
- Saving and Running the file should produce an output in the Python Shell like the output below.
Arranging numerical data in a String
- We can create (or define) a new empty variable named data to hold all the temperature, pressure and battery data using the statement data = “”
- We need to add or concatenate all the data together into one long String.
- To do this we need to:
- convert the numerical data to String using the str() function
- then add each data String together using the plus (+) symbol
- The final code is presented below.
- We also added a print(data) statement so that we can see data variable.
- Save and Run the program. The result should be like the following.
Adding spaces between data values
- The issue with the format of this data is that there is no gap between the data.
- To put a gap between the data we need to add a space using this additional code + “ “ +
- See the example below as a reference.
- The output of this program in the Shell is reproduced here.
Adding a human readable header to our data
- To make the data easier to read we can add an additional print() statement to act as a header for our data.
- print(“Temp, Pres, Bat”)
- With the additional print() header, it makes the data easier to read.
- We also remove the data = “” statement, because we don’t really need it in Python.
Comma separated data
- Rather than having a space separating the data it is better to use a comma (,).
- Replace all the “ “ code with “,” as in the example below.
- After saving and running the program you should get an output like the one below.
- This is an important first step in preparing data to be saved to a file.
Saving data to a file
- Data can be saved in a text file so that we don’t lose the data, and so that other programs on our computer can access the data.
- With bigger data sets you can also save data in a specialised database or data warehouse.
- Saving data to a file is much easier and it also allows us to more easily check the data for integrity.
- In this example we will be creating an empty file named data.txt and then saving new data to this file.
- Open the Terminal.
- When you open the Terminal you will be in your pi home directory. You should see the tilde (~) symbol.
- The path to the pi home directory is /home/pi yet for simplicity this is abbreviated to ~.
- Enter the command ls to list all the contents of the pi home directory.
- In my pi home directory, my project folder is named botanica-park-lake
- To move into the botanica-park-lake directory we enter the command cd botanica-park-lake
- cd is short for Change Directory
- One inside the botanica-park-lake directory we can also use the ls command to list all contents.
- Enter the command ls
- You should see a range of files similar to the ones displayed below.
Creating an empty file in the Terminal
- Before we can save data to a file, we need to create an empty file.
- Enter the command touch data.txt to create an empty file named data.txt
- To verify that the file has been created enter the command ls to list all directory contents.
- You should see the new file data.txt in the listing.
- Another way to list files is to add the option -l. This option lists the files in long format.
- Enter the command ls -l
- The file size value in bytes will be displayed in the column just before the month (e.g. Apr, May).
- We can see that the file size of data.txt is 0 bytes.
- For comparison, the file size of the birds.html file is 414 bytes.
Writing data to a file
- To write the data to a file we need to add three lines of code at the end of the python file.
- f = open(‘/home/pi/botanica-park-lake/data.txt’,’a’)
- f.write(data)
- f.close()
- In the line f = open()
- we identify the path to the file, in this case /home/pi/botanica-park-lake/data.txt
- we also say that we want to append (a) new data to the file. This continually adds new data to the end of the file.
- f.write(data) – this statements writes the data to the file.
- f.close() – this statement closes the file. By closing the file other programs can then access or read the file.
- If we run this file we won’t see any changes to the output in the Python Shell.
- To see the new data we need to view the contents of the data.txt file. Hopefully it should hold some data now.
- To view the contents of a text file in Linux we enter the command cat data.txt
- cat is short for concatenate.
- We can see that the output shows that we have some numerical data stored in the file.