Exploring soil moisture sensors

Connecting a Soil Moisture sensor to the Things network using Node-RED

• In this lesson we are going to use Node-RED as our main programming tool.
• Node-RED is a block or node based programming language.
• Each block has a discrete function and some blocks can contain Javascript code or be configured in different ways.
• Blocks are connected together using wires.
• Node-RED and the programs on the Raspberry Pi are always being updated, so it’s normally good practise to update our Raspberry Pi on a regular basis.

Overview

• In this lesson we are going to connect to a soil moisture sensor.
• The soil moisture sensor is connected to an Arduino MKR 1300 microcontroller (a small computer).
• The data will be transmitted to the LoRa network.
• LoRa is short for Long Range radio communication.
• The data is then sent to The Things Network, which is a community run server that collects data from sensors and makes the data available for download.
• We will download the data to our Raspberry Pi, and in future lessons display that data in a Dashboard using Node-RED.
• All these skills are very important in helping to address Climate Change and general Environmental Sustainability issues the world faces.

Learning Objectives

• Learn how to connect to The Things Network using MQTT using Node-Red
• Learn how to extract specific data from JSON formatted data packets

Updating and Upgrading the Raspberry Pi

• Open the Raspberry Pi Terminal.
• Enter the command sudo apt-get update

• This command will update all the software libraries on the Raspberry Pi, so that if we need to install software the Pi will install the latest software packages.
• Enter the command sudo apt-get full-upgrade -y
• This command will upgrade the entire operating system on the Raspberry Pi.
• The -y option tells the computer to answer Yes to any questions.
• The upgrade will take between 5 and 30 minutes to complete.
• It is always to do this once every month.

Running Node-RED on the Raspberry Pi

• Open the Terminal.
• To start Node-RED enter the command node-red

• The display on your Raspberry Pi will be different, but similar enough.
• When using Node-RED it is important to stop Node-RED when you are shutting down your computer.
• To close Node-RED open a new Terminal window and enter the command node-red-stop
• To re-start Node-RED enter either node-red or node-red-start

• It takes Node-RED approximately 20 seconds to boot up.
• To see the Node-RED interface we need to use a Web Browser.
• Open the default Web Browser on the Raspberry Pi.
• In the Search Bar enter localhost:1880 (or http://localhost:1880)
• The number 1880 represents a Port. Your computer listens and talks on different ports to make communication between devices and systems easier.
• Localhost is an abbreviation for 127.0.0.1 which is a address reserved for the host computer (the Raspberry Pi)

• Node-RED can be installed on Windows, Mac or other Linux machines.
• In a few moments the Node-RED programming environment will be displayed.
• One the left is the Node-RED Pallet containing an assortment of Nodes.
• In the middle is the Node-RED Flow Editor
• On the right are different debugging and system related tools.

Connecting to The Things Network using MQTT

• To download the data generated by our sensor we can use MQTT.
• MQTT is short for Message Queuing Telemetry Transport.
• Using this protocol we can subscribe to newly published data. As soon as new data arrived it is delivered to our device, just like a daily news feed or other subscription.
• To subscribe to our sensor data we need to use a mqtt in node
• Search for the mqtt in node in the pallet on the left and drag it into the Editor Window.

• If the mqtt in node is difficult to find you can enter mqtt in the search bar at the top of the pallet.
• In this example we can see that two results are returned, one for mqtt in and one for mqtt out.

• Note that the mqtt in node has two coloured shapes attached to the node:
  • the red triangle means that the node still needs to be configured
  • the blue circle means that the node need to be Deployed before it will work
• To configure the node, double click on it.
• This will open a Sidebar that will pop out from the right hand side.
• For the Server parameter, we will select Add new mqtt-broker and then click on the Edit tool, which looks like a pencil icon.

• The Edit tool will open up another Sidebar listing the Properties for the mqtt-broker node.
• Enter the following details under the Connection tab:
  • Name – soil-7
  • Server – au1.cloud.thethings.network
  • Port – 1883
  • Protocol – MQTT V3.1.1 (if you don’t see the Protocol parameter you don’t need to change any other settings)
• Now click on the Security tab.

• Enter the following details in the Security tab:
  • Username – whi-soil-7-moisture-mkr-1300@ttn (the username is simply a unique name given to the soil moisture sensor on The Things Network. The name is short for Whittlesea soil moisture sensor #7 running on an Arduino MKR 1300 microcontroller)
  • Password - NNSXS.WL5EA6KF6TWUZCALAIIEWNX7YED4ADI5QQOTK7A.XYRLCUHY6WGY5Z5O6TS7A7UFAACBNY72WIZPAITY6K3LTYLOVZXA (Enter this passwork exactly as shown. This key is generated on The Things Network Console. We will demonstrate this in an upcoming lesson)
• Click on Update to commit these changes.

• Once the Server has been configured there are some other minor configurations in the mqtt in node:
  • Topic - # (the hash symbol # indicates that we are subscribing to all topics)
  • QoS (Quality of Service) – 0 (only zero is supported by Node-RED. Normally higher numbers reflect better error detection)
  • Output – a parsed JSON object (all the data sent from the mqtt in node will be in JSON format. JSON format is similar to a dictionary. There is always an attribute and value pair. For example {colour:blue}. JSON objects are enclosed in curly brackets).
  • Name – soil-7 (this node will be subscribing to data from soil moisture sensor #7)
• Click on Done when finished.

• The mqtt-in node has the name soil-7 displayed.
• Note that the red triangle has disappeared because the node is now correctly configured.
• There is still a blue circle because we haven’t Deployed our code.
• Click on the Red Deploy button to start the program.

• If the mqtt in node is working correctly it should display a connected message supported with a green square.
• Note that if the program is running the Deploy button will be shaded out.
• As soon as you make any change to your program the Deploy button will be active again.
• Remember that while editing your program, the old program will still be running in the background on the Node-RED server.

Viewing MQTT data using the Node-RED Debug Node

• The mqtt in node has a small connecter on the right-hand side of the node.
• This allows us to connect the output data flow to other nodes using wires.
• From the Node-RED pallet find the debug node and drag into into the Editor Window.
• Then using a mouse connect the mqtt in node to the debug node, as shown below.

• To activate the program click on the Deploy button.
• To see the output of the debug node, click on the debug tab in the far-right window. It is the button that looks like an insect with legs.
• Once selected the word debug should appear in the window.

• You may receive an error message when the Deploy button is clicked.
• Just ignore this so long as the mqtt in node is still showing the connected status.

• The soil moisture sensor data sends data every 10 minutes.
• So after a few minutes you should see your first JSON object payload come through.

• By clicking on the drop down arrows you can expand or collapse different sections of the JSON object.
• In a section of the JSON object named decoded_payload there is data from the sensor:
  • bat – battery voltage data
  • humid – humidity data from within the sensor housing
  • soil – soil moisture sensor data (from under my fig tree)
  • temp – temperature data from within the sensor housing
• Note that the full path is msg.payload.uplink_message.decoded_payload.bat (.bat for battery data, .humid for humidity data, etc)
• Using this path we can start to extract specific data from the JSON object message.

Extracting data using the Function node

• Now that we are receiving all the JSON formatted data we can start to extract specific data sets.
• In this example we will extract the battery data.
• Find the function node in the pallet and drag it into the Editor window.
• Also add a debug node and connect them up as shown.

• Double click on the function node. We need to add some code so that the battery data can be extracted.
• Enter the following code:
  • var bat
  • bat=msg.payload.upload_message.decoded_payload.bat (puts the battery data in the variable bat)
  • msg.payload = bat (this then assigns the bat data to the msg.payload. Messages (msg) also contact other attributes such as msg.topic. Here we are only interested in the msg.payload attribute.
  • return msg (this sends the msg to the connecting node to the right. In this case it is the debug node which will display the output in the Debug window on the far right.

• Note – if you can’t see the underscore symbol (_) it may be just a bug.
• The underscore symbol is still there, just invisible.
• Another work around is to create the underscore symbol in a text editor and use the right-mouse button copy and paste function.
• Click on Done when finished.
• Deploy the code.

• To make the Debug window easier to view you can click on the Rubbish bin icon to clear the Debug window.
• Now you just need to wait for a new mqtt message to arrive.

• When the new message arrives, you will see two JSON outputs in the Debug Window.
  • One shows the output from the Debug node connected directly to the mqtt in node (soil-7)
  • The second will show the output from the Debug node connected to the output of the function node (battery). In this case we can see the numerical output 563, which is the battery voltage measurement.

• Congratulations, now you know how to extract JSON data using Node-RED.
• In our next lesson we will extract some other useful data and also create a Dashboard using Node-RED.

Graphing data from a soil moisture sensor using Node-RED

Overview

• In our previous lesson we learned how to connect to a sensor on The Things Network and extract relevant data.
• In this lesson we will learn how to graph the data using a Chart node in Node-RED and display the charts in a special Dashboard.
• Graphing data is very important because it allows you to see trends in the data which may not be obvious if you are only looking at individual data readings.

Learning Objectives

• Learn how to configure the Chart node in Node-RED.
• Learn how to configure the Node-RED Dashboard.
• Learn how to save and retrieve Node-RED flows.

Starting Node-RED

• To start Node-RED open the Terminal and enter node-red
• To see the instance of Node-RED open the default web browser and enter the URL localhost:1880

Installation of the Node-RED Dashboard pallet

• In Node-RED click on the hamburger icon (three horizonal lines) on the right hand side of the screen next to the Deploy button.
• Click on the option Manage palette.

• A new window will open named User Settings.
• The Pallet tab will show what Dashboard pallets are installed.
• In this example we can see that node-red-dashboard (version 3.0.4) is already installed (in use).

• If the Node-RED-dashboard was not installed, we would click on the Install Tab.
• In the search bar we would type in the pallet name dashboard and review the search responses.
• In this example there are several pallets returned, however in this case we would only installed the pallet named node-red-dashboard.
• A simple Google search would confirm what pallet to install.
• If this pallet was not installed the button on the right of the pallet would have the text install
• Installing a new pallet takes approximately one minute. The new pallet will automatically appear in Node-RED.

Configuring the Chart Dashboard Node

• From the Dashboard pallet add a chart node

• Wire up the chart node to the output of the battery function node.
• Note that at this stage the chart node has a red triangle indicating the this node needs to be configured before it will work.

• Double click on the chart node to bring up the Edit chart node window.
• The first action is to create a new Group for all the Dashboard components.
• Click on the Edit tool (pencil icon) on the right of the Group property.

• A new dashboard group config node window will open.
• In the Name property enter Soil Moisture Sensor.
• A Group can have multiple Tabs. We will only have one Tab.
• Click on Edit tool to the right of the Tab property.

• We will have the Tab the same name as the Group, Soil Moisture Sensor.
• Click on the Red Update button.

• This will bring us back to the Add new dashboard group config node window.
• Click on the red Add button.

• Back in the main Edit chart node window we need to configure some formatting for the chart.
• The chart we created is connected to the battery function node and will receive battery data.
• Change the Label name to Battery data.
• Change the x-axis (which is a measure of time) to show data for the last 1 week(s).
• Then click Done.

• Note that the chart node (named Battery data) still has a blue dot above it.
• This indicates that we need to Deploy our changes.
• Click on the Deploy button.

• As soon as we have deployed out Node-RED code the chart will start plotting data.
• The soil moisture sensor only sends a packet of data every 10 minutes, so we will have to be patient.

Viewing the Chart Node in the Dashboard

• The Dashboard we just created can be access using the Browser on the Raspberry Pi.
• Within the Browser click on the Plus (+) symbol to open up a new tab.

• Enter the URL localhost:1880/ui
  • localhost just means the local computer (your Raspberry Pi) of the IP address 127.0.0.1
  • 1880 is a port number. Different web applications use different ports. Port 80 is used for HTTP requests.
  • ui is short for User Interface

• If you are patient, you will gradually see data appear in your Dashboard.
• Try to keep your Raspberry Pi running all week.
• If you close Node-RED you will probably lose all this data.
• In future lessons we will look at ways to storing this data more permanently (so called persistent charts).

Adding a Chart for Soil Moisture

• Now that we understand how to create a chart for the Dashboard, we can easily replicate this process and add another chart to plot soil moisture from our sensor.
• Drag a function node and a debug node from the Node-RED pallet and wire them up as shown so that the new function node receives data from the soil moisture sensor on The Things Network.
• If you can’t find the relevant node you can search for it.

• To make programming easier we will copy (recycle) some of the code from the battery function node.
• Double click on the battery function node to bring up the Edit function node window.
• Highlight and copy all the code.

• Double click on the new function node to brig up the edit window.

• Paste the copied code inside.
• Edit the code to replace all references to bat (short for battery) to soil.
• Change the Name of the function node to Soil moisture.
• Once these changes have been made click on Done.

• The package of data that we receive from the Things Network is a little cryptic, but Node-RED makes it easy to decode.
• Here is the breakdown of the statement msg.payload.uplink_message.decoded_payload.soil
  • msg – the package of information sent from node to node in Node-RED.
  • payload – one of the message components, probably the most important, packed with lots of information.
  • payload_uplink – this smaller package is from the Things Network
  • decoded_payload – this smaller package again contains data from the sensor
  • soil – data specially related to the soil moisture sensor
• It may be easier to visualise the structure of this data by clicking on the drop down arrows in the Node-RED debug window.

• Once the soil moisture function node is Deployed you should start to see soil moisture data appear in the debug window.
• In the example below the battery data is 583 and the soil moisture data is 2149.
• This soil moistures sensor is a little different to most.
  • 0 to 2000 – wet soil
  • 2000 to 6000 – moist soil good for most food crops
  • 6000 and above – soil is getting too dry. Time to water

Creating a chart for soil moisture data

• Add a chart node and wire it up to the Soil moisture function node.
• Double click on the node to configure it as follows.
• Select Group – Soil Moisture Sensor
• Change Label to Soil moisture
• Change x-axis time period to 1 week(s)
• Then click on Done to commit changes.

Editing the Dashboard layout

• We have connected two charts to the Node-RED Dashboard.
• Node-RED gives us the ability to configure the Dashboard.
• On the right hand side of the Node-RED window click on the drop down arrow to allow for configuration of the Dashboard.

• In the example below you can see under the Layout tab,
  • Group - Soil Moisture Sensor, and below this
  • Tab - Soil Moisture Sensor, which contains our charts
    • Soil moisture
    • Battery data

• If we want the Battery data chart to be above the Soil moisture chart we simply change their position by dragging the Battery data icon up.
• See the example below how the position of these two charts has been reversed.

• Another small addition is to add a spacer.
• This will help keep the two charts separate when displayed in the Dashboard.
• To add a spacer hover over the Soil Moisture Sensor tab.
• This will reveal a button name spacer.
• Click on it to add the Spacer and position it by dragging it.

• Once you have made these changes don’t forget to click on Deploy.
• To see your changes go back to the Node-RED Dashboard in your Browser.

Saving your code in Node-RED

• It is always important to save your programs.
• Once saved you need to store a backup of your program either in email, on a USB memory stick, or on another computer.
• To save our program click on the hamburger button (three horizontal lines) and select Export.

• Select the following options:
  • current flow – only the program we are currently working on. You may wish to save all your Node-RED flows.
  • JSON – to save the information in JSON format (remember attribute:value pair)
  • formatted – the option at the bottom right of the window makes the JSON data easier to read.
• Now click Download. Alternatively you can also copy your flow to the Clipboard.

• If this display or options appear different on your Raspberry Pi you may need to do a full-upgrade to your operating system to install the latest versions of all software.
• Clicking on the Download button will save a copy of your Node-RED flow to the Downloads directory on your Raspberry Pi.
• The Downloads directory path is /home/pi/Downloads.
• You can use the File Manager to rename or copy the file to a USB memory stick.

Creating a Dashboard for a soil moisture sensor using node-RED

Overview

• In this lesson we will learn how to compete our Dashboard for displaying data from our soil moisture sensor.
• We will also learn how to save and retrieve our Node-RED flows.
• When you complete this lesson, you can leave your Raspberry Pi running and see how the sensor data changes with time.
• Some examples are presented below.

Learning Objectives

• Learn how to build the soil moisture sensor dashboard in Node-RED.
• Learn how to save and retrieve Node-RED flows.

Adding sensor temperature and humidity to Dashboard

• In the previous lesson sensor data was processed for battery and soil moisture.
• In this lesson we will add the sensor temperature and humidity data values.
• Copy the Soil moisture node by clicking on the function node and then using the keyboard shortcut Ctrl+C.

• If the Copy is successful a message will pop up saying ‘1 node copied’

• Move the mouse to a different are of the Edit window and press Ctrl-V to paste the copied node.
• Carefully move the new mode to a suitable location.

• To edit the copied node double click on it using the mouse.
• This will open the Edit function node window.
• Change the Name to Sensor temperature
• Change the JavaScript code to the example below then click on Done.

• Drag a chart node from the pallet and add it to the main edit window.
• Wire up the function node and the chart node using the mouse to join node outputs to node inputs.

• Double click on the chart node to open the Edit chart node window.
  • Change group to Soil sensor.
  • Change label to Sensor temperature
  • Change x-axis to 1 week(s).
  • Then click Done.

• In the main edit window click on Deploy to commit these changes to your program.

• Repeat this process for the Sensor humidity data.

• When editing the function node for Sensor humidity, use the following code.

Editing the Dashboard layout

• Double click on each Chart node.
• Then click on the Edit button to the right of each Group.
• Check that the Width for the Chart is set to 6.
• This will ensure that charts in the Dashboard are large (6 units), not small.

• To edit the order of Charts in the Dashboard click on the drop-down arrow beneath the three horizontal lines.
• Select Dashboard.

• Within a group, you can drag individual charts to re-arrange their order.
• Note that the charts shown below are associated with Group – Soil sensor.
• Click Deploy to commit changes.

Saving and retrieving Node-RED flows

• To save a flow click on 3 horizontal lines the (hamburger symbol) and select Export

• In the Export nodes window select:
  • current flow – export only the one flow window. You can also select all flows.
  • JSON – arranges the flow information in JSON format
  • Formatted – formats the JSON code with indentations for readability
• Then click on Download. This will download the file to the Downloads directory on the Pi.

• Open the File Manager on the Raspberry Pi and navigate to the Downloads directory.
• Find the most recent flows.json file.
• Right mouse button click on the file and rename it.

• Name the file based on the project (e.g. Soil-moisture-sensor) and also add a date.

Information not saved in the Node-RED file

• As a security measure, the username and password in the mqtt in node is not saved
• Save this information either in a book or in a text file.
• In this example the security information can be shared, so we can keep it in a text file.
• We will need this information if we need to reload (retrieve) our flow in Node-RED.

Retrieving saved flows in Node-RED

To import a saved flow, select Import from the hamburger menu.

• From the Import nodes window:
  • click on select a file to import – and select your file
  • import to – new flow – so that the imported file enters with a new tab
  • then click on Import