• 

  Augmented Reality Gardener logo

• 

  View of the tomato garden.

• 

  Azure IoT Serverless Architecture: Augmented Reality Gardener

• 

  Everything started from a seed!

• 

  Automated Watering. PID controller gradually achieves target soil moisture.

• 

  Baby tomatoes

• 

  Tomatoes seed kit.

• 

  Soil Moisture Machine Learning prediction. Sample results from an Artificial Neural Network regression model.

• 

  Artificial Neural Networks had min Time-Series error. Cross-Validation and Hyper-parameter tuning model training vs other regression models

• 

  MiFlora BLE Sensor. Tracks plant data (Soil: Humidity, Conductivity, Solar exposure, Temperature)

• 

  Water Flow Sensor (Hall Effect). Measures the amount of water provided in mL.

• 

  Soil moisture % ML Service. The best trained Neural Network model, uses weather forecast & data mining to predict the next 48 hours.

• 

  Solenoid Valve. Provides water from the tank to the plant.

• 

  Relay Switch. Controls the 12 volt water valve.

• 

  ESP32 LoRa Wifi. Communicates between Windows IoT core (WiFi) and the remote valve (LoRa) at a kilometer-range (LPWAN)

• 

  Raspberry Pi 2B. Runs the Windows IoT Core operating System.

• 

  Raspberry Pi 3B. Runs the Hass.io operating system.

• 

  Arduino MKR1300 Lora. Communicates with ESP32 (LoRa) and controls the solenoid water valve at a kilometer-range (LPWAN).

• 

  Radana tomatoes. Sweet flavored and pear-shaped.

• 

  Water tank. Holds 5 liters of water.

• 

  Green Envy tomatoes. Grape sized cherry tomatoes.

• 

  Zlatava tomatoes. Round/heart shaped.

• 

  Common tomatoes. Seed extracted from supermarket's tomatoes.

• 

  Taiko tomatoes. When ripe, has bright yellow banana color.

• 

  Lemon cucumber.

Inspiration

"Augmented Reality Gardener" takes care of the crops for you!

It is an automated serverless plant care management system, that monitors plant-related sensor data, uses weather forecasts, Data Mining and Machine Learning to predict & perform irrigation tasks using PID (Proportional Integral Derivative) watering control.

Features wireless, Low Power Wide Area Networks (LPWAN) kilometer-range IoT communication with LoRa (Long Range) message-passing. Presents real-time data using Augmented Reality.

What it does

Monitors

Locally, using HomeAssistant:

• Soil: Humidity, Temperature, Fertility and Sun levels.

On the cloud:

• Air: Atmospheric pressure, Relative humidity.
• Weather: Local (lon/lat) weather future reports for rain/temp/humidity.

Data Mining:

• Aggregates & interpolates (forward/backward fills missing) weather data, using Dark Sky API (https://darksky.net/dev).

Predicts:

• Future metrics using multi-variate time-series prediction with Machine Learning (non-linear Artificial Neural Network regression).

Acts:

• Gradually waters the plants with soil moisture% sensor feedback (using a software PID controller to open a water valve).

Repositories

Augmented Reality IoT Gardener (https://bitbucket.org/account/user/temp3rr0rprojects/projects/AUG ):

1. Gardener Azure ML C2D Function (https://bitbucket.org/temp3rr0rprojects/gardener-azure-ml-c2d-function ): Invokes the Time-Series Azure ML Studio service and sends a Cloud2Device (C2D) message with the predicted statistics of the soil moisture % of the next 48 hours.

2. Gardener Computer Vision (https://bitbucket.org/temp3rr0rprojects/gardener-computer-vision ): Detects a model image in a video feed. Draws real-time computer vision information & the model's translated geometry rectangle of the 3D plane. Also attaches the latest sensor/plant data from CosmosDB.

3. Gardener IoT Hub Device (https://bitbucket.org/temp3rr0rprojects/gardener-iot-hub-device ): Azure cloud I/O communication (D2C and C2D) with .Net Core 2.0 and CosmosDB storage. Proportional Integral Derivative (PID) controller: Performs gradual watering using Soil Moisture % readings (https://en.wikipedia.org/wiki/PID_controller ).

4. Gardener Azure ML Studio Data Mining (https://bitbucket.org/temp3rr0rprojects/gardener-azure-ml-studio-data-mining ): A python script for data mining of weather data (from Dark Sky API), used in Azure Machine Learning studio. Forward/backward fill interpolation is applied to missing data.

5. Gardener PID Controller (https://bitbucket.org/temp3rr0rprojects/gardener-pid-controller ): Gardener Proportional Integral Derivative (PID) controller: Performs gradual watering using Soil Moisture % readings. Proof of concept project for simulation & calibration runs.

6. Gardener IoT Edge ESP32 (https://bitbucket.org/temp3rr0rprojects/gardener-iot-edge-lora-esp32 ): RestFul API Control end-point for water valve and water flow sensing on a TTGO ESP32 LoRa SX1276 868/915MHz Bluetooth WiFi.

7. Gardener IoT Edge LoRa MKR1300 (https://bitbucket.org/temp3rr0rprojects/gardener-iot-edge-lora-mkr1300 ): LoRa end-point for water valve control on an Arduino MKR 1300.

Technologies

• Augmented Reality
• Internet Of Things (IoT)
  • Windows IoT Core Operating System (OS) (https://developer.microsoft.com/en-us/windows/iot)
  • Home Assistant Hass.io OS (https://www.home-assistant.io/hassio/)
• Data Mining:
  • Time-series data interpolation and forward/backward fill.
  • Weather Forecasts (48 hour window)
• Machine Learning (ML):
  • Artificial Neural Networks (ANN)
  • Hyper-parameter Tuning
  • Cross-Validation ML Training
  • Multivariate Time-series Regression
  • Real-time Soil Humidity predicton (48 hour future window)
• Control Theory:
  • Proportional Integral Derivative (PID)
• Computer Vision:
  • Oriented FAST and rotated BRIEF (ORB) feature & descriptor detection
  • Hamming distance descriptor matching
  • Homography using Random sample consensus (RANSAC)
  • 3D-to-2D plane perspective transform & warping
  • Canny Edge detection
  • Classification & match thresholding
• Communication:
  • LoRa (Long Range) for wireless, Low Power Wide Area Networks (LPWAN) IoT kilometer-range message passing interface
  • Representational State Transfer Application Programming Interface (RESTful API)
  • Advanced Message Queueing Protocol (AMQP)
  • Device-To-Cloud (D2C)
  • Cloud-To-Device (C2D)
  • Machine-To-Machine (M2M)
  • Bluetooth Low Energy (BLE)
  • WiFi
• Azure:
  • IoT Hub
  • Functions
  • Machine Learning Studio Service
  • Cosmos DB

Languages, SDKs & Libraries

• C#:
  • Net.Core 2.0
  • Azure.Documents
  • Linq
  • Json
  • Net.Http
  • Timers
  • Asynchronous Tasks
  • Collections.Generic
• Python:
  • numpy
  • cv2 (Open Computer Vision)
  • pandas (for Data Mining)
  • requests
  • datetime
• Arduino Ino
  • LoRa.h
  • WiFi.h
  • WiFiClient.h
  • WebServer.h
  • ESPmDNS.h
  • aREST.h
  • SPI.h

Hardware

• Tomatoes! Plant Theatre Wacky Tomato Kit (https://www.amazon.co.uk/gp/product/B01897NVEC/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1)
• MiFlora Plant Sensor (https://xiaomi-mi.com/sockets-and-sensors/xiaomi-huahuacaocao-flower-care-smart-monitor/)
• Solenoid Water Valve (https://www.amazon.de/gp/product/B07171LNYR/ref=oh_aui_detailpage_o08_s00?ie=UTF8&psc=1)
• Hall Effect Water Flow Sensor (https://www.adafruit.com/product/828)
• TTGO ESP32 LoRa SX1276 868/915MHz Bluetooth WiFi (https://www.amazon.com/ILS-Bluetooth-Internet-Antenna-Development/dp/B07BK225LT)
• Arduino MKR WAN 1300 (LoRa connectivity) (https://store.arduino.cc/mkr-wan-1300)
• Raspberry Pi 3B (https://www.amazon.co.uk/gp/product/B01CD5VC92/ref=oh_aui_detailpage_o06_s00?ie=UTF8&psc=1)
• Raspberry Pi 2B (https://www.amazon.co.uk/gp/product/B00T2U7R7I/ref=oh_aui_detailpage_o04_s00?ie=UTF8&psc=1)
• 5 Liter water container
• Relay Switch (https://www.adafruit.com/product/3191)

Built With

• arduino
• augmented-reality
• azure
• computer-vision
• control-theory
• cosmosdb
• data-mining
• esp32
• iot
• lora
• machine-learning
• neural-networks
• pid
• raspberry-pi