"Augmented Reality Gardener" takes care of the crops for you!

It is an automated serverless plant care management system, that monitors plant-related sensor data, uses weather forecasts, Data Mining and Machine Learning to predict & perform irrigation tasks using PID (Proportional Integral Derivative) watering control.

Features wireless, Low Power Wide Area Networks (LPWAN) kilometer-range IoT communication with LoRa (Long Range) message-passing. Presents real-time data using Augmented Reality.


What it does


Locally, using HomeAssistant:

  • Soil: Humidity, Temperature, Fertility and Sun levels.

On the cloud:

  • Air: Atmospheric pressure, Relative humidity.
  • Weather: Local (lon/lat) weather future reports for rain/temp/humidity.

Data Mining:

  • Aggregates & interpolates (forward/backward fills missing) weather data, using Dark Sky API (


  • Future metrics using multi-variate time-series prediction with Machine Learning (non-linear Artificial Neural Network regression).


  • Gradually waters the plants with soil moisture% sensor feedback (using a software PID controller to open a water valve).


Augmented Reality IoT Gardener ( ):

  1. Gardener Azure ML C2D Function ( ): Invokes the Time-Series Azure ML Studio service and sends a Cloud2Device (C2D) message with the predicted statistics of the soil moisture % of the next 48 hours.

  2. Gardener Computer Vision ( ): Detects a model image in a video feed. Draws real-time computer vision information & the model's translated geometry rectangle of the 3D plane. Also attaches the latest sensor/plant data from CosmosDB.

  3. Gardener IoT Hub Device ( ): Azure cloud I/O communication (D2C and C2D) with .Net Core 2.0 and CosmosDB storage. Proportional Integral Derivative (PID) controller: Performs gradual watering using Soil Moisture % readings ( ).

  4. Gardener Azure ML Studio Data Mining ( ): A python script for data mining of weather data (from Dark Sky API), used in Azure Machine Learning studio. Forward/backward fill interpolation is applied to missing data.

  5. Gardener PID Controller ( ): Gardener Proportional Integral Derivative (PID) controller: Performs gradual watering using Soil Moisture % readings. Proof of concept project for simulation & calibration runs.

  6. Gardener IoT Edge ESP32 ( ): RestFul API Control end-point for water valve and water flow sensing on a TTGO ESP32 LoRa SX1276 868/915MHz Bluetooth WiFi.

  7. Gardener IoT Edge LoRa MKR1300 ( ): LoRa end-point for water valve control on an Arduino MKR 1300.


  • Augmented Reality
  • Internet Of Things (IoT)
  • Data Mining:
    • Time-series data interpolation and forward/backward fill.
    • Weather Forecasts (48 hour window)
  • Machine Learning (ML):
    • Artificial Neural Networks (ANN)
    • Hyper-parameter Tuning
    • Cross-Validation ML Training
    • Multivariate Time-series Regression
    • Real-time Soil Humidity predicton (48 hour future window)
  • Control Theory:
    • Proportional Integral Derivative (PID)
  • Computer Vision:
    • Oriented FAST and rotated BRIEF (ORB) feature & descriptor detection
    • Hamming distance descriptor matching
    • Homography using Random sample consensus (RANSAC)
    • 3D-to-2D plane perspective transform & warping
    • Canny Edge detection
    • Classification & match thresholding
  • Communication:
    • LoRa (Long Range) for wireless, Low Power Wide Area Networks (LPWAN) IoT kilometer-range message passing interface
    • Representational State Transfer Application Programming Interface (RESTful API)
    • Advanced Message Queueing Protocol (AMQP)
    • Device-To-Cloud (D2C)
    • Cloud-To-Device (C2D)
    • Machine-To-Machine (M2M)
    • Bluetooth Low Energy (BLE)
    • WiFi
  • Azure:
    • IoT Hub
    • Functions
    • Machine Learning Studio Service
    • Cosmos DB

Languages, SDKs & Libraries

  • C#:
    • Net.Core 2.0
    • Azure.Documents
    • Linq
    • Json
    • Net.Http
    • Timers
    • Asynchronous Tasks
    • Collections.Generic
  • Python:
    • numpy
    • cv2 (Open Computer Vision)
    • pandas (for Data Mining)
    • requests
    • datetime
  • Arduino Ino
    • LoRa.h
    • WiFi.h
    • WiFiClient.h
    • WebServer.h
    • ESPmDNS.h
    • aREST.h
    • SPI.h


Built With

Share this project: