A healthy and sustainable agriculture sector is key to ensuring we can feed the current and future population. As arable land becomes scare and resources limited, farmers are faced with the challenge to continually optimise their operations and improve productivity.
Whilst there has been a lot of noise about the huge potential of the Internet of Things (IoT) and precision agriculture, SenseAg founders Mike and Anthony identified that there was little progress integrating this technology into intuitive end-to-end solutions for farmers.
Mike and Anthony, both self proclaimed ‘tech geeks’ with a keen interest in using technology for practical business applications, saw an opportunity to integrate the emerging technologies of remote sensing, low-power and long-range communications, Apache Spark Streaming, machine learning and responsive web/mobile applications to optimise farming operations. In doing so, they set out to build a solution enabling farmers to make real-time data-driven decisions while focusing on farming, not collecting and interpreting various sources of data, ultimately assisting them to achieve more with less.
This led to the creation of SenseAg!
What it does
The SenseAg platform provides farmers with real-time, decision imperative and actionable information based on their farm’s key metrics, as measured by SenseAg’s distributed network of sensors and analysed using Apache Spark on the cloud. Information is delivered to the farmer through a variety of channels, including a web application, smartphone apps and alerts/notifications (smartphone, SMS or email), as defined by user needs, thresholds and risk factors.
Through the continuous collection of environmental data by sensors deployed across the farm, the SenseAg platform constantly analyses the information received to identify threshold and rate-of-change events which exceed statistical variances, in addition to providing a real-time geographical representation of key metrics. Additionally, SenseAg can pull information from outside sources (i.e. weather reports) to compare against and inform data collected locally. This combined functionality allows farmers to manage their operations more efficiently and enhance productivity, through the reduction/optimisation of inputs (reducing cost of water, fertilizer, labour etc.) and thus potentially increasing yields (increased revenue and/or competitiveness).
Being a cloud-based solution with both web and mobile applications, SenseAg enables effective decentralised operations (multiple sites) and 24/7 remote monitoring (offsite management or travelling staff), no matter where they are in the world.
How we built it
SenseAg is truly a full-stack solution involving many languages/skills, including:
- Custom printed circuit boards (PCBs) for hardware integration (communications, sensors, power management etc.)
- Custom sensor firmware built in C
- Sensor gateway logic built in Python and JS
- Business Logic centralised into Apache Spark for scalability and written in Scala
- Operational data storage with the MongoDB NoSQL database engine
- Smartphone Applications compiled with MeteorJS using the Apache Cordova architecture
- Visualisations and Geospatial built using custom d3js
- Full end-to-end AES encryption
While the concept came about in April this year, we started building the SenseAg solution in late May. Since the initial build, we have gone through numerous iterations to optimise and enhance each of the components listed above. In that time we have had to learn/extend many skills, noting that neither of us:
- Had much experience in electronics design
- Had significant C skills
- Had any experience with Apache Spark Streaming
- Had built a mobile application
- Had much experience with encryption technologies
- Had time to work on this project full-time
By focusing on small iterations, undertaking significant research and using collaborative tools such as Trello and Slack to help manage our workflow and priorities, we are confident that we have built a platform that delivers real value to a real business problem.
Challenges we ran into
The key challenge faced in building SenseAg was the sheer number of technologies involved in building an end-to-end solution, particularly given that much of it is emerging and therefore with little documentation. To mitigate some risk and ensure we had a thorough understanding of the opportunities and challenges of each technology, both Mike and Anthony worked to become proficient across the entire stack which increased the workload significantly, however we felt that the upfront investment and knowledge transfer would reap rewards as the solution was further developed. From an Apache Spark point of view there have been a few challenges:
How to architect a Spark Streaming job to manage streams of data which arrive at intervals different to the overall Spark Streaming batch interval.
We were hoping that the Spark 'Continuous Applications' arriving in Spark 2.0 would help alleviate this problem (particularly the delta detection), however upon testing in a preview Spark 2.0 release, we deemed it not ready for production. In the end we had to resort to stateful streaming to identify which records have changed and only output if a change is detected for a given sensor station.
How to deal with the time dimension in Spark Streaming.
After testing different solutions, the decision to use an epoch based time standard end-to-end has proven the easiest way to manage event time as it can be easily parsed (date objects proved to be too onerous).
Accomplishments that we're proud of
While still in its early stage of development, the SenseAg platform is proof that Internet of Things (IoT) doesn't have to be a nebulous concept, and in fact can be used to solve real-world problems – we were just two guys with some passion, an idea and some spare time.
Mike is a Spark contributor, and is very proud of the code developed for SenseAg and submitted for this competition. Whilst there are alternative tools that could perform the calculations and workflow that Apache Spark provides, few are as intuitive, have a great API and fulfil so many roles in a data science pipeline.
We are also proud that our first generation custom sensors not only worked flawlessly, but out-performed commercially available alternatives trialled, having successfully captured and transmitted data for the past 6 weeks in real-world conditions.
What we learned
Deploying our hardware solution in a harsh outdoor environment was not a simple endeavour, with some lessons learnt including optimal sensor placement, IP ratings, communication range optimisation, energy harvesting, protection from weather and farm machinery, and accessibility/maintainability.
In terms of our development, we found it extremely beneficial to have designed the entire end-to-end workflow as this allowed us to quickly troubleshoot likely points of failure and target our debugging.
Finally and most importantly, we learned that this solution really works in a farm environment, and is able to provide farmers actionable information with little effort on their part.
What's next for SenseAg
It’s early days but we’re confident that SenseAg has an exciting future ahead. We are currently trailing our solution on a large real Australian farm and continue to evolve platform based on user feedback and our own observations. This includes the addition of other features that will assist in day-to-day operations and decision making.
The platform is currently being upgraded to a microservices architecture which will enhance manageability/scalability moving forward.
As more data is collected, smarter self-learning alerts will be implemented using the Apache Spark ML (Machine Learning) for which we already have a few ideas, particularly focusing on increasing the efficiency of water usage via machine-learning based watering systems, in an effort to move to complete automation.