Wyatt's best friend at Northwestern is a farmer and Matt comes from a big farming area. Seeing the political disconnect between the resources "Big Ag " and "Small Ag" have has inspired us to create tools to level the playing field.
What it does
AgConnect connects small town farmers with big time data analysis, allowing them to very easily discern how other fellow farmers from their county have fared over the last 10 years based off of weather data, including temperature and precipitation. Through our beautiful and interactive graphs the farmers can see yields and profits from the past and make more accurate predictions on the future based on future weather patterns.
How we built it
We used a lot of python scripting to deal with the unnecessarily and obnoxiously senseless NASS API, moving and shaping huge amounts of data around (Up to 500,000 rows worth at a time). On the frontend, besides the classic bootstrap theme, we utilized d3.js in order to make our graphics pretty and interactive. And to merge this front end with the python backend we used Django. In order to get the field data from the farmers we use the Climate API. In addition, we use NOAA weather data files in order to check those trends vs the Google Maps reverse-geocoded GPS Coordinates we queried. In order to use some parts of the Climate API most effectively and so that both of us could have access to the same local server, we used ngrok in order to tunnel our local server onto the web.
Challenges we ran into
NASS API is absolutely atrocious. The government somehow decided it would be a good idea to make the "state_id"s run from 1 to 56 with 6 random states being skipped along the way. In addition, the county_id system made little sense, and when combined with the query parameters often having no impact on the query result, it made for a very frustrating time to make even simple HttpRequests. In addition, we had a lot of trouble finding a good Weather API that was both good and free.
Accomplishments that we're proud of
We are very proud of how much we learned this weekend and how beautiful we realized data could truly be. Matt is also very excited that he now has Ubuntu on his computer and really harnessed that new found power this weekend!
What we learned
We learned to never use another government API again after the disaster that is NASS. I also really learned how much data molding is necessary for dealing with large amounts of data like we did, especially when they are as inconsistently labled as NASS's rows are.
What's next for AgConnect
We are going to try and improve our data scraping methods to increase efficienecy and widen the scope of AgConnect in the future so we can touch the lives of even more agricultural professionals. We also plan on making the site look less bootstrappy and maybe get some sleep.