As data scientists we are constantly trying to improve process and usability when it comes to data. We saw the current manual reports as an opportunity to automate some of the data process. In addition, its clear that it would be helpful to have a webpage where stakeholders can access solid waste information in a succinct way. Finally, when we see data that has not yet been fully analyzed, we get excited because we know its an opportunity. We think that further analysis could help the city accurately predict future solid waste volumes. Furthermore, we believe that work with forecasting waste and clustering curbside violators can help the city develop new outreach and education programs.
What it does
DASHBOARD The application replaces the components of the current monthly dashboard.
ANALYTICS The application incorporates least absolute deviation auto-regressive forecasting as well as hierarchical and k-means clustering.
DATA WAREHOUSING The application removes the need to update static excel spreadsheets. Instead the user is able to enter new rows which can then be saved back to a database.
How we built it
We used a combination of a virtual machine, R, and RShiny to build the application.
Challenges we ran into
None of us had ever hosted and installed a shiny server instance. Initially it was steep learning curve, as was allowing the database to reach our application. Furthermore, three of us worked on separate parts of the Application. Getting it all integrated came down to the last few minutes.
Accomplishments that I'm proud of
We were able to tackle aspects of both problems and meet some of our stretch goals. We all learned some new techniques that will help us on future projects. We met and understood the needs of the stakeholders before we began to tackle the problem in attempt to tailor the solution for their needs.
What I learned
It is not easy to put together a full service analytics tool in less than 48 hours. We learned to use new packages within r, including adding d3 based interactive plots. We were able to fully deploy a shiny server instance. Additionally, we were able to develop a SQL database on the same machine which allows us to efficiently store and update data. Lastly, we learned to geocode on using automated api calls and to put that data into an interactive map.
What's next for Solid Waste Analytics Tool
First off, we would like to take more time to build in historical data for solid waste management. This will allow the application to pull more history and will vastly improve the accuracy of the forecasting.