Sonia: I am very passionate about caving and data so when she got an email about a Hawaiian cave database a week before Technica, I found her project and assembled a team! The project continuation needed:
- conversion of cave registries to a uniform CaveXML format
- a public web search interface, for those who can't program
What it does
This software will scrape cave registry websites and create searchable database of caves.
How I built it
Emma: The cave data is first scraped from a Scottish cave registry using Python and Beautiful Soup. The name of the cave, tags associated with the cave type, length of the cave, vertical range, altitude, and other attributes of the caves are scraped and stored in a Pandas dataframe. The data is saved as a .csv file and then converted to XML so that the tool can be used more broadly with different datasets. The XML constructor used was created using CaveXML as defined by Norbert Schörghofer.
Sonia: Using Python, I converted the CaveXML data file to an SQLite database class with a query function. The Flask app creates an instance of this class and runs the query it receives a get request from the Angular app front end and returns a json string.
Lacey & Kajol: The frontend, homepage and main search page, was created using AngularJS and Typescript.
Challenges I ran into
Our team ran into challenges with the integration of the back end and the front end for the web search interface. Web development is not an area that our team had prior experience with, which made this part of our project especially challenging.
Accomplishments that I'm proud of
Emma: This was my first hackathon and I'm a beginner programmer, so I'm proud of participating in my first hackathon and successfully completing two small components of the project (web scraping and XML conversion).
Lacey: Even though I wasn't able to contribute much to the final code, I celebrated tiny victories, such as being able to get some kind of response from a button. This was also my first hackathon, and I'm glad to have had the chance to learn a bit about a language and framework that were new to me.
Sonia: This is my second hackathon and I am also a beginner programmer. What I initially thought was maybe too easy of a project proved me wrong, but nonetheless I am proud of how far our team got and how much we learned.
What I learned
Emma: I learned how to use the beautiful soup module in Python to scrape data from the web. I also gained familiarity with using GitHub and learned how to parse csv data into an XML format.
Lacey: This was my first time using AngularJS and Typescript, so I devoted a lot of time to learning it. It was frustrating at times, but I believe it was worth it, since I have never thought to choose these tools for a project.
Sonia: I furthered my ability to wrangle data in Pandas dataframes, used Flask for the second time, and worked with front end integration for the first time ever (AngularJS, Typescript).
What's next for CaveXML Search
We would like to fully integrate the the backend with the web search interface by collecting user input for query parameters, and then displaying a formatted view of the returned data. We would also like to expand the number of web sources included in the database.