We set out with the goal to create a twitter bot that could learn to tweet like a person who's tweets are objectively the most "trendy".
What it does
Through clever web-scraping, TrendingSimulator will fetch the "Top Trending" subjects or a subject, profile, or hashtag of your choice. It will take these trends, analyze and parse the related tweets, then create tweets based on that information.
How we built it
- Using Selenium we are able to scrape different twitter pages related to the tags we specified.
- Using markovify we can generate natural or close to natural language based on the scraped tweets.
- Caching of scraped data and Google API results into a SQLite to improve efficiency of queries.
- Google Cloud Vision was used to label and tag images. These results are then loaded into our database. These image tags are then later used to find the most relatable image for the generated tweet.
Challenges we ran into
- Twitter has a long response time when applying for a developer account
- Google Cloud API tagging
- There was nobody on our team with experience using the Google Cloud Vision API.
- Posting images to twitter due to lack of OS control by Selenium WebDriver
- Speed of scraping and the problem of small data sets
- We were able to setup caching to help reduce the time needed for repeated queries. Doing this helped us persist our scraped data. This also helps with reducing Google API calls.
What we're proud of
As a team we worked really well together even though it was our first hackathon. Every person's different skill sets came together to make a project. We're proud of the scope of our project as it fit perfectly into the 36 hour time frame that we were given. In terms of individual feats we're proud that we were able to complete all functionalities planned from the beginning. Integration between individual parts of the application came together smoothly.
What we learned
Walking in as a team we didn't have much experience with hackathons. We also didn't have a single language for all of us to work in. We decided python would be an appropriate language for all of us to collaborate with. As a group we worked together as a team to split up and distribute tasks to work more efficiently with our tighter time-frame.
Once we get access to the Twitter API we can more quickly gather a large data set, thus giving us better results from our bot. Further documentation of the code and optimization is something that we would like to work on even once the hackathon is over. We also want to see our application have further support with other languages.