Crowdsourced search engine | COVID-19

Inspiration

Coronavirus tech handbook is a crowdsource information about the SARS-CoV-2 coronavirus. It was developed at Newspeak House, a hackspace for politics in London, England. It aims to provide a space for technologists, civic organizations, public & private institutions, researchers, and specialists of all kinds to collaborate on a rapid and sophisticated response to the coronavirus outbreak and subsequent impacts. As one of the volunteers of this handbook, we created a chatbot that takes content from the handbook allowing users to conversationally ask questions about the many topics that are of points of interest during the COVID-19 pandemic. This tool is currently a stand-alone web-app and can also be deployed on Facebook - increasing awareness amongst people and allowing the handbook community to get critical feedback on what sort of data users are searching for.

What it does

The web-application connects the handbook community to it's end-users creating demand and supply of crowdsourced information - where the consumers give valuable insights to the librarians of the handbook to added material that is verified and vetted by the community. The users pick the category they want to query and proceed to ask their questions. If their query is not satisfied, they can report their question to the handbook community helping them understand what sort of information users are searching for. Every query is linked to a Whatsapp group that helps users connect to people from the community or other people of the same interest - this helps bring people together and provide a platform where they can discuss their queries with other people.

How we built it

We built the application on Django. We built a python scraper to convert the raw data to a structured JSON. Using sentence transformers, we then created embeddings for every heading created by the tech-book moderators. The user types in their query after selecting a category (as of now, we have Mental-health, unemployment, retail, data-tools, and education) it gets embedded and the chatbot tries to find an appropriate response which is shown on the Django chatbot.

Challenges we ran into

Building the scraper to connect the tech-book is challenging
We could not finish the messenger (Facebook) API on time so we made a simple Django interface (chatbot)
Improving similarity matching. Some queries do not match articles accurately - we are currently collecting more data from the coronavirus tech handbook community

Accomplishments that we're proud of

Connecting the user with our community is our priority. When a user asks a question, they are given the option of joining a Whatsapp group which has people that are interested in the same topics the user queried about. We believe this will bring people together to share their experiences or queries. Structuring the document was/is challenging. Once we converted 5 categories, we used the Bert tokenizer for our chatbot which turned out to be less accurate. After exploring may solutions to match queries with responses, we finally improved on our accuracy although there is a lot of room for improvement. We're also proud of our clean interface!

What we learned

Structuring documents is generally a hard task and we are working with the tech-book community to help covert data to a JSON.

What's next for Community-powered search engine | COVID-19

We're working with the coronavirustechhandbook community to improve our chatbot.
We're also working on expanding the platforms the chatbots are accessible on (Facebook, Twitter, etc)
Sphinx search to better match results.
Creating a feedback loop to connect the community with the data from the user's search.
Improving conversations on Whatsapp groups by using a Whatsapp business API