MR 2: HEARMES

A distribution system for stories shared by migrants and refugees to credible media outlets

Comment

Inspiration

Rabbi Shaul, one of our inspiring keynote speakers for this weekend, shared the pivotal insight that what feeds the separating notion of an “Us” and a “Them” is the lack of depth, lack of a human component when we talk about unfamiliar people.

We propose a solution that humanizes migrants and refugees through a paradigm shift of how we interact with their stories, away from a context of strong negative associations. Instead, we allow their unique background to be shared with readers by injecting snippets of their writing and personality into relevant content you are already consuming, making it as easy as possible for you to make a human connection based on shared interests.

What it does

Submitting a story is as easy as handwriting it and scanning to upload, where a computer vision API converts the image to text format. Using natural language processing, our system extracts keywords relevant to the story and also determines, through a proprietary method, the most compelling phrase in the story that can be placed into related content throughout the web. Here you can see the text format of this refugee’s letter.

Supporting websites can search through our code for keywords relevant to their content and obtain an HTML snippet that is easily embedable. Here is an example - as a jazz enthusiast, I’m reading a recent review in a prominent newspaper, when I stumble upon this attention-grabbing quote. I’m driven to click to read more, and just like that I’ve had a moment of connection with a migrant in my area!

How we built it

Using Django API as a MVC Model, we connected our Python-Backend with the HTML Frontend. The model sends requests to our Microsoft Azure server, holding all APIs credentials. With them (Computer Vision API, Text Recognition API, Blob Storage, Translate API), we analyze the handwritten text into computer text. With this text we use AI to read the sentiment, the main keywords, and the two most important sentences. Those sentences are calculated using our own algorithm:

  1. We take the first and last part of the paragraph
  2. We scan for keywords in it, each keyword gets a score.
  3. The sentence that has more keywords and higher score, wins and get elected.

Those sentences are then injected into an HTML snippets that can be later be used and embedded into websites.

Also, websites can search through this snippets based on keywords, to find quotes relevant to their content.

Challenges we ran into

Concept-wise: Initially, Hearmes was going to resemble a blog, but we realized that people that weren't explicitly inclined to reading about that kind of content in general were most likely not going to read it. We thought long and hard about how to draw in the more neutral readers and we decided on placing "snack bites" of the stories being uploaded in the forms of advertisements and embedding them into relevant news articles topic-wise

Tech-wise: We couldn't find a reliable API to read handwritten Arabic into the computer. Also, there was no API to extract the most important phrases from a block of text, so we had to sit couple of hours and figure out ourselves how to achieve that. We did it :) (It actually got posted, although were surfing on 52kbs)

Accomplishments that we're proud of

Finishing the project in 16h, and actually doing something important and meaningful

What we learned

Using Django, using Azure and more about the needs of the refugees.

What's next for HEARMES

Support different languages once tech that supports automation and OCR becomes available in APIs Connecting with existing organizations, refugee camps, and migrants to collect stories to share Developing partnerships with ad services to place Hearmes throughout a variety of web content Establish mechanism to facilitate delivery of help to refugees and migrants through HEARMES

Built With

Share this project:

Updates