Inspiration

Most news sites today are built to attract viewership through flashy media ads and video content. However, for a visually impaired user, this makes navigating the site and finding relevant content extremely difficult. Screen readers are not built to handle this and the sites themselves are not designed with them in mind. Furthermore, few developers on these sites are asked to cater to these needs as they drive fewer clicks through revenue.

Rather than make it the onus of each of these publication sites to do this job, we decided that we could set up a standardized, streamlined, and accessibility-driven portal to allow readers to effectively navigate to the content that they read. We do this through the help of natural language processing and computer vision.

Once the user has been successfully led to the article that they want to read through the guidance of this featurized metadata, we then ultimately redirect the user to the main origin site so that the user can both enjoy the article in its original context and the makers of the site and writers of the article get their due credit.

Walkthrough

The what

We've built a full-stack solution that is capable of fetching current news articles from all major publications (i.e. WSJ, New York Times, Fox News, Wired, CNN, etc.). It then fetches the raw text for these articles and then passes them through a summarizer that synthesizes the article into one sentence. This sentence is then fed to the screen readers as they traverse these articles through the FE React.JS app that we have made. Furthermore, we also run computer vision algorithms on the images from these articles so that those who cannot see the images can still understand the tone and overall idea the photographer was attempting to capture.

The how

In a nutshell, our BE service is responsible for coalescing the information from all the different APIs we interface with to understand as much as possible about an articles semantic and visual data. We then send these features back in a well-formatted manner for the FE to receive. The FE then uses a combination of the accessibility-friendly Stardust UI Framework, in combination with aria HTML tags and accessibility minded website design, to offer an experience that is delightful and friendly to users regardless of their visual capabilities.

The why

It is news streamlined and distilled. From every source. For everybody.

How we built it

NewsAPI - Current news

NewsPlease - Scrape the text from article urls

IBM Watson - Computer vision to featurize the images

Aylien - Summarize the article

Amazon Lambda, ElastiCache, API Gateway

Stardust UI Framework

React.JS

Node

Challenges we ran into

  • API costs were extremely high for the types of algorithms we were trying to run on the cloud. NLP and, particularly, CV algorithms are heavy on the servers and as such, the SaaS providers charge a premium for them. We've tried to ameliorate some of these costs by using a server level cache for our results using Amazon ElastiCache.
  • Caching also helped us overcome our next big obstacle which was performance. Running these algorithms in real-time, especially on non top-tier server really takes a lot of time and has a strong negative impact on UX. As we can anticipate a user's clicks for news, we can run cron jobs on a scheduled basis that constantly keep our cache refreshed and available for the next customer.

Accomplishments that we're proud of

We were able to successfully integrate 4 different APIs into a demoable product. It was a huge uphill battle understanding the intricacies of each of these APIs, not to mention the countless other alternatives we explored and decided against or gave up on because of recurring issues. Furthermore, linking them together on a BE that is running on multiple different micro-service as Lambda functions and storing results in ElastiCache means that the request are passing through quite a large number of moving pieces in the newseye network before reaching our FE web app. We are proud that we were able to set this up to work and communicate with one another in the first place, and from there on, the mission of the project drove us to finalizing it in a productizable project.

Interfacing with Aylien's text summarization and Watson's computer vision APIs through Amazon Lambda were great forays into applying ML algorithms in real-world production environments. Finally, connecting all of this data back through accessible, clean Stardust UI components was a sobering reminder that the data garnered from lengthy articles and processed by advanced algorithms can still be distilled into an easy-to-understand manner that anyone can use.

What we learned

  • How to deploy microservices on the Amazon Lambda stack
  • How to use expensive ML algorithms in a performant, cost-efficient manner
  • How to design and build accessible, screen reader friendly components

What's next for newseye

  • Expanding our ability to cache our BEs with more news sites that we have learned from
  • Adding the abilities to navigate by categories of articles
  • Making the caching run as a headless service
  • Cache the computer vision processing
  • Built With

    Share this project:

    Updates