BrightBoard

Our API architecture, green arrows = images, black = text, red = knowledge
A screenshot of our UI with lots of knowledge messages

Inspiration

Brightboard is about using technology to make learning magical. We set out to transform the classroom experience from simple passive engagement with a lecture into a magical "living in the future" moment - one that makes learning more effective and encourages discovery.

Brightboard hides a lot of incredible technology behind an intuitive and simply user interface. It transforms data into knowledge without requiring the student to do anything other than explore what they love the most. With the help of Cisco's Spark, it creates a community of knowledge with incentives to share and discuss discoveries - both in the classroom and beyond.

In short, Britghtboard is a new vision for how to learn in a connected age - we have more information at our disposal than ever before but making those critical associations with what we learn in class is still far too difficult.

What it does

Brightboard interprets on-demand snapshots of a whiteboard or powerpoint presentation from an Android Application. These snapshots parsed by a number of web services for image identification, text processing, entity extraction, thematic analysis, and key word checks (see below for the technical details). The results of these APIs are standardized into a Messaging syntax with is passed to a simple and intuitive single-page web application. From the web application user's can interact with knowledge alerts, and even share those alerts with others through a Cisco Spark chatroom.

How we built it

Despite the simple frontend there's a ton of technology involved in this project. Behind the scenes the following services all work together to produce meaningful semantic knowledge from harvested images: 1) Microsoft Cognitive Vision APIs (Both image analysis and OCR) 2) Microsoft Cognitive Knowledge APIs 3) Microsoft Language APIs 4) Cisco Spark Developer Kit 5) Amazon Web Serices - EC2, EBS, etc. 6) Amazon Comprehend API 7) Google NLP API 8) Wikipedia API 9) ArXiv.org API 10) CrossRef API

Our core server is a flask application supported by Flask-SQLAlchemy. The server interacts with Amazon webservices via a boto client, and with most other APIs via REST.

On the front end is a JSViews + Bootstrap single-page, responsive web application.

Additionally, we developed a full Android application to support our project. The Android application allows converts, appropriately scales, and then uploads images directly to our Amazon EC2 instance.

The general flow of information through the application is as follows: 1) The Android Application takes a picture of a whiteboard or powerpoint presentation (other data such as books, video screenshots, or notes are also supported). It uploads this picture to a Flask Application running on AWS EC2.

2) The Flask application processes the image - extracting any sub-images of interest (e.g. a picture on a powerpoint slide). It forwards these processed images to two Microsoft Cognitive Vision APIs. One API provides us with identification of landmarks and famous figures, the other provides Handwriting recognition.

3) Recognized text is then fed to other Azure/Cognitive machine learning and language processing APIs. These services extract relevant features from the text, and identify proper nouns, specialized vocabulary, and terms of art.

4) Further text analysis is provided through AWS's Comprehend Service and Google's NLP processing services. The AWS Comprehend Service is used primarily for identifying locales and other keywords in unstructured text, the Google endpoint is used primarily for identifying academic authors and proper names.

5) The result of this text parsing is converted to useful information. This includes queries to a RESTful database of countries, local analysis against a periodic table data structure, and requests to the Wikipedia API along with the CrossRef and ArXiv APIs for academic papers.

6) All this information is consolidated into JSON objects which are fed to a web frontend and displayed to the student. Each message provides basic information on a topic related to the ongoing lecture as well as links where the student can go to learn more information and explore their passions.

7) The service integrates with a custom-built CiscoSpark chat bot, turning exploratory learning into a collaborative experience. When a student clicks the "Share on Spark" button a message is posed in a Spark 'classroom' team where classmates can engage with the content or discuss more actively. All posts are anonymous to encourage students to feel comfortable sharing what they've learned even if they wouldn't feel comfortable doing so during class.

Challenges we ran into

This project turned out far larger in scope than we anticipated. Although we ended up with a project we're quite proud of, the reality of juggling all of the modular components and APIs meant that communication and coding to a higher standard than we're used to a hackathons was important.

We also ran into some trouble making decisions about what information was most important and how to prevent information overload. In the end, the Cognitive APIs and, especially, the Knowledge API, helped up focus on the most important information on a whiteboard and ignore a lof of extraneous data. However, throughout the hackathon we struggled to build models that would be useful in practice.

Accomplishments that we're proud of

Getting a viable project that works fairly reliably is something that we're all proud of. More importantly, we're incredibly proud of how well our team came together. As a group we run the gamut of hackathon experience from members who have been 10+ to first time hackers. However, throughout the Hackathon we found opportunities to work together and leverage our skills to both learn and develop software. Everyone has code in the final product and everyone has something in this to be proud of.

What we learned

We learned a ton about working with image recognition, understanding themes in 'dirty' / incomplete data, and building modular code such that multiple developers could work simultaneously without stepping on each other's toes.

What's next for Brightboard

Brightboard has a lot of potential to change the way learning happens and make classrooms more engaging and productive for modern students. All of us, especially Edward, are super excited about the opportunity to keep working with this prototype and to take advantage of some of today's most exciting tech to make our everyday lives incredible. Tools like Brightboard are an inevitability and the chance to experience the cutting edge of a maturing field is something we'll all be able to apply to our future work on this and other projects.