Inspiration

Zoom University has been too long and with everyone having been in social isolation for over a year it has been getting harder and harder to focus. With minimal interaction and lengthy lectures class is becoming uninteractive repetition. We thought there should be a faster and more fun way to learn.

What it does

Study Pal allows you to upload a file such as an image or an audio file of a lecture. We will use OCR and speech to text to transcribe your media. Afterwards we will route it through ML to create a study guide consisting of a summary, key points, external resources and a quiz all generated through code based on your lecture content.

Key Features

  1. User Authentication through firebase
  2. Study Guide and File model in Firestore
  3. Quiz models stored on CockroachDB
  4. Optical character transcription for image formats jpg/png/pdf/jpeg
  5. Speech to text transcription for audio formats like wav
  6. Natural Language Processing to generate: a. Summaries b. Quizzes with Answer Key c. Keywords
  7. Web Search API to find relevant external resources and youtube links
  8. PDF generation
  9. Shared documents and profiles for users and study guides

How we built it

Create-react-app, Jupyter notebook, Google Colab, Google Cloud Functions

Accomplishments that we're proud of

First time using Cloud Functions Working Quiz generator with high accuracy using ML

Challenges we ran into

  • Cloud SDK hates windows paths with spaces
  • eslint setup for cloud functions is hard with yarn
  • yarn and npm conflicts
  • merge conflicts
  • Trying out way too many new technologies at once
  • Debugging GCP was painful as the firebase emulator doesn’t cover ML capability.
  • SerpWow ratelimited two of us
  • CORS

What we learned

Having a logic and frontend flow diagram was helpful to directionality. We used GCP more than ever before. First time doing redux for one of us

What's next for Study Pal Beta

Use Google Pub/Sub more thoroughly to separate the layers and support async calls with large payloads. Provide status indicators and polling flows for user friendliness.

Tech

  • Firebase (Auth)
  • Firebase Storage (Files)
  • CockroachDB / Firestore (User/Resource Model)
  • Firebase (Hosting)
  • Cloud Functions (Process Transcription background)
  • Google Cloud Pub/Sub (Async Tasks TODO)
  • Google Memory Store (Caching SerpWow API TODO)
  • Google Cloud Vision (OCR Vision)
  • Google Cloud Speech to Text (Transcription)
  • Google Collab (Create Q&As)
  • Google Cloud Natural Language (Keyword Summary)
  • Serpwow (External links/resources)
  • TailwindCSS, BaseWeb (UI)
  • PDF Kit (PDF Generation)
  • Flask, NodeJS (Backend)

Built With

+ 4 more
Share this project:

Updates