Upload a photo or take a photo of a flyer to try it out on the Web App: https://caughtup-parser.web.app/home
We wanted to make scheduling easier. Image a world without the hassles of planning.
What it does
The android, ios, and web application allows users to upload or take a photo with their camera and we create a calendar event for them.
How we built it
The client applications allows users to upload a photo or take a photo image containing text describing an event which is then uploaded as an jpg to Firebase Storage. It then makes a request to Google Cloud Functions to retrieve that image from storage, extract text from the image, then extract event data from that image, then return the event info item back to the client applications. The client applications then display the extracted event with Title, start time, and end time. Once here the user can select to save the event to their device's calendar on android & ios.
- We use the Flutter software developer kit to create the android and iOS front ends.
- We used Angular for the Web App front end
- We deployed the Angular app online with GitHub pages
- We created a firebase (google cloud) server project to leverage accurate google vision api in the cloud
Challenges we ran into
- Setting up a teammates' computer
- Coming up with the idea to work on this project took us about 5 hours.
- Extracting text from android and ios devices were in accurate so we decided to use Google's cloud vision api which were more accurate than the local device parsing.
- For 3 hours we couldn't figure out why text images on firebase storage couldn't be retrieve by the Vision API. we found out one of us was looking for a .png image rather than a .jpg image
- Once we had the extracted text we displayed a messy blob of data to the user to parse into an event. we found this more difficult than just typing the event. So, we decided to parse the information for them. We looked for packages to help extract event titles and date times from a corpus of text.
Accomplishments that we're proud of
- Android, iOS, Web App, and Server
- Teaching and introducing one of our teammates to how angular is used in conjunction with a backend server API
- Using Google Cloud Vision API
- Using Firebase Storage
- Using Cloud Functions
- Staying up all night to make this happen. Persevering and coming up with alternative solution to problems when it seemed like there was no hope for this project (the struggles getting Google Vision API to work as intended and once it work it gave us a disgusting mess to deal with)
What we learned
- How to use Google Cloud API
- How to set up a server with google cloud credentials
- How to deal with billing info for Google Cloud
- How to upload pictures from a website
- How to use device camera's from a website
- How to teach someone
- How to sleep on the floor of Surge
- How to explain to people in a productive manner that their solution will not work
What's next for Android, iOS, and Web App | Flyer Event Extractor
- We're going to integrate it with a larger Android, iOS, and Web application named CaughtUp that we're creating with the goal to maximize the amount of time people spend with the ones they care about.
Server (Cloud Functions) Journal
- 2020.02.15 4:41 am Working CloudFunctions
- 8:30 am Recieve images from clients, call metod to retrieve images Detect image from image in firestore
- 2:30 pm String to Event Package and Function
- 2020.02.15 3:00 am Working return for string with expected results
Angular Web App Journal
- 2020.02.15 12:44 am Initial Commit
- 2:21 am Working webcam
- 3:11 am installed firebase
- 3:16 am firebase module
- 5:40 am Deployed to github pages
- 10:34 am Created Cloud Service
- 2020.02.16 3:00 am updated result keys
Flutter Mobile App Journal
- 1:30 am Initial Flutter Mobile App Project
- 2:06 am Upload image from gallery or camera
- 2:21 am Created fragment for selecting text
- 2:37 am Added horizontal buttons
- 2:42 am Moved select buttons to top of page
- 3:34 am Updated Android app package & registered firebase application
- 5:43 am Initial access to retrieve user device calendars
- 6:47 am Upload image to Firebase storage
- 6:55 am Added loading indicator for upload image and updated gradle
- 3:45 am updated card design and show raw text
- 3:50 am added toasts to notify user of current states
- 4:07 am selectable text