The project was inspired by seeing the power of images, which are being used not only to capture memories but also for publicity, as posters, for information sharing, and a lot more. A static image is not actionable and doesn't has hyperlinks like a PDF document. With this project, we change that and harness the true power of a jpeg/png image by making it dynamic.
What it does
The project converts a static image into a dynamic image which can be directly clicked for actionable items. For example: if the image contains a website link, then the user can directly tap the link on the image and the project will automatically open the website in the default browser. Similar dynamic functionality has been developed for calling a phone number and for composing emails too.
How I built it
The project has been built as an iOS app using Swift. We have used the Google Cloud MLKit to perform Text Extraction on the image. Next, we input the detected text to the native NSDataDetector class and detect the type of data (eg. web link, email address, or phone number). After detecting the type of data, we define particular actions for each data type and trigger them if the user taps on the particular data point. A Tap Gesture Recogniser has been used to recognise user taps and trigger the defined actions for specific data types.
Challenges I ran into
- Setting up MLKit with CocoaPods. Ran into some linker issues that were hard to debug. Solved them through compiler flags.
- Classifying extracted text into specific categories (web link, email address, phone number) and overlaying buttons to make the image actionable.
Accomplishments that I'm proud of
- Finished the project successfully
- Our project works perfectly in all the use cases that we've tested, including the Hack The North poster itself!
- Fixed a lot of bugs that were really hard to diagnose.
What I learned
- MLKit and Text Extraction from Image
- Text Classification into data types
- Defining specific actions through deep linking
What's next for Dynamic Image
- To further polish it and launch it on the App Store as a utility app
- Implement additional use cases