Harmonies

Because your images need some magic

Comment 1

Inspiration

Harmonies is an online photo editor that aims to simplify the process of editing photos. Now, you can use the same advanced tools from photoshop by dragging and dropping easily into the canvas. We take advantage of the capabilities of computer vision to help our users edit photos in an appropriate way.

What it does

Harmonies app helps the designers, and editors to create a full, rich experience for the users or customers. In addition to the regular editing tools (like Crop, Rotation, Drawing, and Shapes); we provide the user with three powerful computer vision techniques to cut, color, and add images.

Image Colorization: Harmonies deals with the process of taking a grayscale input (black and white) image and then produces a colored image that represents the semantic colors and tones of the input.

Image Segmentation: Harmonies benefits from the concept of Image Segmentation to extract some parts from the image and return a png photo.

Deep Painterly Harmonization: Harmonies produces significantly better results than photo compositing or global stylization techniques and that enables creative painterly edits that would be otherwise difficult to achieve.

gif

How we built it

Front-End: We used React for front-end development. It gives the privilege to have a single page application (SPA) with a clean modern design that is easily maintainable.

Back-End: The technologies used in the backend are node js, express, MongoDB. We established our frontend and backend communication using JWT tokens.

RESTful APIs We have used Flask library to create a web API for both the segmentation and the coloring models. This API takes base64 images as inputs and runs preprocessing on these images then feed them to each model depending on the request URL. This API was then deployed to Azure web services via a git repository and integrated with the front-end editor.

Image Colorization: In this part we reimplemented Colorful Image Colorization using PyTorch for images auto colorization.

Image Segmentation: In this part we reimplemented Rethinking Atrous Convolution for Semantic Image Segmentation using PyTorch for auto-cropping a person from an image. We used the same concept of image segmentation and instead of adding masks, we return a PNG photo.

Deep Painterly Harmonization: In this part we reimplemented Deep Painterly Harmonization using PyTorch to add harmonies to the adjusted element.

CV technical details are fully described on our Github Repository

Challenges we ran into

The biggest challenge we faced was that the team worked together remotely, spread over different time zones. Moreover, it was difficult to:

  • Create a complete Machine learning web application using React and Flask, and dealing with different APIs and data types.
  • Developing the application workflow to be fully automated.
  • One big challenge is not having an NVIDIA GPU on my device, one way of solving this problem was using the cloud for testing and inference.

Accomplishments that we are proud of

We are proud that we have participated in this competition competing against people from all over the world. Furthermore, this Hackathon helped with meeting other incredibly talented people like us, working as a team, and taking on challenges that put our problem-solving skills to the test.

This is our first time as a team and we have successfully created a full MVP during the Hackathon. Moreover, our model has been deployed as a real-life project, and it could be used easily.

What we learned

  • How to integrate the backend API with a frontend making it secure with JWTs.
  • Deploying a machine learning model to the cloud.
  • Get into the team-work technique and sharing ideas.
  • Working completely remotely with a team for the first time.

What's next for Harmonies

The future of Harmonies depends on two main pillars, the first is the technical pillar and the second is the commercialization pillar.

For the technical part, we will retrain our model with a bigger dataset to get better results with image colorization and auto-cropping. We are also studying adding some more computer vision features like image enhancement and converting images into pictures. We will optimize the code to reduce the run time.

For the commercialization part, we are thinking about how to cover the cost of the cloud by adding some ads to the service, make premium packages, or making use of the imaging data.

In the future, we will develop a mobile version of our website to support more users.

+ 6 more
Share this project:

Updates