I was inspired by my lens projects: Melt Cup and Rainbow Teddy Bear

What it does

What this lens does is detect person, and then apply a green color effect to his/her body like an alien, or incredible Hulk, or the Munsters family. Here I used segmentation and machine learning with SnapML, also adding color correction (green), a mask, and edge detection (green).

How we built it

1) I followed all the instructions in the Custom Segmentation template guide; 2) then I opened the segmentation_training.ipynb notebook; 3) I opened google colaboratory; 4) I made some small changes to segmentation of person and you can see in this link person_training.ipynb; 5) Following the step 1, I downloaded the person_segmentation.onnx, then I opened the Lens Studio program and loaded this model; 6) Finally I have added the "Color correction effect", so I selected the green color, and I applied a mask the image or video, 7) My full Lens Studio project on Github you can download at this link Green Person - Github; and 8) My Lens Project is public, unlocked, and you can get here this link Green Person

Challenges we ran into

There were complicated challenges, but satisfying to do! 1) Google Colaboratory offers little time to experiment for free (4-5 hours per day); 2) In Lens Studio documentation I didn't find enough technical support about machine learning and effects on Lens Studio, I hope they publish some tutorial or more documentation on machine learning (how to solve technical problems, how to make a machine learning template, or how to use other machine learning tools); 3) I not found enough machine learning webinars with Lens Studio and taught by technical staff; and 4) Also experimenting with different effects was difficult.

Accomplishments that we're proud of

I finally made my Lenses with Lens Studio as I want, and using Machine Learning with snap ML I achieved a good person segmentation thanks to the artificial intelligence training, plus the excellent tools of Lens Studio, now I just have to experiment to improve my model

What we learned

I also learned how to experiment with various types of effects and color corrections that can be applied to the lens I'm developing. Hopefully, the technical staff of Lens Studio will develop more tools or improve the version. Eg, I experimented with color correction, then added green color to the mask of the detected person. even, I improved this effect by adding the edge detection effect with green color.

What's next for Green Person

As I said, it remains to improve the model increasing training or adding more images, also I want experiment with other effects or color correction to my green persons lenses. I don't know, but I would like technical staff to do a webinar to teach me how to make my own effects and load them into Lens Studio.

Built With

  • google-colaboratory
  • lens-studio
  • python
Share this project: