Teleport

Live Video Conferencing
AR embedding of video within your own environment
Deep Learning based Facial Segmentation

Inspiration

Having decided to come to a far off place, 8,000 miles away from home to participate in a hackathon at Harvard without a team and having no one to support me in the midst of heavy competition. I missed my family. And the whole point of understanding that being a global citizen means cities that are meant to make people closer, and transportation systems that enable the proximity of 2 individuals, it was almost near impossible to fly 8,000 miles now to just meet and talk to my family.

Let's say in a village, city or country, you want to travel. You can travel either by road (though cars/uber/bus), trains and flights. Taking the example of an Uber ride from Amherst to Harvard, it took 45 minutes of waiting and spending $180.

Why? Because there were no drivers around.

Why? Because those Uber drivers have less incentive to be in that area when compared to popular cities like NYC and Boston.

Why? Because BIG cities are the hubs to work and people come here to do that.

Why? Only certain places have access to those opportunities to connect.

And so, we wanted to change that. Just because a country lacks good public transportation system, lack of Infra and other things, shouldn't stop people to connect. So, how do we enable EVERY place in the world, and give it the ability to go anywhere you want?

Thinking out of the box, you need Dr. Strange's portal to teleport. Oh wait, teleportation is possible? With the advent of the metaverse it is. It'd be nice if our family could check in on how we're doing thousands of miles away without disturbing us 24/7.

What it does

Teleport, is a solution, that is powered using Deep Learning to segment your face, using an IoT Network to create a reverse proxy for your localhost computer that essentially negates paying a cloud provider for server space to run this and connect , and finally using Augmented Reality, bring an immersive experience using ONLY your phones (no need for big expensive Meta Quests/ Apple Vision Pro) to connect with the people you love, business deals or just checking in by "teleporting" you to their world!

How we built it

By first creating a video conferencing tool using Node.js and Express.js. And then creating a separate module using Python to make use of dlib, a Machine Learning framework for facial detection, and doubling down by using Nvidia SegFormer Deep Learning model to only segment the face part to be displayed for the AR feature. And used AR.js to embed the AR/VR module to be loaded on a browser.

Challenges we ran into

Initially, creating a web socket that concurrently handles the connections between 2 people via browser was a challenge. Then while using primitive ML models for facial segmentation, the model would not recognize parts of the face and it had black patches within the face, which is why we integrated Deep Learning to make it more robust. And one of the biggest problems, is integrating the video that we get from the model segregated face to appear on a browser in an AR context.

Accomplishments that we're proud of

I did all this despite my team not being there due to emergency, as a full stack developer with no prior knowledge on Deep Learning, Integrating a IoT network, and AR embeddings, I was able to learn and implement these modules in a very short time (<8 hours)

What we learned

Deep Learning, Machine Learning, IoT Networks, AR Integration on Environment

What's next for Untitled

Create an end-to-end solution where all of them can be accessed in a single website URL. Like a metaverse based connection platform but uses live AR to make it immersive in your own world space, Make Meeting in your space.