Jarvis 1.0 using realsense

Intelligent virtual assistant for PC

Comment

Inspiration

Tony Stark Virtual assistant

How it works

Pretty much like siri

Challenges I ran into

To make my accent work ;) also to minimize crashes

Accomplishments that I'm proud of

Works well for a prototype version

What I learned

face tracking and figuring out emotions

What's next for Jarvis 1.0 using realsense

emotions based analysis, better personalization using machine learning

Smart digital assistants have already created a buzz among smartphone users, after the launch of siri. Google Now and Microsoft's cortana also joined the battle of smart AI based assistants.

I was invited by intel to participate in Ambassador track in their realsense challenge. I took bit of inspiration from the famous movie Iron Man and created the first prototype of my own version of Jarvis program as shown in the movie.

The program key functionalities (version 1.0) were as follows :

Jarvis like personal assistant, very rough prototype a lot of work is to be added to it in subsequent releases. (developed in 8 days of work a lot more to come in the final version) please watch the section after in the video below.

OPENING AND STARTING COMMANDS

Open/Show any folder (e.g say, open documents, show music etc but dont say open/start music); Open/Start any default/installed desktop or metro application (e.g say, open skype, open photos, start paint etc.) in you have an application installed with a long name like intel realsense sdk 2013 just say intel and it will search the desktop shortcuts and windows search and open it for you. Search/such/Google -- opens google in default web browser with your query e.g search center of earth , or search lion i included such because of accent detection problem

SOCIAL COMMANDS

These are the social commands the way jarvis will respond to questions will depend on your gesture mood, but this functionality was causing crashes very often so use it wisely, although i handled the exception with static answers, so if you are getting same replies you can understand that the system has thrown an exception at some stem. How are you doing: How are you: Whats up: will you marry me: Tell me a joke / Joke: *tells you a random joke *

CUSTOM COMMANDS

Minimize -- minimize all windows mute / /jarvis mute -- windows speaker mute unmute / Jarvis unmute kill power -- shuts down the system lock -- locks the pc time -- tells you current time date -- tells you current date

TRANSLATE COMMAND

provide the sentense you want to translate in the following format: "Translate hello friends to Spanish" -- and the system will speak up the translated voice

INTELLIGENCE RESPONSE

The system is fully capable of fetching context related data to any query in a very nice format just remember to turn intelligence on, it gets data from different sources and provides you result in a nice ui container. opens browser if unable to find related data. If you use intelligence you have to be a bit patient with the rendering of the results it will take some time, and you cannot crosstalk, you have to click stop listening manually if you dont want your speech to be detected. e.g. : Lion , Love, Lord of rings , population of delhi etc. anything can be passed as a query. but don't bombard queries simultaneously as the app takes some time to render and fetch data and it might hang the system. If you want to use it without intelligence it can be easily used as a personal assistant, and you can use all of the above commands except queries, in this mode you can crosstalk without manually stopping the listening

Built With

Share this project:

Updates