Problem Statement: Due to the current global pandemic all of us are now working and learning from home. We believe that being able to work and learn from home is going to be the backbone of future econony and many applications can be made on this concept. This is a problem that affects everyone, from children at middle-school attending online classes and giving crucial tests that determine their cognitive development to professionals managing projects and hiring new talent to companies through online meetings and assesment. Due to the extended period of lock-down over 8 months and projected lockdown for possibly another year, this is the perfect time for us to transition to working and learning from home. Moreover working and learning from home has its own benefits on environment and economy due to less travelling and more inclusive workforce. But this transition requires the appropriate tools in terms of software and hardware improvements to current available options. It is easy to get distracted or attempt cheating or malpractice during online assessments and interviews. In this hackathon what we propose is an Arificial Intelligence based online assessment and learning platform combined with a user-friendly and effecient python desktop app that helps you keep track of your progress in online assessment.
Tech Stack used for Solving Problem Statement: Our proposed solution can be divided into two parts, the first being creating appropriate object detection and image tracking that is light weight, has high FPS and accurate at the same time, the second part is designing a user-friendly app that displays necessary and meaningful information from the AI-model to the user about the attentiveness. For the attentiveness AI-model we decided to use eye tracking, head movement tracking in two axis (left, right, up and down) and also mouth movement tacking, also in this solution it is possible to independently adjust the threshold for the 3 parameters seperately thus avoiding false results. Thus, it was a challenge to integrate all of the above models into a single manageable model. This app is able to work on any computer with a simple webcam. Both the implementation of AI based monitoring process and creating of python app are discussed below.
Creating and working of python app: Modules and libraries used: PyQt5
The Python app is developed with PyQt5 module which allows quick prototyping of apps. Great care was taken to make the app attractive and user-friendly aspossible. Multi-threading concept is used heavily to run app GUI, Quiz-timer, and also AI monitoring parallely without causing conflict with each other. The App contains many screens with detailed explanation of information about progress during test and even practice tests. The different screen features of the app are login screen, Main menu, Revision (practice test) section, AI monitored Examination section, Exam report section, and leaderboard section.
Login Screen: The login screen contains the team logo and sign in section, you can also register as new user. The login system is fully operational and the username and password are stored in a dedicated SQL database. The advantage of using this type of system is that the app can be used by multiple users and their learning and assessment process can be tracked individually. This system is also password protected and prevents you from creating multiple accounts with same username.
Main menu: The main menu is from where all the different features of our app disscused can be acessed. The important sections are discussed in detail below.
Quick revision Section: The Quick revision section contains practice test for learning concepts without assessmnet. This contains questions which you can answer and also get hints from explanation button. The answers are also revealed instantly for efficient learning experience.
Examination Section: This section contains test for assessment purposes,they are mcq type questions, the tests are timed and remaining time is displayed by quiz-timer, the test is also monitored through webcam by the Assessment monitoring AI system (AMAS), and if malpractice or lack of focus is detected then warnings are given and the total warnings are displayed in the warnings section. At the end of the test the score is dispalyed along with test analysis data.
Exam Report Screen: In the Exam report screen we can see the performance position of the current user, also a graph shows the learning improvement of the user as the user atempts more tests. This acts as a motivation for the user.
Schedule and Syllabus Screen: Once the application is in real world use the Schedule and Syllabus section can be used by teachers and exam-setters to update the timing and content respectively for each test.
Leaderboard Screen: In the leaderboard section we can see the performane scores of other users who have attempted the test. The leaderboard is updated dynamically and shows the latest performance scores of the multiple users.
Implementing AI based monitoring processes: Modules and libraries used: Tensorflow 2.0, OpenCV, Dlib(deeplibrary)
The AI-model problem can be broken down into 3 categories, Eye movement tracking, Mouth movement tracking, Head movement tracking. The solution and implementation of each of the above on a live video feed from a webcam is explained below.
Eye movement tracking: This can be done using segmentation and face landmark detection.The motion of the centre of eye frame is calcualted. If the position of the center of the eyeballs changes with respect to the rectangular coordinates of the face above a certain minimum distance threshold value (which can be adjusted), then a warning is sent through the python app for looking left or right during the assesments.
Mouth movement tracking: Mouth movement tracking is done in a similar manner to eye tracking here also we use Dlib’s facial key points. When sitting facing the webcam the distance between the lips key points is calculated and averaged for 100 frames, if the candidate opens his/her mouth the distance increases and if it increases more than a threshold value (which can be adjusted), then a warning is sent through the python app for talking during the assessment.
Head movement tracking: This can be done by keeping track of certain landmark points on 3D space and the angle of movement is calculated with respect to the center point, in two axis (left, right, up and down),if the angle is greater than a given threshold value (which can be adjusted), then a warning is sent through the Python app indicating to concentrate on the screen and avoid looking away from the assessment.
Face, eye and mouth movement tracking: This is what our AI assessment monitoring system sees, it keeps track of eye, mouth and face motion landmarks to determine if the test taker is not focussing or cheating. If abnormalities are detected then it posts it as warnings to the app GUI.
Log in or sign up for Devpost to join the conversation.