Inspiration

With the rapid shift to online learning and assessments, maintaining academic integrity has become increasingly challenging. Traditional proctoring methods often lack sophistication and can be intrusive, leading to privacy concerns and reduced trust. We were inspired to create Pixtral Proctor to leverage the power of multimodal LLMs to provide a smarter, more efficient, and less invasive way to detect cheating during online exams.

What it does

Pixtral Proctor takes three live feeds—screen recording, webcam footage, and microphone audio—and passes them to the Pixtral multimodal LLM. Pixtral determines whether there are any signs of the user cheating: for example, it will detect if the user is on their phone when they're not supposed to be. This information will then be used to focus an exam proctor's attention on the potentially cheating user and allows them to directly communicate. How we built it

We used the Remix.run framework to implement the two interfaces: one of them is a small page that a user keeps open to allow their screen and webcam feeds to be sent to the proctor, and the other is the proctor's dashboard, which aggregates all the incoming feeds and provides tools for monitoring and communication. We integrated the Pixtral multimodal LLM to analyze the live feeds in real-time, enabling prompt detection of any suspicious activities.

Challenges we ran into

The most significant challenge of the project was sending live video feeds across the internet. Forwarding two streams of video for each user in a large-scale setting is bandwidth-intensive and requires efficient real-time processing to prevent lag or quality loss. Additionally, ensuring the security and privacy of the data while integrating the multimodal LLM added layers of complexity to the development process.

Accomplishments that we're proud

• Successful Integration: Seamlessly integrated a multimodal LLM into a live proctoring system.
• Real-Time Processing: Achieved efficient real-time analysis of video and audio feeds without significant
• User-Friendly Interface: Developed intuitive interfaces for both students and proctors.
• Scalability: Created a solution capable of handling multiple users simultaneously without compromising performance.

What we learned

• Technical Skills: Enhanced our understanding of real-time video streaming and web
• AI Integration: Learned how to effectively integrate AI models with live data
• Team Collaboration: Improved our teamwork and problem-solving abilities under tight deadlines.
  -User Experience: Gained insights into balancing functionality with user privacy and comfort.

What's next for Pixtral Proctor

Feature Expansion: Implement additional features like eye-tracking and behavior analysis for more accurate detection.
Optimization: Enhance performance to handle even larger numbers of users seamlessly.
User Feedback: Gather feedback from pilot users to refine and improve the system.
Partnerships: Seek collaborations with educational institutions to deploy Pixtral Proctor in real-world settings.

Built With

• pixtral
• react
• remix
• typescript