Inspiration: The inspiration for our project stemmed from the innovative concepts introduced by Rabbit OS, particularly the concept of Large Action Model (LAM). LAM, a foundational model designed to understand human intentions on computers, intrigued us with its potential applications in smartphone interactions.
Learning Experience: Throughout the development process, we delved into various technologies and tools to bring our project to life. One key learning was mastering the Android Debug Bridge (ADB), which enabled us to control smartphone functionalities using taps, swipes, and other gestures. Additionally, we familiarized ourselves with OpenCV, leveraging its capabilities to mark grid lines and numbers on screenshots captured from the smartphone.
Project Development: Our primary objective was to integrate the principles of LAM into smartphone control, utilizing Gemini Vision Pro as our core framework. Gemini Vision Pro offered the advantages of being both free and highly accurate, crucial factors in ensuring the effectiveness of our project. We focused on optimizing response times to ensure seamless execution of commands.
Challenges Faced: A significant challenge we encountered revolved around streamlining the process of sending screenshots with prompts to the Gemini Vision Pro model. Initially, sending a single screenshot was straightforward; however, incorporating both the current and previous images with the prompt posed a more complex problem. We tackled this challenge by utilizing REST API calls but faced hurdles in ensuring the smooth transmission of data.
