-
the small dot is the drone flying
-
arch diagram
Inspiration
I’m a computer science graduate student, but I’ve always been fascinated by aeronautics and aviation. Although I didn’t get the opportunity to formally work in that field, I’ve always wanted to build something related to aviation for myself. This project gave me the chance to combine what I know—AI systems and software—with my passion for aviation. The result is a simulation of how a drone can operate when controlled by AI, essentially creating a foundation for a self-driving drone. That’s the inspiration behind this work.
What it does
In simple terms, it makes the drone fly. For example, if a user says, “Go survey the plot at coordinates X and Y,” the AI uses the available telemetry to determine the drone’s position, altitude, velocity, and other physical aspects required for the task. It then generates a structured command output that the drone can understand and act on.
Whenever a request or command comes from the user, the AI module translates it into actionable instructions. These commands are passed to the drone, and all the actions are executed within the WebOps application, where the simulation takes place.
How I built it
There are three important aspects of my project:
LLM Architecture – I used two LLMs to handle real-world data and commands.
Shadow LLM: Collects all telemetry from the drone environment (such as blocks, location, and state). It has the overall knowledge of the drone’s environment and pushes this data into Redis.
Action LLM: Reads the processed telemetry data from Redis. Based on the drone’s current state, it decides what actions to take, such as altitude, velocity, and survey tasks.
Redis – Serves as the central data bus for communication. It stores telemetry, commands, and state information so both LLMs can work in sync.
Webots Simulation – A custom simulation environment for the drone. It includes obstacles, blocks, and a rectangular arena where the drone can fly and react to commands.
By combining these components, I created a loop where the Shadow LLM provides state awareness, the Action LLM generates structured commands, Redis coordinates the flow, and Webots simulates the drone’s behavior.
Challenges I ran into
The biggest challenge was making the entire ecosystem work together correctly. This project was fairly complicated for me, and I ran into multiple issues along the way. Sometimes the drone wouldn’t respond as expected, the LLM failed to interpret telemetry properly, or Redis didn’t deliver the data reliably. Even when commands were generated, they didn’t always translate into the correct drone actions.
Right now, the drone still doesn’t fly perfectly — it doesn’t always behave the way I want — but each failure has helped me better understand how the system works and how to improve it.
Accomplishments that I’m proud of
Right now, I can’t say I’m NOT fully proud of the implementation because the integration didn’t work as expected and the drone doesn’t actually fly yet. What I am proud of, however, is the thought process behind the project — the idea that AI could control a drone and make it fly autonomously. Even though the current build still needs a lot of physics, fine-tuning, and improvements, I’m glad I took the first step and built a foundation. For me, the accomplishment is more about the concept and architecture I envisioned rather than the current working state.
What I’ve learned
Through this project, I’ve learned a lot about what not to do when building an application. I discovered the pitfalls of structuring systems the wrong way, and those lessons will guide me when I refine the project into a working version. Even though this first attempt isn’t fully successful, it has taught me valuable lessons that will help me avoid the same mistakes in the future. I’m confident that with these insights, I’ll be able to improve the system and get my first drone flying soon.
*What’s next *
The ultimate goal is simple: make the drone fly. Everything I’ve built so far is laying the groundwork for that moment, and the next steps will be focused on turning the concept into reality.
Built With
- langchain
- python
- redis
- webots
- websockets