Just something I thought of on the bus. I'd done something with distributed tasks before but not cloud-based, so I thought this was an interesting way to reinvent it. Vince got on board and we started hacking on it.
What it does
- Receives task configuration from the web
- Shows task status on the web
- Installs docker containers across many AWS instances
- Automatically scales AWS spot instances based on the best price
How I built it
Firebase is used as the database for the whole thing. There's an “orchestrator” backend application which polls the database occasionally and tries to figure out the most optimal distribution for new tasks, and checks to see whether it needs to scale. There's an “instance daemon” which receives tasks from the orchestrator. Finally, there's the web interface made with Firebase (of course) and React.js.
Challenges I ran into
Algorithms, mostly. Managing all that state is not trivial.
Accomplishments that I'm proud of
It works! We can ship docker containers to our instances and intelligently distribute them, and display the progress.
What I learned
What's next for computespot
- Making the instance scaling more reliable
- Implement down-scaling as well (shrinking)