a resilient large file downloader for unstable connections

i spent most of my waking hours downloading a 6–7 GB file

Comment

This project, in solving a niche, immediate problem, addresses a broader issue of reliable data transfer in low-connectivity environments around the world.

To be honest, I spent the majority of Saturday trying to download the 6.9 GB file from AppLovin, only to find out that I underestimated the challenge (or was too sleepy to figure it out by the time I got the file). I don't want to leave without submitting a project though.

Inspiration

The project was born out of necessity because I was planning to attempt the AppLovin query challenge. Google Drive and USB drives were blocked by the organization policy of the laptop I was using, so only my phone could access the Google Drive file. In addition, since I received better connection in the residential streets right outside the Palace of Fine Arts, it was more convenient to carry around a phone than a laptop.

What it does

My phone can connect to XfinityWifi in the streets of San Francisco, and it's a lot faster (30 Mbps vs like the 1 Mbps I was getting in the afternoon). XfinityWifi has surprisingly good coverage in the residential streets—I guess there are a lot of Comcast customers. I wanted to download the AppLovin data.zip file while walking around, but every time I entered an extended period of no connection, the Google Drive app would restart the entire download. Why??

This also has applications in less developed areas of the world. Google Drive clearly does not handle unstable connections well, so while my project is niche and specific to Google Drive, it solves its problem well, rather than being a solution in search of a problem.

How we built it

The way the script works is that it uses HTTP ranges to download 10 MB chunks at a time. That way, if a chunk fails, there are many checkpoints instead of having to restart the entire process. It is beyond me why the Google Drive app does not do this.

The software is a Node.js script that runs in Termux on my phone. For concatenating the chunks and transferring the file to my laptop, it uses an express.js server that serves the chunks one after another, so it ends up as one file on the laptop. Ironically, I need to turn on my phone's hotspot and have my laptop connect to it for the two devices to connect. The transfer speed is around 5 MB/s.

Because my devices were struggling with the internet in the hacking area, I was moving around outside, limited to using my phone. Because of this, I had Google Jules (powered by Gemini) write the code because it can test and upload code in a VM, rather than being connected to "Cal Hacks 12.0 - EXT" and having to fight hundreds of hotspots. One might think that this means I didn't do anything, but I disagree: Writing code is really a small part of the entire process—it's tedious, sure, but code also has to be reviewed and tested in the field, which currently can't be easily automated with AI, but are still major components of software development. That's what I did on my phone while walking around: encountering bugs, inspecting Jules' code (as if it were my teammate/coworker), providing review feedback, and suggesting fixes and improvements.

Challenges we ran into

Getting a direct download link to the file had an extra step because Google Drive shows a "this file is too big to scan" message, which requires some session token handling.

Besides the wifi, apparently Google Drive also ratelimits downloads of a given file. A file's ratelimit is likely global—the API route is not authenticated, so I likely locked out other participants from downloading AppLovin's data.zip—and seems to count each range request as one download, even if it only downloads a tiny fraction of the entire file. My solution was to make a copy of the zip in my school account's Google Drive, which was grandfathered into having a slightly larger quota (15 GB I think) than new accounts do (2 GB). Therefore, this solution is not sustainable.

Another challenge was that I didn't intend on working alone. I was invited by a friend, but he wasn't accepted, and I checked in late, after all the team formation events.

Accomplishments that we're proud of

I am not proud of this project. It is not my best work. :(

What we learned

It would've been nice if I had scouted out places nearby in case the wifi gets spotty or bathrooms full.

What's next for What's the best way to tell people to turn off their hotspot

This isn't the first time and won't be the last time I'll be downloading a large file on unstable internet, so the script might come in handy in the future.

Built With

Share this project:

Updates