Crumbs - crux of the solution
There isn't a single unified experience to share files or content across the various online storage providers. Everyone has their own choice of the app and collaboration for productivity is painful considering each of the providers have their own unique way of dealing with sharing and consuming content. There's also the problem with data privacy and the impending question behind everyone's mind as to who else is able to see your data. It was with these two objectives, that we starting building CRUMBS.
What it does
CRUMBS lets you build groups with your friends for sharing content and data using the existing stroage infrastructure. We're inventory light as it leverages the existing infrastructure already provided by them and help you link their accounts to the CRUMBS app. Once done, a group between 3 friends could be using like Google a Drive, Dropbox or OneDrive by contributing a part of their overall storage (lets say 1 GB each of total 3 GB). It's more like Bring Your Own Storage (BYOS) - another lingo. Anyways, CRUMBS lets you seamlessly share the content between them by abstracting the storage layer and the files that are uploaded are split, encrypted and distributed across the linked accounts. So in case, there's a data breach the files are practically useless since it's only a piece of the data. Even we would not be able to have access to the files since it ultimately resides on the cloud with indivdiual storage providers. We store just the meta-data of the file split and upload info.
How we built it
The architecture is just like a traditional 3 tier system - client (front-end), middleware and the server (backend). Front end has been designed with reactjs as of now to give a full fledged experience on bigger form factors like laptops/desktops. Login to the application is done with FB Account login. Now, any further interactions (like creating groups, linking your DropBox, GoogleDrive etc..) at the client end is redirected to the middleware that is built with Nodejs. All the file upload, split process and upload takes place here. The backend is an AWS EC2 instance (Amazon Linux) that's installed with SQLite (for the moment) to maintain the transactional intergrity like account info, file info etc.
Challenges we ran into
We ran into the usual 0 to 1 problem of getting it started from the grounds up since there were a lot of moving parts and multiple integrations. We were confused as to the choice of technology that should be adopted for file splitting and merging. A huge learning curve also happened with the usage of REST APIs provided by GoogleDrive and DropBox for seamless upload and download. A mistake was also done in the earlier phase of moving the entire uploaded file to the EC2 instance and then performing the operations there. This lead to various timeouts happening with the function which is now resolved by performing the file operations at the inidividual client end. Last but not the least, issues existed with having multiple dev environment flavors between us - the classic Linux and Windows problem of compatibility and node module compilation and usage.
Accomplishments that we're proud of
- In about 30 days we were able to get the basic skeleton (UI + backend) and the core idea up and running. The files are now being properly uploaded and merged on download across the linked accounts in a group.
- Keeping the performance in mind, we decided to build our own node modules for the upload and download process and the choice was C++11. There was a lot of problem initially to build the node modules rather than choosing an existing file split and merge module from npmjs. Really good learning and the end our own module which can now be extended from a functionality perspective.
- A good learning and finally a satisfactory outcome of integrating the front-end which now relays the message to the backend via the custom middleware. This has been a great journey of integration.
What we learned
- We learnt how to use the FB account login API and more importantly reactjs for designing solid UI experiences.
- A choice was made on making our own node module rather than the readily available modules. This helped us learn the process of building custom modules which were suited to our needs.
- Walking the whole nine yards in terms of creating a 3 tier system comprising of front-end, middle ware and the backend.
What's next for Crumbs
- Integration of other popular storage providers like Outlook, AWS S3 etc to increase the reach of usage.
- We now store the tokens of the drives in a SQLite database which does not expire in our backend EC2 instance. This is not a secure way and instead it would be better to have this encrypted and kept in a blockchain thereby transferring the ownership to the clients who use the service.
- Improve the UI and UX to make it more user friendly and an addictive app.
- There are now size restrictions (100 MB) kept in the upload process as timeout benchmarking was not properly done. Remove these restrictions to make it a truly unlimited storage engine.