Goliath

Our Package Icon

Inspiration

We thought that it was weird that Python had no user friendly way of using thread-pools, so we thought that we should implement this functionality that seems very useful to us.

What it does

When handling large sets of data, the thread-pool model can often do wonders for parallelizing and thus speeding up a program. However, Python's native support for concurrency is more like polling than threading; it doesn't properly take advantage of multiple CPU cores. This can be frustrating when working in Python with a task that would be easily threadable in other languages.

Goliath enables Python to distribute a set of function calls over a set of servers. This essentially simulates the thread-pool model as a pool of servers, each maintaining a pool of Python worker processes. Additionally, these servers can be reached over the Internet, enabling a many-to-many relationship between clients requesting work and servers providing workers; each client can have work distributed across multiple servers, and each server can handle work from multiple clients.

Goliath abstracts this entire model and aggregates the results from the servers, finally returning the list of results to the coder.

How we built it

We built this using the asyncio library which let us set up thread pools and let us manage them asynchronously

Challenges we ran into

Creating the package had its challenges such as figuring out how to make it run the code we have provided.

Accomplishments that we're proud of

-Developing a custom communication protocol between the client and remote servers.

What we learned

We learned a lot about networking as well as asynchronous communication.

What's next for Goliath

Next, we want to make our package be more efficient with its resources, so it reuses its workers and it separates dynamic from static variables.

Built With

asyncio
python

Submitted to

HackIllinois2020

Created by

I worked on setting up the code into a package on pip, I helped handle exceptions, I created the graphic for our package and I helped test the functionality

ilievartic Vartic
I worked on the interface between programmer and the background networking operations, which in turn would poll remote servers to determine how to distribute tasks. I've rarely worked on low-level networking projects, so I learned a lot

manswami23 Swaminathan
I wrote a lot of the Lieutenant and Commander classes, helped set up pip package building, and defined lots of the custom schema/syntax.

Logan Pulley
I worked on testing, debugging, and assisted in adding experimental features.

Deepan Venkatesh
ZacharyOldham Zachary Oldham