Inspiration
Lack of open-ended CSV tools that perform operations based on the parameters set by the user inspired us to create this CLI tool where, from number of inputs to orientation of output is set and defined by the user.
What it does
It takes a number of arguments from the user, depending on which appropriate function/transformation (i.e. filtering, formatting, joining, aggregating) is performed and the transformed CSV is output and stored locally which can further be saved in local Database. Certain commands are dedicated to "help" the user with how a certain transformation command is to be inputted.
How we built it
We used the "typer" package of Python for collecting the input arguments efficiently and giving informative output to the user. For the actual operations, widely renowned "pandas" library was used and to provide the support for various delimiters "Sniffer" of "csv" library was used. "sqlalchemy" is used for connecting to the local mysql database. Try catch blocks were for providing informative error handling.
Challenges we ran into
When certain conditional arguments to be taken from the user, a lot of erroneous situations can occur, so providing error handling for most of those scenarios was very difficult especially when "Date" as an parameter is considered because it is very difficult to separate date from a normal string. Keeping "filtering" operation open-ended towards user was difficult as collecting boolean arguments is as good as parsing a string in programming language as a lot of unnecessary spaces might be there, separating column name from operand etc. made it challenging.
Accomplishments that we're proud of
Successfully completing all the tasks and brownie points mentioned such as handling different formats of csv files, connecting and adding data to database. We didn't limit the functionality of the app to the problem statement, and allowed users to specify more than 2 csv files for joining, multiple columns for aggregating, to building a highly customizable command line tool.
What we learned
Following efficient coding practices to build a user friendly command line tool in python, effective error handling techniques, considering diverse use cases.
What's next for TwinTowers_Web
Packaging the app and enabling the user to call the command line through an entrypoint command instead of running the python file. Also, a common public database can be incorporated with CSVs of robust nature so that hardcore testing can be performed.
Log in or sign up for Devpost to join the conversation.