HPCC Database Management

Directed by HPCC, we learned a new language, Enterprise Control Language (ECL). We manipulated 3 music-related datasets containing metadata such as Artist names, Song Names, and release years.

Comment

Inspiration

We were inspired by the challenges provided by HPCC, we wanted to learn more bout a new programming language, ECL, while also developing our skills in managing large datasets.

What it does

Filters, Sorts, Manipulates, and Interprets data provided by HPCC concerning popular music catalogs

How we built it

  • Friday: Attended info sessions Set up ECL development environment for both VS Code and ECL IDE Started splitting up tasks

  • Saturday: Finished splitting up tasks Completed Challenges 1, 2, 3, and 4 Learned ECL through HPCC's documentation Learned and used ECL Watch to view and interpret the manipulated data Attended various workshops and the ESports Competition

  • Sunday: Fixed up some bugs Chose representative Prepared to present our project to the sponsor

Challenges we ran into

  • Narrowing down project ideas and challenges to compete for
  • Setting up the environment for the HPCC Challenge
  • Learning the syntax for TRANSFORM, PROJECT, and TABLE functions
  • Visualizing and combing through data
  • Sorting with unique factors (such as the length or number of words in the title)
  • Editing data to remove duplicates and improperly formatted data

Accomplishments that we're proud of

  • Completing all of the outlined Challenge Parameters
  • Learning and applying a new language, ECL
  • Better understanding of database management and HPCC
  • Managing time pressures to deliver a functional product

What we learned

  • The outline of how High-Performance Computing Clusters worked
  • How to install ECL extensions in VS Code
  • Basic Functionality of ECL (sorting and filtering data)
  • How to use RECORD structures to outline new datasets
  • Creating TRANSFORM functions to transform data from one dataset to another
  • Using PROJECT to apply changes to every item in a dataset
  • Creating tables using RECORD, cross-tab, and TABLE
  • Using TRIM and DUP to clean up data
  • Interactions between local desktop, Thor server, and Roxie to access and alter data
  • Displaying Data on the HPCC playground

Built With

  • ecl
Share this project:

Updates