Aggregating DB, API & file data for ML pattern detection

Inspiration

Conversations with Peter Nguyen among others sparked interest (excuse the pun) in applying our awesome data integration products to creating a big data set for use with ML analysis... Extending our traditional DCI products' capabilities to access data from databases, REST APIs, SaaS applications and other datasources, with MoveIT's abilities as a secure MFT solution, gives us more complete coverage of the data integration space, allowing us to demonstrate broader range of implementation patterns. The explosion in the number of data scientists and the rise of ML-based analysis, driven by increasing awareness that there really is business value tied up in the massive datasets our customers and partners have lends itself to an ML-based demo use case. This also allows us to focus on using all our data products from an ETL perspective rather than a BI-tool perspective - bring-your-own-ETL rather than bring-your-own-BI - a growing space that our DCI demos have arguably under-served up to now...

What it does

How I built it

Challenges I ran into

Accomplishments that I'm proud of

What I learned

What's next for Aggregating DB, API & file data for ML pattern detection

Built With

• apachespark
• arc
• etl
• hdp
• moveit
• openedge
• pasoe