We work at a fund management company where we manage funds of large institutes, there we have a proprietary algorithm, a experienced team and beefy hardware. These things are worth dedicating for financial institutes who are willing to put lot of money but for individual investors who are interested in investing in the retail platform of lending club it is difficult so to solve that problem we built lending ninja.
What it does
It tracks and analyzes the performance of loans being posted on a lending platform and help investors design a loan selection strategy that meets their return targets. The robo advisor leverages large amounts of loan characteristics and performance data and uses advanced portfolio analysis and statistical techniques to make recommendations.
How we built it
We used Apache spark hosted on AWS EMR to perform ETL operations on data. ETL operation involved loading data, cleaning up, cumulating the performance, running model to derive score and defaults and calculate lifetime valuation of loans. After ETL we load the data in mariadb column store for to efficiently analyze prehistoric data. We have other modules which doesn't not belong to analytics layer store their data in mongodb. we use python flask to build rest apis and some ready to use ansible roles to deploy the things to aws.
Challenges we ran into
Dealing with Spark and getting the data ready for analytics, the spark jobs took decent time to run, also we were using zeppline notebook which is in beta so sometimes it just got stuck and we had to restart the things. We also had to generate lot of graphs and we were using nvd3 with angular for first time so sometimes the things got little nasty.
Accomplishments that we're proud of
What we learned
Team work, Spark and Angular :P
What's next for LendingNinja
Getting it to market and improving the modeling algorithms and improving the UX.