J and J

Big data meets Google Cloud Computing meets RNN Neural Nets

Comment

Inspiration

We wanted to expand our big data analytics skill using Finra's large dataset.

What it does

It displays court cases and SEC reports in a semantic layout Allows for searching of court cases and significant people in SEC reports Real-time natural language parsing using Google Cloud Services to analyze court information Compiled metadata from entity analysis is fed into a neural net, which attempts to map connections and detect patterns between the two data sets.

How I built it

Recurrent neural net from TensorFlow. Entity analysis was done using Google Cloud Services. We converted XML data into JSON, to make all data sets consistent and easy to feed into our nodejs app. Write a dynamic angular framework on a single web page that responds to a mix of sampled data from a set that is processed live and a set that has been pre-processed.

Challenges I ran into

Some data were in XML and some were in JSON. Getting 70gbs of data onto a machine with adequate processing power. XML data was not consistently formatted Neural Net training ran slowly

Accomplishments that I'm proud of

Successfully set up a Natural Language Parser. Dynamic design of the website. Efficient and fast processing of data.

What I learned

First time using Google Cloud Platform. Very cool. Improved on web design and layout skills.

What's next for J and J

Unsupervised neural net would be very interesting. It would be able to detect patterns that we otherwise wouldn't notice or think of checking.

Built With

Share this project:

Updates