Big Data technologies will become the main tool to manage risk in the financial industry, but because of the technique threshold of big data technology itself, so far it has not been popular within the financial industry.

What it does

Through the integration, analysis of financial big data, Diting Tech aims to improve the efficiency of the financial capital market. Firstly, We boost behavior finance research. Behavior Finance focus on explaining and predict the financial market from the perspective of microcosmic individual behavior and psychological motivation. But it's difficulty to make a comprehensive analysis about the individual behavior as the data is hard to acquire, store and deep analysis. With the help of the full stack spark architecture, we have solved this problem from crawling, storage, analysis, investment strategy research and visualization. Secondly, We help industry get comprehensive scope about company from unstructured text. Obtaining the real-time and comprehensive information about target stock is important in Finance. But this real time information not only is hard to obtain but also is buried in unstructured text. With help of the full stack spark architecture, we design a system which can crawl and analyze massive news from media website, social media and forums in a real-time, comprehensive manner. Finally, We introduce the Big Data technology built on Spark, into the financial market, and show the power of big data in financial risk management by developing investment strategy based on big data analysis, which offer a new version for the financial practitioners.

How we built it

Technical Framework is built on Spark. In our project, We use Spark to do crawling, cleaning, and computing task. Our Architecture can be divided into three layers.

  1. The The bottom Architecture is responsible for data crawling storage and ETF task.
  2. The middle Architecture is responsible for extracting important statistics and global variables. It includes statistics computing, graph computing(like PageRank,Important Nodes),sentiment analysis, knowledge graph construction and other machine learning task. In this part, we mainly do this for the next part. The next part is about how we design investment strategy, but it may relay on some variables, which can be extracted from original big data. This extractor is just the Middle part.Also, after Middle part, the scale of data can be reduced a very small level, which can be downloaded by our finance analysts and they can analyze them in their PC. This extractor part improve our system portability.
  3. The top Architecture is responsible for how we use the statistics and variables from the middle part to construct investment strategy. This is our final goal. We will introduce this part in detail in part 4.

The following figure gives a propect about how our system is build and run.

The Technical Framework

Industry Reply

  1. Fullgoal Fund Management Co., Ltd. was established in 1999, it is one of he first 10 fund management companies approved by the China Securities Regulatory Commission . Its current assets under management reached 200 billion yuan.
        “Based on traditional financial model,This industry has entered the intense competition, and tapping new signal becomes a new source of profit. We worked with Baidu before, but they only provide a click data. I personally think that analyzing millions of online investors is a very good attempt.” from Xu Ruohua, quantitative investment director
  2. KN capital is a pioneer in China's quantitative trading, in February 2013 it issued China's first fully automated sunshine private fund and so far KN has issued 20 products. The funds under management reached 10 billion.
         "I feel this is very interesting, personally feel that your service company of this product should be raised funds and private equity funds. if the product is good. it is very helpful for institutions like ours. "
    From Xu Xu, KN capital Executive Director

What's next for Diting Tech

More data, More refined financial risk analysis.

Share this project: