China Wholesale Food Prices & Supply Forecast

Drivers behind the proposed Cognitive Hackathon

proposed research is significant as most of Hong Kong's basic food commodities (grain, meat, fruits, eggs, diary and vegetables) are China imported. Domestic events there such as sale of large quantities of state reserve grain stocks, governmental agricultural policy directives (land use conversion) adverse weather inadvertently leads to volatile wholesale food & commodity price and supply in Hong Kong access to location specific price and supply forecast information for commodities such as perishable crops, along with weather forecasts; affords price dispersion uniformity and removes price uncertainty along the entire agricultural value chain producers gain bargaining power with the ability to sell at markets where they can maximise their profits, supermarkets have greater insights into medium to long term purchase buying window beyond seasonal produce

What it does - Objectives

Development of a price and supply commodity framework to monitor, predict price & supply of basic food commodities at a country (China) and regional level (City). Be able to send subscribers prediction of 3 days advance average wholesale food prices

Scope/extent of the technical works

Discover which indicators (extreme weather, policy changes, consumer sentiment, etc) which may potentially impact the price and supply of food commodity products in China and thus impact Hong Kong wholesale food prices Provision of cloud based distributed databases to store data for real-time analysis Data collection of indicators from official sources and indicators derived from social media data Data Processing: Cleaning of data Data Analysis and Association Rule Mining Prediction Model: feed data into deep learning algorithms, train & test models Data visualisation of analysis with geo-location Documentation Presentation of findings

Project deliverables: prototype that incorporates those technologies

Apache Spark cloud computing with cognitive API for weather analytics
Data Collection: social media sentiment analysis, web crawl daily and weekly price data and stocks from government data sources
Data Processing: Cleaning of data: Social media data categorisation and keywords analysis, indicators that impact wholesale price & supply information
Association Rule Mining: Price Analysis
Prediction Model: Data in CSV format fed into deep learning models, namely Support Vector Regression, - FeedForward Neutral Networks and Echo-state neural networks using IBM SPSS modeller and time series analysis of price data
Data visualisation: Maps/visualisation with daily/weekly wholesale price & supply forecast index at national and regional level
Documentation:Data collection & processing, Data analysis and prediction models, Visualisation of data
Presentation: Map based visualisation of forecasts

Proposed Team (projected) 9 members

Team Leader - The role of the team leader - graduate teaching assistant, Dr. George Ng is Project manager - to organize the project and drive its completion in time, as well as monitoring its progress, defining properties for clustering and help with analysis, price prediction with Recurrent Neural Networks. He is also responsible of making sure every team member implements a fair share of the project

Team Members Assigned Tasks (Tentative)

Team Member: Online information processing and clustering with Spark, parallelizations of ML algorithms Team Member: Price prediction with Time Series Analysis Team Member: Crawling, anomaly detection in online content and price sequences Team Member: Querying Twitter, price prediction with Recurrent Neural Networks Team Member: Organizing documentation, Interacting with APIs on social media, Visualizing results, Designing of web services and interfaces Team Member: Data crunching & preprocessing (e.g. for pdfs), Sentiment extraction from tweets, checking possible links in changes of sentiment to changes in price sequences. Team Member: Data gathering, crowdsourcing possibilities, association rule learning Team Member: Research of relevant data sources, price prediction with Machine learning methods (different types of regression, e.g. SVR)