Poster

Authors:

Will Jaekle, David Boles, Emmanuel Peters

cslogins:

wjaekle, dboles, apeter

Introduction

We are trying to use past in game item prices and trade volume to predict market trends in the game _ Old School Runescape _. We hope to show that the in game economy is less complex than its real world counterpart and complex models can maintain predictive capabilities. While our data set is substantially simpler than existing financial data, we also hope to develop a better understanding of what makes a successful financial model. Our intended implementation of direct price prediction is a regression problem; however a portfolio manager might also be solved using ordinal regression.

Related Work

Jiang, Z., Xu, D., & Liang, J. (2017). A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem. ArXiv.org. https://doi.org/10.48550/arXiv.1706.10059 Summary: Using a Deep learning framework (a CNN, RNN and LSTM) for portfolio management in High volatility cryptocurrency markets. Small trading periods and volatility of cryptocurrency is similar to our game market environment and thus lends itself to possibly similar performance. Implementation: https://github.com/ZhengyaoJiang/PGPortfolio

Racca, A., & Magri, L. (2021). Robust Optimization and Validation of Echo State Networks for learning chaotic dynamics. ArXiv.org. https://doi.org/10.48550/arXiv.2103.03174 Summary: Proposed validation strategies when using ESN’s to predict chaotic time series data. Proposed Recycle Validation as well as chaotic versions of pre-existing validation algorithms.

Data

We obtained the prices and trade volumes of each item, aggregated to hour long intervals for the last year. This data was obtained through an API provided by the Runescape Wiki Data Collection Project. Each data point was obtained as a json file which was then paginated. Each resulting table has roughly 8000 observations across ~3700 features. The paginated data is 170MB of gzip-compressed parquet relational files. Trade volume has no missing values, however for intervals where the item in question was not traded, pricing values are missing.

This data is polled from players using the _ Runelite _ 3rd party client which represents about 70% of the player population; however, these trades involve two players, therefore the sampling we have is a larger percentage of all completed trades. Furthermore price fluctuations would trigger unfilled trades- increasing trade volume, which increases the likelihood the price fluctuation is represented in our data.

Methodology

Preprocessing and Feature Engineering:

The Data is created such that missing values in the volume data represent a zero in reality, and these will be inserted accordingly. Missing values in pricing are caused by a lack of trading during the given time window and are somewhat more challenging to handle, however linear interpolation would be a reasonable approach to eliminating these. There are many items which were introduced in the last year, these will need to be dropped during training.

Feature Engineering

We are interested in engineering summary metrics at each time step to provide the model. Market volatility and market valuation/throughput are both reasonable to consider. Additionally using clustering we could create summary features representing the overall behavior of specific groups of items. Lastly, we will add a feature representing day of the week and a feature representing hour of day to aid in representation of daily and weekly cycles that affect the market.

Model:

Our methodology is subject to change, however we are looking towards Echo State Networks as a potential architecture. ESNs have shown promise when applied to chaotic time series data, however their interpretability is limited, so other approaches, such as attention may be interesting to pursue as well. If feasible our goal would be to predict the entire economy state n time steps ahead, however if this is unwieldy the target variables could be reduced by clustering products with similar trends.

Metrics

There are a number of benchmarks we can compare our model to, we seek to compare our model to the baseline prediction of no change in values, we can adjust this prediction to represent the overall market trend as well, which will be slightly more accurate. And of course we can compare this to the idealized case of a perfect prediction. In these applications we would use MSE, however this would undervalue high volume items with lower prices-so the loss, at each step, must be normalized by the cost of each individual item. This issue might also be avoided by normalizing item prices in the data pre-processing step. By using the benchmark models above we can see if our model is correctly identifying global trends, and item specific trends. Our base goal is to simply outperform the MSE of the baseline model, which predicts no change, with statistically significant consistency. For a real-world market this performance would be an exceptional outcome. Our target goal is to outperform the MSE of the baseline model which accounts for overall market valuation. Our stretch goal involves extending our prediction length beyond a few hours to a usable time frame on which to make investments, as buy and sell times often take this long, our stretch goal would be to outperform the base model of predicting no change on the order of days.

Ethics

Who are the major “stakeholders” in this problem, and what are the consequences of mistakes made by your algorithm?

The stakeholders involved in applying this model to make in game purchases are the model user, players who participate in the market with the intention of profiting, players who use the market as a means of progressing in the game - not treating items as investments (which would be nearly the entire player base), The company which maintains the game and is financially invested in the health of the in-game economy, Jagex. If the model were to become open source the Runescape economy would evolve quickly and irreversibly. Players might try to emulate the results of the model, and if successful, a subset would attempt to automate the trades using scripts. This breaks in game rules and the effect of high frequency trading could change market dynamics to increase instability, leading to traditional players spending more on in-game items, potentially slowing their perceived progress. Making this form of botting profitable would also lead to an uptick in interest in botting which could negatively impact the player experience.

Additionally, the Runescape economy is so small that individuals and groups routinely affect item values by noticeable amounts; doing so maliciously has been a problem in the past and is against game rules. A model retrained on data created after it has been deployed could more easily result in price manipulation.

More likely, however, would be a mistake in prediction, where the investor purchases a product and the price drops, resulting in a loss. This is expected to some degree, however in cases where external forces change market value, or an item's price is artificially inflated by a malicious actor, an error could result in the near elimination of the user's portfolio value- this outcome is the most likely consequence.

Why is Deep Learning a good approach to this problem?

This problem revolves around a chaotic time series. Regression has been shown to have limited success with such predictive tasks despite the observations not being independently distributed. In order to best predict the prices of products the vast search space of potential feature interactions needs to be considered, this is only really possible in a complex deep learning network. Additionally the work around echo state networks has been tailored to chaotic time series, which provides an advantage over other methods, such as regression, which make assumptions about the data's distribution.

Division of Labor:

Preprocessing: Will Feature engineering: Emmanuel + Will Initial model architecture: David + Will Model tuning: David + Emmanuel Creation of baseline models: David Research: David + Will + Emmanuel Presentation script: Will Final write up:: David + Emmanuel