This sprint’s primary objective was to analyze data which is to be believed useful for identifying price trend to enhence performance of Orbit. The current orbit generates events with asset-specific data not including trend-related data. The Orbit’s approach seems to be good as it is, but that is not the best.
No asset is free from the big trend. If market is down, an asset’s price is likely to be down. For a short period, the asset can beat the market, but it could not last for a long time. So adding such information(Trend-related data) to Orbit is crucial to increase the winning probability. The trend data we are looking for is to indicate symptom of trend forming at beginning. The most valuable information for our purpose might be the measure of each trend’s strength. Strong trend tend to have higher probability of forming trend(Most of trend forming is likely yo fail). With that information, we can assume how the trend will last.
Data for Trend Identification
- We need a different type of information for trend. The word, trend, is often used in our conversation because Orbit’s core concept is coming from the trend. We all know the meaning and concept of the trend, but it is hard to define the trend technically. The puzzling part is when to identify trend. After a trend is formed, it is easy to detect, but it is tough to tell at the beginning of the trend.
- We have been investigating data we already have one by one through data analysis, understanding data, checking correlation with the trend, etc. This is a long-running process, but we have no other options.
- Since the data engineering for the trend has started, the number of data processing tasks has been increased. Most of the tasks take some time to be done due to complicated data processing. For example, aggregating whole asset data takes few hours in 32-thread server machine.
- Without the data, we can not do data engineering. And furthermore, data processing is a routine job. So we need to handle it in an automated manner; otherwise our productivity will go down very much because we have to deal with tedious jobs.
- We have developed three airflow plugins to make our daily life more comfortable. Airflow will generate data as scheduled without our interventions.