The editorial board of QuantStart wrote a
material about what you should know when you start developing your own system for testing trading strategies. We discussed some of the issues raised in the article earlier on the blog, so this time we prepared an adapted retelling of theses on what problems developers face, what is the difference between back-testers of different types, and what are their pros and cons.
What is a backtester
A backtest is the application of the rules of a trading strategy to a set of historical data on the prices of financial instruments. The essence of the approach is that if we develop a mechanism for determining the moment for entry and exit from a position (buy / sell), for example, stocks from a certain portfolio, and apply the resulting rules to historical data, this will give an idea of the effectiveness of the trading strategy “in the past ".
One day, someone said that "all models are wrong, but some are useful." This phrase is great for backtesting. Systems for historical testing of financial strategies help determine whether the existing set of rules should be applied to real trading. If we know how a strategy could lead in the past, it will help filter out bad strategies without the need for real financial losses.
The problem is that the result of backtesting has nothing to do with the results of real trading on the exchange. This is just a model of reality. A model that often contains many assumptions.
There are two main types of backtesters - cyclic (for-loop) and event-driven (event-driven).
When developing such systems, there always arises the need for a compromise between accuracy and implementation complexity. These two types of backtesters represent the whole range of options for such a compromise.
Backtesting challenges
Testing for historical data carries a lot of difficulties. All of them are related to the fact that the whole process is only a simulation of reality. Here are just some of them:
- In-sample testing - a problem occurs when using the same data for training trading models and for their further testing. In this case, the productivity shown is significantly depreciated - the result is achieved on a previously known data system. In reality, the data will often differ significantly from training data. In essence, this is a type of retraining.
- Survivor's error - stock indexes (for example, S & P500) are characterized by the process of listing and delisting, when certain stocks and financial instruments appear or are excluded from them. If these changes are not taken into account during the backtesting, then a strategy that does not take into account the shares of companies that were excluded from the index due to low capitalization can be considered successful. To avoid such problems when running backtests on long time intervals, you need to use data that is not subject to the error of the survivor.
- Errors of forecasts (look-ahead bias) - data from the future may also affect the result of the backtest. For example, consider the case when a linear regression index is calculated at a certain time interval. If this indicator is then used in the same sample, it turns out that data from the future penetrated into it, which means that the resulting efficiency of the strategy is largely depreciated. Event-oriented back-testers help solve this problem.
- Changing the market mode - the parameters of the financial market are not stationary. This means that the processes resulting in the movement of stock prices do not rely on parameters that are constant over time. This fact complicates the generalization of parametrized models (many trading strategies are special cases of such strategies), which leads to the fact that the effectiveness of the strategy on historical data is much better than with real trading.
- Transaction costs — Many cyclical back-testers do not take into account even the most basic information about transaction costs, such as various commissions and fees. Often this is the fault of the authors of scientific works, who prefer not to stoop to such trifles. It is very easy to find a strategy that is very profitable in ideal conditions of no cost. The problem is that when trading in real conditions such strategies can be deeply unprofitable. It is extremely important to take into account the spread, market situation, various fees, slippage (in transactions with high-volatility assets, the real price of the transaction may differ slightly from that assumed when placing an application - both in the favorable direction and in the minus).
There are also other issues that are not so often discussed, but nevertheless crucial for creating a quality working back tester. Among them:
- OHLC data is information about the opening price, the highest price of a financial instrument during a trading session, its lowest value and the closing price of the trading period (open-high-low-close chart, OHLC). Usually it is imported from sources like Yahoo Finance. In this case, it can be a combination of different data feeds. This means that it will be difficult to get extreme values (including High and Low prices) in a real-time trading system. This should also be taken into account in the trading model.
- Capacitive limitations - when backtesting there is a great temptation to use an infinite amount of money. In reality, the amount of funds available for trading is always limited (as is the possible amount of borrowed funds for margin trading). It is also important not to forget about the average daily volume limit (Average Daily Volume, ADV), especially for low-liquidity stocks, when there is a high risk that the operations of the trading system will lead to a real price change. Such an impact on the market should also be considered.
- Selection of benchmark - it is necessary to answer the question of whether the benchmark is chosen correctly with which the tested strategy will be compared. For example, if you trade in commodity futures that are neutral to the S & P500 index, is it worth using this index as a benchmark? It is likely that a basket of other commodity funds will be a more correct choice.
- Robustness - if you change the start time of the strategy during the backtest, how much does this affect the result? For long-term strategies, the time to start their work should not seriously affect productivity - it does not matter if the back test started on Monday or Thursday. If it is too sensitive to the initial conditions, it means that there is no possibility to predict its possible productivity at the start of real trading.
- Retraining and variance - we have already touched upon the issues of retraining above, but this is a broader problem inherent in all supervised machine learning methods. This problem can be solved only through careful use of cross-validation techniques. And even in this case, you should be extremely careful when adapting strategies to noise in test data sets.
- Psychological tolerance - psychology is often ignored when creating automated trading systems, because its influence should be minimized by algorithms. However, people remain people, even when they trade not with their hands, but with the help of robots. As a result, this may affect the results. For example, if during a backtest a deposit drawdown of 50% may seem an acceptable risk, in reality the loss of half the value of assets turns out to be a much more traumatic experience. Keeping from unplanned actions in such a situation is not easy.
On this, with the problems of testing everything on history, we now turn to the description of the systems themselves for such tests.
Two kinds of backtesters
First, consider cyclic systems. This is the simplest type of backtester, which is most often described in various blog posts on finance.
Cyclic backtesters
They work like this - the system simply iteratively passes every trading day (or the OHLC bar), performs calculations related to the price of the desired asset (for example, calculates moving average closing prices), and then performs the corresponding operation (entering a long or short position). Further iterations continue. In the process, information about the performance is saved in order to build an equity curve (equity curve).
Here is the pseudo-code of this algorithm:
for each trading bar: do_something_with_prices(); buy_sell_or_hold_something(); next_bar();PythonCopy
As you can see, the system is extremely simple, which makes such back-testers an excellent option for getting the first estimates about the prospects of the trading system.
pros
Cyclic backtester is very easy to implement using almost any programming language, and do it quickly. This is useful when you want to test the effect of many different parameters.
Minuses
The main disadvantage is unrealistic results. Often in cyclic back-testers there is not even basic functionality for accounting for transaction costs, it has to be implemented separately. Also usually used only MARKET-orders.
Also, the ability to reuse code for a test and productive system is minimal, so you have to write it again. This increases the likelihood of software errors.
Cyclic back testers are prone to prediction errors. They should be used solely as a filtering tool to discard obviously unsuccessful strategies. At the same time, it is important to maintain an extremely skeptical attitude towards strategies that have shown good results. It is important to remember that in real life strategies rarely show themselves better than during backtesting.
Event-oriented systems
Systems of this type are on the opposite side of the spectrum. They are much closer to reality. Usually they work in huge while loops, during which in a special “event queue” “events” are constantly searched for, including:
- ticks - the emergence of new market data;
- signal events - the appearance of trading signals;
- order event - an order for a transaction is ready to be sent to the broker system;
- transaction event - receipt from the broker of information on the execution of the order.
When a certain event is recognized, it is transmitted to the appropriate module in the infrastructure of the trading system for further processing and potentially generates new events that again fall into the queue.
The pseudocode of such a backtester looks like this:
while event_queue_isnt_empty(): event = get_latest_event_from_queue(); if event.type == "tick": strategy.calculate_trading_signals(event); else if event.type == "signal": portfolio.handle_signal(event); else if event.type == "order": portfolio.handle_order(event); else if event.type == "fill": portfolio.handle_fill(event) sleep(600);
As you can see, the system is extremely dependent on the processing module of the portfolio - this is the real heart in event-oriented systems.
pros
This type of back tester has many advantages:
- Reducing the likelihood of forecasting errors - thanks to a structure that involves the phased transfer of messages, forecasting errors are less common in event-oriented systems, at least at the trade level. However, the probability of their occurrence is still preserved, since errors may be contained in the model itself.
- The possibility of code reuse - to use the strategy in real trading, you only need to replace the data processing module and the request execution engine. The description of the strategy, the module of risk management and position management, the code for assessing the productivity of the system - all this can be used without changes. This reduces the likelihood of new bugs.
- Portfolio Level - Event-oriented strategies are easier to think at the portfolio level. As a result, this facilitates changes in strategy and the development of hedging methods.
- “Correct” risk management - in such systems it is easier to “modularize” risk management. A trader can easily use techniques like the Kelly Criterion, as well as include various alerts, set limits (for example, on volatility).
- Remote deployment and monitoring - the modular principle of writing code simplifies its deployment in the cloud or in a collocation scheme.
Minuses
The advantages of event-oriented systems are understandable, but there are also certain disadvantages. Among them:
- Complex code - development of a fully test-covered system will take weeks and months of full-time operation.
- Object orientation - a modular system design requires an object-oriented approach to programming. So, we need a language that supports these principles. Unit testing will not make it any easier.
- High entry threshold - a beginner in programming will fail to create such a system. It will take quite a significant engineering experience that will allow you to understand how to write code, implement logging, conduct unit tests, implement version control and continuous integration practices.
- Low speed - an approach in which messages within the system are sequentially transmitted within its various levels, slows down the speed of execution of requests compared to the vectorized cyclical approach. Calculating various combinations of parameters can be time consuming.
Other materials on finance and stock market from ITI Capital :