Backtesting VaR models: Quantitative and Qualitative Tests
Carlos Blanco and Maksim Oks
This is the first article in a two-part series analyzing the accuracy of risk measurement models. In this first article, we will present an overview of backtesting methods and point out the importance of conducting regular backtests on the risk models being used. In the second article, we will present an alternative to measuring VaR using a top-down or “macro” approach as a complementary tool to traditional risk methodologies.
Should risk models be accurate?
Firms that use VaR as a risk disclosure or risk management tool are facing growing pressure from internal and external parties such as senior ...view middle of the document...
The traditional excuse given by many risk manages that “VaR models only measure risk in normal market conditions” or “VaR models make too many wrong assumptions about market or portfolio behavior” or “VaR models are useless” should no longer be taken seriously, and risk managers should be accountable to implement the best possible framework to measure risk, even if it involves introducing subjective judgment into the risk calculations. It is always better to be approximately right than exactly wrong.
Determining the accuracy of VaR models
How can we assess the accuracy and performance of a VaR model? To answer this question, we first need to define what we mean by “accuracy.” By accuracy, we could mean:
- How well does the model measure a particular percentile of or the entire profit-and-loss distribution?
- How well does the model predict the size and frequency of losses?
Many standard backtests of VaR models compare the actual portfolio losses for a given horizon vs. the estimated VaR numbers. In its simplest form, the backtesting procedure consists of calculating the number or percentage of times that the actual portfolio returns fall outside the VaR estimate, and comparing that number to the confidence level used. For example, if the confidence level were 95%, we would expect portfolio returns to exceed the VaR numbers on about 5% of the days.
Backtesting can be as much an art as a science. It is important to incorporate rigorous statistical tests with other visual and qualitative ones.
Simple Backtesting: VaR estimates vs. P&L -20%-15%-10%-5%0%5%10%15%10/07/9301/28/9405/23/9409/13/9401/04/9504/27/9508/18/9512/11/9504/02/9607/24/9611/14/9603/07/9706/30/9710/21/9702/11/9806/04/9809/25/9801/18/9905/11/9909/01/9912/23/9904/14/0008/07/0011/28/0003/21/0107/12/0111/02/0102/25/0206/18/0210/09/02VaR (-)ETLVaR (+)ETGP&L
The simplest backtest consist of counting the number of exceptions (losses larger than estimated VaR) for a given period and comparing to the expected number for the chosen confidence interval.
A more rigorous way to perform the backtesting analysis is to determine the accuracy of the model predicting both the frequency and the size of expected losses. Backtesting Expected Tail Loss (ETL) or Expected Tail Gain (ETG) numbers can provide an indication of how well the model captures the size of the expected loss (gain) beyond VaR, and therefore can enhance the quality of the backtesting procedure.
* If we do not have ETL, VaR(+) and ETG data, we can perform the analysis with VaR data exclusively, but we would have limited information to extract conclusions.
Statistical tests help us check whether the risk model is accurately capturing the frequency, independence or magnitude of exceptions, which are defined as losses (gains) exceeding the VaR estimate for that period.
When we test a certain hypothesis in statistics, we can make two types of errors: Type I errors occur when we reject the model which is...