By Ashutosh Dave and Udisha Alok
Measuring the dispersion in a financial dataset is one of the most important tasks in quantitative finance. Whether it is for signal generation in a trading strategy or for risk management, standard deviation is the most popular volatility measure used in the financial markets. In this blog, we unravel this metric and discuss its various use cases working with real financial data in Python.
We'll cover the following:
- Definition of standard deviation
- What is standard deviation?
- Standard deviation formula
- Examples of standard deviation from real life
- Why use standard deviation?
- Standard deviation in Finance and Trading
Definition of Standard deviation
In statistics,
The standard deviation (σ) is a measure that is used to quantify the amount of variation or dispersion of data from its mean.
In other words, the standard deviation gives us information about the magnitude of the average deviation from the mean of the data. So, if the values in a dataset lie close together, the standard deviation would be small.
On the other hand, if the values are spread out, the standard deviation would be larger.
What is Standard Deviation?
Let's get back to the very basics.
What is meant by deviation from the mean?
Deviation here tells us how far an observation is from the mean. It can be positive as well as negative, based on whether the observation is greater or lesser than the mean respectively.
What if we measure the total deviation by summing all such values i.e., \(x_{1}-\overline{x}, x_{2}-\overline{x}, ..., x_{n}-\overline{x}?\)We’ll end up with zero because the positive and negative values will cancel each other out! What if we take the absolute values and sum them instead? We could. Either that or we could square the differences and aggregate them. The latter is preferred because it has attractive mathematical properties.
So, to repeat, we square the differences to get rid of the positive and negative signs and compute their average. This resultant quantity is called the variance, which captures the dispersion in the data.
Standard deviation is a standardized version of the variance, obtained by taking the positive square root of the variance and hence the name, standard deviation. In the next section, we will discuss the formula to calculate the standard deviation.
Standard deviation formula
The formula for calculating the standard deviation (denoted by σ) is as follows:
\(\sigma = \sqrt\frac{{\sum_{i=1}^{N}(x_{i}-\mu)^2}}{N}\)
Where,
\(x_i\) = value of the \(i^{th}\) point in the data set\(\mu\) = mean of the data points, calculated as \(\frac{1}{N}\sum_{i=1}^{N}x_i\)
N = total number of data points
Examples of standard deviation from real life
The term standard deviation sounds like something you hear in a statistics class, but don’t dismiss it as an overly technical term just yet. It can be used in different aspects of our lives.
A teacher can use the standard deviation of marks of her students in an exam as a metric to assess the overall level of understanding of the subject. If the mean and standard deviation are both high, it indicates that, on an average, students have a good understanding of the subject.
However, there would be many students who have scores that are much above and much below the average scores. In case the mean is high and standard deviation is low, it indicates that the average scores are similar to the previous case.
The low standard deviation tells her that most students have scores that are close (i.e. slightly above and slightly below) to the mean. In weather forecasting, it can be used to compare the weather patterns in two or more regions.
If we compare the standard deviation of temperatures in Jaisalmer (which has extreme weather) with Mumbai (which has moderate weather), we would find that the former has more variability in temperature around the mean.
Why use standard deviation?
Unit of standard deviation
The unit of standard deviation would be the same as the unit of our data. This makes it easier to interpret compared to the variance. In the next section, we do a detailed comparison between these two measures of dispersion.
Standard deviation vs Variance
The variance \((σ^2)\) of a random variable X is given by the formula below:\(Variance = \frac{\sum_{i=1}^{N}(x_i-\mu)^2}{N}\)
As we can see, by its very construction, the variance is in the square of the original unit. This means that if we are dealing with distances in kilometres, the unit of variance would be in square kilometres.
Now, square kilometres may be easy to visualize as a unit, but what about \(year^2\) or \(IQ^2\), if we are working with ages or IQs of a group? They are harder to interpret.Hence, it makes sense to use a measure that can be comparable to the data on the same scale/units, like the standard deviation.
Standard deviation is calculated as the square root of variance. It has the same unit as our data and this makes it easy to use and interpret.
For example, consider a scenario where we are looking at a dataset of the heights of residents of a neighbourhood. Assume that the heights are normally distributed with a mean of 165 cm and standard deviation of 5 cm.
We know that for a normal distribution,
- 68% of the data points fall within one standard deviation,
- 95% within two standard deviations, and
- 99.7% fall within three standard deviations from the mean.
Thus, we can conclude that the height of almost 68% of the residents would lie between one standard deviation from the mean, i.e., between 160 cm (mean – sd) and 170 cm (mean + sd). You can read more about normal distribution here.
Standard deviation for sample data - Bessel's correction
When calculating the standard deviation of a population, we use the formula discussed above. However, we modify it slightly when dealing with a sample instead.
This is because the sample is much smaller compared to the entire population. In order to account for differences in a randomly selected sample and the entire population, we ‘unbias’ the calculation by using '(n-1)' instead of 'n' in the denominator of equation 1. This is referred to as Bessel's correction.
Thus, we use the following formula to calculate the sample standard deviation (s).
\(s = \sqrt\frac{{\sum_{i=1}^{n}(x_{i}-\overline{x})^2}}{n-1}\)
Where,
\(x_i\) = value of the \(i^{th}\) point in the sample\(\overline{x}\) = sample mean
n = total number of data points in the sample
Do note that as the sample size n gets larger, the impact of dividing by '(n-1)' or 'n' will become lesser.
Standard deviation in Finance and Trading
Standard deviation as a measure of volatility
In trading and finance, it is important to quantify the volatility of an asset. An asset’s volatility, unlike its return or price, is an unobserved variable.
Standard deviation has a special significance in risk management and performance analysis as it is often used as a proxy for the volatility of a security. For example, the well-established blue-chip securities have a lower standard deviation in their returns compared to that of small-cap stocks.
On the other hand, assets like cryptocurrency have a higher standard deviation, as their returns vary widely from their mean.
In the next section, we will learn to compute the annualized volatility of stocks in Python.
Computing annualized volatility of stocks using Python
Let us now compute and compare the annualized volatility for two Indian stocks namely, ITC and Reliance. We begin with fetching the end of day close price data using the yfinance library for a period of the last 5 years:
Date Adj Close 2021-10-19 245.949997 2021-10-20 246.600006 2021-10-21 244.699997 2021-10-22 236.600006 2021-10-25 234.350006
Date Adj Close 2021-10-19 2731.850098 2021-10-20 2700.399902 2021-10-21 2622.500000 2021-10-22 2627.399902 2021-10-25 2607.300049
Below, we calculate the daily returns using the pct_change() method and the standard deviation of those returns using the std() method to get the daily volatilities of the two stocks:
Date Adj Close Returns 2016-10-25 511.991608 NaN 2016-10-26 508.709717 -0.006410 2016-10-27 506.127686 -0.005076 2016-10-28 509.144104 0.005960 2016-11-01 507.237701 -0.003744 ... ... ... 2021-10-19 2731.850098 0.008956 2021-10-20 2700.399902 -0.011512 2021-10-21 2622.500000 -0.028848 2021-10-22 2627.399902 0.001868 2021-10-25 2607.300049 -0.007650
Date Adj Close Returns 2016-10-26 508.709717 -0.006410 2016-10-27 506.127686 -0.005076 2016-10-28 509.144104 0.005960 2016-11-01 507.237701 -0.003744 2016-11-02 494.086243 -0.025928
In general, the volatility of assets is quoted in annual terms. So below, we convert the daily volatilities to annual volatilities by multiplying with the square root of 252 (the number of trading days in a year):
The annualized standard deviation of the ITC stock daily returns is: 27.39% The annualized standard deviation of the Reliance stock daily returns is: 31.07%
Now we will compute the standard deviation with Bessel's correction. To do this, we provide a ddof parameter to the Numpy std function. Here, ddof means 'Delta Degrees of Freedom'.
By default, Numpy uses ddof=0 for calculating standard deviation- this is the standard deviation of the population. For calculating the standard deviation of a sample, we give ddof=1, so that in the formula, (n−1) is used as the divisor. Below, we do the same:
The annualized standard deviation of the ITC stock daily returns with Bessel's correction is: 27.39% The annualized standard deviation of the Reliance stock daily returns with Bessel's correction is: 31.07%
Thus, we can observe that, as the sample size is very large, Bessel's correction does not have much impact on the obtained values of standard deviation. In addition, based on the given data, we can say that the Reliance stock is more volatile compared to the ITC stock.
Note: The purpose of this illustration is to show how standard deviation is used in the context of the financial markets, in a highly simplified manner. There are factors such as rolling statistics (outside the scope of this write-up) that should be explored when using these concepts in strategy implementation.
The z-score
Z-score is a metric that tells us how many standard deviations away a particular data point is from the mean. It can be negative or positive. A positive z-score, like 1, indicates that the data point lies one standard deviation above the mean and a negative z-score, like -2, implies that the data point lies two standard deviations below the mean.
In financial terms, when calculating the z-score on the returns of an asset, a higher value of z-score (either positive or negative) means that the return of the security differs significantly from its mean value. So, the z-score tells us how well the data point conforms to the norm.
Usually, if the absolute value of a z score of a data point is very high (say, more than 3), it indicates that the data point is quite different from the other data points.
We use standard deviation to calculate the z-score using the following formula in case we have sample data:
\(z = \frac{x_i - \overline{x}}{s}\)
Where,
\(x_i\) = a single data point
\(\overline{x}\) = the sample mean
s = the sample standard deviation
Below we calculate and plot the z-scores for the ITC stock returns using the above formula in Python:
From the above figure, we observe that around March of 2020, the ITC stock returns had a z-score reaching below -3 several times, indicating that the returns were more than 3 standard deviations below the mean for the given data sample. As we know that this was during the sell-off triggered by the COVID pandemic.
In addition, a standardized measure like the z-score is used widely to generate signals for mean-reverting trading strategies such as pair trading.
You can read more about the pair trading strategy here.
Also, one can use the zscore function from the scipy.stats module to calculate the z-scores as follows:
Date Adj Close Returns Returns_zscore 2021-10-19 2731.850098 0.008956 0.380491 2021-10-20 2700.399902 -0.011512 -0.665617 2021-10-21 2622.500000 -0.028848 -1.551575 2021-10-22 2627.399902 0.001868 0.018247 2021-10-25 2607.300049 -0.007650 -0.468222
Value at Risk
Value at Risk (VaR) is an important financial risk management metric that quantifies the maximum loss that can be realized in a given time with a given level of confidence/probability for a given strategy, portfolio or trading desk.
It can be computed in three ways, one of which is the variance-covariance method. In this method we assume that the returns are normally distributed for the lookback period. Understand how VaR calculation can help enhance your skills in financial risk management.
The idea is simple. We calculate the z-score of the returns of the strategy based on the confidence level we want and then multiply it with the standard deviation to get the VaR. To get the VaR in dollar terms, we can multiply it with the investment in the strategy.
For example, if we want the 95% confidence VaR, we are essentially finding the cut-off point for the worst 5% of the losses from the returns distribution. If we assume that the stock returns are normally distributed, then their z-scores will have a standard normal distribution. So, the cut-off point for the worst 5% returns is -1.64:
Thus the 1-year 95% VaR of a simple strategy of investing in the ITC stock is given by:
VaR = (−1.64) ∗ (s) ∗ investment
where, s is the annualized standard deviation of the ITC stocks.
z_score_cut_off
-1.6448536269514722
VaR = z_score_cut_off * annual_standard_deviation * initial_investment VaR
-45045.34407051503
Thus, we can say that the maximum loss that can be realized in 1 year with 95% confidence is INR 45045. Of course, this was calculated under the assumption that ITC stock returns follow a normal distribution.
Confidence intervals
Another common use case for standard deviation is in computing the confidence intervals.
In general, when we work with data, we assume that the population from which the data has been generated follows a certain distribution and the population parameters for that distribution are not known. These population parameters have to be estimated using the sample.
For example, the mean daily return of the ITC stock is a population parameter, which we try to estimate using the sample mean. This gives us a point estimate. However, financial market forecasts are probabilistic, and hence, it would make more sense to work with an interval estimate rather than a point estimate.
A confidence interval gives a probable estimated range within which the value of population parameter may lie. Assuming the data to be normally distributed, we can use the empirical rule to describe the percentage of data that falls within 1, 2, and 3 standard deviations from the mean.
- About 68% of the values lie between -1 and +1 standard deviation from the mean.
- About 95% of the values lie within two standard deviations from the mean.
- About 99.7% of the values lie within three standard deviations from the mean.
The 95% confidence interval of the ITC stock daily returns is: [-0.03,0.03]
Thus, we can say that based on the data we have, the mean daily return of the ITC stock is 95% likely to be a value between -3% and +3% (assuming the ITC stock returns are normally distributed).
Conclusion
In this blog, we have seen how standard deviation captures the dispersion in a given dataset with ease. We saw that interpreting the standard deviation is much more intuitive compared to variance and why it remains the most popular measure of dispersion in the world of quantitative finance.
Furthermore, we discussed some use cases of standard deviation in quant finance and trading such as estimating the unobservable volatility of asset returns, the z-score, computation of VaR for risk management and determining confidence intervals for unknown population parameters.
We hope that you have enjoyed reading this blog and learnt a few new concepts along the way! Till next time, keep learning!
If you are a trader, a programmer, a student or someone looking to pursue and venture into algorithmic trading then our comprehensive 6 month Executive Programme in Algorithmic Trading (EPAT) taught by industry experts, trading practitioners and stalwarts like Dr. E. P. Chan, Dr. Euan Sinclair to name a few - is just the thing for you.
Disclaimer: All investments and trading in the stock market involve risk. Any decision to place trades in the financial markets, including trading in stock or options or other financial instruments is a personal decision that should only be made after thorough research, including a personal risk and financial assessment and the engagement of professional assistance to the extent you believe necessary. The trading strategies or related information mentioned in this article is for informational purposes only.
