Co-variance: An intuitive explanation!
source link: https://towardsdatascience.com/co-variance-an-intuitive-explanation-a67f6c80674d?gi=620e6dc933fb
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
A comprehensive but simple guide which focus more on the idea behind the formula rather than the math itself — start building the block with expectation, mean, variance to finally understand the large picture i.e. co-variance
Apr 5 ·8min read
Introduction
Contrary to the popular belief, a formula is much more than just mathematical notations. It tries to express an idea, which get hidden under the math and is not evident unless you really look for it. The main problem with this kind of representation (as it usually happens with me), is that after sometime you tend to forget the formula. So, here is my attempt to explain one topic such that it sticks with the audience. Before diving right into it, I will try to explain some prerequisite topics. If you are already familiar with them, feel free to skip. If not, ride along :)
Expectation and Mean
Let’s start with a relatively easier topic which is the one of the basic blocks required to understand co-variance. In probability theory, expectation represents “the expected value of a discrete random variable , which is the probability-weighted average of all its possible values” and it is formalized as,
the expectation over all values of ‘X’ and their probabilities ‘p_i’Here ‘X’ is the variable which can take several shapes — ‘x_i’, where each have it’s own probability ‘p_i’ of occurrence. Notice expected value is a single number representation of all the values a variable can take considering their probabilities. One special case to remember is when all ‘p_i’ are equal i.e. probability of occurrence of all values are equal. In this case, expected value transforms into mean or average. To give an example, suppose a variable which simulates the rolling of an unbiased dice, so the possible values it can take can be 1 to 6. Also the probability of occurrence of any of these numbers will be equal. Coming back to the generalization, the transformation of expectation to mean is showcased below,
Notice as equal weights are given to all of the values of the variable, the mean is proportional to the values itself, hence it tends to incline towards denser concentration of points. See below a simulation of distribution of points (blue dots) and how change in their position leads to change in the mean (red dot) itself.
Changing points and chasing mean!Also be vary of practical simulations, as most of the time they differ from the theoretical simulations. Consider the dice roll example, where we very easily stated that they have equal probability, but a programmed simulation may show some variations. Below, I simulated 10,000 rolls of an unbiased dice. Look at the occurrence distribution of the dice faces.
Now compare the theoretical and practical calculation of mean notice there is a difference, even though small, but in practical scenario this will do.
Variance
Observe the following plots, can you find anything common in them?
Changing points and … wait a sec!?The answer is — all of them have the same mean! But they look so different, right? And what is so different in all of the them? It seems that they have different ‘spread’ or ‘width’. Variance is basically the measure of this spread or width of the data. In statistics, “variance is the expectation of the squared deviation of a random variable from its mean .” Let’s try to fit this definition to our understanding of expectation,
And just like that we have the formula of variance! Notice first we compute the mean of all the values of ‘X’. Then we find the numerator, which is square of the difference of each value with this mean. The square part is required as we don’t care about the direction of spread, hence we don’t want the spread in opposite directions i.e. with different polarity, to cancel out each other. Some may say, if we square to find the numerator, why not later take a square root? And this idea is exactly represented by standard deviation. So in other words, variance is the square of the standard deviation. With this in mind, let’s look at the same plots as before (now separated and static), but now with variance and standard deviation computed.
Now we are ready for the main topic but before that there is one more interesting derivation of variance. This isn’t required to understand co-variance but curious readers may want to see it anyways. It represents an idea that, “variance of a variable is expectation of the squared variable minus the square of the expectation itself”. It is derived below,
While we are at it, let’s compute the variance of the dice roll simulation from before. Also let’s compute the same variance in three different ways, one representing lazy python way and the remaining two representing the formulas we discussed.
Note that the variance is same to a certain decimal value, the small difference there is due to the floating point errors. Also, in Python prefer the way 1, I have coded way 2 and way 3, just to showcase the formulas we discussed.
Co-variance
Till now we have been looking at only one variable at a time i.e. our data was 1D or 1-dimensional. Co-variance is defined for higher dimensional data. So as the name suggests, instead of just one variable, it considers multiple (exactly 2) variables and compute variance. Before going further, let’s discuss the data first. When I say 2D, I mean each instance of data is represented by two numbers. And in basic planar geometry, we know two numbers can be associated with a point, hence each instance is represented by a single point. A sample data with 10 instances and their visualization on a 2D plane is shown below,
Recommend
-
129
Kotlin generic variance modifiersPublished in
-
103
The ultimate guide to understanding generics, covariance, contravariance, and type projections in Kotlin. Learn two simple subtyping rules that explain why generic variance works the way that it does.
-
83
A high bias low-variance introduction to Machine Learning for physicists. This website contains Python notebooks that a...
-
21
An Intuitive Explanation of NeoDTI NeoDTI is a task-specific node embedding learning algorithm for link prediction on heterogeneous networks. Introduction In the previous stories of this series,...
-
28
An Intuitive Explanation of EmbedS Most GDL techniques ignore the semantics of the network. EmbedS aims to represent semantic relations as well. Introduction In the previous stories, we discusse...
-
16
Intuitive explanation of Neural Machine Translation A simple explanation of Sequence to Sequence model for Neural Machine Translation(NMT) What is Neural Machine Translation? Neural machine tr...
-
25
An Intuitive Explanation of Field Aware Factorization Machines From LM to Poly2 to MF to FM to FFM
-
10
An Intuitive (and Short) Explanation of Bayes’ Theorem Bayes’ theorem was the subject of a detailed article. The essay is good, but over 15,000 words long — here’s the condensed v...
-
5
Intuitive Explanation of the Gini Coefficient The Gini coefficient is a popular metric on Kaggle, especially for imbalanced class values. But googli...
-
4
Intuitive Explanation of Arithmetic, Geometric, & Harmonic Mean If you Google for an explainer on the differences and use cases for the arithmetic mean vs geometric mean vs harmonic mean, I feel like everyt...
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK