4 Worksheet 04: Prying open our imperfect model, Part 1
Before proceeding, make sure that you have finished the preceding worksheets.
In this worksheet, we will dig a bit deeper into what rnorm() is actually doing. Why should we do this?
First, I want to make sure that we have a common understanding of some technical terms. The English language and our intuition can fail us, especially in light of the previous worksheet. You should somehow feel that our English vocabulary is limited and it is difficult to achieve a common understanding of things if we don’t agree on technical definitions.
Second, prying open what rnorm() does is an important step toward building in reflection as part of the model-building process. Commentators reflecting on The Global Financial Crisis of 2008 have been blaming the models we use. As you start working on this worksheet and beyond, you should be able to somehow determine the virtues and vices of their reflections.
4.1 What does rnorm() actually do?
As we have used it, the command rnorm() had three inputs: n, mean, and sd. n is the number of simulated draws. The mean and sd are usually called parameters. One way to think about parameters is that these are the constants needed to be able to produce one simulated draw.
You must understand how the simulated draws were obtained. The simulated draws were obtained via random sampling for a normal distribution. Another way to say this is that we have independent and identically distributed (IID) draws from the normal distribution.
4.2 The definition of independence in a special case
I will use the definitions of independence found in Wooldridge (2020). I will take for granted that you have encountered the following concepts, but I link some old slides in case you need a refresher. Of course, you can also use Appendix B of Wooldridge (2020).
- Random variables and their distributions (slides 1 to 8 here or slides 1 to 8 here)
- Discrete random variables (slides 12 to 13 here)
- Continuous random variables (slides 64 to 65 here)
Roughly speaking, independence is about the “absence of a relationship” between two or more random variables. Making precise the meaning of “absence of a relationship” is the difficult part. We are talking about relationships among random variables. In a sense, the relationship is subject to “chance fluctuations”. But to start with a definition requires us to think about the distribution of two or more random variables.
For now, we take the simplest case of two discrete random variables. We need a concept to describing the distribution of both random variables at the same time, if we are to think about the “absence of a relationship”.
The joint probability mass function of two discrete random variables \(X\) and \(Y\) is given by \[f_{X,Y}\left(x,y\right)=\mathbb{P}\left(X=x\, \mathsf{and}\, Y=y\right),\] where \(x\) and \(y\) are given numbers. What is being encoded is information about joint probabilities, i.e. how likely is it that both \(X=x\) and \(Y=y\) happen at the same time. Make sure to read \(X=x\) as \(X\) taking on a value equal to \(x\).
Two discrete random variables \(X\) and \(Y\) are independent whenever \[\mathbb{P}\left(X=x\, \mathsf{and}\, Y=y\right)=\mathbb{P}\left(X=x\right)\mathbb{P}\left(Y=y\right). \tag{4.1}\] The right hand side of the equation involve marginal distributions. These marginal distributions encode probabilities from one of the random variables only.
To obtain marginal probabilities from the joint probabilities, you have to observe that when \(X=x\), for example, \(Y\) could be taking certain values. So we need to match \(x\) with any other values that \(Y\) could take. These become the possible cases when \(X=x\). Since these cases are disjoint or mutually exclusive, we must have \[\mathbb{P}\left(X=x\right)=\sum_y \mathbb{P}\left(X=x\, \mathsf{and}\, Y=y\right).\] Similarly, we also have \[\mathbb{P}\left(Y=y\right)=\sum_x \mathbb{P}\left(X=x\, \mathsf{and}\, Y=y\right).\] Therefore, marginal probabilities involving summing up joint probabilities in a particular way.
In order to know how likely \(X=x\) and \(Y=y\) happen together, we only need two pieces of information: How likely is \(X=x\)? How likely is \(Y=y\)? This is one way to think about “absence of a relationship”. The possible values of \(X\) and \(Y\) can be related numerically, but that has nothing to do with independence. Instead, independence places restrictions on the probabilities of events!
To check if two discrete random variables \(X\) and \(Y\) are independent, then you have to show Equation 4.1 holds for all values \(x\) and \(y\). If it happens that Equation 4.1 holds for all values \(x\) and \(y\), then \(X\) and \(Y\) are independent.
To check if \(X\) and \(Y\) are not independent, you just have to find a pair of values \(x\) and \(y\) such that Equation 4.1 does not hold.
Feel free to refer to the old slides I have related to the topics so far:
4.3 Exercise A
Consider the following joint distribution of \(X\) and \(Y\). Here \(X\) and \(Y\) both take two possible values. The set of possible values for both random variables is \(0\), \(1\). To read the entries of the table, start from \(\mathbb{P}\left(X=0\, \mathsf{and}\, Y=0\right)=0.24\), and continue further.
| \(Y\) | ||||
|---|---|---|---|---|
| \(0\) | \(1\) | |||
| \(X\) | \(0\) | \(0.24\) | \(0.16\) | \(0.4\) |
| \(1\) | \(0.36\) | \(0.24\) | \(0.6\) | |
| \(0.6\) | \(0.4\) |
- Verify whether the marginal probabilities are correct.
- Verify that \(X\) and \(Y\) are independent.
4.4 Equivalent definition of independence in a special case
We can also frame independence in terms of how additional information regarding a random variable, whether actually observed or hypothetical, affects the probability distribution of another random variable. Intuitively, we are framing “absence of a relationship” in terms of how information about one random variable affects how likely certain events involving another random variable would be.
This means we have to introduce a concept that takes into account the arrival of information whether observed or hypothetical. This leads to the concept of conditional probability.
The conditional probability that \(Y=y\) given that \(X=x\) is given by \[\mathbb{P}\left(Y=y | X=x\right)=\frac{\mathbb{P}\left(X=x\, \mathsf{and}\, Y=y\right)}{\mathbb{P}\left(X=x\right)},\] provided that \(\mathbb{P}\left(X=x\right)\neq 0\).
If, in addition, you assume independence of \(X\) and \(Y\), then we can write
\[\mathbb{P}\left(Y=y | X=x\right)=\frac{\mathbb{P}\left(X=x\, \mathsf{and}\, Y=y\right)}{\mathbb{P}\left(X=x\right)}=\frac{\mathbb{P}\left(X=x\right)\mathbb{P}\left(Y=y\right)}{\mathbb{P}\left(X=x\right)}=\mathbb{P}\left(Y=y\right),\] for any \(y\) and for any \(x\) such that \(\mathbb{P}\left(X=x\right)\neq 0\).
Therefore, a consequence of independence is that for any \(y\) and for any \(x\) such that \(\mathbb{P}\left(X=x\right)\neq 0\), we have \[\mathbb{P}\left(Y=y | X=x\right)=\mathbb{P}\left(Y=y\right).\]
The intuition here is that independence of \(X\) and \(Y\) has the following consequence: knowing that \(X=x\) (we observed it or hypothetically we are thinking about it) did not alter probabilities of events involving the random variable \(Y\). In that sense, there is an “absence of a relationship” between \(X\) and \(Y\).
4.5 Exercise B
Let \(x\) and \(y\) be any given values. Show that if \(X\) and \(Y\) are independent discrete random variables, then \[\mathbb{P}\left(X=x | Y=y\right)=\mathbb{P}\left(X=x\right),\] provided that \(\mathbb{P}\left(Y=y\right)\neq 0\).
4.6 Exercise C
Return to the joint distribution in Exercise A of this worksheet.
- Use the equivalent definition of independence to determine whether or not \(X\) and \(Y\) are independent.
- Draw a picture of the conditional probability distributions \(Y|X=1\) and the marginal probability distribution of \(Y\). What do you notice?
- Draw a picture of the conditional probability distributions \(Y|X=0\) and the marginal probability distribution of \(Y\). What do you notice? Try to articulate how knowing that \(X=0\) alters how likely the possible values of \(Y\) could occur.
4.7 The meaning of identical distribution
The technical definition of identical distribution can be found in slide 14 here. Take a moment to understand the requirement. \(X\) and \(Y\) having the same distribution does not mean literally that \(X\) has the same value of \(Y\). The requirement is about the equality of probabilities of events.
For the discrete case, it is the set of possible values that \(X\) and \(Y\) can take should be equal. In addition, the probabilities at a particular value based on the distribution of \(X\) should be the same as the probabilities at that same value based on the distribution of \(Y\).
4.8 Exercise D
Consider the following joint distribution of \(X\) and \(Y\). Here \(X\) and \(Y\) both take three possible values. The set of possible values is \(-1\), \(0\), \(1\). To read the entries of the table, start from \(\mathbb{P}\left(X=-1\, \mathsf{and}\, Y=-1\right)=0.1\), and continue further.
| \(Y\) | ||||
|---|---|---|---|---|
| \(-1\) | \(0\) | \(1\) | ||
| \(-1\) | \(0.1\) | \(0.2\) | \(0\) | |
| \(X\) | \(0\) | \(0.2\) | \(0.3\) | \(0\) |
| \(1\) | \(0\) | \(0.1\) | \(0.1\) |
- Are \(X\) and \(Y\) independent?
- Do \(X\) and \(Y\) have identical distributions?
4.9 Exercise E
Below you have the following joint distribution of \(X\) and \(Y\). The difference this time is that you do not know the joint probabilities. You only know two pieces of information. The marginal distribution of \(X\) is given. In addition, \(X\) and \(Y\) are IID random variables.
| \(Y\) | |||||
|---|---|---|---|---|---|
| \(-1\) | \(0\) | \(1\) | |||
| \(-1\) | ? | ? | ? | \(0.1\) | |
| \(X\) | \(0\) | ? | ? | ? | \(0.3\) |
| \(1\) | ? | ? | ? | \(0.6\) | |
| ? | ? | ? |
- To make sure we are on the same page, what is the marginal distribution of \(X\)?
- What should be the entries to replace the entries marked “?” so that \(X\) and \(Y\) are independent and identically distributed random variables?
4.10 Exercise F
You can do this exercise without the help of generative AI first. Afterwards, ask generative AI for their responses to the exercise. Document your prompts and validate the responses from your end. After that, compare and discuss your findings with a classmate.
- Create a joint distribution of two random variables which are independent but not identically distributed.
- Create a joint distribution of two random variables which are not independent but identically distributed.
- Create a joint distribution of two random variables which are not independent and not identically distributed.
4.11 Connection to stock returns
You might wonder why we spend so much time on fleshing out what IID means. One reason is that we will be looking at departures from IID or violations of IID.
Perhaps the most pressing reason is for you to think of the toy example in Exercises D or E as the joint distribution of returns of two stocks or the joint distribution of returns of the same stock at two different points in time. Think of \(-1\) as total loss of your investment in a stock, \(0\) as no profits and no losses, and \(1\) as doubling your investment in a stock. Hopefully, the discussion on joint probabilities and independence will now have some “life”.