11 Worksheet 11: The truth about linear regression

Before proceeding, make sure that you have finished the preceding worksheets.

Let us start with Section 4.8. To understand what the technique of linear regression of \(Y\) on \(X\) is doing, it pays to know some important features of the joint distribution of \((X,Y)\).

To facilitate exploration, think of the actions you apply to data. I purposely selected a non-financial dataset to illustrate the main ideas without the baggage of financial data.

Most of what you see in this worksheet is a review of what you learned in your introductory econometrics course. But in case you were lost in that course, now is the time to get it right.

11.1 Labor force participation

This portion is inspired by Chapter 4 of Hendry and Nielsen (2007).

11.1.1 Constructing a custom dataset by shopping

You will be collecting data from IPUMS USA, specifically data from the American Community Survey (ACS). The linked website has a lot of supporting documentation and you are encouraged to look into the details.

Note

If you already have an account, you can skip this note. If not, continue reading.

Create an account at IPUMS. Click on the link to IPUMS earlier to learn more about this data repository. To create an account, click on Register at the top part of the page. After that, click on “Apply for Access”. Fill in the Required details only. For Occupation Category, select Undergraduate Student. For Specific Occupation Title, put in Undergraduate Student.

You will need access to IPUMS later and you will be working on some data analysis tasks with IPUMS soon. Don’t delay the creation of the account as it may take time for the access to be approved.

Once approved, you can now “shop” for your data!

Exercises

Go to website of IPUMS USA. Create your custom data set.
Click on SELECT SAMPLES and make sure that the only box that is checked is ACS 2024. Click on SUBMIT SAMPLE SELECTIONS.
Now, you will be selecting variables. You have the option to choose from variables measured at the household level or at the person level. You can also search directly. Given the question posed, how would you conduct the search?
The bottom line is that we will use variables with the names EMPSTAT and EDUC (why?). Select them by checking the boxes beside these variable names so that they will be “added to cart”.
Once finished, click on VIEW CART. You will notice that there are more variables included than those you have selected. Explore these variables.
Afterwards, click on CREATE DATA EXTRACT.

Congratulations, you have successfully created a data extract from IPUMS! But this is a very large dataset. We have the option to reduce it in size for our purposes.

Under OPTIONS, click on SELECT CASES to reduce the size of the dataset. Check the boxes for EDUC, EMPSTAT, and GQ. Select ” Include only those persons meeting case selection criteria”. For each variable listed, you can choose observations which obey pre-specified criteria. So that we are working on the same dataset, choose only those observations which fulfill the following criteria:

Households under the 1970 definition
Employment status is employed, not employed, or not in the labor force
No missing information about educational attainment

Afterwards, click on SUBMIT. Once you are back to your cart, put a description of your extract and then click on SUBMIT EXTRACT. Some time would have to pass before you actually get the links:

Download the dataset. Expect a file with file extension .dat.gz. This is a compressed file similar to zipped files. You have to decompress this file and it will produce a .dat file.
Open the command files. You have many options and you should choose the R command file. Feel free to explore others at your own leisure. These commands should be at the beginning of your R script to analyze the IPUMS data extract.
View the codebook. Open both the Basic and DDI links to see what they contain. These are the different formats of presenting documentation related to the data extract.
Download the codebook. You must right click DDI and save link as (or save target as) an XML document. If your browser does not have the option to save an XML document, adjust the type of document to All Files. Make sure the file extension is .xml.

To look for data you have downloaded before, click on MY DATA. Requested data files will stay in the system for a maximum of 72 hours.

Exercise

It is your turn to shop for a dataset. Modify your data requests so that you can answer the question ““How does female labor force participation change with educational attainment?”

Keep the resulting dataset for future exercises.

11.1.2 Analysis

Now that you have the dataset, let us proceed with an analysis which could answer the question “How does labor force participation change with educational attainment?”

The IPUMS dataset is going to be loaded differently and we follow the instructions at the IPUMS website.

Important

Here you have to copy and paste code and execute the commands in R/RStudio. Don’t use the browser and make sure that you are not working on other things as the dataset has millions of observations!

Do not upload the dataset into the cloud because the dataset is not yours and can have some private information.

# Based on R command file from IPUMS
library(ipumsr)
ddi <- read_ipums_ddi("usa_00017.xml") # You must change the name of the xml file to your situation 
data <- read_ipums_micro(ddi)

We only have two variables and both of them are categorical in nature. Along with the codebook, explore what you think the commands below are producing.

table(data$EDUC)
table(data$EMPSTAT)

Observe that you have the frequency distributions of each variable separately. In this sense, these tables are not yet in the direction of finding “relationships”.

Observe also that there are quite a lot of categories for each variable. For the sake of reducing the size of the problem a bit and to illustrate how to redefine variable and process data for further use, we will reduce the number of categories.

We reduce the number of categories in EMPSTAT to two: in the labor force or not in the labor force. We also reduce the number of categories in EDUC to three: elementary or no schooling, high school, or college. Pay attention as to how these variables are recoded.

data$EDUCN <- with(data, 1*(EDUC <= 2) + 
  2*(EDUC == 3 | EDUC == 4 | EDUC == 5 | EDUC == 6) + 
  3*(EDUC >= 7 & EDUC <= 11))
data$EMPSTATN <- with(data, 1*(EMPSTAT <= 2) +
  0*(EMPSTAT == 3))

In order to explore the “relationship” between labor force participation and educational attainment, we can summarize the information by looking at a joint frequency distribution. In the context of categorical variables, we can construct a cross-tabulation or a crosstab.

xtab.EMP.ED <- with(data, table(EMPSTATN, EDUCN))
xtab.EMP.ED

We can add margins to the crosstab.

with(data, addmargins(xtab.EMP.ED))

Exercise

Why are they called margins and what information do they have?

We can also express everything in terms of relative frequencies:

ptab.EMP.ED <- with(data, prop.table(xtab.EMP.ED))
ptab.EMP.ED

Let \(X\) be EDUCN and \(Y\) be EMPSTATN. \(Y\) can take values 0 or 1, while \(X\) can take values 1, 2, or 3. The distinct values that \(X\) and \(Y\) can take can be written as \(x_1,\ldots, x_K\) and \(y_1,\ldots, y_J\).

Important

We are using uppercase letters for the variables and the lowercase letters for the values these variables could take. Therefore, \(x_1\) no longer means the first observation of the variable \(x\). It will now represent one (not necessarily the first) possible value the variable \(X\) could take.

Wooldridge (2020) does not make this distinction too strongly, because you are supposed to be aware of this from the context.

Another important point of note moving forward: both \(X\) and \(Y\) here are categorical in nature. What happens when either one of these variables is continuous?

Exercise

What are the values of \(J\) and \(K\) in our example?

Each cell of the crosstab is \(\widehat{f}\left(x,y\right)\), defined as the relative frequency of people who reported \(X=x\) and \(Y=y\). Note that \(x\) and \(y\) are dummy arguments. The joint relative frequencies satisfy: \[\sum_{j=1}^J\sum_{k=1}^K \widehat{f}\left(x_k, y_j\right)=\sum_{k=1}^K\sum_{j=1}^J \widehat{f}\left(x_k, y_j\right)=1.\]

Exercise

Check that this is the case for our example.

We can also compute the marginal relative frequency distributions for both variables. Try to figure out the information in these frequency distributions.

with(data, addmargins(ptab.EMP.ED))

These marginal relative frequency distributions are defined as: \[\begin{eqnarray*} \widehat{f}_X\left(x\right) &=& \sum_{j=1}^J\widehat{f}\left(x, y_j\right) \\ \widehat{f}_Y\left(y\right) &=& \sum_{k=1}^K\widehat{f}\left(x_k, y\right) \end{eqnarray*}\]

Exercise

Check that this is the case for our example.

Our question “How does labor force participation change with educational attainment?” would be hard to answer with the joint and marginal relative frequency distributions. Why? Because the question somehow requires us to consider what would be the labor force participation of workers who have a certain level of educational attainment. For example, we can calculate the labor force participation of workers who have a high school education. But we can do this for all levels of EDUCN. The result is an example of a conditional relative frequency distribution.

ctab.EMP.ED <- with(data, prop.table(xtab.EMP.ED, margin = 2))
ctab.EMP.ED

These conditional relative frequencies are defined as: \[\widehat{f}_{Y|X=x} \left(y|x\right) = \frac{\widehat{f}\left(x,y\right)}{\widehat{f}_X\left(x\right)}.\] What do you notice about how these conditional relative frequencies varying with \(x\)? We can even plot a graph:

plot(c(1,2,3),ctab.EMP.ED[2,])

Exercise

What do you notice about the pattern produced by the graph?
Why did we not execute plot(c(1,2,3),ctab.EMP.ED[2,])?

Note

We could also have defined conditional relative frequencies of educational attainment given labor force participation, specifically: \[\widehat{f}_{X|Y=y} \left(x|y\right) = \frac{\widehat{f}\left(x,y\right)}{\widehat{f}_Y\left(y\right)}.\]

But given our question for this example and the definitions of \(X\) and \(Y\), it makes more sense to consider \(\widehat{f}_{Y|X=x} \left(y|x\right)\) than \(\widehat{f}_{X|Y=y} \left(x|y\right)\).

To get the conditional relative frequencies \(\widehat{f}_{X|Y=y} \left(x|y\right)\), you need

with(data, prop.table(xtab.EMP.ED, margin = 1))

These conditional relative frequencies have the following properties, beyond the definition:

The sum of the conditional relative frequencies for a fixed category is equal to 1, i.e. \[\sum_{j=1}^J \widehat{f}_{Y|X=x} \left(y_j|x\right) = 1.\]
With weights equal to the marginal relative frequencies of the conditioning variable, the weighted average of the conditional relative frequencies is equal to the marginal relative frequency, i.e. \[\sum_{k=1}^K \widehat{f}_{Y|X=x_k} \left(y|x_k\right) \widehat{f}_X\left(x_k\right)=\widehat{f}_Y\left(y\right).\]

Exercise

Check that these properties hold for our example.

We can calculate summaries of these relative frequency distributions:

Sample mean: \[\widehat{\mathbb{E}}\left(Y\right)=\sum_{j=1}^J y_j\widehat{f}_Y\left(y_j\right)\]
Sample variance: \[\widehat{\mathsf{Var}}\left(Y\right)=\sum_{j=1}^J \left(y_j-\widehat{\mathbb{E}}\left(Y\right)\right)^2\widehat{f}_Y\left(y_j\right)\]
Sample covariance: \[\widehat{\mathsf{Cov}}\left(X,Y\right)=\sum_{j=1}^J\sum_{k=1}^K \left(x_k-\widehat{\mathbb{E}}\left(X\right)\right) \left(y_j-\widehat{\mathbb{E}}\left(Y\right)\right)\widehat{f}\left(x_k,y_j\right) \]
Sample conditional mean: \[\widehat{\mathbb{E}}\left(Y|X=x_k\right)=\sum_{j=1}^J y_j\widehat{f}_Y\left(y_j|x_k\right)\]
Sample conditional variance: \[\widehat{\mathsf{Var}}\left(Y|X=x_k\right)=\sum_{j=1}^J \left(y_j-\widehat{\mathbb{E}}\left(Y|X=x_k\right)\right)^2\widehat{f}_{Y|X=x_k}\left(y_j|x_k\right)\]

Exercise

The summaries you have seen, like the sample mean, does not exactly look like what you are used to. Show that \(\widehat{\mathbb{E}}\left(Y\right)\) is the same as \(\overline{Y}\) you might have encountered in statistics before.

In terms of R implementation, we can compute the conditional means and variances of participation gives educational attainment as follows:

with(data, tapply(EMPSTATN, EDUCN, mean))
with(data, tapply(EMPSTATN, EDUCN, var))

In addition, there are connections between the conditional and unconditional (or marginal) summaries. First, observe that \[\begin{eqnarray*}\widehat{\mathbb{E}}\left(Y\right) &=& \sum_{j=1}^J y_j \widehat{f}\left(y_j\right)\\ &=& \sum_{j=1}^J\sum_{k=1}^K y_j\widehat{f}\left(x_k,y_j\right) \\ &=& \sum_{j=1}^J\sum_{k=1}^K y_j\widehat{f}_{Y|X=x_k}\left(y_j|x_k\right)\widehat{f}_X\left(x_k\right) \\ &=& \sum_{k=1}^K\sum_{j=1}^J y_j\widehat{f}_{Y|X=x_k}\left(y_j|x_k\right)\widehat{f}_X\left(x_k\right) \\ &=& \sum_{k=1}^K \widehat{\mathbb{E}}\left(Y|X=x_k\right)\widehat{f}_X\left(x_k\right) \\ &=& \widehat{\mathbb{E}}\left[\widehat{\mathbb{E}}\left(Y|X\right)\right] \end{eqnarray*}\]

As a result, the sample (unconditional or marginal) mean is the sample average of the sample conditional means. Second, it can be shown that \[\widehat{\mathsf{Var}}\left(Y\right) = \widehat{\mathbb{E}}\left(\widehat{\mathsf{Var}}\left(Y|X\right)\right)+ \widehat{\mathsf{Var}}\left(\widehat{\mathbb{E}}\left(Y|X\right)\right).\] This equality links the sample variance to the sum of the sample average of the sample conditional variances and the sample variance of the sample conditional means.

Exercise

There are other ways to present information contained in these conditional relative frequencies. We can consider the odds for labor force participation for different values of EDUC, e.g. for people who attained a high school level, the odds for participation is \[\frac{\widehat{f}_{Y|X=2}\left(1|2\right)}{\widehat{f}_{Y|X=2}\left(0|2\right)}.\] There are other odds you can report.

Can you communicate what information the odds contain?
We can also plot the odds as a function of the level of EDUC. Similarly, we can also plot the logarithm of the odds as a function of the level of EDUC. What information do these plots contain?
Study the lines of R code below and relate them to this exercise:

# Odds for participation
ctab.EMP.ED[2,]/ctab.EMP.ED[1,]
# Plotting the odds
plot(c(1,2,3),ctab.EMP.ED[2,]/ctab.EMP.ED[1,])
# Plotting the log odds
plot(c(1,2,3),log(ctab.EMP.ED[2,]/ctab.EMP.ED[1,]))

11.1.3 Population-level concepts

So far, what we have illustrated are some calculations based on the data that we have. Another way to say this is that we used a sample to calculate a variety of relative frequency distributions and summaries related to these distributions.

From your previous statistics course, you must have discussed that relative frequencies tend to stabilize to probabilities when the sample size is large and under some conditions. As a result, it makes sense to think about the corresponding population counterpart of these relative frequencies.

Another way to motivate the ensuing discussion is that we can imagine that participating in the labor force as a coin toss. What we get to observe in the data are like outcomes of a coin toss. As a result, the language of random variables, specifically a binary or Bernoulli random variable, becomes very useful. A coin toss could either be heads or tails. We use the language of probability to represent these uncertain outcomes.

What is different this time is that the outcomes of a coin toss will vary depending on what value \(X\) will take. In effect, you allow for a coin with different probabilities of obtaining heads or tails depending on the level of educational attainment. Admittedly, what is described here is fictional, but then again we are on the path towards constructing a model.

Note

Focus on Appendix B. Wooldridge (2020) uses the phrase “density” even in the discrete case. To avoid confusion, cross that off and replace it with “mass”, especially if working with the discrete case. Leave “density” alone for the continuous case.

Your task is to see the parallels and to make sure you can distinguish the sample and population concepts. For the moment, skip the parts where independence is being discussed.

Read the first part of Section B-1, the move on to Section B-1a.
Read Section B-2a up to the end of page 688. Specifically pay attention to joint distribution, joint probability mass function, and marginal probability mass function.
Read Section B-2b in page 690. Specifically pay attention to conditional distribution and conditional probability mass function.
Read Section B-3a. Skip the parts which involve continuous distributions.
Read Section B-3b up to Example B.5.
Read Section B-3d up to B-3g.
Read Section B-4a and B-4b in pages 697-698 up to “… as we will see shortly”. Specifically pay attention to the population covariance. Continue with Section B-4c (B.29) for the definition of the population correlation coefficient.
Read Section B-4e in pages 700-702. Specifically pay attention to the conditional expectation and how it can be thought of as a function.
1. When working with the conditional expectation, there are many senses of the idea.
2. One is that \(\mathbb{E}\left(Y|X=x\right)\) is a number for a given \(x\).
3. Another one is to think of \(x\) as a dummy argument. Provided that \(x\) is a feasible and possible value that \(X\) takes, then \(\mathbb{E}\left(Y|X=x\right)\) is a function of \(x\).
4. Since \(x\) is just one of the values that \(X\), we are uncertain as to what value \(\mathbb{E}\left(Y|X=x\right)\) will take. We can write this situation as \(\mathbb{E}\left(Y|X\right)\) being a random variable with its own distribution and will have its own features and summaries.
Read Section B-4g in page 704. Specifically pay attention to conditional variance.

11.2 A first step towards an econometric model

Based on the analysis earlier of the dataset from IPUMS, we come to the conclusion that labor force participation and educational attainment are “related” in the following senses:

The conditional relative frequency distribution of EMPSTATN given EDUCN varies with the value taken by EDUCN.
The conditional relative frequency distribution of EMPSTATN given EDUC is not the same as the marginal relative frequency distribution of EMPSTATN.
The sample conditional mean of EMPSTATN given EDUCN varies with the value taken by EDUCN.
The sample conditional mean of EMPSTATN given EDUCN is not the same as the sample mean of EMPSTATN.
The sample conditional variance of EMPSTATN given EDUCN varies with the value taken by EDUCN.
The sample conditional variance of EMPSTATN given EDUCN is not the same as the sample variance of EMPSTATN.
EMPSTATN and EDUCN have nonzero sample covariance. As a consequence, they have a nonzero sample correlation coefficient.

These observations are a starting point for constructing a model for the relationship between labor force participation and educational attainment. A model can be thought of as a set of statements and assumptions about the joint distribution of random variables at the level of the population.

In contrast, Wooldridge (2020) in his glossary, explicitly defines an econometric model as “an equation relating the dependent variable to a set of explanatory variables and unobserved disturbances, where unknown population parameters determine the ceteris paribus effect of each explanatory variable”. To some extent, this is a good starting definition but what makes this definition unsettling is that it is not obvious where disturbances come from, why these are not observable, how parameters become relevant, and how these parameters represent ceteris paribus effects.

Because of our main question and our exploration of the data, we will be focusing on the conditional distribution of labor force participation given educational attainment. Below you will find different ways of setting up a model for this conditional distribution.

Important

Notice that we are formulating a model for the conditional distribution, or if you wish, the conditional probability mass function, rather than the conditional relative frequency distribution. The latter was used as inspiration for the conditional distribution.

If you have not read the relevant parts of Appendix B, stop and read those first.

One formulation is to treat \(f_{Y|X=x}\left(y|x\right)=\mathbb{P}\left(Y=y|X=x\right)\) as a general or unspecified function of \(x\). For our example, \(x\) could take on three feasible or possible values (or support points), since \(X\) could be 1, 2, or 3. As a result, we have \[\begin{eqnarray*}\mathbb{P}\left(Y=1|X=1\right) &=& p_{1|1} \\ \mathbb{P}\left(Y=1|X=2\right) &=& p_{1|2} \\ \mathbb{P}\left(Y=1|X=3\right) &=& p_{1|3} \end{eqnarray*} \] with \(p_{1|1}\), \(p_{1|2}\), and \(p_{1|3}\) left unknown. The entire function is the parameter.

Exercise

For our example so far, why didn’t we have to say anything about \(\mathbb{P}\left(Y=0|X=x \right)=p_{0,x}\) for \(x=1,2,3\)?
There are restrictions on the values of \(p_{1|1}\), \(p_{1|2}\), and \(p_{1|3}\). What are these?

Another formulation could be to work on the conditional expectation of labor force participation given educational attainment and treat this conditional expectation as a general or unspecified function of \(x\). Since \(Y\) is binary, we can conclude that for all support points \(x\), \[\begin{eqnarray*} \mathbb{E}\left(Y|X=x\right) &=& 0\cdot \mathbb{P}\left(Y=0|X=x\right)+1\cdot \mathbb{P}\left(Y=1|X=x\right) \\ &=&\mathbb{P}\left(Y=1|X=x\right).\end{eqnarray*}\] Therefore, in this special case of \(Y\) being a binary 0/1 variable, the conditional expectation has the same information as our first formulation.

Exercise

We could also have worked on the conditional variance. In the special case of \(Y\) being a 0/1 variable, prove that \[\mathsf{Var}\left(Y|X=x\right)=\mathbb{P}\left(Y=1|X=x \right)\left[1-\mathbb{P}\left(Y=1|X=x \right)\right].\]

We could also make additional assumptions about the form of \(\mathbb{P}\left(Y=1|X=x \right)\) instead of leaving its values unspecified and almost unrestricted. Suppose we write \[\mathbb{E}\left(Y|X=x\right)=\mathbb{P}\left(Y=1|X=x \right)=\beta_0+\beta_1 x. \tag{11.1}\] \(\beta_0\) and \(\beta_1\) are left unknown and these are the parameters. The first equality is always true given the structure of our example. The second equality is an assumption.

Exercise

You will explore what is implied by the assumption on the conditional expectation. Read Section B-4f, specifically Properties CE.1, CE.2, and CE.4.

Review (B.27) of Section B-4b. We will be working on \(\mathbb{E}\left(XY\right)\).
Apply Property CE.4. In the formula for that property, let the role of \(Y\) be played by \(XY\) and \(X\) be played by \(X\). Write down the immediate consequence of applying CE.4 to \(\mathbb{E}\left(XY\right)\).
Next, apply Property CE.2. Write down the immediate consequence. Afterwards, substitute the assumption about \(\mathbb{E}\left(Y|X=x\right)\) and simplify by applying Property E.3 in page 693.
Repeat the previous two items but this time for \(\mathbb{E}\left(Y\right)\).
Compute \(\mathsf{Cov}\left(X,Y\right)\). What do you notice?

Make a conclusion about the implications of imposing the assumption that \(\mathbb{E}\left(Y|X=x\right)=\beta_0+\beta_1 x\). What meanings do \(\beta_0\) and \(\beta_1\) have?

We could also make a different assumption about the form of \(\mathbb{P}\left(Y=1|X=x \right)\) instead of stating that it is equal to \(\beta_0+\beta_1x\). Suppose we write \[\mathbb{E}\left(Y|X=x\right)=\mathbb{P}\left(Y=1|X=x \right)=\frac{\exp\left(\beta_0+\beta_1 x\right)}{1+\exp\left(\beta_0+\beta_1 x\right)}. \tag{11.2}\] \(\beta_0\) and \(\beta_1\) are left unknown and are the parameters. The first equality is always true given the structure of our example. The second equality is again an assumption.

Exercise

You might wonder about the differences between the assumptions regarding \(\mathbb{E}\left(Y|X=x\right)=\mathbb{P}\left(Y=1|X=x \right)\) found in Equation 11.1 and Equation 11.2.

There are supposed to be restrictions on \(\mathbb{P}\left(Y=1|X=x\right)\) given its meaning. Which of the two models you have seen so far obeys these restrictions? Explain.
If Equation 11.1 holds, compute \[\mathbb{P}\left(Y=1|X=x+1 \right)-\mathbb{P}\left(Y=1|X=x \right).\] Would the value of \(x\) change the computed value?
If Equation 11.2 holds, compute both \[\mathbb{P}\left(Y=1|X=x+1 \right)-\mathbb{P}\left(Y=1|X=x \right)\] and \[\log\left[\frac{\mathbb{P}\left(Y=1|X=x \right)}{\mathbb{P}\left(Y=0|X=x \right)}\right].\] How is the latter connected to a previous exercise on odds?
Given the previous item, provide an interpretation of \(\beta_1\). Is it related to \(\mathsf{Cov}\left(X,Y\right)\)?

The model in Equation 11.1 is called a linear probability model . A discussion can also be found in Wooldridge (2020) Section 7-5. The model in Equation 11.2 is called a logit model or a logistic regression model. A discussion can also be found in Wooldridge (2020) Section 17-1.

It probably should not be very surprising that if you return to our dataset and run a linear regression of labor force participation on educational attainment, then we have a procedure to estimate \(\beta_0\) and \(\beta_1\) in the linear probability model.

Exercise

Can you articulate an intuitive reason why you would expect what was described to happen?
Run the regression just described.
Take a stab at providing an interpretation to \(\widehat{\beta}_1\).

If you want to estimate \(\beta_0\) and \(\beta_1\) in Equation 11.2, you need a new command called glm(). This command is a very popular command for estimating coefficients of generalized linear models.

glm(EMPSTATN ~ EDUCN, family = binomial(link = "logit"), data = data)

We have to determine how to interpret the results and how they differ from least squares.

Finally, we have to determine if all the results can actually be used to answer the main question: “How does labor force participation change with educational attainment?”

11.3 Optimal prediction

This section is based on Section 1.3 of Pua (2024) and we provide some answers to the following questions:

Why would the objective function used in least squares be a good idea?
Can the meaning of the regression coefficients extend to outside the sample that we have?
What would be necessary so that we can ably talk about cause and effect?

11.3.1 Setup and core ideas

The setting we consider for the prediction problem is as follows. There is some random variable \(Y\) whose value you want to predict or guess. You may or may not have additional information about another random variable \(X\). In the end, you have to specify a prediction rule or a formula that you can use to make predictions. We write this rule as a function of \(X\), say \(g(X)\).

An example of a prediction rule is an unknown real number \(\beta_0\). In this case, \(g(X)=\beta_0\), so it does not matter what \(X\) is and your prediction is always some constant \(\beta_0\). You make a prediction error or forecast error \(Y-\beta_0\), but this error is also a random variable having a distribution.

Next, you need some criterion to assess whether your prediction is “good”. This criterion is called a loss function. We will focus on a particular loss function called the squared error loss function. It is defined as \(L\left(z\right)=z^{2}\), where \(z\) is just a dummy argument¹. Losses will typically depend on the errors which were made. As a result, losses will have a distribution too.

Because losses are random, we need to specify a non-random criterion which we could minimize. Traditionally, we target expected losses. This may seem arbitrary but many prediction and forecasting contexts typically use expected losses. Form an expected squared loss \(\mathbb{E}\left[\left(Y-\beta_0\right)^{2}\right]\). This expected loss is sometimes called mean squared error (MSE).

11.3.1.1 Best constant prediction

You can show that the population mean \(\mathbb{E}\left(Y\right)\) is the unique solution to the following optimization problem: \[\min_{\beta_0}\mathbb{E}\left[\left(Y-\beta_0\right)^{2}\right] \tag{11.3}\]

We also have a neat connection with the definition of the population variance. \(\mathsf{Var}\left(Y\right)\) becomes the smallest expected loss from using \(\mathbb{E}\left(Y\right)\) as your prediction for \(Y\).

Exercise

Do you see any similarities with what you have seen before?
How about the connections and parallels to simple linear regression as a summary?
What are the the differences?
What statistical procedure do you apply to data which can be justified by best prediction?

11.3.1.2 Best linear prediction

What if we want to incorporate additional information from \(X\) in our prediction rule? We have to change the specification of our prediction rule. Earlier, we considered \(g(X)=\beta_0\) as a prediction rule. We now specify \(g(X)=\beta_0+\beta_1 X\) as a prediction rule.

We repeat the process of calculating the MSE but for a new specification of the prediction rule. Under an MSE criterion, the minimization problem is now \[\min_{\beta_{0},\beta_{1}}\mathbb{E}\left[\left(Y-\beta_{0}-\beta_{1}X\right)^{2}\right], \tag{11.4}\] with optimal coefficients \(\beta_{0}^{*}\) and \(\beta_{1}^{*}\) satisfying the first-order conditions: \[\mathbb{E}\left(Y-\beta_{0}^{*}-\beta_{1}^{*}X\right) = 0, \ \ \mathbb{E}\left[X\left(Y-\beta_{0}^{*}-\beta_{1}^{*}X\right)\right] = 0\]

As a result, we have \[\beta_{0}^{*} = \mathbb{E}\left(Y\right)-\beta_{1}^{*}\mathbb{E}\left(X\right),\qquad\beta_{1}^{*}=\dfrac{\mathbb{E}\left(XY\right)-\mathbb{E}\left(Y\right)\mathbb{E}\left(X\right)}{\mathbb{E}\left(X^2\right)-\mathbb{E}\left(X\right)^2}.\]

In the end, we can rewrite as \[\beta_{0}^{*} = \mathbb{E}\left(Y\right)-\beta_{1}^{*}\mathbb{E}\left(X\right),\qquad\beta_{1}^{*}=\dfrac{\mathsf{Cov}\left(X,Y\right)}{\mathsf{Var}\left(X\right)}. \tag{11.5}\]

Exercise

Take note of how we computed linear regression in lm().

Do you see any similarities with what you have seen before?
How about the connections and parallels to simple linear regression as a summary?
What are the the differences?
What statistical procedure do you apply to data which can be justified by best prediction?

11.3.1.3 Best prediction

What if we want to incorporate additional information from \(X\) in our prediction rule, but this time we want to leave \(g(X)\) unspecified?

Under an MSE criterion, the minimization problem is now \[\min_{g(X)}\mathbb{E}\left[\left(Y-g(X)\right)^{2}\right].\] We now impose a restriction to make some progress, without introducing too many new mathematical ideas.

Let us assume that \(X\) is a discrete random variable with support points \(x_1,\ldots, x_K\). We do not need to assume that \(Y\) is also discrete. If we want to find the best function \(g\) which will minimize mean squared error and \(X\) takes on a finite number of support points, we can look at this case by case.

Let \(x_k\) be any one of those support points. For the subpopulation where \(X=x_k\), how would we optimally predict the \(Y\) of any arbitrarily selected unit from that subpopulation? In other words, we are looking for the number \(g\left(x_k\right)\) to minimize MSE. This is nothing but best constant prediction in Section 11.3.1.1. We know from Section 11.3.1.1 that the best constant prediction would be population mean for that subpopulation, i.e., \(\mathbb{E}\left(Y|X=x_k\right)\).

Because the argument in the previous paragraph applies to all support points, the choice of \(g\) which will solve the minimization problem would have the same core structure. The result is that under an MSE criterion, \(\mathbb{E}\left(Y|X\right)\) minimizes the expected squared loss seen earlier.

Observe that \(\mathbb{E}\left(Y|X\right)\) may be different from \(\beta_0^*+\beta_1^* X\). Clearly, for best prediction, the better choice is \(\mathbb{E}\left(Y|X\right)\). For best linear prediction, the better choice is \(\beta_0^*+\beta_1^* X\). Therefore, depending on the class of predictors under consideration, you have different best choices.

You should also note that \(\mathbb{E}\left(Y|X\right)\) and \(\beta_0^*+\beta_1^* X\) may be different from each other. They may be similar with slight differences, but they will still be different from each other. The really special situation is when, by design or by assumption, \(\mathbb{E}\left(Y|X\right)=\beta_0^*+\beta_1^* X\). This is usually the starting point of the simple linear regression model.

If this special situation does not hold, the best we can say is that \(\beta_0^*+\beta_1^* X\) is the best linear approximation to \(\mathbb{E}\left(Y|X\right)\), i.e. \(\beta_0^*\) and \(\beta_1^*\) solves the mnimization problem: \[\min_{\beta_{0},\beta_{1}}\mathbb{E}\left[\left(\mathbb{E}\left(Y|X\right)-\beta_{0}-\beta_{1}X\right)^{2}\right].\]

11.3.2 Connections to linear regression

By now, you should have noticed that the tools which have been introduced to you before and in this review can be theoretically justified for prediction use cases. As a result, findings based on the tools you have learned so far could be expressed in the language of prediction. If we make the assumption (or if it happens) that \(\mathbb{E}\left(Y|X\right)=\beta_0^*+\beta_1^* X\), then your interpretation of \(\beta_0^*\) and \(\beta_1^*\) could be in terms of averages. Otherwise, your interpretation of \(\beta_0^*\) and \(\beta_1^*\) will be in terms of approximations of subgroup averages. These approximations could be very poor approximations.

Of course, the justification requires us to maintain additional assumptions:

The data that we observe is but one of the many realizations or draws from the joint distribution \((X,Y)\). This aspect becomes more refined as you progress towards more complicated settings.
The metric to judge whether you have made a good prediction is an MSE criterion. You have to accept this whether you like it or not, especially if you apply the tools. You also have to judge whether accepting this criterion is appropriate for your case.
A very technical requirement is that these expected values actually exist. You might argue that we are talking about them already, so they must exist, right? In some application areas like climate and finance, there are tools which don’t rely on the existence of these expected values. In addition, their use cases make it hard to justify the use of expected values for prediction.

11.3.3 Which prediction stance should you adopt?

The short answer is that it depends on your research question. The longer answers are about considering the following factors:

How different are \(\mathbb{E}\left(Y|X\right)\) and \(\beta_0^*+\beta_1^* X\) from each other – This is very hard to tell and require you to understand the features of the joint distribution at the population level.
How rich is the support of \(X\) – If there are many possible values of \(X\), then it becomes rarer or harder to find a specific value of \(X\). Subgroup averages are harder to compute if you do not observe a particular subgroup. But for \(\beta_0^*+\beta_1^* X\) has the advantage of not necessarily taking subgroup averages, because it uses shape to “fill up the spots”.
How large is your dataset – The larger your dataset is, the more you will be able to cover the population of interest. With more data, it becomes more feasible to use \(\mathbb{E}\left(Y|X\right)\).

The bottom line is that

You may have to assess how far \(\mathbb{E}\left(Y|X\right)\) is from \(\beta_0^*+\beta_1^* X\) using the data.
Sometimes you might not be interested in the prediction rules themselves. You might be interested in “marginal effects”, say the slope \(\beta_1^*\) or the derivative of the conditional expectation with respect to a change in the value \(X=x\), i.e. \(\partial \mathbb{E}\left(Y|X=x\right) / \partial x\) or the difference in the conditional expectation with respect to a change in the support point (\(x_l\) to \(x_j\)), i.e. \(\mathbb{E}\left(Y|X=x_j\right)-\mathbb{E}\left(Y|X=x_l\right)\).

References

Hendry, David F., and Bent Nielsen. 2007. Econometric Modeling: A Likelihood Approach. Princeton University Press.

Pua, Andrew Adrian Yu. 2024. Econometrics. https://ecometr.neocities.org.

Wooldridge, Jeffrey M. 2020. Introductory Econometrics: A Modern Approach. 7th edition. Cengage Learning.

Remember the discussion on dummy arguments before?↩︎