Before we get into talking about linear regression, you may recall that we recently have discussed advanced machine learning techniques such as neural networks and support vector machines, but these are not always the most appropriate tool for modeling data. Machine Learning models are big, complicated, and almost impossible to interpret. While they have great capacity and are sometimes the only solution to difficult problems, their downsides can be substantial depending on what our goals are. Additionally, it is often the case that we want to understand our models or have some measures of how valid they are.
For example, the goal of a physicist may be to understand their model of a particle interaction. If they create a neural network to do this, we may have low error in the system but our human understanding of the laws of nature is no better. Likewise, an economist may be more concerned with understanding the general relationship between political uncertainty in elections, uncertainty in public policy, and uncertainty in financial markets, than creating a complex interactive system that prevents generalizations that could be applied to governance, electioneering, or market trading. So, for the time being, let us return from the vast jungles of machine learning to the tamed and comfortable community of traditional statistical models, the first of which will be linear regression.