Welcome to this course on improving your statistical questions. Approximately three years ago, I recorded a similar course, which was called Improving Your Statistical Inferences. And where this previous course focused more on calculating and interpreting statistics such as p-values, and confidence intervals. In this course, I want to take one step back, and focus a little bit more on the types of statistical questions that you can ask. The goal of this course is nicely summarized by this quote by John Tukey. He says far better an approximate answer to the right question, which is often vague than an exact answer to the wrong question, which can always be made precise. I like to think about this as the probability, or the risk of making a Type 3 error. You might have heard about type 1 errors and type 2 errors, which are false positives and false negatives. And a type 3 error is sometimes jokingly called the error, or probability, of having solved the wrong problem. So the goal here is to make sure that you focus on answering a question that's actually interesting, instead of answering the wrong problem. If we want to generate scientific knowledge, we need description, prediction, explanation and exploration, and all these approaches are valid questions that you can ask of your data. As an example of descriptive modeling, we can think of how we use statistical models to summarize data. Very often, the focus in these kinds of questions lies on the measurement level. We focus less on the constructs, or the underlying theory, where we try to accurately measure something and summarize the data, so that other people can use it. An example of a descriptive research question is researched by Kinsey, who studied the sexual behavior and experiences of Americans in a time that very little was known about this topic. He used interviews that provided the statistical basis to draw conclusions about what actually happened, which, at the time, challenged conventional beliefs about sexuality. We can also focus on explanatory modeling. In this case, we use statistical models to test causal assumptions, or explanations that we derive from theories. Explanatory modeling typically starts with a theory. From this theory, we derive a statistical hypothesis. And on this basis, we try to collect some data, from which we draw statistical inferences, and then eventually theoretical inferences. An alternative to explanatory modeling is predictive modeling. Here, the focus lies on trying to predict future, or otherwise unknown observations. So we used statistical models very often trained on some data that we already collected, to try to make predictions about what will happen in unseen datasets. There are some interesting differences between explanatory modeling, which is very common in social sciences, and predictive modeling, which is more common in other fields, such as machine learning. Explanatory modeling requires sample size justifications based on, for example, statistical power. If your goal is to observe a statistically significant effect, then you want to design a well powered study. In terms of predictive modeling, very often, this requires large hold-out datasets. A model is trained on an initial data set, and then it's tested on a much larger data set that wasn't used to generate the predictive model to begin with. Another interesting difference is that in explanatory modeling, we very often rely on well controlled experiments. After all, we want to be able to pinpoint the cause of something. Very often, derived from a theory, and test whether this causal mechanism actually works as we predicted. In predictive modeling, we very often want to make predictions about real-world behavior, which is much more messy. So in this sense, we use a context that also represents this more often messy reality, so well controlled environments are less important. A final difference is that in explanatory modeling, we focus on interpreting each significant causal factor. Because we want to know all the things that significantly explain some variance in the outcome. In terms of predictive modeling, we accept that it's very often difficult to interpret the models. They might predict very well, we don't always know what's going on. And in some cases, we might even remove statistically significant factors from a model, because this improves prediction. Finally, we have exploratory research. The goal of exploratory research is to gain familiarity with phenomenon. Very often, early in research lines, it makes sense to just explore. It answers the question, what's actually going on here? A good example of this is research that's done by Milgram and Sabini, who wondered what happens if you ask people in the subway to give up their seat. Apparently, Milgram's mother-in-law was complaining that nobody gave up their seats for elderly anymore, and Milgram wondered whether this was indeed the case. He sent out some students into the subway to just ask can I please have a seat? The students came back with quite surprising stories. They found it very uncomfortable to actually ask strangers to give up their seats. Milgram was not really sympathetic to this idea, up until the point that he himself went out into the subway and asked people this question. And figured out that breaking implicit norms is actually a very stressful, and sort of annoying thing to do. So this led him to understand the importance of implicit norms in society, a topic he studied quite a lot after this. Another goal of exploration is very often to build a model. We don't know enough to actually test the hypothesis at some points, so the goal is to have some tentative model building, trying to build a theory that actually can be tested later on. One approach is to use data visualization. Another quote by Tukey says that the greatest value of a picture is when it forces us to notice what we never expected to see. This is a nice illustration of how you can use data visualization to learn something that you didn't expect from data that you have collected. Science is both loosening and tightening. This is framework by Klaus Fiedler. He says that exploration very often leads to hypothesis generation in this loosening stage. And in this loosening stage, anything goes. You can create ideas based on whatever you want to do. But exploration is typically one of the things that we do, we explore the situation we're interested in, and then we use this to generate ideas that we later want to test. Now in this more tightening stage, this is where we ask questions. Can we actually confirm or reject certain predictions that we have? But in this loosening stage, we're more exploring the field, and trying to figure out what's actually going on? Very often, I see people who are almost forced into this hypothesis testing straight-jacket. They feel that they have to test some sort of hypothesis in their research, when, actually, what they really want might be to just explore, or maybe just to describe things, or maybe build predictive models. We often pretend that everything that we do starts with some sort of theory, and we derive an experiment from the theoretical predictions. But very often, the theory that we have is not strong enough to lead to testable predictions to begin with. In reality, very often, especially early on in research lines, we have more of a cyclical approach, where we might test something, do an experiment inform a theoretical idea that we have. Use this to then generate another experiment, look at the data, and then have this sort of cyclic approach that's more common in this loosening stage of doing research, before we have a theory that really makes us feel we understand our topic well enough, so that we can actually test it. Something that have noticed when I read statistics papers, is that statisticians very often want to tell researchers what it is that they actually want to know. I jokingly call this the statisticians fallacy. Let's give one of the most well-known examples of this. This is a quote by Jacob Cohen from 1994, where he says what's wrong with null hypothesis significance testing? Well, among other things, it does not tell us what we want to know, and we so much want to know what we want to know, that out of desperation, we nevertheless believe that it does. What we want to know is given these data, what is the probability that the null hypothesis is true? Now, it only took him a year and some comments on this paper that he wrote, to acknowledge that well, maybe this was slightly overgeneralizing. He writes to those who rushed to the defense of null hypothesis significance testing, I can see that there are circumstances in which the direction, not the size of an effect is central to the purpose of research. An example is a strictly controlled experiment, such as a clinical trial. So we went from a situation where null hypothesis significance testing is not the answer that you want, to a statement about how it could be an answer that you want if you run a strictly controlled experiment, which is quite relevant if, like myself, this is actually mainly what you do. Let's take a look at some other examples of statisticians telling us what we want to know. Here's Colquhoun saying what you want to know is that when a statistical test of significance comes out positive, what is the probability that you have a false positive? So this is known as a false positive report probability. Another statement by Kirk says what we want to know is the size of the difference between a and b, and the error associated with our estimate. So he's calling for an interpretation of effect sizes and confidence intervals. Blume, who study statistics mainly from a likelihood perspective, asks, what we really want to know is how likely it is that the observed data are misleading? This example from by Bayarri and colleagues says we want to know how strong the evidence is, given that we actually observe the value of the test statistic that we did. So this is calling for an interpretation of the strength of evidence, for example, by using a Bayes factor. And finally, we have this example from Mayo, who says we want to know what the data say about a conjectured solution to a problem. What erroneous interpretations have been well ruled out? This is calling for a severe test of your hypothesis. Now it might very well be that you want to know all the things on the previous slides, and even more than these. The relative weight with which you want to know these answers depend on, for example, the research phase, and your philosophy of science. Very early on in research lines, other questions are more important than later on in research lines, when you already have well developed models. So I think at the very core of this lecture is that I think that instead of others telling you what you want to know, you should be able to justify the question you're asking.
