Most social scientists do not do experiments, but there is no better preparation for research than to gain a solid grounding in the LOGIC OF EXPERIMENTS.

We refer to the application of this knowledge as "research design" and to the variety of standard ways to do this as "research designs."

A primary goal of research design is to ensure that you avoid threats to the validity of your work. In this context we mean whether your research is worth anything, whether it it correct. But the standard definition of "validity" also implies: we are concerned about whether you found out what you think you found out.

Do an experiment in the room. Split into four random groups. Groups one and three get a pre-test. Groups one and two get the treatment. Groups three and four get no treatment. All groups get the post-test.
Deck of cards at each place.
All: The cards in this deck will describe a simple game. The next card in the deck will describe a scenario and ask you to decide on what your move in the game will be and to write it down. Then you will turn that card over and follow the instructions on the next card, and so on.

GROUP ONE

Pretest In this game you are matched with another player anonymously. You are given $40. You may split the money with the other person however you would like, including the option of keeping it all for yourself. On the next card, write down how much you would like to give to the other person.
Treatment/Test In this game you are matched with the student sitting closest to you in class today. You are given $40. You may split the money with the other person however you would like, including the option of keeping it all for yourself. On the next card, write down how much you would like to give to the other person.

GROUP TWO

Treatment/Test In this game you are matched with the student sitting closest to you in class today. You are given $40. You may split the money with the other person however you would like, including the option of keeping it all for yourself. On the next card, write down how much you would like to give to the other person.

GROUP THREE

Pretest In this game you are matched with another player anonymously. You are given $40. You may split the money with the other person however you would like, including the option of keeping it all for yourself. On the next card, write down how much you would like to give to the other person.
Test In this game you are matched with another player anonymously. You are given $40. You may split the money with the other person however you would like, including the option of keeping it all for yourself. On the next card, write down how much you would like to give to the other person.

GROUP FOUR

Test In this game you are matched with another player anonymously. You are given $40. You may split the money with the other person however you would like, including the option of keeping it all for yourself. On the next card, write down how much you would like to give to the other person.

There are lessons for us in why we have all four groups.

Here's our question: does thinking about a concrete person who is physically nearby have an effect on how you play the dictator game? The subjects in group two are the ones we study with this in mind. Suppose they average giving 63% of the initial endowment away. We can report that this group is significantly less self-interested than the base line rational actor model would suggest. But can we say anything else?

The "treatment" that we think we are studying is the person being told that the other player is the person sitting next to them.

We call the treatment the INDEPENDENT VARIABLE - meaning that it is free to vary separate from the causal relations in the experiment. It is the "cause" and how much the subjects give to the other player is the outcome or effect or DEPENDENT VARIABLE.

"True experiments, with randomized assignment and full control by the
researcher, produce knowledge that has high internal validity. This means
that changes in the dependent variables were probably caused by—not merely
related to or correlated with—the treatment. Continual replication produces
cumulative knowledge with high external validity—that is, knowledge that
you can generalize to people who were not part of your experiment." Bernard 113

Threats to Validity: What con confound our results

CONFOUND: This comes from the concept "confounding variable" or "confounding factor" and it means that some external factor you are not considering has a relationship with both your independent and your dependent variable. The result is you think that something is going on between your IV and your DV but there's not - it's just this third, outside factor.

Suppose you have a study and you think you have found a causal relationship between A and B

History. Suppose you wonder whether higher levels of social media usage cause higher levels of political engagement among African American youth. You have access to high school and college student surveys from 2013 and 2015 that have very good measures of social media usage and political engagement.
Maturation. You want to do evaluation research on a substance abuse program. You follow a cohort of drug users and non-users who are in programs and not in programs to see how much the programs help folks get their lives together. The people are followed for 6 years. What matures? People. Organizations and programs. "Maturing out" in substance abuse, gang, etc. research.
Testing and instrumentation. We plan to do home-based interview study about attitudes toward pets. We send out interviewers out to random addresses. After 1000 doors are knocked on, we find we have done 600 interviews. Of the 400 failed contacts, 200 were due to the interviewer leaving because of a vicious dog.
Regression to the mean.
Selection bias.
Mortality.
Diffusion.

Later: sampling on the dependent variable.