Causal validity (internal validity) 12
Cross-population generalizability (external validity) 12
Descriptive research
Evaluation research
Explanatory research
Exploratory research
Generalizability 12
Illogical reasoning 5
Measurement validity 12
Overgeneralization 5
Resistance to change 6
Sample generalizability 12
Science 7
Selective (or inaccurate) observation 5
Social science 7
Validity 11

"Now will you tell me where we are?" asked Tock as he looked around the desolate island.

"To be sure," said Canby; "you're on the Island of Conclusions. Make yourself at home. You're apt to be here for some time."

"But how did we get here?" asked Milo, who was still a bit puzzled by being there at all.

"You jumped, of course," explained Canby. "That's the way most everyone gets here. It's really quite simple: every time you decide something without having a good reason, you jump to Conclusions whether you like it or not. It's such an easy trip to make that I've been here hundreds of times."
Norton Juster, The Phantom Tollbooth


In general, I won't just repeat what's in the text book. I assume that you treat a reading assignment in that book as an assignment to learn the material presented there (to be clear, it's the concepts and not the examples that you are expected to "learn"). If we start class with "are there any questions?" that would be the time to say "yes, I did not really understand what the textbook authors meant when they said that…."

That said, let me list the things you ought to take away from this week and this week's reading:

Name and explain four threats to right answers about the social world.
Name and describe the differences among four types of research.
Name and explain three things good research needs.

and then there is this assignment to define the key terms listed at the end of the chapter:

  • Causal validity (internal validity) 12
  • Cross-population generalizability (external validity) 12
  • Descriptive research 8
  • Evaluation research 10
  • Explanatory research 10
  • Exploratory research 9
  • Generalizability 12
  • Illogical reasoning 5
  • Measurement validity 12
  • Overgeneralization 5
  • Resistance to change 6
  • Sample generalizability 12
  • Science 7
  • Selective (or inaccurate) observation 5
  • Social science 7
  • Validity 11

You should know that the point is NOT to copy the definitions you might find in the book. The point is to come up with your own definition, the one you would use with your younger sister or that uncle who always wants to know what you are studying or maybe a prospective employer who asks you about it in a job interview.

One way to build up your scientific vocabulary is to listen for when these ideas or words and phrases show up in things you read or hear. It's especially valuable when the concept is there, but the words are not - you "spot" the idea being used even though the words were not said. It's a bit like

Last time, I talked about the fact that we do "science" as a way of countering a whole series of things that have tended, historically and at present, to lead us to the wrong answers about the world.

This time we want to think about what makes good science - what makes our researching "sciency"?

Four Kinds of Research

Before we get started, a side story. New X-Files TV show just started the other night. I asked about this in my other class and joked about extra terrestrials and a few students coughed and asked why I was joking. They said, quite seriously, "you know there are ETs, don't you? They ARE real. In fact, they are here at Mills." They seemed really serious and a few other students nodded in assent. How do you know? I asked. "There are shows about it, duh. They wouldn't make shows about it if it were not real. And movies."

As a teacher, this line of thinking did not impress. But they were pretty rock solid in their conviction. I remembered a thing called the Thomas theorem, named after sociologist W I. Thomas:

The Thomas theorem is a theory of sociology which was formulated in 1928 by W. I. Thomas and D. S. Thomas (1899–1977): “ If men define situations as real, they are real in their consequences. ” In other words, the interpretation of a situation causes the action.

These folks were defining the situation - little green people at Mills - as real, so it's probably real in its consequences.

So I decided to research this. First I decided I needed to convince President DeCoudreaux to provide us funding to study this. On our own nickel we did some preliminary research: we went into the Mills email system and randomly selected 50 email addresses and sent them an email inquiring as to whether there ETs where real and whether there were any at Mills. Twenty nine answered (58%) and of these 17 (59%) said of course ETs were real and 15 (52%) said they were at Mills.

Since the survey was RANDOM I don't think it's likely that I accidentally emailed all the X-file fans and Trekkies and Star Wars fanatics. ON the other hand, it could be that people who DO NOT believe in aliens thought my little survey was stupid and they just deleted the email.

In other words, even though the SAMPLE was random, there might have been SELF-SELECTION BIAS in terms of who responded.

So, I do not feel safe claiming that 59% of the Mills community believes in aliens or that 52% think there are aliens among us here.

BUT I am confident that my DESCRIPTIVE research suggests that there are 17/50=34% or about a third of the Mills community that believes in aliens. What I've done is just assume that everyone who did not answer would have said no to my questions.

This is taking a conservative approach - since the thing I'm studying is belief in aliens I want to do all my adjustments in the direction of "no finding" so that if, after all that, I still have some indication of the thing I'm studying, then I can think, maybe it does exist after all.

GOOD SCIENCE stacks the deck against itself.

But I still had to mindful of something: even though I've established that a sizable part of the Mills community is alien-positive, I don't know anything about Oaklanders, college students, women, Californians, Americans, or humans. My work so far does not GENERALIZE beyond the community I am studying (and this is defined not even by all of the Mills community but rather, people at Mills who were in my SAMPLING FRAME on the email list from which I selected my 50).

But anyway, back to the story. I took my DESCRIPTIVE data into one of Alecia's open office hours and said I'd like some funds to study this. She looked at me sort of funny and I thought for a moment that I'd blown it. If SHE believes in aliens and that they are at Mills, maybe I should have FRAMED my research the other way round: "President DeCoudreaux, you should know that almost 2/3 of the Mills community does not believe in aliens!" Luckily, she was just a little surprised that I'd come to her office hours in gym shorts. We got past that and she gave me $2500 to study the belief in aliens at Mills.

And so I decided my next step would be to do some EXPLORATORY research. Who are these folks? Do they know each other? Are some of them aliens? What do they think aliens look like? How do they know that there are aliens here?

For this part of my research I was able to use the list of emails of the 15 people who knew about the aliens at Mills.

We decided we would do "in depth interviews" with them. Now what do we mean by "in depth interviews"? Was there an alternative to do really shallow interviews? Or really casual interviews?

Hi, my name is Dan and I'm doing a research project on extra-terrestrials at Mills. Seen any? Thanks.

So, me and my team sat down and thought about what we would ask them. We realized we needed to learn a few things about aliens and time travel and all the rest. So we set about doing a little background research.

The Shoulders of Giants

We went to the library to talk to Michael Beller. He was helping out a history student so we had to wait and then we looked at one another and realized we were a little nervous about this, thinking that if there really were extra terrestrials on campus, it was pretty likely he was one - so we didn't reveal our project, we just asked about how to use Google scholar and then we learned a few things about extra-terrestrials on earth. Our sources suggested that belief in alien life on earth is usually limited to just a few people out of a hundred. We realized we had to approach these folks carefully so we did not frighten them.

We developed an interview guide that would lead to a conversation of about an hour. It included some "innocent" background questions and then it got into the subject: "These days," the question began, "more and more people are open minded about things that used to be considered scientific impossibilities." And then we list a few: life after death, clean diesel, flying cars, extra-terrestrials, sane Republican presidential candidates. Our plan was, if they checked yes on extra-terrestrials, we'd launch into some follow up questions. If not, we had a few plain vanilla wrap up questions and we were out of there. But almost everyone checked the ET box.

And here's what we learned in our EXPLORATORY research. Folks believe in aliens either because they have met them or they are themselves from outer space. Not all aliens, apparently, are little green men (the proper name, apparently, is cis-specied - the ones that look just like us are trans-specied).

Another thing we found out was that most of the extra-terrestrials on campus were unhappy, some to the point of depression. At least, that's what they said and that's how they seemed to us. Most importantly, perhaps, a lot of them were leaving Mills. We had an extraterrestrial retention problem on our hands.

Around this time, I run into my colleague Professor Shelton and we get to talking. I mention this little project Maggie and I are doing and she gets a little twinkle in her eye. "It's not the first time, you know," she said.

So she goes on to explain how she came across some files in the Mills archives not long ago about a project on belief in and presence of alien life forms at Mills from the late 1980s. And then, when she told her colleague Professor Gordon about what she'd found, he reveal that there was yet another study from the 1960s and he was quite sure about this because he himself had conducted the study! The funny thing is, though, she said, "I remember quite distinctly that the Mills extra-terrestrials in those studies were noted to be much happier than the other Mills students of the time. All the reports mention this.

So all the sudden we had a mystery on our hands. Why were the ETs leaving?


We have a few hypotheses. Maybe the ETs don't feel like they fit in. Maybe they are homesick. We start thinking about ways to study this.

Our first idea is that we should talk to the ETs we can find and ask them if they feel like they fit in. So we go over the to ET lounge and talk to a few. A number of them say they don't. They talk about being shy, worrying that they stand out, that they are not smart enough to be here.

We are pretty sure we are onto something when we note that 75% of the ETs we talk to mention this fitting in thing.

We are talking about the findings in a meeting one day and this student overhears us (in the faculty dining room) and says "maybe you have some internal validity problems?" We ask what she means. Did you think about the fact that you might have been talking specifically to the few who feel that way based on how you found them? Did you think about the fact that maybe they were expressing the same feelings that all students, both terrestrial and extra feel? I think you need to go back to the drawing board.

So we did. We changed our strategy and managed to get a list of all the extra terrestials and we get a random set of terrestrial students and we ask them the same questions and it turns out there's not much difference between them - but there are a bunch of both that don't feel like they fit in, they are not "cool."

Since our study was carefully done, we think we might be onto something.

Then we run into a history student who says: aren't you trying to figure out why they are leaving now, but they did not leave before? Can you answer that without thinking about before?

First project just looked at the ETs at the beginning of the semester and then at the end of the semester and we found a difference so we knew that something about the way the school was run was having a negative effect on the ETs.


Turns out they are very worried about grades. And so we suspect: grade anxiety is the problem.

But when we talk to a colleague, she tells us that she's found grade anxiety is just a normal part of being a student. So this can't explain the peculiar problem with our ETs.


Anyway, long story short, the president was thrilled with our research. She got on the horn with some wealthy alumnae and got a load of cash for an astronomy department on the assumption that a lot of the extra-terrestrials would actually like to major in star-gazing or at least be able to minor in it.

So, we hired some astronomy professors, turned the garages up behind Mary Morse into an observatory (we bought this one from Vassar College for pennies on the dollar).

We decided to create a new summer program for the extra-terrestrials too. Since we already had the SAW program we had a supply of T-Shirts and so we named the new program the Summer Astronomy Program. We were sure it would help make the Mills experience a more positive one for the ETs.

CAUSAL VALIDITY: we need to meet multiple criteria: time sequence, nonspurious association, mechanism/theory,



Student Life was allowed to hire some astronomers but someone there misread the memo and they ended up hiring new RAs - resident astrologers instead.

Three years into the program we put together an EVALUATION RESEARCH project. We

A lit review or a course is meant to help you climb up onto the shoulders.

A Top Ten List (sort of)

Just because it impressed you or opened your eyes, does not mean it makes sense to do research on it. There are lots of things that you might want to learn more about, and you should. But the "research topic" shows up AFTER you know a whole lot about it, not at the start.

Don't study yourself. Study humans.

Don't study images of women in magazines.