Replicability in Psychology

To judge whether or not a study is important

A study that helps advance scientific progress by supporting or shaping scientific theories.

(Glossary) pertaining to a study whose results are obtained again when the study is repeated.

Are the results of the study a fluke or will researchers get the same result if they do the study again?

It gives a study credibility

1. Use inferential stats 2. Conduct the same study again

Use of the theories of probability to decide whether or not a study's result is "statistically significant"

When the result found in a sample is so extreme that it is unlikely to have happened by chance

It predicts the likelihood that the result was sampled from a population in which there is no difference, no correlation, or no relationship

If they were to repeat the study exactly (with the same sample size and a similar set of participants), they would get similar results a high percentage of the time.

reliable

According to the analysis, there is a strong chance that the result would be stat sig again if the study were to be repeated

a replication study

In order to have findings published, researchers want to make sure their results are significant and look presentable.

Three

1. Direct Replication 2. Conceptual Replication 3. Replication-Plus-Extension

researchers repeat an original study as closely as they can to see whether or not the original effect shows up in the newly collected data

Exact Replication

True

The two studies may have been conducted at different times in the year. The experimenters may have been different people. Some other minor circumstances may have changed.

False

If there were any existing internal threats or flaws, they will be replicated with the same methodology

Researchers study the same research question but use different procedures. At the abstract level, the variables in the study are the same, but the procedures for operationalizing the variables are different

Red vs. Achievement with different color booklets instead of just red lettering

Researchers replicate their original study but add variables to test additional questions

It could extend a finding to a new population

Introduce a situational variable

In the driving and competency design, researchers wasted to see if the participants' ability to drive with cellphones would improve with practice. The degree of practice became the new situational variable.

False

True

Scientific literature

Scientific literature consists of a series of related studies, conducted by various researchers, that have tested similar variables. They are composed of several studies on a particular topic, often conducted by many different researchers.

A way of mathematically averaging the results of all the studies that have tested the same variables, to see what conclusion that whole body of evidence supports. The process of collecting all possible studies on a particular research question and combining them mathematically to study the overall trend in the data.

True

1. Collect all possible examples of a particular kind of study 2. Average all the effect sizes to find an overall effect size

True

Publication bias; Sig relationships are more likely to be published than null effects

Instead of being published, studies sit forgotten in the researchers' filing cabinets, and this allows meta-analysis to (possibly) overestimate the true size of an effect because the null effects have not been included

Contact colleagues requesting both published and unpublished data for their project

True

True

True

They report on the latest studies as well as the provide a sense of what the entire literature says on a particular topic

External

False

Conceptual replications and replication-plus-extension studies

True

True

How the participants were obtained for the study

A study's similarity to real-world contexts

Mundane realism

When the sample for a study has been selected from a population at random (using probability sample), the results from that sample may be generalized to the population it was drawn from. It also makes sense that if a study's sample includes only men, researchers may not generalize its results to women, etc. However, the statements are only sometimes correct because the importance of external validity depends on a researcher's priorities. Whether a researcher strives for external validity in a study depends on what research mode he/she is operating in: theory-testing mode or generalization mode.

1. Theory testing mode 2. Generalization mode

Testing association or causal claim to investigate support for a theory

Internal

Researchers want to generalize the findings from the sample in their study to a larger population

True

True Examples on page 390

A sub discipline of psychology that makes generalization mode its primary business. Cultural psychologists are interested in how cultural background and environment shapes the way a person thinks, feels, and behaves.

Western Educated Industrialized Rich Democratic

False

When a study takes place in the real world

Settings in which people experience authentic emotions, motivations, and behaviors

True

True

"Is that result stat sig?" "Was the result replicated?" "That was a single study. How does the study fit in with the entire literature?" "How well do the methods of this study get at the theory they were testing?" "Was the study conducted in generalization mode? If so, were the participants sampled randomly?" "Would the results also apply to the other settings?" "Would that result hold up in another cultural context?"

"This single study is definitive" "This study has nothing to do with the real world." "That's only a theory" "This is a bad study because they didn't use a random sample." "This is a bad study because they used only North Americans as participants." "They used thousands of participants. It must have great external validity." "That psychological principle seems so basic, I'm sure it's universal."

The first step to establishing a study's importance is to use inferential statistics or replication studies to determine whether the findings are replicable. If inferential stats show that a study result is sig, researchers can conclude that if they ran the study in exactly the same way, we would probably obtain the same result. Replication studies are a concrete way of establishing a study's replicability. There are three main types of replication studies. -Direction replications: attempt to repeat the original study exactly -Conceptual replication: same conceptual variables as the original study, but they operationalize the variables differently -Replication Plus extension studies: repeat the original study but introduce new participant variables, situations, or levels of the IV. Meta-analyses collect the effect sizes from all published (when possible, unpublished) studies that have tested the relationship between a set of variables, using stat tools to compute the average effect size for a particular research question. Meta analyses are powerful tools that help quantify whether an effect exists in the literature, and if so, how large the effect is and what moderates it. When researchers are in generalization mode, they care whether the samples they are studying are representative. Whether the data from sample A are applicable to population A and if it could be applicable to population B. When researchers are in theory-testing mode, researchers design studies that test a theory to find out whether the study turns out the way the theory predicts it would. -If the study doesn't turn out the same way, the theory must be modified and the new theory must be tested in a new study. -Generalization is left for later studies which will test whether the theory holds in a sample that is representative of another population. Question two assumptions about generalization: 1. Important research should use random samples from populations 2. Important research should apply to people from both sexes and all ethnicities, social classes, states, etc. --In some studies, it is important to use random samples and a variety of people, but diverse, rep samples are primarily important when researchers are in generalization mode. --In theory testing mode, researchers do not (yet) consider whether their samples are rep of some population, so external validity is much less important than internal validity. Researchers who make frequency claims are always in generalization mode, but they are also in generalization mode when they ask whether an association or causal claim can be generalized to a different group of people. Cultural psychologists have documented how psychological discoveries are not always applicable cross-culturally. Even seemingly basic cognitive or visual processes may not operate the same way in different cultural contexts. The work of cultural psychologists serves as a reminder that when researchers collect data only from WEIRD samples, they cannot assume that the theories they have developed in theory-testing mode will apply to all human beings. Question the assumption that to be important, research must be conducted in settings that resemble the real world. -Some research has high ecological validity, other research takes place in labs. -While some experiments might be high in experimental realism, they may not resemble other real world settings outside of the lab. -Research from "artificial settings" help researchers test theories in the most internally valid way possible, and their lessons usually apply to other real-world situations.