Research Design

question

Typical Research Process
answer

The typical quantitative study involves a series of steps, one of which is the statistical analysis
question

step 1:
answer

research questions and hypotheses
question

step 2:
answer

operationalize and choose measures
question

step 3:
answer

choose a research design
question

step 4:
answer

analyze data
question

step 5:
answer

draw conclusions
question

causal relationship
answer

-the goal of virtually every quantitative research study is to determine between two variables -ideally a causal relationship would be established -that is, does a change in X definitively lead to a predictable change in Y
question

Confounding Variables
answer

-Variables always have multiple causes -we would like to isolate only the two variables of interest and exclude the effects of all other variables
question

Unwanted variables
answer

are called confounding or nuisance variables
question

Internal validity
answer

the validity of findings with the research study; the technical soundness of a study, particularly concerned with the control of extraneous influences that might effect the outcome
question

External validity
answer

the degree to which the findings can be inferred to the population of interest or to other populations or settings; the generalizability of the results
question

Both internal and external validity
answer

are import in a study but they are frequently at odds with one another in planning and designing a study
question

Internal validity is
answer

considered the basic minimum for experimental research
question

Internal validity facts
answer

-is the basic minimum without which any study is not interpretable -particularly important in experimental studies -did, in fact, the experimental treatment (X) produce a change in the dependent variable (Y) -lends itself to highly controlled, laboratory settings
question

To gain internal validity
answer

the researcher attempts to control everything and eliminate possible extraneous influences
question

Threats to Internal Validity
answer

history, maturation, testing, instrumentation, statistical regression, selection bias, experimental mortality, interaction among factors
question

History
answer

events occurring during the experiment that are not part of the treatment
question

maturation
answer

biological or psychological processes within participants that may change due to the passing of time (age, fatigue, hunger)
question

Testing
answer

the effects of one test upon subsequent administrations of the same test
question

Instrumentation
answer

changes in testing instruments, raters, or interviewers including lack of agreement within and between observers
question

Statistical regression
answer

the fact that groups selected on the basis of extreme scores are not as extreme on subsequent testing
question

Selection bias
answer

identification of comparison groups in other than a random manner
question

Experimental mortality
answer

loss of participants from comparison groups due to nonrandom reasons
question

Interaction among factors
answer

factors can operate together to influence experimental results
question

External validity facts
answer

-generalizability of results to what populations, settings, or treatment variables can the results be generalized -concerned with real-world applications -random selection is the key to controlling most threats to external validity
question

External validity is generally
answer

controlled by selecting subjects, treatments, experimental situations, and tests to be representative of some larger population
question

Two types of External Validity
answer

Population validity and Ecological Validity
question

Population Validity
answer

refers to the extent to which the results can be generalized from the experimental sample to a defined population
question

Ecological Validity
answer

refers to the extent to which the results of an experiment can be generalized from the set of environmental conditions in the experiment to other environmental conditions
question

Common sources of error
answer

many possible sources of error can cause the results of a research study to be incorrectly interpreted. the following sources of error are more specific threats to validity of a study than those described previously
question

Threats to External Validity
answer

interaction effects of testing, selection bias, reactive effects of experimental setting, multiple-treatment interference
question

Interaction effects of testing
answer

the fact that the pretest may make the participants more aware of or sensitive to the upcoming treatment
question

Selection bias is
answer

when participants are selected in a manner so they are not representative of any particular population
question

Reactive effects of experimental setting
answer

the fact that treatments in constrained laboratory settings may not be effective in less constrained, real-world settings
question

Multiple-treatment interference
answer

when participants receive more than one treatment, the effects of previous treatments may influence subsequent ones
question

Additional Threats to External Validity
answer

-Reaction effects: Hawthorne effect, placebo effect, John Henry effect, rating effect -Experimental effect: experimental bias effect
question

Hawthorne Effect
answer

-a specific type of reactive effect in which merely being a research participant in an investigation may affect behavior -suggests that, as much as possible participants should be unaware they are in an experiment and unaware of the hypothesized outcome
question

Placebo effect
answer

participants may believe that the experimental treatment is supposed to change them, so they respond to the treatment with a change in performance
question

John Henry Effect
answer

a threat to internal validity wherein research participants in the control group try harder just because they are in the control group
question

Rating effect
answer

variety of errors associated with ratings of a participant or group -halo effect -overrater error -underrater error -central tendency error
question

Experimenter bias effect
answer

the intentional or unintentional influence that an experimenter (researcher) may exert on a study
question

Types of designs
answer

-non-experimental -true experimental -quasi-experimental
question

Non-experimental designs/ Descriptive Research
answer

-weak experimental designs in terms of control -no random sampling -threats to internal and external validity are significant problems -many definite weaknesses -example: one-group pretest/posttest design
question

True experimental designs
answer

-best type of research design because of their ability to control threats to internal validity -utilizes random selection of participants and random assignment to groups -example: pretest/posttest control group design
question

Quasi-experimental designs
answer

-lack either random selection of participants or random assignment to groups -lack some of the control of true experimental designs, but are generally considered to be fine example: nonequivalent group design
question

Research Design Basics
answer

-research design, or blueprint, for a study can be drawn out using a simple set of symbols proposed by Campbell & Stanley (1963)
question

X =
answer

an intervention was administered (i.e., a treatment, program, IV, etc)
question

O =
answer

data were collected (i.e., observed, measured, etc.)
question

Between Subjects design
answer

-also called between groups or cross-sectional -each participant is in one (&only one) of the treatments/conditions -different groups of participants are in each treatment/condition -typically used to study \”differences\”– when in application, a participant will usually be in one treatment/condition or another
question

Characteristics of between-subject design
answer

-the defining characteristic of a between-subjects design is that it compares separate groups of individuals -it allows only one score per participant (every score represents a separate, unique participant) -each score represents a separate participant, a between subjects design is often called an independent-measures design
question

More characteristics of between-subject design
answer

-between-subjects experimental design requires a separate, independent group of individuals for each treatment condition -individuals are assigned to groups using a procedure that attempts to create equivalent groups
question

Goal of between-subjects experiment
answer

is to determine whether differences exist between two or more treatment conditions (a researcher may want to compare two teaching methods (two treatments) to determine whether one is more effective than the other)
question

Advantages of between-subjects design
answer

-each individual score is independent of the other scores -participant’s score is not influenced by such factors as: -practice or experience gained in other treatments -fatigue or boredom from participating in a series of treatments -contrast effects that result from comparing one treatment to another
question

Disadvantages of between-subjects designs
answer

-large number of participants (problem with special populations) -individual differences: -characteristic that differ from one participant to another are called individual differences -individual differences can become confounding variables -individual differences can produce high variability in scores
question

Confounding variables in between subjects design
answer

Individual differences: -participant characteristics differ from one group to another -participants in one group may be older, smarter, taller, than the participants in another group environmental variables: -characteristics of the environment differ between groups -one group may be tested in a large room and another group in a smaller room
question

Equivalent groups
answer

in a between-subjects experimental design, the researcher does have control over the assignment of individuals to groups -the separate groups must be: – created equally -treated equally (except for the treatment conditions -composed of equivalent individuals
question

Limiting confounding by individual differences
answer

-random assignment,matching groups (matched assignment), holding variables constant
question

Random assignment (randomization)
answer

a random process is used to assign participants to groups
question

Matching groups (matched assignments)
answer

involves assigning individuals to groups so that a specific variable is balance or matched across the groups (IQ)
question

Holding variables constant
answer

simply hold the variable (restrict the participant to those with with IQs between 100-110)
question

Within and between treatments variability
answer

advantage: -variability between treatments -it can be increased by increasing differences between conditions (levels) disadvantages: -variability within treatments -it is caused by individual differences -should be minimized
question

Within and between treatments variability (2)
answer

minimizing variability within treatments -standardize procedures and treatments setting -limit individual differences by holding a participant variable constant -random assignment and matching -sample size: -using a large sample can help minimize the problems associated with high variability
question

Other threats to internal validity of between-subject designs
answer

assignment bias, differential attrition, on diffusion or imitation of treatment, compensatory equalization, compensatory rivalry, resentful demoralization
question

Assignment bias
answer

-groups of participants are different before the treatments -the group assignment process produces groups within noticeably different characteristcs
question

Differential attrition
answer

-attrition refers to participant withdrawl from a research study before it is completed -differential attrition refers to differences in attrition rates from one group to another and can threaten the internal validity of a between-subjects experimental (effectiveness of a dieting program)
question

Diffusion or limitation of treatment
answer

refers to the spread of the treatment effects from the experimental group to the control group (new depression therapy)
question

Compensatory equalization
answer

occurs when an untreated group learns about the treatment being received by another group and demands the same or equal treatment (watching Batman in violent TV group)
question

Compensatory rivalry
answer

occurs when an untreated learns about the treatment received by another group and then works extra hard to show that they can perform just as well as the individuals receiving the special treatment
question

Resentful demoralization
answer

-opposite of compensatory rivalry -occurs when an untreated group learns about the treatment received by another group and is less productive and less motivated because they resent the expected superiority of the treated group
question

Applications and statistical analyses of between-subjects design
answer

-comparing only two groups of participants -comparing means for more than two groups
question

Comparing only two groups of participants
answer

-this design is referred to as the single-factor two-group design or simply two group design -an independent-measures t test is used to determine whether there is a significant difference between the means
question

Comparing means for more than two groups
answer

-single factor multiple group design may be used and analysis of variance (ANOVA) would be used for statistical analysis -adding extra groups to a research study tends to reduce the differences between groups
question

Within-Groups Designs
answer

-also called within-subjects, repeated measures, or longitudinal, each participant is in all (every one) of the treatment/conditions -one group of participants, each one in every treatment/condition -typically used to study \”changes\” when, in application, a participant will usually be moving from one condition to another
question

Characteristics of within-subjects design
answer

-a within subjects experimental design compares two or more different treatment conditions (or compares treatment and control) by observing or measuring the same group of individuals in all of the treatment conditions being compared -a within-subjects design looks for differences between treatment conditions within the same group of participants -a within subjects design is often called a repeated-measures design because the research study repeats measurements of the same individuals under different conditions -it is used in experimental situations comparing different treatment conditions and also to investigate changes occurring over time
question

Advantages of within-subjects designs
answer

-it requires relatively few participants -it essentially eliminated all of the problems based on individual differences that are the primary concern of a between-subjects designs -within-subjects design has no differences between groups -each individual serves as his or her own control or baseline
question

Disadvantages of within-subjects
answer

-time related problems: -participant attrition, history, maturation, instrumentation, statistical regression
question

Disadvantages of within subjects design
answer

carry over effects, progressive error
question

Carryover effects
answer

-changes in behavior or performance that are caused by participation in an earlier treatment condition -carryover effects exist whenever one treatment condition produces a change in the participants that affects their scores in subsequent treatment conditions (e.g. new skill from treatment 1 can influence results in treatment 2)
question

Progressive error
answer

changes in participant’s behavior or performance that are related to experience over time in a research study but not related to a specific treatment or treatments (e.g. practice effects and fatigue)
question

Dealing with time related threats and order effects
answer

-controlling time: if the different treatment conditions are scheduled over a period of weeks, the chances greatly increase that the results will be influenced by some outside event (history) or maturation or change in the measurement instrument When a within subjects design is not a good idea: -e.g. comparing two methods of teaching reading to first-grade children (carryover effects)
question

Dealing w/ time related threats and order effects
answer

Counterbalancing: -involves changing the order in which treatment conditions are administered from one participant to another -the goal is to use every possible order of treatment with an equal number of individuals participating in each sequence -the purpose of counterbalancing is to eliminate the potential for confounding by disrupting any systematic effects from factors related to time or the order of treatments -e.g. with two treatments one half of the participants begins in treatment 1, then moves to treatment 2 and the other half begins in treatment 2, then receives treatment1
question

Applications and statistical analyses of within-subjects designs
answer

Two treatment designs, multiple treatment designs
question

Two treatment designs
answer

-a repeated-measures t test or an analysis of variance can be used to evaluate the statistical significance of the mean difference -if the data are measured on an ordinal scale, a Wilcoxon test can be used to evaluate significant differences
question

Multiple-treatment designs
answer

-with too many treatment conditions, the distinction between treatments may become too small to generate significant differences in behavior -statistical analysis – repeated-measures analysis of variance to test for any significant differences among the treatment means
question

Matched-subjects designs
answer

-each individual in one group is matched with a participant in each of the other groups -the matching is done so that the matched individuals are equivalent with respect to a variable that the researcher considers to be relevant to the study (e.g. IQ) -maintains all the advantages of between-subjects and within-subjects designs without the limitations of either (e.g. eliminates individual differences, time-related factors and order effects)
question

Properties of True Experiments
answer

-True experiments provide the highest degree of internal validity possible -Virtually all confounding variables should be controlled via random assignment -Because confounds are controlled, a causal link can only be claimed when using a true experimental design
question

Limitations of Random Assignment
answer

-If random assignment is so advantageous, what prevents its use? -It may not be ethical to expose subjects to or withhold from them a treatment –e.g., What are the effects of alcohol use during pregnancy? -Some research questions are not amenable to random assignment –Comparisons of naturally existing groups (e.g., males and females, ethnicity, socioeconomic differences)
question

If only True experiments can be causally interpreted, why even bother running non-experiments?
answer

1st Remember that we can’t always run a true experiment ! Lots of variables we care about can’t be RA & manip – gender, family background, histories and experiences, personality, etc. Even if we can RA & manip, lots of studies require long-term or field research that makes ongoing equivalence (also required for causal interp) very difficult or impossible. We would greatly limit the information we could learn about how variables are related to each other if we only ran studies that could be causally interpreted.
question

Categorical vs. Continuous Variables
answer

-In practice, it is not usually necessary to make such fine distinctions between measurement scales -To distinctions, categorical and continuous are usually sufficient -Categorical variables consist of separate, indivisible categories -Continuous variables yield values that fall on a numeric continuum, and can (theoretically) take on an infinite number of values
question

Nominal Scale
answer

-Observations fall into different categories or groups -Differences among categories are qualitative, not quantitative -Examples: Gender Ethnicity Counseling method (cognitive vs. humanistic) Retention (retained vs. not retained)
question

Ordinal Scale
answer

-Categories can be rank ordered in terms of amount or magnitude -Categories possess an inherent order, but the amount of difference between categories is unknown -Examples: Class standing Letter grades (A,B,C,D,F) Likert-scales (SD, D, N, A, SA)
question

Interval Scale
answer

-Categories are ordered, but now the intervals for each category are exactly the same size -That is, the distance between measurement points represent equal magnitudes (e.g., the distance between point A and B is the same as the distance between B and C) -Examples: Fahrenheit scale of measuring temperature Chronological scale of dates (1997 A.D.) Standard scores (z-scores)
question

Ratio Scale
answer

-Categories are ordered, but now the intervals for each category are exactly the same size -That is, the distance between measurement points represent equal magnitudes (e.g., the distance between point A and B is the same as the distance between B and C) -Examples: Fahrenheit scale of measuring temperature Chronological scale of dates (1997 A.D.) Standard scores (z-scores)
question

LOOK AT SLIDES
answer

62,61,55,54,53,52

Get instant access to
all materials

Become a Member