Research Design
Unlock all answers in this set
Unlock answersquestion
Typical Research Process
answer
The typical quantitative study involves a series of steps, one of which is the statistical analysis
question
step 1:
answer
research questions and hypotheses
question
step 2:
answer
operationalize and choose measures
question
step 3:
answer
choose a research design
question
step 4:
answer
analyze data
question
step 5:
answer
draw conclusions
question
causal relationship
answer
-the goal of virtually every quantitative research study is to determine between two variables -ideally a causal relationship would be established -that is, does a change in X definitively lead to a predictable change in Y
question
Confounding Variables
answer
-Variables always have multiple causes -we would like to isolate only the two variables of interest and exclude the effects of all other variables
question
Unwanted variables
answer
are called confounding or nuisance variables
question
Internal validity
answer
the validity of findings with the research study; the technical soundness of a study, particularly concerned with the control of extraneous influences that might effect the outcome
question
External validity
answer
the degree to which the findings can be inferred to the population of interest or to other populations or settings; the generalizability of the results
question
Both internal and external validity
answer
are import in a study but they are frequently at odds with one another in planning and designing a study
question
Internal validity is
answer
considered the basic minimum for experimental research
question
Internal validity facts
answer
-is the basic minimum without which any study is not interpretable -particularly important in experimental studies -did, in fact, the experimental treatment (X) produce a change in the dependent variable (Y) -lends itself to highly controlled, laboratory settings
question
To gain internal validity
answer
the researcher attempts to control everything and eliminate possible extraneous influences
question
Threats to Internal Validity
answer
history, maturation, testing, instrumentation, statistical regression, selection bias, experimental mortality, interaction among factors
question
History
answer
events occurring during the experiment that are not part of the treatment
question
maturation
answer
biological or psychological processes within participants that may change due to the passing of time (age, fatigue, hunger)
question
Testing
answer
the effects of one test upon subsequent administrations of the same test
question
Instrumentation
answer
changes in testing instruments, raters, or interviewers including lack of agreement within and between observers
question
Statistical regression
answer
the fact that groups selected on the basis of extreme scores are not as extreme on subsequent testing
question
Selection bias
answer
identification of comparison groups in other than a random manner
question
Experimental mortality
answer
loss of participants from comparison groups due to nonrandom reasons
question
Interaction among factors
answer
factors can operate together to influence experimental results
question
External validity facts
answer
-generalizability of results to what populations, settings, or treatment variables can the results be generalized -concerned with real-world applications -random selection is the key to controlling most threats to external validity
question
External validity is generally
answer
controlled by selecting subjects, treatments, experimental situations, and tests to be representative of some larger population
question
Two types of External Validity
answer
Population validity and Ecological Validity
question
Population Validity
answer
refers to the extent to which the results can be generalized from the experimental sample to a defined population
question
Ecological Validity
answer
refers to the extent to which the results of an experiment can be generalized from the set of environmental conditions in the experiment to other environmental conditions
question
Common sources of error
answer
many possible sources of error can cause the results of a research study to be incorrectly interpreted. the following sources of error are more specific threats to validity of a study than those described previously
question
Threats to External Validity
answer
interaction effects of testing, selection bias, reactive effects of experimental setting, multiple-treatment interference
question
Interaction effects of testing
answer
the fact that the pretest may make the participants more aware of or sensitive to the upcoming treatment
question
Selection bias is
answer
when participants are selected in a manner so they are not representative of any particular population
question
Reactive effects of experimental setting
answer
the fact that treatments in constrained laboratory settings may not be effective in less constrained, real-world settings
question
Multiple-treatment interference
answer
when participants receive more than one treatment, the effects of previous treatments may influence subsequent ones
question
Additional Threats to External Validity
answer
-Reaction effects: Hawthorne effect, placebo effect, John Henry effect, rating effect -Experimental effect: experimental bias effect
question
Hawthorne Effect
answer
-a specific type of reactive effect in which merely being a research participant in an investigation may affect behavior -suggests that, as much as possible participants should be unaware they are in an experiment and unaware of the hypothesized outcome
question
Placebo effect
answer
participants may believe that the experimental treatment is supposed to change them, so they respond to the treatment with a change in performance
question
John Henry Effect
answer
a threat to internal validity wherein research participants in the control group try harder just because they are in the control group
question
Rating effect
answer
variety of errors associated with ratings of a participant or group -halo effect -overrater error -underrater error -central tendency error
question
Experimenter bias effect
answer
the intentional or unintentional influence that an experimenter (researcher) may exert on a study
question
Types of designs
answer
-non-experimental -true experimental -quasi-experimental
question
Non-experimental designs/ Descriptive Research
answer
-weak experimental designs in terms of control -no random sampling -threats to internal and external validity are significant problems -many definite weaknesses -example: one-group pretest/posttest design
question
True experimental designs
answer
-best type of research design because of their ability to control threats to internal validity -utilizes random selection of participants and random assignment to groups -example: pretest/posttest control group design
question
Quasi-experimental designs
answer
-lack either random selection of participants or random assignment to groups -lack some of the control of true experimental designs, but are generally considered to be fine example: nonequivalent group design
question
Research Design Basics
answer
-research design, or blueprint, for a study can be drawn out using a simple set of symbols proposed by Campbell & Stanley (1963)
question
X =
answer
an intervention was administered (i.e., a treatment, program, IV, etc)
question
O =
answer
data were collected (i.e., observed, measured, etc.)
question
Between Subjects design
answer
-also called between groups or cross-sectional -each participant is in one (&only one) of the treatments/conditions -different groups of participants are in each treatment/condition -typically used to study \"differences\"-- when in application, a participant will usually be in one treatment/condition or another
question
Characteristics of between-subject design
answer
-the defining characteristic of a between-subjects design is that it compares separate groups of individuals -it allows only one score per participant (every score represents a separate, unique participant) -each score represents a separate participant, a between subjects design is often called an independent-measures design
question
More characteristics of between-subject design
answer
-between-subjects experimental design requires a separate, independent group of individuals for each treatment condition -individuals are assigned to groups using a procedure that attempts to create equivalent groups
question
Goal of between-subjects experiment
answer
is to determine whether differences exist between two or more treatment conditions (a researcher may want to compare two teaching methods (two treatments) to determine whether one is more effective than the other)
question
Advantages of between-subjects design
answer
-each individual score is independent of the other scores -participant's score is not influenced by such factors as: -practice or experience gained in other treatments -fatigue or boredom from participating in a series of treatments -contrast effects that result from comparing one treatment to another
question
Disadvantages of between-subjects designs
answer
-large number of participants (problem with special populations) -individual differences: -characteristic that differ from one participant to another are called individual differences -individual differences can become confounding variables -individual differences can produce high variability in scores
question
Confounding variables in between subjects design
answer
Individual differences: -participant characteristics differ from one group to another -participants in one group may be older, smarter, taller, than the participants in another group environmental variables: -characteristics of the environment differ between groups -one group may be tested in a large room and another group in a smaller room
question
Equivalent groups
answer
in a between-subjects experimental design, the researcher does have control over the assignment of individuals to groups -the separate groups must be: - created equally -treated equally (except for the treatment conditions -composed of equivalent individuals
question
Limiting confounding by individual differences
answer
-random assignment,matching groups (matched assignment), holding variables constant
question
Random assignment (randomization)
answer
a random process is used to assign participants to groups
question
Matching groups (matched assignments)
answer
involves assigning individuals to groups so that a specific variable is balance or matched across the groups (IQ)
question
Holding variables constant
answer
simply hold the variable (restrict the participant to those with with IQs between 100-110)
question
Within and between treatments variability
answer
advantage: -variability between treatments -it can be increased by increasing differences between conditions (levels) disadvantages: -variability within treatments -it is caused by individual differences -should be minimized
question
Within and between treatments variability (2)
answer
minimizing variability within treatments -standardize procedures and treatments setting -limit individual differences by holding a participant variable constant -random assignment and matching -sample size: -using a large sample can help minimize the problems associated with high variability
question
Other threats to internal validity of between-subject designs
answer
assignment bias, differential attrition, on diffusion or imitation of treatment, compensatory equalization, compensatory rivalry, resentful demoralization
question
Assignment bias
answer
-groups of participants are different before the treatments -the group assignment process produces groups within noticeably different characteristcs
question
Differential attrition
answer
-attrition refers to participant withdrawl from a research study before it is completed -differential attrition refers to differences in attrition rates from one group to another and can threaten the internal validity of a between-subjects experimental (effectiveness of a dieting program)
question
Diffusion or limitation of treatment
answer
refers to the spread of the treatment effects from the experimental group to the control group (new depression therapy)
question
Compensatory equalization
answer
occurs when an untreated group learns about the treatment being received by another group and demands the same or equal treatment (watching Batman in violent TV group)
question
Compensatory rivalry
answer
occurs when an untreated learns about the treatment received by another group and then works extra hard to show that they can perform just as well as the individuals receiving the special treatment
question
Resentful demoralization
answer
-opposite of compensatory rivalry -occurs when an untreated group learns about the treatment received by another group and is less productive and less motivated because they resent the expected superiority of the treated group
question
Applications and statistical analyses of between-subjects design
answer
-comparing only two groups of participants -comparing means for more than two groups
question
Comparing only two groups of participants
answer
-this design is referred to as the single-factor two-group design or simply two group design -an independent-measures t test is used to determine whether there is a significant difference between the means
question
Comparing means for more than two groups
answer
-single factor multiple group design may be used and analysis of variance (ANOVA) would be used for statistical analysis -adding extra groups to a research study tends to reduce the differences between groups
question
Within-Groups Designs
answer
-also called within-subjects, repeated measures, or longitudinal, each participant is in all (every one) of the treatment/conditions -one group of participants, each one in every treatment/condition -typically used to study \"changes\" when, in application, a participant will usually be moving from one condition to another
question
Characteristics of within-subjects design
answer
-a within subjects experimental design compares two or more different treatment conditions (or compares treatment and control) by observing or measuring the same group of individuals in all of the treatment conditions being compared -a within-subjects design looks for differences between treatment conditions within the same group of participants -a within subjects design is often called a repeated-measures design because the research study repeats measurements of the same individuals under different conditions -it is used in experimental situations comparing different treatment conditions and also to investigate changes occurring over time
question
Advantages of within-subjects designs
answer
-it requires relatively few participants -it essentially eliminated all of the problems based on individual differences that are the primary concern of a between-subjects designs -within-subjects design has no differences between groups -each individual serves as his or her own control or baseline
question
Disadvantages of within-subjects
answer
-time related problems: -participant attrition, history, maturation, instrumentation, statistical regression
question
Disadvantages of within subjects design
answer
carry over effects, progressive error
question
Carryover effects
answer
-changes in behavior or performance that are caused by participation in an earlier treatment condition -carryover effects exist whenever one treatment condition produces a change in the participants that affects their scores in subsequent treatment conditions (e.g. new skill from treatment 1 can influence results in treatment 2)
question
Progressive error
answer
changes in participant's behavior or performance that are related to experience over time in a research study but not related to a specific treatment or treatments (e.g. practice effects and fatigue)
question
Dealing with time related threats and order effects
answer
-controlling time: if the different treatment conditions are scheduled over a period of weeks, the chances greatly increase that the results will be influenced by some outside event (history) or maturation or change in the measurement instrument When a within subjects design is not a good idea: -e.g. comparing two methods of teaching reading to first-grade children (carryover effects)
question
Dealing w/ time related threats and order effects
answer
Counterbalancing: -involves changing the order in which treatment conditions are administered from one participant to another -the goal is to use every possible order of treatment with an equal number of individuals participating in each sequence -the purpose of counterbalancing is to eliminate the potential for confounding by disrupting any systematic effects from factors related to time or the order of treatments -e.g. with two treatments one half of the participants begins in treatment 1, then moves to treatment 2 and the other half begins in treatment 2, then receives treatment1
question
Applications and statistical analyses of within-subjects designs
answer
Two treatment designs, multiple treatment designs
question
Two treatment designs
answer
-a repeated-measures t test or an analysis of variance can be used to evaluate the statistical significance of the mean difference -if the data are measured on an ordinal scale, a Wilcoxon test can be used to evaluate significant differences
question
Multiple-treatment designs
answer
-with too many treatment conditions, the distinction between treatments may become too small to generate significant differences in behavior -statistical analysis - repeated-measures analysis of variance to test for any significant differences among the treatment means
question
Matched-subjects designs
answer
-each individual in one group is matched with a participant in each of the other groups -the matching is done so that the matched individuals are equivalent with respect to a variable that the researcher considers to be relevant to the study (e.g. IQ) -maintains all the advantages of between-subjects and within-subjects designs without the limitations of either (e.g. eliminates individual differences, time-related factors and order effects)
question
Properties of True Experiments
answer
-True experiments provide the highest degree of internal validity possible -Virtually all confounding variables should be controlled via random assignment -Because confounds are controlled, a causal link can only be claimed when using a true experimental design
question
Limitations of Random Assignment
answer
-If random assignment is so advantageous, what prevents its use? -It may not be ethical to expose subjects to or withhold from them a treatment --e.g., What are the effects of alcohol use during pregnancy? -Some research questions are not amenable to random assignment --Comparisons of naturally existing groups (e.g., males and females, ethnicity, socioeconomic differences)
question
If only True experiments can be causally interpreted, why even bother running non-experiments?
answer
1st Remember that we can't always run a true experiment ! Lots of variables we care about can't be RA & manip - gender, family background, histories and experiences, personality, etc. Even if we can RA & manip, lots of studies require long-term or field research that makes ongoing equivalence (also required for causal interp) very difficult or impossible. We would greatly limit the information we could learn about how variables are related to each other if we only ran studies that could be causally interpreted.
question
Categorical vs. Continuous Variables
answer
-In practice, it is not usually necessary to make such fine distinctions between measurement scales -To distinctions, categorical and continuous are usually sufficient -Categorical variables consist of separate, indivisible categories -Continuous variables yield values that fall on a numeric continuum, and can (theoretically) take on an infinite number of values
question
Nominal Scale
answer
-Observations fall into different categories or groups -Differences among categories are qualitative, not quantitative -Examples: Gender Ethnicity Counseling method (cognitive vs. humanistic) Retention (retained vs. not retained)
question
Ordinal Scale
answer
-Categories can be rank ordered in terms of amount or magnitude -Categories possess an inherent order, but the amount of difference between categories is unknown -Examples: Class standing Letter grades (A,B,C,D,F) Likert-scales (SD, D, N, A, SA)
question
Interval Scale
answer
-Categories are ordered, but now the intervals for each category are exactly the same size -That is, the distance between measurement points represent equal magnitudes (e.g., the distance between point A and B is the same as the distance between B and C) -Examples: Fahrenheit scale of measuring temperature Chronological scale of dates (1997 A.D.) Standard scores (z-scores)
question
Ratio Scale
answer
-Categories are ordered, but now the intervals for each category are exactly the same size -That is, the distance between measurement points represent equal magnitudes (e.g., the distance between point A and B is the same as the distance between B and C) -Examples: Fahrenheit scale of measuring temperature Chronological scale of dates (1997 A.D.) Standard scores (z-scores)
question
LOOK AT SLIDES
answer
62,61,55,54,53,52
