|
| |
MODULE R13
EXPERIMENTAL RESEARCH AND DESIGN
Experimental Research - An attempt by the researcher to
maintain control over all factors that may affect the result of an experiment. In doing
this, the researcher attempts to determine or predict what may occur.
Experimental Design - A blueprint of the procedure that enables
the researcher to test his hypothesis by reaching valid conclusions about relationships
between independent and dependent variables. It refers to the conceptual framework within
which the experiment is conducted.
Steps involved in conducting an experimental study
 | Identify and define the problem. |
| Formulate hypotheses and deduce their consequences. |
| Construct an experimental design that represents all the elements, conditions, and
relations of the consequences. |
1. Select sample of subjects.
2. Group or pair subjects.
3. Identify and control non experimental factors.
4. Select or construct, and validate instruments to measure outcomes.
5. Conduct pilot study.
6. Determine place, time, and duration of the experiment.
| Conduct the experiment. |
| Compile raw data and reduce to usable form. |
| Apply an appropriate test of significance. |
Essentials of Experimental Research
| Manipulation of an independent variable. |
| An attempt is made to hold all other variables except the dependent variable constant - control. |
| Effect is observed of the manipulation of the independent variable on the dependent
variable - observation. |
Experimental control attempts to predict events that will occur in the
experimental setting by neutralizing the effects of other factors.
Methods of Experimental Control
| Physical Control |
 | Gives all subjects equal exposure to the independent variable. |
| Controls non experimental variables that affect the dependent variable. |
| Selective Control - Manipulate indirectly by selecting in or out variables that
cannot be controlled. |
| Statistical Control - Variables not conducive to physical or selective
manipulation may be controlled by statistical techniques (example: covariance). |
Validity of Experimental Design
| Internal Validity asks did the experimental treatment make the difference in this
specific instance rather than other extraneous variables? |
| External Validity asks to what populations, settings, treatment variables, and
measurement variables can this observed effect be generalized? |
Factors Jeopardizing Internal Validity
| History - The events occurring between the first and second measurements in
addition to the experimental variable which might affect the measurement. |
Example: Researcher collects gross sales data before and after a
5 day 50% off sale. During the sale a hurricane occurs and results of the study may be
affected because of the hurricane, not the sale.
| Maturation - The process of maturing which takes place in the individual during
the duration of the experiment which is not a result of specific events but of simply
growing older, growing more tired, or similar changes. |
Example: Subjects become tired after completing a training
session, and their responses on the Posttest are affected.
| Pre-testing - The effect created on the second measurement by having a
measurement before the experiment. |
Example: Subjects take a Pretest and think about some of the
items. On the Posttest they change to answers they feel are more acceptable. Experimental
group learns from the pretest.
| Measuring Instruments - Changes in instruments, calibration of instruments,
observers, or scorers may cause changes in the measurements. |
Example: Interviewers are very careful with their first two or
three interviews but on the 4th, 5th, 6th become fatigued and are less careful and make
errors.
| Statistical Regression - Groups are chosen because of extreme scores of
measurements; those scores or measurements tend to move toward the mean with repeated
measurements even without an experimental variable. |
Example: Managers who are performing poorly are selected for
training. Their average Posttest scores will be higher than their Pretest scores because
of statistical regression, even if no training were given.
| Differential Selection - Different individuals or groups would have different
previous knowledge or ability which would affect the final measurement if not taken into
account. |
Example: A group of subjects who have viewed a TV program is
compared with a group which has not. There is no way of knowing that the groups would have
been equivalent since they were not randomly assigned to view the TV program.
| Experimental Mortality - The loss of subjects from comparison groups could
greatly affect the comparisons because of unique characteristics of those subjects. Groups
to be compared need to be the same after as before the experiment. |
Example: Over a 6 month experiment aimed to change accounting
practices, 12 accountants drop out of the experimental group and none drop out of the
control group. Not only is there differential loss in the two groups, but the 12 dropouts
may be very different from those who remained in the experimental group.
| Interaction of Factors, such as Selection Maturation, etc. - Combinations of
these factors may interact especially in multiple group comparisons to produce erroneous
measurements. |
Factors Jeopardizing External Validity or Generalizability
Pre-Testing -Individuals who were pretested might be less or more
sensitive to the experimental variable or might have "learned" from the pre-test
making them unrepresentative of the population who had not been pre-tested.
Example: Prior to viewing a film on Environmental Effects of
Chemical, a group of subjects is given a 60 item antichemical test. Taking the Pretest may
increase the effect of the film. The film may not be effective for a nonpretested group.
Differential Selection - The selection of the subjects determines
how the findings can be generalized. Subjects selected from a small group or one with
particular characteristics would limit generalizability. Randomly chosen subjects from the
entire population could be generalized to the entire population.
Example: Researcher, requesting permission to conduct
experiment, is turned down by 11 corporations, but the 12th corporation grant permission.
The 12th corporation is obviously different then the others because they accepted. Thus
subjects in the 12th corporation may be more accepting or sensitive to the treatment.
Experimental Procedures - The experimental procedures and
arrangements have a certain amount of effect on the subjects in the experimental settings.
Generalization to persons not in the experimental setting may be precluded.
Example: Department heads realize they are being studied, try to
guess what the experimenter wants and respond accordingly rather than respond to the
treatment.
Multiple Treatment Interference - If the subjects are exposed to
more than one treatment then the findings could only be generalized to individuals exposed
to the same treatments in the same order of presentation.
Example: A group of CPA’s is given training in working with
managers followed by training in working with comptrollers. Since training effects cannot
be deleted, the first training will affect the second.
Tools of Experimental Design Used to Control Factors Jeopardizing
Validity
| Pre-Test - The pre-test, or measurement before the experiment begins, can aid
control for differential selection by determining the presence or knowledge of the
experimental variable before the experiment begins. It can aid control of experimental
mortality because the subjects can be removed from the entire comparison by removing their
pre-tests. |
However, pre-tests cause problems by their effect on the second
measurement and by causing generalizability problems to a population not pre-tested and
those with no experimental arrangements.
| Control Group -The use of a matched or similar group which is not exposed to the
experimental variable can help reduce the effect of History, Maturation, Instrumentation,
and Interaction of Factors. The control group is exposed to all conditions of the
experiment except the experimental variable. |
| Randomization - Use of random selection procedures for subjects can aid in
control of Statistical Regression, Differential Selection, and the Interaction of Factors.
It greatly increases generalizability by helping make the groups representative of the
populations. |
| Additional Groups - The effects of Pre-tests and Experimental Procedures can be
partially controlled through the use of groups which were not pre-tested or exposed to
experimental arrangements. They would have to be used in conjunction with other pre-tested
groups or other factors jeopardizing validity would be present. |
The method by which treatments are applied to subjects using these
tools to control factors jeopardizing validity is the essence of experimental design.
Tools of Control
Internal Sources |
Pre-Test/
Post Test |
Control
Group |
Randomization |
Additional
Groups |
History |
|
X |
|
|
Maturation |
|
X |
|
|
Pre-Testing |
|
|
|
X |
Measuring
Instrument |
|
X |
|
|
Statistical
Regression |
|
X |
X |
|
Differential
Selection |
X |
|
X |
|
Experimental
Mortality |
X |
|
|
|
Interaction
of Factors |
|
X |
X |
|
External Sources |
|
|
|
|
Pre-Testing |
|
|
|
X |
Differential
Selection |
X |
|
X |
|
Procedures |
|
|
|
X |
Multiple
Treatment |
|
|
|
|
Experimental Designs
Pre-Experimental Design - loose in structure, could be biased
Aim
of the Research |
Name
of the Design |
Notation
Paradigm |
Comments |
| To attempt to
explain a consequent by an antecedent |
One-shot
experimental case study |
X » O |
An approach that
prematurely links antecedents and consequences. The least reliable of all experimental
approaches. |
| To evaluate the
influence of a variable |
One group
pretest-posttest |
O » X » O |
An approach that
provides a measure of change but can provide no conclusive results. |
| To determine the
influence of a variable on one group and not on another |
Static group
comparison |
Group 1: X » O Group 2: - » O |
Weakness lies in no
examination of pre-experimental equivalence of groups. Conclusion is reached by comparing
the performance of each group to determine the effect of a variable on one of them. |
True Experimental Design - greater control and refinement, greater
control of validity
Aim
of the Research |
Name
of the Design |
Notation
Paradigm |
Comments |
| To study the effect
of an influence on a carefully controlled sample |
Pretest-posttest
control group |
R - - [ O » X » O
[ O » - » O |
This design has been
called "the old workhorse of traditional experimentation." If effectively
carried out, this design controls for eight threats of internal validity. Data are
analyzed by analysis of covariance on posttest scores with the pretest the covariate. |
| To minimize the
effect of pretesting |
Solomon four-group
design |
R - - [ O » X » O
[ O » - » O
[- » X » O
[ - » - » O |
This is an extension
of the pretest-posttest control group design and probably the most powerful experimental
approach. Data are analyzed by analysis of variance on posttest scores. |
| To evaluate a
situation that cannot be pretested |
Posttest only
control group |
R - - [ X » O
[ - » O |
An adaptation of the
last two groups in the Solomon four-group design. Randomness is critical. Probably, the
simplest and best test for significance in this design is the t-test. |
Quasi-Experimental Design - not randomly selected
Aim
of the Research |
Name
of the Design |
Notation
Paradigm |
Comments |
| To investigate a
situation in which random selection and assignment are not possible |
Nonrandomized
control group pretest-posttest |
O » X » O O » - » O |
One of the strongest
and most widely used quasi-experimental designs. Differs from experimental designs because
test and control groups are not equivalent. Comparing pretest results will indicate degree
of equivalency between experimental and control groups. |
| To determine the
influence of a variable introduced only after a series of initial observations and only
where one group is available |
Time series
experiment |
O » O » X » O » O |
If substantial
change follows introduction of the variable, then the variable can be suspect as to the
cause of the change. To increase external validity, repeat the experiment in different
places under different conditions. |
| To bolster the
validity of the above design with the addition of a control group |
Control group time
series |
O » O » X » O » O O » O » - » O » O |
A variant of the
above design by accompanying it with a parallel set of observations without the
introduction of the experimental variable. |
| To control history
in time designs with a variant of the above design |
Equivalent
time-samples |
[X1 » O1] »[X0 » O2] » [x1 » O3] |
An on-again,
off-again design in which the experimental variable is sometimes present, sometimes
absent. |
Correlational and Ex Post Facto Design
Aim
of the Research |
Name
of the Design |
Notation
Paradigm |
Comments |
| To seek for
cause-effect relationships between two sets of data |
Causal-comparative
correlational studies |
-»
Oa ¥ Ob
«- |
A very deceptive
procedure that requires much insight for its use. Causality cannot be inferred merely
because a positive and close correlation ratio exists. |
| To search backward
from consequent data for antecedent causes |
Ex post facto
studies |
|
This approach is
experimentation in reverse. Seldom is proof through data substantiation possible. Logic
and inference are the principal tools of this design |
Leedy, P.D. (1997). Practical research: Planning and design (6th
ed.). Upper Saddle River, NJ: Prentice-Hall, Inc., p. 232-233.
SELF ASSESSMENT
1. Define experimental research.
Define experimental design.
2. List six steps involved in conducting an experimental study.
3. Describe the basis of an experiment.
4. Name three characteristics of experimental research.
5. State the purpose of experimental control.
6. State three broad methods of experimental control.
7. Name two type of validity of experimental design.
8. Define eight factors jeopardizing internal validity of a research
design.
9. Define four factors jeopardizing external validity.
10. Describe the tools of experimental design used to control the
factors jeopardizing validity of a research design.
11. Define the essence of experimental design.
12. Name and describe the four types of experimental designs.
        
|
