|
| |
| Practical Guidelines for Planning Evaluations Part 2 |
|
Stufflebeam’s
Structure
 | Focusing the evaluation |
| Collecting information |
| Organizing information |
| Analyzing information |
| Reporting information |
| Administering the evaluation |
Planning
| Specifying information needs in advance |
| Identifying appropriate sources of information |
| Identifying appropriate methods and instruments for
collecting information |
| Determining appropriate conditions for collecting
information |
| Determining appropriate methods and techniques for
organizing and analyzing information |
| Determining appropriate ways to interpret and report
evaluation findings |
Collecting Information
| Sampling procedures |
| How information will be collected |
| When information will be collected |
Sampling
| Randomization |
| Stratifying |
| Proportional |
| Clustering |
| Systematic |
| Purposive |
Randomization
| Specify the population |
| Determine sample size |
| List all members of population |
| Assign all individuals on the list a consecutive number
from zero to the required number |
| Select an arbitrary number in the table of random numbers |
| For the selected number, look at only the appropriate
number of digits. Work through the column until you attain the desired sample size |
Stratifying
| A procedure that includes identifying subgroups from the
population in the proportion that they exist in the population. The purpose is to
guarantee the desired distribution among the selected subgroups. |
Proportional
| Individuals are selected at random from the subgroup in
proportion to the actual size of the group in the total population. This can provide a way
to achieve even greater representativeness in the sample of the population. This method is
used in combination with stratified and cluster sampling. |
Clustering
| Groups of elements (clusters) are selected from the
population instead of individuals. This is the most used method of sampling in educational
research. |
Systematic
| A procedure in which a sample is drawn from certain
intervals on a list. The researcher takes every nth name (n = size of population divided
by desired sample size) on a list of the population until the desired sample size is
reached. |
Purposive
| A type of non-probability sampling characterized by the
use of judgment and a deliberate effort to obtain a representative sample |
Sample Size Determination
| Nature of population |
| Type of investigation |
| Degree of precision required |
Data Gathering Procedures
| Instrument development |
| Instrument description |
| Validity |
| Reliability |
Validity
| Construct Validity - Is the underlying construct or
theoretical foundation of the survey consistent with research and information on the
topic? |
| Content Validity - Does the content of the items in the
instrument accurately reflect the underlying construct? |
| Criterion-related Validity - Does the instrument contain
the proper criteria for measuring the traits or constructs of interest? |
| Face Validity - Does the instrument look like it will
measure what it is supposed to measure? |
Reliability
| Test - Retest |
| Alternative Forms |
| Split Halves |
| Internal Consistency |
Can an Instrument Be...
| Valid but not reliable? |
| Reliable but not valid? |
Questionnaire
| A written or printed form used in gathering information
on some subject or subjects consisting of a list of questions to be submitted to one or
more persons |
Advantages
| Economy |
| Uniformity of questions |
| Standardization |
Disadvantages
| Respondent’s motivation is difficult to assess,
affecting the validity of response. |
| Unless a random sampling of returns is obtained, those
returned completed may represent biased samples. |
Factors Affecting Returns
| Length of questionnaire |
| Reputation of sponsoring agency |
| Complexity of questions asked |
| Relative importance of study as determined by potential
respondent |
| Extent to which respondent believes his responses are
important |
| Quality and design of questionnaire |
| Time of year questionnaires are sent out |
Characteristics
| Deals with a significant, clearly stated topic |
| Seeks information not obtainable from other sources |
| As short as possible |
| Attractive in appearance, neatly arranged, and clearly
duplicated or printed |
| Directions clear and complete, important terms defined, each question has only one idea |
| Objective questions |
| Good psychological order, general to more specific |
| Easy to tabulate and interpret |
Limitations of Inferring Attitude
| May hide real attitude and express socially acceptable
opinions |
| May not really know feelings about a social issue |
| May never have considered the idea seriously |
| May not know attitude about a situation in the abstract
and so may be unable to predict his reaction or behavior until confronted with a real
situation |
Interview
| A direct face-to-face attempt to obtain reliable and
valid measures in the form of verbal responses from one or more respondents. |
Advantages
| Allow interviewer to clarify questions |
| Can be used with young children and illiterates |
| Allows informants to respond in any manner |
| Allows interviewers to observe verbal and non-verbal
behavior of respondents |
| Means of obtaining personal information, attitudes,
perceptions, and beliefs |
| Reduces anxiety so that potentially threatening topics
can be studied |
Disadvantages
| Unstructured interviews often yield data too difficult to
summarize or evaluate. |
| Training interviewers, sending them to meet and interview
their informants, and evaluating their effectiveness all add to the cost of the study. |
Structured Interviews
| Same questions presented in same manner and order to each
subject |
| Choice of alternative answers restricted to predetermined
list |
| Same introductory and concluding remarks |
| More scientific in nature |
| Introduce controls that permit formulation of scientific
generalizations |
Unstructured Interviews
| Few restrictions |
| If preplanned questions are asked, they are altered to
suit the situation and subject |
| Subjects are encouraged to express their thoughts freely |
| Few questions asked to direct answers |
| Sometimes, information is obtained in such a casual
manner that respondents are not aware they are being interviewed |
Other Data-Gathering Tools
| Inventories |
| Scales |
| Direct observation |
| Semantic differential |
| Q methodology |
| Delphi technique |
| Nominal group technique |
| Conference |
Focus Groups
| A group discusses certain topics, commonly for one to two
hours. The interviewer/leader/moderator raises various issues, focusing the discussion on
matters of interest to the researcher according to an outline. Analysis of gathered
information attempts to discern patterns and trends that develop among participants and
across focus groups. |
Characteristics
| Involve people. It must be small enough for everyone to
have opportunity to share yet large enough to provide diversity of perceptions. |
| Conducted in series. Multiple groups with similar
participants are needed to detect patterns and trends across groups. |
| Possess certain characteristics. Participants are reasonably
homogeneous and unfamiliar with each other. |
| Provide data. Focus groups pay attention to perceptions
of users and consumers of solutions, products, and service. They are not intended to
develop consensus, to arrive at an agreeable plan, or to make decisions about which course
of action to take. |
| Produce qualitative data. |
| Focused discussion. |
Advantages
| It is a socially oriented research procedures |
| The format allows the moderator to probe |
| Discussions have high face validity |
| Discussions can be relatively low cost |
| The format can provide speedy results |
| They enable the researcher to increase the sample size of
qualitative studies |
Disadvantages
| The researcher has less control in the group interview as
compared to the individual interview |
| Data are more difficult to analyze |
| The technique requires carefully trained interviewers |
| Groups can vary considerably |
| Groups are difficult to assemble |
| The discussion must be conducted in an environment
conducive to conversation |
Questions
| Use open-ended questions |
| Avoid dichotomous questions (yes or no) |
| Why? is rarely asked |
| Use "think back" questions |
| Carefully prepare focus questions |
| Ask uncued questions 1st, cued questions 2nd |
| Consider standardized questions |
| Focus the questions |
| Be cautious of serendipitous questions |
Focus Group Analysis
| Consider the words |
| Consider the context |
| Consider the internal consistency |
| Consider the frequency or extensiveness of comments |
| Consider the intensity of comments |
| Consider the specificity of responses |
| Find the big ideas |
Systematic Notification Procedure
| Set meeting times for group interviews |
| Contact potential participants via phone (or in person) |
| Send a personalized invitation |
| Phone (or contact) each person the day before the
focus group |
Selection Strategies
| The list |
| Piggyback focus groups |
| On location |
| Nominations |
| Snowball samples |
| Screening/selection services |
| Random telephone screening |
| Ads in newspapers and bulletin boards |
Incentives for
Participation
| Money |
| Food |
| Gifts |
| Positive, upbeat invitation |
| Opportunity to share opinions |
| Enjoyable, convenient and easy to find meeting location |
| Involvement in important research project |
| Build on existing community, social or personal
relationship |
Data Analysis
Procedures
Total Design Method
| Principle
is to increase perception of possible rewards, decrease perceived costs, and encourage the
likelihood that recipient of questionnaire trusts the rewards will on balance outweigh
costs
|
Increase Rewards
| Make answering interesting |
| Use consulting approach |
| Support values |
| Token incentives |
Decrease Costs
| Time |
| Embarrassment |
| Mental effort |
| Subordination |
Promote Trust
| Show investment |
| Legitimate, trustworthy sponsorship |
TDM Elements
| Respondent-friendly questionnaire |
| Multiple contacts - initial mailing, reminder postcard,
replacement questionnaire, second replacement questionnaire, telephone reminder |
| Letterhead stationary, personalization, emphasis on
social utility, and respondent’s importance |
| Cover design |
TDM Limitations
| "One size fits all" approach inconsistent with
exchange theory |
| Some elements may be superfluous |
| Ignores some past research, notably financial incentives |
Time-Related
Project Management
| PERT Systems - Program Evaluation and Review Technique |
| Gantt Charts |
PERT Network
"Blueprint" of the
network of activities and events for a given project
| Outlines a comprehensive project - useful for internal
planning for a complex evaluation |
| Can estimate time-frames if the amount of time is
indicated on the solid lines |
Gantt Chart
Simple
display that includes proportionate, chronologically scaled time frames for each
evaluation task
| Communicates evaluation plans with a non-technical
audience |
| Helps in time management |
Needed Resources
| Evaluation staff salary and benefits |
| Consultants |
| Travel and per diem (staff and consultants) |
| Communications (postage, telephone calls, etc.) |
| Printing and duplication |
| Data processing |
| Printed materials |
| Supplies and equipment |
| Subcontracts |
| Overhead (facilities, utilities) |
Quality Control
| Evaluating the evaluation plan |
| Monitoring adherence to the evaluation design |
| Revising the evaluation design as needed |
Total Design Method
Principle is
to increase perception of possible rewards, decrease perceived costs, and encourage the
likelihood that recipient of questionnaire trusts the rewards will on balance outweigh
costs
Increase Rewards
| Make answering interesting |
| Use consulting approach |
| Support values |
| Token incentives |
    
| |
|
