Data Tools 2
Home Up Data Tools Data Tools 2 Reliability Historical Descriptive Experimental Qualitative

Research Design in Occupational Education
Copyright 1997. James P. Key. Oklahoma State University
Except for those materials which are supplied by different departments of the University
(ex. IRB, Thesis Handbook) and references used by permission.





Inventories are instruments that attempt to "take stock" of one or more aspects of an individual’s behavior rather than to measure in the usual sense. Inventories have been used in educational research to obtain trait descriptions of certain defined groups such as underachievers, dropouts, members of minority groups, etc. They have also been used in research concerned with interrelationships between personality traits and such variables as intelligence, achievement, and attitudes.




The information obtained may be superficial or biased.
The difficulty of validating inventories limits their use as scientific instruments.

To validate an inventory, one demonstrates that the scores obtained on it agree highly with some other reliable device for measuring the factor under study.



A scale is a set of numerical values assigned to subjects, objects, or behaviors for the purpose of quantifying the measuring qualities. Scales are used to measure attitudes, values, and interests. They measure the degree to which an individual possesses the characteristic of interest.

Used as a measuring instrument.
Used to indicate a measuring instrument.
Used to indicate the systematized numerals of the measuring instrument.
Tests are scales, but scales are not tests.

Attitude is an integral part of the personality (as to the way we think, feel, perceive, and behave toward a cognitive objective). Attitude scales have three major types which are the summated rating scales, equal appearing interval scales, and cumulative or Guttman scales.

The summated rating scale is a set of attitude items, all of which are considered of approximately equal attitude value and to which the subjects respond with degrees of agreement or disagreement (intensity). The scores of the items of such a scale are summed or summed and averaged to yield an individual’s attitude score.


U is the universe of items - means that there is no scale of items as such, but the individuals responding to the items are scaled
Intensity of attitude expression - Respondents are asked to indicate whether they agree very strongly (7), agree strongly (6), agree (5), disagree (3), disagree strongly (2), disagree very strongly (1), no response (4)

Equal-Appearing Interval Scales are used to scale the attitude items. Each item is assigned a scale value, and the scale value indicates the strength of attitude of an agreement response to the item. The universe of items is considered to be an ordered set, items differ in scale value. The scaling procedure finds these scale values. In addition, the items of the final scale to be used are so selected that the intervals between them are equal, a most important and desirable psychometric feature.

Cumulative or Guttman Scale consists of a relatively small set of homogeneous items that are unidimensional (or supposed to be). Unidimensional scale measures one variable and only one variable. The scale gets its name from the cumulative relation between items and the total scores of individuals.

Summated Rating Scales concentrate on the subject and their places on the scale, most useful in behavioral research. Equal-Appearing Interval Scales concentrate on the items and their place on the scale. Cumulative Scales concentrate on the scalability of sets of items and on the scale position of individuals.

The value scale is a culturally weighted reference for a thing or things for people, for institutions, or for some kind of behavior. Simply put, values express the good, bad, shoulds, and oughts of human behavior.

The objective scale and items are independent and non-independent. The types of objective scale include:

Agreement-Disagreement Items

Those permitting one of two possible responses

Those permitting one of three or more possible responses

Those permitting more than one choice of three or more possible responses

Rank Order Items
Forced Choice Items


Direct Observation

Direct observation is a measuring instrument used to measure such traits as self-control, cooperativeness, truthfulness, and honesty. In many cases, systematic direct observation of behavior is the most desirable measurement method. An investigator identified the behavior of interest and devises a systematic procedure for identifying, categorizing, and recording the behavior in either a natural or "staged" situation.

Systematic Direct Observation

Selecting the aspect of behavior to be observed. An observer cannot notice everything that happens. Select first the behavior upon which the investigator wishes to focus.
Defining the behavior that fall within a category. Know in advance what will or will not be classified as aggressive behavior, problem-solving behavior or any other classification of interest.
Training observers. Observers must be trained for uniformity of interpretation and standard application of the observation categories.
Quantifying observations. An observation system must include a standard method for counting behaviors.
Developing procedures to facilitate recording. Procedures to facilitate recording must be developed to avoid errors of selectivity of memory. A useful technique is to develop a coding plan that enables observers to record their observations with a single letter or digit rather than in narrative form.


Semantic Differential

The semantic differential (SD) is a method of observing and measuring the psychological meaning of words, usually concepts. An actual SD consists of a number of scales, each of which is a bipolar adjective pair. The bipolar adjectives are usually seven-point rating scales. Each scale measures one or two of the basic dimensions or factors that Osgood and his colleagues have found to be behind the scales: Evaluative, Potency, Activity.

Evaluative scales include pairs such as good-bad, bitter-sweet, large-small, and dirty-clean. A second cluster has adjectives that seem to share strength or potency ideas (strong-weak, rugged-delicate). A third scale is called activity because its adjectives seem to express motion and action (fast-slow, hot-cold).

The first step in construction and use of SD is to choose the concepts or other stimuli to be rated with bipolar adjectives. The researcher needs to choose a number of concepts that are relevant to the research problem. A sample of concepts must be judiciously chosen to represent some part of the semantic space. The second step is to select appropriate scales or adjective pairs. Two main criterion determine the selection including factor representatives and relevance to the concept used.

It is a combination of the usual types of rating scales with factor analysis.
The technique is extremely flexible and simple to construct, administer, and score.
The semantic differential is subject to all of the limitation of rating scales, the possibilities of faking responses, acquiescing (tendency to place marks in the middle position), and having to mark a concept on a meaningless scale (is honesty more purple or green?).
Validity and reliability of semantic differential scales are generally satisfactory. The validity studies show correlation coefficients of approximately .80 between the semantic differential ratings and Thurstone, Likert, and Guttman scales. The test-retest reliability of the semantic differential is reported to be about .90.
The semantic differential is a useful technique for measuring attitudes toward objects.


Below describe how you feel about agriculture by placing a check in one of the seven spaces between each word pair.

traditional    _____  _____  _____  _____   _____  _____ progressive

simple _____  _____  _____  _____  _____  _____ complicated

like me _____  _____  _____  _____  _____   _____ unlike me

friendly _____  _____  _____  _____  _____   _____ unfriendly

challenging _____  _____  _____  _____  _____   _____ simple

serious _____  _____  _____  _____  _____   _____ humorous

stale _____  _____  _____  _____  _____  _____ fresh

work _____  _____  _____  _____  _____  _____ fun

relaxed _____  _____  _____  _____  _____   _____ tense

clear _____  _____  _____  _____  _____  _____ confusing

unstructured _____  _____  _____  _____  _____   _____ structured

bright _____  _____  _____  _____  _____  _____ dull

systematic _____  _____  _____  _____  _____   _____ unsystematic

masculine _____  _____  _____  _____  _____   _____ feminine

active _____  _____  _____  _____  _____  _____ passive

accepting _____  _____  _____  _____  _____   _____ rejecting

closed _____  _____  _____  _____  _____  _____   open


Q Methodology

Q methodology is used for the study of intrapersonal relations. The Q technique is a set of procedures used to implement Q methodology. It is well suited to small samples and makes use of the correlation between persons rather than the correlation between tests. The Q technique centers particularly in the sorting of decks of cards called Q sorts and in the correlations among the responses of different individuals to the Q sorts.

Steps in the application of Q methodology

Begin with the development of items, usually 60 to 150, from theoretical formulation.
Each statement is printed on a separate card and presented to respondents who sort them into piles. Generally, there are 11 piles, ranging from "most applicable to oneself" to "least like oneself."
The Q technique requires each respondent to sort his deck of cards into a symmetrical distribution with the number of cards to be sorted into each pile fixed by the investigator.

                                        Most Applicable                                                     Least Applicable














# cards to be placed in each pile














After sorting, each statement is assigned a score corresponding to the pile into which it is placed.
If a number of individuals are involved in a study using just one sort, the correlation between the individuals for each item or statement can be computed and a factor analysis run to determine which individuals are responding similarly.
An alternative procedure is to have each person sort the deck numerous times under different conditions, such as for idealized and actual self. The correlation between two sorts can then be used as a measure of adjustment.


It is theoretically oriented.
Sortings can be analyzed quite objectively without entirely sacrificing the richness of the usual clinical and much less objective methods.
Q can test the effects of independent variables on complex dependent variables.
It is useful in exploratory research.
It is interesting to subjects because the method is realistic as well as challenging.



It is not a method well suited to cross-sectional or large sample purposes.
Q placements violate the independence assumption in all forced-choice procedures.
Q is a forced-choice procedure which is unnatural because it requires the subject to conform to an unreasonable requirement.
Important information on elevation and scatter may be lost with the forced Q procedure.


The Conference technique is a face-to-face discussion of a topic of interest.

Experts are brought together at a common site.
The group brainstorms to generate as many ideas on the problem as possible. The only rule regarding this step is that there are no negative reactions to any suggestions.
The experts then evaluate and rate the suggestions.
The most popular responses are determined, and an arbitrary number are chosen based on natural breaks or logic.
Finally, the group discusses the strengths and weaknesses of the top suggestions and ranks the final choices.

One major drawback to this method of data gathering is the influence of personalities as a strong factor in determining consensus.


Delphi Technique

The Delphi technique is used in the planning process, especially with appraising the future political, economic, and social environment, ascertaining the role of the organization in this environment, and anticipating and perceiving the needs and requirements of client groups.

The Delphi technique is a means of securing expert convergent opinion without bringing the experts together in face-to-face confrontation. This opinion of experts is usually gained through the use of successive questionnaires and feedback with each round of questions being designed to produce more carefully considered group opinions.


A questionnaire is mailed to respondents who remain anonymous to one another. The first questionnaire may call for a list of opinions involving experienced judgment, a list of predictions, or a list of recommended activities.
On the second round, each expert receives a copy of the list and is asked to rate or evaluate each item by some such criterion as importance, probability of success, etc.
The third questionnaire, which includes the list and ratings, indicates the consensus, if any, and asks the experts either to revise their opinion or specify their reasons for remaining outside the consensus.
The fourth questionnaire includes lists, ratings, consensus, and minority opinions. It provides the final chance for revision of opinions.


It allows planners to get the views in a broad perspective rather than from an isolated point of view.
Delphi in combination with other tools is a very potent device for teaching people to think about the future of education in much more complex ways than they ordinarily would.
It is a useful instrument even for a general teaching strategy.
It is a planning tool which may aid in probing priorities held by members and constituencies of an organization.
Delphi saves time and travel which are required to bring people together for a conference.
Delphi prevents personality biases from affecting the results.


Interpretation of the participants’ responses and the meaning of the importance of the factors in planning is difficult.
It is unknown how the findings can be generalized to Delphis which cover a 30 year extension into the future.
Delphi at present can render no rigorous distinction between reasonable judgment and mere guessing.
It is difficult to determine the amount of bias injected into the results by the person administering the Delphi.


Nominal Group Technique

The committee chairman reiterates that the role of everyone present is to contribute his perceptions, expertise, and experience to the identification of priority problems. He emphasizes that the purpose is to identify and describe priority problems and needs. He indicates that each member is to work individually without interacting verbally with each other. The committee chairman distributes the problem or need identification form to the members and ask them to respond in writing to the questions or statement on the form, giving an example of the kind of response.

Without discussion, silently and independently, each member lists on the form the problems and/or needs for approximately 10 minutes. The chairman enforces silence by requesting that those who have stopped writing not talk with others and think more deeply for other possible items.
The recorder now asks each member to state an item from their list. The items are recorded until all lists have been included. Discussion of items in not allowed and no concern is given to overlap of items at this time. However, members are encouraged to generate new ideas on their forms based on items presented by other group members.
The group now discusses for approximately 20 minutes the items listed for the purpose of clarification, elaboration, combination of items, or addition of new items.
Without discussion and acting independently, each group member should select and prioritize the items believed to be most critical. The recorder asks for each member to give the items selected in priority order. The most critical items are determined by the total amount of interest and votes. Discussion of voting and priority items continues until a consensus is reached.


Focus Groups

Focus groups have been a mainstay in private sector marketing research for the past three decades. More recently, public sector organizations are beginning to discover the potential of this procedures. Educational and nonprofit organizations have traditionally used face-to-face interviews and questionnaires to get information. Unfortunately, these popular techniques are sometimes inadequate in meeting information needs of decision makers. The focus group is unique from these other procedures; it allows for group interaction and greater insight into why certain opinions are held. Focus groups can improve the planning and design of new programs, provide means of evaluating existing programs, and produce insights for developing marketing strategies.


Involve people. It must be small enough for everyone to have opportunity to share insights and yet large enough to provide diversity of perceptions. Focus groups are typically composed of 6 to 10 people, but the size can range from as few as 4 to as many as 12.
Conducted in series. Multiple groups with similar participants are needed to detect patterns and trends across groups.
Possess certain characteristics. Participants are reasonably homogeneous and unfamiliar with each other.
Provide data. Focus groups pay attention to the perceptions of the users and consumers of solutions, products, and service. They are not intended to develop consensus, to arrive at an agreeable plan, or to make decisions about which course of action to take.
Produce qualitative data.

Focused discussion.

Grudens-Schuck, N., Lundy-Allen, B., & Larson, K. (2004, May). Focus group fundamentals. Ames, IA: Iowa State University Extension. Available at

Larson, K., Grudens-Schuck, N., & Lundy-Allen, B. (2004, May). Can you call it a focus group? Ames, IA: Iowa State University Extension. Available at


It is a socially oriented research procedures.
The format allows the moderator to probe.
Discussions have high face validity.
Discussions can be relatively low cost.
The format can provide speedy results.
Focus groups enable the researcher to increase the sample size of qualitative studies.


The researcher has less control in the group interview as compared to the individual interview.
Data are more difficult to analyze.
The technique requires carefully trained interviewers.
Groups can vary considerably.
Groups are difficult to assemble
The discussion must be conducted in an environment conducive to conversation.


Types of Focus Group Questions

Opening Question. This is the round robin question that everyone answers at the beginning of the focus group. It is designed to be answered rather quickly (within 10-20 seconds) and to identify characteristics that the participants have in common. Usually it is preferably for these questions to be factual as opposed to attitude or opinion-based questions.
Introductory Questions. These questions introduce the general topic of discussion and/or provide participants an opportunity to reflect on past experiences and their connection with the overall topic. Usually these questions are not critical to the analysis and are intended to foster conversation and interaction among the participants.
Transition Questions. These move the conversation into the key questions that drive the study. The transition questions help the participants envision the topic in a broader scope. They serve as the logical link between the introductory questions and the key questions. The participants are becoming aware of how others view the topic.
Key Questions. These questions drive the study. Typically, there are two to five questions in this category. These are usually the first questions to be developed and also the ones that require the greatest attention in the subsequent analysis.
Ending Questions. These questions bring closure to the discussion, enable participants to reflect back on previous comments, and are critical to analysis. These questions can be of three types:
All Things Considered Question. This allows the participants to state their final position on critical areas of concern.
Summary Question.
Final Question.



1. Define inventories.

2. List three advantages of inventories.

3. How do you validate an inventory?

4. Define scale.

5. List three characteristics of scales.

6. Name the three types of scales.

7. Name the three types of attitude scales.

8. Describe value scales.

9. List the types of objective scales.

10. Define direct observation methods.

11. List five steps for systematizing the direct observation methods.

12. Define semantic differential.

13. Name the three clusters of adjectives used in semantic differential.

14. How do you construct and use the semantic differential?

15. List the characteristics of the semantic differential.

16. Define Q methodology.

17. How do you apply the Q methodology?

18. List three advantages of Q methodology.

19. List three disadvantages of Q methodology.

20. List the steps of a conference.

21. Define Delphi technique.

22. List the Delphi procedures.

23. List three advantages and three disadvantages of Delphi technique.

24. List the characteristics of a focus group.

Home Up Data Tools Data Tools 2 Reliability Historical Descriptive Experimental Qualitative