PLEASE NOTE: IF ANY QUESTIONS ARE POSTED TO ME THAT WOULD CHANGE OR CLARIFY THE CONTENT OF THIS SITE, I WILL CORRECT IT HERE ON THIS SITE AT THE TOP.

ATTENTION: THERE ARE UPDATES ABOUT WHAT WILL BE ON EXAM 2 UNDER THE GENERAL EXAM 2 COVERAGE SECTION

FOR BOTH THESE REASONS, CHECK BACK BEFORE EXAM 2!
 
 

EDF 5481 READINGS AND ASSIGNMENTS

OVERVIEW

GENERAL GUIDE:   EXAM TWO ALSO SEE:  GUIDE 1: INTRODUCTION GUIDE 2: VARIABLES AND HYPOTHESES GUIDE 3: RELIABILITY, VALIDITY, CAUSALITY, AND EXPERIMENTS GUIDE 4: EXPERIMENTS & QUASI-EXPERIMENTS GUIDE 5: A SURVEY RESEARCH PRIMER GUIDE 6: FOCUS GROUP BASICS GUIDE 7: LESS STRUCTURED METHODS GUIDE 8: ARCHIVES AND DATABASES

WE WILL NOT ANSWER EMAIL MESSAGES RECEIVED AFTER 10 PM TUESDAY NOVEMBER 12.
WE ARE NOT RESPONSIBLE FOR ANY DELAYS IN EMAIL DUE TO SERVERS (some of which are quite slow).
IF YOU EMAIL WEDNESDAY MORNING, WE WILL NOT HAVE TIME TO RESPOND TO YOU.
THANK YOU FOR YOUR CONSIDERATION.
 
 

STUDY GROUP TIME! WE DO NOT HAVE A REGULAR CLASS WEDNESDAY NOVEMBER 6. RATHER THE PLAN IS FOR STUDY GROUPS TO MEET IN ROOM 126 DURING CLASS TIME. SEVERAL STUDENTS HAVE EXPRESSED AN INTEREST. I HAVE A FEW SUGGESTIONS TO MAKE THE MOST PRODUCTIVE USE OF THAT TIME: If size allows, break into groups of about 4-6. This allows everyone to participate.  Try beginning with the sample questions you see below in this Exam Guide. Assign a different set of questions to each group to work on. Then, after about 90 minutes, bring the groups together to exchange answers and rationales.  Separate groups can also (1) work on sets of concepts or (2) take one of the survey research segments (overall; sampling; question construction; etc) and work on that. The idea is for each group to take a MANAGEABLE amount of material for 90 minutes, then to share information with other groups. This approach has worked well in methods or statistics courses in the past. If there is a question about any of the sample questions that the group(s) feels is not resolved, EMAIL ME OR CHRIS. I should be email accessible Wednesday night and Thursday night.

A NOTE ON SELF-CONFIDENCE

Most students in this course are not only accustomed to getting high grades, but to feeling considerable self-confidence about course material. After all, you are graduate students at a very competitive research university. Thus, it may be a disconcerting surprise to some to feel overwhelmed by the material, and uncertain about one's level of knowledge.

These mixed feelings are common among relative novices in statistics and/or methods courses. Remember! You are learning a new language. In part you are doing so because many professionals in your discipline already speak this language and you want to converse knowledgeably with them. Just as you would not expect to take two months of introductory Spanish or Korean and emerge speaking like a native, this material takes time to assimilate. I don't know whether this will be by the end of the semester or a year down the road (when I began taking math courses during graduate school, things began to "click" somewhere near the end of my third semester course) although my experience is things DO fall into place eventually.

In addition, methods and statistics courses use conceptions of causality in a way different from many of your other courses. These take some getting used to. We have a range of backgrounds in this class, from people who have already had exposure to several methods classes to those with none at all.

Meanwhile, pay careful attention to feedback. If Chris or I say that you are doing well, relax a bit (not too much, of course) because you are, even if you don't feel that way yet. And folks who have already had several methods courses probably already have a higher level of comfort.

NEW INFORMATION INCLUDED ABOUT SOME OF THE EXAM CONTENTS!

The major emphasis on Exam Two is on topics covered by:

• chapters 7, 11 and 12 (basically the attitude inventories and ratings scales sections only in Chapter 12)  in Wiersma;

• chapters 5, 7 (focus on pages 155-161 and 165-173 in Chapter 7), and 8 (pp. 195-197 ONLY but skim the annotated article after that) in McMillan;

• and all lectures, videos, demonstrations, and course Web sites through Guide 6 (with associated links), including any material I have placed in Blackboard/CourseInfo.

EXEMPT from the exam is the focus group DIRECTIONS site (Guide 6 IS on Exam 2) and the eSampling site.

We will continue to reference causal issues through to the end of the semester.

Exam Two includes material on focus groups, which are only mentioned in passing in the texts (see Guide 6).

In some cases you will be asked to choose the sections of a question that you answer, e.g., select three out of four sections. The purpose of this is to allow you to show off the areas that you know the best. DO NOT answer all choices in such instances. No extra credit! We only grade the first number of designated selections if you answer all the selections in these cases. So what can happen is that (for example, in a 3 out of 4 selection question) you get parts 1, 2 and 4 right, but I only grade parts 1, 2, and 3, so your credit is lower than if you had simply answered 1, 2 and 4.

This exam has a more applied focus than Exam 1. That is, at this point, you should be able to apply concepts such as external validity or construct validity to issues in sampling or to questionnaire item construction. Thus, the PROBLEM SOLVER sections altogether will be slightly more than one-third of Exam Two.

I also expect you to be able to compare and contrast advantages and disadvantages of the methodological techniques we have examined to date: experiments; quasi-experiments; standard surveys; and focus group interviews.

Exam 2 is 100 points and should take about one hour to complete. It counts 25 percent toward your final grade. If you want more time than one hour, that is OK too.

As before, Exam 2 is a mix of multiple choice, fill in the blank, and short essay questions. You may add a SHORT explanation to any short-answer question. There are multiple choice, true-false, etc. sections on causality, confounded variables, validity, advantages of different kinds of administered surveys, and samples.

There is ONE problem solver question, worth 38 points. It is an applied example for both structured survey research and for focus groups.
 
 

Exam coverage from the texts is selective. For example, I won't have questions about the literature review sections in either Wiersma or McMillan. Exam Two only covers through survey research and focus groups, so although there are sections on less structured or qualitative designs in some of your chapter readings, we will cover this material on Exam Three.

I don't have articles for you to analyze as McMillan does in his excellent examples but you will have a lengthy problem solver question which will require you to draw upon and analyze using your knowledge.

EXEMPT from the exam is the link on focus group DIRECTIONS (Guide 6 IS on Exam 2) and the eSampling site.
 
 

WHAT WILL BE ON THE EXAM

PROBLEM SOLVER QUESTION: You will assist an investigator in designing a survey. You will also assist in designing a focus group interview.  Question sections require you to know the difference among types of survey administration, understand types of samples, and rewrite questions. There are a lot of similarities to assisting Jerry with his survey on Assignment 3. The most helpful sites to review for this section are:

Guide 5
Sampling Site
Question Construction Site
Questionnaire Example
Assignment 3 Feedback Site

Reminders Site

Review Guide 6 on Focus Groups HERE.

MORE EXACT INFORMATION ON TYPES OF QUESTIONS TO BE POSTED LATER.

There will be questions on causal order and causal plausibility.

Reviews on types of validity: construct, internal, and external. In these questions, you are required to apply issues such as response rates, types of sample designs, and types of study designs (experiments, quasi-experiments, surveys) to different issues in validity. Be sure you can appropriately identify the rules or guidelines applicable to these sets of variables to place them in causal order.

There are several multiple choice questions and true/false questions on sampling and different types of samples.

A section on different levels of variables: nominal, ordinal and interval-ratio that is identical in format to the questions on Exam One (but different variables are used). Be able to give the rationale for your choice in each case.
 

In general, your books and I agree on terms and usage. Please recall there is one important exception for this exam.
 

INTERVENING VARIABLE

I define an intervening variable as one that links the independent variable to the dependent variable. Thus, an intervening variable is part of a causal chain:

The intervening variable comes in between the independent variable and the dependent variable.

INDEPENDENT VARIABLE  INTERVENING VARIABLE  DEPENDENT VARIABLE

PLEASE USE THIS DEFINITION FOR THIS COURSE. PLEASE DO NOT USE THE TEXT DEFINITION.

There are several advantages to using the in-class definition:

• It is consistent with the way that most statisticians use the term in Structural Equation Modeling.

• Using this definition, intervening variables are NOT confounded. In fact, by disaggregating the effects of an independent variable, intervening variables actually unconfound an originally confounded variable.

• Using this definition, intervening variables absolutely CAN be measured. They are critical to use in non experimental research designs. They can specify what it is about the dependent variable that is important.

• Intervening variables inform us about causal sequences or chains, thus explaining the causal process of a phenomenon.

education level  number of science courses elected science attitudes

Logic, necessary conditions, and prerequisites are why educational level affects the number of science courses. In order to take the courses, one must first be in school. If we are looking at ADULTS, it is likely that they first completed their education, along with taking the science courses, before forming adult attitudes (time order).

SECOND EXAMPLE: educational level occupational type income level

Educational level influences someone's income level because people with more education are more likely to obtain jobs that pay a higher salary. A certain educational level is first in time before taking most full-time, year-round occupations.A certain level or specialty in education is also typically a requirement or prerequisite to being hired in a job. And, for most of us, the job is the requirement to receive income.

As you can see, intervening variables, both conceptually and operationally, are very important for all but the very simplest causal assertions. It will be useful to you to practice thinking in these causal chains and speculating about precisely which intervening variables are the most critical in outcomes.

Be sure that you understand the differences among construct validity, internal validity, and external validity. In fact, there will be a series of multiple choice questions with examples of each. Be aware that reliability is a necessary, but not sufficient, condition for construct validity and internal validity.

I want you to know about threats to internal and external validity, and the enhancements that aspects of your designs can make to both these types of validity.

REVIEW  YOUR WIERSMA BOOK, PAGE 104! and

HINT: REVIEW GUIDE 3 HERE: 

AND PARTS OF GUIDE 4 HERE: 

AND CHECK OUT THE GENERIC REMINDERS #9 ON RESPONSE RATES & VALIDITY HERE: 

The key is not just to memorize threats to internal or external validity, but to see how they apply in concrete instances. For example, random assignment of study participants to treatments or interventions controls MANY threats to internal validity, such as maturation, history, or self-selection. Probability sampling is not a guarantee of high external validity, but typically it is a necessary requirement. Without a probability sample, you cannot scale the first hundle to establish high external validity.

HINT: CHECK OUT THIS CLASS WEB SITE FOR THESE TERMS: 

I expect you to know the differences betweenrandom assignment and probability sampling.

There were some problems distinguishing these, especially on Assignment 3.

I expect you to be able to place variables in nonexperimental studies in causal order (independent, intervening, dependent). There will be a series of these questions. You will need to be able to describe the rule that you used to causally order the variables  too.

HINT: CHECK OUT THESE CLASS WEB SITES FOR THESE TERMS

Review: 

Also review: 

Recognize examples of the following and be able to define the following:

Nominal variable
Ordinal variable
Interval-ratio variable

HINT: CHECK OUT THIS CLASS WEB SITE FOR THESE TERMS

 
 

OTHER TERMS TO KNOW

Confounding or confounded variable: a multidimensional variable, for example, educational level which measures social class, cognitive sophistication, and exposure to diversity.

Cross sectional versus longitudinal or panel (time series) survey
Focus group
Pilot test
Randomization
Response rate
Self versus interviewer administered survey
Social desirability
Types of samples:

• probability versus non-probability. Stratified, simple random, systematic and cluster probability samples. Self-selected, grab or convenience, purposive or judgmental, and quota non probability samples.

How does a PILOT TEST differ from a PRETEST?
(Hint: we use the first in surveys and the second typically in experiments.)

IMPORTANT REVIEW NOTE: While randomization or random assignment of subjects or groups to treatments/interventions is very important for internal validity, notice it says NOTHING about how you obtained your subjects or groups in the first place. Thus randomization is NOT connected to external validity.
 

Given a pair or a triplet of variables in nonexperimental data, can you designate which one is the independent variable, which the intervening variable, and which the dependent variable? Can you give the rule behind your decision? By "the rule," I mean one of the decision guidelines in Guides 2 and 3:
 

In Guide 2:

In Guide 3:
 
 

Can you decide if a causal relationship is asymmetric or symmetric?

Can you decide accurately whether a variable is nominal, ordinal, or interval-ratio level? Can you give the rationale behind your decision?

What does it mean to say a variable is "confounded?" Give an example and explain where the problem lies.

Do you understand how response rates influence BOTH internal AND external validity? This is especially true if nonresponse is systematically, rather than randomly, distributed.

Can you recognize the different types of probability and non-probability samples when a sample is described to you (for example, in a journal article)? Do you know when you need more information about a sample than what is provided for you (mode of administration? case base? response rate? sample type? population and sampling frame?)

Do you know why it is generally more advantageous to take a probability sample? And why logistic considerations may preclude your doing do?

See if you can explain the difference between bias and random error.

Review construct validity. Consider how measurement, including question format in surveys and "tests" may contribute to or detract from construct validity.
 

Provide RULES FOR CAUSAL ORDER IN NON EXPERIMENTAL STUDIES that you used to order the variables. (Use the Guidelines from Guide 3.)

Place the independent variable on the far left, then the intervening variable, and the dependent variable on the far right:

• 1. Algebra method experimental intervention, Geographic region, Math test score

• 2. Child physical fitness score, Child years of physical training, Parental attitude

The following questions are designed to look at several interlocking concepts: randomization versus simple random samples, and internal versus external validity.

For each of the following statements about validity, check whether the statement is generally true or generally false. Add a FEW words of explanation if you wish.

1. A very low response rate can threaten BOTH internal validity AND external validity.   CHECK ONE:
       [  ] TRUE      or    [  ] FALSE
 

Random assignment of subjects to experimental treatment groups produces data high in EXTERNAL validity.  CHECK ONE:
         [  ] TRUE      or    [  ] FALSE
 
 

Survey research designs typically have higher INTERNAL validity than true experimental research designs. CHECK ONE:
        [  ] TRUE      or    [  ] FALSE
 
 

Using simple random sampling to obtain your subject pool produces data high in INTERNAL validity. CHECK ONE:
        [  ] TRUE      or    [  ] FALSE
 
 

A multi-stage sample has higher external validity than a single stage sample.
 

DistanceLab wants to conduct a survey of their population of 10,000 Web trainees and it has hired YOU as a consultant.  DistanceLab conducts Web-based courses for technology managers and executives on several topics. The company is considering a telephone survey of trainees.

1. DistanceLab wants to know whether a telephone survey is the right option for them. Compared with other survey methods, what is one ADVANTAGE of a telephone survey?
 
 

2. Briefly describe one DISADVANTAGE of using a telephone survey compared with other survey methods:
 
 

3. A DistanceLab employee suggests mailing a survey to the trainees instead. Your answer is (CHECK THE ONE BEST OR MOST APPROPRIATE ANSWER):

    [  ]A. DistanceLab trainees are too literate
    [  ]B. Mailed response rates are very low
    [  ]C. The costs are too high
    [  ]D. These survey topics are too sensitive

Mailed response rates not only are generally low, but respondents self-select. Less literate--and sometimes less sucessful--are less likely to respond.

4. DistanceLab has an alphabetical list of all trainees with their most recent telephone numbers. After a random start, each 10th person on the list is selected to be interviewed.

   A. What is the name of this kind of sample? SYSTEMATIC SAMPLE WITH A RANDOM START

   B. Did DistanceLab take a probability sample?                    Circle   YES    or    NO

   C. What is the sampling frame for this sample? The alphabetical list of all trainees.
 

   D. Is this a (CIRCLE ONE:)      single stage      or          a multi-stage sample?

As described, ONE list and one sample stage.
 
 

5. Question construction can be difficult and DistanceLab values your good advice. Here are four questions from an earlier survey they conducted. First, BRIEFLY describe any problems you find with each question. (If the question is adequate, simply note this.) Next, rewrite the question to correct any problems that exist (3 points each).

A. What was your age on your last birthday?
   _____Under 21
   _____21-25
   _____26-31
   _____32-40
   _____41-50
   _____over 50

Needless collapse of ratio data into ordinal with UNEQUAL categories. HUGE open final category. The stem is basically OK (although age in years would be better). What was your age in years on your last birthday (record literal response).

B. How long have you been working with DistanceLab?

Another "where were you born?" question. Vague time reference. "How many years have you been working with DistanceLab?" (Depending on study goals, you may want to use "months" and if you believe people have left and returned, either a set of questions or "altogether" added.)

C. Please tell me if you strongly agree, agree, disagree, strongly disagree or are in the middle on the following statement: It is easy to learn new things using computers and the Internet.

Double barrelled question. Break into two questions, one for "computers" and a second for "the Internet".

D. Do you know the one best thing that you like about the DistanceLab program?

Stem of question would lead respondent to answer "yes" or "no."
What is the one best thing that you like about the DistanceLab program?

At this point, a DistanceLab executive complains that he wants more in-depth information about how DistanceLab trainees feel. You propose that DistanceLab conduct several focus groups.

6. BRIEFLY describe one advantage that a focus group would have over a conventional standardized survey in providing in-depth information about DistanceLab trainees.

Depth, a variety of opinions on the topic, the ability to explore interesting attitudes that are expressed
(that's 3).

7. BRIEFLY describe the type of sample you would take to form focus groups of DistanceLab trainees.

Is the sample you describe CIRCLE ONE:    PROBABILITY    or    NONPROBABILITY ?

It most likely will be a nonprobability sample. You might invite individuals who responded to the general survey. Perhaps you might use "snowball sampling" (e.g., having respondents recommend other respondents). Since generalizing is typically not a goal of focus groups, you may value having a diversity of opinions.

8. (2) Is it better to use open or closed questions for a focus group? BRIEFLY, WHY?

You want people to talk. OPEN. Definitely!
 
 

For each of the following questions, indicate (1) whether the variable is nominal, ordinal, or interval-ratio and (2) IN ONLY ONE SHORT SENTENCE describe the reason behind your decision:
 

1. Educational credit hours	2. Smoke cigarettes: yes or no
3. Gender: male or female	4. Scale ranking: very true of me; somewhat true of me; a little true of me; not true of me

 

First decide if the following assertion is largely plausible. Second, describe the causal rule that makes it plausible (or which rule was violated if it is implausible.)

1. In your study, race/ethnicity is related to occupational choice. You conclude that race/ethnicity has a causal influence on occupational factors.

Is this assertion plausible or implausible? Because.....?
 
 

2. In your survey, you find a correlation between a "state anxiety" test score and a motivational test score. You conclude that anxiety causes motivation.

Is this assertion plausible or implausible? Because.....?
 
 

BRIEFLY define AND give an example of ANY THREE (and ONLY three) of the following terms:

1. Demographic variable
2. Double-barrelled question
3. External validity
4. Response rate
 
 

HERE IS AN INSTANCE WHERE (1) YOU WRITE ON ONLY THREE TERMS and (2) YOU MUST ANSWER BOTH PARTS OF THE QUESTION: THE DEFINITION and THE EXAMPLE TO RECEIVE FULL CREDIT.

For each of the following statements, check the ONE BEST or MOST APPROPRIATE answer. Add a few words of explanation if you wish.

Although women are only about 10 percent of the engineering population, I wanted to study enough women to reliably analyze data. Thus, I designed my systematic sample with a random so that women were now 33 percent of a sample of engineers. This type of sampling design is:

[  ]A. A disproportionate stratified sample
[  ]B. An EPSEM sample
[  ]C. No longer a probability sample
[  ]D. Probability proportionate to size
 
 

Which of the following is true about sampling error?

[  ]A. Larger samples produce larger confidence intervals
[  ]B. Larger samples produce smaller sampling errors
[  ]C. Sampling error is beyond any control
[  ]D. Sampling error is systematic
 
 

Which one of the following statements is true?

[  ]A. Sampling error is random, bias is systematic.
[  ]B. Single stage samples are better than multistage samples.
[  ]C. The sample size is more important than the response rate.
[  ]D. The type of sample you take is typically less important than the sample size.
 
 

Measurement error is the most involved with which ONE of the following types of validity?

[  ]A. Construct validity
[  ]B. Criterion validity
[  ]C. External validity
[  ]D. Internal validity
 
 

PLEASE NOTE: IF ANY QUESTIONS ARE POSTED TO ME THAT WOULD CHANGE OR CLARIFY THE CONTENT OF THIS SITE, I WILL CORRECT IT AT THE TOP OF THIS SITE.
 
 

EDF 5481 READINGS AND ASSIGNMENTS

OVERVIEW

UPDATED
Susan Carol Losh  November 11 2002
This page was created with Netscape Composer and
is best viewed with Netscape Navigator
800 X 600 display resolution.