Be sure that you answer ALL PARTS of the assignment. You cannot receive credit for sections of a question that are left incomplete.
 

GENERIC REMINDER  #2

In the world of survey research professionals, "random sample" and "random methods" are essentially laypeople terms. There is no such technical term as "random sample." One goal of this course is to give you practice using professional terms. The correct technical terms that are meant 99 percent of the time are "probability sample" or "probability methods" ("chance methods" is OK too.)

It is incorrect to interchangably use the terms "random sample" and "probability sample" because these are not interchangable terms.
 
 

GENERIC REMINDER  #3

Random Assignment is VERY different from simple random sampling. You lost credit if you used these terms in an interchangeable manner. Simple random sampling is one way to select study participants. Random assignment is how you place these participants in different interventions--once you sampled them in the first place.
 

GENERIC REMINDER  #4

There are many types of probability samples. EPSEM (Equal Probability of Selection Methods) samples are just one type of probability sample. Simple random samples (srs) are just one type of EPSEM sample. Further, srs samples are a minority, because a complete frame listing all elements is necessary. Except for Random Digit Dialing (RDD) and some samples of military or student populations, such complete frames generally do not exist.

The key defining characteristics of a probability sample are KNOWN and NONZERO CHANCES of selection. That's it.

Non-EPSEM probability samples are perfectly OK to do, as long as you apply corrective weights before analyzing the data as a whole. In fact, in some cases, you may want to oversample a small group (e.g., female engineering students) so that you will have a large enough case base to reliably analyze at all.

It is incorrect to interchangably use the terms equal probability sample, simple random sample, and probability sample because these are all different terms with very specific meanings.

A nonprobability sample is a major cause of low external validity because the subjects or respondents do not represent or "stand in for" anyone else but themselves.
 
 

GENERIC REMINDER  #5

Be sure to match the question stem with the provided responses. When a question begins with the phrases "do you..." or "are you," the respondent will expect "yes" or "no" answers because that is conventional English usage (NOTE: English as a second language speakers: be alert to similar nuances in your first language).

When a question stem matches the question responses, the respondent is much more likely to interpret the question correctly and find it easier and faster to answer.
 
 

GENERIC REMINDER  #6

Be VERY alert to "double-barrelled questions" which ask at least two questions in one. These are essentially confounded variables. Asking about "being tired anddepressed" is asking about two separate issues. Although the words "and" & "or" are usually tipoffs, be alert to other possible wordings that confound the question.

For example: "Do you approve or disapprove of a married woman   with young children taking a job outside the home if her husband can support her anyway?" This question, which was really used in the 1950s is "triple-barrelled" with three separate issues: marriage, young children, and husband's economic support. No "and" or "or" separates the triple barrelled clauses.
 
 

GENERIC REMINDER  #7

Avoid collapsing interval-ratio variables into ordinal variables. Avoid grouped categories for interval variables. You will lose valuable information and it will be impossible to retrieve that information.

For example: How many seasons have you been a head coach? _____1  _____2-3  _____4-6 _____7 or more

You have just turned a ratio variable NEEDLESSLY into an ordinal variable. You will not be able to get the mean number of seasons. You can also simply get the "number of seasons" and collapse this variable any way you please in a matter of seconds.

The one exception is income for two main reasons: (1) people are more willing to answer a grouped income category than a precise amount and (2) people rarely know the precise amount of their income because there are so many sources of income such as: wage and salary income; profits from a small or large business; government transfers such as Medicare or Social Security; private pensions; interest on bank or credit union deposits; interest on private loans; stock dividends; rents--and many more. Almost no one will add up all these sources on the spot to arrive at a final accurate figure.

EXAMPLE: if you ask about education in number of years, you can always group categories later on.  ("What was the highest grade or year of education that you completed?")
 
 

GENERIC REMINDER  #8

The term "Likert item" has a very specific meaning. The respondent is given a statement and asked to indicate their degree of agreement or disagreement with that statement. At a stretch you might use degree of satisfaction. The idea is to measure affect about the statement. Because of their use of "yes" and "no" or "agree-disagree" response choices, Likert items are easy and fast to administer, but Likert items are particularly prone to one form of questionnaire response set: acquiescence or "yea-saying" response set. Depending on the number of responses, they also may be prone to extremity response set too.

Likert items DO NOT refer to all types of forced choice or closed questions. It is a misapplication of terminology to do so and survey research professionals will misunderstand you.

Questions such as "how true is each statement of you" are simply "forced choice" or "closed" questions.

GENERIC REMINDER  #9 RESPONSE RATES

What's the importance of the response or completion rate? When you mail out a survey, assign telephone numbers to interviewers, or send an interviewer into the field, not every selected element results in a survey. The percent of selected, eligible elements that actually results in a survey is called the response or completion rate.

Not all nonresponse is problematic. Some letters will be returned because no one lives at that address and some telephone numbers are not operative. Some addresses literally are not (either the address was wrong or the building was torn down.) All these are instances of non-population elements. Some researchers report this number, some do not, but non-population elements, strictly speaking, do not enter the denominator for the response rate.

Other nonresponse occurs: for example, an element exists but is ineligible or unable to participate for some other reason. People who are deaf can't take telephone surveys without special equipment on both ends of the telephone wire. If the questionnaire is not in the respondent's native language and the ethnic group is a tiny fraction of the population, the element may be marked "no interview, other." (If several elements speak this language, the questionnaire will probably be translated and back-translated.) People get sick the day surveys are administered. Depending on whether the respondent is truly ineligible, or was eligible but unable to participate, the case either is "outside the population" or enters the denominator for response rate.

Laypeople imagine that "refusals" are the major reason for nonresponse on surveys, but typically that isn't true. Not-at-homes or "no answers" are usually the largest nonresponse segment.In general public and specialized public surveys, absenteeism has become a highly significant problem. More households are one person households and if that person isn't home, neither an in-person or a telephone interviewer can interview them. Among married couples, the norm is now for both partners to hold outside jobs, thus reducing the time that even one person is at home. Studies indicate that the amount of leisure time in the United States has dropped significantly over the past several decades leaving students and workers with little time to complete a survey. Telephone surveys have been made more difficult because more people use various screening devices or answer services, for example: answering machines; "caller id;" or voice mail. And, of course, people who are illiterate don't return completed self-administered questionnaires.

Survey researchers may attempt a "refusal conversion" on individuals who initially refused to do the survey. As many as half of initial refusals may "convert" following a second attempt to obtain the survey. This is because most people originally coded as refusals, in fact, did not refuse, although they were miscoded that way. Sometimes the person's husband or wife refuses and the respondent never even knew about the contact. Sometimes a person must leave for an appointment, is in the middle of cooking dinner, or someone enters the office. No one with the flu wants to complete a survey at that time. My experience is that "true refusals" are typically less than 15 percent of the original sample.

Both refusals and not-at-homes or absentees are part of the population and enter the denominator for the response rate.

Telephone numbers that ring with no pickup are difficult to interpret. The telephone company does not intercept every non working number (and some telephone companies intercept almost no non working numbers.) Robert Groves, considered an international expert on telephone survey non response, estimates that nearly all numbers that simply ring, after 10 rings, and that are called at least six separate times at different days and times over several days, are non working.

Thus the numerator of the response rate is the number of completed surveys.
The denominator of the response rate is all eligible and sampled members of the population.

How good is "good?" Conventional survey research wisdom says any response rate under 50 percent probably means your sample became a self-selected sample. For a mail survey, 50 percent is actually pretty good. Telephone surveys conducted by organizations such as the University of Michigan's Survey Research Center or the National Opinion Research Corporation (University of Chicago) usually get at least a 60 percent response. Well-conducted telephone surveys that last at least a few weeks (to allow for call backs) can get a 75 percent response. And the face-to-face General Social Survey, an indicator survey that dates back to 1972, gets around 80 percent.

THREATS TO INTERNAL AND EXTERNAL VALIDITY

A low response rate is a threat to BOTH internal AND external validity.It is easy to see why low response contributes to external validity. Your case base is smaller, but, more importantly you are concerned about generalizing. As the response rate shrinks, your project looks more and more like it used a self-selected, hence non probability, sample.

Moreover, non response may not be random. Instead it may be selectively distributed across subsets of the population. Less educated people less often answer self-administered surveys. Very wealthy people may decline to be interviewed by telephone or in person. People who are depressed may refuse to be surveyed. If non response is nonrandom, it also threatens internal validity.More and more, your survey may appear to suffer from a well-known internal validity threat: selective mortality, or the disproportionate loss of individuals from particular treatments or categories. Selective mortality can lead to unknown response biases. (NOTE: to compensate for selective mortality, some researchers "weight up" so that sample proportions, say, for educational level, match the population proportions. THIS DOES NOT SOLVE THE SELECTIVE MORTALITY THREAT!)

Therefore, you must scrutinize the reasons for nonresponse very carefully and see if you can compensate for any of them. Perhaps a translated questionnaire and fluent speaker can be used to interview those who speak a different language. Increase the number of call-backs to locate respondents (or to ascertain if the telephone number is operative). Incorporate one polite refusal conversion attempt for those originally coded as refused. Perhaps a female interviewer will succeed where a male interviewer did not (or vice-versa), or an older or younger interviewer will work out well. Offer to do the survey as an interview for those with literacy problems. Compare your data with any kinds of known records to calculate if your study produced any systematic selective mortality.
 
 

GENERIC REMINDER  #10

The type of the sample you took is generally more important for external validity than the number of cases you have. In their assignments, some individuals felt that they could not generalize well from a small case base (n =100). This is true only for very small samples. And it is not that you cannot generalize (if you took a probability sample), it is that the standard errors are so large for very small samples that your generalizations would be very imprecise or very unstable.

Nonprobability samples threaten external validity more than the case base size.

The standard error in statistics, which you can use if you took a probability sample, adjusts for the number of cases. If you have more cases, your standard error is smaller, your confidence intervals are smaller, and your estimates in generalizing are more precise. Conversely, very small case bases, or n, result in large standard errors, and very imprecise estimates of population parameters.

Obviously, tiny samples, even tiny probability samples, leave you with very limited generalizations. But once you have even 100 cases from a probability sample with a good response rate, you can generalize to your population with known error limits.

The problem for external validity involves not only generalizing to your original population, but to other populations. With your 75 percent response for a survey of Tallahassee, you can generalize to Tallahassee--but what else is possible? Here logic and knowledge usually serve you the best. Perhaps results from Tallahassee are comparable to similar sized cities in the Southeast, such as Albany, Georgia. Results from a good sample at FSU might generalize to other large, public Southeastern Research I universities, such as University of Florida, University of Georgia, or University of North Carolina.
 
 

GO BACK TO ASSIGNMENT 3 FEEDBACK HERE: