EDF 5481 READINGS AND ASSIGNMENTS

ASSIGNMENT THREE SPECS ASSIGNMENT THREE EXAMPLES

OVERVIEW

It has been very enjoyable to watch your study designs evolve over the semester. In general, they are a very good group of papers.

Here are some points to note:

Some people are still confusing independent and dependent variables, and at this stage of the course, this is a VERY serious problem. There WILL be variables to place in the independent -> intervening -> dependent sequence order on Exam Two! So if you are still mixing up cause (independent variable) and effect (dependent variable), review in your reading and at GUIDE 2 and GUIDE 3.

There are also some problems with conceptual and operational variables. Recall that an operational variable describes how you will measure that variable (i.e., concrete operations).  In an operational variable or definition, you explain exactly how you will measure the conceptual entity. GUIDE 2 is a helpful review here too.

Be sure to differentiate between random assignment and a simple random sample. These are two entirely different constructs. You can review the difference in the SAMPLING SITE.

Use the SAMPLING SITE to distinguish between a census and a sample. If you have a small population and it is really feasible to study everyone, then it is perfectly OK to take a census. The big problem comes in if you have a large population or a scattered population so that a sample may be the only practical thing to do.

Was your design appropriate to your research problem? If your dependent variable is stock prices, why survey company CEOs? If you want to see what aspects of an organization impact on individuals, you probably want to design an intervention, not do a survey. If your population is basically illiterate, don't use a self-administered survey!
 

HAD TO DO ALL OF THE FOLLOWING:

• Correctly rewrite Assignment 2. Clear research problem statement and rationale, accurately designated independent and dependent conceptual variables, and TWO DISTINCT unambiguous conceptual hypotheses. It was OK to change variables or hypotheses--as long as all was correctly done.

• Give concrete detail for the operational variables (examples: cite specific tests, a proposed questionnaire, a concrete procedure).

• Have clear and unconfounded conceptual and operational hypotheses. Accurately stated two distinct hypotheses in each case.

• Explained why their choice of research design was the most appropriate. Gave two distinct and appropriate threats to internal validity that directly pertained to THEIR design (not just "generic threats" to internal validity). For example, social desirability is one form of reactivity, and is not a second distinct threat. History or maturation should not be a problem with a true experiment because random assignment will control for these. Gave enough detail so that it was clear that what the student wanted to do in fact matched the design designation they chose.

• Described their sample or census thoroughly with the correct designation of probability or non probability (in the case of a sample) and type of sample (in the case of a sample).Accurately described their sample (for example, did not confuse a sample with a census). Gave one appropriate threat with their design to external validity.

• Accurately diagnosed Jerry's research design problems. Correctly answered questions about sampling and external validity. Addressed ALL FIVE of Jerry's sample questions and appropriately rewrote the questions.

• You STILL have problems with independent and dependent variables.
• Your operational variables weren't.
• Your variables were compound and confounded.
• You weren't sure WHAT kind of design you had!
• Your design was inappropriate for your research problem.
• You mixed up internal validity with external validity.
• You gave no detail at all on what you planned to do so it wasn't clear what kind of design you would use.
• You thought you had a sample when you had a census.
• You thought you had a probability sample but you had a convenience sample instead.

WHO will complete your questionnaire? Will it be self-administered (be sure your target population has the capacity to do so, especially if there are open-ended questions)? Will you use an interviewer or digital recording?
 

As a professional, you will spend considerable amounts of time reading research materials and research studies. Nearly all of them will draw conclusions and many of them will make recommendations for interventions and/or policy continuance or changes.

Therefore it is essential that you are able to accurately assess the methodological soundness of the studies that you read and ascertain whether causal statements can be plausibly made and/or the results generalized past that specific study.

Look for similar problem-solver questions on Exam Two.

Here is Jerry's planned survey:
 

IMMEDIATELY VISIT THIS SITE AND READ THE GENERIC REMINDERS ABOUT THIS ASSIGNMENT AND ABOUT SURVEY RESEARCH IN PARTICULAR:
 

THE REMINDERS WEBSITE

1.  Is Jerry's sample a probability sample? Please answer YES or NO, then explain in a sentence or two:

• WHY or WHY NOT?

Don't confuse Jerry's narrow definition of his population (male, public high school football and basketball coaches) with a sample. Jerry can define his population as narrowly as he chooses. Note, however, that this will get him into trouble with external validity when he wants to generalize later on.

2. Briefly give Jerry ONE SUGGESTION about how he can get a good RESPONSE RATE for his survey.

WE HAD GREAT STUFF ON THIS QUESTION. REMEMBER THAT JERRY HAS RULED OUT A SECOND MAILOUT.Here are some of the answers:

• Have a terrific cover letter, preferably with endorsements from a major sports figure. Bobby Bowden, where are you?

• Have a nice inducement. We know that Jerry's funds are limited, but how about donated gift certificates from a local sports store? Or tickets to a professional game? One person even suggested donated gift certificates for Dairy Queen.

• Keep the questionnaire short and easy to answer. This means nearly all closed questions. Have an attractive questionnaire layout too. Lots of "white space." "White space" is the blank paper surrounding the type and I put it in quotation marks because several people suggested using colored paper, both to make the questionnaire attractive and also so it is immediately recognizable in the stacks of paper on the coach's desk.

• Make some telephone calls to coaches who didn't return their questionnaire (you might have to send out selective second mailings in this case). Possibly even email a reminder to the coaches. If finances permit, send a reminder postcard. Your chances of these reminders working increase if you do so relatively early in the response period, before the questionnaire hits the wastebasket.

3. Do you think Jerry's design in the box above has relatively high or relatively low EXTERNAL VALIDITY?

• Pick "high" or "low" and in then a sentence, explain the basis for your decision.

• Briefly describe ONE possible threat to EXTERNAL VALIDITY in Jerry's design. (You may elaborate on your answer immediately above.)

Jerry does have a probability sample, so he passes this first hurdle. But external validity is about generalizing to other populations and situations, and in that respect, he is abysmally low.

Jerry can ONLY generalize to male public school football and basketball coaches (in his state). Many people commented on the gender and/or public school issue. However, the type of sport is important too. American basketball and football coaches are probably under significantly greater pressures than, say, golf or track coaches in most schools. Unlike other high school sports, football and basketball are often significant producers of school revenue and are much more visable. Jerry's very limited population means he can't generalize much at all.

Jerry only mails out one questionnaire. Most mail surveys have low response rates, especially with only one "wave." Typical is a response rate of 20 percent. Worse yet, there are usually systematic differences between respondents and nonrespondents on mail surveys (the more literate, the more committed to the topic, and those with more time are more likely to respond.) Thus, not only does this feature endanger external validity, but if there are systematic differences between respondents and nonrespondents, there is also bias, and thus threats to internal validity too.

4. Briefly describe ONE problem that you see with the way each one of Jerry's questions 1-5 have been written. Rewrite the question to solve the problem, using techniques of good question construction. If the question is OK, and you don't see any problems with it, then just say so and say the question remains "AS IS".

Please be sure to address ALL FIVE of Jerry's questions.

Well, this really is close to the worst set of questions (for pedagogical purposes, of course) I have ever done! Keep reading for some of the problems and solutions:

(1) How long have you been coaching?

"How long" is multidimensional. When the question is "how long" you MUST include a time frame, such as seasons, months or years. It is OK to include qualifiers such as where or "continuously" although DO consider using separate questions for each detail.

Avoid collapsing categories such as "1-5 seasons." You have just turned an interval variable into an ordinal variable, and this will hamper your efforts to analyze data later. Statistical packages such as SPSS will group categories for you in seconds. However, you cannot turn "1-5" years back into individual years.

EXAMPLE: How many seasons altogether have you been a head coach? _____Number of seasons

(2) What is the primary sport that you coach?

What does "primary" really mean? You're better off seeing which sport the coach coaches--and don't take anything for granted! People enter into a sample in very strange ways, sometimes; that's why you see "neither" as an alternative below:

Are you the head coach for (please check only one alternative:)

[   ] Football
[   ] Basketball
[   ] Both football and basketball
[   ] Neither football nor basketball?
 

(3) How would you describe your mental state (please just check one statement:)

     [    ]A. Every day I wake up happy and ready to tackle the day's work.
     [    ]B. Every day I wake up feeling "blah".
     [    ]C. Every day I wake up so sad I can hardly stand it.
     [    ]D. Other

Oh, OUCH. This is a TERRIBLE question. First of all, even depressed people have a good day every now and then. The first alternative is double barrelled ("happy AND ready to tackle the day's work"), "blah" is slang, and who knows what the limiting point is for each person (alternative #C). The answers are far from exhaustive so expect a big pile of responses in an uninterpretable "other" category.

Jerry is probably best off turning this somehow into a unidimensional question. He can go "happy-sad" (e.g., "think of a typical day last week, where a score of "1" is very happy and a score of "7" is very sad, what would be YOUR score?) or he can try a Likert type item, such as ascertaining the degree of agreement with a statement such as "I generally consider myself a happy person."

I would avoid using a question such as "how do you feel these days?" because you might get the facetious answer such as "with my hands."

(4) Do you agree or disagree with this statement? "These days I am so tired and depressed I can hardly get anything done."

The statement is double-barrelled because tired and depressed are two different things (and being able to get things done is a third "barrel"). At a minimum you must turn this statement into two separate questions.

You may want to add a neutral category.

You may want to add degree of agreement or disagreement.

But you must address the confounded nature of the question.

EXAMPLE: "These days I find it hard to get anything done." Do you strongly agree, agree, disagree, or strongly disagree with this statement?

(5) Sex

Jerry deserves whatever he gets here! First, of course, is that this is not a complete sentence. Does Jerry mean biological sex? Does Jerry just mean sex, period? Who knows?

Please check one: are you _____Male   or   _____Female?

I would avoid using the term "sex" to ask about someone's biology because of the double-meaning. (My favorite respondent answer was "YES!")

The response category "or something else" in this case is probably unwise since less than one percent of the population is "something else" but instead you will receive silly--and uncodable--answers.

Because this is a mail (a male?) questionnaire, Jerry doesn't really know who received it. He just has a set of names. Names such as Jerry, Kerry, Terry, Lee, Leigh, or Jordan could be either sex, so Jerry really needs to double check this one, in case his population frame is incomplete or in error about participant characteristics (in many cases, all you get are names). In fact, it's a good idea to double check, even if you think your interviewer knows, in telephone surveys, OF COURSE in email surveys, and even sometimes in personal interviews.
 

Susan Carol Losh. October 28, 2002
This page was built with Netscape Composer
and is best viewed with Netscape Navigator
600 X 800 display resolution.
 
 
 

EDF 5481 READINGS AND ASSIGNMENTS

OVERVIEW