EDF 5481 READINGS AND ASSIGNMENTS

FEEDBACK ON EXAM TWO FALL 2002

OVERVIEW

Here are the scores from Exam Two. The maximum possible score was 100 points.

An "A" means excellent work. A "B+" means superior work. A "B" means good work. If your grade is lower than B, you may need some tutorial help.

LETTER GRADES ARE APPROXIMATE. I will add the scores from all three exams to create a new TOTAL SCORE in December. This total score will receive an exam total letter grade and this total exam grade counts 75 percent of your final grade. You can calculate an approximation of your percent by adding the scores from Exams One and Two and dividing by 2.   95 percent + is an A; 90-94 percent is an A-; 86-89 percent is a B+; 81-85 percent is a B; grades for averages 80 percent and under will depend on the average percent.

Please read all material on this site. Exam Three will have a few questions about sampling issues.

EXAM PURPOSES

Any exam serves several purposes. First, it should spot overall class problems in comprehension.

Second, an exam should assesses class mastery of the material so that I can judge the pace and sophistication of presented material.

I am especially pleased with class performance on Exam Two. This was an excellent set of exams!

Don't worry if you don't feel confident about the material yet. In any introductory methods course, it is typical to feel uncertain about how well you are doing because the material is new for many and you are basically learning a new vocabulary and grammar. It may take some students a few methods courses to feel very comfortable with the material.

Third, of course, an exam should assess your individual mastery of the material, which I am mandated to do.

If your score is low, you should "trouble shoot" why so that you can create the best learning strategy for you:

• Have you read the material? This includes the two texts and the websites.
• You may need some kind of tutorial help. (I have some suggestions if you ask me.)
• Review the comments that you receive on your assignments.
• Perhaps you need to access the Web resources (sample questions, review, study guides) more often or print them out and actively work with them. These guides should be especially helpful for research novices because they highlight the material that I think is most important for this course.
• Remember to check the Syllabus and Overview sites each week: they are linked to all the study guides for this semester.
• Try outlining the text chapters, too.

You can reach me by email at: slosh@garnet.acns.fsu.edu

Chris has office hours on Tuesdays, 2-5 in 215M Stone.
His email is:
tavani21@hotmail.com
 

As you can see below, students generally did about one-half letter grade better on Exam 2 than on Exam 1. Congratulations!
 

If you scored below 81 on this exam, review the problem-solving suggestions above.
 

	Score 74 77 81 84 86 88 (4) 90 (3) 91 (2) 92 94 (2) 96 97 (2) 98 (3) 99 (1) 100 (2)	Grade C+ B- B B B+ B+ A- A- A- A- A A A A+ A+
Median = 91   Mean = 91  25th percentile = 88  75th percentile = 97  Standard deviation = 6.89

ABBREVIATED PORTIONS ARE AREAS IN WHICH EVERYONE DID WELL. CORRECT ANSWERS ARE ANNOTATED IN RED.
 

PART A: (4 total) (i. Place ANY ONE (and ONLY one) of the following variable sets in this causal order: INDEPENDENT variable, INTERVENING variable (CLASS DEFINITION), then DEPENDENT variable. Place the independent variable on the far left, then the intervening variable, then the dependent variable on the far right. (Below they are in alphabetical order.)

(ii) Next provide ONE RULE for that set of variables that you used to put them in causal order. (You may use more than one rule if appropriate.) Give a sentence or so to explain why.

A. Computer in the household, Elementary school child’s grades, Parental education

B. Geographic region, School average reading test score, Type of school teaching method

Did you remember to use one of the rules from Guide 3 to justify your answer? Did you use material from Guide 2 on designating independent and dependent variables?
 

Review: 

Also review: 
 
 

PART B: (10 total) For each of the following statements, please circle whether TRUE or FALSE is the best or more appropriate answer.
 

Intervening variables help us to disentangle causal effects.	TRUE     OR      FALSE
Experiments control for any confounded variables.	TRUE     OR FALSE
Operational variables are never confounded.	TRUE     OR FALSE
Only experiments allow us to make causal inferences about our results.	TRUE     OR FALSE
Experiments allow us to make stronger causal inferences than surveys do.	TRUE     OR      FALSE

1. TRUE. Using the class (and SEM) definition of intervening variable means that we can isolate, identify, and measure HOW the independent variable exerts influence through intervening or mediating variables.

2. FALSE. Any confounded variables? That's a tall order. Consider this experimental example from Exam 1:
 

3. FALSE. Suppose you have 25 Likert items about prejudice in a row. That is an operational (concrete measure). The measure also has potentially confounded prejudice with acquiescent response set.
 

Did you recall? A "Likert Scale" refers to a VERY SPECIFIC type of measure. The respondent is given a series of statements and is asked how much s/he agrees or disagrees with each statement. You CANNOT use the term "Likert Scale" to refer to any ordinal attitude measure.

4. FALSE. Remember that burning cigarette with the lung cancer picture in Guide 3? If you believe #4 is true, you cannot possibly believe that cigarette smoking causes lung cancer (as 96 percent of Americans do believe) because no society has forced a randomized group to smoke cigarettes for 20 years and another randomized group to abstain. In many cases, we need to apply the Guidelines for Causality in Non Experimental data because experiments are not possible.

5. TRUE. On the other hand, a well-planned experiment does have stronger internal validity and can make stronger causal inferences than other types of research, including well-planned surveys. For example, the results of the Hormone Replacement Therapy experiment released last summer are considered more definitive than the hundreds of thousands of cases observed in previous epidemiological research--and their results were nearly the reverse of the observational studies. Why? It appeared that self-selection (normally controlled in randomized experiments) meant that women who voluntarily saw their doctors for estrogen replacement also took better care of themselves (exercise, diet, etc) than women who did not, and thus these women has better survival rates.

PART C: (10 total) Select ANY TWO (and ONLY two) of the following causal assertions. First decide if the causal assertion is largely plausible OR implausible, that is, whether that causal assertion “makes sense” given guidelines for causality in non experimental data. THEN if it IS causal, decide if the assertion is symmetric or asymmetric. THEN provide a CAUSAL RULE that makes the assertion plausible (if implausible, what rule was violated?) THREE PARTS!
 
 

1. In your analysis of a large national database, you find that age influences extent of Internet use in a large, statistically significant way. You conclude that one’s age affects Internet use.

This causal assertion is (CHECK ONE)  [  ] LARGELY PLAUSIBLE or [   ] LARGELY IMPLAUSIBLE

The causal assertion is (CHECK ONE)     [  ]Asymmetric   [  ]Symmetric   or   [  ]Non causal

because:
 

2. Among a large sample of adults, you find a strong, statistically significant positive correlation between the number of science courses and basic science knowledge. You conclude that exposure to science courses makes individuals more knowledgeable about science.

This causal assertion is (CHECK ONE)  [  ] LARGELY PLAUSIBLE? or [   ] LARGELY IMPLAUSIBLE

The causal assertion is (CHECK ONE)     [  ]Asymmetric   [  ]Symmetric   or   [  ]Non causal ?????

because:
 

3. Teachers are interviewed at the end of the academic year. You find that the more a teacher likes a student, the higher the student’s grade. You conclude that teacher preferences affect student grade point average.

This causal assertion is (CHECK ONE)  [  ] LARGELY PLAUSIBLE or [   ] LARGELY IMPLAUSIBLE

The causal assertion is (CHECK ONE)     [  ]Asymmetric   [  ]Symmetric   or   [  ]Non causal

because:
 
 

THIS QUESTION WAS THE MOST PROBLEMATIC OF THE ENTIRE EXAM. Therefore, unless you stated something clearly incorrect, you almost certainly got 2 or 3 points credit.

What happened? I envisioned this question as a straightforward very short essay on a group self-administered survey, that is, when surveys are handed out to a group (such as your friend's introduction to Educational Psychology class) and the respondents INDIVIDUALLY complete the survey questionnaire.

However, obviously I had NOT described this survey option clearly enough because from responses it appeared that several students thought this was a focus group, and answered accordingly. Others apparently envisioned a group collectively filling out a survey--and answered accordingly. And still others used the definition I had in mind in the first place.

SO! Hopefully, everyone now knows what a group self-administered survey is (review Guide 5 too). Specifically, what are some of the problems with this type of survey? (Its advantages are that it is cheap! cheap! and usually someone is there to generically clarify questions.)

• Typically the samples are grab convenience samples, thus you can't generalize to anyone.
• Respondents are less motivated because they don't get individual attention. They may fail to complete all the questions or not even fill out the survey at all.
• Respondents may hesitate to ask any questions about the survey items for fear of appearing stupid in front of their peers. Or they may be too shy to ask in a group situation.
• Literacy issues could be a problem, as they are for any self-administered survey.
• Because you have people in a group, very often the group is homogenous. For example, if you use classes, you have the typical cluster sample problem: restricted variation on age, educational level, or even interests.

PART E: (9 total) For ANY THREE (and ONLY three) of the following four, indicate (i) whether the variable is nominal, ordinal, or interval-ratio and (ii) IN ONLY ONE SHORT SENTENCE describe the reason behind your decision in each case:

DID YOU REMEMBER TO WRITE PART 2?

1. Answer to the question: “Do you own at least one automobile? Yes or No.”

Anyone who owns one car owns more than someone who has none. Ranked categories, ordinal.

2. Degree type: vocational school; liberal arts college; advanced disciplinary degree (e.g., MA or PhD); advanced professional degree (e.g., law or medicine)

I originally considered these categories as differences in type (a professional degree may involve the same years of school as a PhD but the training in the former typically is much more clinical), thus nominal. However, many people gave convincing arguments (with some category collapses) for ordinal ranked categories. If you had a solid explanation for ordinality, you received full credit; otherwise, you received partial credit.

3. Religious affiliation (e.g., Buddhist, Christian, Jewish, Muslim, Shinto, etc.)

Can't rank the categories into higher or lower, just the same or different. Thus, nominal.

4. Total family expenditures last year in dollars

Common unit with equal intervals (one dollar) and an absolute zero (you can't spend less than $0). Thus interval-ratio (in fact, ratio).

1. Which ONE of the following is true about bias?

    [  ]A. Biased measures are less precise.
    [  ]B. Biased measures systematically differ from the true population value.
    [  ]C. Unbiased measures detract from high construct validity.
    [  ]D. Unbiased measures provide greater external validity.

Remember that scale that always weighs 5 pounds too light? It may be precise but it systematically underestimates weight by 5 pounds. Bias could be precise or it could be unreliable--the big problem is that it is systematic and often unknown.

2. Which ONE of the following general statements is true about samples?

    [  ]A. If you have a probability sample, you will have high external validity.
    [  ]B. Only EPSEM samples allow us to generalize to the larger population.
    [  ]C. Only probability samples allow generalizations with known rates of error.
    [  ]D. The sample size is more important than the method of sample selection.

Review Assignment 3 Feedback and see how narrowly Jerry defined his population. Since he took a probability sample, he can generalize to that population, but it was so narrow he can't discuss women coaches, private schools, or head coaches other than football or basketball (who may be under more stress; hence to eliminate other coaches may produce biassed overestimates of coach burnout).
 

3. Which ONE of the following is true about disproportionate stratified probability samples?

[  ]A. They are useful to study relatively small subgroups.
[  ]B. They give each study participant an equal chance of selection.
[  ]C. You cannot estimate errors or other statistics with this type of sample.
[  ]D. You cannot compare across groups if you use this type of sample design.

Disproportionate samples use UNEQUAL probabilities of selection (thus B is false) but are STILL probability samples (thus C is false). Although you can compare across groups, you will need to apply weights if you want to do population estimates from your entire sample to turn the entire sample back into a "self-weighted" sample (thus D is false). However, you may want to oversample groups with relatively small proportions in your population (female physics or chemistry graduate students, for example) to ensure enough of them to study (choice A).

4. Which ONE of the following terms is the most relevant to the type of sampling you use to select study participants (even if it is only a necessary, not a sufficient condition)?

[  ]A. Construct validity
[  ]B. Criterion validity
[  ]C. External validity  Probability sampling is a necessary condition to generalize.
[  ]D. Internal validity

Recall, however, from Assignment 3, that just doing a probability sample is NOT ENOUGH for high external validity. If you define your population in very narrow terms (as Jerry did), you will have a difficult time finding similar groups or populations to which your findings MIGHT apply. (Also see the problem solver question below on generalizing.)

5 Random assignment of participants to experimental or intervention groups usually controls for which ONE of the following design problems?

[  ]A. Confounded variables
[  ]B. Developmental or maturation effects
[  ]C. Experimental mortality
[  ]D. Mundane reality

These effects should be about the same across all the intervention or treatment groups, e.g., you will have about the same number of early-maturing teenagers in each group.
 

1. Absolutely true, since the researcher will not have to pay travel costs and use interviewer time for travel. Insurance typically will not be needed for the interviewers either.

2. False. Functional illiterates cannot read or write. Therefore, this would cause problems for any form of self-administered survey, but will not if an interviewer reads the survey and writes down the answers.

3. False. As noted in class and in Guide 5, all these features make the respondent more difficult to reach and have increased the amount of telephone numbers that must be called by AT LEAST 20 percent. People often change cell phone companies to take advantage of better rates and typically there are not directories for cell phones because there are so many companies. Voice mail and caller ID mean that many people do not answer their telephones unless they wish to take a call from an individual they know.

4. False. The further removed the respondent is from an interviewer (telephone, self-administered), the easier it is to answer sensitive questions (illegal drugs, abortion, etc.).
 

1. (3) Currently, the COE is considering a mailed survey to its graduate students because the administration wants to keep the costs down. They say a mailed survey “is cheaper.” What is your response to the COE administration about the cost of a mailed survey compared with ONE OTHER option (e.g., telephone, in-person, group administered) for conducting a survey?

• Well, it's probably cheaper than in-person, what with the costs for interviewers.
• But mail surveys really aren't that cheap: paper, envelopes to mail out and mail back, possibly postage (use campus mail, perhaps?), possible inducements (how about a drawing for a free semester's fellowship?), and sending out the whole packet again to people who don't respond the first time.
• If there are required courses, consider a self-administered group survey (stat courses, maybe?)
• And a telephone survey (using student volunteers to keep down costs) may be the best bet.

• Low response rates
• Language problems
• Incomplete returned questionnaires
• No interviewer to help motivate or clarify
• It takes a long time to get the data back: first, you mail the surveys out and you probably need to wait a MINIMUM of 2 weeks for them to be mailed back. A second mailout means a longer time still for data collection.

   a. (2) When all the elements in a given population are studied, this is a CENSUS
(ONE WORD, 6 letters)

   b. (2) When only a subset of a given population is studied, this is a SAMPLE
(ONE WORD, 6 letters)

(A survey is a method of collecting data through anwers to questions. You could do a survey with either a sample or a population.)

However, because there are several hundred graduate students in the COE, you are able to persuade the administration to use a random number table to select only 150 students to study. The students are numbered alphabetically and then a random number table is applied.

   c. (2) What is the technical name of this type of sample?   Simple random sample

  d. (2) Did the COE take a probability sample?     Please  Circle      YES         or         NO

  e. (2) As described, is this an EPSEM sample?     Please  Circle            YES         or         NO

  f. (2) Very briefly describe ONE OTHER POPULATION that you may be able to generalize the results of this study to:

This is THE classic probability sample. Not only does each element have an equal chance of selection, but each combination of elements has an equal chance of selection.

REMEMBER! There is much more to external validity than just using sampling error to do population parameter estimates! You also want to be able to extend your findings to other populations and/or situations. So, what populations or situations might be plausible here? Probably large Research I universities in the Southeast, such as University of Georgia or University of Florida. Although you can't use sampling error, etc., you can at least you can suggest that similar types of universities would find similar results among their College of Education graduate students.

First, the COE is considering a series of 25 questions that all take the same format:

Always true of me    Usually true of me    Sometimes true of me    Rarely true of me    Never true of me

For example, a student would be given the statement “I get nervous when the instructor presents mathematical formulas on the board” and the student would indicate how true the statement was of him or her.

(3) Briefly describe ONE problem the COE may encounter using such a series of 25 questions all with the format above:

This situation BEGS for response set, or the tendency to consistently mark one response category over the entire set of statements, virtually regardless of content. Imagine how much WORSE response set can be if you have, for example, over 400 statements (the MMPI "personality test") which take a consistent set of ordinal categories (true-false; agree-disagree; always true of me, etc.).

(2) Briefly suggest ONE way to correct or alleviate this problem:

• Use reversed items (some positive, some negative).
• Use a wide mix of item formats, including some open-ended questions.
• Avoid all open-ended questions in a situation like this one; they are difficult to code reliably and take a long time to administer. If the survey is self-administered, there are language or literacy problems and most people don't like to write a lot by hand.

REMEMBER TO DO PART 2! SOME STUDENTS FORGOT.

A. How many graduate credit hours do you have?
   _____Under 4
   _____4-9
   _____10-12
   _____13-20
   _____over 20

• 1. The time and place frames are vague. This semester? Altogether? At FSU? Someplace else?
• 2. Never (except income) break up interval-ratio data into ordinal categories (especially unequal interval ordinal categories). You can always collapse categories later. SPSS will tear through a recode statement with over 20,000 cases in less than TWO MINUTES. Leave this answer in the original number of hours.

B. Do you like statistics?  YES/NO STEM WITH NON-YES/NO ALTERNATIVES
   _____It’s one of my favorites COLLOQUIAL & CUTE, AVOID BOTH!
   _____It’s OK                }  THESE ARE BOTH VAGUE
   _____It’s not so good }
   _____It’s the “pits”SLANG! AVOID!

As one student noted: cherry pits? arm pits? (coal pits?) etc. THANKS, DONNA!

I think this is the worst exam sample question I ever wrote. Practically everything is wrong with it--so wrong a lot of people just threw up their hands and said: make this one open-ended "What do you like most about statistics?" or "What is most useful to you about statistics?"

Those are fine solutions.

You could also just make the responses "yes" or "no".

You could turn this item into a Likert statement (e.g., "I like statistics.")

C. Please tell me if you strongly agree, agree, disagree, strongly disagree or are in the middle on the following statement: Computers and the Internet make statistics easier to learn.

Oh ouch, that red "and" is the tip-off: a double-barrelled question, two questions in one. Who knows which one the respondent meant to answer (or both).

Break into TWO separate questions, one on computers and another on the Internet.
 

At this point, an Associate Dean suggests conducting several focus groups to find out how COE graduate students feel about statistics.

6. The Associate Dean tells you that he wants to use a probability sample to select the focus group respondents.

(2) Briefly, what could be one ADVANTAGE of using a probability sample to select focus group respondents?

• You would have a group that represents the population.
• You could generalize to the population of COE students
• You would have a lot of diverse opinions (NOTE: this will only be true if there are a lot of diverse opinions IN THE POPULATION, which may not be the case).
• The selection of focus group respondents will be relatively free of bias.

• If the range of opinions in the population is very restricted, you won't have much of a range of opinions on the topics.
• Generally, we don't generalize from focus groups; we are looking for in-depth opinions and how people debate their opinion on a particular topic. Thus, the COE is going to an awful lot of trouble with no real payback.
• Lots of people won't show up, turning your sample back into a self-selected sample.
• Most of the respondents may have no motivation about the topic and may not even have firm opinions.

(2) The Associate Dean suggests to you that the faculty who teach statistics should be the moderators for the focus group interviews with graduate students. What is your (printable) response?

THE PROBLEM SOLVER QUESTION ENDS

PART I: (8 total) BRIEFLY define AND give an example of ANY TWO (and ONLY two) of the following four terms. BE SURE TO ANSWER BOTH PARTS OF THIS QUESTION FOR EACH TERM YOU CHOOSE.

1. Closed question
2. Cluster sample
3. Intervening variable (class definition)
4. Non-Probability sample
 

Because most students did so well on Exam 2, Course Guides to review are at a minimum. Here are a few sites that may be useful as well as another exposition on randomization and simple random sampling.

Only if you know what are threats to INTERNAL validity and what are threats to EXTERNAL validity can you control those threats when you design your own research or when you evaluate papers and articles.

Thus, you want to know the differences between internal and external validity.

SEE YOUR WIERSMA BOOK, PAGE 104! and

HINT: REVIEW GUIDE 3 HERE: 

AND PARTS OF GUIDE 4 HERE: 

AND CHECK OUT THE GENERIC REMINDERS ON RESPONSE RATES & VALIDITY HERE: 

Random assignment can only control for factors that could be common to all your treatment groups. Thus, it CAN control for maturation and history. By randomly assigning elements from your entire subject pool, you can control statistical regression, differential selection, and self-selection too. There really is no equivalent substitute for these important controls.

However, unfortunately, randomization doesn't solve ALL interal validity problems. If subjects drop out of different treatments because, for example, one treatment is too difficult or too threatening (remember the Milgram film?), although they originally may have been randomly assigned, you may now have a self-selection bias. If experimenters are not "blind" and know the purpose of the study, they may inadvertantly give off cues and threaten internal validity that way.

Some students are still having trouble distinguishing random assignment from a simple random sample.

HINT: ONLY call it a simple random sample (and not "random sampling" or "random sampling methods" and you will cut down on the confusion).

REVIEW THIS CLASS WEB SITE FOR THESE TERMS: 
 
 

And, for more on sampling in general: 
 

Need to review strengths and weaknesses of various ways of administering surveys? CLICK HERE

EDF 5481 READINGS AND ASSIGNMENTS

OVERVIEW

Susan Carol Losh  November 17 2002
This page was created with Netscape Composer and
is best viewed with Netscape Navigator
800 X 600 display resolution.