EDF 5481 READINGS AND ASSIGNMENTS

FEEDBACK ON EXAM ONE FALL 2002

OVERVIEW

Here are the scores from Exam One. The maximum possible score was 100 points.

An "A" means excellent work. A "B+" means superior work. A "B" means good work.

LETTER GRADES ARE APPROXIMATE. I will add the scores from all three exams to create a new TOTAL SCORE in December. This total score will receive a grade and this total exam score grade counts 75 percent of your final grade. You can review the GRADING site HERE.

Please read all material on this site. I will include material on BIAS, INTERNAL VALIDITY, and EXTERNAL VALIDITY, and RANDOM ASSIGNMENT VERSUS RANDOMIZATION on Exam Two because these concepts gave people the most trouble.

EXAM PURPOSES

Any exam serves several purposes. First, it should spot overall class problems in comprehension and correct them. Hence there will be a "replay" of several concepts, such as random assignment, on Exam Two. These are such critical issues in the research process that it will be difficult for you to design valid studies or accurately read journal articles if you don't understand them.

Second, an exam should assesses class mastery of the material so that I can judge the pace and sophistication of presented material. So far, despite our astounding diversity, we are doing well. The median score was 90 out of 100. We have students from Learning and Cognition, Program Evaluation, Instructional Design, Sports Psychology, Sports Administration, Reading Education, Early Childhood Education, Art Education, and more.  One third of our class has a native language other than English. Some people have master's degrees, theses, or publications, others have no research experience at all.

Third, of course, an exam should assess your individual mastery of the material, which I am mandated to do.

If your score is low, you should "trouble shoot" why so that you can create the best learning strategy for you:

• You may need some kind of tutorial help. (I have some suggestions if you ask me.) PLEASE TAKE THIS VERY SERIOUSLY. If you scored below 81, you are in potential trouble. Past experience says you will score worse on Exams 2 and 3 if you fail to get help NOW.
• Review the comments that you receive on your assignments.
• Perhaps you need to access the Web resources (sample questions, review, study guides) more often or print them out and actively work with them. These guides should be especially helpful for research novices because they highlight the material that I think is most important for this course.
• Remember to check the Syllabus and Overview sites each week: they link to all the study guides for this semester.
• Try outlining the text chapters, too.

You can reach me by email at: slosh@garnet.acns.fsu.edu

	Score 56 67 69 80 81 83 84 85 (2) 88 (2) 89 90 (3) 91 92 (4) 93 (4) 94 (2) 95 96	Grade F C C B- B B B B B+ B+ A- A- A- A- A- A A
Median = 90.0    Mean = 87      25th percentile =  84   75th percentile = 93   Standard Deviation = 9.3

If you scored below 81 on this exam, review the problem-solving suggestions above.
 
 

ABBREVIATED PORTIONS ARE AREAS IN WHICH EVERYONE DID WELL. CORRECT ANSWERS ARE ANNOTATED IN RED.

Exam One is 100 POINTS. It is 25 PERCENT of your final grade. Please follow all directions and answer ALL DESIGNATED parts of each question. Numbers in parentheses ( ) beside each question section indicate the points for that section of a question.
 

PART A: (10 total) (1) Place ANY TWO (and ONLY two) of the following variable sets in this causal order: INDEPENDENT variable, INTERVENING variable (CLASS DEFINITION), DEPENDENT variable. Place the independent variable on the far left, then the intervening variable, then the dependent variable on the far right.

(2) Next provide ONE RULE for each set that you answered to causally order all three variables (from GUIDE THREE). You may use a different or the same rule for each set.

A. Gender, Number of Student science courses elected, Adult science literacy score

Examples of rules cited could include time order (STUDENT science courses), giggle factor (if you believe taking lots of science courses turns you into a male, you may also believe that you will turn into a pumpkin on Halloween)
 

B. Size of university in numbers of students, Dominant campus teaching method (Web or print), Reading achievement scores

Examples of rules cited could include ease of change (it's harder to change the size of a university than teaching methods), time order (teaching method comes before achievement score)

C. Parental socio-economic background, Participant completed educational level, Participant occupational type

Examples of rules cited could include time order (obviously factors such as parental education or income precede the participant's own educational level), necessary condition (a particular level of education is typically required for a specific job)

The "general to specific" causal ordering rule was misused several times. This rule occurs when you have both a general measure (e.g., a total IQ score) and a specific instance of the general measure (e.g., an IQ test subscale). It does not apply when the constructs lie in totally different domains, such as university size and teaching method, or gender and number of science courses.
 

---

1. “Corporations who begin a fitness walking program for their employees have employees who score higher on Bandura’s scale of Physical Self-Efficacy.” This is one example of:

[   ]A. A conceptual hypothesis
[   ]B. An operational hypothesis  Notice how concretely the variables are specified.
[   ]C. A null hypothesis

2. “Ethnicity is unrelated to income.” This is an example of:
                                                                                    [   ]A. A conceptual hypothesis
                                                                                    [   ]B. An operational hypothesis
                                                                                  [   ]C. A null hypothesis

3. “Faculty who have mentors assigned to them have more successful academic careers.” This is an example of:

 [   ]A. A conceptual hypothesis  The variables are not concretely defined here.
 [   ]B. An operational hypothesis
 [   ]C. A null hypothesis

4. Even when we randomly assign intact groups to interventions, we often have a quasi experimental design instead of a true experimental design because:

[   ]A. Other factors may differentiate the groups before the random intervention.
[   ]B. Randomization is not a key element in creating a true experimental design.
[   ]C. We will always have regression toward the mean effects.
[   ]D. Actually, randomization guarantees a true experimental design.

The first problem solver study is a good example. Different teachers meant the groups were exposed to different experiences. Starting the study halfway through the academic year meant the groups probably differed in key respects before the study began collecting data.

5. Intervening variables are useful because:

[   ]A. They are able to become confounded variables.
[   ]B. They are multidimensional variables.
[   ]C. They are nominal level variables.
[   ]D. They explain how the independent variable operates.

As mentioned in at least two study guides (including Exam guide 1), please use the class definition of "intervening" which is more consistent with later statistics courses that many students will take.

A confounded variable is one in which several variables are simultaneously embedded. Experimental treatments that either deliberately or inadvertently include too many variables in a single treatment also create confounded variables. In the case of a confounded variable or treatment, when a treatment effect or a correlation coefficient occurs, we cannot specify exactly what the cause is. This is not true of intervening variables, which, instead elaborate or specify what the intermediate stages of a causal system are.

6. Participants in the Milgram “Obedience” documentary film were incorrectly told they were in a teaching and learning experience (rather than an obedience study) because:

[   ]A. It was an attempt to minimize subject reactivity.
[   ]B. Most experiments never use deception in their procedures.
[   ]C. This contributed to high external validity.
[   ]D. This was an example of conceptual variables.

Because participants may change their behavior when they know it is monitored for research, many investigators use deception about the purposes of the study to minimize reactivity.
 

1. (2) Is this study a CHECK ONE:

[  ] TRUE EXPERIMENT        or a         [ X ] QUASI EXPERIMENT     or        [  ] Neither one?

    (3) In a sentence or two, give the rationale behind your answer:

Well, the coin flip was better than human judgement, but the timing of the study and the use of two different teachers means we have a quasi-experiment with intact groups here.

2. (2) This study suffers from statistical regression effects    Check:  [   ]YES   or [  X  ] NO

The groups were not subdivided in any way and both were emotionally disabled classes. Except for the different teachers, there is no reason to believe Class 1 is systematically different from Class 2.

3. (2) This study suffers from history effects.                          Check:  [   ]YES   or  [ X   ] NO

Ultimately, all students received credit for this question thanks to the explanations of several. Perhaps a teacher was pregnant in one class, but not the other, or perhaps one class had a measles epidemic. Although unlikely (wouldn't a contagious disease hit the entire school??), the arguments were plausible enough that I could see the point. Originally, my feeling was the relatively short study period and the same school ruled out most history effects, but then there's those pregnant teachers...

PART D (12 total). BRIEFLY define AND give an example of ANY THREE (and ONLY three) of the following terms:

1. Bias refers to systematic differences between the true population value and the measured population value of a variables. Bias is most often due to problems with the measuring instrument (although I have seen it occur in coding open responses too.) For example, the scale that weighs 5 pounds too light or the car speedometer that underestimates miles per hour. RELIABILITY DIFFERS FROM BIAS.It is very possible to have reliable (i.e., stable or predictable) measures that are still incorrect. A big problem is that bias is often not obvious, so we miss it. For example, the results of many personality tests may reflect question format instead of the construct the researcher was trying to measure.

2. External validity refers to how well we can generalize study results to other samples, populations, or situations. The Milgram series of "Obedience" experiments was largely restricted to white, native-born males so we don't know how well the results would generalize to women or to those from other cultures.

3. Ordinal variable is one in which the CATEGORIES (or, sometimes, the participants) can be rank-ordered. The categories are more than just names, but are not numeric. Order of finish in a foot race or Likert scales are ordinal variables.

4. Regression toward the mean effects (statistical regression) refers to the tendency of those with extreme scores on a variable to "fall back" toward the mean on subsequent measures, regardless of any intervention. For example, someone with a GRE verbal score of 800 might score 780 on a retest, while the recipient of a verbal score of 200 might score arounf 220 the second time they took the test.

PART E (10 total): For ANY FIVE (and ONLY FIVE) of the following, indicate whether the variable is a CONCEPTUAL variable or an OPERATIONAL variable:
 
 

1. Annual dollars of revenue in FSU football ticket sales	[   ] Conceptual OR	[   ] Operational?
2. Best business practices (what are they?)	[   ] Conceptual OR	[   ] Operational?
3. Motivation to learn (measured how?)	[   ] Conceptual OR	[   ] Operational?
4. Science literacy (measured how?)	[   ] Conceptual OR	[   ] Operational?
5. University type as explicitly defined by the Carnegie Report, e.g., Research I, Research II, etc.	[   ] Conceptual OR	[   ] Operational?
6. Verbal scores on the Graduate Record Exam	[   ] Conceptual OR	[   ] Operational?

PART F: (9 total) For ANY THREE (and ONLY three) of the following, indicate (1) whether the variable is nominal, ordinal, or interval-ratio and (2) IN ONLY ONE SHORT SENTENCE describe the reason behind your decision in each case:

Remember to state the reason for your decision. Some students didn't do this and lost credit.

1. Bought a ticket for the Florida Lottery in the past month: yes or no

ORDINAL, "yes" bought more tickets (even if one more) than "no"
 

2. Country of origin (e.g., Brazil, Canada, Chad, China, Germany, Korea, Turkey, USA)

Can't rank the categories in any kind of agreed-upon hierarchy; NOMINAL data
 

3. Number of correct answers on an English reading short-answer exam

You can count the number of correct answers (unit = 1 answer) and can't have less than zero. RATIO

4. Years of age

You can count years of age (unit=1 year) and can't have less than zero. RATIO 

Several students may misunderstand these concepts, so look for some questions on these concepts on Exam 2.

1. In her study of student attitudes about distance learning courses, an instructor used three sections of a college junior level Educational Psychology course. This design poses a threat to:

[   ]A. Construct validity
[   ]B. Internal validity
[   ]C. External validity We're limited in who we can generalize to
[   ]D. Reliability

2. In the experimenter’s study of racial and ethnic stereotypes, he neglected to include a manipulation check. This omission poses a threat to:

[   ]A. Construct validity
[   ]B. Internal validity  If the participants don't notice the treatment at all, can it cause the results?
[   ]C. External validity
[   ]D. Reliability

3. Students from a reading training group are given a weekly reading quiz. But it is nearly impossible to predict from an earlier quiz to how the student will score on the very next quiz. These results pose the most threat to:

                                                        [   ]A. Construct validity
                                                        [   ]B. Internal validity
                                                        [   ]C. External validity
                                                        [   ]D. Reliability

We are not going to be able to conclude anything about the construct measurement or validity, because the results were too unstable, volatile or unreliable to conclude anything at all.

4. The experimental group saw a movie that included several sex scenes and two murders. The control group read a selection from the McMillan Research Methods text. This design poses a threat to:

 [   ]A. Construct validity
 [   ]B. Internal validity
 [   ]C. External validity
 [   ]D. Reliability

This is a confounded treatment, hence a threat to internal validity. CLICK HERE TO BRIEFLY REVIEW.

PART H: (30 total) PROBLEM-SOLVER: Dr. Lord is a researcher in child development who works daily with a group of preschool children from a lower middle class subdivision. All her children are enrolled in a reading readiness program to increase their prereading skills.

 In her research, Dr. Lord has learned that exposure to classical music may increase children’s prereading skills. Through a random number table, Dr. Lord assigns the children she works with to three groups: Group 1 listens to a daily hour of classical music; Group 2 listens to a daily hour of “top 40 tunes;” Group 3 colors instead of listening to music. Dr. Lord continues her other regular activities with all three groups. She measures prereading skills at the beginning of the music exposure study and every month thereafter. After three months, the classical music Group 1 has shown the greatest prereading improvement.

1. (2) Is this study a CHECK ONE:

[  ] TRUE EXPERIMENT        or a         [  ] QUASI EXPERIMENT     or        [  ] Neither one?

    (3) In a sentence or two, give the rationale behind your answer:

One group, randomly divided into three treatments. This is a classic experimental design.

2. (2) Do you think Dr. Lord used a pretest?        Check:        [   ] YES      or      [   ] NO

    (2) In a few words, why or why not?

The description said so.

3. Is maturation (or development) a plausible threat to interpreting Dr. Lord’s study results?

   (2) Check:        [   ] YES      or    [   ] NO

   (2) In a few words, why or why not?

If maturation is a threat to the study results, it means that maturation can explain why the classical music group did better than the other two. However, all three groups were created by random assignment. This means that at the beginning of the study, all three groups will have about the same proportion of early and late developers.

Typically, random assignment will control for maturation or development effects (but, of course, read each study description thoroughly just to be sure.)

4. Are experimenter effects a plausible threat to interpreting Dr. Lord’s study results?

   (2) Check:      [   ] YES      or      [   ] NO

   (2) In a few words, why or why not?

Dr. Lord is both teacher AND experimenter--always a very bad combination. She knows which child received each treatment (she isn't double blind) and she knows what the expected outcomes are. It is entirely possible that Dr. Lord will behave differently in subtle ways in each treatment group and thus influence the results.

5. (2) In general, does Dr. Lord’s study have (Check one:)    [  ] HIGH     or    [  ] LOW
external validity?

    (3) In a few words, give the rationale behind your answer.

One preschool in a lower middle class subdivision. Social class effects may either prevent Dr. Lord from generalizing very far or social class effects may interact with the treatment, for all that we know.

6. (2)   Briefly define the term REACTIVITY:

Study participants change their behavior because they know they are being studied. This can occur through self-monitoring, self-consciousness, social desirability, or other forms of experimenter demand effects.

(3) How big a problem do you think REACTIVITY among the preschool children might be in interpreting Dr. Lord’s results? ---Briefly EXPLAIN why you do or do not think reactivity would affect the study outcomes.

Not a big problem. The wonderful thing about studying students is that students are used to being tested and used to being exposed to (what for them is) new teaching methods all the time, so they are unlikely to notice much out of the ordinary. Unless Dr. Lord makes a big deal about an experiment, most students are unlikely to even realize that they are the subject of a study.

These are preschoolers, remember, and music is typically a sizable part of the preschool day. So is coloring.

The big danger isn't reactivity among the kids. Their parents would sign the consent form (children under 18 can only ASSENT, they can't CONSENT) and the kids probably will notice very little. It's Dr. Lord, who knows what she expects, and which child received which treatment (and who teaches prereading skills on top of it.)

7. (3) Briefly describe ONE FEATURE that you believe could improve EITHER the internal validity OR the external validity of Dr. Lord’s study design. BE SURE YOU DESIGNATE WHAT KIND OF VALIDITY YOU ARE ADDRESSING HERE. Describe how your improvement could create a stronger research design for Dr. Lord’s study.

Lots of excellent suggestions here:
 

• Instead of coloring, have the control group listen to "typical preschool music."
• Don't continue the study for too long--given development effects, we would expect all the groups of kids to show some improvement in prereading skills, no matter what.
• See if the study can be repeated in a wider sample of preschools, incorporating different social class backgrounds.
• Bring in different experimenters who are blind to the purpose of the study.

ALSO, PLEASE REVIEW THE WIERSMA BOOK PAGE 104 FOR THREATS TO  INTERNAL & EXTERNAL VALIDITY.
 
 

RECALL THAT IN MANY CASES, PEOPLE ARE UNAWARE THAT THEIR BEHAVIOR IS THE TOPIC OF A RESEARCH STUDY.
 
 

EDF 5481 READINGS AND ASSIGNMENTS

OVERVIEW

This page was created using Netscape Composer.
It is best viewed using Netscape Navigator
600 X 800 display resolution.
Susan Carol Losh October 7, 2002

   yes...think snow