EDF 5481 READINGS AND ASSIGNMENTS

OVERVIEW

GENERAL GUIDE:   EXAM ONE ALSO SEE:  GUIDE 1: INTRODUCTION GUIDE 2: VARIABLES AND HYPOTHESES GUIDE 3: RELIABILITY, VALIDITY, CAUSALITY, AND EXPERIMENTS GUIDE 4: EXPERIMENTS & QUASI-EXPERIMENTS GUIDE 5: A SURVEY RESEARCH PRIMER

EXAM 1 IS OCTOBER 2. OFFICE HOURS  SUSAN MONDAY 9/30 2-3 WEDNESDAY 10/2 1-3:30. CHRIS TUESDAY 10/1 2-5 215M STB

Susan: slosh@garnet.acns.fsu.edu Chris: tavani21@hotmail.com

WE WILL NOT ANSWER EMAIL MESSAGES RECEIVED AFTER 10 PM TUESDAY OCTOBER 1.umtch
WE ARE NOT RESPONSIBLE FOR ANY DELAYS IN EMAIL DUE TO SERVERS (some of which are quite slow).
IF YOU EMAIL WEDNESDAY MORNING, WE WILL NOT HAVE TIME TO RESPOND TO YOU.
THANK YOU FOR YOUR CONSIDERATION.
 

COVERAGE

DIFFERENCES FROM THE BOOKS

BASIC CONCEPTS

APPLICATIONS

SAMPLE QUESTIONS

This exam covers chapters 1-6 in Wiersma and chapters 1-3, 6,  8 (pp. 180-186 ONLY), and 9 in McMillan, and all lectures, videos (including the Milgram documentary), demonstrations, and course Web sites through Guide 4 and associated links, including any material I have placed in Blackboard/CourseInfo.

Exam One is 100 points and should take about one hour to complete. It counts 25 percent toward your final grade.

In some cases you will be asked to choose the sections of a question that you answer, e.g., select three out of four sections. The purpose of this is to allow you to show off the areas that you know the best. DO NOT answer all choices in such instances. No extra credit! I only grade the first number of designated selections if you answer all the selections in these cases. So what can happen is that (for example, in a 3 out of 4 selection question) you get parts 1, 2 and 4 right, but I only grade parts 1, 2, and 3, so your credit is lower than if you had simply answered 1, 2 and 4.

The exam is a mix of multiple choice, fill in the blank, and short essay questions. You may add a SHORT explanation to any short-answer question.
 
 

WHAT WON'T BE ON THE EXAM

Exam coverage from the texts is selective. For example, although both Wiersma and McMillan have excellent chapters on doing literature reviews, I will not have questions about them on this exam. These are valuable chapters to use in your own research, regardless of any exam questions.

I will not have articles for you to analyze as McMillan does in his excellent examples.Although I DO expect you to know that you must state your research problem early in the Introduction, I will not test you on this part of the chapter. I will have you critique study designs.

I do not expect you to memorize which disciplines are "qualitative" and which are "quantitative," although I DO expect you to know the general differences between these two types of research.
 
 

WHAT WILL BE ON THE EXAM

I expect you to know the differences among reliability, construct validity, internal validity, and external validity. There may be a series of multiple choice questions with examples of each.

I want you to know about threats to internal and external validity, and the enhancements your designs can make to both these types of validity. You will have short essay and multiple choice questions in these areas.

I expect you to be able to place variables in nonexperimental studies in causal order (independent, intervening, dependent). There will be a series of these questions. You will need to be able to describethe rule that you used to causally order the variables too. "The rule" means one of the rules for establishing causality in nonexperimental data that you will find in Guide Three.

CLICK HERE TO REVIEW THESE RULES!

I expect you to know about properties of variable category systems and levels of measurement in data. You will have some examples of these.

I expect you to recognize different types of experimental and quasi-experimental designs and how these relate to threats to internal validty and external validity. You will need to know the advantages and disadvantages of different types of experimental designs.

You need to recognize the difference between a "true" experimental design and other kinds of designs.

You need to be able to identify different kinds of hypotheses: conceptual, operational and null.

Similar to the experimental critique in Assignment 2 , you need to be able to trouble shoot and problem solve different kinds of experimental and non experimental designs.
 

In general, your books and I agree on terms and usage. Here are a few exceptions that you need to know about for Exam 1.
 
 

1. INTERVENING VARIABLE

Recall that here is an important case where we differ:

I define an intervening variable as one that links the independent variable to the dependent variable. Thus, an intervening variable forms part of a causal chain:

INDEPENDENT VARIABLE -------> INTERVENING VARIABLE ------> DEPENDENT VARIABLE

PLEASE USE THIS DEFINITION FOR THIS COURSE.

This definition is also consistent with the way that most statisticians use the term in Structural Equation Modeling. (Some statisticians use the term "mediator variable" or "moderator variable" for intervening variable instead. But be warned! They use the terms mediator variable and moderator variable to designate other meanings too, which makes life confusing.)

Using this definition, intervening variables are NOT confounded. In fact, by disaggregating the effects of an independent variable, intervening variables actually unconfound an originally confounded variable.

Using this definition, intervening variables absolutely CAN be measured. They are critical to use in non experimental research designs. They can specify what it is about the dependent variable that is important.

EXAMPLE: educational level is a cause of science attitudes because educational level influences how many science courses someone has taken, and in turn, the number of science courses affects science attitudes.

EDUCATIONAL LEVEL SCIENCE COURSES SCIENCE ATTITUDES

Intervening variables inform us about causal sequences or chains, thus explaining the causal process of a phenomenon.

EXAMPLE: educational level  occupational type  income level

Although I would like to tell you that it is the mere possession of a degree that will bring you piles of money, unfortunately you must GET A JOB to do so in most cases. One's education prepares one for a certain level of job, which, in turn, produces income.

As you can see, intervening variables, both conceptually and operationally, are very important for all but the very simplest causal assertions. It will be useful to you to practice thinking in these causal chains and speculating about precisely which intervening variables are the most critical in outcomes.

2. REACTIVITY (subject effects)

AGREEMENT:

Both our Wiersma book and I AGREE that reactivity designates changes that people make in their behavior when they know that they are being studied. This is almost certainly due to increased self-awareness and self-monitoring. People also experience evaluator or evaluation apprehension because they may fear that their behavior in the laboratory somehow will be deficient so they change their behavior. The experimental laboratory is probably the most reactive because people have come for an experiment and they know their behavior is being watched, although it can occur in other study designs too.

That is why so many experimenters use deception. They are trying to divert subject attention so that the "true behavior under study" is not altered.

DIVERGENCE:

Wiersma lists reactivity as a threat to external validity, that is, the ability to generalize to other situations.While I don't disagree, I think the far greater threat is to INTERNAL validity.

Reactivity introduces an alternative causal explanation for our results: they occurred, not because of the experimental manipulation, but because people were so self-conscious that they changed their behavior. Reactivity may also statistically interact with the experimental manipulation. For example, if the treatment somehow impacts on self-esteem (say you are told that the stories you tell to the TAT pictures indicate your leadership ability), reactivity may be a far greater problem.

3. QUASI-EXPERIMENT

Our Wiersma book defines quasi-exeriments as research using treatments and intact groups (i.e., groups pre-existing to any treatment or intervention). While I agree that interventions with intact groups probably are quasi-experimental 75-80 percent of the time, there is the other 20-25 percent, outlined in the paragraph below.

I use a more inclusive definition of quasi-experiment: If your study has different levels of treatments or interventions, and participants are assigned to those treatments WITHOUT random assignment, you have a quasi-experiment.

In a "true experiment," subjects are randomly assigned to treatment or intervention groups using a coin flip or some other probability, non human judgment method. Randomization is what makes true experiments so strong in internal validity. It means that on the average at the beginning of a study, all your treatment groups are about the same or "pre-equivalent." Thus, in a posttest only design, we can reasonably state that the differences we find are due to the treatment (all things equal, or "ceterus paribus"). For example, randomization controls for self-selection bias, history, instrumentation, regression effects, and other threats to internal validity.

Randomization just isn't always possible. Some treatment groups are initially formed on the basis of performance (high, medium, low, for example), some variables (e.g., bipolar depressive disorder) just aren't experimentally induced. Individuals who are not in intact groups could enter the treatment levels in your quasi-experiment through self-selection, because of their particular performance category (that bottom quartile in Intramural sports), or because a researcher has "paired" individuals a priori that she or he believes are somehow similar.

What are important are: (1) HOW subjects entered the groups in the first place; (2) What happens in the group; and (3) The length of time groups pre-existed prior to interventions.

If subjects are randomly assigned to groups in the first place (which often happens in school and universities for classes where there are many equivalent sections), the tasks to be performed are virtually identical in each group, and the pre-intervention time is short (probably a few weeks at most), THEN if you randomly assign groups to conditions, you probably have a true experiment.

Consider some alternatives:

• Subjects are assigned to groups using pre-existing knowledge about the subjects and the groups consequently differ on variables related to the study. Sports teams grouped by ability, "tracking" systems in schools, and enlisted versus officers in the military are three examples.

• Even if random assignment originally places subjects in groups, their curricula and itineraries may be different, thus providing subjects in different groups with different experiences.

• Finally, bosses and teachers differ in their approaches, again providing subjects in different groups with different experiences which diverge further as time goes on.

THE BASICS:
HINT: CHECK OUT THIS CLASS WEB SITE FOR THESE TERMS 

Conceptual variable
Operational definition
Operational variable

Independent and dependent variables
Rules for assigning causality in non experimental data

Recognize examples of the following and be able to define the following:

LEVELS OF MEASUREMENT
Nominal variable
Ordinal variable
Interval variable
Ratio variable

HINT: CHECK OUT THIS CLASS WEB SITE FOR THESE TERMS 

Reliability
Bias
Construct Validity
Internal Validity
External Validity

THREATS TO INTERNAL VALIDITY:

SEE YOUR WIERSMA BOOK, PAGE 104! IT IS AN EXCELLENT SOURCE and

HINT: CHECK OUT THIS CLASS WEB SITE FOR THESE TERMS 

HINT: CHECK OUT THIS CLASS WEB SITE FOR THESE TERMS 

Experimental reality (sometimes called artificiality)
Mundane reality

Manipulation check

Solomon Four Group Designs are more expensive because they require more subjects and conditions than other types of experimental treatments. But, many researchers believe the advantages are worth the expense. Notice how in addition to controlling other threats to internal validity, this design also controls for pretest effects, and pretest/treatment effects.

HINT: CHECK OUT THIS CLASS WEB SITE FOR THESE TERMS 

Hypothesis
Conceptual Hypothesis
Operational Hypothesis
Null hypothesis

Confounding or confounded variable: a multidimensional variable, for example, educational level which measures social class, cognitive sophistication, and exposure to diversity.

Randomization

Does the CONTROL GROUP simply have "nothing happen"? Or is it an ALTERNATIVE TREATMENT GROUP?

"Levels of treatment" (THIS DIFFERS FROM: Variable Levels of Analysis)
Experimental demand ("demand effects")
"Organismic" variable (or naturalistic variable)

How does a PRETEST differ from a PILOT TEST?

Interaction effect (also known as "statistical interaction")
Repeated measures design (Wiersma sometimes calls it "time series")

IMPORTANT NOTE: While randomization or random assignment of subjects or groups to treatments/interventions is very important for internal validity, notice it says NOTHING about how you obtained your subjects or groups in the first place. Thus randomization is NOT connected to external validity.

Randomization is different from simple random sampling so do not confuse them (although both are probability-based methods).
 
 

HERE for a fast review.

 
 

Given three variables in nonexperimental data, can you designate which one is the independent variable, which the intervening variable, and which the dependent variable? Can you give the rule behind your decision?

Can you decide if a causal relationship is asymmetric or symmetric?
Do you remember what asymmetric (CAN designate a causal variable) and symmetric (CAN'T designate a causal variable) are?

Can you decide accurately whether a variable is nominal, ordinal, or interval-ratio level?
This one is much more difficult than you may think.

Do you know the difference between a conceptual variable or definition and an operational variable or definition? When writing a research report, where is the appropriate spot to use each kind of variable or definition?

What are the key components of an informative research problem statement? By the end of your second written page, what should be accomplished in describing your research problem?

Can you write a clear hypothesis linking two variables? Can you convert this to a null hypothesis (if the data allow)?

What distinguishes a "true" experiment from a "quasi" experiment?

What does it mean to say a variable is "confounded?" Give an example and explain where the problem lies.
Can an hypothesis be confounded too?
 
 

Place the following three variables in this causal order (the original order is alphabetical):
independent variable, intervening variable (IN-CLASS DEFINITION), and dependent variable.

Provide TWO RULES FOR CAUSAL ORDER IN NON EXPERIMENTAL STUDIES that you used to order these variables.

Place the independent variable on the far left, then the intervening variable, and the dependent variable on the far right:

• 1. Computer software type, student learning preference, student reading achievement test score
• 2. American University grades. country of origin, TOEFL score,
• 3. College football player season touchdown score, professional draft pick status. size of undergraduate college

Country of origin clearly (and unambiguously) is the first independent variable and could affect TOEFL score. By the time the international student has come to this country, they ALREADY took the TOEFL test, so both country and TOEFL could affect grades (neither TOEFL nor grades could EVER affect country of origin--logic).

The football player is already at a college, so size could affect touchdown score, but not the other way around (time order). Both college size and player performance would affect pro draft pick status. Since draft pick status is the outcome, it must be the dependent variable.

 An independent variable is a cause, a dependent variable is an effect, and an intervening variable comes in between.

Ease of change and time order are critical rules. Education DOES NOT change someone's ethnicity. Teacher salary or satisfaction CANNOT change geographic region. Since both of these are the most difficult to change, they must be independent variables.

The "general to specific" causal ordering rule  occurs when you have both a general measure (e.g., a total IQ score) and a specific instance of the general measure (e.g., an IQ test subscale). It does not apply when the constructs lie in totally different domains, such as ethnicity and education, or geographic region and teacher satisfaction.

1. “Art students who participate in their own grading process are more motivated about course material.” This is one example of:
                                                                                [   ]A. A conceptual hypothesis
                                                                              [   ]B. An operational hypothesis
                                                                                [   ]C. A null hypothesis

Variables such as "grading process" or "motivation" are "generic"--there are no concrete operations mentioned.

2. “There is no gender difference in basic science knowledge.” This is one example of:

          [   ]A. A conceptual hypothesis
          [   ]B. An operational hypothesis
        [   ]C. A null hypothesis

No relationship between the variables is postulated so this is a null hypothesis.

3. The Obedience documentary film, while striking, did, however have problems with:

[   ] A. External validity
[   ] B. Face validity
[   ] C. Internal validity
[   ] D. None of the above

On the "face of it," Milgram's measures were definitely related to harming others and making study participants believe that they could have or had harmed others. The study treatments were varied and participants were randomly assigned to them. Although Milgram made more effort than most laboratory experiments to get a range of participants, he did move off the Yale University campus, and used adults as well as college students, his subjects were apparently all European White background and all male. We simply have no idea whether women or non Whites (Blacks, Asians, Hispanics, etc.) would have comparable responses.

Coach Large wanted to examine how a new visualization technique affects Intramural player performance. At the very beginning of the academic year, he divided his total group of players into two teamsusing a random number table. Group One received the visualization technique while Group Two was told to “think positive thoughts.” At the end of the season, Group One won significantly more games than Group Two.

1. Is this study a CHECK ONE:             [  ] TRUE EXPERIMENT or  [  ] QUASI EXPERIMENT?
    In a sentence or two, what is the difference between a “true experiment” and a  “quasi experiment?”

Coach Large had an experimental design, because he used a random number table to randomly assign players to the two experimental conditions. There is no indication of initial intact groups so we have a "true experiment."

2. Did this study use a Solomon 4 Group design?

No way, there are only two groups!

3. Did Coach Large have high external validity?           Check:   [   ] YES   or   [    ]NO

No, because he only studied students who signed up for Intramural sports that season at one particular school. Thus, he is very limited in being able to generalize. However, because Coach Large used random assignment at the beginning of the academic year, his study has high internal validity.

4. Describe TWO DIFFERENT threats tointernal validity that you see in this study design. Briefly describe what each threat is (e.g., testing, regression effects, etc.) and then tell WHY this is a problem for the internal validity of this design.

Experimental mortality could be a problem. If Group One found the visualizing too difficult, more people would drop out, creating a variation of a "self selection" effect.

The control group was uninformative. It really didn't have much to do, so we have no idea how much more effective visualization techniques are than other methods a coach might try (e.g., repeated practice drills, rotation of team leaders). This is akin to saying a new antihistamine is more effective than a "sugar pill" but not whether it is more effective than other (maybe cheaper) antihistamines that are already on the market and in common use.
 

Taking a probability sample means you can generalize with known error rates to a defined population. Varying time and place will allow you to generalize to more situations. Thus both address external validity.

If experimental subjects don't notice your manipulations, it is unlikely your treatments will have much effect! Having reliable and valid instruments means you can draw better cause-effect conclusions because you know what your measures mean. Both address internal validity.

Briefly describe the POST-TEST ONLY DESIGN.
Which research method can use this design (experiment? survey? ethnography? etc.)?
Diagram this design (Wiersma's terminology is fine.)
Briefly describe two problems with a POST-TEST only design.

Study participants only receive an assessment of after treatment outcomes. There is no prior measurement or pretest. In (more usually) experiments and (far less often) quasi-experiments, the intervention or treatment is applied and then measurements are taken on the dependent or outcome variable. Because subjects typically are randomly assigned to treatment groups, the assumption is that they are basically alike on the average prior to the treatment.
 
 

Group #	Pretest	Treatment	Posttest
One	No	R X1	01
Two	No	X2	02

Without a pretest, you cannot measure improvement or net gain or loss in the dependent variable.

Pretest scores may have an important impact on how the treatment is received (e.g., better students may take greater advantage of a new technique than poorer students).

Pretests allow the researcher to establish a "benchmark" or baseline. For example, you may discover that students have absolutely no prior knowledge about a particular topic.

1. Choice of response alternative to the statement “I am anxious:” “A lot like me;” “Somewhat like me;” “A little like me;” or “Not at all like me.”

Ordinal. You can rank the five categories from "a lot like me" to "not at all like me" but you don't have a common unit (such as a year or a dollar) between adjacent categories.

2. Gender, male or female.

Nominal. You can't rank the gender categories "male" and "female" as higher or lower or more or less. There is no agreed upon rank order.

3. Part-time employed versus full-time employed faculty status.

Only two categories, but still ORDINAL because full-time employed faculty work more hours than part-time faculty.

4. Student graduate program: Art Education; Early Childhood; Instructional Design; Learning and Cognition; Math Education; Reading Education; Sports Administration; Sports Psychology.

Graduate program consists of nominal categories.

5. Total number of graduate semester hours elected at Florida State University.

This is interval-ratio data: it has an equal unit (a semester hour) and even a fixed zero (0 credit hours).

1. Because the treatment was administered first, it is the independent variable and the relationship is asymmetric. (Time order.)

2. Oops. Symmetric. We can't designate a causal variable in this observational study without repeated time series measures. It is likely that students performing well are more positive toward school. Conversely, a positive attitude probably helps the student get better grades.

3. This one is FOR REAL. Why? Students with summer birthdays are usually more immature than those with spring and fall birthdays because they are several months younger. A few months can make a big difference in children's behavior. Clearly the student's birth preceeded any other measure you can take so it is the independent variable. The relationship is asymmetric (time order). Notice in some cases (like this one) how easy it is to establish causality in observational data.

4. Symmetric again. This time we have observational data and we CAN'T establish an independent variable. Better students are probably encouraged by others and have higher degree aspirations. Students with higher aspirations know they need good grades...The logic could go either way.

Statistical regression is a common occurrence that happens with individuals or groups who initially score at the extremes (very well or very poorly). More measures, even reliable ones, have a certain amount of error variance in them and group and or individual scores also have a certain amount of error variance. For example, you probably do better on an exam when you are well-rested and worse if you have a bad cold.

A school originally designated as "failing" because its average standardized achievement test scores are so low the first time a test is administered cannot do any worse the second time. Due to error variance, in fact, it will probably score a little higher the next time students take the test. Conversely, a school with the top average score probably can't do any better and on the following administration, due to error variance, will probably do a little worse. In other words, the very high and very low scoring entities tend to fall toward the middle or "regress to the mean" on subsequent administrations of the measure. Thus statistical regression is a "normal" phenomena.

If you have selected a wide variety of individuals or groups to study, you probably won't notice much statistical regression. The danger occurs when the researcher only selects extremes to study. After the treatment is administered, the low scoring group looks like it is doing better and the high scoring group looks like it is doing worse. This is due to statistical regression rather than the intervention (but the researcher often interprets the results as though they were due to the intervention.)

The most dangerous case is when only very low scoring groups or individuals are selected for the intervention. The well-meaning (but poor methodological) researcher wants to help those who appear to need it the most. Sure enough, on the next administration of the outcome measure, the group looks like it is doing better and by mistake the investigator attributes the improvement to the treatment, not to regression toward the mean--which would have happened in any event, treatment or not.

The solution? Generally, avoid selecting extreme groups to study. If you must, split the extreme group of participants in half randomly. Half receive the treatment, half do not. That way, at least, you have a partial control for statistical regresssion effects.

1. Is abstract, we don't know exactly how math achievement will be measured. CONCEPTUAL.

2. Scores on a specific test are OPERATIONAL. You have specified exactly how the variable will be measured.

3. OPERATIONAL AGAIN. A specific teaching method is exactly how you will measure teaching choices of method.

4. Calculator? Computer? Electronic slide rule? Who knows. The lack of a concrete operation for measurement makes this one CONCEPTUAL.

1. Case study design
2. External validity
3. Maturational threat to internal validity
4. Null hypothesis
5. Pretest-Posttest experimental design

HERE IS AN INSTANCE WHERE (1) YOU WRITE ON ONLY THREE TERMS and (2) YOU MUST ANSWER BOTH PARTS OF THE QUESTION: THE DEFINITION and THE EXAMPLE TO RECEIVE FULL CREDIT.

I'm going to let you look these up for the most part but I will do one as an example:

A null hypothesis postulates no relationship between the independent or dependent variables (or no relationship among the variables for a symmetric relationship).

EXAMPLE OF NULL HYPOTHESIS: Gender is unrelated to basic science knowledge score.

EDF 5481 READINGS AND ASSIGNMENTS

OVERVIEW

Susan Carol Losh  September 22 2002
This page created with Netscape Composer and
is best viewed with Netscape Navigator
800 X 600 display resolution.