This was a very good group of papers and I very much enjoyed reading them.

Assignment 3 WILL BEGIN WITH A STATEMENT OF THE RESEARCH TOPIC OR "PROBLEM." This way, you will have a chance to rewrite this assignment and the Research Topic Statement will count about 1/4 of the grade for Assignment 3.

Almost everyone knows what an independent and a dependent variable is.

BUT a few people don't seem to be quite sure. Review the ON PROOF and CAUSALITY site HERE.
 
 

BUT

Some people are using:

"Fuzzy" operational variables (see below). In an operational variable or definition, you MUST explain exactly how you will measure the conceptual entity.

Vague or indeterminate variables such as "personality," "best practices as defined by field experts" or "perceptions toward companies." It is time to specify exactly what you mean by personality (for example, trait anxiety), best business practices (for example, specific programs for employee improvement) or motivation (for example, achievement motivation). Notice that all these specified examples outside the parentheses are still conceptual and you have not explained how you plan to measure them.

If you plan to use some kind of test, you MUST give a minimum of information about that standardized test besides its name, for example, length (e.g., "20 true-false questions"), how widely it is used (e.g., "the WAIS is the most widely used standardized intelligence test for adults in the United States") and any other information about it, including any known information about reliability and validity. If YOU are constructing this test anew, you need to say so and convey a generic, brief description of content.

Compound variables, such as "personality and motivation" or "gender, race, age, and SAT score." Gender is ONE variable, race is ONE variable, age is ONE variable, and SAT score is yet a FOURTH variable. Each variable stands for one and only one entity. Keep it simple.

"Variables" that don't vary, e.g., "all individuals with annual incomes under $25000. When variables are turned into constants, you don't really have hypotheses (which state relationships between variables) any more. You also create a real danger of committing the "fallacy of the invisible control group" (see below).
 

Virtually everyone wrote good hypotheses, conceptual, operational, and null.
 
 

REMEMBER!

Hypotheses link ONLY ONE independent with ONLY ONE dependent variable. Too many variables at a time lead to confounded variables and treatments.

Statements of your expected univariate results ("At least 75 percent of students will pass the math test.") are hunches about fact, they are not causal explanations. Unless your research area is in a truly beginning or exploratory stage, estimates of univariate fact are usually not what scholars recognize as hypotheses.

Oh no! The Fallacy of the Invisible Control Group!

Did you know? Most adult women in the United States say that their families are more important to them than their jobs.

Well, dang! Just can't put women in high positions at work, their families come first every time!

Oops, I bet that you didn't know that MOST MEN say their families are most important to them too. In fact, the proportions of men and women "putting family first" over their jobs are nearly identical.

Why didn't you know that the sexes were nearly identical in this respect? Because most studies never ask men about their families and jobs. Due to the researcher's own stereotypes and assumptions about sex differences, they only asked women. The results were used to justify exclusionary social policies and family hostile policies at work--and they were based ON FLAWED RESEARCH FINDINGS.

These are the null hypotheses that really were not. If you have a null hypothesis that states "there is no sex difference on job and family priorities," it means you must ask BOTH SEXES about these priorities. You can't just ask women and assume you already know all about men!

You see, there was an implicit control group all along, wasn't there? That was how men were supposed to feel about families and jobs. The problem was WE JUST DIDN'T MEASURE THAT IMPLICIT CONTROL GROUP; instead we ASSUMED what that missing control group was like and wrote up our findings accordingly. This is FLAWED RESEARCH! And it goes on all the time.

Did you want to find out about how ethnicity affects attitudes toward home schooling among parents who do? Suppose you discover that Blacks and Hispanics believe academic content is more important than religious issues in the decision to home school. There is an implicit--and missing--control group(s) here. The implicit assumption is that Euro-Whites or Asians value religious issues more (even if you didn't intend to say so), or at least more than Blacks and Hispanics do.

How can you correct for the fallacy of the invisible control group? Simple. MAKE THE MISSING CONTROL GROUP VISIBLE. EXPLICITLY BRING THEM INTO THE COMPARISONS. You may find out your results aren't so striking after all--but they probably will be a lot more accurate.
 

Everyone recognized that Kim's "experimental design" was in trouble. It wasn't a true experimental design (no randomization), and it wasn't even a quasi-experimental design with comparatively high internal validity. Thus, in no way could Kim conclude that her new software program was successful, even for the treatment group. Treats to internal validity included designated groups (the investigator purposely formed homogeneous groups with respect to ability), statistical regression toward the mean effects (the subjects, being in the bottom quartile probably could do no worse, the high group couldn't do much better and would probably do a little worse at the next measurement), a "control group" that was no control group(it not only wasn't equivalent, it received no treatment at all), and various subject effects (reactivity: social desirability, evaluation apprehension, demand effects). Kim's design needed a fast infusion of random assignment, and either heterogeneous groups, or randomly dividing homogeneous groups into different treatments.

Further, Kim COULDN'T really measure improvement--she has no pretest (in fact, her posttest is curiously undefined too). If Kim initially gave some type of achievement test, before the intervention began, she MIGHT be able to make some kind of statement about improvement. Grades are a poor measure here, they probably aren't as objective as a pretest could be, and Kim was hardly blind to the treatments instituted and their purposes.
 

HAD TO DO ALL OF THE FOLLOWING:

• Correctly rewrite Assignment 1. Clear research problem statement and rationale, accurately designated independent and dependent conceptual variables, and TWO DISTINCT unambiguous conceptual hypotheses.

• Give concrete detail for the operational variables (examples: cite specific tests, a proposed questionnaire, a concrete procedure).

• Had unconfounded and singular variables in each hypothesis.

• Have clear and unconfounded operational and null hypotheses. Accurately stated two distinct null hypotheses.

• Accurately diagnose Kims research design (not a true experiment, low internal validity, no way to measure improvement), accurately cite two distinct threats to internal validity (review from Guide 4 HERE), and gave a correct suggestion to improve Kim's design.

• Recognized that randomization can remove threats to internal validity. Non-random assignment itself alone is not a threat to internal validity; it CREATES threats to internal validity. REVIEW THE WIERSMA BOOK PAGE 104. The entire page is devoted to threats to internal validity and the surrounding pages go into them in more detail.

• Had at least a score of 8 on Part 3 for a straight A.

• You still had problems with independent and dependent variables.
• Your variables were compound and confounded.
• Your operational variables were not specific and concrete.
• You treated lack of randomization as a threat to internal validity instead of a CAUSE of threats to internal validity.
• You forgot about Kim's "control group" and cited maturational or developmental factors as a threat to internal validity IN THIS CASE. (The high ability group will undergo similar development and probably experience a similar history since it is being studied at the same time.)
• You overlooked the fact that Kim had no pretest for either group so that she would have a very hard time discussing "improvement" in the students. If Kim decided to use grades, issues of bias and reliability begin to become important (not to mention that Kim was hardly "blind" to the treatment conditions!)

Susan Carol Losh. September 24 2002
This page was built with Netscape Composer
and is best viewed with Netscape Navigator
600 X 800 display resolution.

 

EDF 5481 READINGS AND ASSIGNMENTS

OVERVIEW