Assignment 4 Feedback: Research Problem & Ethnography Critique

THIS PAGE IS COMPLETED. THANK YOU FOR YOUR PATIENCE.

EDF 5481 METHODS OF EDUCATIONAL RESEARCH FALL 2002
DR. SUSAN CAROL LOSH

OFFICE HOURS EXAM WEEK: MONDAY, 12/9 1:30-3:30 AND WEDNESDAY, 12/11 3-5:30
CHRIS HOURS: TUESDAY 12/10 2-4:30 PM

EDF 5481 READINGS
AND ASSIGNMENTS
ASSIGNMENT FOUR SPECS
ASSIGNMENT FIVE SPECS
OVERVIEW
NOW POSTED: EXAM 3 STUDY GUIDE
Sooner than you think!
Wednesday December 11 at 5:30

IF I HAVE REQUESTED A COPY OF THIS, OR A PREVIOUS, ASSIGNMENT ON DISKETTE, I WOULD LIKE TO RECEIVE THEM BY OUR EXAM, DECEMBER 11, 2001. THANKS VERY MUCH!

GENERAL FEEDBACK ASSIGNMENT FOUR
LESS STRUCTURED RESEARCH STATEMENT
ETNOGRAPHIC DESIGN CRITIQUE

GUIDE 1: INTRODUCTION
GUIDE 2: VARIABLES AND HYPOTHESES
GUIDE 3: RELIABILITY, VALIDITY, CAUSALITY, AND EXPERIMENTS
GUIDE 4: EXPERIMENTS & QUASI-EXPERIMENTS
GUIDE 5: A SURVEY RESEARCH PRIMER
GUIDE 6: FOCUS GROUP BASICS
GUIDE 7: LESS STRUCTURED METHODS
GUIDE 8: ARCHIVES AND DATABASES

This assignment is worth 5 PERCENT toward your final grade.
Remember! I use plus and minus grading on assignments and for the final grade.

This Feedback page is generic. If you feel it does not address your grade on your paper, please make an appointment and we will go over your paper.

STUDY DESIGNS

It has been very enjoyable to watch your study designs evolve. Everyone is much more sophisticated about study design than they were three months ago. The designs are a lot of fun to read (how about focus groups with overweight high school kids about what they like in school vending machines?) Or ethnographies that involve participant observation at fitness centers?

At this point, nearly everyone is able to write a comprehensive, yet succinct, research statement. When you embark on your research, this paragraph should be placed somewhere on the first page of your paper, to alert your readers to the topic, its importance, and what you plan to do. Don't forget this skill when you write your thesis or dissertation prospectus!

I think that you can now fully appreciate my statement at the beginning of the semester: that your research problem statement will be the most difficult task of the semester. Even while wading through the mountains of terminology, and the strengths and weaknesses of different designs, it is surprisingly difficult to clearly and briefly state what your research is about, why the topic matters, and what, specifically, you will do.

Congratulations! You will now find that writing the rest of your research paper is much, much easier, because you truly know what your research topic is and what you plan for it. This will facilitate your catching redundancies, to only address the issues that directly relate to your research problem, and to systematically plan your data collection and subsequent analysis.

NOTE: You DID have a succinct introductory paragraph on your research problem--right? By this time, one short paragraph should suffice. Save the detail and the citations for your literature review that comes later.

Here are some points to note:

Match your study design to what you want to know. You generally cannot infer the internal states of individuals or cognitions (e.g., beliefs and attitudes) through field observations because you typically observe external behaviors. Because external behaviors can be multiply determined by many internal states, they will not give much insight into specific internal states. Ethnographies typically examine a group culture so they will not be much help in assessing attitudes or beliefs either. Both ethnographies and field observations are useful to study interaction among people, however.

Don't expect to use an oral history to learn much about actual interaction. What you will get instead are people's perceptions of interaction (useful, but probably not what you wanted to know in this case).

Alternatively, If you want more information about internal states, you may gain more insight through in-depth interviews or an oral history than through a field observation grid.

An oral history is different from a structured survey! It is much longer and uses virtually all open-ended questions. It is very unlikely that you would do oral histories with 60 students in an online course, 40 team members on a sports team, or 20 clinic members. Each oral history typically takes hours at a minimum, and some take weeks, even months. Remember, you are basically doing a type of biography and that's not a form someone can fill out in 10 minutes.

Consider what is the best sample for YOUR study purpose. Even if you take a probability sample, you will not be able to generalize from "a sample of one" (e.g., a single school, hospital, or company). A purposive sample may be your best choice in this instance, or perhaps some type of quota sample. Avoid a grab sample, however, if this is at all possible.

And, of course, you did have to give some information about whether you took a sample or a census, what kind of sample you took, and why this sample was appropriate for what you wanted to do.

Examine the answer to Juanita's proposed ethnography, question 2 BELOW.

There are many different kinds of secondary analysis. Secondary analysis is a generic term that refers to analyzing data that were originally collected for a different purpose. Thus, my current statistical analysis of how gender and education influence cognitive complexity in Internet use with the General Social Survey data is secondary analysis, because that was not the original purpose of this GSS module. Your analysis of interaction patterns in television situation comedies is also secondary analysis because these comedies were originally made to entertain (and attract advertising dollars for revenue.)

Note the differences however. I am not really "content-analyzing" the General Social Survey data, which remains a structured survey. I am building some new indices and then analyzing those but the original categories remain the same. Anyone who works with this subset of te GSS will have to take the precoded responses as given. There is very little depth to the answers because this survey almost entirely utilized closed questions. However, when I work with the NSF Surveys of Public Understanding of Science and Technology, I have some of the actual verbatim responses which I can recode into totally new and different categories.

Contrast my use of the GSS with your study of interaction patterns on television. You are content analyzing. You are taking originally unstructured data, creating your own coding scheme, and creating an entirely new and unqiue set of data. Someone else might do a content analysis of the same comedies and create a totally different schema, thereby constructing a second original dataset.

We both did secondary analysis. But most practitioners would argue that you did a content analysis and I did not.

THE A OR A- PAPER

HAD TO DO ALL OF THE FOLLOWING:

WAS COMPLETE. This means that every section of the assignment was completed.

Wrote a clear AND CONCISE (no long, long, long paragraphs, especially on paragraph one) description of their research topic, why this topic is important to study and what they planned to do. (Maybe three sentences--NOT three paragraphs!)

Accurately designated their choice of research design and described why it was appropriate. For example, participant or field observation is not generally very good at diagnosing internal states such as beliefs or attitudes. A survey design would be more appropriate here. Oral histories are lengthier and go into much more detail than a short, structured survey.

Gave enough accurate detail so that it was clear that what the student wanted to do in fact matched the design designation they chose. This typically meant giving an example or two of what would be coded or observed, providing a small coding grid, or describing the type of field notes that would be taken.

Described their sample or census thoroughly. (A few folks didn't mention how they would obtain schools, administrators, or individuals at all.) Described why a probability or non-probability sample would be better to take in their particular instance.

Received a score of at least 12 points out of the 14 available on Juanita's ethnography. Accurately diagnosed Juanita's research design problems. Used less structured design terminology accurately. Recognized why a non-probability "sample of one" may be adequate for design purposes. At least partly addressed the issue of "what is scientific" in part 6.

You lost credit if:

You STILL have problems with different types of samples.
You had a lot of information about your research problem but not very much about what your study was and how your would conduct it.
You weren't sure WHAT kind of design you had!
You had so little detail that it wasn't clear what kind of design you would use. For example, your design appeared to describe a structured survey or an in-depth interview which you designated as an oral history.
You thought a less structured design wasn't scientific because it didn't produce numeric variables to analyze.
You didn't mention how you would obtain the units you planned to study.

PART 2: JUANITA'S PLANNED ETHNOGRAPHY

As a professional, you will spend considerable time reading research materials. Nearly all of them will draw conclusions and many of them will recommend interventions and/or policy continuance or changes. Therefore it is essential that you are able to accurately assess the methodological soundness of the studies that you read and the kinds of conclusions that you can draw.

This becomes particularly important with less structured study designs. Because hypotheses often are generated well into the data collection phase and because many important insights come from naturalistic field observations, internal validity is typically lower for less structured designs than for designs such as experiments, quasi-experiments and surveys, which more often employ a priori hypotheses to test. This doesn't mean such studies are "unscientific" (see below).

Look for similar problem-solver questions on Exam Three.

Here is Juanita's planned ethnography:

A PROPOSED ETHNOGRAPHY

Juanita wants to study processes of decision-making at a local health clinic.
Juanita obtains a list of all health clinics in Northwest Florida. Because of her graduate school schedule, she must remain in the immediate vicinity. She examines several aspects of the clinics, including the demographics of the clinic administrators, staff, and clientele, and whatever information is available about each one. She discusses using a clinic locale with four different Northwest Florida clinics. Two immediately tell her "we are sorry but we don't want our clinic to be studied right now." She finally selects The Whole Health Clinic from the remaining two clinics because its demographic mix of clients resembles the average for the district and its size is right at the median size ftor the district. She presents a proposal to the clinic and it is accepted! Juanita is allowed (with possible student assistants) to study Whole Health for the next several months.
HERE ARE SOME OF JUANITA'S PROPOSED ACTIVITIES
Juanita examines some of the following variables:
Time spent talking between clients and staff.
Time spent talking among staff.
Time spent talking between staff and administrators.
Time spent in the staff lounge.
Who speaks first to whom.
Juanita also plans to spend some time participating in the less technical activies that staff engage in, such as taking client histories and profiles, answering the telephone, and attending staff meetings. She also will spend time waiting in the waiting room, talking with clients about their clinic experiences, and going with them to pick up medications. She will take extensive field notes. She also plans to examine a sample of client records, do in-depth interviews with clients, staff, and administrators, and ascertain how well both clients and staff follow-up on medical care.

EVERYONE MUST ANSWER THE FOLLOWING QUESTIONS BASED ON JUANITA'S LESS STRUCTURED DESIGN

1. Juanita's prospectus states: I am interested in how aspects of staff-client interaction may relate to how well clients follow staff directions pertaining to their health care.

What is the technical term for this type of statement?

Typically a statement such as this is called a working hypothesis or working design.

Because the naturalistic field is so rich, one must have some focus to begin. Another term also used in this instance is foreshadowed problem.

2. Is Juanita's sample a probability sample? Please answer YES or NO, then explain in a sentence or two:

WHY or WHY NOT?

Was the type of sample Juanita employed generally a good sample or a bad sample for the purposes of her study? Explain your reasoning briefly.

Virtually everyone recognized that Juanita had a (pick one, each has applicable aspects:) convenience, purposive, or judgement sample. These are non-probability samples. Worse yet, half the clinics (not hospitals) she contacted refused to participate. This is not unusual. Remember in an ethnography, it's not just a quick "in-out" survey. Researchers are frequently on site for months, even years. They may be "behind the scenes" (another great participant observation phrase) or "backstage". Organizations may feel uncomfortable about having research done in such "close quarters."

Although not a probability sample, Juanita has a reasonably good sample for her purposes. She is attempting to select a clinic with a mix of patients, a "so-called typical" site. In fact, were she to take a probability sample, with just ONE clinic, she might not have such a mix. For example, she might end up with a nearly all-White, all-Black, all-Asian, or all-Hispanic clinic, for example, and not have the "mix" she would like to see.

When you only have ONE unit (e.g., a clinic or a school), generalizing is not going to be an issue, whether you take a probability or a non-probability sample.Obviously, you can't establish a valid standard error on a sample of one! This "sample of one" may be the best that Juanita (or any other researcher) will be able to do.

3. Briefly describe ONE activity a researcher such as Juanita could use to measure how much the clinic staff interact with the clinic clients.

If you want to study interaction, I recommend interviews AS A LAST RESORT. You aren't getting "what really happened". You are getting someone's perception of what really happened. Even in-depth interviews won't help that one.

What can you do instead? You OBSERVE. We got some great suggestions here:

The patient logs and records, stating how much time staff spent with patients.

A variety of grids to measure--

Minutes of interaction
Sociograms of who interacts with whom
Face to face one on one interaction (say, versus telephone or group meetings)
Non-verbal behavior such as smiles, frowns, arms folded over chest, etc.

4. Here are some techniques that Juanita COULD use in her study:

Content analysis
Gleaning
Oral histories
Participant observation

Of these four methodological techniques, which ONE appears to be the MAJOR technique that Juanita proposes to use most extensively in her study? Briefly give the rationale behind your choice.

Participant observation, hands down. Juanita is interested in (a) interaction and (b) compliance with medical directions. With oral histories, she gets people's views on these issues, but not what really happened (see above). And if I were Juanita, I certainly wouldn't take people's word for it on compliance with medical directions! She will be doing some in-depth interviews, and a lot of content analysis. But her participation in the daily life of the clinic means that participant observation will be her primary mode of data collection.

Most ethnographies will combine all four techniques, but may vary the emphasis.

5. Briefly describe ONE way in which an assistant can be helpful to Juanita while she is studying in the field.

Again, we got lots of great suggestions. It goes to show that we can all use research assistance!

A big help is lots of multiple tasking--reliability checks in observations, in coding, in data entry.

Teamwork helps, since Juanita cannot be everywhere simultaneously. For example, Juanita can be back observing in the staff lounge while the assistant is out in the waiting room.

Objectivity. The assistant can help keep Juanita grounded if she begins to "go native".

"Boy-talk." This is a health clinic. There will be areas where Juanita probably can't observe (such as intimate examinations). Plus, male patients or staff may feel more comfortable with Juanita's male assistant.

Helping the research forge forward. For example, the assistant can do early coding, looking for themes in the field notes and documents. Identification of these themes may create structure as the research moves forward.

Helping with logistics. An assistant can transcribe field notes, enter material in the computer, and even see that Juanita gets a cup of coffee now and then.

6. Juanita shows her drafts to a colleague as she is writing a report on Whole Health. Much of Juanita's report uses anecdotes, some simple percentages, and a lot of description. Her colleague states the report doesn't seem scientific at all.

What is your BRIEF rejoinder to Juanita's colleague's comment?

It all goes back to "what is science?"

We come back full circle to material at the beginning of the semester.
Science is not the quantitative level or qualitative level of your variables. Those qualities will affect the type of statistical analyses that you use but that isn't really science either. And numbers don't "prove anything" (although they can disconfirm.) More structured designs have biases too: by failing to ask inclusive research questions, the study designs can lead us in the direction that the researcher wants, but this may not necessarily be the most accurate direction. Each type of design we have studied this semester has its strengths and its weaknesses.
Science wants to make cause and effect statements, and to explain relationships among observed phenomena--but so do many other epistemological systems, such as religion or astrology.
Science is a process, not an outcome. It is about how you think, how you collect evidence, and how you interpret evidence. As McMillan notes, early on, science is a systematic, objective, and logical method of inquiry. Science is rigorous in measurement. Most of all, at its best, science is a self-correcting process of inquiry. That is why it is so important to consider alternative causal factors, to measure and test for them when you can, and to rule out alternative explanations for your findings whenever possible.
As such, it is evident that Juanita has taken great care to select the best sample that she can, to be both empathetic yet objective in the methods she employs, and to accurately describe. She is not trying to establish strong cause-effect connections as an experiment might or to generalize to a population as many surveys do. Yet she is following scientific procedures. Thus (optimally) Juanita will have a solid description of the patient-staff interaction and patient adherence to health routines at the Whole Health Clinic. Her study will probably generate more testable hypotheses for future research. I definitely call what Juanita is doing "science"--and I hope that you will too!

LOOK OVER YOUR ASSIGNMENT CAREFULLY. I MADE SEVERAL COMMENTS ON EACH ASSIGNMENT.

EDF 5481 READINGS
AND ASSIGNMENTS

OVERVIEW

Happy Holidays!

Susan Carol Losh. December 3 2002
This page was built with Netscape Composer
and is best viewed with Netscape Navigator
600 X 800 display resolution.