EDF 5481 READINGS  AND ASSIGNMENTS

GUIDE 7: AN INTRODUCTION TO LESS STRUCTURED METHODS

OVERVIEW

A refresher from our overview:
 
 

I don't like the terminology "quantitative-qualitative."

It confuses the level of the variables (such as nominal or interval) with the way you conducted your study (e.g., experiment, ethnography), and that is just plain inaccurate.

Many historical studies, for example, are highly quantitative. Field studies may gather information on quantitative variables. The computer programs used for content analyses can produce highly quantitative results. Similarly, you might have a categorical independent or dependent variable in an experiment, a quasi-experiment, or a survey.

Of course, the level of the variables, whether quantitative or qualitative, also has absolutely nothing to do with causality or internal validity either. For example, gender, a nominal variable, is typically the independent variable in many studies. Causality depends on the threats to internal validity or possibly the construct validity of your variables, but not at all on whether a variable is nominal, ordinal, or interval.

Yet, we suspect that differences exist among methods called "quantitative" and those called "qualitative." What is the key to some of those differences?

I believe that key is STRUCTURE. From the very beginning, the methods we call "quantitative" are much more structured: how they are conceptualized; how scholars measure the variables concerned, how the data are gathered; how the data are analyzed, and how reports are written.

Less structured ("qualitative") research is much more fluid. The timetable for these kinds of studies is more flexible. The scholar may alternate among data collection, data analysis, and theorizing, or start in one area (religious congregation directives on religion, science, and politics) and end up somewhere else entirely different (the role of group cohesion).

A huge reason for this fluidity is that once you enter "the field," whether this is an organization or a set of records, you will almost certainly encounter many unanticipated events, patterns or sequences. To capture knowledge about these, you end up revising your research design.

To researchers accustomed to more structured methods, such as experiments or surveys, less structured methods seem sloppy, over-intuitive, and even downright mystical. When does data collection stop? Many unstructured researchers give answers more structured researchers consider vague: when the data repeat themselves; when no new information is uncovered; when no more new insights occur; or, most aggrevating of all, "you just know."

If your goal is to draw tight cause-effect connections, clearly this is not the best way to proceed because of the typical lack of standardization and the less precisely defined variables. However, you may have different goals in mind:

You want to describe a culture or a subculture, particularly from the point of view of its population.

You want to describe observed relationships among variables but not, at this point in time, to make definitive causal statements about these relationships.

You want to generate rather than to test hypotheses.

You don't exactly understand all the parameters of the situation under study and you want to understand more about them.

You want to conduct your study in a natural setting, rather than in a laboratory.

You want the study participants to define their lives for you.

You want to examine SOCIAL INTERACTION among people who know each other. 

Back in Chapter One (p. 13), William Wiersma had a small table comparing more and less structured research designs. I want to amplify on that section of text below:
 

Less structured designs often become more structured as the study continues and the research team learns more about the situation. Descriptions of situations may give way to more structured field observation codes, in which observers are able to check off behaviors and themes that regularly occur.

Eventually, the researcher may be able to specify when an event occurs or when it does not. This increase in precision means that occurrences could be coded as dichotomous 0-1 or "dummy" variables, "yes" or "1" for an occurrence and "no" or 0 for its absence. If understanding proceeds to this stage, the analyst can then create countable or ratio variables for the number of instances of an event. The analyses then begin to resemble those from more structured methods, such as analysis of variance.

Thus, the analysis for a less structured study can use any applicable statistic. For analyses such as analysis of variance, t-tests, or regression, you need an interval-ratio dependent variable and you need to be able to specify cause and effect. If your data allow you to meet these (and the required other statistical assumptions), you can use these analytic methods. They are not particular to any particular method of collecting data or conducting studies.

Statistical analyses, of course, can be hampered if only one group or organization is studied, or a number of case histories are selected in a haphazard or "grab" manner. Probability sampling is more the exception than the rule in less structured designs. This is partly because most occur in a naturalistic field and access is often a problem.

The analyses for ethnographies or case histories often resemble those done for focus groups. A few descriptive statistics such as means or percentages may be provided. A lot of the data presentation is anectdotal. Much of it describes situations in the study setting. Causality is suggested, not assured. The end point of the analyses and discussion may be to suggest variables and hypotheses for new studies.
 

Historical research often deals with events of the past or threads of the present in natural settings.

Before I married an historian, I must confess to considerable naïveté when it came to historical data. I imagined history as a study of facts--which were known to historians, to be altered only if new facts came to light. I WAS WAY WRONG! First, most historians are sticklers for examing the original documents and want to draw their own conclusions from them.

Second, interpretation and reinterpretation constantly occurs within history. The same facts are often interpreted in many different ways. Were the immigrant schools of New York City about 100 years ago a way to "Americanize" immigrants or did they serve as propaganda to support existing elite structures in the Eastern United State (or both)? What were the goals of immigrant schools? How were these goals achieved? Historians pore over documents, examining them first one way, then another. It is not so much what facts are, but what facts mean that is important.

Historians examine many types of written records and record searches: diaries;, schedules; newspapers and magazines; census data; television programs; letters; and cultural artifacts of various kinds (art work; music and song).

Historians often interview individuals who may be connected with the events under study, such as European Holocaust survivors, children and grandchildren of the Rosewood, Florida attacks, or early members of a group or organization. These use in-depth (qualitative) interviews. Questions are open-ended. Comparable to focus groups, the researcher starts with several unstructured questions. Which questions are asked, and their order, depend on the information that is uncovered  during the interviews. This information may lead to the formulation of new research questions.

Ethnographies describe cultures and/or subcultures. Ethnographers often collect records and field notes, do oral histories, and utilize field and participant observation to collect data. Typically one group or organization (a case history) is studied in considerable detail. The emphasis is  holistic and contextual. Ethnographies occur in natural settings.

Frequently ethnographies utilize participant observation, or a methodology in which individuals act as data-collecting researchers while participating in the daily events of the group or organization. Participant observers detail the routine rounds of daily life, they conduct in-depth interviews, describe leaders, and summarize group activities.

Compared with other types of research, a major focus of many ethnographies is interaction among individuals or group interaction. Life in groups often receives scanty notice in experimental work, which often uses "groups" comprised entirely of strangers. Surveys, of course, utilize individuals as the source of information, even if individuals report about a group, such as a household. Individual reports about groups, of course, only reflect that respondent's perspective. Ethnographies, with their focus on "who said what or did what to whom" provide a refreshing contrast. On the other hand, it is very difficult to observe groups because typically considerable amounts of action simultaneously occur.

Old magazines are often stored in library archives. Old reruns of situation comedies apparently "never die." The researcher who wants to trace idealized fictitious images of the American family can start with "I Love Lucy" television films from the 1950s and sample from every decade through to "Everyone Loves Raymond" in 2002. The minutes of a corporation may stretch back hundreds of years.

Content analysis involves studying content in media or records, typically using a framework that the researcher provides. The data are pre-existing to the particular study and were gathered for original purposes unrelated to the purposes of the study. Thus content analysis is one form of secondary analysis. Content analyses use naturalistic data, very often generated by a group or formal organization.

The researcher provides the themes for the content analyses, which are often unrelated to the orginal purpose that generated the data (for example, early television producers were unaware of the gender composition or gender stratification in their situation comedies). The researcher may provide a coding grid or transfer data to one of several computer programs (see below) that will test for themes, do word counts, theme counts, and other analyses.

Gleaning

In the course of our daily lives, we leave a lot of debris behind us. Consider the contents of our garbage cans. The Project du Garbage at the University of Arizona certainly has! Wrappings from the foods that we eat as well as leftovers, the cosmetics and colognes that we apply, discarded magazines and pamphlets, tossed out solicitations for credit cards, charitable contributions, mail-order catalogues, newspapers. Take a good look at the contents of your can before you toss things into the trash and consider what kind of information your trash gives about you.

Because garbage can be toxic and infectious, the Arizona researchers "suit up" in biohazard suits. Do NOT try this at home! But you can look at the "gleanings" from everyday life without going through the trash. Gleaning can be particularly helpful in telling us about culture. Do teenagers claim they are interested in "higher culture," new art or music, while records of the books, magazines, cassette or video tapes, and CDs checked out, rented, or sold tell a different story? Examine best-seller book and music lists (which typically are now computer generated from bar codes and much more accurate than the subjective impressions of store owners and managers used in the past). Frequent second-hard shops to see which articles are discarded.

There are more subtle indicators of the gleanings of everyday life. Consider the fingerprints and noseprints left on glass display windows as clues to the popularity of exhibits and the age of visitors. Tiles that are worn on a floor or paths worn in the grass indicate popular routes.

We will have an entire course section on analyzing data that originally were collected for other purposes. In particular, we will examine a variety of online databases and archives. Content analysis is one type of secondary analysis. So are online analyses of data archives.

The "funnel approach"

The funnel approach (so-called because the research endeavor begins in a very broad way, then narrows as more information is accumulated) is common in less structured designs. Typically, the researcher begins with very general questions ("what are the major themes in religious school classes?"). These serve to begin delineating the type of sites to study (a variety of Judeo-Christian religious denominations, to be expanded to Islamic denominations at a second stage of research) and the initial type of data collection (ethnographies and participant observation). As research progresses, the research questions are refined, the sites are narrowed, and the type of data to be gathered becomes more precise (content analyses of religious school lessons; congregant participation in religious school lessons.)

Holistic

Rather than defining and measuring individual variables, holistic research attempts to examine the entire situation. Thus, it may describe a situation in considerable detail: who the actors are; how they interact; the "scenery" of the setting without delineating separate variables and the relationships among them. Holistic research is often contextual, that is, it insists that data must be examined in the context in which it exists or occurs. For example, lessons in religious school would be interpreted in the context of the religious congregation, its general denomination, its epistomology, its sermons and religious literature.

Participant Observation terms

Participant observation tends to have one of the more colorful terminologies in methods. For example, observers who do not reveal their research goals to participants are said to be "under cover." Researchers who empathize so closely with the individuals and situations that they study are said to "go native," borrowing a term from anthropology. Group members who have a mutual trust with the researcher and who often provide in-depth information are called "key informants."

Participant observation also presents ethical issues that are less prominent in some other types of research. The researcher may fear that revealing his or her true purpose in the situation will be obtrusive and cause reactivity. Participants may become self-conscious or alter their normal modes of behavior if they are aware their behavior is under scrutiny. Some researchers study behavior that is illegal (e.g., drug use) or of dubious morality. Signed "consent statements" might implicate participants as criminals or hurt them professionally. The researcher must balance their ethics with data collection purposes.

Phenomenological

There are several aspects to the phenomenological approach in research. Phenomenological research tends to be holistic. It also tends to be interpretive, i.e., not just concerned with facts and findings, but the meaning of these facts and findings in the larger context of the individual life, group or organization. This kind of research is often empathetic, taking the view of the participants in the situation.

Unobtrusive Measurement

Many researchers are concerned that their presence in the field will be obtrusive, or noticable. This is particularly problematic with research that occurs in the natural field, such as case histories or ethnographies. Obtrusive measurement may lead to reactivity, particularly social desirability. People may want to appear more organized, more competent, or more tolerant. You will recall that this is one major problem with videotaping. People become self-conscious about their appearance or what they say in a manner that does not seem to happen with audiotaping.

Of course, some research, such as content analysis, may contain elements of social desirability when the data are laid down in the first place (e.g., "no one commits suicide in death certificates," individuals resign instead of being fired.)

There are two main ways we address obtrusive measures in less structured work. One way is how researchers behave. An attempt is made to be low key and non-judgmental in oral histories or in-depth interviews. Fear of reactivity is a major reason why participant observers go "under cover" and race to lavatories to scribble field notes. The second is in the measures that are used. "Gleanings" for example, which use the debris of everyday life, do not intrude on the lives of those whom we study. We may be able to study organizational records without bothering study participants.

Working or emergent design

This is your preliminary plan for data collection. These methods are LESS structured, but that does not mean UNstructured or no structure. When you enter the field for the first time, you are likely to be overwhelmed, especially because you typically have no control over who enters or leaves the scene, or the events that occur within it. It is literally impossible for you to see everything (all at once OR in sequence), or to record everything.

Thus, you need some focal points to get started. Your design may designate key individuals or activities to study. It may include a sequence of the areas of the setting that will be studied.

Over time, you probably will gain a better idea of variables and sequences of behavior. You may be able to formulate hypotheses and define conceptual and operational variables. At that point the study design may change. For example, you may be able to create grid sheets of behaviors or occurrrences to check off as you observe them.

Working hypotheses

Working hypotheses are much less precise than the relatively structured conceptual, operational, or null hypotheses that you constructed earlier this semester. You may not know exactly which variables you will study. Indeed, you may not even be interested in variables per se at this stage. Working hypotheses are more likely to take the form of questions about phenomena rather than relationships among well-defined variables.
 
 

External validity is typically less an issue in less structured research for several reasons:

• There may be no probability sampling--or even a systematic attempt to produce a generalizable sample. Cases often depend heavily on ACCESS (who lets you in or who lets you examine records) and AVAILABILITY (old records may be gone; some behaviors may only be observed by insiders, e.g., a church executive council meeting).

• The case base is often limited. In field studies (ethnographies), there may only be ONE organization (although several people may be studied over a long time with many instances of coded behaviors).

• The argument for generalizing is often made through logic, rather than through sampling error. In the manner of purposive sampling, the researcher argues their locale is typical of a certain type of setting (religious congregations in mid-size Southeastern--and possibly Midwestern--communities.)

• Multiple observers. The researcher may assess consistency or agreement across INDEPENDENT observers.

• Multiple measures. Sometimes called TRIANGULATION, the researcher tries to measure a concept in as many ways as possible (interpersonal cohesion? OBSERVER "feeling thermometer" on friendliness; the observation of an informal mentor; how long it takes 50 percent of the congregation to leave after the service; CODER estimates of verbal interaction based on the entire congregation folder). Triangulation does two things at once: it assesses whether a phenomenon has occurred and often the degree of the phenomenon so it addresses reliability. It is also assessing the CONVERGENT VALIDITY of a construct, i.e., whether different measures appear to assess the same variable.


Triangulation means that you used at least two different kinds of methods to measure a phenomenon. It does NOT mean you used two or more instances of the same kind of measure, such as two self-report surveys.

Subjectivity is a constant problem in less structured research designs.

We can think of statistical significance testing, random assignment in experimental treatments, replication, or probability sampling as safeguards against our own biases and what we want to believe. All of us start a research project with hunches and our own beliefs about how we think cause and effect operate in the situations that we study.

The problem is that our subjective beliefs can easily influence which phenomena we choose to observe, the categories that we construct, and the interpretation that we place on our observations or collected records. In participant observation, we may become so empathetic with those whom we study that we lose our own objectivity.

It is a delicate balance that we need to try to preserve between empathy and interpretation and objectivity. I think this balance is what can make less structured research so difficult. It helps to have multiple observers. It helps to have regular meetings among all those involved with the research project. These meetings should follow a semi-structured agenda that allows input from all participants on the projects, sharing ideas and interpretations and receiving feedback from other participants.

By now, you are getting the idea that less structured designs are far from easy. You are absolutely right! The researcher must be far more careful to try to specify concepts as clearly as possible (none of those "my concept is whatever this standardized test says" statements will work here). Because creating and testing hypotheses on the same set of data can produce tautologies, the researcher must bend over backwards to provide rigorous and varied evidence for his or her assertions.

Recall that because so much of less structured designs is interpretive and holistic, objectivity can be a definite problem. No matter what one's empathy about the people who are studied, the researcher must remember that she or he IS a researcher.

On the other hand, when it is possible, try to imitate the clarity and objectivity  of more structured designs. If you can draw a probability sample of records or organizations, do so! If you are able to create a patterned grid to measure at least some of the behaviors that you observe and count their occurrence early in the research process, do it! If either prior theory or experience in the field allows you to create more precise hypotheses, create them!

For example, many computer programs are now available to assist you with coding data from content searches or ethnographic studies. Examples include:

NUD*IST

AskSam, and

THE ETHNOGRAPH.

PLEASE SEND ME AN EMAIL TO LET ME KNOW ABOUT ANY OTHER SITES OF THIS TYPE THAT YOU RUN ACROSS!
Thanks.
 

EDF 5481 READINGS  AND ASSIGNMENTS

OVERVIEW

Happy Happy

This page was built with Netscape Composer.
It is best displayed in Netscape Navigator,
600 X 800 display resolution.
Susan Carol Losh
Updated October 7 2006