INF5220 - Autumn 2006

Lecture October 25th: Quality

Question: Should we just trust qualitative researchers (e.g. based on their ”political credentials” and demonstration of ”involvement”)? Or is there a requirement upon the qualitative research community to generate credible and valid knowledge?

(of course these are rhetoric questions, and the answer should be “no” to the first and “yes” to the second, in case you wokdered).

 We will discuss quality criteria, concretely around validity and reliability, as well as evaluation criteria for three types of scientific writings: research proposals, thesis/dissertations, scientific papers.

 The paper by Checkland & Holwell (on Action Research) advocated the recoverability criterion. The argument was that qualitative research can not achieve (and should not aim for) ”replicability” (according to the natural science ideals). Still, one should aim higher than merely claiming to offer ”plausible” results. Their solution is that the research process should be made recoverable (rather than replicable):  you should make clear to interested observers the thought processes and models which enabled the team to make their interpretations and draw their conclusions.

 Recoverability (transparency) goes for both documentation (reliable documentation of the empirical material) and argumentation (valid, consistent, coherent line of argument in analysis and drawing of conclusions).  The two terms reliability and validity are usually connected to each of these two parts of the research process (reliable documentation and valid argumentation).

 What does the terms validity and reliability mean?

 Validity has to do with what we might call ‘truthfulness’

Reliability has to do with consistency of ’measurements’

Example: repeated readings of two termometers in boiling water (100 ºC):

 

A

96

96

96

96

96

96

96

96

B

101

100

99

98

100

100

101

99

 

A is reliable, but gives invalid results

B is unreliable, but gives relatively valid results

 

 

Reliability (of research procedures):

The aim is reliable collection and documentation of empirical material. This criterion pertains to your data collection techniques (the practices of observations, interviews, note-taking/recording, the descriptions you produce). It is crucially important to document the procedures of data collection, and to make conscious choices on how you represent your data.

 

Silverman’s advice is: Use low inference descriptors and standardised data collection procedures.

  • Use low-inference descriptors, e.g. verbatim accounts of what people said, extracts from field notes.
  • On the contrary, ”high-inference descriptors” may be your ’polished’ account of what you see in your material, your interpretations (”The girl was flirting with the boy”).
  • Provide excerpts from field notes (when appropriate)
  • Give details on the relevant context of observations and how you recorded and handled the notes.
  • Have you standardised your data collection? Describe your field-note conventions (if you used standardised templates for data collection, your research group’s practices for sharing and discussion of analytic memos, etc.)

 

In a previous book on research methods (Silverman, 1993) he advices you to use 4 sets of fieldnotes:

The optional reading for today by Ulrike Schultze gives extensive detail of fieldwork practicalities, and uses these analytically in a genre called confessional ethnography. She writes:

”I thus endeavored to record incidents as objectively as possible. This meant that I carefully described events from the perspective of a distant observer, presenting them in their rich detail and including myself as a subject in the scene. I made my impressions, reactions and interpretations as explicit as possible”…(p.16)

 

There can be said to be several levels of abstraction in any situation (in which we do fieldwork and in general). We should train ourselves to be able to distinguish between observable “data” and our processing of these “data”. The different levels are: the observed (what I register or sense of what was said or done), the interpreted (how I experience the situation, interpret or guess what is going on), my reaction (my thoughts, emotions or bodily reactions, based on my interpretation), and behaviour (the actions that follow from my interpretation).

 

Validity of conclusions:

The validity criterion pertains to the representativeness of data and the truthfulness of the interpretation of them. It concerns the quality of argumentation and conclusions.

 

Questions that try to assess the validity can be: Is the account true? (Are the deviant or contrary cases excluded? Is a ’glossy picture’ presented? ). Is it clear how the data included in the account (”story”) was selected? (Are only a few favourable cases or instances picked?)

 

Silverman discusses what he calls the common problem of ’anecdotalism’: that a few exemplary (and/or entertaining) instances are offered, without reasons for selecting them, or a discussion of the typicality or representativeness of them.

 

What does the data ”mean” or ”tell”? Last time we discussed a concrete example of data-driven analysis. Your research focus is a significant determinant for what you see in the data. Theory-driven analysis is more common in our field (informatics) than it is in Silverman’s book, which makes for a different set of problems (e.g. the issue of “labeling” – the theory’s concepts are mapped onto empirical examples from the fieldwork (the findings are labeled as A or B etc), but without going further and using the theory for something. The theory is illustrated, exemplified, but doesn’t give analytic leverage to the study).

 

Back to the book: Commonly advocated methods to secure validity are triangulation and respondent validation.  Triangulation means combination of different methods that give you different but complementary views on “reality”. Silverman asks: Can a ”true fix” on ”reality” be achieved separately from ways of looking at it? Remember that results from methods are situated and context-bound. Respondent validation means to take your accounts and interpretations back to the subjects to have them verify (possibly correct) and thus validate your data. Question: But are they really in a privileged epistemological position to give the “right” interpretation?

 

Silverman’s advice instead:

 

The reading for today (Klein and Myers, 1999) offer criteria (seven principles) for conducting and evaluating interpretive research. These are based on the perspective of hermeneutics, but to some degree generally applicable.

 

  1. The fundamental principle of the hermeneutic circle: ..”we come to know about a complex whole from preconceptions about the meanings of its parts and their interrelationship.” Understanding requires a number of iterations from part to whole etc.
  2. ”The principle of contextualization requires the subject matter be set in its social and historical context so that the intended audience can see how the current situation emerged”. Thus ”a key task becomes one of seeking meaning in context”
  3. The principle of interaction between the researcher(s) and the subjects. ”In social research, ’data’ are not just sitting there waiting to be gathered, like rocks on the seashore. Rather, interpretivism suggests that the facts are produced as part and parcel of the social interaction of the researchers with the participants.”
  4. The principle of abstraction and generalization. ”Unique ideas and instances can be related to ideas and concepts that apply to multiple situations.” Validity does not depend on representativeness of cases but on the ”plausibility and cogency of the logical reasoning used in describing the results from the cases, and in drawing conclusions from them.”
  5. The principle of dialogical reasoning. Confront your preconceptions with data that emerges (prejudice is unavoidable). Make the fundamental philosophical assumptions as transparent as possible
  6. The principle of multiple interpretations. Study the multiple viewpoints and possibly conflicting interpretations of the subjects
  7. The principle of suspicion. False preconceptions (look for ’biases’ and ’distortions’ in narratives). ..”to ’read’ the social world behind the words of the actors”

 

Evaluation criteria

See Silverman: Table 15.1, 15.2 and 15.3. Different genres have different evaluation criteria:

 

Summing up