Qualitative Reliability, Validity, and Generalizability

As I am continuing to develop my idea that I published yesterday, Research Design: Communities of Practice for Autoethnographers, I want to clarify one of the issues that some of my wonderful colleagues commented about, namely the issue of Reliability, Validity, and Generalizability.

While I used those terms in that area based on Creswell’s Research Design: Qualitative, Quantitative, and Mixed Methods Approaches work (pg. 190), I put it there primarily as a place-holder so those in my program who will offer me some comments on it will know that I will, in some way, address it. What I actually had in mind was Lincoln and Guba’s Naturalistic Inquiry, where they give 4 areas of concern for qualitative Trustworthiness:

  1. “Truth value”
  2. Applicability
  3. Consistency
  4. Neutrality

These four were offered to address the quantitative internal validity, external validity, reliability, and objectivity.

I will address these as I engage with my research, and share aspects of them here.

My Autoethnographic Writing in Yvonna Lincoln’s Session

(This is the result of an assignment Yvonna Lincoln gave us during her pre-conference session on experimental writing. It is not finished, and is intended for us to explore this method. I will develop this into the “What did I learn in this? section for my paper.)

I have 3 days. That is it. No more without discussing it and gaining  agreement with my colleagues and faculty tutor. How am I going to finish my first research project in my doctoral program? 3 months into the program, and already original (if not perfect) research.

Yes, I will finish it. I always finish it. Working full-time, teaching 3 university classes, studying toward a PhD, and preparing for 3 conference presentations within a two week period of time? I can do that. Of course I can do that! I have always finished before . . .

I interviewed 3 people about their intentions and what they learn and what they hope their learners will learn from their autoethnographic work. Yes, and how technology figures into the equation, as I am in an E-research and Technology Enhanced Learning doctoral program.

We had 4 weeks for the project, and on top of everything else, I have not only gotten my research design written, approved by the ethics committee, requested interviewees (I got 3 though I only tried for 2), adjusted my transcripts after my interviewees did not like my notes of our discussion, and then open and axially coded my data – I also analyzed the data and wrote an initial overview of my findings.

I realize the more time would not have helped much; what I need it help with is writing a more concise and comprehensive consent form, resources for transcription, feedback with being more rigorous with the strategies of inquiry and to what extent coding should be done (Strauss, Corbin, Berg, Wolcott) and using whose framework (Stake, Yin) and how mixing frameworks can be useful), and assistance with selecting and then using qualitative analysis software (Nvivo, Atlas ti, MaxQDA).

Is my research perfect? Of course not. Complete? No.  Ready for conference or publication submission? Not quite yet.

But what value was the project? Immeasurable.

I never would have supposed I could have learned so much about research apart from doing it (this is my first time doing it to  this extent individually). I realized not only I can do it, but that I love it. I learned that my initial framework for autoethnography was somewhat limited and is now larger, as I learned that other people think about it in other ways. I learned that it is really important to love the topic of the research, as that will be the only tings to sustain me in times of overwhelming despair. I can even see this research developing into my first step toward a dissertation (called a thesis in the UK).

I learned a lot, learned there is a lot more I do not know, and that research really can influence and support practice.

Experimental Writing with Yvonna S. Lincoln

I am attending a pre-conference with Yvonna Lincoln at the QI2009 conference today. As we are nearly ready to begin, she just distributed a workbook (that she was clear to tell us we will not work in), but that contains examples of the different materials and media that she will discuss.

“Who brought data in?” Of course, none of us did not since that was omitted with in the description for this session. That is ok, she said, as it is now time for autoethnography!

She enjoys experimental writing, as opposed to that difficult, academic-style writing.  This session is all about experimental writing as the product of research, not as creative fiction done outside a research perspective.

Good gods, she just turned on an overhead. Not a projector, but an overhead. No, this is not necessarily the way that she always works; we were told at this conference that there are no projectors available, but only overhead machines. Glad I printed my slides on overheads!

Why Write Experimentally?

  1. To capture the emotional and psychological impact and residue of fieldwork (Brady, 2000, Handbook of Qualitative Research 3rd Ed.). I really like this concept, the “residue of fieldwork.” Never heard of that before, and my initial thinking is that this will be
  2. To create a “re-thinking of presence.” this is all about admitting that the researcher  is there and we were affected by the research and experience. This inserts a sense of reality, and not a sterile writing and sense that everything is gods-eye view of reality
  3. Writing experimentally permits a criticism of the “rhetorics of scientific authority, [the] techniques of vision, and [the] discursive mechanisms of relating time and space” – Clough, 2000.

Somebody just knocked a cup of coffee (under a seat) over and onto somebody’s Apple laptop. Ouch.

Criteria for Judging Experimental Writing (there are no firm criteria)

  1. It should “motivate cultural criticism. . . .”
  2. Experimental writing should “be open to the future;” it should help readers to pose alternative—and perhaps more just—scenarios for tomorrow – this hints at some present and talks to us about a presence that is less than satisfactoy and points to a more just future
  3. It should prompt “theoretical reflection,” never straying far from theory. . .
  4. If experimental writing is literary in form (e.g., poetry, plays, short stories) it must meet criteria for good literature in that specific genre (M. Allen, n.d.). Mitch suggested that if you want to turn fieldwork into a play, then we need to think about what makes for a good and strong play. This makes me think about Michael Hemmingson’s writing.

This is the first time I am liveblogging a conference this year, and am looking forward to the paper I am presenting tomorrow, where I discuss liveblogging from the perspective of autoethnography.

I think that my liveblogging work is changing. I used to capture what the person said and then include my own thoughts on the topic. However, with my increasing work with autoethnography and how I believe liveblogging cannot be a neutral act as I as author and participant am writing this material from my own perspective—I am the one who chooses what to include from the presentation.

Thinking about this more, I think liveblogging is more accurately about sharing elements of the author’s perspective of an experience, insofar as the author is able to type some of what thoughts and experiences can be captured. From this perspective, the liveblogging experience is my work, and my words. This is where the co-creation from the experience comes from; I am discussing my experience and thoughts from this session, and cannot claim it may or may not represent the experiences of anybody else in the session.

Yvonna just talked about experimental writing and how, when it is done well, then that does not make this easy, as good writers are often still very deep and complicated.

The first kind of experimental text she showed was from Carol Rambo Ronai, Betty St. Pierre, and Patti Lather. There are multiple things moving; things come together and then come apart, and then perhaps come together again.

Yvonna is not very loud. Somebody requested she speak into the microphone (too lecture-like, but with the air conditioning unit and the loud presenter next door, this may be a good idea). She does speak there, but only time to time. As her voice does not seem to project very much, I wonder what the environment where she commonly presents is like?

With experimental writing, the concept of layered texts shows how complicated texts and experiences are. I like the section heading for the booklet she passed out: :Skirted, Pleated and Layered Texts.”

Poetry is a form of experimental text. She favors ethnographic poetry, such as by Laurel Richardson and Hartnett. This is significant because it clearly states that there is never a detached researcher.

I really like that—there never is a detached researcher. That is what John Creswell talks about as being one of the 3 components of a research design, namely by explaining and discussing what worldview the researcher has when writing.

She is discussing Laurel Richardson’s need to write an ethnographic poem, Louisa May, based on a transcription that just did not seem to easily be written in any other way.

I am glad we have the hand-out booklet that includes all the examples she is discussing.

About poetry, she said

Captures the emotional residue of fieldwork; makes it plain that fieldwork is not all “science,” but rather involves the entire person of the fieldworker.


The “voyeur’s gaze” is turned back onto the researcher herself of himself. The “subject” becomes the subject of the research. Done in order to theorize more deeply on some social situation, or the effect of the researcher’s presence in that situation.

Somebody in the audience just shared the story when a poem was rejected, and the feedback was that the poem beautified a tragic story. It seems to be that the poem must have been wonderful. Yvonna suggested these journals as ones that publish poetry — QSE, Qualitative Inquiry, Humanistic Anthropology, and IJQI.

Choose different articles and review them from different paradigms as a way of better understanding these different perspectives.

It seems useful to give students the opportunity to explore various ways of experience. Some students are instrumentalists and some are interested in learning anything they can. There was a discussion about how to deal with experimental coursework. This makes me think about WIFM? – What’s In It For Me? Help students meet their needs, though consider Mezirow’s Transformative Learning as a way to help challenge learner’s worldviews to invite them to think more creatively and expand their horizons. Seems the issue is about helping students to learn in ways that may engage them in ways that the learners did not previously consider.

Autoethnography is a theoretical way of the researcher researching one’s own experiences. Carolyn Ellis, Stacy Holamn Jones, Brian Alexander are all examples of this.

Really glad my netbook computer has a good, long battery.

Ethnographic Fiction (why fictionalize science?)

  1. To disguise identities—names are changed to protect the innocent . . . if there are any
  2. To create more seamless story when pieces of narrative are missing
  3. To provide the mood, context, tempo, climax, and other literary elements of fiction

Performance Ethnography

  1. The giving of voice to interviewees
  2. the re-creation of tough, hard, profoundly moving life experience so some group
  3. The demonstration of the moral dimension of some issue or context
  4. the call to arms for a moral, more just world
  5. as a way of understanding “the substance of what we [are] saying”
  6. as a means of demonstrating that culture does not pre-exist human beings. culture is created and performed every day
  7. giving a sense of immediacy

Poetry, fiction, and drama – it has to be good science and good literature

Autoethnography, autobiography, memory work, visual/photographic experimentation as less bounded by these criteria and rules. They have to be readable, engaging, and theoretically grounded. I really like these criteria.

We now have an assignment:

Write and be prepared to read what we write to the class here. What is an experience we have had that is powerful or painful. Try writing a short piece (in a little less than an hour), write about something to explore the emotional residue of some research project on which you have been working. I am going to do this as another post.

Just as I was preparing to work on this, I saw my friend and colleague Johnna Parker, who I have not seen in years and who told me about getting to UIUC from flying into Indianapolis, and not into Chicago. So nice to catch up . . .

First Thoughts from the 5th International Congress of Qualitative Inquiry (QI2009)

QIlogo2009-286-1 Here I am at the University of Illinois at Urbana-Champaign, all checked into the hotel and unpacked, and ready to attend this evening’s Pre-Congress Reception, from 7:00-9:00 in the Levis Faculty Center. I just heard about this reception yesterday, and think it is a nice (though not widely publicized) way to begin this, my first International Congress of Qualitative Inquiry.

Then, off to try to get my first full night’s sleep all week so I can be prepared for the two Pre-Conference Workshops I registered to attend tomorrow:

  1. New Experimental Writing Forms with Yvonna Lincoln
  2. Writing Autoethnography and Narrative in Qualitative Research with Art Bochner and Carolyn Ellis

More about the final preparation I am doing for my own two paper sessions (that I will present on Friday) tomorrow!