Research Evaluation Tools for Articles in the Social Sciences (DRAFT)

UPDATED on September 9, 2013:

These tools were again revised, with the current version on my Research Tools page.
Feedback is always appreciated.

UPDATED on August 21, 2013:

Based on some very helpful feedback, I revised these two tools:

As before, feedback will be most appreciated.


I am in the process of creating 2 tools to help my graduate research students assess and evaluate research studies, and am interested in getting some feedback on them. They are:

While I have seen various tools for specific purposes, I have not seen many that were intended for general use in the social sciences. Furthermore, while these cannot be applied to every qualitative or quantitative study in the social sciences, they are intended to be applicable to most of them.

Do these work? Are they helpful? Is there anything major missing or that should be combined, edited, or refined? Any feedback at all will be most appreciated.

Once I finalize these, I will make them freely available under a Creative Commons license.


I Completed My Analysis

I am happy to say I have just completed the analysis of my data for my doctoral thesis!

Let me clarify what I mean. By analysis, I mean making sense of the 23 interviews I completed by coding them, grouping similar concepts together, and then putting these concepts in a coherent order to present for my readers. That may not sound like a lot, but with hundreds of pages of interview transcripts and over 1000 codes to navigate and organize, it is a significant  accomplishment.

While I have written up my analysis along the way (cf. Richardson’s work on writing as a method of inquiry), I hope to have my full draft analysis completed in another week or so. As I am engaging in narrative inquiry, this will be, in all likelihood, my longest thesis chapter.

I just sketched a tentative timeline of thesis work for the next week, so will keep my fingers crossed to maintain its trajectory (which I will do via Twitter).


Spotlight: Journal Publishing

This is an interdisciplinary panel session on journal publishing, with Dorothy Becvar, Ron Chenail, Roy Ruckdeschel, Ian Shaw, Harry Torrance, and Donna Mertens.

While I have published a few articles with co-authors, I have not yet published a work with myself as the sole author. I think this will have to be a goal for myself for submitting 2 articles I have been working on for publication consideration. Wonder why I have had a block for doing the formal submission

Dorothy is speaking about the editor as mentor, and in this capacity she is speaking about her epistemological (and ontological) perspective. She has a responsibility to the journal publisher, and she also see a responsibility to mentoring the authors.

Ron is speaking about how his open access journal just celebrated its 20th year anniversary. Ron claims that he wants to appeal to everybody’s internal reviewer. He is now also proposing the concept of an association of qualitative editors. There are other examples of groups of ediitors in other fields.

Ian is speaking about a journal that is aspiring to be an applied, as well as an international journal. Being an applied qualitative research journal can be a challenge, as well as those that are international journals that seek to reach to one form of audience or another. Issues in and around applied work are at times distinguished by more “pure” research.

Harry is acknowledging how educational research is more eclectic and thus accepted in UK-based journals. The British Educational Research Journal is the premiers British educational research journal, and is committed to being wide, broad, and general. Likewise, it also has an international readership, and they do publish works from outside the UK. This journal, even though it publises a wide-range of work, gets relatively little qualitative research. Look at the journal, read what it publishes, and then select a journal to submit to based on how your work crosses those boundaries and fits into what goes into it. He then gave a list of British journals that are friendly to qualitative work:

  • British Educational Research Journal
  • Brtish Journal of Educational Studies
  • British Journal of Sociology of Education
  • Cambridge JOurnal of Education
  • Gender and Education
  • Journal of Educational Policy
  • Journal of Education for Teaching
  • Journal of Philosophy of Education
  • International Journal of Research and Method in Education
  • Oxford Review of Education
  • Pedagogy, Culture, and Society
  • Race, Ethnicity, and Education
  • Teaching and Teacher Education
  • Qualitative Research

Donna is now speaking about how her journal in Mixed Methods. They are an international journal, and publish articles that advance aspects of mixed methods. She then discussed the Pragmatic paradigm and the Dialectical approach and then the Transformative paradigm (Mertnes, Harris, Holmes, and Brandt, 2007) as various sorts of frameworks for the sorts of articles the journal accepts.

Roy is now speaking about the importance of speaking about citing other articles from that journal. Additionally, look at the editors and the board people, to know what they write about and who they write for. Most of the times articles are submmited, the response is usually to revise and then address each issue that is identified and then resubmit.

Ron just did a plug for an online course he offers, which is Appraising Qualitative Research. This course teaches people how to assess (and hopefully review) qualitative research articles.

Wonderful time for discussion and Q&A. Glad to have had the opportunity to speak to Ron and Harry, both of whom I had been meaning to meet and speak to for some time.


Grounded Theory Methodologies for Social Justice Projects

In the second of the two pre-conference sessions at ICQI, this one by Kathy Charmaz, I am looking forward to understanding grounded theory by one of the known experts in this methodology. She studied with both Strauss and Glaser.

Nice to have intros all the way around, with who people are and what they do. Always seems like such a basic thing, but I find it as a way of forcing the articulation of self-identity. Living with my developing understanding of identity, it is always interesting to consider these sorts of experiences. People here in the session all seem so interesting, with fascinating areas of current and doctoral studies. Lots of people here from qualitative nursing studies and social work.

Kathy is speaking about grounded theory and its relationship to social justice. She sees this broadly, from inequalities to disabilities to the justice system to eradication of oppression. Grounded theory has flexible guidelines, so it fits nicely in these areas. While grounded theory comes out of sociology, it can be applied in different areas and the strategies used by anybody. Grounded theory was influenced by a reaction to grand theories — grounded theory seeks to theorize about more focused  situations. Grounded theory is also an iterative process to go back and forth between the data and the analysis.

Grounded theory is a systematic approach to inquiry. It is:

  • inductive (you do not have the framework in mind before you begin)
  • comparative (compare bits of data with other data)
  • interactive (constantly interact with your data) — a lot of grounded theory is focused around documents

If you do interviews, then transcribe your own interviews. This helps you to understand your interview styles and then better understanding your data analysis.

It is common that people claim they use grounded theory though they really are not. This may be because they claim it to link this to the literature, as well as  because many people commonly do not understand it. It will  be helpful to clearly state whose work you are following if you use grounded theory (e.g., Glaser, Strauss and Corbin, Charmaz, etc.). Glaser adopted the language of quantitative research and brought it to his qualitative work. The theoretical sampling is not to try to get the equal sampling that is usually representative of the large group — this is done after the analysis is started, so you categorize your work and then check it out.

If you do multiple interviews of the same people, mention the number of interviews and then the number of people who were interviewed in total.

Try not to read all the theories that are out there before engaging in the grounded theory, as those theories may influence what you find, rather than as research in and for itself.

Social justice inquiry:

  1. takes an explicit value stance
  2. analyzes power in multiple forms
  3. attends to fairness, equity, equality, democratic process, status, hierarchy, and individual rights and obligations
  4. requires looking at both situated realities and ideals
  5. addresses contested meanings of “shoulds” and “oughts”
  6. prompts reassessment of our roles as national and world citizens
  7. explores tentions between complicity and consciousness, choice and constraint, indifference and compassion, inclusion and exclusion, poverty and privilege, and barriers and opportunities
  8. aims to create good societies and a better world

Kathy spoke extensively around research involved with social justice topics, as well as how to do effective social justice studies.

Ahh, now the practicalities of grounded theory strategies. Kathy believes there are only 2 levels of coding that are needed. Initial coding is the first level. She (following Glaser) uses gerunds (-ing end of verbs that act as nouns) which then helps to see things across data (based on the work of Glaser). Kathy also recommends line-by-line coding, as it helps you see things that you would otherwise miss. Your area that you are trying to investigate would drive the sort of coding you use.

Then, for the guidelines for initial coding, ask:

  • what is this data a study of?
  • what does this data suggest? Pronmounce?
  • From whose point of view?
  • What theoretical category does this datum indicate?
  • What might be lefft unstated?

I asked about the fourth bullet, as I wanted  to clarify that the theoretical category (such as identity) was not equated with theoretical framework (such as transformative learning). Those are different concepts, as grounded theory develops one’s own theory or framework about a phenomenon. Very useful distinction.

Kathy then suggested some ways of coding:

  • remain open
  • stay close to the data
  • keep your codes simple and precise
  • construct short codes
  • preserve actions
  • compare data with data
  • move quickly through the data

We just did an example of first level, gerund-based line-by-line or within-paragraph coding. Using these gerunds in this way for coding certainly helps to demonstrate what is within the situation.

Then, after that initial level of coding, time for Selective of Focused Coding (which is to select the most important codes that were already identified) that leads to the categories that begin to appear.

Then, clustering is the next step. This is not a formal part of traditional grounded theory work, though has been developed by Kathy as she used to teach writing and she found this visual organization of larger categories (hubs) with the focused codes coming out from the center (like spokes). This cluster diagram, like mindmapping or brainstorming, is useful to determine if we are really getting the main points of the analysis.

Next, comes Memo-writing when you write memos based on things that are happening, what you notice, what you wish you could have noticed earlier, etc. You have material in narrative form, and is the step between coding and writing the first draft of the paper. This involved defining the categories from the data that were collected, and not from the literature (or anything  outside of the experience). This is the key aspect, it seems to me, about grounded theory–take the categories only from the data. There is some very valuable discussion about the necessity of memo-writing. This seems like a wonderful example of a theshold concept as per Jan Meyer and Ray Land.

Finally comes the Theoretical Sampling, by which is meant “seeking pertinent data to develop your emerging theory.” This is when the researcher elaborates and refines the categories that constitute the theory.

While I feel there is a lot more to grounded theory than I learned, I do recognize that I know enough to feel confident to try using grounded theory (as I have Kathy’s book Constructing Grounded Theory: A Practical Guide through Qualitative Analysis, though never quite understood it as now I think I do now).

Another fantastic pre-conference workshop that was, again, worth the price of admission!!


Focus Groups: Inquiry, Pedagogy, and Praxis

I am attending a pre-conference workshop at the International Congress of Qualitative Inquiry (ICQI10) on Focus Groups presented by Greg Dimitriadis and George Kamberelis. Nice that we are going around the room and introducing ourselves and mentioning what our experiences are. I mentioned that I am doing some organizational focus group work within an organization, and I want to better target my questions and how to engage my population, as well as how to better write up the results.

Goodness, the slides are being presented in Normal View, and not in full-screen presentation mode. Eeeeccckkkk! Somebody just asked about showing this in slide show format, and alas there are some difficulties making this full-screen. Perhaps it is just a borrowed Mac and projector issue (benefit of the doubt?).

The nature of focus groups:

  • collective conversations or group interviews
  • small or large
  • can be “managed” or “self-organizing”

The emphasis of the presenters is to focus about strategic uses of focus groups, namely depending on if the reviewer has an agenda and uses strict questionning to follow that agenda, or if it is more open-ended to allow for the population to go where it develops.

As part of the introduction, there is a summary of who has used focus groups, and Freire was mentioned (as a Marxist revolutionary). Hmm, not quite how I would describe him.

There is also some talk about focus groups that allow researchers to “explore constitutive power of discourse in people’s lives.” Interesting perspective. Will have to think about this more.

Interesting discussion about similarities and differences between focus groups and interviews. They both have value for different purposes. Reminds me of issues of incommensurability.

The presenters are talking about focus groups being increasingly scrutinized by institutional (ethical) review boards. This may be partly since people in review boards do not understand what focus groups are all about (e.g., the perspective that this may be a privacy issue since there is a perspective that focus groups are used to save time by not intervieiwing people in individual interviews). Some of the reasons for this seem to include:

  • fears of violation of privacy (cannot guarantee privacy that participants will leave and externally discuss what others have said)
  • fear of generating dangerous knowledge (this can lead to the possibility of violence over what internal members have said)
  • fear of collective action

This research at times has to be defended because:

  • data affords insights not afforded other data
  • pedagogical value (for participants)
  • invite / cite political action (for participants)

I just asked a question about privacy concerns, such as how to handle it when the participants will not consent to be recorded out of fear of retribution. One of the presenters has not encountered this, though the other one mentioned that it is impportant to do whatever the population being studied feeld comfortable with. Thus, I will continue to take hand-notes depending on if my population declines being recorded.

It seems one of the major issues is over how participatns and other people envision and imagine risk (even for the people involved), and not just on the perspective of the IRB).

Somebody is sharing an interesting strategy that is being used about how the intended population to be studied has a unique body of knowledge or experiences that we cannot get to in any other way than by speaking to these people. Good way of using this reason to convince an IRB to approve this sort of research if there is value in getting these experiences.

One of the ways of getting focus groups to work is when the facilitator strategically decides how much or how little to participate.

Paulo Freire (Reading the Word / Reading the World) is an example of the focus on pedagogy (as pedagogy and politics are linked):

  • Goal is praxis (to help people interrogate and change the conditions of their lives)
  • Use of “generative” words and phrases (examples) – usually a small number of these to help participants generate other words to counter the prevailng narrative
  • Cultivation of conscientization (moments for the group to move beyond)

Collective testimony leads to personal and collective empowerment:

  • authority of the researcher decentralized
  • safe spaces to talk about shared, real-life struggles
  • colleactive identity helps people “reclaim their humanity”
  • what seems uniquely personal gets theorized and validated
  • power (and its abuse)  is made visible
  • subjugation is given a face (not just a knot in the stomach)
  • individuals and collectives are given a voice
  • survival and resistance strategies are generated
  • new narratives are generated (equipment for living) — based on Freud’s notion that what happened is not as important as the stories we generate about what happened — very useful concept
  • complexity and contradiction or experience made visible

There seems to be some agreement amongst the participants that this approach to focus groups, namely around issues of creating safe spaces for the participants and a flattening of the power relationships is quite interesting.

Ahh, the Internet wifi service now appears to be working. Hurray! Glad I asked about this this morning before the sessions started and found somebody who could contact the university IS group to solve this issue.

General advice:

  • create safe and comfortable spaces “of” the people
  • create a sense of the festive (pizza, potluck dinners)
  • exploit pre-existing networks (e.g., Radway)
  • encourage collegiality, solidarity
  • use prompts not questions (open ended)
  • encourage (and allow) participants to “take over”

The concept of community can be a challenge in focus groups, as that may encourage groupthink or inclusion / exclusion depending on the issue at hand.

I asked about how to handle the issue in a focus group if the facilitator allows the participants to discuss and move along as they feel comfortable, and in the process a few dominant voices tend to control the conversation or otherwise silence other discussion. It was suggested that there then be a conversation about the reason for the focus group and how it is valuable to hear from all members of the group as that allows for a wider perspective of experiences to be expressed. I am thinking especially about how to implement this in the work that I am doing. My experiences have at times been when people tend to become very vociferous within the session, and that does not as easily allow for diversity, equality of voice, and perspective to be raised (and thus allowing the discussion to develop in other and quite interesting ways).

There is some interesting work over what does not get said. I asked about this and if the presenters can speak about how to write this up. They will come back to this topic.

There are some really interesting examples of the findings and how they wrote these up from the examples of real interview transcripts that they shared. This section was titled “Indexicality, Complexity, Nuanced, Contradictory” with the findings as “Emerging Patterns.”

There was a fascinating example of when what appears to be background chatter in one focus group, after it was analyzed, demonstrated that there was more that was occurring at the time than the researcher was initially aware. This is the example of how the silence in the focus group really points to the need for interpretation and explanatory understanding that the researcher brings to the experience.

I asked another question about how to organize the discussion / findings (such as by thematic analysis, areas that were not discussed, conclusions, etc.); is there a standard way they conceive of presenting the findings? There was an example of straight findings that were responded to from 3 distinct audience groups in very different ways from their own narratives. It seemed that however the findings were initially presented, there would still be different audience responses to the results that could bring the researcher back into the raw data to reinterpret it again in a newly-informed perspective.

I feel like I got everything I wanted to get from this discussion; this was extremely helpful for my theory as well as practice. Excellent session.


Preparing for ICQI2010

I am preparing my 2 paper presentations for the Sixth International Congress of Qualitative Inquiry, ICQI2010, and here are their abstracts as listed in the book of abstracts:

Keefer, Jeffrey M., Lancaster University
Learning Goals and Personal Learning in Practitioner Autoethnographic Inquiry: A Case Study

Autoethnography is an increasingly used method of inquiry in the social sciences that focuses upon an experience, or case, in the life of the researcher, who then studies this with the hope of presenting it with vivid and rich descriptive details. It is often intended that the reader be able to share in the experience, so the research is intentionally related to common cultural perspectives. There is limited research around why autoethnographers engage in this method, especially in the context of trying to better understand their goals and intentions for their own work, as well as what is hoped for within the audience. This exploratory case study research will explore the researcher’s learning goals for his or her readers, as well as how the researcher’s own learning develops through the process of conducting the autoethnographic research.

Keefer, Jeffrey M., Lancaster University
The Design and Initial Development of an eLearning Course to Organize and Frame a Qualitative Research Design: The Learning Qualitative Project

There is increasing attention to the challenges faced by faculty who teach qualitative research method courses. Whether lecturers do not have sufficient background to teach in this area, students are not ready to take these courses or their expectations differ from what is taught, logistical institutional factors inhibit teaching and learning, or challenges in the theory-practice relationship pose struggles to coursework, the struggles with teaching qualitative methods seem nearly endless (Hurworth, 2008). With so much need, it was decided to create a freely available eLearning course to assist faculty and students alike. A qualitative research design eLearning course was developed to address some of the basic elements of a research design, and the first version of this is being tested with feedback being used to improve the course’s usefulness.

Hope to see some colleagues, old and new, later this week!


Research Interests for the Residential Next Week

researchI am traveling to England on Sunday for my residential at Lancaster University, and the focus this year is on the methodologies and ideas for the upcoming doctoral thesis (doctoral dissertation in the US) that we hope to begin after the next year of coursework.

One of the assignments we have for next week is a welcome one — we should consider our research interests and prepare a 5 minute presentation to discuss them, their context, and possible methodology. While none of this will be set in stone, it will be the first time we have discussed this together with our cohort.

I have been thinking about this, and believe I will develop this here on my blog over the next few days. I have some ideas, though need to reflect on to process them and determine how I can move forward . . .