Summary & Review of Article - "What Theory is Not" by R. I. Sutton & B. M. Shaw

The content of this blog post is based primarily on the following article:

Sutton, R. I., & Staw, B. M. (1995). What theory is not. Administrative Science Quarterly, Vol. 40, No. 3, pp. 371-384.

In the article "What theory is not" by Sutton and Staw (1995), it is noted that many researchers mistake references, data, variables/constructs, diagrams, and hypotheses for theory. The authors also urged journals and editors to be more receptive to papers that investigate a part of rather than an entire theory, and utilise illustrative (qualitative) rather than definitive (quantitative) data. 

Sutton and Staw (1995) explained that:

• References are not theory, because researchers need to explain which concepts and arguments are adopted from sources and how they are linked to the theory.  
• Data are not theory, because data merely describes which empirical patterns were observed and theory explains why empirical patterns were observed. It does not constitute a theory. Researchers who use qualitative data must develop causal arguments (theory) to explain why findings are observed. 
• Variables/constructs are not theory. The key issue is why certain variables are more important and thus chosen, not what variables/construct are in the theoretical model. 
•  Diagrams are not theory, because they don't explain why. However, the authors acknowledged that a good theory is often representational and verbal.
•  Hypothesis (or predictions) are not theory, because hypotheses are statements about what is expected to occur, not why it is expected to occur.

A strong theory answers why and delves into the underlying processes. According to the authors, strong theories are missing in many quantitative research papers, as they seem to be overly concerned with methodology. There is a need to rebalance the selection process between theory and method. However, the authors also noted that theory is often over-emphasized in qualitative research. In light of these findings, the authors argued that the best papers are those that strike a fine balance between theory and method.

Understanding "Unit of Analysis"

I had quite a hard time understanding the entire concept of "unit of analysis" when it was first being introduced by Kerry, but gradually began to grasp the concept when Kerry kept prompting me to change my unit of analysis for my research project on self-efficacy beliefs. Kerry's intention of wanting me to change my unit of analysis wasn't because he felt that the individual level of analysis was inappropriate for the purpose of my research, but he wanted me to understand what the unit of analysis is, and how I could go about using it to analyse a journal article, research, or theory.

Units of analysis are issues we examine to account for differences among separate entities.  The biggest misconception that most people have regarding the unit of analysis is that it refers to the level of observation in a study. I fell into this trap as well. When Kerry mentioned that the unit of analysis could be at the individual, group, organisational, institutional, or societal level, I immediately jumped to the conclusion that the unit of analysis refers to what the researcher is studying. It is not the sample that you're studying which determines your unit of analysis, it is the analysis that you do in your study that determines what the unit is. If one were to liken the unit of analysis to the unit of observation/sampling, then there would only be one "unit of analysis" throughout the study. However, in qualitative research, it is possible for a study to have different units of analysis. In fact, researchers can position their research in different ways based on different units of analysis. For example, I might decide to position my analysis based on individual self-efficacy beliefs. In this case, the unit of analysis is at the individual level. However, when analysing my collected data, I might decide to compare the average level of self-efficacy. In this case, the unit of analysis is at the group level.

In varying the unit of analysis, Kerry urged us to be consistent and anticipate what conclusions we wish to make with regard to each unit of analysis. He has seen many examples of research where statements of a particular unit of analysis are actually based on another unit of analysis. Essentially, in exploring the different units of analysis, we might risk drawing invalid conclusions where the units of analysis do not match.

Elements of a Good Proposal

A research proposal is important because it assesses the researcher's expertise in the area in which he/she is conducting research, his/her knowledge of the existing literature, and how the research project will enhance the current pool of knowledge. Potential supervisors, admissions officers, and/or research funders usually base their decisions on research proposal, and emphasise on the quality and originality of ideas, the researcher's skills in critical thinking, and the feasibility of his/her research project.

With reference to the aims and importance of a research proposal, I will now proceed to discuss "what makes a good proposal"? I will do so in three steps, as shown below:

(1) According to Kerry's good proposal template©, a good proposal addresses the following questions:

1. What is my key academic paper?
2. What is the problem (mistake/gap) in that paper?
3. What is the "model" of the process you are looking at?
4. What is my "best guess" / theory for how I would expect the gap to work?
5. Why is the gap important for cause and effect issues?
6. What evidence do I need to "fill the gap"?
7. What is a reasonable way to gather the evidence I need?

(2) Translating these questions into specific elements of a good proposal, we get:

• Question 1 refers to the underlying motivation of the research project. 
• Question 2 refers to the research gap present in existing literature.
• Question 3 refers to the cause and effect linkages within the field of research.
• Question 4 refers to the theory proposed by the researcher (theoretical process). 
• Question 5 refers to the link between the research gap and the cause-and-effect linkages. 
• Question 6 refers to evidence that addresses the research gap/problem.
• Question 7 refers to the method used to collect data (methodology) to support the theory proposed earlier.

(3) The last step is to assign proper headings to the elements identified in step (2):

• Introduction
  - Research Issue
  - Literature Review
  - Research Question(s)
• Purpose of Study
• Research Design & Methodology
  - Sample
  - Data Collection
  - Data Analysis
  - Pilot Study
  - Ethical Consideration(s) 

 A PhD or Honours thesis proposal might even include the following items:

• Proposed Outcome(s)
• Limitation(s)
• Dissemination
• Timeline
• Resources
• References
• Appendices

This list is not exhaustive, but it attempts to capture the key elements that researchers usually include in their research proposals. A proper research grant proposal might even include additional sections such as "budget", "project plan", or "staff and institutional qualifications", but for the purpose of this course and my Honours thesis, I will not have to worry about funding.

Instead of giving us strict headings for the proposal, I like how Kerry prompts us with questions that help us to understand the essential components of a research proposal. In particular, the questions "What is my "best guess" / theory for how I would expect the gap to work?" and "Why is the gap important for cause and effect issues?" have constantly reminded me that the theory and the research gap are extremely important for high quality and interesting research. I will now proceed to write a complete proposal for the qualitative research project of this course.

The Ladder of Inference - Don't be too quick to jump to conclusions!

I came across this simple yet insightful TED-Ed video entitled "Rethinking Thinking" by Trevor Maber, and was introduced to the "ladder of inference", an idea developed by American business theoriest Chris Argyris, and subsequently thrust into the corporate world by Peter Senge via his book "The Fifth Disciple: The Art and Practice of the Learning Organisation".

The ladder is often used as a tool to help an individual understand how and why he/she thinks as he/she does about an issue. It can also help the individual understand why others think differently about an issue and empathise with their perspectives. The conscious use of the ladder of inference has been shown to be a valuable resource for understanding the source of differences in opinions. I identified many similarities between the research process and the steps involved in the ladder of inference, and one of the most important lessons which I learned while comparing both is - never be too quick to jump to conclusions! And one of the ways to avoid jumping to conclusions is to always be clear about the underlying assumptions and conclusions made. 

The steps involved in the ladder of inference, adapted from:
http://gwynteatro.wordpress.com/2011/03/13/climbing-the-ladder-of-inference/

Consultants advocate the use of the ladder to help leaders or management draw better conclusions, make better decisions, or challenge other people's conclusions based on the facts and information available. They also encourage the use of the ladder in analysing hard data, for example, a set of sales figures, the consumption habits of the general populace, or to test assertions. At the individual level, the ladder of inference can also be used to help validate or challenge other people's assertions or conclusions.

In research, although not many researchers or scholars have mentioned the use of the ladder in their thought processes, I have noticed many similarities between the steps expounded in the ladder of inference and how researchers go about doing research.

The first step of the ladder of inference is the data. There is a vast amount of data in the world, and each individual has a limited capacity to absorb them. Hence, the individual selects the data, and proceeds to add meaning to the data he/she has selected. Based on the meaning he/she has added, assumptions are made and conclusions are drawn, and action is taken based on the conclusions made. The cycle repeats when the results/consequences of the actions become data for another round of climbing the ladder. Over time, it is believed that the conclusions formed contribute to the foundation of the individual's beliefs, assumptions, and even values. They also play an influential role in filtering the data selected and adding meaning to the data when the process is repeated time and again.

In fact, I think the reason why the ladder of inference is highly applicable to researchers and their thought processes is because the logic behind the ladder is built on the assumptions about the human behaviour. In general, humans tend to:   

• assume that others see the world as they do. Hence, if there are any disagreements, they are usually concentrated on the conclusions. The issue here is, humans assume that everyone selects the same data and adds the same meaning to the data. Reality is, none of us do.; 
• take short-cuts around the ladder (e.g. jump to conclusions), and are unconscious of the steps they have climbed on the ladder; and
• assume that any conclusions made are the "truth". But what is the truth? 

Similarly, in research, the data at the bottom of the ladder represents the pool of information that may be relevant to the researcher's research interests. Going up the ladder, the researcher then selects the relevant information from the pool of information available. The researcher proceeds to learn what the selected information describes about the phenomenon or issue he/she is studying, and from thereon, interpret and evaluate whatever he/she has noticed. Any assumptions or beliefs that the researcher holds greatly influence whatever he/she notices from the selected information. Following which, the researcher seeks to link the issue he/she is studying to the information he/she has selected, identifying any inconsistencies and consistencies between both, and finally, coming up with a theoretical framework on which the issue will be investigated. Having established a theoretical framework, the researcher then proceeds to investigate the issue and provide explanations and conclusions regarding his findings, and at the end of it, he/she will usually provide some limitations and recommendations for future research.

I found the following questions derived from the ladder of inference extremely useful when doing research:

• What are the facts that I should be using?
• Are the facts relevant?
• Are there other facts I should have considered?
• What data have I chosen to use and why?
• Have I selected data rigorously?
• Why have I chosen this course of action/approach?
• Are there other actions/approaches which I should have considered?
• What belief/assumption led to this particular action?
• Were the beliefs/assumptions well-founded?
• What am I assuming, and why?
• Essentially, are my assumptions valid?
• Why did I draw that conclusion?
• Is the conclusion sound?

The temptation to skip various steps of the ladder and jump straight to making conclusions is something that the ladder of inference would like to remind people about (and prevent them from doing so). In order to address and overcome this temptation, the ladder of inference advocates and emphasises critical thinking in accepting the existence of perceptions, rather than simply trusting any data that one has encountered. The next time you do any form of research, it might be a good idea to go through your thinking process following the steps of the ladder of inference. Having done that, let me know if the ladder of inference did assist you in gaining further insights about your thought processes and how you go about doing research.

Research = Conversation

"I think of research as a conversation, and it really is very much like a conversation. No single person dominates it, but what does happen is when you interject something, when you contribute something to a conversation, you want to be understood, you want to be heard, you would like people to pay attention, you would like it to have some influence on the way the conversation goes. You don't control it." — Daniel Kahneman

I particularly like how researchers (see Anne Huff, 1999; McMillen & Hill, 2004) liken research to an unending conversation. Because conversation is such a familiar activity that takes place on a daily basis,  it has enabled me to better understand the processes involved when doing research.

Perhaps the most important and striking similarity between both is that they are interactive processes. The Merriam-Webster Dictionary defines conversation as an "oral exchange of sentiments, observations, opinions, or ideas". In other words, it is a bi-directional flow of opinions and ideas, such that all parties involved in the conservation bring something to the table, discuss and deliberate on the opinions and ideas provided, reach an agreement or consensus (this might not happen), and after which, gain something from the conversation that occurred. Anne Hull (1999) considered conversation the essence of society, citing the Oxford English Dictionary which defined conversation as "the action of living or having one's being in a place or among persons."

Many novice researchers, like myself, find it hard (initially) to picture research as a conversation — after all, our first impression of research is that it is a solitary self-discovery journey where researchers read up on their research topics/areas, proceed to carry out surveys/questionnaires/experiments, and finally, pen their research reports or theses. It seems as though we, under the guidance of our supervisors, are the only ones who are trying to make sense of the information out there, with the ultimate aim of presenting and sharing our findings to our research communities.

Little did we know that every available information out there — the facts, opinions, and findings in journal articles, books, or even artefacts — is the result of multiple, never-ending conversations among researchers. Research is built on conversation and interaction, it requires both elements, and at the same time, it is the product of both elements. The paragraphs of words that we encounter in research journals, books, and so on — were, more often than not, carefully chosen to convey the right message to the right audience. The citations and references made in journal articles to ascertain, challenge, or support the researcher's arguments are fundamentally scholarly dialogues, which help to shape the boundaries, define the key issues, and deepen understanding in various academic disciplines.

In the article "Why teach 'research as a conversation' in freshman composition courses? A metaphor to help librarians and composition instructors develop a shared model", McMillen & Hill (2004) proposed several features that are common to both research and conversation. Essentially, they pointed out that research and conversation are both:

• interactive and recursive processes;
• context-sensitive and -situated; and
• avenues through which people construct meaning. 

They also added that learning how to research is equivalent to learning how to converse in a second language, and because conversation is such a familiar day-to-day activity, they came out with a teaching model based on the idea of a conversation to introduce students to research. An inherent assumption underlying the teaching model is that it is much easier to understand what conversation entails than to comprehend the formal structure of academic discourse embodied in research.

Getting started with research — joining the conversation

How do you get enough background information on the topic so that you can participate in the conversation?

How can researchers join the conversation?

A researcher, no matter how experienced or skilful he/she is, would need to make sense of the research conversations that are taking place in that discipline, before deciding where and how his/her ideas fit into the pool of conversations. Most researchers usually do this by reading up extensively on the topic. They then proceed to write a literature review, which reveals a research gap in current literature. When analysing each journal article or book, the researcher annotates the information presented. Essentially, he/she determines where the author is coming from, examines the author's arguments, as well as the author's area of expertise. With an in-depth understanding of the research area, the researcher can then proceed to critique other researchers' work in the same area, and put forth his own arguments and findings through a research problem/research questions. That is when the researcher enters the conversation.

What is the researcher's role?

The researcher's role throughout the conversation (or research) is to keep the bi-directional flow of ideas, opinions, suggestions, and arguments going. He/she could choose to either explore (breadth) or focus (depth) in a particular area, and this will be done through continuous readings. As he/she keeps the conversation going, he/she may agree or disagree with certain sources of information. Slowly but surely, trends and patterns will start to appear. Coupled with the researcher's own ideas, all these bi-directional dialogues will be compiled and put into place. This ultimately paves the way for the final research project/thesis/presentation.

References:

Huff, A. S. 1999. Writing for scholarly publication. Thousand Oaks, California: Sage Publications, Inc.

McMillen, P. S., & Hill, E. 2004. Why teach “research as a conversation” in freshman composition courses? A metaphor to help librarians and composition instructors develop a shared model. Research Strategies, 20(1): 3-22.