December 25, 2015
Doing research simply means the systematic use of some set of theoretical and empirical tools to try to increase our understanding of some set of phenomena or events.
Joseph E.Mcgrath, the author of article “Methodology matters:doing research in the behavioral and social sciences” says.
In fact both of the two first articles are addressing the fact that what you are going to get out of a study totally depends on the method you are using and your philosophical stance. However the first article “Selecting Empirical Methods for Software Engineering Research” concentrates on software engineering research which makes it more sensible for me, while the second reading talks about research in social sciences.
The first reading mentions that we can not certainly say which research method is suitable for which research problems and various local factors should be considered when selecting a method, including available resources, access to subjects, opportunity to control the variables of interest and skills of the researcher. Furthermore, it precisely mentions that because each method has its own flaws, comprehensive research strategies which benefits from multiple research methods are more viable, so that weaknesses of each used method could be addressed and compensated by use of other methods.
First we must see what kind of research question we are asking. Potential questions include:
- Exploratory questions: like “does X exist?” or “what is X like?”
- base-rate questions: like “how often does X occur?”
- relationship questions: like “is X related to Y?”
- causality questions: like “does X cause or prevent Y?”
- design questions: like “what’s an effective way to achieve X?”
one important factor is your philosophical stance, which dramatically affects how evidences and responses to your research question(s) satisfies you. Here are four important philosophical stances:
- positivism: science is the process of verifying theories by testing hypotheses derived from them.
- constructivism: science is the process of seeking local theories that emerge from data and elaborate that.
- critical theories: theories are assertions of knowledge to be critiqued in terms of how they shape that power.
- pragmatism: theories are the products of a consensual process among a community of researchers, to be judged for their practical utility.
To classify possible research methods, we could introduce 5 major classes for software engineering research, however not all researchers in this area will necessarily have consensus on the names and domains of these classes:
- controlled experiment: An experiment in which one or more independent variables are manipulated to measure their effect on one or more dependent variables. It demonstrates the research hypothesis by testing it on a representative population. It enables to take control over unwanted variables, however if we ignore critical variables, then our experiment will not be comprehensive.
- case study: It is an empirical inquiry that investigates a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident. The problem with case study is that data collection and analysis is more open to researcher bias because of the nature of study, but the advantage is that it could be applied within all four mentioned philosophical stances.
- survey research: It is used to identify the characteristics of a vast population of people usually by analyzing data collected from questionnaires or structured interviews or even data logging techniques. The ideology is to well-sampling from population and trying to generalize analysis results to the whole population. So again there’s a challenge of selecting biased samples and the other challenge is the way questions and structures are defined, so the answers bring us valid data.
- ethnography: Its goal is to study a community of people to understand how the members of that community make sense of their social interactions so it focuses on the sociology of meaning through field observation. It best describes the constructivism stance and the challenge is possible preconceptions on data observation and analysis.
- action research: It attempts to solve a real-world problem while simultaneously studying the experience of solving the problem, so researchers using this method tend to enhance the situation while studying it which is a considerable difference from other methods.
- mixed methods: As we mentioned, all methods have some weaknesses so using a hybrid method would be desirable. They usually employ both quantitative and qualitative data analysis and provide vaster results. However resolving contradictions in results might be a challenge in mixed methods.
Furthermore, the second reading says that a research process always involve with a content of interest, some ideas that give meaning to it and some techniques that enable the researcher(s) to study them. The author depicts three domains of research in behavioral and social science including:
- substantive domain: From which we draw contents that seem worthy of our study and attention. In this domain, the phenomena itself and patterns of phenomena are being studied. The phenomena of interest involve the states and actions of some human systems and the conditions and processes that give rise to and follow from those states and actions.
- conceptual domain: From which we draw ideas that seem likely to give meaning to our results. Properties of the states and actions of those human systems that are the focus of study. Relations refer to any of a variety of possible ways in which two or more elements can be connected.
- methodological domain: From which we draw techniques that seem useful in conducting the proposed research. It makes use of basic sets of elements or tools by which social and behavioral scientists systematically gather empirical information.
Then the author mentions 8 research strategies and groups them into 4 quadrants as following:
- Field strategies including field study and field experiment. Both emphasize that the behavior system under study is natural, in the sense that it would occur whether or not the researcher were there and whether or not it were being observed as part of a study.
- Experimental strategies including laboratory experiment and experimental simulation. In contrast to those of Quadrant 1, they involve concocted rather than natural settings. The laboratory experiment and the experimental simulation are strategies that involve “actor-behavior-context” systems that would not exist at all were it not for the researcher’s interest in doing the study.
- Respondent strategies including sample and judgement study. They concentrate on the systematic gathering of responses of the participants to questions or stimuli formulated by the experimenter, in contrast to the observation of behaviors of the participants within an ongoing behavior system. Studies are usually done under neutral conditions of room temperature, lighting, chair comfort to nullify any effects of the behavior setting or context on the judgments that are the topic of study.
- Theoretical strategies including formal theory and computer simulation. The inclusion of these two strategies reminds us of the importance of the theoretical side of the research process. One of the more powerful general strategies for research is the simultaneous use of one of the theoretical strategies and one of the empirical strategies.
The author of this paper try to highlight the fact that results of our experiment always depend on methods, and like first reading, it verifies that a combination of multiple methods should be used. It also mentions the important fact that the results of a study should not be interpreted in isolation. Researcher should always consider other evidences and studies on the same research question.
In the 3rd reading, the author mentions some conceptual models of CSCW which are largely descriptive and then talks about her framework, which is currently descriptive as well, but supposed to be more developed by further investigation. She defends her model of “coordinated action” by claiming that it frees us from having to decide on only one common field of work and one “clear-cut goal”. She also mentions that they chose the word “action” to emphasize the importance of goal-directedness implied by the word “work”. MoCA has 7 dimensions as following:
- Synchronicity: According to Johansen’s matrix, it concerns a continuum of coordinated action ranging from being conducted synchronously, or at the same time, to asynchronously, or at different times.
- Physical Distribution: Again similar to Johansen’s matrix, this continuum concerns that if all actions are occurring in the same geographic location or at different places. It emphasizes that working from different locations for a long time will definitely be a big problem.
- Scale: Addresses number of participants involved in collaboration. Because a lot of articulation work needs to be done to organize members of a teamwork.
- Number of Communities of Practice: It focuses specifically on the notion of different cultural communities, so that when people from different disciplines come together and want to collaborate, how could we manage and resolve these differences.
- Nascence: Discusses Unstablished v.s. established coordinated actions.
- Planned Permanence: It refers to planned or intended permanence which is less addressed in similar works and it’s important because it’s usually impossible to say until when a coordinated action will continue and also hard to say when things are at a stable situation.
- Turnover: Refers to the rapidity with which participants enter and leave. This dimension cover collaborations that range from closed, private collaborations where participants leave slowly, if at all, to collaborations that are fully open and public so that might have many participants.
You can watch her presentation about this work in Stanford seminar:
As a new researcher in software engineering areas, I found the first reading surprisingly useful. It drew a comprehensive picture of the art of doing empirical research in software engineering, however many key points of this reading is not just limited to this area. It elegantly shows us how to propose a valid and valuable research question as the key starting point of doing research. Then describes what to consider when looking for an answer to this question and emphasizes that the way we think and our philosophical expectations and stances will have a considerable impact on potential answers. Then highlights the importance of having a theory which is like a lens through which the world is observed. Then provide nice information about various empirical methods in software engineering and how they could be combined together to compensate each other’s shortfalls. And finally provides information about how to collect data and how to validate results based on the stance and method we have used. The second reading also does same effort and in some cases provide more details on methods which the first reading didn’t discuss in-depth. It also reminisces me how closely interconnected social sciences and collaborative software engineering topics are. Finally, the final reading represents a framework of coordinated action which tries to add more value to related works and models around CSCW studies by discussing some aspects (dimensions in the author’s words) that were less focused by other works while addressing many useful works that have been done in the past.