Overview of Assessing
|As mentioned in the home page of this Web
site, the research literature can provide a useful mountain lookout that
broadens and deepens your perspective of a topic and allows you to scout
promising routes to distant objectives, but it can also be a snake pit
filled with groundless rhetoric, invalid findings, and misleading conclusions.
In addition, the available research on a given topic usually varies some
in the questions or hypotheses addressed, the contexts in which the research
is conducted, the interventions that are applied in experiments and demonstrations,
and the research methods used. All these can affect the results and
thus should be taken into account during the next step of the review process,
synthesis across studies.
What causes invalid results and misleading conclusions in research studies? Some are caused by incompetence, when the researchers are not adequately prepared to conduct the study. Some are due to constraints in resources (money, staff, time) and access that force scholars to conduct research that is less thorough and rigorous than they would desire. Some result from honest mistakes and others from carelessness. Finally some are due to duplicity--deliberately skewing the research so that it serves only to buttress preconceived opinions.
Most research journal editors have two knowledgeable scholars review each article submitted for publication. This is supposed to identify weaknesses and used either to request improvements or to reject the submitted article. Ulrich's International Guide to periodicals indicates which use peer review. University presses and commercial publishers of text books usually require peer review; most other commercial publishers do not use it.
Peer review definitely helps screen out the worst research, but there is compelling evidence that it allows considerable flawed research to reach print. For instance, in a study of 114 research articles sampled from 44 journals, the average expert ratings of the problem statements, research procedures, data analysis, and summary and conclusion were each “mediocre” (Ward, Hall, and Schramm, 1975).
The merit of a research study can be considered like a chain hanging from a stout crossbeam, holding up a heavy pallet of knowledge. If any one link has a major crack, the pallet is in danger of crashing to the floor. The links include the background preparation for the study, the conceptual framework, the questions or hypotheses addressed, the contexts prevailing during the study, the interventions actually applied in experiments and demonstrations, the methodology of the study, the results, and the conclusions inferred from the study.
The following are some general guidelines that should help when assessing individual research studies. The guidelines will be discussed broadly so that they are generally applicable to both quantitative and qualitative research, although not every discussion will be applicable to all research approaches.
This is the first of eight brief lessons on assessing research literature. The other lessons are indicated below. For moderate competency in conducting literature reviews, one should master all the lessons.
Lesson A-2: Assessing
the background of the study
Last Update: June 29, 2000