NRP/513 Version 2 Overall Critique of a Quantitative Research
NRP/513 Version 2 Overall Critique of a Quantitative Research
Aspect of the Report
Critiquing Questions
Title
Is the title a good one, succinctly suggesting key variables and the study population?
Abstract
Did the abstract clearly and concisely summarize the main features of the report (problem, methods, results, conclusions)?
Introduction
Statement of the problem
Was the problem stated unambiguously, and was it easy to identify?
Is the problem significant for nursing?
Did the problem statement build a persuasive argument for the new study?
Was there a good match between the research problem and the methods used—that is, was a quantitative approach appropriate?
Hypotheses or research questions
Were research questions and/or hypotheses explicitly stated? If not, was their absence justified?
Were questions and hypotheses appropriately worded, with clear specification of key variables and the study population?
Were the questions/hypotheses consistent with existing knowledge?
Literature review
Was the literature review up-to-date and based mainly on primary sources?
Did the review provide a state-of-the-art synthesis of evidence on the research problem?
Did the literature review provide a strong basis for the new study?
Conceptual/theoretical framework
Were key concepts adequately defined conceptually?
Was a conceptual/theoretical framework articulated—and, if so, was it appropriate? If not, is the absence of a framework justified?
Were the questions/hypotheses consistent with the framework?
Method
Protection of participants’ rights
Were appropriate procedures used to safeguard the rights of study participants?
Was the study subject to external review by an IRB or ethics review board?
Was the study designed to minimize risks and maximize benefits to participants?
Research design
Was the most rigorous design used, given the study purpose?
Were appropriate comparisons made to enhance interpretability of the findings?
Was the number of data collection points appropriate?
Did the design minimize biases and threats to the internal, construct, and external validity of the study (e.g., was blinding used, was attrition minimized)?
Population and sample NRP/513 Version 2 Overall Critique of a Quantitative Research
Was the population identified? Was the sample described in sufficient detail?
Was the best possible sampling design used to enhance the sample’s representativeness? Were sampling biases minimized?
Was the sample size adequate? Was the sample size based on a power analysis?
Data collection and measurement
Were the operational and conceptual definitions congruent?
Were key variables measured using an appropriate method (e.g., interviews, observations, and so on)?
Were the specific instruments adequately described and were they good choices, given the study population and the variables being studied?
Did the report provide evidence that the data collection methods yielded data that were reliable, valid, and responsive?
Procedures
If there was an intervention, was it adequately described, and was it rigorously developed and implemented? Did most participants allocated to the intervention group actually receive it? Was there evidence of intervention fidelity?
Were data collected in a manner that minimized bias? Were the staff who collected data appropriately trained?
Results
Data analysis
Were analyses undertaken to address each research question or test each hypothesis?
Were appropriate statistical methods used, given the level of measurement of the variables, number of groups being compared, and assumptions of the tests?
Was the most powerful analytic method used? (e.g., did the analysis help to control for confounding variables)?
Were Type I and Type II errors avoided or minimized?
In intervention studies, was an intention-to-treat analysis performed?
Were problems of missing values evaluated and adequately addressed?
Findings
Was information about statistical significance presented? Was information about effect size and precision of estimates (confidence intervals) presented?
Were the findings adequately summarized, with good use of tables and figures?
Were findings reported in a manner that facilitates a meta-analysis, and with sufficient information needed for EBP?
Discussion
Interpretation of the findings
Were all major findings interpreted and discussed within the context of prior research and/or the study’s conceptual framework?
Were causal inferences, if any, justified?
Was the issue of clinical significance discussed?
Were interpretations well-founded and consistent with the study’s limitations?
Did the report address the issue of the generalizability of the findings?
Implications/recommendations
Did the researchers discuss the implications of the study for clinical practice or further research—and were those implications reasonable and complete?
General Issues NRP/513 Version 2 Overall Critique of a Quantitative Research
Presentation
Was the report well-written, organized, and sufficiently detailed for critical analysis?
In intervention studies, was a CONSORT flowchart provided to show the flow of participants in the study?
Was the report written in a manner that makes the findings accessible to practicing nurses?
Researcher credibility
Do the researchers’ clinical, substantive, or methodologic qualifications and experience enhance confidence in the findings and their interpretation?
Summary assessment
The following questions address statistical conclusion validity, internal validity, construct validity, and external validity. These questions come from Box 10-1 in chapter 10.
1. Was there adequate statistical power? Did the manner in which the independent variable was operationalized create strong contrasts that enhanced statistical power? Was precision enhanced by controlling confounding variables? If hypotheses were not supported (e.g., a hypothesized relationship was not found), is it possible that statistical conclusion validity was compromised?
2. In intervention studies, did the researchers attend to intervention fidelity? For example, were staff adequately trained? Was the implementation of the intervention monitored? Was attention paid to both the delivery and receipt of the intervention?
3. What evidence does the report provide that selection biases were eliminated or minimized? What steps were taken to control confounding participant characteristics that could affect the equivalence of groups being compared? Were these steps adequate?
4. To what extent did the study design rule out the plausibility of other threats to internal validity, such as history, attrition, maturation, and so on? What are your overall conclusions about the internal validity of the study?
5. Were there any major threats to the construct validity of the study? In intervention studies, was there a good match between the underlying conceptualization of the intervention and its operationalization? Was the intervention “pure” or was it confounded with extraneous content, such as researcher expectations? Was the setting or site a good exemplar of the type of setting envisioned in the conceptualization?
6. Was the context of the study sufficiently described to enhance its capacity for external validity? Were the settings or participants representative of the types to which results were designed to be generalized?
7. Overall, did the researcher appropriately balance validity concerns? Was attention paid to certain types of threats (e.g., internal validity) at the expense of others (e.g., external validity)?
