Evidence-Based Methodology: Difference between revisions
No edit summary |
ShoaibIqbal (talk | contribs) new writeup |
||
| Line 1: | Line 1: | ||
[[File:18 Evidence of Impact.mp4|thumb|Dr Peter Wyman, in interview with Tim McGowan, says that rigorously tested programs that follow key findings of evidence offer our best chances of success in prevention. ]] | [[File:18 Evidence of Impact.mp4|thumb|Dr Peter Wyman, in interview with Tim McGowan, says that rigorously tested programs that follow key findings of evidence offer our best chances of success in prevention. ]] | ||
Evidence-based methodology means treating a programme's impact as a question to test, not an assumption to repeat. A design can look sound on paper and still fail once measured, so the commitment is to keep measuring, not to stop once a plausible theory is in place. | |||
Wyman et al. (2020) built this commitment into the Wingman-Connect trial from the outset. The study was not a simple before-and-after check: it was "conducted from October 2017 to October 2019" as a trial with an active comparison condition, and its stated aims went beyond outcomes alone. "The primary hypothesis tested was that Wingman-Connect would reduce suicidal ideation, depression symptoms, and job-related problems. A secondary objective was testing the guiding network health theoretical model" (Methods). That second aim matters: the trial was designed to test whether the proposed mechanism was real, not only whether the numbers moved. | |||
The results honoured that distinction. Rather than stopping at "it worked," the authors traced why: "participants' perceptions of being embedded in a more cohesive, healthy class accounted for significant portions of Wingman-Connect's impact on reducing suicidal ideation and depression symptoms" (Results). Evidence-based methodology, on this model, means testing the theory of change itself, not just its endpoint. | |||
This same commitment shows up in an unrelated setting, evidencing that it is a methodology and not a single result. McNeill et al. (2024) evaluated a graduate nursing course on evidence-based practice using a validated instrument before and after teaching, and reported that "the results indicated a significant difference pre vs post course in the areas of EBP use... EBP Skill... and EBP Communication." But they did not stop at the positive result: the same data showed that "22 out of the 24 students that reported feeling overwhelmed by the course were MSN students," and the authors concluded plainly that "the facilitation, evaluation, and redesign of EBP curriculum need to be informed by data gathered from formal assessments." A programme that works overall can still surface a problem in the same measurement; evidence-based methodology means acting on both findings, not just the favourable one. | |||
Two honest caveats belong here, and naming them is itself part of the methodology. The Wingman-Connect trial is a randomized design and carries strong evidentiary weight, but it comes from a single research programme. The McNeill et al. study is a single-cohort, pre-post design with no control group, a weaker tier of evidence than a trial; its conclusion about curriculum should be read as a prompt for further testing, not as settled fact. Grading the strength of evidence, rather than treating all findings as equal, is not a footnote to evidence-based methodology. It is the methodology. | |||
''' | ==== '''Context: Evidence-Based Methodology in Schools''' ==== | ||
Schools need a way to judge whether a claimed evidence base actually meets the standard, not just whether it sounds plausible. Santiago-Rosario et al. (2023) provide exactly this for one common schoolwide framework, concluding after systematic review that "PBIS can be designated an evidence-based practice for reducing exclusionary discipline and improving social, emotional, and behavioral outcomes" (p. 1). That is a formal grading of evidence strength, not a general endorsement. The same standard applies to network-based prevention: Wyman (2014) reports that "training for high school student peer leaders to prepare them to modify norms through their natural social networks (Sources of Strength) has increased schoolwide help-seeking acceptability, coping norms, and engagement of adults to help suicidal peers" (p. S254) — a specific, tested claim, not an assumption. A school adopting either framework should expect this same level of documented evidence, not take effectiveness on faith. | |||
==== '''Context: Evidence-Based Methodology among Remote Workers in IT and Technology''' ==== | |||
No network-health programme for IT or software teams has been tested with a trial design comparable to Wingman-Connect's, so evidence-based methodology here means being honest about what standard of proof exists rather than borrowing one from elsewhere. The closest available evidence is observational rather than experimental. Yang et al. (2022) used a large natural experiment, not a randomized trial, to show that "firm-wide remote work caused the collaboration network of workers to become more static and siloed, with fewer bridges between disparate parts" (Abstract). That finding is credible and large-scale, but it documents an effect of remote work, not the impact of any specific intervention designed to counter it. Applying a network-health programme to distributed IT teams without testing it against this same bar — a comparison condition, a measured mechanism, replication beyond one firm — would be exactly the kind of unevidenced adoption this finding warns against. | |||
''' | ==== '''References''' ==== | ||
McNeill, C., George, N., Stephens, U., & Walker, T. (2024). Teaching evidence-based practice to MSN, DNP and PhD students: Lessons learned. ''ABNFF Journal, 1''(1), 53–60. | |||
Santiago-Rosario, M. R., McIntosh, K., Izzard, S., Cohen Lissman, D., & Calhoun, E. (2023). ''Is positive behavioral interventions and supports (PBIS) an evidence-based practice?'' Center on PBIS, University of Oregon. | |||
Wyman, P. A. (2014). Developmental approach to prevent adolescent suicides: Research pathways to effective upstream preventive interventions. ''American Journal of Preventive Medicine, 47''(3, Suppl. 2), S251–S256. <nowiki>https://doi.org/10.1016/j.amepre.2014.05.039</nowiki> | |||
Wyman, P. A., Pisani, A. R., Brown, C. H., Yates, B., Morgan-DeVelder, L., Schmeelk-Cone, K., Gibbons, R. D., Caine, E. D., Petrova, M., Neal-Walden, T., Linkh, D. J., Matteson, A., Simonson, J., & Pflanz, S. E. (2020). Effect of the Wingman-Connect upstream suicide prevention program for Air Force personnel in training: A cluster randomized clinical trial. ''JAMA Network Open, 3''(11), e2022532. <nowiki>https://doi.org/10.1001/jamanetworkopen.2020.22532</nowiki> | |||
Yang, L., Holtz, D., Jaffe, S., Suri, S., Sinha, S., Weston, J., Joyce, C., Shah, N., Sherman, K., Hecht, B., & Teevan, J. (2022). The effects of remote work on collaboration among information workers. ''Nature Human Behaviour, 6'', 43–54. <nowiki>https://doi.org/10.1038/s41562-021-01196-4</nowiki> | |||
Revision as of 13:49, 22 July 2026
Evidence-based methodology means treating a programme's impact as a question to test, not an assumption to repeat. A design can look sound on paper and still fail once measured, so the commitment is to keep measuring, not to stop once a plausible theory is in place.
Wyman et al. (2020) built this commitment into the Wingman-Connect trial from the outset. The study was not a simple before-and-after check: it was "conducted from October 2017 to October 2019" as a trial with an active comparison condition, and its stated aims went beyond outcomes alone. "The primary hypothesis tested was that Wingman-Connect would reduce suicidal ideation, depression symptoms, and job-related problems. A secondary objective was testing the guiding network health theoretical model" (Methods). That second aim matters: the trial was designed to test whether the proposed mechanism was real, not only whether the numbers moved.
The results honoured that distinction. Rather than stopping at "it worked," the authors traced why: "participants' perceptions of being embedded in a more cohesive, healthy class accounted for significant portions of Wingman-Connect's impact on reducing suicidal ideation and depression symptoms" (Results). Evidence-based methodology, on this model, means testing the theory of change itself, not just its endpoint.
This same commitment shows up in an unrelated setting, evidencing that it is a methodology and not a single result. McNeill et al. (2024) evaluated a graduate nursing course on evidence-based practice using a validated instrument before and after teaching, and reported that "the results indicated a significant difference pre vs post course in the areas of EBP use... EBP Skill... and EBP Communication." But they did not stop at the positive result: the same data showed that "22 out of the 24 students that reported feeling overwhelmed by the course were MSN students," and the authors concluded plainly that "the facilitation, evaluation, and redesign of EBP curriculum need to be informed by data gathered from formal assessments." A programme that works overall can still surface a problem in the same measurement; evidence-based methodology means acting on both findings, not just the favourable one.
Two honest caveats belong here, and naming them is itself part of the methodology. The Wingman-Connect trial is a randomized design and carries strong evidentiary weight, but it comes from a single research programme. The McNeill et al. study is a single-cohort, pre-post design with no control group, a weaker tier of evidence than a trial; its conclusion about curriculum should be read as a prompt for further testing, not as settled fact. Grading the strength of evidence, rather than treating all findings as equal, is not a footnote to evidence-based methodology. It is the methodology.
Context: Evidence-Based Methodology in Schools
Schools need a way to judge whether a claimed evidence base actually meets the standard, not just whether it sounds plausible. Santiago-Rosario et al. (2023) provide exactly this for one common schoolwide framework, concluding after systematic review that "PBIS can be designated an evidence-based practice for reducing exclusionary discipline and improving social, emotional, and behavioral outcomes" (p. 1). That is a formal grading of evidence strength, not a general endorsement. The same standard applies to network-based prevention: Wyman (2014) reports that "training for high school student peer leaders to prepare them to modify norms through their natural social networks (Sources of Strength) has increased schoolwide help-seeking acceptability, coping norms, and engagement of adults to help suicidal peers" (p. S254) — a specific, tested claim, not an assumption. A school adopting either framework should expect this same level of documented evidence, not take effectiveness on faith.
Context: Evidence-Based Methodology among Remote Workers in IT and Technology
No network-health programme for IT or software teams has been tested with a trial design comparable to Wingman-Connect's, so evidence-based methodology here means being honest about what standard of proof exists rather than borrowing one from elsewhere. The closest available evidence is observational rather than experimental. Yang et al. (2022) used a large natural experiment, not a randomized trial, to show that "firm-wide remote work caused the collaboration network of workers to become more static and siloed, with fewer bridges between disparate parts" (Abstract). That finding is credible and large-scale, but it documents an effect of remote work, not the impact of any specific intervention designed to counter it. Applying a network-health programme to distributed IT teams without testing it against this same bar — a comparison condition, a measured mechanism, replication beyond one firm — would be exactly the kind of unevidenced adoption this finding warns against.
References
McNeill, C., George, N., Stephens, U., & Walker, T. (2024). Teaching evidence-based practice to MSN, DNP and PhD students: Lessons learned. ABNFF Journal, 1(1), 53–60.
Santiago-Rosario, M. R., McIntosh, K., Izzard, S., Cohen Lissman, D., & Calhoun, E. (2023). Is positive behavioral interventions and supports (PBIS) an evidence-based practice? Center on PBIS, University of Oregon.
Wyman, P. A. (2014). Developmental approach to prevent adolescent suicides: Research pathways to effective upstream preventive interventions. American Journal of Preventive Medicine, 47(3, Suppl. 2), S251–S256. https://doi.org/10.1016/j.amepre.2014.05.039
Wyman, P. A., Pisani, A. R., Brown, C. H., Yates, B., Morgan-DeVelder, L., Schmeelk-Cone, K., Gibbons, R. D., Caine, E. D., Petrova, M., Neal-Walden, T., Linkh, D. J., Matteson, A., Simonson, J., & Pflanz, S. E. (2020). Effect of the Wingman-Connect upstream suicide prevention program for Air Force personnel in training: A cluster randomized clinical trial. JAMA Network Open, 3(11), e2022532. https://doi.org/10.1001/jamanetworkopen.2020.22532
Yang, L., Holtz, D., Jaffe, S., Suri, S., Sinha, S., Weston, J., Joyce, C., Shah, N., Sherman, K., Hecht, B., & Teevan, J. (2022). The effects of remote work on collaboration among information workers. Nature Human Behaviour, 6, 43–54. https://doi.org/10.1038/s41562-021-01196-4