Improve and innovate through evaluation, research, and quality improvement

Core Function: Assurance

Why This Is Important

Ongoing evaluation can help local health departments assess the impact of opioid prevention efforts, as well as reveal opportunities for refinement. While engaging in a robust evaluation of SUD- and OUD-related efforts may be beyond the capacity of many local departments, the systematic collection and analysis of information about substance use-related programs and policies can help to reduce uncertainty, improve effectiveness, and facilitate decision-making.

What’s Involved

  • Evaluate services, policies, plans, and laws continually to ensure they are being delivered as intended and producing positive outcomes.
  • Involve community members in all evaluation processes.
  • Value and use both qualitative and quantitative data, including input from people with lived experience, to assess effectiveness.

Key Considerations

  • Outcome evaluation looks at results, but results don’t tell the whole story. An outcome evaluation alone, without the context provided by an accompanying process evaluation, won’t help you understand why a program did or did not work.
  • Stakeholder involvement, including the voices of people with lived experience, is a crucial part of any evaluation effort. People who have survived overdose and their families can be invaluable partners in understanding what your data tell you. One focus group conversation with participants at a recovery support center or treatment facility could help you unpack the data you are collecting, what they mean, and how to use them to improve service delivery.
  • Evaluating naloxone distribution can be complex, so don’t go it alone! Some identified challenges related to naloxone use include:
    • Tracking when, where, and by whom naloxone is actually administered.
    • Tracking the outcome of naloxone administration (e.g., whether the person survived the overdose, the survivor received medical attention/follow-up support).
    • Finding out if naloxone was involved in a non-fatal overdose (e.g., post-overdose visits don’t always ask survivors or family members this directly)
    • Reaching and getting naloxone into the hands of people who need it most, particularly if they are from a culture that is less willing to seek help.

Consider partnering with an evaluator to help you develop the protocols and processed needed to collect this information. Working with law enforcement and first responders to document when naloxone is used on calls is another way to get data.

Get Started!

  • Make the case for the importance of data collection. Though funders almost always want clear evidence of how grant funding is being used, partners may be reluctant to collect or share this information if it presents their agency or community in a negative light. Work with partners to resolve challenges and ambivalence related to data collection and convey the importance of accurate reporting to keep resources available for prevention efforts.
  • Begin setting up memos of agreement or data sharing agreements with key partners. Data-sharing agreements document what data is being shared, for what purpose, and how the data will be used. This will help to clarify expectations and prevent miscommunication. The agreement should meet the needs of both the agency providing the data as well as the agency receiving the data.
  • Take an inventory of existing data, and begin making a plan for how you will fill identified data gaps. Involve community members in the development of this plan, including defining evaluation questions, indicators of success, and realistic timeframes for getting the work done.

Evaluation in Action: New Bedford Health Department

When the City of New Bedford Health Department was awarded a 2019 grant of more than $1.8 million from the Substance Abuse and Mental Health Services Administration to continue its work to fight opioids, evaluation was built into their plan from the start. Funded through the First Responders-Comprehensive Addiction and Recovery Act, the 4-year grant would support the health department’s TRAIN (Teach, Reach, and Initiate Naloxone) Project—a program designed to prepare first responders and key personnel in New Bedford and two neighboring communities to administer naloxone and provide post-overdose follow-up to affected individuals and their families.

Because the project included multiple, moving parts and involved multiple players—the New Bedford Health Department, the police department, and the Fishing Partnership Support Services—the TRAIN Project first developed a clear evaluation plan; this helped to ensure that everyone involved was on the same page.

To inform the plan, the team conducted key informant interviews with various community stakeholders from the three communities that would be served by the project, including representatives from law enforcement, treatment and recovery agencies, organizations serving the fishing community, and organizations serving the Spanish- and Portuguese-speaking communities.

The team also worked with health department leadership to develop a logic model—a graphic roadmap that laid out what the project hoped to accomplish and the concrete steps they would take to get there. For each of the project’s key goals (training, naloxone distribution, and post-overdose follow-up), the logic model presented a statement of need, core objectives, a detailed list of corresponding activities, and clear process and outcome measures.

Once the logic model was in place, the health department turned its attention to developing a set of data collection tools to track program delivery. The department also worked together, and in collaboration with project partners, to develop a practical data collection approach that wouldn’t present an undue burden. Wherever possible, they streamlined the collection process, recognizing that theirs was just one of several grants for which partners were collecting data. They also vetted individual survey questions to make sure they were being asked in culturally sensitive ways.

Finally, the team developed a centralized data repository to store the collected data for each of the program’s three components. They also trained designated staff to input the data, thus helping to ensure consistency across components. The repository not only facilitated project reporting, but also supported data analysis over time.