09 Jun You CAN Evaluate Your Program! How to Conduct an Environmental Scan and Needs Analysis (Part 1 of 11) By: AGS Staff

Have you been tasked with evaluating a program and don’t know where to begin? If so, you aren’t alone. Many people struggle with program evaluation. This new three-part series on evaluation will prepare you to design and implement a strong evaluation comprised of quantitative and qualitative data analysis. You will also understand how to secure a third-party evaluator, if you need one. This post focuses on how to conduct an environmental scan and needs assessment.

Environmental Scan vs. Needs Assessment:

While the terms needs assessment and environmental scan are often used interchangeably, there are differences. A needs assessment collects, analyzes, and reports data about needs or gaps related to an issue, program, or target population. An environmental scan examines issues and resources that exist at the program-, organization-, or community-level. Environmental scans can assist with strategic planning, developing/implementing new programs, or revising current programs for continuous quality improvement. We will use the term “scan” to refer to the combined needs assessment and environmental scan. Scanning serves several purposes:

• Increase understanding of the needs of a target population, program, organization, or community.
• Raise awareness of issues, risks, strengths, assets, and protective factors.
• Identify gaps in services or resources.
• Understand how programming compares with others in the community, region, state, or nation.
• Inform program strategic planning—outcomes, activities, services, policies, and procedures—resulting in improved performance.
• Increase engagement with diverse, varied stakeholders, building stronger relationships and partnerships.

How to Conduct a Scan:

Comprehensive scanning includes multiple data collection methods, including literature reviews, existing data, observation, surveys, focus groups, and interviews. Comprehensive scanning typically occurs most often in times of uncertainty, such as when a new program is being designed. However, various components of a scan can benefit a program or organization each year. For example, you may create a schedule to collect data from current and potential partners and other stakeholders every three years and participant data annually via observation, surveys, focus groups, and/or interviews. If you don’t need or are unable to conduct a comprehensive scan, you may choose to implement components that best fit your needs.

We will use the following scenario to build out the steps of conducting a scan.

Your organization provides recovery housing with mental health services. You have been tasked with conducting a scan for the purpose of improving this program.

Step 1: Define the Purpose and Scope of the Scan

Asking questions will help you focus the scan to avoid wasting time and other resources. How will you use the results? Do you want depth or breadth? What topics do you want to investigate? While your process should have flexibility, the plan should have focus. Going back to the example, to address the purpose of program improvement, you need to understand the other programs that exist in your community, region, and state and how well you serve current clients. The scan’s focus is two-fold: 1) learn about available recovery housing programs with mental health services and their strengths and weaknesses; 2) learn how well your program supports recovery for clients.

Step 2: Identify Research Questions

Narrow your topics until you have several research questions. Without specific questions, you can find yourself down rabbit holes and waste valuable time. Going back to the example, research questions might include:

• What recovery housing programs with mental health services exist in the community and region?
• What are the strengths of these programs (including yours)?
  • Who is the target population? What outreach methods are effectively used?
  • What programming and services are offered?
  • What treatment options are used, including evidence-based practices?
• What are the limitations or gaps of these programs (including yours)?
  • Is there a segment of the target population with need that is un- or under-served?
  • What programming, services, and treatments are not offered, particularly best practices?
  • According to partners and stakeholders, what are the unmet needs of the target population?
• What are your program’s sobriety rates?
  • What is the percentage of clients who remain in the program for more than 30 days?  
  • What is the percentage of current clients who successfully graduate from the program, having fully completed the program components? 
  • What is the percentage of graduates who remain sober six months after successfully graduating from the program? 
• What do clients, employees, partners, and stakeholders believe are your program’s strengths and weaknesses?
  • According to clients and former clients, what are the strengths and weaknesses?
  • According to employees, what are the strengths and weaknesses?
  • According to stakeholders, what are the strengths and weaknesses?
  • According to partners, what are the strengths and weaknesses?

Step 3: Identify Methods and Collect Data

The methods that you choose will to some degree be dependent on the amount of time available. Surveys that you create and interviews take more time, which needs to be accounted for in your data collection timeline. Also, the amount of time required for you to analyze and interpret the data will increase as you use more data collection methods. Looking back at our example, the methods could include:

• Internet search for a list of recovery housing program and their target population, programs, services, and treatment options
• Survey or phone interview with other programs willing to participate
• Survey or interviews with key stakeholders
• Survey, focus group, or interviews with partners
• Survey or focus group with current and former clients
• Employee survey
• Existing organizational data regarding your program’s sobriety rates

Know what information you want to collect before reaching out to stakeholders, other programs, or partners. Otherwise, you may face engagement fatigue, which occurs when respondents encounter too many questions or those that are redundant or insincere. Only ask for information that you need and that you were unable to answer using other methods. With stakeholders, you might consider incorporating Four Simple Questions: (1) Who cares about __ and why? (2) What work is already underway and by whom? (3) What shared work could unite us? and (4) How can we deepen our connection?

Step 4: Store, Analyze, and Interpret the Data

It is important to store data systematically and link them to your questions. You can create a spreadsheet to house and organize your data. If you don’t know how to create this type of tool, stay tuned for a future blog post on that topic. Keep in mind that data storage and data presentation are two separate functions, so your spreadsheet will look messy and difficult to decipher. The spreadsheet should not have merged cells, colors, subheadings, or more than one piece of data in each cell (for example, gender and age), or more than one type of data in each cell (letters and numbers) so that you can use Excel’s built-in features to analyze and present the data. We will address how to analyze quantitative and qualitative data in future blog posts in this series.

Step 5: Present the Results and Findings

As you plan the presentation of the results (quantitative data) and findings (qualitative data), consider your audience(s) and how each will use the information. Remember to include the stakeholders who participated as you think about the audience. You may write a report or create a slideshow that uses graphs, charts, maps, or infographics. Use this information to inform strategic planning and ongoing decision-making processes.

Subscribe for the next post in this series about how to gather existing data! AGS offers several on-demand webinars on a variety of topics to support the full grant cycle, in addition to an Evaluation Training Series! Check out our website to learn more and sign up for our training newsletter.

If you are interested in grant services, training, or federal review services, or are interested in our career opportunities,  Julie Assel, CGMS, GPC, President/CEO will be happy to talk with you about this opportunity and provide you with a quote for grant services.

GPCI Competencies

AGS blogs, funding alerts, and trainings are aligned with the Grant Professional Certification Institute’s Competencies and Skills.

• Competency #4: Knowledge of how to craft, construct, and submit an effective grant application. Skill 4.11: Identify evaluation models and components appropriate to grant applications.
• Competency #8: Knowledge of methods and strategies that cultivate and maintain relationships between fund-seeking and recipient organizations and funders. Skill 8.4: Identify methods for collaborative efforts among the grant manager, program manager, and support staff during funder site visits and site evaluations.
• Competency #5: Knowledge of post-award grant management practices sufficient to inform effective grant design and development. Skill 5.4: Identify methods of establishing transitions to post-award implementation that fulfill project applications (e.g., document transfer, accuracy in post-award fiscal and activity reporting).