16 Jun You CAN Evaluate Your Program! How to Gather Data through Observation By: Michelle Dykes-Anderson, Ed. D, GPC, CFRE

Observation is a method to gather data by watching events or behaviors that can give information beyond what you can draw from numbers and is helpful in several situations:

• To collect data that is unavailable through other methods. People are sometimes unable or unwilling to participate in surveys or interviews.
• To understand an ongoing situation or process. For example, you want to identify efficiencies/inefficiencies in the process of college registration process as students meet with advisors to create a semester schedule.
• To know more about a physical setting. For example, you want to determine if a residential rehabilitation center’s facilities are conducive to recovery.
• To understand more about interactions. For example, you want to determine if a motivational guest speaker sparks interest in at-risk youth in a college preparatory program.

Types of Observation:

There are many types of observation, each with its own pros and cons.

Overt and Covert Observation

In overt observations, everyone is aware of being observed, while in covert observation, no one is aware of being observed. The observer may use a hidden camera or pretend to be part of the group. Covert observation is sometimes used because people tend to alter their behavior if they know that they are observed. However, this can lead to ethical considerations like a need for informed consent and invasion of privacy.

Participant and Non-Participant Observation

In participant observation, the observer blends in with the people being observed. For example, the observer receives food assistance and observes how all participants are treated as they wait for and receive services. In non-participant observation, the observer is separate from the participants. For example, the observer stands to the side and observes how participants are treated as they receive food assistance services. A benefit of participant observation is the observer can gather deeper, more complex data and more fully understand the situation and its dynamics. However, non-participant observation allows the observer to use notetaking, recorders, or other tools. A combination of participant and non-participant observation is most effective.

Direct and Indirect Observation

In direct observation, the observer watches the event, activity, or interaction as it occurs. For example, an observer watches an occupational therapist utilize an evidence-based practice to see if it is used with fidelity. In indirect observation, the observer watches the results of the event, activity, or interaction. For example, occupational therapists observe the amount of food left on the plates of children with sensory processing disorder to determine if an activity with a goal of trying a new food was effective. Direct observation offers real-time information but does not gather data about anything outside of the observation period. As a result, the observer may miss important information. Direct observation can be at risk of bias because people tend to alter their behavior if they know they are being observed. Indirect observation is not prone to bias, but only offers limited information. Again, a combination of direct and indirect observation is most effective.

Simple and Behavioral Observation

In simple observation, the observer notes the number of times that a phenomenon occurs without explaining why. For example, an observer may note the number of elementary school students referred for dental procedures during a dental screening outreach program. In behavioral observation, the observer interprets behavior or phenomenon. For example, the observer may note the number of students who don’t know how to floss their teeth and infer that as a cause. Behavioral observation can be open to bias. Participant and non-participant observation are most effective when used together.

Collecting and Interpreting Observational Data:

1. Identify one or more focus areas. For example, you want to know how well an evidence-based practice is implemented by mental health practitioners. Your focus areas might include practitioners’ knowledge/skills and interactions between practitioners and their clients.
2. Select a data collection system. There are three main systems, which can be combined: field notes, observation guides, and recording sheets/checklists. Field notes allow full open-ended narrative, either written or recorded. Observation guides list behaviors, interactions, and/or processes that you will observe as headings, leaving space for open-ended narrative. Finally, recording sheets/checklists have preset characteristics, behaviors, and interactions, with space to select those that occur during the observation. Recording sheets/checklists will not be appropriate for all observations. They may be a good fit to determine if a practice is implemented with fidelity, for example.
3. Identify the observation locations. Choose the number of sites that will allow you to generalize to your overall population and understand the phenomenon.
4. Schedule the observation. Time your observation to include the sequence of the phenomenon that you want to study.
5. Document findings. Fully describe the setting, individuals present, and the general mood. Consider if your presence influenced any behaviors. Include all three data types:

• Descriptive Data: Write down what you see and hear. You might write down that you see five college students waiting to see the same advisor in the hallway. The students check their watches, fidget, and sigh periodically. The average wait time is more than 45 minutes. One student left.
• Inferential Data: Interpret the motives behind what you see and hear. You infer that the students are frustrated at having to wait.
• Evaluative Data: Interpret the motives behind what you see/hear, then make a judgment. You determine that advising sessions are more effective by scheduling appointments.

Benefits and Limitations of Observation:

While each type has its own set of pros and cons, it is important to note the benefits and limitations of observation in general.

Benefits

• You can collect data as an activity or event occurs.
• You can include nonverbal communication in your data.
• You can test a hypothesis in a real-world setting.
• If the topic is sensitive, and people are uncomfortable answering questions or will likely give you biased answers, you can still gather data. For example, if you want to know instructor’s opinions of women in STEM fields, you can observe their interactions in the classroom.
• Observational data is often more reliable than self-reported data.
• Observation has strong validity.

Limitations

• Observation can be time consuming and expensive, if it takes the observer away from other duties.
• To get reliable data, you should repeat the observation several times before people behave normally if they know that they are being observed. Unobtrusive observation becomes more difficult over time because people will start to notice you.
• If the people being observed have unique characteristics, the data lacks external validity and cannot be generalized to the overall population. To generalize, you would need to observe several different groups. Often, the observer has no control over who happens to be present that day, and it is difficult to know what unique characteristics of those being observed.
• The observer’s opinions can subconsciously alter the findings.
• Some phenomena cannot be observed, such as child abuse or sexual behaviors.

Subscribe so you don’t miss the next post in this series about how to create surveys! AGS offers several on-demand webinars on a variety of topics to support the full grant cycle, in addition to an Evaluation Training Series! Check out our website to learn more and sign up for our training newsletter.

If you are interested in grant services, training, or federal review services, or are interested in our career opportunities,  Julie Assel, CGMS, GPC, President/CEO will be happy to talk with you about this opportunity and provide you with a quote for grant services.

GPCI Competencies

AGS blogs, funding alerts, and trainings are aligned with the Grant Professional Certification Institute’s Competencies and Skills.

• Competency #4: Knowledge of how to craft, construct, and submit an effective grant application. Skill 4.11: Identify evaluation models and components appropriate to grant applications.
• Competency #8: Knowledge of methods and strategies that cultivate and maintain relationships between fund-seeking and recipient organizations and funders. Skill 8.4: Identify methods for collaborative efforts among the grant manager, program manager, and support staff during funder site visits and site evaluations.
• Competency #5: Knowledge of post-award grant management practices sufficient to inform effective grant design and development. Skill 5.4: Identify methods of establishing transitions to post-award implementation that fulfill project applications (e.g., document transfer, accuracy in post-award fiscal and activity reporting).