CDC Program Evaluation Framework Action Guide — page 1

CDC’s

Program

Evaluation

Framework

Action Guide

Office of Policy, Performance,

and Evaluation

CDC Program Evaluation Framework Action Guide — page 2

CDC’s Program Evaluation Framework Action Guide

Centers for Disease Control and Prevention. CDC’s Program Evaluation Framework Action

Guide. Atlanta, GA: Centers for Disease Control and Prevention; 2024

Please contact

cdceval@

cdc.gov

for more information

or with questions about this resource.

CDC Program Evaluation Framework Action Guide — page 3

Table of Contents

CDC’s Program Evaluation Framework Action Guide............................................................... 1

Overview of the Program Evaluation Framework Action Guide ............................................ 4

Topics Covered ...................................................................................................................................... 4

How to Use the Action Guide .................................................................................................................. 5

What is Program Evaluation? .............................................................................................. 6

Types of Program Evaluations ............................................................................................ 7

How is Program Evaluation Different from Research, Surveillance, and Monitoring? ........... 9

Program Evaluation Compared with Research ....................................................................................... 9

Program Evaluation Compared with Surveillance ................................................................................... 9

Program Evaluation Compared with Performance Measurement ........................................................... 9

Cross-Cutting Actions ...................................................................................................... 10

Evaluation Standards ....................................................................................................... 11

Understanding Approaches to and Practical Application(s) of the CDC Evaluation

Framework ...................................................................................................................... 12

Step 1: Assess Context .................................................................................................... 13

Key Takeaways for Step 1 ..................................................................................................................... 13

Overview and Importance .................................................................................................................... 13

Implementing Step 1 with Illustrative Examples ................................................................................... 14

Applying the Cross-Cutting Actions and Evaluation Standards ............................................................. 20

Step 2: Describe the Program ........................................................................................... 21

Key Takeaways for Step 2 ..................................................................................................................... 21

Overview and Importance .................................................................................................................... 21

Implementing Step 2 with Illustrative Examples ................................................................................... 21

Developing the Narrative ...................................................................................................................... 24

Applying the Cross-Cutting Actions and Evaluation Standards ............................................................. 25

Step 3: Focusing the Evaluation Questions and Design ..................................................... 26

Key Takeaways for Step 3 ..................................................................................................................... 26

Overview and Importance .................................................................................................................... 26

Implementing Step 3 with Illustrative Examples ................................................................................... 26

Applying the Cross-Cutting Actions and Evaluation Standards ............................................................. 32

Step 4: Gather Credible Evidence ..................................................................................... 33

Key Takeaways for Step 4 ..................................................................................................................... 33

Overview and Importance .................................................................................................................... 33

Implementing Step 4 with Illustrative Examples ................................................................................... 33

Applying the Cross-Cutting Actions and Evaluation Standards ............................................................. 39

Step 5: Generate and Support Conclusions ...................................................................... 40

Key Takeaways for Step 5 ..................................................................................................................... 40

Overview and Importance .................................................................................................................... 40

Implementing Step 5 with Illustrative Examples ................................................................................... 41

Applying the Cross-Cutting Actions and Evaluation Standards to Step 5 .............................................. 47

Step 6: Act on Findings ..................................................................................................... 48

CDC Program Evaluation Framework Action Guide — page 4

Key Takeaways for Step 6 ..................................................................................................................... 48

Overview and Importance .................................................................................................................... 48

Implementing Step 6 with Illustrative Examples ................................................................................... 48

Applying the Cross-Cutting Actions and Evaluation Standards to Step 6 .............................................. 53

Toolkit Appendix: Worksheets .......................................................................................... 54

Step 1: Assess Context ......................................................................................................................... 54

Step 2: Describe the Problem ............................................................................................................... 62

Step 3: Focusing the Evaluation Question and Design .......................................................................... 67

Step 4: Gather Credible Evidence ......................................................................................................... 70

Step 5: Generate and Support Conclusions .......................................................................................... 74

Step

Act on Findings ...................................

.......................................................................................... 74

References ...................................................................................................................... 79

CDC Program Evaluation Framework Action Guide — page 5

Overview of the Program Evaluation Framework Action Guide

The

Program Evaluation Framework Action Guide

serves as a comprehensive resource to assist new or

early career evaluators with applying the CDC Program Evaluation Framework across a range of

programs and settings. This Action Guide acts as a complement to the

CDC Program Evaluation

Framework, 2024

and serves as a how-to guide for planning, designing, and implementing evaluations in

a practical way.

A key theme across effective evaluation planning and implementation is the integration of the

cross-

cutting actions

and

evaluation standards

across all steps of the framework. To ensure that evidence is

actionable for decision-making and supports continuous program improvement, the cross-cutting

actions and evaluation standards are designed to improve the quality of evaluations by guiding decisions

throughout each step of this process.

Figure 1:

CDC Program Evaluation Framework

Topics Covered

The following foundational objectives are covered in this Action Guide:

•

Defining program evaluation, its importance, and the types of program evaluation

•

Distinguishing program evaluation from other related areas such as surveillance, research, and

program monitoring

•

Defining

cross-cutting actions

and

evaluation standards

•

Understanding how to apply evaluation standards and integrate cross-cutting actions from the

CDC Program Evaluation Framework into your program evaluation approaches

•

Identifying the approach and practical application(s) of the evaluation framework

CDC Program Evaluation Framework Action Guide — page 6

How to Use the Action Guide

This guide is a starting point for planning and implementing program evaluations. It provides practical

applications of the

CDC Program Evaluation Framework

to any program evaluation. Although the CDC

Program Evaluation Framework is often discussed as a linear process, conducting program evaluation is

iterative. This Action Guide is intended to support your current evaluation need(s) and help you develop

robust program evaluations. The guide is organized by each step of the Framework and can be used as a

stand-alone resource.

The following features are included within each step to help with navigation:

Key takeaways

. Important tips to help you understand the main objectives of each step

Overview and importance

. Explanation of each step at a high level and discussion of the

benefits of implementing this step within the evaluation process

Implementation with illustrative examples

. Explanation of how to carry out each step and

make important decisions. Integration of a case example, CDC’s Colorectal Cancer Control

Program (CRCCP), is used throughout to illustrate the application of each step of the

Framework, as well as other examples and callout boxes to reiterate important concepts

Applying the cross-cutting actions and evaluation standards

. Explanation of how to integrate

these foundational principles throughout each step of the framework

Appendix:

The appendix includes fillable or downloadable

Worksheets

that can be adapted to

meet specific evaluation needs and skillsets

CDC Program Evaluation Framework Action Guide — page 7

What is Program Evaluation?

Program evaluation uses systematic data collection and analysis to assess the effectiveness and

efficiency of one or more programs, policies, and organizations (Kidder DP, Fierro LA, Luna E, et al.,

2024). Program evaluation can help clarify how to improve existing programs and build upon their

strengths, identify why a program is or is not being implemented as planned or producing intended

results, and why certain trends or patterns are observed in existing data sources.

program

is defined as

any set of related activities undertaken to achieve an intended outcome

. In this

context,

program

is used to describe the object of evaluation, which could be any organized public

health action.

Program evaluation is one of the

ten essential functions of public health.

Figure 2

shows how evaluation

is part of a lifecycle of continuous program improvement providing the feedback loop to assess the

program to inform decision-making for action.

Figure 2:

CDC’s Performance Improvement Framework

Program evaluation helps to provide answers to important questions regarding:

Program implementation (Are program activities being completed as planned?)

Effectiveness (Is the program achieving what was intended?)

Attribution (Did the outcomes achieved happen because of the program?)

Contribution (Are factors that could contribute to outcomes identified?)

Efficiency (Is the program operating using the appropriate resources?)

CDC Program Evaluation Framework Action Guide — page 8

Figure 3

demonstrates how program evaluation can answer those questions through the methodical and

intentional engagement with interest holders that leads to a deep understanding of the program (i.e.

logic model), what will be evaluated, and how to evaluate it (i.e. evaluation plan). Program evaluation

produces findings that:

•

Translate evidence to recommendations for action

•

Demonstrate accountability to funders, policymakers, and participants of the program

•

Document progress and ensure optimal use of resources

•

Help inform decisions about areas for program improvement

Figure 3:

Detailed flow of the activities for planning and implementing an evaluation aligned

with the CDC Program Evaluation Framework steps and cross-cutting actions.

Types of Program Evaluations

There are many types of evaluations that can be used for different purposes.

•

Formative evaluation

is typically conducted to assess whether a program, policy, or

organizational approach is feasible, appropriate, and acceptable before it is fully implemented.

It can include process or outcome measures and focuses on learning and improvement (OMB,

2021).

•

Process/Implementation evaluation

assesses how well program implementation followed the

original plan. It often includes information on content, quality, quantity, and structure of what is

being assessed (Kidder DP, Fierro LA, Luna E, et al., 2024).

•

Outcome evaluation

measures how well a program, policy, or organization has achieved its

intended outcomes. It cannot determine what causes the specific outcomes (causality), only

whether they have been achieved (Kidder DP, Fierro LA, Luna E, et al., 2024).

CDC Program Evaluation Framework Action Guide — page 9

•

Impact evaluation

compares the outcomes of a program, policy, or organization to estimates of

what the outcomes would have been without it. It usually seeks to determine whether the

activities caused the observed outcomes (Kidder DP, Fierro LA, Luna E, et al., 2024).

•

Economic evaluation

examines programmatic effects relative to program costs. Common

approaches include cost analysis, cost-benefit analysis, and cost-effectiveness analysis (Kidder

DP, Fierro LA, Luna E, et al., 2024).

CDC Program Evaluation Framework Action Guide — page 10

How is Program Evaluation Different from Research, Surveillance, and

Monitoring?

While evaluation is often used interchangeably with these other ways of using systematic approaches to

answer questions, they each have their own distinct purpose.

Program Evaluation Compared with Research

Research and evaluation are scientific activities that use similar methods, though the purpose is

different. Research seeks to contribute to generalizable knowledge and test a hypothesis, whereas

evaluation seeks to continuously improve programs and produce findings and recommendations for

decision-making (Kidder DP, Fierro LA, Luna E, et al., 2024). Evaluation focuses on determining the value

or significance of a program and providing information to help the program improve.

Program Evaluation Compared with Surveillance

Surveillance is the ongoing, systematic collection, analysis, and interpretation of health-related data

(CDC, 2001). Surveillance data are often used as data sources for program activities. However,

surveillance data alone may be insufficient to appropriately answer certain types of evaluation

questions.

Program Evaluation Compared with Performance Measurement

Performance measurement is the ongoing monitoring and reporting of program accomplishments,

particularly progress toward pre-established goals (M-21-27, 2021). These data can be used to identify

increasing or decreasing performance that may warrant further investigation. Program evaluation helps

you to identify the reasons behind these changes and potential areas of improvement.

CDC Program Evaluation Framework Action Guide — page 11

Cross-Cutting Actions

Cross-cutting actions are the foundational principles that are woven into every step of an evaluation.

Evaluations that integrate them are likely to produce more rigorous evidence that is informed by many

perspectives and is meaningful, informative, timely, and actionable (Kidder DP, Fierro LA, Luna E, et al.,

2024). Applying these cross-cutting actions also generates insights that may further health equity.

The cross-cutting actions include the following:

Engage collaboratively.

Evaluators can facilitate co-ownership of the program evaluation with

interest holders. By doing so, the evaluator can increase the validity of evaluation findings and

improve the likelihood that results are used by interest holders (Kidder DP, Fierro LA, Luna E, et

al., 2024). Strong evaluations proactively engage people with a range of perspectives who may

typically be excluded. This collaboration starts at the beginning of the evaluation planning

process and continues into the implementation and interpretation phases.

Advance equity

Health equity

is defined as the state in which everyone has a fair and just

opportunity to obtain their highest levels of health (Burton DC, Cardo D, et al., 2024). Evaluation

can contribute to advancing equity and the elimination of health inequities in several ways:

•

By using collaborative and equitable evaluation approaches to create environments

where everyone is respected and heard.

•

By considering the effects of potential decisions, and their impact on the advancement

of equity, throughout each step of the framework and when applying the standards.

•

By proceeding with cultural responsiveness and recognition of shared experiences,

integrating the uniqueness of each context into the design and implementation of an

evaluation.

•

By conducting evaluations that uncover factors perpetuating health inequities.

Learn from and use insights

. Evaluators also serve as facilitators for continuous learning, use of

findings, and improvement through evaluation. Successful evaluators build relationships,

cultivate trust, and model the way for interest holders to see value and utility in evaluation

insights.

This Action Guide provides you more information on how to integrate these cross-cutting actions into

your program evaluation approaches in the sections below.

CDC Program Evaluation Framework Action Guide — page 12

Evaluation Standards

Evaluation standards describe what constitutes a high-quality evaluation by guiding decisions at each

step. These standards are intentionally broad to ensure flexibility to the unique circumstances and

context that apply to each evaluation. Each evaluation should balance these standards, along with other

factors, depending on the evaluation and the needs of the situation (M-20-12, 2020).

The US federal evaluation standards are:

Relevance and Utility

. Evaluations must address the information that is important to the

interest holders to be useful. The evaluation findings must be actionable and available in time

for use as well as presented in a way that is understandable, culturally responsive, and

informative for interest holders to act.

Rigor

. Evaluations must produce findings that interest holders can confidently rely upon while

also providing clear explanations of limitations. The rigor of an evaluation is highly dependent

on thoughtful planning and implementation of the underlying design and methods, as well as

how findings are interpreted and reported. Credible evaluations must be planned, implemented,

and interpreted by qualified evaluators in collaboration with interest holders. The most

appropriate design and methods to answer the evaluation questions must be used while

balancing the evaluation goals, scale, timeline, feasibility, and available resources.

Independence and Objectivity

. Evaluations must strive to be as independent and objective as

possible for interest holders, experts, and the public to accept their findings. The

implementation of evaluation activities must be appropriately insulated from political and other

undue influences that may affect their objectivity, impartiality, and professional judgement.

Evaluators must strive for objectivity in the planning and conducting of evaluations and in the

interpretation and dissemination of findings. Conflicts of interest, bias, and other partialities

must be avoided to enhance objectivity; evaluators can

regularly assess their potential biases

which may affect whom they choose to engage in an evaluation, what they pay the most

attention to, and what they may be overlooking as a result.

Transparency

. Evaluations must be transparent throughout the planning, implementation, and

reporting phases to enable accountability and help ensure that aspects of an evaluation are not

tailored to generate specific findings. Decisions regarding the evaluation's purpose and

objectives, the range of interest holders who will have access to findings, the design and

methods, and the timeline and strategy for releasing findings must be clearly documented

before conducting the evaluation. Once evaluations are complete, comprehensive reporting of

the findings should be released in a timely manner and provide sufficient detail so that others

can review, interpret, or reproduce the work.

Ethics

. Evaluations must be conducted to the highest ethical standards to maintain trust in the

process and products. Evaluations must be planned and implemented to safeguard the dignity,

rights, safety, and privacy of participants, interest holders and/or affected entities. Evaluators

must abide by current professional standards pertaining to treatment of participants.

Evaluations must be equitable, fair, and just, and must consider cultural and contextual factors

that could influence the findings or their use.

CDC Program Evaluation Framework Action Guide — page 13

Understanding Approaches to and Practical Application(s) of the CDC Evaluation

Framework

The CDC Evaluation Framework consist of six steps:

Assess Context

Describe the Program

Focus the Evaluation Questions and Design

Gather Credible Evidence

Generate and Support Conclusions

Act on Findings

These steps are guided by the integration of three

cross-cutting actions

and five

evaluation

standards

. The framework is a practical and iterative process for designing and implementing good

program evaluations. For more information on each step, as well as the associated evaluation

standards and cross-cutting actions, please refer to the

CDC Program Evaluation Framework, 2024

CDC Program Evaluation Framework Action Guide — page 14

Step 1: Assess Context

Key Takeaways for Step 1

•

Evaluations are influenced by the context in which the evaluation is situated.

•

Key products include: 1) Evaluability assessment that examines evaluation readiness, 2)

Relevant interest-holder mapping, and 3) Documentation of place-based context, evaluation

capacity assessment, and evaluator readiness assessment.

Overview and Importance

Understanding a program’s context sets the stage for meaningful, actionable, and culturally

responsive evaluation (Kidder DP, Fierro LA, Luna E, et al., 2024). This can include the various features

of an evaluation’s setting, such as location and environment, people and their cultural values,

historical circumstances, roles that power and privilege play, and other pertinent characteristics.

Tip: What Happens When Context

is not Considered in the Evaluation?

•

Only the evaluator or evaluation team’s culture, context, and values may be reflected in the

evaluation.

•

The wrong interest holders could be involved, and their priorities could be misinterpreted.

•

Program participants are not engaged in a way that is respectful and this could increase

harm.

•

The program is not accurately described.

•

Data are not collected in a way that is culturally appropriate and enhances multicultural

validity.

•

Reporting results in modes that are only accessible by the dominant culture or those in

positions of power.

Adapted from: Snow, B. (2016).

Context in Evaluation

. Dr. Beth Snow Blog. Retrieved from: Context in

Evaluation | Dr. Beth Snow (drbethsnow.com)

Four factors to consider when assessing context include:

Readiness for Evaluation

People

(interest holders),

Place

Evaluation Capacity

(as well as assessing the evaluator’s readiness and understanding how to apply

the

cross-cutting actions

and

evaluation standards

to this step)

The main products of step 1 include the following:

Evaluability assessment

to examine whether a program is ready to be evaluated

Relevant interest holder mapping

of the four main types of interest holders

Documentation

of place-based context and evaluation capacity assessment

Evaluator reflection

to assess your readiness to conduct evaluation

CDC Program Evaluation Framework Action Guide — page 15

Implementing Step 1 with Illustrative Examples

Readiness for Evaluation

Evaluability assessments are a type of pre-evaluation method used to determine which aspects of a

program are ready for evaluation.

Table 1.1

describes the components of an evaluability assessment. The results of the assessment will

help evaluators understand if a program is ready for evaluation, and if not, identify which components

to rework.

Use Worksheet 1A

to apply the evaluability assessment concepts.

Table 1.1

Common Components of an Evaluability Assessment and Example Strategies

Evaluability

Components

Description

Key Question(s) to Consider

Example Strategies

Program

Intent and

Logic Model

Determine program

objectives and

expectations and

depict the

relationships

between/among

inputs, activities, and

expected outcomes.

•

What is the program?

•

Is the program logic clear,

rational, and

understandable? Are

there any apparent gaps?

•

Do interest holders

understand their role(s)?

•

Review programmatic

materials for program goals.

•

Conduct literature reviews to

develop a foundation for the

Logic Model or Theory of

Change (Mayne, 2015; Chen,

2018; King, 2021).

•

Interview interest holders to

assess interest, buy-in, and

perception of the program’s

role.

Program

Plausibility

Determine if the

programmatic goals,

outcomes, and the

feasibility of

measuring progress

towards

programmatic goals

are clearly defined.

•

What would success look

like?

•

Are program expectations

realistic?

•

Observe the program in action

to compare the expected and

actual implementation.

•

Review goals and outcomes to

determine if they are realistic

and measurable.

Data

Accessibility

Determine the

likelihood that data

potentially needed for

the evaluation are

acquired feasibly

given resource and

time constraints.

•

Are you collecting data on

what you want to

achieve?

•

What data are available?

•

What new data are

feasible to collect for the

evaluation?

•

Identify available data,

collection tools, and

monitoring systems.

•

Identify the types of resources

needed to support data

collection.

Program

readiness

Determine if the

program needs to be

adjusted and/or if

additional resources

are needed. If no

adjustments are

needed, then you are

ready for evaluation.

•

What adjustments need to

be made to prepare the

program for evaluation?

•

Assess the program scope and

magnitude of resource needs.

•

Identify program training

needs for personnel.

•

Determine program

partnership needs.

•

Assess if program goals and

objectives are realistic or need

to be adjusted.

Sources:

CDC Program Evaluation Framework Action Guide — page 16

Armstead, T. Lee, R. (2013). Evaluability Assessment Guidance for DELTA FOCUS Recipients. Internal report (CDC).

Unpublished.

D’Ostie,R.L., Dagenais, C., Ridde, V. (2013). An evaluability assessment of a West Africa based non-governmental

organizations (NGO) progressive evaluation strategy.

Evaluation and Program Planning

, 36(1), 71-79.

Farmer, H. (2018). Conversations to have when designing a program: Fostering evaluative thinking.

Better Evaluation

Blog

. Retrieved from

Conversations to have when designing a program: Fostering evaluative thinking - Blog post on

Better Evaluation

People

Interest holders

are people or organizations who are invested in and may be affected by the evaluation.

Table 1.2

shows four types of interest holders to keep in mind when implementing Step 1.

Table 1.2:

Type of Interest Holders and Implementation

Considerations

People Who...

Potential Interest Holder

Implementation Consideration

are served or

affected by the

program

•

Past, current, and future

program participants

•

Employers or associates of

program participants

•

Local recipients of your

funds

•

Populations affected by the

problem

Individuals or groups who directly or indirectly

receive program services may be most interested in

aspects of the evaluation that are related to

improvements or modifications in program services.

plan or implement

the program

•

Local and national

professional organizations

•

State or local health

departments and health

commissioners

Individuals or groups who have a professional role in

the program may be most interested in how to

improve the process for implementing the

program’s services, and the outcomes that are a

result of the program.

might use the

evaluation findings

•

Program designers,

implementers, and

evaluators

•

Local government, state

legislators, and governors

•

Universities and

educational groups

Individuals or groups who have authority to make

decisions about the program and individuals and

groups who have a general interest in the results

because they design, implement, evaluate, or

advocate on behalf of the program being evaluated

or similar programs.

are skeptical about

the program

•

Past, current, and future

program participants,

employers or associates of

program participants, and

developers of similar,

complementary, or

competing programs

Individuals or groups who are opposed to the

program may be most interested in knowing if the

outcomes can be attributed to the program and if

there is a cost-benefit of the program,

Adapted from:

Bryson, J.M., Patton, M.Q., Bowman, R.A. Working with evaluation stakeholders: A rationale, step-wise approach and toolkit.

Evaluation and Program Planning

. 2011;34(1):1-12. doi:https://doi.org/10.1016/j.evalprogplan.2010.07.001

Each interest holder group may have different interests, needs, concerns, power, priorities, and

perspectives that may need to be understood to ensure relevance and use of evaluation findings (Bryson

JM, Patton MQ, Bowman RA, 2011). While there are many techniques evaluators can use to conduct

interest holder identification, worksheets 1B and 1C are two examples.

Use

Worksheet 1B

to identify

interest holder needs for each step, cross-cutting actions, and evaluation standards.

Use

Worksheet 1C

CDC Program Evaluation Framework Action Guide — page 17

identify interest holder’s stakes in the evaluation and the interest they seek to serve as well as

their desired level of involvement.

Tip: Collaborative Engagement and Interest Holders

In addition to identifying the interest holders and their possible needs, as it relates to the

evaluation, also consider how and when they will be engaged. Principles for engaging interest

holders can be rooted in Community-Based Participatory Research and include:

•

Using a participatory approach that is empowering to program participants (e.g.,

Participatory Learning and Action Tools).

•

Program participants and program evaluators having an equal contribution.

•

Using a co-learning process, where both the program participants and evaluators learn from

each other.

•

Using a systems development and local community capacity building model to enhance the

evaluation capacity for both the program evaluator and program participants.

•

Creating balance between evaluation and action.

Adapted from:

Wallerstein, N., Duran, B., Oetzel, J.G., Minkler, M. (Eds.). (2018). Community-based

participatory research for health: Advancing social and health equity (3

edition). Jossey-Bass.

ISBN-13:978-1119258858

Source:

Napier, A., Simister, N. (2017). Participatory Learning and Action. Intrac for Civil Society Retrieved from

Participatory-learning-and-action.pdf (intrac.org)

Case example: Interest Holder Identification for Colorectal Cancer Control Screening Program

(CRCCP)

An example of how and when to engage interest holders can be found in

Table 1.3

. This is an adapted

case example from the Division of Cancer Prevention and Control’s CRCCP evaluation plan for engaging

interest holders in the evaluation. The CRCCP funds recipients through a cooperative agreement to

partner with health systems and their primary care clinics to implement evidence-based interventions

within clinics to increase colorectal cancer (CRC) screening among priority populations and ultimately

reduce CRC incidence and mortality.

In the case example, the interest holders were engaged throughout evaluation planning,

implementation, and the dissemination of findings to ensure the following:

•

Interest holder needs were prioritized

•

Multiple perspectives for data collection and analysis procedures were shared and considered

•

Findings would be useful for program improvements and policy change

Table 1.3

, interest holders are identified as “internal” or “external” to the CDC or organization and

then categorized with an “X” according to when they were engaged in the evaluation.

CDC Program Evaluation Framework Action Guide — page 18

Table 1.3:

Interest Holder Engagement Matrix

Internal Interest Holders

(Federal)

Describe

the

Program

Focus the

Evaluation

Questions

and Design

Collect

Credible

Evidence

Justify

Conclusions

Act on

Findings

U.S. Federal Agencies

Division of Cancer

Prevention and Control

Leadership

Division of Caner

Prevention and Control

Program Consultants

External Interest Holders

Describe

the

Program

Focus the

Evaluation

Questions

and Design

Collect

Credible

Evidence

Justify

Conclusions

Act of

Findings

CRCCP Recipient

Programs

Information Management

Services, Inc. (IMS) Data

Coordinator

National Association of

Chronic Disease Directors

(NACDD)

National Partners

General Public

Adapted from:

Division of

Cancer Prevention and Control Program. (2020). “CRCPP Evaluation Stakeholder

engagement” table in the CDC Evaluation Plan: DP20-2002 Cancer Control Program. Internal Report (CDC):

unpublished

Place

It is important to consider the place-based context in which the program and evaluation are conducted.

This includes recognizing the program and community history, and power dynamics. Existing systems in

a community affect how evaluators engage with interest holders, design the evaluation, and

communicate findings, especially in communities that are marginalized (Kidder DP, Fierro LA, Luna E, et

al., 2024).

There are two place-based contexts that are important to consider in Step 1:

Program Features

and

Program Environment.

As you think about place-based context, use the questions in

Table 1.4

as a starting point for

understanding programmatic features and the programmatic environment.

See

Worksheet 1D

for

additional guidance.

CDC Program Evaluation Framework Action Guide — page 19

Table 1.4.

Key Questions to Consider for Identifying Place in Context

Programmatic Features

Programmatic Environment

What is the program, and why was it

developed?

What is the history of the community, program,

organization and of the evaluation within it?

What are the historical, economic, health, and social

dimensions of the communities?

Who funds the program?

What are the strengths of this context? Are there any

conditions or circumstances that are problematic? What are

the ongoing political, social, or economic conditions that

might influence the program?

Who is involved in the program development

and implementation?

How is power distributed among persons who interact with

or influence the program or who might be engaged in the

evaluation (e.g. funders, planners, implementers)?

What is the program user's commitment to

the program?

How does the organization typically learn (e.g., about what

they are doing well, areas for improvement) and what is the

general level of receptiveness to learning from mistakes?

What are the demographics of the user

group, including income, education, gender,

race or ethnicity, and other identities?

What are the spoken or unspoken rules about evaluation at

the organization?

Who has authority and decision-making

power? What do decision-making processes

tend to look like?

What are the spoken or unspoken rules about identifying

and using data for action at the organization? How does the

organizational mission support or oppose evaluation?

Evaluation Capacity

Evaluation capacity is the program’s existing capacity to “do” and “use” evaluation. Specifically,

Individual

and

Organizational

are two types to consider

Culture

is a cumulative body of learned and shared behavior, values, customs, and beliefs common to a

particular group or society (Kirkhart, 2010; American Evaluation Association, 2011). Evaluation can be

part of organizational culture. Organizations that have a strong evaluation culture seek out information

about their programs and use that information to improve them. As you think about evaluation culture

and capacity, use the questions in

Table 1.5

as a starting point, then

use

Worksheet 1E

to apply the

concepts

Table 1.5.

Key Questions for Understanding Individual and Organizational Evaluation Capacity

Individual Evaluation Capacity

Organizational Evaluation Capacity

How knowledgeable is the individual, or evaluation

team, about the different evaluation approaches and

methods?

What financial resources are available for the

evaluation?

How much experience does the individual or

evaluation team have with developing evaluation tools

and templates for use by the organizations?

What human resources (e.g., time and

internal/external evaluation staff) are available for

the evaluation?

How much experience does the individual or

evaluation team have with analytical and facilitation

skills?

What is the organizational culture with respect to

evaluation and using evaluation findings?

What is the individual’s attitude toward evaluation?

What mechanisms already exist to share products

from the evaluation with others in the organization

who could benefit?

CDC Program Evaluation Framework Action Guide — page 20

Individual Evaluation Capacity

Organizational Evaluation Capacity

Does the individual view the evaluation as important

and valuable?

Are there opportunities within the organization to

reflect on insights that arise throughout the course

of the evaluation?

Adapted from:

Buetti, D., Bourgeois, I., & Jafary, M. (2023). Examining the competencies required by evaluation capacity

builders in community-based organizations.

Evaluation and Program Planning

, 97, 102242.

Tip: Learn from and Use Insights about Evaluation Capacity

As you think through the existing evaluation capacity of the individual and/or organization, also

think about how the findings from the evaluation will be used. Do the evaluators have knowledge

and experience to develop the evaluation recommendations and influence use of findings? In the

past, how has the organization used evaluation findings?

Evaluator Readiness

The lens that evaluators use in the evaluation is a direct reflection of their personal experiences and

context. These experiences may reflect an individual’s privilege and/or unintentional bias and may lead

the evaluator to make decisions that perpetuate health inequities (Miranda-Hartsuff, et al., 2024). To

help prevent this, evaluators may reflect on how their own personal context influences their evaluation

practices.

Use

Worksheet 1F

as a starting point to inventory and assess your own cultural dimensions.

Reflective practice is a tool that can be used to better understand unintended consequences of the

evaluators and their role in influencing the evaluation.

Table 1.6

lists several tools that evaluators can use for self-reflection. The information from these self-

reflections can help evaluators understand how their own culture might affect what they ask, how they

ask it, what they perceive, and whether they might be inadvertently favoring certain voices over others

in this evaluation context. Then, they can consider how to adjust their thoughts and practices and use a

collaborative approach to include multiple perspectives and contexts to potentially decrease biases in

the evaluation.

Table 1.6.

Tools for the Evaluator to Engage in Reflective Practice

Tool

Description

Self as an Instrument

Portfolio

Set of self-calibration questions for identifying a person’s own strengths, gifts,

constraints, etc. that is brought to a particular research setting.

Johari Window as a Skill-

Building Resource

Communication model that can be used to cultivate multilateral self-

awareness through a four-paned window metaphor.

Integral Evaluation Model

Framework for mindfully scanning and monitoring the human-system

dynamics through the individual and collective vantage points and across the

subjective, behavioral, intersubjective/cultural, and social systems

environments.

Source:

Symonette, H. (2009). Cultivating self as responsive instrument: Working the boundaries and borderlands for ethical

border crossing

. Handbook of Social Research Ethics

. (pp.279-294).

https://doi.org/10.4135/9781483348971.n18

Keep notes to document your thoughts and reflections that emerge during the evaluation. Use

Worksheet 1G

to engage in reflective practice.

CDC Program Evaluation Framework Action Guide — page 21

Tip: Reflective Practice and Cultural Humility

The process of reflective practice is meant to highlight how culture and context emerge during the

evaluation process and can enhance the evaluator’s cultural awareness and humility.

Applying the Cross-Cutting Actions and Evaluation Standards

There are several key questions that an evaluator can use to make sure that they have successfully

integrated the

cross-cutting actions

and applied the

evaluation standards

within Step 1 of the

framework.

Refer to Table 2

in the

CDC Program Evaluation Framework, 2024

for how to consider

applying these actions and standards to assessing context.

CDC Program Evaluation Framework Action Guide — page 22

Step 2: Describe the Program

Key Takeaways for Step 2

•

Logic models are a useful tool for communicating and understanding your program to show the

connection between a program’s activities and intended outcomes.

•

Key products of this step are 1) A one-page graphic depiction of the program to be evaluated

(e.g., logic model, program roadmap), and 2) An accompanying narrative of the program.

Overview and Importance

In Step 2, describe the program to be evaluated by identifying the intended outcomes and the key

activities expected to lead to those outcomes. Providing a description of the program that is clear and

concise with enough detail can help to facilitate an understanding of the program roadmap. This step is

vital because it lays the foundation for the rest of the evaluation. Investing time and effort to accurately

describe the program will help facilitate success as you further plan and implement the evaluation.

There are two key parts to describing the program (Kidder DP, Fierro LA, Luna E, et al., 2024):

logic model

program roadmap

, which is a 1-page graphic depiction of the relationship

between a program’s activities and its intended outcomes, typically including arrows showing

the connections between/among each of these. The most common elements are:

•

Activities

•

Short-term, intermediate, and long-term outcomes

narrative description

providing more detail on the program, which typically includes the

following:

•

Need

•

Inputs

•

Activities

•

Outcomes

•

Contextual factors

•

Stage of development

Tip:

Rely on insights about program context from

Step 1

when developing your program description.

Implementing Step 2 with Illustrative Examples

Developing a Logic Model

A logic model helps visualize the connection between the program activities and the changes that are

intended to result from them.

Tip

: Engaging interest holders in the process of describing the program allows for greater

understanding and agreement of the program activities and outcomes.

Your program’s logic model can be useful for your interest holders and broader program. Engage your

interest holders in developing your logic model and program description from the start.

CDC Program Evaluation Framework Action Guide — page 23

Logic models differ widely in format and level of detail; however, the core elements in logic models

include:

•

Activities

: strategies the program implements to effect change and achieve intended outcomes.

•

Outcomes

: who or what is expected to change due to the program’s efforts. Outcomes are

logically sequenced to show the temporal sequence, with shorter-term (often more specific)

outcomes leading to intermediate then longer-term outcomes.

Additional terms you may see in logic models are

•

Inputs

: resources needed for program activities such as personnel, partners, materials, funding,

equipment, data/surveillance, and the existing evidence-base.

•

Contextual factors

: factors outside the program which may affect the ability to achieve the

desired outcomes.

Outputs are the products of program activities. As an indicator of the activities, outputs fit best in the

narrative section that describes the evaluation measures.

There are several possible approaches to developing a logic model. A good place to start is by identifying

activities and outcomes that may be contained in the program descriptions, mission/vision statements,

or program planning documents. You can either start with the activities and ask “so what” to generate

the outcomes or start with the desired outcomes and ask “how to” achieve them through program

activities. See

Figure 4

for a logic model example from CDC’s Data Modernization Initiative.

Figure 4:

Example

Logic Model for the CDC Data Modernization Initiative

CDC Program Evaluation Framework Action Guide — page 24

Similar Terms for Logic Model

The logic model term is commonly used in evaluation, though there are other related terms you may see

•

Program Roadmap

•

Theory of Change (Mayne, 2015; Chen, 2018; King, 2021)

•

Theory of Cause

•

Theory of Action

•

Concept/Conceptual Maps

•

Outcome Maps

•

Logical Frameworks (LogFrames) (Bamberger, 2020)

Logic Model Examples and Visualization Techniques

Case example: Logic Model for Colorectal Cancer Control Screening Program

(CRCCP)

You can see an example of a logic model for a public health program in

Figure 5

, which uses the

concepts we have highlighted in

Worksheet 2A

in the Appendix. The CRCCP is a program aimed at

increasing colorectal cancer (CRC) screening in primary care clinics (CDC, 2020). The need for the CRCCP

program centers around CRC as a leading cause of death among cancers in the United States. While this

is not stated explicitly in the visualization (logic model) it is reflected in the long-term outcomes. The

context is positioned underneath as foundational components of the program (and logic model) as a

whole.

You can follow the inputs for the program, left to right, through each of the components of the logic

model. You could “read” one portion of the CRCCP logic model as:

•

the program has all of their indicated inputs including CDC funding, the recipient

organizations, clinics, evidence-based interventions, data, skills, tools, etc.,

then

they will be

able to establish partnerships with primary care clinics (activity).

•

they establish partnerships with primary care clinics,

then

they will be able to increase CRC

screening in partner clinics (short-term outcome).

•

CRC screening in partner clinics is increased,

then

they will be able to increase the number of

cancers prevented (intermediate outcome).

You would continue reading across the logic model in the same fashion.

Depending on how you (and possibly your interest holders) design the logic model, it may read

differently. For example, if the CRCCP logic model did not include inputs, it would read as, “If

partnerships are established, clinic assessments conducted, EBIs implemented, patients’ linkage to

colonoscopies facilitated, and data quality, program monitoring, and evaluation are conducted, then

clinic-level CRC screening rates will increase. If CRC screening rates increase, then the number of

cancers prevented, and early-stage CRC diagnoses will increase. If those increase, then it will result in a

reduction of CRC incidence and mortality.”

CDC Program Evaluation Framework Action Guide — page 25

Figure 5:

Example logic model adapted from Colorectal Cancer Control Screening Program (CRCCP)

Apply communications best practices when developing your logic model:

•

Visualize relationships between and among groups using arrows

•

Use space and color to signal clear groupings of logic model components

•

Be careful when putting text on colored spaces

•

Fit the logic model on a single page (not including the narrative)

Note how the

Logic Model

from the

Laboratory Leadership Service (LLS) Fellowship

program applies a

number of these practices to make a somewhat complex logic model easier for the reader. It also

provides text along the components to facilitate reading the model from left to right.

A template for developing logic models is provided in

Template 2A

along with

Worksheet 2B

Key

Components of a Logic Model

to help you identify the program components.

Developing the Narrative

The program narrative explains the components of the logic model in more detail and includes the

following:

Need

: The issue, challenge, or opportunity the program is intending to solve or is contributing to

solving.

Inputs

: Resources needed for conducting program activities such as personnel, partners,

materials, funding, equipment, data (such as surveillance data) and the existing evidence-base.

Activities

: Identifies what the program is implementing to effect change and achieve intended

outcomes.

Outcomes

: Identifies who or what is expected to change due to the program’s efforts.

CDC Program Evaluation Framework Action Guide — page 26

Contextual factors:

Factors outside of the program that might affect achieving the desired

outcomes.

Stage of development:

The evolution of the program, as all programs change from planning

stages to maturity.

Understanding contextual factors and stage of development are essential when conducting an

evaluation. The sections below provide more detail on these important components.

Identifying and Describing the Contextual Factors

Programs operate in the real world, which is a complex environment. As such, incorporating how

external elements may facilitate or hinder your program’s ability to achieve its goals into your program

description or logic model, helps account for and interpret findings accurately. Identifying and describing

contextual factors that will influence whether your program is able to implement the activity and

achieve the outcomes is helpful for transferring learnings from the evaluation to another context.

During this process, draw on insights from your efforts assessing context in Step 1. Understanding the

relationship of the context to the program functions and products allows you to understand potential

impacts on the program’s success.

Identifying and Describing the Stage of Development

A program’s stage of development is a factor in determining the type of evaluation to be conducted

(process, outcome). Align your evaluation purpose and questions with the program’s stage of

development (see Step 3: Focus the Evaluation Questions and Design). The most common stages of

program development include:

Planning:

The program being evaluated is early in its development and its activities have likely not been

implemented or tested yet. Some piloting may/ be or have been occurring.

Implementation:

The program and its activities are already being implemented at the time of

evaluation.

Maintenance:

The program is more mature, having been implemented for an extended period. The

program or the component being evaluated has produced some if not most of the intended likely

outcomes.

Applying the Cross-Cutting Actions and Evaluation Standards

As with all the evaluation framework steps, consider how to apply the

cross-cutting actions

and the

evaluation standards

when developing your logic model and evaluation to Step 2.

See Table 3

CDC

Program Evaluation Framework, 2024

for how to consider applying the cross-cutting actions and

standards to your program description.

Tip:

Once you feel you have completed a draft of your logic model, be sure to review it for critical

content, usability, and ease of understanding. Use

Worksheet 2D

Logic Model Review Checklist

as a

helpful guide in addition to the other materials in this toolkit section.

CDC Program Evaluation Framework Action Guide — page 27

Step 3: Focusing the Evaluation Questions and Design

Key Takeaways for Step 3

•

Collaboratively engage with interest holders to focus the evaluation efforts and develop the

most appropriate evaluation design.

•

The key products of this step include 1) Purpose statement explaining why the evaluation is

being performed, how the findings are likely to be used, and who is likely to learn from or use

the findings, 2) Statement about the type of evaluation that will be conducted (e.g., process,

outcome), 3) List of intended users and use of the evaluation findings, 4) List of evaluation

questions, and 5) Description of the overarching evaluation design.

Overview and Importance

Step 3 of the Evaluation Framework identifies what aspects of the program to evaluate. It uses the

priorities and information needs from interest holders identified in Step 1 and the logic model from Step

2 to focus the evaluation questions and design. Step 3 prioritizes what is important to understand and

identifies a culturally responsive evaluation design that accommodates the program context and

resources while also incorporating relevant

evaluation standards

The main products of this step are the following:

Statement identifying the evaluation’s purpose, how the findings are likely to be used, and who

is likely to learn from or use the findings.

Statement about the type of evaluation that will be conducted (e.g., process, outcome).

List of intended users and use of the evaluation findings.

List of evaluation questions.

Description of the overarching evaluation design that will answer the evaluation questions.

Implementing Step 3 with Illustrative Examples

Develop Evaluation Purpose Statement

Before conducting an evaluation, identifying its main goal or purpose sets the tone for how the

evaluation will be conducted. The evaluation purpose statement can be influenced by the following:

•

Intended users of the evaluation: those who have a specific interest in the evaluation and a clear

use in mind.

•

Intended uses of the evaluation: the ways in which the intended users plan to use the

knowledge and findings gained from the evaluation.

•

Program components: contextual information of the program to assess evaluation readiness.

•

Feasibility to conduct the evaluation: the resources available to conduct the evaluation.

Table 3.1.

Key questions to consider when identifying the purpose

Criterion

Question

Intended users of the

evaluation

•

Who will use and/or act on the evaluation findings? This includes funders, program

implementers, community members, etc.

Intended uses of the

evaluation findings

•

What information is important to interest holders?

o

What do they want to gain from the evaluation?

o

When are the evaluation findings needed?

CDC Program Evaluation Framework Action Guide — page 28

Criterion

Question

Program components

•

What is the stage of program development (i.e. planning phase, implementation

phase, maintenance phase)?

•

What is the overall context?

•

Which components of the program are best suited for evaluation?

Feasibility to conduct the

evaluation

•

What resources (i.e. time, money, effort) are needed for the evaluation?

(feasibility)

o

What financial resources are available?

o

How much staff time will be needed to invest in the evaluation?

o

How intense is the program?

Use

Worksheet 3A

to apply the concepts outlined in Table 3.1.

Case example: Evaluation Purpose for Colorectal Cancer Control Screening Program

(CRCCP)

The CRCCP funds recipients through a cooperative agreement to partner with health systems and

their primary care clinics to implement evidence-based interventions within clinics to increase

colorectal cancer (CRC) screening among priority populations and ultimately reduce CRC incidence and

mortality. The CRCCP’s evaluation team plans to conduct a five-year multi-level process and outcome

evaluation across all recipient programs. The purpose of their evaluation includes the following:

Improving recipient programs

Strengthening CDC’s accountability to the public and Congress and recipients’ accountability to

CDC

Informing future programmatic planning and decision-making

For additional examples of evaluation purposes and associated uses,

see Table 4

in the

CDC Program

Evaluation Framework, 2024

Determine Type of Evaluation to Conduct

The evaluation purpose provides clarity on the type of evaluation to conduct. The most common types

of evaluations include formative, process/implementation, outcome, impact and economic.

Table 3.2

compares the evaluation types and when to use them.

Table 3.2.

Comparison of Evaluation Types

Evaluation Type

What it Shows

When to Use

Why it is Useful

Examples

Formative

Evaluation

Whether the

proposed program,

policy or

organizational

approach is

feasible,

appropriate, and

acceptable to the

population of reach

before it is fully

implemented

During the

development of a new

program, policy, or

organizational

approach

When an existing

program is being

modified or used in a

new setting

When you want to

focus on the who,

Allows for

modifications

before full

implementation

begins

Assessing if the program

can be implemented as

planned

Assessing if the program

will be accepted by the

community

CDC Program Evaluation Framework Action Guide — page 29

Evaluation Type

What it Shows

When to Use

Why it is Useful

Examples

what, when, where,

and how

Process Evaluation

The extent to which

the program,

intervention,

regulation, operation,

or policy is being

implemented as

planned

Beginning of program

or policy

implementation

During operations of

existing program or

policy

Provides early

indications of gaps or

issues that can be

addressed early in the

process

Allows for programs

to monitor activity

efficiency

Assessing reach of activities

to priority populations

Assessing resource

mobilization/allocation

Assessing threshold level of

participation or exposure to

intervention

Assessing access to and/or

quality of services

Assessing process for

implementing program,

policy, intervention, or

organizational approach

Outcome

Evaluation

The degree to which

the program, policy,

or organization has

achieved its intended

outcomes or results

It cannot determine

what caused specific

outcomes (causality),

only whether they

have been achieved

After an intervention,

policy or program has

been implemented

with the priority

population

Identifies whether

the program or policy

is achieving its goals

Assessing change in

knowledge, attitudes, and

behaviors in priority

populations

Assessing change in

policies, regulations, or

social norms in priority

populations

Assessing change in

incidence, mortality, and

morbidity

Economic

Evaluation: (cost

analysis, cost-

benefit analysis,

cost-effectiveness

analysis, cost-

utility analysis)

Examines program

effects relative to the

costs of the program

At the beginning of a

program or policy

implementation

During operations of

an existing program

At the end of

program or policy

intervention

Helps understand the

cost of implementing

a program and can

assess program

effects relative to the

cost to produce them

Assessing if the value of the

program’s outcomes

exceeds the cost

Impact Evaluation

Compares the

outcomes of a

program, policy, or

organization to

estimates of what the

outcomes would have

been without it.

Usually seeks to

determine whether

activities caused the

observed outcomes

At the end of a

program

During operation of a

program at

appropriate intervals

Provides evidence for

use in policy and

funding decisions

Assessing the extent to

which the outcomes can be

related to the program as

opposed to other external

factors (i.e.

attribution/causality)

Adapted from

CDC Program Evaluation Framework Action Guide — page 30

Salabarria-Pena, Y. et al

(2007).

Practical Use of Program Evaluation among Sexually Transmitted Disease (STD)

Programs, Atlanta (GA): Centers for Disease Control and Prevention.

Kidder DP, Fierro L, Luna E, et al. CDC Program Evaluation Framework, 2024.

MMWR Recomm Rep.

2024;73(No. RR-

6):1-37. doi:

http://dx.doi.org/10.15585/mmwr.rr7306a1

. Retrieved f

https://www.cdc.gov/mmwr/volumes/73/rr/rr7306a1.htm?s_cid=rr7306a1_w

Case Example: Evaluation Focus Area for Colorectal Cancer Control Screening Program (CRCCP)

Table 3.3

provides a case example of the CRCCP’s focus area for the evaluation and the type of

evaluation to be conducted. The CRCCP’s evaluation team plans to conduct a five-year multi-level

process and outcome evaluation of recipient’s programs. Based on their logic model (

Figure 5

), the

evaluation team is focused on understanding program implementation and identifying a change in clinic-

level CRC screening rates over time.

Table 3.3

Case Example of CRCCP’s Focus Area and Evaluation Type

Evaluation Focus

Process

Outcome

Partnerships with health systems and primary care clinics

Implementation of evidence-based interventions within partner clinics

Increase in clinic-level colorectal cancer screening rates

Reduction in colorectal cancer incidence and mortality

Develop Evaluation Questions

Evaluation questions identify the aspects of the program that will be investigated (Wingate L, Schroeter

D, 2007). These questions are usually broad in scope, open-ended, and either process or outcome

focused. They help to define the boundaries of the evaluation that align with its purpose, the intended

uses and information needs, and interest holder priorities.

Tip:

Evaluation questions

ARE NOT

the same as data collection questions (i.e. surveys, key informant

interviews, etc.) which are usually specific, direct, and meant to collect a single data point.

Case Example: Evaluation Questions for Colorectal Cancer Control Screening Program (CRCCP)

CRCCP’s evaluation questions are based on the main program activities and outcomes that align with the

program logic model and evaluation purpose. See

Table 3.4

for a case example of CRCCP’s alignment of

evaluation questions with the evaluation focus and type of evaluation question.

Table 3.4

CRCCP’s Evaluation Focus, Evaluation Questions and Type of Evaluation Question

Evaluation Focus

Evaluation Questions

Process

Outcome

Partnerships with health

systems and clinics

What are the characteristics of the recipient’s partner health

system and clinics?

What is the annual and overall reach of CRCCP?

Are recipients partnering with clinics serving populations of need

with low colorectal cancer (CRC) screening rates for

completeness?

Implement Evidence-

Based Interventions (EBI)

What EBIs are recipients implementing in the clinics?

How are EBIs implemented in the clinics?

CDC Program Evaluation Framework Action Guide — page 31

Evaluation Focus

Evaluation Questions

Process

Outcome

What are the costs and cost effectiveness of EBIs being

implemented by CRCCP recipients?

Increased CRC Screening

To what extent are screenings completed among patients who

receive a screening referral?

To what extent have clinic screening rates and the number of

CRC screenings changed over time?

Reduction in CRC

Incidence and Mortality

What is the long-term impact of the CRCCP on lives saved?

For additional examples of evaluation questions for process and outcome evaluations,

see Table 5

in the

CDC Program Evaluation Framework, 2024

Developing and prioritizing evaluation questions can be influenced by several factors (CDC Asthma

Control Program, 2013)

•

Interest holder priorities and when information is needed.

•

Appropriate fit within the program’s description, goals, and stage of development.

•

Relevance with evaluation purpose and providing information useful to interest holders.

•

Resource availability

in time, funding, and staffing.

•

Timeline for conducting the evaluation.

•

Availability of similar insights from prior evaluations or other evidence activities.

•

Provided insights for advancing health equity.

In addition to developing and prioritizing the evaluation questions, assessing the effectiveness and

appropriateness of the questions will help capture the most important information. There are

certain characteristics that make a good evaluation question as highlighted in

Table 3.5

Evaluation Question

Characteristic

Definition

Table 3.5:

Characteristics of good evaluation questions

Evaluative

Provides determination on merit, worth or significance and can directly inform decisions about

the program.

Pertinent

Clearly related to the purpose and intended user’s information needs.

Reasonable

Linked to what a program can practically and realistically achieve based on scope, stage of

development, resources.

Specific

Clearly identify what will be investigated during the evaluation.

Answerable

Questions reflect type and quantity of data that can be feasibly collected, analyzed, and

interpreted.

Complete

Thoroughly addresses the purpose of the evaluation, intended users’ information needs, and

purposefully selected from a range of topics appropriate for the evaluation.

Adapted

from Wingate, L., Schroeter, D. (2007).

Evaluation questions checklist for program evaluation

Use

Worksheet 3B

to assess evaluation questions that combines key factors and evaluation question

characteristics.

CDC Program Evaluation Framework Action Guide — page 32

Determine Evaluation Design

The evaluation design is the overarching structure for the evaluation (Kidder DP, Fierro LA, Luna E, et al.,

2024). Selecting the appropriate evaluation design is guided by the evaluation purpose, key evaluation

questions, and evaluation context, including available resources and constraints.

Figure 1:

Schematic of how to identify an appropriate evaluation design (Kidder DP, Fierro LA, Luna E, et

al., 2024)

The schematic above demonstrates that interest holder engagement influences the evaluation purpose,

key evaluation questions and the understanding of the

evaluation context.

Collaboratively engaging

interest holders contributes to the credibility of the evaluation and ensures that the selected design will

be able to be answer the key evaluation questions with scientific integrity within the real-world

constraints.

Tip:

Select evaluation questions that can be answered in a timely manner. Engage collaboratively

with all interest holders (i.e. funders, implementers, community members, etc.) to develop

questions that will answer their specific information needs. This will ensure all perspectives are

represented in the evaluation questions.

Tip:

Implement the evaluation design in a manner that upholds the evaluation standards to the

greatest extent possible. The choice of design has implications for what will count as evidence, how

that evidence will be gathered, and what claims can be made [CDC Program Evaluation Framework,

2024].

The three types of scientific designs commonly used in evaluation are experimental, quasi-experimental,

and observational as outlined in

Table 3.6

CDC Program Evaluation Framework Action Guide — page 33

Research Design

Example

Definition

Strengths

Limitations

Table 3.6:

Types of Evaluation Designs

Experimental

Randomized

controlled trial

(RCT)

Units of study are

randomly assigned to a

control group or

a treatment/experimental

group (Shackman, 2020)

May demonstrate

causal relationship

between activities and

outcomes

Considered a rigorous

approach that may

answer questions of

causality

Not always

practical due to

costs

May not be

ethical to

randomly assign

people if the

program or

intervention

involves

treatment or

another service

Quasi-

experimental

Time series

Compare between non-

equivalent groups that are

not randomly assigned

(Pell Institute Evaluation

Toolkit 2024).

Enables

experimentation

when random

assignment isn’t

possible or ethical

Additional

analyses are

needed to control

for extraneous

factors that can

influence findings.

Observational

Case studies

or cross-

sectional

Does not use comparison

or control groups.

(University of Washington

Training Evaluation Tools,

2024)

Simple design and

may require least

resources to conduct.

Inability to infer

causality

Applying the Cross-Cutting Actions and Evaluation Standards

As with the other steps in the evaluation framework, integrating

the cross-cutting actions

throughout

each step will increase the likelihood that the evaluation is credible, equitable, and findings are used. In

addition, applying the

evaluation standards

will help to improve the evaluation quality and use by

reducing options at each step to make the evaluation manageable (Office of Management and Budget,

2020).

See table 6

in the

CDC Program Evaluation Framework, 2024

for key questions for integrating the

evaluation standards and cross-cutting actions for Step 3.

CDC Program Evaluation Framework Action Guide — page 34

Step 4: Gather Credible Evidence

Key Takeaways for Step 4

•

Determine the evidence needed to answer the evaluation questions, including what data will be

collected, how, when, and from whom (or what).

•

A key product of this step is a data collection strategy defining expectations for credible

evidence, methods that will be used, indicators and associated measures of interest, and data

sources.

Overview and Importance

Step 4 of the Evaluation Framework builds on the evaluation design from Step 3 to determine the

evidence needed to answer the evaluation questions. During this step, evaluators will develop a data

collection strategy, which involves the selection of data sources and associated measures that align with

the evaluation purpose and questions.

Implementing Step 4 with Illustrative Examples

Establish Expectations

Evaluators will want to collaborate with interest holders and set expectations up front that are realistic

and will produce the evidence needed to address their information needs.

Examples of Engaging with Interest Holders During this Step

•

Collaboratively establish and manage expectations about the evidence that is needed to

answer the evaluation questions, and the type and level of results that will be used to

answer those evaluation questions.

•

Identify indicators that are relevant and align with the evaluation purpose and questions.

Interest holders can validate indicators or suggest alternatives.

•

Discuss expectations of what constitutes success or a positive finding and the rationale for

such expectations.

•

Identify what data collection methods are practical, what data sources are reliable, and

other expectations about the quality, type, and quantity of data needed to build credible

evidence.

Choose Appropriate Methods

An evaluator can choose quantitative (numeric) or qualitative (narrative) data collection methods. Both

quantitative and qualitative methods have benefits and limitations to their use.

Quantitative data methods

rely on numerical data to draw conclusions and compare results. However,

these methods are less able to capture unknowns about a particular topic or context and have less

flexibility to be adaptive.

Qualitative data methods

rely on words to gather deeper insights on topics or concepts that are not

well understood. They are also useful when an understanding/study of evaluation participants’

experience is needed to provide context on a topic. However, qualitative data are unable to offer

comparability.

CDC Program Evaluation Framework Action Guide — page 35

Tip:

Engage with interest holders on the most appropriate methods that will ensure your evaluation

gathers credible evidence.

Choosing the appropriate data collection method may be influenced by several factors (Mwita, 2022):

•

Resource availability (e.g., time, cost, staffing)

•

Ethical considerations

•

Size and scope of the evaluation

•

Validity (e.g., how well the evaluation measures what’s intended)

•

Reliability (e.g., ability to replicate findings)

In determining the type of data to collect, evaluators and interest holders can explore data that is

already available that may be able to answer the evaluation questions. Available data may include

surveillance data, census data, or other datasets.

In some cases, an evaluator may use a mixed methods approach, purposefully integrating both

qualitative and quantitative methods. Using a mixed methods approach can improve the accuracy of

results by providing deeper insights and understanding on the “why” behind concepts, topics, or context

more comprehensively (Kidder DP, Fierro LA, Luna E, et al., 2024). Ensure the data collection method

selected to be used is culturally responsive to produce trustworthy and accurate data. See

Table 4.1

for

a description of common data collection methods and some scenarios on when to use them and

use

Worksheet 4A

to assess which data collection methods and data sources are best to use in your

evaluation.

Tip:

Engage interest holders to develop culturally responsive data collection methods.

Table 4.1:

Data Collection Methods

Data Collection

Method

Type

Description

When To Use

Survey or

Questionnaire

Quantitative

Collection of data by gathering

standardized information from all

respondents. Typically involves

various sampling methods and

techniques. Surveys can be in

person or virtual through various

methods such as interviews, phone,

mail-in, text, email, website, or

social media.

•

When you need to collect

information quickly.

•

When you want to monitor

changes or trends over

time.

•

When you want to identify

individual’s knowledge,

attitudes, or behaviors.

Interviews

Qualitative

Structured conversations to gather

information. Interviews can be

conducted individually or in group

settings, in person, telephone,

video, or groupf settings.

•

When you need rich,

detailed information.

•

When you need deeper

understanding of

individuals’ experiences,

motivations, or emotions

on a particular topic or

issue.

•

When you need deeper

understanding of the

context

CDC Program Evaluation Framework Action Guide — page 36

Data Collection

Method

Type

Description

When To Use

Observations

Qualitative

Conducted by watching or listening

to behaviors. There are several

considerations for conducting

observation, including ethics,

sampling methods, and objectivity.

•

When you need to explore

a new topic.

•

When you need to

understand individuals’

behavior or context as it

occurs naturally.

•

When you need to discover

silent norms and values.

Focus Groups

Qualitative

Interactive discussion involving

small groups of participants (often

around 6-8) on specific topics under

the guidance of a trained

moderator.

•

When you need to explore

a new or specific topic of

which little is known.

•

When you need to gain a

range of views about issues

in a single data collection.

Case Study

Qualitative

In-depth exploration of individuals,

groups, events, programs, sites, or

locations. Relies on various sources

of information such as interviews,

observations, documents, etc.

•

When you want to gain

concrete, in-depth

knowledge about a specific

topic.

•

When conducting extensive

research is not feasible

Document or

Record Reviews

Qualitative

Assessing existing documents to

gain historical understanding of the

program, participants, etc.

•

When you need to gather

background information on

the program.

•

When you need answers to

specific evaluation

questions related to

“what”.

Source: Hennink M, Hutter I, Bailey A. Qualitative Research Methods. SAGE Publications. 2011

Identify Relevant Indicators

Indicators differ from measures and metrics. They:

•

Are measurable statements that measure a construct or concept (e.g., knowledge of health risks

of smoking or awareness of smoking campaign)

•

Help evaluators understand whether interventions or programs have achieved progress

•

Help programs identify areas for improvement and make decisions based on credible evidence

•

Help ensure that the evaluation is credible and effective

•

Align with the evaluation focus area and evaluation questions

The program roadmap or logic model contains the activities and outcomes, which can be helpful for

guiding indicator development as well as the expectations established with the interest holders. These

include

activity indicators

, which measure the qualities of program implementation efforts and

associated

output indicators

that measure products of the program activities, as well as

outcome

indicators

measuring whether the program effects were achieved. Similarly, indicators will have an

associated measure/metric that directly aligns to it and quantifies it. Use

Worksheets 4B

and

assess selecting relevant indicators.

Developing Indicators

CDC Program Evaluation Framework Action Guide — page 37

•

Identify and define the expected changes or outcomes that the program is hoping to achieve.

•

Choose indicators that are directly related to the evaluation purpose and questions. Clearly outline

how the outcome or activity will be measured to make sure that the program constructs align with

the measures.

•

Use

Specific, Measurable, Achievable, Relevant, Timebound, Inclusive and Equity (SMARTIE)

criteria

to develop the indicator and ensure there is sufficient specificity and detail to facilitate accurate

data collection (CDC, 2024).

•

Align the indicators with the evaluation questions to ensure there are appropriate measures to

answer the questions.

Determine Appropriate Data Sources, Data Quantity, and Data Quality

When selecting data sources, consider the relevance of the source to the purpose of the evaluation.

Primary data

are new data specifically collected for that evaluation, and

secondary data

are existing,

previously collected data. Using multiple sources of data enhances the credibility of the evaluation by

incorporating varying perspectives.

Data quantity refers to the amount of data that needs to be collected to answer the evaluation

questions. Data quality refers to the appropriateness and integrity of data used in the evaluation

(Kidder DP, Fierro LA, Luna E, et al., 2024). In considering data quantity and quality, evaluators may:

•

Assess the amount of data needed to answer the evaluation questions sufficiently (i.e. what

information is important to know based on Step 3)

•

Balance the amount of data to be collected with avoiding burden on the respondents or those

involved in data collection

•

Ensure that the sample size is sufficient to do a detailed analysis

Data collection protocols that are thorough and clear with well-defined indicators will improve the

likelihood that the evaluation will collect quality data. Additionally, the data collection instrument

design, procedures, trainings, source selection, coding, and error checking are other factors that also

affect high quality data.

Data collection plans may need to be updated or modified during an evaluation, and documenting these

data collection changes will be a useful resource when reviewing, analyzing, and interpreting data. See

Table 4.2

for considerations when checking for data quality.

Table 4.2:

Assessment of Data Quality

Criteria

Questions

Accuracy

•

Are there errors or inaccuracies?

•

Are the data collection methods reliable?

•

How were the data validated?

•

Were there any limitations or biases?

•

How did you minimize inaccuracies?

Completeness

•

Are there missing data or gaps in the data?

•

Were the procedures for data collection designed in a way that made sure the information

captured is complete?

Consistency

•

Do the data align with previous records related to the focus and are there inconsistencies?

•

Were protocols for data collection consistently followed across all sources?

•

Were there sufficient data checks?

•

If there were discrepancies, were the data resolved or explained to justify the discrepancy?

CDC Program Evaluation Framework Action Guide — page 38

Criteria

Questions

Timeliness

•

How often are the data collected and updated?

•

Were the data collected according to the timeline established in the evaluation plan?

•

Were there delays in data collection? How were the delays addressed?

Relevance

•

Do the data relate back to the objectives/goals of the evaluation?

•

Were there any adjustments made to data collection methods at any point during

collection? Did they enhance relevance to the objectives?

•

Were interest holders effectively engaged to ensure data are relevant for all parties?

Source: Kidder DP, Fierro L, Luna E, et al. CDC Program Evaluation Framework, 2024.

MMWR Recomm Rep.

2024;73(No.

RR-6):1-37. doi:

http://dx.doi.org/10.15585/mmwr.rr7306a1

. Retrieved from

https://www.cdc.gov/mmwr/volumes/73/rr/rr7306a1.htm?s_cid=rr7306a1_w

While data quality and quantity are important, also consider the cultural context and norms when

making decisions regarding gathering data. Evaluators ensure persons providing data know and

understand their rights, any risks, and how data will be handled, stored, and used. Evaluators also gather

data that is secure, confidential, and compliant with program standards. Working collaboratively with

interest holders on data collection and data security procedures will help align the procedures with

cultural norms and the project setting (Kidder DP, Fierro LA, Luna E, et al., 2024).

Develop Data Collection Instruments and Implement Data Collection

Data collection instruments lay the foundation for data and information gathering. This section explores

key steps in developing data collection instruments such as:

•

Defining the data needs

•

Determining if there are existing instruments that can be adapted to fit the evaluation context

or if there is a need to develop new instruments

•

Developing a protocol guide for the instrument use to ensure consistency and completeness of

data

•

Developing questions or prompts that align with the previously selected indicators

•

Developing the format of response options for the items (such as open-ended, multiple choice,

or ranking)

•

Piloting the data collection instruments with interest holders to make sure the questions are

clear

•

Revising the data collection instrument based on feedback from the pilot test

•

Finalizing the data collection instrument

•

Training the data collection team on how to use the instruments and any associated protocols

•

Developing a data management plan

•

Developing a quality assurance plan

•

Monitoring the data collection process

Once these steps are completed, the evaluator can dive into implementation and data collection. Use

Worksheets

, and

to create a data collection implementation plan to ensure the collection

of meaningful insights in a timely manner.

CDC Program Evaluation Framework Action Guide — page 39

Tip:

Simplicity is key! Use language that is clear and concise. Test the data collection instruments so

you can be sure any challenges are addressed early before full implementation.

Case example: Evaluation Plan Matrix for Colorectal Cancer Control Screening Program

(CRCCP)

Using an evaluation matrix brings all the components discussed above together in one table to make it

easier for the evaluator and interest holders to see how the evaluation questions, indicators, measures,

data collection sources, and methods align with the evaluation focus from Step 3.

Table 4.3

provides an

example using the case example of Colorectal Cancer Control Program (CRCCP)’s evaluation matrix. Also,

see table 7

in the MMWR.

Use Worksheet 4H

as a template for an evaluation matrix that can be adapted

to fit program needs.

Table 4.3:

CRCCP Evaluation Plan Matrix

Evaluation

Focus Area

(Step 3)

Evaluation

Questions

(Step 3)

Evaluati

on Type

(Step 3)

Indicator

(Step 4)

Measures

(Step 4)

Data

Collection

Source

(Step 4)

Data

Collection

Method

(Step 4)

Data

Collection

Timing

(Step 4)

Establishing

partnerships

with health

systems/clinics

What is the

annual and

overall reach

of CRCCP?

Process

Partnerships

established

with health

systems and

clinics

• # health systems recruited

• # clinics recruited

• # clinic patients aged 45-75

• # primary care providers within

clinics

• Geographic location of clinics

mapped with population density

overlay

Clinic data

U.S. Census

data

Survey

baseline

and

annually

Establishing

partnerships

with health

systems/clinics

Are recipients

partnering

with clinics

serving

populations

of need with

low colorectal

cancer (CRC)

screening

rates?

Process

Reach of

CRCCP among

clinics serving

priority

populations

• #/% clinics located in high burden

areas

• #/% of clinics located in counties

with high CRC incidence and death

rates

• #/% clinics that are Federally

Qualified Health Centers (FQHCs)

• #/% clinics with uninsured

populations >20%

• Avg. clinic baseline screening rate

Clinic data

US Cancer

Statistics

Small Area

Health

Insurance

Estimates

Social

Vulnerability

Index

Survey

baseline

and

annually

Evaluation

Focus Area

(Step 3)

Evaluation

Questions

(Step 3)

Evaluati

on Type

(Step 3)

Indicator

(Step 4)

Measures

(Step 4)

Data

Collection

Source

(Step 4)

Data

Collection

Method

(Step 4)

Data

Collection

Timing

(Step 4)

Implementing

evidence-

based

interventions

(EBIs)

How are EBIs

implemented

in the clinics?

Process

Implementa-

tion of EBIs

• Descriptions of EBI delivery

protocols

• Avg. # ways patient reminders

sent per clinic

• Avg. # ways provider reminders

sent per clinic

• #/% clinics reducing structural

barriers in more than one way

•

Avg. frequency of

provider assessment and

feedback per clinic

Interview

and focus

group

transcripts

Clinic data

Focus

groups

and

Interviews

Survey

Periodic

baseline

and

annually

Implementing

EBIs

What are the

costs and cost

effectiveness

of EBIs being

implemented

by CRCCP

recipients?

Process

Cost

effectiveness

of EBIs

• EBIs implemented

•

Costs of implementing EBIs

•

Screening prevalence changes

Cost data

Clinic data

Cost

collection

tool

Survey

Periodic

Adapted from:

Division of

Cancer Prevention and Control Program. (2020). “CRCPP Evaluation Stakeholder engagement” table

in the CDC Evaluation Plan: DP20-2002 Cancer Control Program. Internal Report (CDC): Unpublished

Applying the Cross-Cutting Actions and Evaluation Standards

There are several key questions that an evaluator can use to ensure successful integration of the

cross-

cutting actions

and application of the

evaluation standards

during Step 4 of the framework.

Use Table 8

in the

CDC Program Evaluation Framework, 2024

to determine if you have effectively applied the cross-

cutting actions and evaluation standards.

CDC Program Evaluation Framework Action Guide — page 41

Step 5: Generate and Support Conclusions

Key Takeaways for Step 5

•

Focuses on generating answers to the evaluation questions. These answers are presented as

evaluation conclusions that align with the questions in Step 3 and demonstrate how the

conclusion is supported by the data collected in Step 4.

•

The key products for this step are 1) Data analysis and interpretation plan, and 2)

Recommendations based on analytic findings.

Overview and Importance

Figure 2:

Key Steps for Implementing Step 5

Step 5 involves generating evaluation conclusions, which are answers to the evaluation questions

supported by the data collected in Step 4. This step involves the following:

•

Reviewing the evidence expectations identified previously

•

Planning data analysis

•

Conducting data analysis

•

Interpreting the analytic findings

•

Forming recommendations

Data analysis and interpretation in this step transform the raw data into meaningful and actionable

recommendations. As you begin Step 5, refer to the expectations of interest holders established in Step

4. Specifically, engage interest holders in:

•

Discussing the analysis plan (may include quantitative and qualitative analysis techniques)

•

Interpreting analyses

•

Drawing evaluative conclusions

•

Testing the feasibility of potential recommendations to ensure conclusions are responsive to

the context and underlying data

•

Clearly planning for how and when you will involve interest holders in each of these activities.

CDC Program Evaluation Framework Action Guide — page 42

Implementing Step 5 with Illustrative Examples

Review Expectations with Interest Holders

The expectation setting with interest holders that happened in Step 4 can be the basis for developing

the data analysis plan. Review what the interest holders stated regarding:

•

What evidence will be used to answer the evaluation questions

•

What expectations they have about the type, quality, and quantity of data needed

•

What indicators and measures are most important

•

What changes, trends, or patterns suggest the program is on track or doing well

Planning Data Analysis

A data analysis plan includes how and by whom the data would be organized, analyzed, and synthesized.

A plan that is feasible, appropriate, and aligns to the evaluation questions and data collection methods

identified in earlier steps is more likely to produce findings that will be used by the interest holders.

Evaluators will want to collaboratively engage interest holders to obtain agreement on the techniques

for analysis and interpreting findings before data collection begins to promote transparency.

Develop the data analysis plan early because it will outline the series of choices you will make. The plan

will also include the program’s context, external factors, and implementation changes.

Prepare your

data analysis plans before data collection begins to ensure the data collected can be analyzed and meets

the needs of interest holders, including program leadership. Involve interest holders in reviewing these

plans and the expectations for success established in Step 4.

Tip: Questions to consider while planning for evaluation data analysis include the following:

•

Who will do the analysis?

•

What method(s) will you use to analyze the data? (for example, descriptive statistics,

inferential statistics, content analysis, thematic analysis)

•

What skills and expertise are required for the analysis methods selected?

•

Do those responsible for the analysis have the necessary skills and expertise? If not, how

could they obtain these skills?

•

How will the results of the analysis be validated?

Table 5.1

is a sample template to document which data will be analyzed, how, when, and by whom.

Align the data analysis plan to the evaluation questions, indicators, and data collection methods

identified in Steps 3 and 4. To prepare for analysis, you can prepare templates that specify the data

expected for each analysis and prepare a codebook for qualitative data. See

Worksheet 5A

in the

Appendix for a corresponding worksheet to complete for your evaluation.

CDC Program Evaluation Framework Action Guide — page 43

Table 5.1:

Sample data analysis plan

Evaluation

Question

Data to be Analyzed

Analysis to be

Performed

Person(s) Responsible

Due Date

Conducting Data Analysis

As you begin the data analysis process, follow the data analysis plan, and document your progress along

the way. Engage interest holders throughout the data analysis process and incorporate their

perspectives. Analyze the data within the program's context, considering external factors and

implementation challenges which may affect interpretation of the findings.

CDC Program Evaluation Framework Action Guide — page 44

Your evaluation may include quantitative and qualitative data sources, in which case you will conduct

quantitative and qualitative analyses. Common steps for quantitative and qualitative analysis are

described below.

Common steps for quantitative data analysis

▪

Clean and pre-process raw data for accuracy and consistency. Check for missing data,

standardize data formats, and make any other corrections needed.

▪

Import data into a statistical software program such as SAS, STATA, SPSS, R, or Excel.

▪

Analyze data based on the indicators defined in Step 4.

▪

Calculate descriptive statistics to describe, summarize, and compare key characteristics

about the data (totals, frequency counts, percentages). For some indicators, data will need

to be stratified or grouped based on variables of interest first.

▪

Explore distributions and variations within the data. Analyze trends, patterns, and

relationships within the data.

▪

Draw conclusions from the findings.

▪

Conduct advanced statistical analyses for evaluation questions looking to determine the

association between program activities and the intervention or desired outcomes (changes

in knowledge, attitudes, behavior, health status, or systems changes).

Common steps for qualitative data analysis

▪

Transcribe recordings from interviews or focus groups and/or enter narrative comments

from surveys or field notes into a word processing or qualitative analysis software such as

nVivo, Excel, MAXQDA, or ATLAS.ti.

▪

Review and annotate the data to understand and familiarize yourself with the content. This

could involve reading interview transcripts or listening to recordings.

▪

Organize the data: for instance, by date, by data collection type, or question asked.

▪

Code the text to identify and label key themes (trends or ideas that appear throughout) that

correspond to your evaluation questions.

▪

Group text by key themes.

▪

Review the themes and codes for refinement and to identify any sub-themes that emerge.

▪

Draw conclusions from the findings.

▪

More advanced qualitative analysis techniques include the use of multiple coders,

calculation of interrater reliability, and within-or between-case analysis.

Adapted from:

Centers for Disease Control and Prevention. Evaluation Brief 19: Analyzing Qualitative

Data for Evaluation. Evaluation Brief 20. Analyzing Quantitative Data for Evaluation.

CDC Program Evaluation Framework Action Guide — page 45

Case example: Evaluation Data Analysis Plan for Colorectal Cancer Control Screening Program

(CRCCP)

An example of a data analysis plan can be found in Table 5.2. The CRCCP evaluation team identified

multiple quantitative and qualitative analysis methods to analyze the primary and secondary data and

to answer their evaluation questions, and Table 5.2 shows the connection among the evaluation

questions, indicators, methods, and techniques.

Table 5.2: Data Analysis Plan for the CDC Colorectal Cancer Control Screening Program (CRCCP)

Evaluation

Questions

(Step 3)

Evaluation

Type

(Step 3)

Measures

(Step 4)

Data

Collection

Source

(Step 4)

Data

Collection

Method

and

Timing

(Step 4)

Data Analysis

(Step 5)

What is the

annual and

overall reach of

CRCCP?

Process

• # health systems recruited

• # clinics recruited

• # clinic patients aged 45-75

• # primary care providers within clinics

• Geographic location of clinics mapped with

population density overlay

Clinic data

U.S. Census

data

Survey, at

baseline and

annually

Quantitative

(frequencies,

percentages,

averages)

GIS mapping

How are EBIs

implemented in

the clinics?

Process

• Descriptions of EBI delivery protocols

• Avg. # ways patient reminders sent per

clinic

• Avg. # ways provider reminders sent

per clinic

• Avg. frequency of provider assessment

and feedback per clinic

• #/% clinics reducing structural barriers

in more than one way

Interview

and focus

group

transcripts

Clinic data

Focus groups

and

Interviews,

periodic

Survey, at

baseline and

annually

Qualitative (thematic

analysis)

Quantitative

(frequencies,

percentages,

averages)

What are the

costs and cost

effectiveness of

EBIs being

implemented

by CRCCP

recipients?

Process

•

EBIs implemented

•

Costs of implementing program activities

•

Screening prevalence changes

Cost data

Clinic data

Cost

collection

tool, periodic

Survey,

periodic

Quantitative (e.g.,

costs

per patient screened,

average cost per fecal

test completed,

incremental cost

effectiveness ratio)

What is the

long-term

impact of the

CRCCP on lives

saved?

Outcome

•

# cancers averted

•

# lives saved

•

# late-stage cancers averted

Clinic data

Patient-

level

screening

data

Natural

disease

history

model

One-time

special study

– survey,

disease

model

Simulation modeling

CDC Program Evaluation Framework Action Guide — page 46

Interpreting Analytic Findings

The interpretation process is where meaning is made from the analytic results. When reviewing results,

consider how the findings compare to the expectations for success outlined with interest holders in Step

4. Findings also can be reviewed alongside the existing evidence base or applicable scientific theories or

models.

A shared interpretation process allows evaluators and interest holders to collaborate and discuss the

strengths, limitations, and interpretations of the findings within context. Seek perspectives from various

interest holders, especially those directly impacted by the program and familiar with the program

context. The evaluator may have to balance interpreting findings with including interest holder

perspectives as this will influence the interpretation of findings. When different but equally well-

supported conclusions exist, each could be presented with a summary of its strengths and weaknesses.

The culmination of this step is generating evaluative conclusions.

Tip:

Involve "cultural interpreters" in the data interpretation process. These are interest holders who

are part of the data interpretation process to verify that data represents what was captured as well

as the context.

Source:

Hood, S., Hopson, R., Kirkhart, K. (2015). Culturally Responsive Evalua. on.

10.1002/9781119171386.ch12.

There are several ways to conduct a collaborative process of reviewing analytic results, translating

findings, and validating the interpretation, including:

•

Using data dashboards to share and generate discussion on interim findings

•

Establishing collaborative interpretation activities such as data walks

•

Hosting roundtable discussions

•

Facilitating data storytelling (for instance, by using narratives and visualizations to convey

insights from data)

•

Re-purposing project meetings to discuss findings throughout the evaluation

CDC Program Evaluation Framework Action Guide — page 47

Case Example of a Data Gallery Walk

Purpose and goals:

To collectively review data from a community health assessment of two

communities, examine differences, and generate insights and action steps.

Participants:

Youth co-investigators

Format of activity:

Participants rotate between 3 health topic stations to review display posters

showing bar charts of health indicators, maps, and quotations from the community health

assessment. An additional open-ended station will provide participants an opportunity to propose

additional community health needs. Use discussion questions in small group discussions about the

data.

Data point:

Percentage of adults 18 years and older reporting poor mental health in past 30 days in

each community.

Possible discussion questions:

What are some ways to interpret these data? What questions do

these data recall? What surprises you about these data? What patterns do you see in the data?

What gaps? What do you find interesting in these data? What assumptions might these data help to

test? What other data would you want to have a fuller picture of the current context? Whose

perspective is reflected in these data? Whose is missing?

Source:

Springer, et al., Exploring Models for Youth Engagement in Community Health Planning: The Youth-led

Community Health Learning Initiative

It is important to think through the best approaches for your evaluation’s unique interest holders, and

the format may vary between different groups of interest holders. At the end of the process, answers to

the following questions may be generated.

•

How do the findings compare to the expectations for success established among interest

holders prior to data collection?

•

Do any changes need to be made because of the evaluation findings?

See

Worksheet 5B

in the Appendix for a template to guide the planning of a collaborative discussion of

evaluation results (such as data walks, data gallery walks, data parties, data chats, and dashboard

reviews).

Forming Recommendations

After interpreting the analytic results of the evaluation and forming evaluative conclusions, the next

step is to outline recommendations for action. Ensure recommendations are clear and concise,

evidence-based, and grounded in the program context. Facilitating recommendation discussions with

interest holders allows you to gauge the feasibility of putting them into action. Prioritize and outline

which recommendations are more critical and feasible to address first, who will address them, on what

timeline, and provide multiple options for action when possible. Establish a process to document and

share recommendations with interest holders, as well as a process to follow up on progress. Use

Worksheet 5C

in the Appendix to plan and implement actionable recommendations based on your

evaluation results.

CDC Program Evaluation Framework Action Guide — page 48

Applying the Cross-Cutting Actions and Evaluation Standards to Step 5

As with the other steps, integration of the

cross-cutting actions

and the

evaluation standards

are

important to increase the likelihood the evaluation results are credible and produce actionable findings.

Use Table 9

in the

CDC Program Evaluation Framework, 2024

to assess integration of the cross-cutting

actions and evaluation standards for step 5.

CDC Program Evaluation Framework Action Guide — page 49

Step 6: Act on Findings

Key Takeaways for Step 6

•

Engage interest holders to translate evidence into actionable decision-making.

•

Key products of this step are 1) the anticipated use of the evaluation findings by interest

holders, 2) communication, reporting and dissemination strategy, and 3) relevant evaluation

product(s) on which the interest holders can act.

Overview and Importance

Figure 3:

Key Steps for Implementing Step 6

Facilitating the use of evaluation findings includes planning, collaboration, and commitment from the

evaluators and interest holders to act on these findings and recommendations. Since evaluation findings

do not translate into action automatically, this step is an essential element in the evaluation cycle.

The components of this step are:

Planning how the findings will be used

Preparing the findings for use

Facilitating the insights into actions

Implementing Step 6 with Illustrative Examples

Planning the Findings for Use

Planning for how the evaluation findings and recommendations will be used happens throughout the

evaluation process. For example, when evaluators collaborate with interest holders to develop the

evaluation design, questions, and indicators (Steps 3 and 4), they also consider the use of evaluation

findings. Questions to consider at each step of the evaluation process to prepare for use of the

evaluation findings include the following:

•

Who will use the findings?

•

What data, insights, or recommendations will be used?

•

When will they be used?

•

How will they be used?

Use

Worksheet 6A

to anticipate how evaluation findings will used. Consider which formats will resonate

the most with various audiences. For example, some interest holders may like the visual aspect and near

real-time results found in data dashboards, while others may prefer reading through a more detailed

CDC Program Evaluation Framework Action Guide — page 50

report that provides more information on methods and background. Regardless of the format type,

tailor communication products to the audience, use plain language, and be culturally relevant.

Tip: Strategies to encourage evaluation use

▪

Integrate evaluation at the beginning of the program and identify ways to promote learning and

use of evaluation findings throughout the program.

▪

Develop a strong relationship between the evaluation team and the program team through

consistent communication and collaboration.

▪

Facilitate trust with interest holders and program recipients through transparency, engagement,

and open communication.

▪

Identify leaders who will champion evaluation use.

▪

Cultivate an environment that promotes evaluation capacity building and a team culture

receptive to feedback.

Source: McWhorter, A. Case Study on Evaluation Use.

Preparing the Findings for Use

Planning ways to communicate evaluation findings and updates throughout each step of the evaluation

process helps to build trust between the evaluators and interest holders. This could involve planned

periodic discussions to share updates, seek feedback, or to review interim findings.

Existing structured approaches that encourage use of findings throughout the evaluation process

include the

Knowledge to Action

(Centers for Disease Control and Prevention, 2014) and

Data to Action

Frameworks

(Zakocs et al., 2015).

Table 6.1

lists steps and considerations for how to complete a rapid

feedback cycle, based on the Data-to-Action Framework.

Table 6.1:

Data-to-Action Rapid Feedback Cycle Considerations

Steps

Considerations

Clarify intent

Draft a short protocol that explains:

•

Purpose

•

Intended users

•

Questions to be answered

•

Low-cost data collection strategies

•

Staff roles

Collect “good

enough” data

Collect and analyze data quickly.

Produce brief

memo

Draft a brief memo that highlights the major findings with a narrative and visuals. Distribute the

memo after the data have been collected and analyzed.

Engage in reflective

debrief

Engage with intended users about the findings. Center the discussion around these three

questions:

•

What are we learning?

•

What are the implications for the project?

•

What actions are required?

Make decisions

Informed by the discussion in Step 4, make decisions to:

•

Gather more information before taking actions,

•

Take no action, or

•

Take action: make adjustments to the program

CDC Program Evaluation Framework Action Guide — page 51

Adapted from:

Zakocs R, Hill JA, Brown P, Wheaton J, Freire KE. The Data-to-Action Framework: A Rapid Program Improvement

Process. Health Educ Behav. 2015 Aug;42(4):471-9. doi: 10.1177/1090198115595010. PMID: 26245935; PMCID: PMC5990006.

Case Example: Colorectal Cancer Control Screening Program (CRCCP) Evaluation Uses

The CRCCP evaluation focused on the timely and meaningful use of evaluation findings to inform

continuous program improvements, maximize accountability to interest holders, and demonstrate

program effectiveness. Use of evaluation findings was a priority and varied by interest holders. In

Table

6.2

, the evaluation team anticipated potential uses of evaluation findings by interest holders with the

understanding that some interest holders may develop new uses for evaluation findings that help to

inform program policies, program implementation, resource allocation, and replication of promising

practices.

Table 6.2:

Case Example - Anticipating Evaluation Use by Interest Holder

Interest Holder

Use of Evaluation Findings

Federal agencies (e.g.

Department of Health and

Human Services and the U.S.

Congress)

•

To assess CRCCP reach to priority populations and primary outcome of interest-

changes in CRC screening rates.

•

Cost and Cost-effectiveness of CRCCP strategies and activities that may impact funding

allocation.

•

Success stories of individuals that may promote the benefits of the program

CDC leadership (Center for

Chronic Disease and Health

Promotion, Division of Cancer

Prevention and Control

Leadership)

•

Monitor recipient progress and performance for purposes of program improvement,

accountability and decision- making.

•

Recipients spend rates and return on investment study results to inform future funding

decisions and practices.

Division of Cancer Prevention

and Control program

managers

•

Inform technical assistance (TA) and guidance to funded recipients and monitor

recipient progress that may lead to improved recipient implementation of activities

and performance.

•

These interest holders will use dashboards populated with various program data (e.g.,

budget, partnerships, reach, EBI implementation, screening rates, follow up care) to

monitor progress and inform their TA. They will use this information to learn about

individual recipient challenges, successes, and TA needs.

CRCCP funded recipients

•

The evaluation team will provide recipients with information on program reach,

implementation activities, and program effectiveness.

•

CDC will work with recipients to conduct an annual clinic data review process to

examine data quality and program progress.

•

Information will be used to improve data quality, programs, and accountability.

National partners

•

The Evaluation Team will publish results of various analyses that will be of interest to

national partners (e.g., American Cancer Society, National Association of Community

Health Centers).

•

Interested in return-on-investment studies and studies of specific strategies identified

as promising practices that may influence broader replication in the field.

General public

•

Providing information on who was served and what was achieved to demonstrate

effective and efficient use of public dollars.

Adapted from:

Division of Cancer Prevention and Control Program. (2020). CRCCP Evaluation Plan: DP20-2002

Cancer Control Program. Internal CDC report: unpublished

Worksheet 6B

in the Appendix provides a tool to plan the dissemination of various information from

your evaluation with various audiences. Some considerations include which format will resonate most

with the audience, desired timing, and the best person to deliver the message.

CDC Program Evaluation Framework Action Guide — page 52

There are several considerations to keep in mind when deciding how to promote the results of the

evaluation.

Table 6.3

outlines common communication components to consider. If you are required to

develop a final report for the evaluation refer to the common components of an evaluation report in

Table 6.4

Tip: Ways to Share and Disseminate Findings

•

Dashboards

•

Data visualizations

•

Infographics

•

Social media

•

Briefs and one-pagers

•

Podcasts

•

Success stories

•

Newsletters

•

Brochures

•

Presentations

•

Posters

•

Evaluation report

Table 6.3:

Communication Components to Consider When Developing Messages to Promote Results

Communication

Components

Definition

Timing

The timeframe for when the message will be delivered.

Style

The type of communication style that will be used when delivering the message. Communication

styles can be mixed and must be tailored to each audience.

Tone

The tone of voice or tone of writing sets the mood for the message and how it will influence the

audience. Tone of voice includes the pitch, volume, speed, and body language that is used when

delivering a message verbally. Tone of writing is how the writer expresses their attitudes and

feelings through word choice and syntax.

Source

The individual who sends the message.

Format

The method that will be used to share the message with the audience. Examples are listed in Table

5 of the

CDC Program Evaluation Framework, 2024

Channel

The way that the message will travel between the sender and receiver. The channel can be either

spoken or written, depending on the message format that was chosen.

Context

The setting where the message will be delivered and the expectations of those who will receive

the message. For example, if you are delivering the message in a community center with interest

holders from the community, you must determine ahead of time their expectations like how to

dress, words to include/exclude, if a translator is needed, if the visuals are culturally appropriate,

etc.

CDC Program Evaluation Framework Action Guide — page 53

Table 6.4

. Common Components of an Evaluation Report

Component

Explanation

Title page

Title page should include the following:

•

Title that identifies the type of evaluation and what was evaluated

•

Date (month and year)

•

Authors listed with affiliations

Acknowledgements

No more than half a page with a list of contributors, their affiliation, and how they

contributed

Abstract or Executive Summary

Describe the evaluation questions, methods, key findings, and recommendations (200-

300 words)

Table of Contents

Table with the main report sections and page numbers for those sections

Evaluation Purpose and

Evaluation Questions

Overview of the program being evaluated (1 page), the evaluation purpose (1 page), and

the evaluation questions (1 page). In this section, you will also want to identify the

framework that was used to guide the evaluation (such as the CDC Evaluation

Framework) and how it was used.

Methods

This section must describe the evaluation methods and analysis that were used, the

rationale, and the strengths and weaknesses of those methods (1-3 pages).

Key Findings, Conclusions,

Interpretation, and

Recommendations

This section must include the key findings based on the data and aligned with the

evaluation questions (3-4 pages), the interpretation of those findings (1-3 pages), and

actions that are based on the findings and conclusions (1-3 pages).

References

Include any references cited in the text using the appropriate citation style for your

paper (such as AMA or APA) accordingly.

Appendix or Annex (if needed)

Determine if an appendix or annex are needed. Examples of what to include:

•

Detailed evaluation timeline with the roles listed for each evaluation team

member.

•

Detailed description of the evaluation tools that were used to collect data.

Facilitating Insights to Action

Evaluators serve as facilitators for interpreting the findings, uncovering, and applying insights, and

preventing misuse. Evaluators are also in the position to identify how to encourage interest holders to

use the evaluation findings and how to support the users after they receive the evaluation findings.

Possible questions that evaluators can consider are listed in

Table 6.5

See

Worksheet 6C

for a template.

Table 6.5:

Questions that Evaluator(s) can consider when facilitating a follow-up plan with interest

holders

•

What support do you think you need to understand and apply the findings to your work?

•

Following the review of the evaluation results, how much time would you need before you act on the findings and

recommendations?

•

Who will serve as the champion for the findings and recommendations and prevent misuse?

•

How much influence do you expect the evaluation findings to have?

•

How have past evaluations been used?

•

What types of data and findings are needed to support decision-making, and what types of data and findings are

viewed by leadership as credible?

•

Who needs to be involved for the evaluation to be influential?

•

How will the interest holders know afterwards if the evaluation was used as intended?

Adapted from:

International Development Research Center. (2012). Identifying the Intended User(s) and Use(s) of an

Evaluation. Retrieved from

idrc.pdf (betterevaluation.org)

Also refer to the Action Plan (

Worksheet 5C

) developed during Step 5 to generate discussion and plan

follow up on the evaluation findings and recommendations.

CDC Program Evaluation Framework Action Guide — page 54

Applying the Cross-Cutting Actions and Evaluation Standards to Step 6

Use Table 10

in the

CDC Program Evaluation Framework, 2024

for questions to consider when applying

and integrating the

cross-cutting actions

and

evaluation standards

to this step. Applying the evaluation

standards to Step 6 will increase the likelihood that findings are used throughout the evaluation process

and prevent misuse.

CDC Program Evaluation Framework Action Guide — page 55

Toolkit Appendix: Worksheets

Step 1: Assess Context

Worksheet 1A.

Evaluability Assessment

Evaluability

Component

Evaluability Statement

Not at all

true

Somewhat

true

True

Not

applicable

Program Intent

and Logic

Model

The program logic is clear,

rational, and understandable.

Program Intent

and Logic

Model

There are no apparent gaps.

Program Intent

and Logic

Model

There is shared understanding

among program leadership and

staff about core elements of the

program and the context in

which the program operates.

Program

Plausibility

Program expectations are

realistic.

Data

Accessibility

Existing data are valid,

reliable, and readily

available.

Data

Accessibility

New data can be feasibly

collected.

Data

Accessibility

There are systems in place to

track performance measures

Program

Readiness

There are sufficient resources

(personnel, funding, etc.) for

the evaluation.

Program

Readiness

The timing of the evaluation is

commensurate with the

timeframe for the program and

when implementation or

outcomes can be measured.

Directions

: Indicate to what extent each statement is true. Use this template as a starting point for

conducting evaluability assessments. The questions listed are not an exhaustive list but can be used as a

guide.

Adapted from

: Zandniapou, L, JBS International. 2014. “Impact Evaluability Assessment Tool”.

Washington, DC: Corporation for National Service.

CDC Program Evaluation Framework Action Guide — page 56

Worksheet 1B.

Identifying Interest Holder Priorities

Step 2

Step 3

Step 4

Step 5

Step 6

Interest Holder

What is most

important to

the interest

holder about

the program

and the

outcomes it is

expected to

produce?

What

evaluation

question(s)

does the

interest

holder

want

answered

and why?

Does the interest

holder prefer

specific types of

data (qualitative,

quantitative)? If

so, what type

and what do you

prefer about that

data?

How would

the interest

holder

define

success for

this

program?

How does the

interest

holder

anticipate

using the

results of this

evaluation?

How would

you like to

learn about

the evaluation

findings (e.g.,

report,

presentation)?

Directions

: Complete this table by identifying the interest holder in the first column and then answering

the questions for steps 2-6. You also could share this with the interest holder and have them complete

the table. Feel free to add additional rows to accommodate the number of interest holders. Remember

to weave in the cross-cutting actions and evaluation standards into how an evaluation is designed and

implemented and considered by interest holders when completing this table.

CDC Program Evaluation Framework Action Guide — page 57

Worksheet 1C.

Interest Holder Planning Matrix

Level of

Involvement

Level of

Involvement

Level of

Involvement

Level of

Involvement

Level of

Involvement

Interest Holder

Inform

We will keep

you informed

of the

evaluation’s

progress and

findings.

Consult

We will keep

you

informed,

listen to you,

and provide

feedback on

how your

input

influenced

the

evaluation.

Involve

We will work

with you to

ensure your

concerns are

considered

and reflected

in options

considered,

make sure

you get to

review and

comment on

options, and

provide

feedback on

how your

input is used

in the

evaluation.

Collaborate

We will

incorporate

your advice

and

suggestions to

the greatest

extent

possible and

give you

meaningful

opportunities

to be part of

the evaluation

decision-

making

process.

Empower

This is your

evaluation.

We will offer

options to

inform your

decisions.

You will

decide and

we will

support and

facilitate

implementing

what you

decide.

Name and

Evaluation Step

Directions:

Complete the matrix by filling in the name of each Interest Holder, and the step(s) that they

would like to be involved in. Use an “X” to mark their level of involvement. Revise this document, as

needed.

Adapted from

Bryson J M, Patton M Q, Bowman R A (2011).

Working with evaluation stakeholders: A

rationale, stepwise approach and toolkit

. Evaluation and program planning, 34(1), 1-12.

CDC Program Evaluation Framework Action Guide — page 58

Worksheet 1D.

Template for Place-Based Context

Place-Based Context Areas

Programmatic Features

Responses

What is the program, and why was it

developed?

Who funds the program? How is the program

funded?

Who is involved in the development and

implementation of the program?

What is the user’s commitment to the program?

What are the demographics of the user group,

including income, education, gender, race or

ethnicity, and other identities?

Who has authority and decision-making power?

How is this process done?

Programmatic Environment

Responses

What is the history of the community, program,

organization and of the program within it?

What are the strengths of this context? Are

there any conditions or circumstances that are

problematic?

How are power and privilege distributed across

the program?

What are organizational learning/evaluation

norms for the organization?

What are the spoken or unspoken norms about

evaluation at the organization?

What are the spoken or unspoken norms about

identifying and using data for action at the

organization?

How does the organizational mission either

support or oppose evaluation?

Directions

: Read through each question in the first column and provide your response in the second

column. Feel free to modify this table, as needed.

CDC Program Evaluation Framework Action Guide — page 59

Worksheet 1E.

Template for Individual and Organizational Evaluation Capacity

Beginner

Intermediate

Expert

Has basic knowledge

and limited experience

Has practical and

applied experience

Recognized as an

authority

Individual Evaluation

Capacity

How knowledgeable is

the evaluator, or

evaluation team, about

the different

evaluation approaches

and methods?

How much experience

does the evaluator, or

evaluation team, have

with developing

evaluation tools and

templates for use by

the organizations?

How much experience

does the evaluator, or

evaluation team, have

with analytical and

facilitation skills?

None

Some

Many

There are no resources

available

Resources can be

accessed but are

limited

Any resource can be

accessed and is

available for use

Organizational

Evaluation Capacity

What financial

resources are available

for the evaluation?

What human resources

(such as time and

personnel) are

available for the

evaluation?

What is the

organizational culture

with respect to

evaluation and using

evaluation findings?

CDC Program Evaluation Framework Action Guide — page 60

None

Some

Many

There are no resources

available

Resources can be

accessed but are

limited

Any resource can be

accessed and is

available for use

Organizational

Evaluation Capacity

Are there opportunities

within the organization

to reflect on insights

that arise throughout

the course of the

evaluation?

Directions

: Read through each question in the first column and then rank the answer to the question,

according to the 3-point Likert scale. Feel free to modify this table as needed.

CDC Program Evaluation Framework Action Guide — page 61

Worksheet 1F.

Cultural Identity Inventory and Assessment

Cultural

Dimensions

Manifestations

Potential

Biases

Potential

Biases

Vantage

Points

Strength

Challenge

Continuing

Learning

What values,

actions or

messages are

associated with

this cultural

dimension? Does

this dimension

interact with

other dimensions?

If so, how?

What

privileges do

you have?

How have

you

responded?

What have

you been

denied?

How have

you

responded?

How do

you

understand

this aspect

of yourself?

How do

you think

others view

you?

Is your

culture

dimension a

strength or

challenge

(or both) to

the

evaluation?

What do I

need to do to

continue to

learn about

this cultural

dimension?

Gender

identity

Race or

ethnicity

Class

Sexual

orientation

Citizenship

Religion

Physical or

mental

ability

Other?

Directions

: Complete the table by filling in each column box for the associated cultural dimension. You

may add or remove cultural dimensions, as needed.

Adapted from

Wallerstein, N., Duran, B., Oetzel, J.G., Minkler, M. (Eds.). (2018). Community-based

participatory research for health: Advancing social and health equity (3

edition). Jossey-Bass. ISBN-

13:978-1119258858

CDC Program Evaluation Framework Action Guide — page 62

Worksheet 1G.

Practicing Reflective Practice

Reflection Question

Relevant Step, Standard, or

Cross-Cutting Action

Reflection Note

How might my cultural identity

influence the evaluation?

What am I seeing, noticing, or

feeling?

What am I learning? Why is this

important?

What are the implications for

practice?

Directions

: Complete the table by filling in the relevant step, evaluation standard, or cross-cutting action

in the second column that is related to each question listed in the first column. In the third column,

write down your reflections. If you require more space to write, please duplicate this table. Keep in mind

that reflective practice should be ongoing, and you do not necessarily have to complete the table at the

beginning of an evaluation. Instead, consider revisiting the table and filling it out and/or updating it

across multiple timepoints.

CDC Program Evaluation Framework Action Guide — page 63

Step 2: Describe the Problem

Worksheet 2A.

Raw Material for Your Logic Model

Activities

What will the program and staff do?

Outcomes

What changes do you hope will result in someone

or something other than the program and the

staff?

Directions

: Complete this table with interest holders as a mapping exercise to understand the outcomes

and activities of the program.

CDC Program Evaluation Framework Action Guide — page 64

Worksheet 2B.

Program Description “Cheat Sheet”

What activities

do/will we do to

move target

audiences to act?

(Note that you have

already identified

the activities in

Worksheet 2A)

Which target

audiences need to

act?

What kind of action

do I need these

target audiences to

take?

What is the “big public health

problem” I’m addressing?

(outcome)

(Note that you have already

identified the outcomes in

Worksheet 2A)

CDC Program Evaluation Framework Action Guide — page 65

Worksheet 2C.

If…Then Statements: Reading and Writing a Logic Model.

Intended Results

Inputs

Activities

Short-Term

Outcomes

Intermediate

Outcomes

Long-Term

Outcomes

If you have…

…then you can…

If you accomplish

your planned

activities to the

extent intended,

then it will result

in…

…and in time will

result in…

…and in our

ultimate effects.

With these

resources,

we can implement

these activities,

which will lead to

these changes,

which lead to,

which will in turn

lead to.

Directions:

Use the Worksheet to practice how to read a logic model and talk through it narratively. Two

different formulations for statement structure are provided in the Worksheet. Try both and see which

makes sense for you and the program/logic model in question. These are not the only ways to construct

statements for your logic model but serve as starting point. Practice by writing statements

using Models

2A and 2B

from Step 2 of the Toolkit. Example statement formulation: If the program has ____ [input]

and ____[input], it can (do) ____ [activity] and ____ [activity], which will result in ____ [outcome], which

will in turn result in ____ [outcome], and ____ [outcome].

CDC Program Evaluation Framework Action Guide — page 66

Worksheet 2D.

Logic Model Review Checklist

Prompt

Response

Does your logic model have a title at the top of the

page?

[Optional] Is a short description needed to

introduce the program reflected in the logic

model?

If yes, include under the title towards the top of

the page.

Do you have clear headers for each component of

your logic model (inputs, activities, outcomes,

context, etc.)?

Is plain language used throughout? See

www.plainlanguage.gov/resources

for more on

using plain language.

Are there abbreviations or acronyms?

If yes, ensure they appear unabbreviated the first

time they appear in the logic model.

Do you have a narrative accompanying your logic

model?

Directions:

When reviewing your logic model draft, consider the following and check to see if they are

reflected in your logic model. This checklist is only a starting point for your review. You can also utilize

the other resources throughout this toolkit section to draft a program description.

CDC Program Evaluation Framework Action Guide — page 67

Template 2A:

https://www.cdc.gov/evaluation/media/pdfs/2024/12/LMTemplate2A-Action-Guide-

12032024-1.pdf

CDC Program Evaluation Framework Action Guide — page 68

Step 3: Focusing the Evaluation Question and Design

Worksheet 3A.

Evaluation Purpose Identification

Intended User and Intended Uses

Response

Intended user(s)

Important information the intended user(s)

wants to obtain from the evaluation

Intended uses of the evaluation findings

Date evaluation findings are needed

Feasibility

Response

Stage of program development

Intensity of the program

Financial resources needed for the evaluation

Staff needed for the evaluation

Percent of time staff can invest in the evaluation

Context important to the evaluation

Components of the program (for example, logic

model) best suited for evaluation

Evaluation Purpose

Response

Evaluation purpose(s) based on intended users,

intended uses, and feasibility

Directions

: Provide an answer for each statement in the first column.

CDC Program Evaluation Framework Action Guide — page 69

Worksheet 3B.

Evaluation Questions Criteria Matrix

Does this evaluation question meet

this criterion?

Yes

Does not meet criterion

but merits inclusion

because….

Evaluation Question:

Evaluative

•

Does this question provide

interest holders with answers

that will be useful to inform next

steps for the program?

Pertinent

•

Is this question relevant to the

priorities and information

interest holders want to obtain

from the evaluation?

•

Is this question directly linked to

the program’s purpose, goals,

activities, or outcomes (does it

come from the logic model)?

•

Is this question relevant to the

evaluation purpose?

Reasonable

•

Is this evaluation question linked

to what the program can

practically and realistically

achieve or influence (scope of

influence and resource

availability)?

•

Is this evaluation question

appropriate for the program’s

stage of development?

Specific

•

Is the evaluation question

clearly worded (words are

clearly defined)?

•

Is the evaluation question

narrow in scope (addresses

specific program components or

elements of program

performance to be

investigated)?

Answerable

CDC Program Evaluation Framework Action Guide — page 70

•

Can the evaluation question be

answered objectively with

available and attainable data?

•

Can the evaluation question be

answered ethically and

respectfully?

•

Can data for the evaluation

question be obtained with

accuracy acceptable to interest

holders?

•

Can the question identify health

inequities and/or disparities?

Complete

•

Does this question, in

combination with the other

questions proposed for the

evaluation, address the

evaluation purpose and all

program components deemed

important by the interest

holders?

•

Does this question, in

combination with the other

questions proposed for the

evaluation, provide enough

information which can be acted

upon to make program

improvements?

Directions

: Read each question in the first column that is underneath each blue highlighted evaluation

question characteristic and provide a “yes” or “no” response based on the evaluation questions for your

evaluation. Use the last column to provide an explanation of why an evaluation question must still be

included even if it doesn’t meet the criteria.

Adapted from:

Wingate, L., & Schroeter, D. (2007).

Evaluation questions checklist for program

evaluation

Step 4: Gather Credible Evidence

Worksheet 4A.

Identifying Data Collection Sources/Methods

Question

Response

What is the purpose of this data collection?

Who will use the data?

What data collection methods and data sources

are credible and culturally responsive to the

interest holders?

Do we have appropriate resources? Funding?

Staff? Training and background knowledge?

How long will the data collection period be?

When will results be needed?

How often will we need to collect data?

What ethical considerations are there?

Are there existing data that can answer the

evaluation questions? If so, what are the data

sources?

Directions

: Answer the questions in the first column to help determine the data sources and methods

needed for the evaluation.

Worksheet 4B.

Setting Expectations about Indicators

Excellent –

define

Good –

define

Adequate –

define

Poor –

define

Notes

Indicator 1

Indicator 2

Indicator 3

Directions:

Use the headers to define what constitutes an indicator being excellent, good, adequate, or

poor. Determine where the indicators identified in the evaluation belong within the defined range. Add

more rows to accommodate for the number of indicators for your evaluation.

CDC Program Evaluation Framework Action Guide — page 72

Worksheet 4C.

Selecting Relevant Indicators

Key Objective/Focus

Area

Evaluation Question

Identify Key

Performance Indicator

Indicator Justification

Directions:

List the objective or focus area that the evaluation will address. Once you have identified the

focus area, list the associated evaluation questions identified in step three and then identify indicators

associated with the focus areas and questions. Write indicators so that they effectively help evaluators

measure progress towards the objective or program success. Once you have identified the indicators, it

may be helpful to provide a brief justification for how the indicator aligns with the evaluation focus

areas. If you are finding it difficult to justify, then that may indicate that you need to rework the

indicator.

Worksheet 4D.

Data Collection Plan; Milestones for Data Collection Activities

Phase 1

Phase 2

Phase 3

Phase 4

Example:

Week One: Administer

emails for survey/focus

group participation

Week Three: Conduct

the focus group

Directions:

Please indicate any key milestones or deadlines that you have for data collection activities.

Add more rows as necessary.

Worksheet 4E.

Data Collection Plan; Allocation of Resources

Personnel

Resources Needed

Budget

Example: Facilitator

Example: Equipment to Record

the Session

Example: Incentives for

Participants

Directions:

Please outline how you will allocate resources and list any resources that you will need to

accomplish implementation, who will be needed to implement, and what you will need to budget for

implementation. Add more rows as necessary.

CDC Program Evaluation Framework Action Guide — page 73

Worksheet 4F.

Contingency Plan for Data Collection

What are the potential

implementation risks to data

collection?

How do you plan to mitigate?

Who will be responsible?

Directions:

Please use the first column to identify the implementation risks then use the second two

columns to create a contingency plan for mitigating those risks for data to remain valid. Add more rows

as necessary.

Worksheet 4G.

Communications Plan for Data Collection

Communication Frequency

Communication Methods

Example: Weekly check-ins with the interest

holders

Example: Virtual Meetings via Teams

Directions:

Please create a communication plan and indicate how often you will communicate with key

partners and team members and what the best method of communication will be. Add more rows as

necessary.

CDC Program Evaluation Framework Action Guide — page 74

Worksheet 4H.

Evaluation Matrix

Concept/

Construct

Evaluation

Questions

Indicator(s)

Measure /

Metric

Data

Collection

Source

Data

Collection

Method

Data

Collectio

Timeline

Data

Analysis

Which

concept will

you evaluate?

What do

you want

to know?

What data

will you

need to

address the

evaluation

questions?

How do

you plan

measure

these

data?

What will

be the

baseline?

Where

will you

get this

data

from?

How will

you get

this data?

When

will you

collect

this

data?

How will

you

organize

and

interpret

the data

collected?

Directions

: Please use the information from the previous worksheets to complete the evaluation matrix.

CDC Program Evaluation Framework Action Guide — page 75

Step 5: Generate and Support Conclusions

Worksheet 5A.

Data Analysis Plan

Evaluation

Question

Data to be

Analyzed

Analysis to be

Performed

Person(s)

Responsible

Due Date

Directions

: For each data point to be analyzed, identify the analysis to be performed, person(s)

responsible, and due date.

Adapted from

: Wilce, M., et al. Planting the Seeds for High-Quality Program Evaluation in Public Health.

Atlanta, GA: Centers for Disease Control and Prevention.

CDC Program Evaluation Framework Action Guide — page 76

Worksheet 5B.

Collaborative Interpretation of Evaluation Results

Purpose and goals (what do you hope to achieve

or learn from the process)

Target participants (which interest holders will

you engage in this activity)

Format of activity (i.e., data walk with posters,

data chat, dashboard review)

Logistics decisions (any logistics decisions aimed

at making participating easier on the participants

attending)

Data point 1 (which data point will you discuss,

and how will it be visually displayed)

Discussion questions for reviewing data point 1

(which questions will you ask participants to

guide the discussion toward your learning goals)

Data point 2

Discussion questions for reviewing data point 2

Data point 3

Discussion questions for reviewing data point 3

Data point 4

Discussion questions for reviewing data point 4

Directions

: This worksheet can be used to plan a collaborative discussion of analytic results with interest

holders. Some options for format include data walks, data gallery walks, data parties, and data chats.

This is a coordinated event where interest holders come together collaboratively to discuss and make

meaning of analytic results and discuss implications for practice.

Adapted from

: Murray, B., Falkenburger, E., Saxena, P. (2015). Data Walks: An Innovative Way to Share

Data with Communities. Urban Institute.

CDC Program Evaluation Framework Action Guide — page 77

Worksheet 5C.

Forming Actionable Evaluation Recommendations

Program:

Evaluation Purpose:

Evaluation Result (Describe the evaluation result that needs action):

Directions:

Use this worksheet to identify the plan of action needed to inform collaborative

recommendation discussion with interest holders. Write the name of the program, evaluation purpose

and evaluation result at the top. The descriptions in the columns above can be used as a guide for laying

out how what information is important to include. Add more rows as necessary.

Plan for

Action

Act and

Monitor

Progress

Change

Needed

Activities

Implement

Change

Person

Responsible

Resources

Required

Due By

Indicators

that Change is

Implemented

Data Sources

Indicators

Monitor

Success

Change

Data

Sources

Describe

key

change(s)

you want

to achieve

based on

this

finding.

List

activities

that need

to be

carried

out to

make the

change

happen

in the

program.

List the

person(s)

who will

assure

each

activity

occurs.

List

resources

required

for the

activity.

Assign a

due date

by which

the activity

will be

completed.

Describe

how you will

know that

the change is

implemented

as planned.

Describe

what data

you will need

to have to

know change

implemented.

Describe

how you

will

know

the

change

program

working

or not.

Describe

the data

you will

need

measure

success.

CDC Program Evaluation Framework Action Guide — page 78

Step 6: Act on Findings

Worksheet 6A.

Planning for Action with Interest Holders

Interest

Holder

Insight

What do you

think will be

learned?

Who do you think

will use these

findings?

When will

findings be

used?

How will the

findings be

used? *

Directions:

Use this worksheet to identify how to facilitate action of the evaluation findings

Adapted from

International Development Research Center. (2012). Identifying the Intended User(s) and Use(s)

of an Evaluation. Retrieved from

idrc.pdf (betterevaluation.org)

*Use can be defined as:

(1). Instrumental Uses: modify the object of the evaluation (for example, make changes to program

activities; continue or end the program; change funding allocation; make staffing decisions; inform

professional development; fulfill accountability requirements; improve technical assistance; determine

recipient training needs; change evaluation strategies or performance measures; and improve

evaluation data reporting).

(2). Conceptual Uses: leads to new understanding; process use (for example, change program support

structures; provide feedback to recipients; build evaluation capacity; mitigate risk to the program; and

influence approach to other programs).

(3). Enlightenment Uses: Adds knowledge to the field (for example, develop best or promising practices;

publish and present; and model effective working relationships between evaluation and programmatic

teams).

Worksheet 6B.

Planning for Dissemination for Various Audiences

Information

and Purpose

Audience(s)

Possible

Formats

Possible

Messengers

Timing

Person

Responsible

Directions:

Use this worksheet to identify the dissemination plan.

Source:

Template adapted from Learning & Growing through Evaluation: State Asthma Program

Evaluation Guide Appendix D. Strategic Evaluation Plan Outline.

CDC Program Evaluation Framework Action Guide — page 79

Worksheet 6C.

Questions that Evaluator(s) can consider when developing a follow-up plan with Interest

Holders

Question

Response

What support do you think you need

to grasp and apply the findings to

your work?

Following the review of the

evaluation results, how much time

would you need before you act on

the findings and recommendations?

Who will serve as the champion for

the findings and recommendations,

and prevent misuse?

How much influence do you expect

the evaluation findings to have?

How have past evaluations been

used?

What type of data and findings need

to support decision making, and

what type of data and findings are

viewed by leadership as “credible”?

Who needs to be involved for the

evaluation to be influential?

How will the Interest Holders know

afterwards if the evaluation was

used as intended?

Directions:

Use this worksheet to identify important considerations when following up with interest

holders to facilitate action on evaluation findings.

Adapted from:

International Development Research Center. (2012). Identifying the Intended User(s) and

Use(s) of an Evaluation. Retrieved from

idrc.pdf (betterevaluation.org)

CDC Program Evaluation Framework Action Guide — page 80

References

•

Kidder DP, Fierro L, Luna E, et al. CDC Program Evaluation Framework, 2024.

MMWR Recomm

Rep.

2024;73(No. RR-6):1-37. doi:

http://dx.doi.org/10.15585/mmwr.rr7306a1

. Retrieved from

https://www.cdc.gov/mmwr/volumes/73/rr/rr7306a1.htm?s_cid=rr7306a1_w

•

Office of Management and Budget.

Evidence-Based Policymaking: Learning Agendas and Annual

Evaluation Plans

. OMB M-21-27 (2021). Retrieved from

https://www.whitehouse.gov/wp-

content/uploads/2021/06/M-21-27.pdf

•

Wanzer, D.L. (2021). What Is Evaluation? Perspectives of How Evaluation Differs (or Not) From

Research

American Journal of Evaluation

42(1), 28-46.

https://doi.org/10.1177/1098214020920710

•

Centers for Disease Control. (2021). Updated Guidelines for Evaluating Public Health

Surveillance Systems: Recommendations from the Guidelines Workgroup

. MMWR

Recommendations and Reports

,50(RR-13): 1-35.

PMID: 18634202.

•

Government Accountability Office (GAO). (2021). Program Evaluation Key Terms and Concepts.

GAO-21-404SP. Retrieved from

https://www.gao.gov/assets/gao-21-404sp.pdf

•

Burton, D.C., Cardo, D., et al

(2024). Principles of Health Equity Science for Public Health Action.

Public Health Reports

,139:277-283.

•

Association AE. (2011). A

merican Evaluation Association Public Statement on Cultural

Competence in Evaluation.

•

Centers for Disease Control and Prevention. (1999

Framework for program evaluation in public

health.

MMWR: Recommendations and Reports

,48:1-40.

•

Snow, B. (2016).

Context in Evaluation

. Dr. Beth Snow Blog.

Retrieved from: Context in

Evaluation | Dr. Beth Snow (drbethsnow.com)

•

Armstead, T., Lee, R. (2013).

Evaluability Assessment Guidance for DELTA FOCUS Recipients.

Internal report (Centers for Disease Control). Unpublished.

•

D’Ostie-RL Dagenais, C., Ridde, V. (2013). An evaluability assessment of a West Africa based non-

governmental organizations (NGO) progressive evaluation strategy

Evaluation and Program

Planning

, 36(1), 71-79.

•

Farmer, H. (2018, February 24). Conversations to have when designing a program: Fostering

evaluative thinking

Better Evaluation Blog

. Retrieved from:

Conversations to have when

designing a program: Fostering evaluative thinking - Blog post on Better Evaluation

CDC Program Evaluation Framework Action Guide — page 81

•

Bryson, J.M., Patton, M.Q., Bowman, R.A. (2011). Working with evaluation stakeholders: A

rationale, step-wise approach and toolkit.

Evaluation and Program Planning

,34(1):1-12.

doi:https://doi.org/10.1016/j.evalprogplan.2010.07.001

•

Wallerstein, N., Duran, B., Oetzel, J.G., Minkler, M. (Eds.). (2018). Community-based

participatory research for health: Advancing social and health equity (3

edition). Jossey-Bass.

ISBN-13:978-1119258858

•

Napier A, Simister, N. (2017). Participatory Learning and Action. Intrac for Civil Society. Retrieved

from Participatory-learning-and-action.pdf (intrac.org)

•

Centers for Disease Control and Prevention. (2020).

Colorectal Cancer Control Program (CRCCP)

Evaluation Plan.

Internal report (Centers for Disease Control): unpublished

•

Fitzpatrick, J.L

(2012). An introduction to context and its role in evaluation practice

New

Directions for Evaluation

,135:7-24.

•

Kirkhart, KE. (2010

Eyes on the prize: Multicultural validity and evaluation theory.

American

Journal of Evaluation

,31(3):400-413.

•

Buetti, D., Bourgeois, I., Jafary, M. (2023). Examining the competencies required by evaluation

capacity builders in community-based organizations.

Evaluation and Program Planning

, 97,

102242. https://doi.org/10.1016/j.evalprogplan.2023.102242

•

Miranda-Hartsuff, P., Hillard, T., Walters, H., Snipes, A., Reddock, E., Santos, E., Elam P. (2023).

Chapter 2: Positionality, Reflexivity, and Strengthening the Process of Conducting Culturally

Responsive Racially Equitable Evaluations (CREE) as Practitioners of Color

In Adedoyin, A.,

Amutah-Onukagha, N., Jones, C. (Eds.).

Culturally Responsive & Equitable Evaluation: Visions and

Voices of Emerging Scholars

. Congella, Inc. 2024. ISBN:978-1-7935-5864-0

•

Symonette H. (2009). Cultivating self as responsive instrument: Working the boundaries and

borderlands for ethical border crossing

. Handbook of Social Research Ethics

. (pp.279-294).

https://doi.org/10.4135/9781483348971.n18

•

Zandniapour, L., JBS International. (2014).

Impact Evaluability Assessment Tool.

Washington, DC:

Corporation for National Service.

Microsoft Word -

Laying_the_Goundwork_Before_Your_First_Evaluation_Tool

•

Centers for Disease Control and Prevention. (2021). Comprehensive Cancer Control Branch

Program Evaluation Toolkit.

https://www.cdc.gov/cancer/ncccp/pdf/CCC-Program-Evaluation-

Toolkit-508.pdf

•

Meyer, M.L., Louder, C., Nicolas, G. (2022). Creating with, not for people: Theory of change and

logic models for culturally responsive community-based intervention.

American Journal of

Evaluation

, 43(3), 378-393. DOI: 10.1177/10982140211016059

•

Chen, H.T., Pan, H.L.W., Morosanu, L., Turner, N. (2018). Using logic models and the action

model/change model schema in planning the learning community program: A comparative case

study.

Canadian Journal of Program Evaluation

, 33(1), 49-68. DOI: 10.3138/cjpe.42117

•

Barbrook-Johnson, P., Penn, A. (2021). Participatory systems mapping for complex energy policy

evaluation.

Evaluation

, 27(1), 57-79. https://doi.org/10.1177/1356389020976153

•

Mayne, J. (2015). Useful theory of change models.

Canadian Journal of Program Evaluation

30(2), 119-142. DOI: 10.3138/cjpe.230

•

Arensman, B., Van-Waegeningh, C., Van-Wessel, M. (2018). Twinning “Practices of Change” With

“Theory of Change”: Room for Emergence in Advocacy Evaluation.

American Journal of

Evaluation

, 39(2), 221-236. https://doi.org/10.1177/1098214017727364

•

Newcomer, K., Olejniczak, K., Hart, N. (2022). Learning agendas: Motivation, engagement, and

potential.

New Directions for Evaluation

, 63–83.

https://doi.org/10.1002/ev.20495

•

Jones, N.D., Azzam, T., Wanzer, D.L., Skousen, D., Knight, C., Sabarre, N. (2020). Enhancing the

Effectiveness of Logic Models.

American Journal of Evaluation

, 41(3), 452-470.

https://doi.org/10.1177/1098214018824417

•

Peyton, D.J., Scicchitano, M. (2017). Devil is in the details: Using logic models to investigate

program process.

Evaluation and Program Planning

, 65, 156-162.

•

Montague, S., Porteous, N.L. (2013). The case for including reach as a key element of program

theory.

Evaluation and Program Planning

, 36(1), 177-183.

DOI:10.1016/j.evalprogplan.2012.03.005.

•

King, J. (2021). Expanding theory-based evaluation: Incorporating value creation in a theory of

change.

Evaluation and Program Planning

, 89, 101963.

DOI:10.1016/j.evalprogplan.2021.101963

•

Wright, D.S., Grunbaum, J.A., Dawkins N, et al. (2011).

Ensuring accountability in public health

prevention research: evaluating the prevention research centers program at the Centers for

Disease Control and Prevention

. Journal of Public Health Management Practice, 17(2), E1-E9

•

McColluch C, Davis M, Araujo A, et al. (2022). Lessons learned for public health workforce

development: An evaluation of the Centers for Disease Control and Prevention’s laboratory

leadership service fellowship.

Evaluation and Program Planning

, 95, 1-10.

https://doi.org/10.1016/j.evalprogplan.2022.102147

•

Centers for Disease Control. (2018). CDC Program Evaluation Framework Checklist for Step 2.

Checklist for CDC Eval Framework Step 2: Describe the Program

•

National Science Foundation. (2002). The 2002 User Friendly Handbook for Project Evaluation.

https://www.nsf.gov/pubs/2002/nsf02057/nsf02057.pdf

CDC Program Evaluation Framework Action Guide — page 83

•

Evergreen, S.D. (2014). Presenting Data Effectively: Communicating Your Findings for Maximum

Impact . Los Angeles: SAGE. Print.

•

Centers for Disease Control and Prevention. Division of Heart Disease and Stroke State Heart

Disease and Stroke Program Evaluation Guide: Developing and Using a Logic Model.

State Heart

Disease and Stroke Prevention Program Evaluation Guide: Developing and Using a Logic Model

Evaluation Guide

•

Weinschenk, S. (2020).

100 Things Every Designer Needs to Know about People

. (2nd ed).

Hoboken: Pearson Education, Limited Print.

•

Lien, A.D., Greenleaf, J.P., Lemke, M.K., Hakim, S.M., Swink, N.P., Wright, R., Meissen, G. (2011).

Tearless Logic Model

. Global Journal of Community Psychology Practice

, 2(2), 1-12.

http://www.gjcpp.org/

•

Bamberger, M,, Mabry, L. (2020

). RealWorld evaluation.

(

Vols. 1-0

). SAGE Publications, Inc.,

https://doi.org/10.4135/9781071909607

•

Salabarría-Peña, Y., Apt, B.S., Walsh, C.M. (2007).

Practical Use of Program Evaluation among

Sexually Transmitted Disease (STD) Programs, Atlanta (GA)

: Centers for Disease Control and

Prevention.

•

Wingate, L., Schroeter, D. (2007). Evaluation questions checklist for program evaluation.

Retrieved from

http://wmich.edu/evaluation/checklists

•

Centers for Disease Control. National Asthma Control Program. (2013). Good Evaluation

Questions. A Checklist to Help Focus Your Evaluation Questions. Retrieved from

AssessingEvaluationQuestionChecklist.pdf (cdc.gov)

•

Shackman, G. (2020). What is Program Evaluation? A Beginner’s Guide. The Global Social Change

Research Project. Retrieved from

What is Program Evaluation? A Beginner's Guide

•

The Pell Institute for the Study of Opportunity for Higher Education, the Institute for Higher

Education Policy, and Pathways to College Network. (2024

Evaluation Toolkit. Retrieved from

Choose an Evaluation Design « Pell Institute

•

University of Washington I-Tech Fostering Healthier Communities. (2024). Training Evaluation

Framework and Tools. Retrieved from

Evaluation-Design-and-Methods.pdf

CDC Program Evaluation Framework Action Guide — page 84

•

Office of Management and Budget, Executive Office of the President, OMB M-20-12, Phase 4

Implementation of the Foundations for Evidence-Based Policymaking Act of 2018:

Program

Evaluation Standards and Practices

. 2020; Retrieved from

M-20-12 (whitehouse.gov)

•

Office of Chief Operating Officer. (2023).

Memoranda of Understanding/ Memoranda of

Agreement

. Retrieved from Memoranda of Understanding/Memoranda of Agreement | Services

| OCOO (cdc.gov)

•

Mwita, K. (2022). Factors to Consider When Choosing Data Collection Methods.

International

Journal of Research in Business and Social Science

, 11(5): 532-538.

DOI:

10.20525/ijrbs.v11i5.1842

•

Centers for Disease Control and Prevention.National Center for Cancer Control and Prevention.

(2024). From SMART to SMARTIE Objectives. Retrieved from

SMART to SMARTIE Objectives

•

Hennink M, Hutter I, Bailey A.

Qualitative Research Methods

. SAGE Publications. 2011

•

Centers for Disease Control and Prevention. (2021).

Learning and Growing through Evaluation:

Asthma Program Evaluation Guide. Module 1: Planning Evaluations

. Atlanta, GA: Centers for

Disease Control and Prevention, National Center for Environmental Health, Division of

Environmental Health Science and Practice, Asthma and Community Health Branch.

http://www.cdc.gov/asthma/program_eval/guide.htm

•

Centers for Disease Control and Prevention. (2018).

Analyzing Qualitative Data for Evaluation.

(Evaluation Brief No. 19). U.S. Department of Health and Human Services. Retrieved from:

https://www.cdc.gov/healthyyouth/evaluation/ pdf/brief19.pdf

•

Centers for Disease Control and Prevention.

Analyzing Quantitative Data for Evaluation

(Evaluation Brief No. 20). U.S. Department of Health and Human Services. Retrieved from:

https://www.cdc.gov/healthyyouth/evaluation/ pdf/brief20.pdf

•

Hood, S., Hopson, R., Kirkhart, K. (2015). Chapter 12: Culturally Responsive Evaluation: Theory,

Practice and Future Implications. In Newcomer, K.E., Hatry, H.P., Wholey, J.S. (Eds.),

Handbook

of Practical Program Evaluation

(pp. 281-317). Jossey-Bass. ISBN: 9781119171386.

DOI:10.1002/9781119171386.ch12.

•

Krause, H., Richburg-Hayes, L. (2023). The Data Equity Framework: A Concrete and Systematic

Equity-Oriented Approach to Quantitative Data Projects. https://doi.org/10.31235/osf.io/sqt4u

•

Murray, B., Falkenburger, E., Saxena, P. (2015). Data Walks: An Innovative Way to Share Data

with Communities. Urban Institute. Retrieved from

Data Walks: An Innovative Way to Share

Data with Communities

•

Springer, A.E., Marshall, A.N., Randolph, R., Wilkinson, A.V., Rosenbluth, B., Cortez, A., Greene,

M., Ganzar, L.A., Millea, S., Levulett, A., Delgado-Ochoa, M., Sebastian, N. (2022). Exploring

CDC Program Evaluation Framework Action Guide — page 85

Models for Youth Engagement in Community Health Planning: The Youth-led Community Health

Learning Initiative. Progress in Community Health Partnerships: Research, Education, and Action,

16(2), 155–168. Johns Hopkins University Press. https://doi.org/10.1353/cpr.2022.0020

•

Wilce, M., Fierro, L.A., Gill, S., Perkins, A., Kuwahara, R., Barrera-Disler, S., Orians, C., Codd, H.,

Castleman, A.M., Nurmagambetov, T., Anand, M. (2021). Planting the Seeds for High-Quality

Program Evaluation in Public Health. Atlanta, GA: Centers for Disease Control and Prevention,

National Center for Environmental Health, Division of Environmental Health Science and

Practice, Asthma and Community Health Branch. Retrieved from

http://www.cdc.gov/asthma/program_eval/PlantingSeeds_eTextbook-508.pdfCenters for

Disease Control and Prevention. (2014).

Applying the Knowledge to Action (K2A) Framework:

Questions to Guide Planning

. Atlanta, GA: Centers for Disease Control and Prevention, U.S.

Deptartment of Health and Human Services.

•

International Development Research Center. (2012

). Identifying the Intended User(s) and Use(s)

of an Evaluation

. Retrieved from idrc.pdf (betterevaluation.org)

•

Zakocs, R., Hill, J.A., Brown, P., Wheaton, J., Freire, K.E. (2015). The Data-to-Action Framework: A

Rapid Program Improvement Process.

Health Educ Behav

. 42(4):471-9. doi:

10.1177/1090198115595010. PMID: 26245935; PMCID: PMC5990006.

CDC Program Evaluation Framework Action Guide

Prompt to Clone This CDC Program Evaluation Framework Action Guide

Apply this example's core structure, strategic pacing, and structural hierarchy directly to your own topic.

What are you generating today?