Advancing Educational Equity: Tools for Evidence Building

Changemakers in education recognize that when students are taught in inclusive and supportive environments and receive high-quality, challenging, and engaging instruction, they are better positioned to reach their full potential. That is why Mathematica is working in partnership with the Bill & Melinda Gates Foundation to help education grantees generate timely and actionable evidence that will lead to program improvements and inform the field on how to implement and scale solutions that address education’s most persistent challenges.

As the evaluation technical assistance partner for two portfolios of investments in Middle Years Math and Secondary Writing, Mathematica developed a set of tools to support grantees through a measurement and evaluation (M&E) process. The goal of these portfolios is to develop, refine, and scale evidence-based solutions that demonstrate success in improving educational outcomes for students who are Black, Latino, and/or experiencing poverty.

To expand the reach of this work and promote the adoption of evidence-based solutions more broadly, Mathematica has adapted the suite of M&E tools for a diverse set of users—organizations, funders, research partners, and other stakeholders. Although these tools were designed to offer users in education a road map for implementing M&E in their own work, individuals serving in other fields may use or adapt these tools for their own learning needs.

What is the Measurement and Evaluation process?

The M&E process is designed to promote rapid innovation and scaling of promising solutions through generating timely and actionable evidence about what works for whom, and in what context. It uses an iterative approach to evidence building, in which the focus and design of the research is aligned to the solution’s phase of development. Checkpoints are built in throughout to encourage users to pause and reflect on what has been learned so far and to refine the solution and the M&E plan as needed.

The M&E process centers on equity and community voice as a core principle. Across all steps, organizations are encouraged to meaningfully engage community stakeholders to ensure the solution builds on the strengths and assets and meets the needs of the community it is intended to serve.

Assess Measurement and Evaluation Needs and Interests

These tools introduce the iterative approach to evidence building and help organizations articulate their M&E goals, interests, and needs.


Right-Sizing Research to Inform Design and Decision-Making, Part 1

In this video, Mathematica experts introduce an iterative approach to evidence building that they created to help organizations design, refine, and test solutions that seek to address educational and other social challenges.

Kids in a math class

Introducing the evidence-building pipeline and development framework

Play Video

M&E Needs Inventory

Go to resource

The Measurement and Evaluation Needs Inventory is a tool designed to assist an organization in conducting a self-assessment of its existing measurement and evaluation capacity, with the goal of identifying priority areas for support from an external research partner.

 

Prior Research Table

Go to resource

The Prior Research Table is a tool designed to help organizations document previous research on solutions to inform future evaluations.

 

Plan Measurement and Evaluation

These tools help organizations develop their own measurement and evaluation plan. The checklists in particular provide guidance on working through the evidence-building phases. For definitions of terms used in the tools, please refer to this glossary.


M&E Plan Template

Go to resource

The Measurement and Evaluation Template is a road map organizations and funders can use to develop a detailed measurement and evaluation plan with support from a research partner.

 

M&E Checklist: Design the Solution
(Phase 1)

Go to resource

Phase 1 of the Measurement and Evaluation Checklist guides users through the process of co-designing a solution with a partner and developing a well-defined theory of change for how that solution is expected to improve outcomes for students in a priority community.

 

M&E Checklist: Refine the Solution
(Phase 2)

Go to resource

Phase 2 of the Measurement and Evaluation Checklist guides users through the steps to refine a solution based on lessons learned during implementation. By following these steps, users will develop descriptive evidence that the refined solution was successfully implemented in a priority community.

 

M&E Checklist: Assess for Early Evidence of Success (Phase 3)

Go to resource

Phase 3 of the Measurement and Evaluation Checklist guides users through the steps to conduct a small pilot study and demonstrate that a solution is associated with improved outcomes for students in a priority community.

 

M&E Checklist: Validate Effectiveness (Phase 4)

Go to resource

Phase 4 of the Measurement and Evaluation Checklist guides users through the steps to conduct a large pilot study and demonstrate with a high degree of confidence that a solution leads to improved outcomes for students in a priority community.

 

Guide to Measuring Implementation:
The 3 Us

Go to resource

This guide is designed for organizations implementing solutions, with support from a research partner. It provides guidance on when and how to measure usability, usefulness, and utilization—key indicators of success during early implementation.

 

Guide to Setting Targets

Go to resource

This guide is designed to support organizations at different stages of solution development and implementation define a set of targets for each research question they will answer. Targets enable organizations or their funders to define success.

 

Sample Size Guide

Go to resource

This guide is designed for funders and organizations implementing solutions with support from a research partner. It provides an overview of the sample size considerations for each of the evidence-building phases.

 

Conduct Measurement and Evaluation

These tools assist organizations executing their M&E plan, including collecting the data needed to address each research question.


Rightsizing Research to Inform Design and Decision Making, Part 2

In this video, Mathematica experts use a fictitious case study to describe an iterative approach to evidence building they created to support organizations, research partners, and funders develop, refine, and test solutions.

Two middle school girls in class

A Case Study

Play Video

 

Rightsizing Research to Inform Design and Decision Making, Part 3

In this video, Mathematica experts provide a deep dive into each of the five measurement and evaluation criteria that are part of an evidence-building pipeline they developed to support organizations, research partners, and funders develop, refine, and test solutions.

Two middle school boys in class

Measurement and Evaluation Criteria

Play Video

Analyze and Report Results

These tools guide organizations through the final step of the M&E process, which includes analyzing collected data, determining next steps, and reporting the findings.


M&E Reporting Template

Go to resource

Organizations, research partners, and funders can use the Measurement and Evaluation Reporting Template to summarize and share findings from a study.

 

Related Staff

To connect with one of our experts on this project or to get more information about these tools, please send an email to info@mathematica-mpr.com.

Kate Place

Kate Place

Senior Researcher

View Bio Page
Virginia Knechtel

Virginia Knechtel

Principal Researcher

View Bio Page
Kara Conroy

Kara Conroy

Researcher

View Bio Page
Megan Shoji

Megan Shoji

Senior Researcher

View Bio Page
Ryan Ruggiero

Ryan Ruggiero

Research Analyst

View Bio Page
Alexandra  Resch

Alexandra Resch

Senior Director of Learning and Strategy, Human Services

View Bio Page
Gregory Chojnacki

Gregory Chojnacki

Senior Researcher

View Bio Page
Lindsay Fox

Lindsay Fox

Senior Researcher

View Bio Page