Advancing Educational Equity: Tools for Evidence Building
Changemakers in education recognize that when students are taught in inclusive and supportive environments and receive high-quality, challenging, and engaging instruction, they are better positioned to reach their full potential. That is why Mathematica is working in partnership with the Bill & Melinda Gates Foundation to help education grantees generate timely and actionable evidence that will lead to program improvements and inform the field on how to implement and scale solutions that address education’s most persistent challenges.
As the evaluation technical assistance partner for two portfolios of investments in Middle Years Math and Secondary Writing, Mathematica developed a set of tools to support grantees through a measurement and evaluation (M&E) process. The goal of these portfolios is to develop, refine, and scale evidence-based solutions that demonstrate success in improving educational outcomes for students who are Black, Latino, and/or experiencing poverty.
To expand the reach of this work and promote the adoption of evidence-based solutions more broadly, Mathematica has adapted the suite of M&E tools for a diverse set of users—organizations, funders, research partners, and other stakeholders. Although these tools were designed to offer users in education a road map for implementing M&E in their own work, individuals serving in other fields may use or adapt these tools for their own learning needs.
What is the Measurement and Evaluation process?
Mathematica’s suite of measurement and evaluation (M&E) tools provides a road map for generating timely and actionable evidence about what works for whom, and in what context. Whether you are a developer refining evidence-based solutions, a funder exploring a great new idea, or a researcher, you can use these tools to promote rapid innovation and scaling of promising solutions.
Each step of the M&E process has tools to help guide you to a deeper understanding of your solution. There are built-in checkpoints to encourage users to pause and reflect on learning and to refine the solution and the M&E plan as needed.
Additionally, equity and community voice are a core principle embedded in the M&E process. Organizations should meaningfully engage communities across all steps to ensure the solution builds on the strengths and assets and meets the needs of the community they will serve.
Assess Measurement and Evaluation Needs and Interests
These tools introduce the iterative approach to evidence building and help organizations articulate their M&E goals, interests, and needs.
Right-Sizing Research to Inform Design and Decision-Making, Part 1
In this video, Mathematica experts introduce an iterative approach to evidence building that they created to help organizations design, refine, and test solutions that seek to address educational and other social challenges.
Introducing the evidence-building pipeline and development framework
M&E Needs Inventory
The Measurement and Evaluation Needs Inventory is a tool designed to assist an organization in conducting a self-assessment of its existing measurement and evaluation capacity, with the goal of identifying priority areas for support from an external research partner.
Prior Research Table
The Prior Research Table is a tool designed to help organizations document previous research on solutions to inform future evaluations.
Plan Measurement and Evaluation
These tools help organizations develop their own measurement and evaluation plan. The checklists in particular provide guidance on working through the evidence-building phases. For definitions of terms used in the tools, please refer to this glossary.
Menu of Measures: High-Quality Measures of Middle Years Math Student Outcomes
This resource provides educators, school district leaders, researchers, and funders a set of measures and approaches to assess the extent that math instruction and interventions change what students know and how they feel about math, as well as how they use math outside of the classroom.
M&E Plan Template
The Measurement and Evaluation Template is a road map organizations and funders can use to develop a detailed measurement and evaluation plan with support from a research partner.
Design Research Plan Template
This template is for organizations that want to incorporate the needs and preferences of the communities they intend to reach into their program, practice, or product’s design. It includes space for organizations to document a description of their program, practice, or product; their theory of change; their learning objectives and approach; and a timeline to carry out the work.
M&E Checklist: Design the Solution
(Phase 1)
Phase 1 of the Measurement and Evaluation Checklist guides users through the process of co-designing a solution with a partner and developing a well-defined theory of change for how that solution is expected to improve outcomes for students in a priority community.
M&E Checklist: Refine the Solution
(Phase 2)
Phase 2 of the Measurement and Evaluation Checklist guides users through the steps to refine a solution based on lessons learned during implementation. By following these steps, users will develop descriptive evidence that the refined solution was successfully implemented in a priority community.
M&E Checklist: Assess for Early Evidence of Success (Phase 3)
Phase 3 of the Measurement and Evaluation Checklist guides users through the steps to conduct a small pilot study and demonstrate that a solution is associated with improved outcomes for students in a priority community.
M&E Checklist: Validate Effectiveness (Phase 4)
Phase 4 of the Measurement and Evaluation Checklist guides users through the steps to conduct a large pilot study and demonstrate with a high degree of confidence that a solution leads to improved outcomes for students in a priority community.
Guide to Measuring Implementation:
The 3 Us
This guide is designed for organizations implementing solutions, with support from a research partner. It provides guidance on when and how to measure usability, usefulness, and utilization—key indicators of success during early implementation.
Guide to Setting Targets
This guide is designed to support organizations at different stages of solution development and implementation define a set of targets for each research question they will answer. Targets enable organizations or their funders to define success.
Sample Size Guide
This guide is designed for funders and organizations implementing solutions with support from a research partner. It provides an overview of the sample size considerations for each of the evidence-building phases.
Conduct Measurement and Evaluation
These tools assist organizations executing their M&E plan, including collecting the data needed to address each research question.
Rightsizing Research to Inform Design and Decision Making, Part 2
In this video, Mathematica experts use a fictitious case study to describe an iterative approach to evidence building they created to support organizations, research partners, and funders develop, refine, and test solutions.
A Case Study
Rightsizing Research to Inform Design and Decision Making, Part 3
In this video, Mathematica experts provide a deep dive into each of the five measurement and evaluation criteria that are part of an evidence-building pipeline they developed to support organizations, research partners, and funders develop, refine, and test solutions.
Measurement and Evaluation Criteria
Analyze and Report Results
These tools guide organizations through the final step of the M&E process, which includes analyzing collected data, determining next steps, and reporting the findings.
M&E Reporting Template
Organizations, research partners, and funders can use the Measurement and Evaluation Reporting Template to summarize and share findings from a study.
Related Staff
To connect with one of our experts on this project or to get more information about these tools, please send an email to info@mathematica-mpr.com.
