Supporting Families Through Equity-Infused Program Change

[JULIE SANON]

We bring no programs or services into a community that we have not first heard from those residents within the community that this is something that they're interested in seeing within their community. So voice and choice, or that coming to the table together, is important. It's a roundtable. So that table is a table that everybody is welcome at. Everybody's voice is important. Everyone at the table is interested in coming to a solution about whatever it is that we are attempting to solve for. Agape does not have the answer. We're not the savior. We are a part of this process equally.

[J.B. WOGAN]

Hi, I'm J.B. Wogan from Mathematica, and welcome back to On the Evidence.

On this episode, we're going to talk about a cross-sector partnership that enhanced a local community-based organization's capacity to support children and families experiencing poverty in Memphis, Tennessee. That partnership involved the Memphis-based nonprofit, Agape Child and Family Services, as well as Mathematica and the Annie E. Casey Foundation. Lessons from that partnership appear in a new guide for Mathematica and the Annie E. Casey Foundation about how community-based organizations like Agape can use equity-infused rapid-cycle learning to identify and address program needs.

We're fortunate to have an all-star guest lineup for this episode. I'm joined by Allison Holmes, Julie Sanon, and Tosin Shenbanjo. Julie is the Chief Operations Officer at Agape Child and Family Services. Allison is a Senior Research Associate at the Annie E. Casey Foundation, and Tosin is a Researcher at Mathematica and the lead author of the guide I just mentioned.

Welcome, Julie, Allison, and Tosin. Thank you all for being here.

[TOSIN SHENBANJO]

Thanks, J.B.

[JULIE SANON]

Thanks, J.B.

[ALLISON HOLMES]

Thank you for having us.

[J.B. WOGAN]

All right so, Julie, I want to start with you. For listeners who aren't in Memphis, tell us what Agape Child and Family Services does and who you serve.

[JULIE SANON]

So when you think of Agape Child and Family Services, or actually when you think of Memphis, you think of business, you think of basketball, blues, and barbecue. So Agape Child and Family Services is a 53-year-old faith-based, not-for-profit organization. We have a two-gen lens or philosophy wherein we are serving families in three large communities of Memphis through a whole family service delivery model. We have approximately 150 collaborative partners that we work with. We're in 17 different schools, five apartments, and church-based service delivery sites. We are committed to matters of equity, inclusion, and goodness for the families we serve; and we serve with a voice-and-choice foundation to our work.

[J.B. WOGAN]

Okay, excellent, I love that mission. Julie, I know that in 2001, Agape began offering two-generation programs; and you've just referenced that a little bit in your previous answer, but these two-generation programs that serve children and families, or whole families, who are experiencing poverty. So I want you to share a little bit more about the thinking behind that shift. What led to that shift?

[JULIE SANON]

Sure, so in the Memphis community, there was and continues to be a significant amount of children that enter the foster care system. For Agape, our commitment was to go upstream. How do we help to stop the flood of children entering the foster care system? Within those three communities that we serve, they're actually the larger communities where children are entering the foster care system. So our purpose was to be able to walk along with families, empowering them, educating them, but ultimately ensuring that families remain intact. So that is the start of Agape and Agape's power lines community network work.

[J.B. WOGAN]

Okay, great. Julie, I want to make sure our listeners have a really concrete idea of what a two-generation program or strategy looks like in practice. So what are some examples of two-generation programs that Agape offers today?

[JULIE SANON]

So what we call our philosophy of two-gen is a cradle-to-career and beyond. We have several different portals of entry that families come in for services. They may come in from an early childhood language development or brain development standpoint. It may be within those 17 schools that we're working in supporting youth as it regards attendance matters, behavior matters, and parental engagement. It may be from the trained therapists that we have in the three communities working with families, specifically with children in the schools as it regards suicide ideation. It may be safety and security matters. It may be workforce development matters. It may be social capital, economic development. There's just a plethora of different portals of entry.

Most recently, we've started to address matters as it regards hope. So we have staff that are also in those same communities working in collaboration with faith-based organizations in matters of hope for families.

[J.B. WOGAN]

Huh, okay, hope, that's interesting. I haven't heard that before. I hope we can hear a little bit more about those programs focused on hope.

Allison, I want to turn to you now. Since the Annie E. Casey Foundation works with organizations across the country, could you put Agape and its two-gen strategy or two-gen philosophy in the context of the national landscape? Like is Agape's use of two-gen programming emblematic of how many nonprofits operate today? For listeners who don't work in this space, what do you think they should know about Agape's approach and how it compares to others in the field?

[ALLISON HOLMES]

That's such a great question. So if we think about just the nonprofit landscape broadly, it is widely heterogenous. So I think there's just within nonprofits generally, I think there's something that's very emblematic of how Agape is; and that's especially their mission-oriented approach that Julie so beautifully describes. A little bit that I think we'll maybe get into a bit later and part of how Annie E. entered the conversation is how they're grappling with data. That's kind of something that cuts across all nonprofits no matter your size, no matter your sector, no matter what you're doing. I think that's something that's there.

From the two-gen side of things, the two-gen landscape in some ways is almost more diverse because all of those multitude of things that Julie just described for what they're approaching, you could have another two-gen approach or organization that is offering a completely different set of services and approaching families – again, maintaining that holistic lens that you heard Julie describing but in a very different way. So that's sort of the fascinating part of the two-gen landscape is that it is so, so broad; and there are many entry points coming into that entire context. So in some ways, they're just a fantastic example of just the diversity that is there and how you can enter into that space.

I think what makes them really stand out though within that landscape because there are so many different entry points and so many ways that you can approach it – age, programming platforms, all of those different variation points, and especially when I'm thinking of what stands out especially from an evidence-building perspective is their community center focus. You heard a bit about that in their origin story, of how they even came to a two-generation purpose. It really came from flipping that lens and looking at what the community needs and why we should be there and how we should enter this work.

Kind of mirroring that and continued in our conversation is this deepening of an equity lens and practice. So I think that really helps them stand out. In my conversations with them because I fund research an evaluation, so as a grant maker – there's all kinds of grant makers out there – but my work is supporting research and evaluation. So something that also stands out and is something that is something that all two-gen programs are really working on is pushing on data capacity and those opportunities for evidence building. So that's something that is both common but also makes them stand out because of their intense drive to continue to work on that and also explore in that space.

Also something I really want to lift up for this is the overall culture that they have and their willingness to learn, and kind of along that side of learning is also taking risks. So it's a little bit of that double-edged sword, but they really lean into that. So this combination of equity-centered, data-informed, and real like learning culture – all of those things kind of coming together I think sets them apart not only in a nonprofit landscape but as well as the two-gen landscape and how they fit into that universe.

[J.B. WOGAN]

Okay, great. There was one thing that you said that I wanted to follow up about. I think you said something about many nonprofits grapple with data capacity or the need for data. I'm going to infer here – and tell me if I'm wrong – that one of the issues is that nonprofits may not have the staffing or the resources to collect data or analyze data in the way that they would like because they operate on such tight budgets; and yet it would also be something really valuable that if they could have that capacity, they might be able to improve. Is that a fair statement? Tell me a little bit more about what you meant by other nonprofits grappling with data.

[ALLISON HOLMES]

Yes, so data being both a tool and like a mechanism within nonprofit operation – so even when I have a privilege of entering into a space to have opportunity for research and evaluation, one of the first things we need to look into is to see what kind of administrative data – what kind of operational data do you have. I think one of the most interesting things that I've seen is regardless of budget, regardless of size, there is always an ongoing – it's a cyclical process of grappling with various elements of data. So who has it? How do we get it? What are we looking at? Where are we going with it? Why are we here?

And it varies over time. So no matter how large you are or what sector you're in, that element is there because you need data for your operation. You need data for funding. I'm a funder. I'm going to ask you questions about that. So it's part of communications. It's how, whether you're getting it from a funding source or donors, it plays into that. So kind of with that said, I see it across the sector regardless of size that it really comes out.

[J.B. WOGAN]

Okay, perfect. So I do want to get to Tosin and get more into the meat of this new guide for practitioners. But before I do, Julie and Allison, could you speak to what motivated the partnership with Mathematica in the first place? I have a feeling that this is actually going to tie well to that follow-up question I just asked.

[JULIE SANON]

So I'll go first, Allison. As you've shared already, Agape is an organization that strives to be data informed. We look at big data. We look at small data from the standpoint of ultimately, how do we make a difference for the families that we're serving.

So Allison, in several conversations, met with the CEO, David Jordan. From those conversations, there was a mission. There was alignment. From that, we had further conversations about how might we work in collaboration, and we've done that for years now. Then next comes Mathematica; and as it regards how do we improve processes, we're always about that – how do we do it better? Again, not for the sake of just improving but for the sake of the impact to families. Then, how can we communicate that impact to the families as well as to potential funders as well as collaborative partners? How are we able to share those successes both small and large?

So it was a perfect partnership that came about with both Annie E. Casey Foundation and with Mathematica. So I would say from an Agape standpoint, we've been years into the making. We've been in years of training and creating a culture where staff is receptive to data, that there is a need for data, that it's important, and that it helps us on this pathway to be able to document what we're doing.

Lastly, I would say that as we work in collaboration now – even from a national standpoint -- it also helps us be good partners, good thought partners to those that are also traveling in this same way. So that data helps us to be able to help others.

[J.B. WOGAN]

I think we'll talk later about how there are generalizable or universal lessons that are applicable or relevant beyond just the Agape or Memphis context.

Allison, I'm curious, from your standpoint, what's the origin story of this partnership -- why Mathematica, how did this all come together and why?

[ALLISON HOLMES]

Yeah, it really was I think exactly what Julie said. It was an alignment of priorities, and there was a time that we were building a relationship before we entered into work. I just really want to lift that up as an important part of kind of the partnership-building process – that it wasn't an immediate jumping in and trying to push something forward. There was a lot of time of getting to know each other – of what the Foundation was interested in, where are we going – as well as from the Agape standpoint, getting to know Agape. What work are they interested in? Where are their priorities? Where are they? And really building that out over time.

When I look back, it was nearly a year of really sorting out and building that relationship and seeing where a potential opportunity would be. We thought there might be opportunity, and then the pandemic happened; and like many things in the world, our plans were slightly waylaid with that, but I think – this is just my reflection.

Julie, I would definitely be interested in your reflection as well.

But because we had a relationship that we were building – this partnership and our learning goals around – even though we had this external force that was changing the world, we were able to kind of pivot and make an adaptation.

[JULIE SANON]

Yes.

[ALLISON HOLMES]

And there were other learning opportunities that we, again, understood each other that we could then enter into. So I think that was a really important part – how we were able to be responsive to the changes that were happening. Because we had a strong relationship, I think that opened up another opportunity for exploring ways of doing more equity-centered grant making, as something that our Foundation has been exploring what those opportunities could look like. Agape was a really fantastic partner and interested in exploring that as well.

So what that looked like in practice was in this new concept post-pandemic, we're adapting our new idea of what this new learning goal would be and what a project could look like. We developed that concept together. So that was co-created of what this work would be. Then from that, then we co-created the RFP. So a lot of that was listening to Agape, hearing what would be the important parts that were there, put that into word form, and then got feedback. Then we made those adjustments before sending out requests and getting bids for that. Mathematica was on that list.

But the review process was joint. So Agape did their review of all of the responses that we received, and then we made decisions together in selecting a partner for this work. So there was a whole process building up into this before we launched into the project, which I thought is really important for lifting up. I think set up a nice foundation for everything that we were able to achieve in this project.

[J.B. WOGAN]

Okay, thank you for that context.

Go ahead, Julie.

[JULIE SANON]

I would agree. That coming together and coming to the table together was very important.

I think another piece that I would add to that, Allison, is we were ready. I think organizations that want to be a part of this kind of very good work, you have to be ready for it. Throughout the organization, everybody has to be ready to take this journey; and we were at that point.

[ALLISON HOLMES]

Mm-hm, yeah.

[J.B. WOGAN]

Tosin, you've been very patient. I appreciate that. I want to you now.

[TOSIN SHENBANJO]

Yeah, of course.

[J.B. WOGAN]

As I mentioned in the intro, Mathematica has published a new guide describing the way we supported Agape's process of improvement. There are two terms that lay audiences may need you to define in order to understand what is in the brief. Number one, what is rapid-cycle learning; and, number two, what is equity-infused rapid-cycle learning?

[TOSIN SHENBANJO]

Yeah, and I love that, Allison, you were just talking about co-creation because that is a key theme in rapid-cycle learning and equity-infused rapid-cycle learning.

I think I'll start off talking first about traditional research and evaluation. I think that oftentimes for organizations, that can feel pretty siloed and not collaborative. So somebody is typically coming in, and there's a set of research questions that they're answering; and they're doing the methods to answer those question and analyzing the data to understand what's happening. Then, sort of giving that back as a gift to the organization.

I think that with rapid-cycle learning, it really shifts that by making that more of a collaborative and inclusive process. So it's really a way to still identify program opportunities and address them, but in a way that's quicker. So you're testing strategies more quickly to address the opportunities, understanding how those strategies worked, taking that data, refining the strategies, and doing that process again. So the timeline is a little bit shorter than what traditional evaluation typically relies on. Then it's collaborative; so the process of identifying what those program opportunities are and developing the strategies to address the is a collaborative process across different partners, across different organizations, and very co-creative as Allison was describing with the partnership between Casey and Agape when that started.

So with equity-infused rapid-cycle learning, rapid-cycle learning generally already includes inclusion principles, especially those around collaboration and co-creation. But equity-infused rapid-cycle learning basically takes that a step further by more intentionally and formally integrating principles around equity or principles around inclusion.

For example, for us with our work with Agape, when we started to think about what some of the program challenges and opportunities were that we wanted to address through this work with Casey and Agape, we collaborated with Agape staff members, participants, and community partners to bring them to the table to understand what their perspectives were about challenges. With equity-infused rapid-cycle learning, you're embedding those principles throughout multiple stages of the rapid-cycle learning process – including before it actually begins.

So for example, one thing we talk about in our guide is thinking through before you start power imbalances and how those power imbalances can affect the collaboration that's going to take place through rapid-cycle learning and then thinking through ways to address those power imbalances in a way that will allow all of the people at the table to contribute their expertise meaningfully.

So I think that at its core, rapid-cycle learning basically allows people who are using the method to address opportunities in the right way for the right people and thinking through how to bring those people to the table to address those opportunity areas.

[J.B. WOGAN]

Okay, perfect, so with those definitions out of the way, let's get into some of the specifics.

Julie and Tosin, could you speak to some of the ways that you infused equity in the process of learning and improving programs that Agape runs. Julie, since Tosin just went, perhaps we can start with you; and then I'd love to hear from others, including Tosin, about additional specifics or examples.

[JULIE SANON]

Again, I will add that Agape has a mantra of voice and choice. So we bring no programs or services into a community that we have not first heard from those residents within the community that this is something that they're interested in seeing within their community. So voice and choice, or that coming to the table together, is important. It's a roundtable. So that table is a table that everybody is welcome at. Everybody's voice is important. Everyone at the table is interested in coming to a solution about whatever it is that we are attempting to solve for. Agape does not have the answer. We're not the savior. We are a part of this process equally.

So as it regards our work with Mathematica, with both our Stars effort as well as our TeamWorks effort – and I say Stars, that is our in-school effort where we're in 17 different schools. TeamWorks is our workforce development effort where those that are seeking gainful employment, suitable employment so that they're able to more appropriately take care of their families, are able to do so. It's bringing them to the table. It's ensuring that what they're seeking to have answers to that we're hearing their voice – that we're hearing their voice, that we're getting their information in this process, that the feedback comes to them.

So everybody continuously has this information. It's this trial and error, but we're doing it together. That is what I would offer, and that's the uniqueness. And Mathematica was there at the table as well to help walk through this process, which made it, I believe, more comfortable for the families that we were serving.

Tosin, you might add to this; but it wasn't just an Agape and "them," if you would; but Mathematica was there also, in some ways maybe even as a buffer for them. But that would be what I would add.

Tosin?

[TOSIN SHENBANJO]

Yeah, I completely agree, Julie.

I think for me to answer your question, J.B., that is at the top of my mind as really throughout the process we were bringing additional voices to the conversation, bringing as many people as we could to the table.

For example, when we first started the work with Stars and TeamWorks, we wanted to understand what were some of the challenges that families and staff were experiencing as a result of the pandemic. So early on in the work to help us understand what was going on, we worked with Agape leadership to identify Agape staff members, participants, and community partners who could be part of a workgroup, which was basically an advisory group that was helping us set up and inform the work.

So the workgroup helped us understand what some of those challenges and program opportunities were. So we had a series of workgroup meetings with them where we were doing that, and that was really important for setting the stage for making sure that the work we would continue to do and the strategies we would develop were aligned to the right opportunity areas.

So it was really important for us, like you were saying, Julie, to not just have staff there but to also have community partners there that worked with Stars and TeamWorks who have their own unique perspectives about what's challenging due to COVID. It was very important for us to have participants there as well because they were the ones who were directly affected by programming and who have those close relationships with staff members. So that was one way that we wanted to make sure that we were bringing those voices to the conversation.

Then when we started to develop those strategies to address the program challenges and test them and then collect data to see how they were working, we also engaged Agape staff throughout that process. We kind of called them in our guide "champions of change." So we had two implementation teams -- one for Stars, one for TeamWorks – that included staff across levels from both of programs. They were involved in helping us develop the strategies that we were going to test, refine them, think about how we were going to test them, help us refine our approach to collecting data which was an important piece of this work as Allison mentioned, and then also interpreting the data.

So we partnered with those implementation teams throughout, and I think it was important to partner with them not just to formally carry out that equity principle because they're the experts of this work. They're the experts with participants and what's going on, on the ground. So those are two examples of ways that we brought those people to the table.

I think that another thing that we were sure to do as well was after we had those conversations about what was challenging, we wanted to make sure that there was opportunity for more staff members and participants to offer additional perspectives about challenges. So after those additional workgroup meetings, we had additional conversations through interviews with staff members and with participants to get more clarity on the challenges and, again, to make sure that we were focusing on the right program opportunities at that time.

I think that that leads me to sort of the second point that I want to raise up to answer your question, J.B., about how we were including those equity principles. I think for us it was really important to have root cause conversations at the beginning of this work. So I think that when there are challenges that programs are trying to address, they look one way at the surface; but then upon having continued conversations and assessing root causes, they might look another way.

So with our work group, we were having those root cause conversations; and we were facilitating different activities to make sure that the staff and the participants and the community partners could meaningfully contribute their ideas about what they thought were the root causes of the challenges. I think that one key takeaway for us was that these root cause conversations allowed us to make sure that our challenges didn't have the deficit focus perspective because I think initially, we were talking about how family engagement was challenging during COVID, which is understandable because they were stretched thin and families had a lot going on.

But then in having those root cause conversations with everyone at the table, we were able to get down to what Agape could do to address family engagement and to make sure that our perspectives around what was challenging weren't focused on what families were or weren't doing but were focused on what Agape as a collective organization could do to address those challenges. So that's another point I wanted to highlight.

Julie, I don't know if you have anything that you want to add to that.

[JULIE SANON]

That's good. That's good, Tosin. I agree.

[J.B. WOGAN]

Allison, before I move on to the next question, is there anything as you were listening to Tosin and Julie speak, was there anything that it brought to mind to you. Anything that you wanted to flag from your observations with this work?

[ALLISON HOLMES]

I think maybe just two points. So, one, there's like a really key theme as you listen to both Julie and Tosin describe when the programmatic effort as well as the evaluation approach, that when we're talking about bringing in these different perspectives and hearing the community voice, the equity principle in that is addressing blind spots. So just to really underscore that. There was such a richness in being able to combine what we were seeing at the programmatic level and how Agape is approaching their work and then being able to combine that with the evaluation approach and really bringing this 360 perspective to round everything out and cover potential blind spots because each person is bringing different perspectives given where they sit and what their experiences are.

So I just wanted to underscore that because this is really what is valued in this work and what they're describing. So I just wanted to underscore that part.

Then maybe just to highlight a piece of those findings that we had because it's just such a pandemic story. But we approached this project as an opportunity to look at some of the adaptations that Agape was making in response to the pandemic. A lot of the issues that were revealed were not new issues, right? These were just things that had been existing, but the pandemic was really just shining a light on things that were already there. We saw this in so many other circumstances across the country in human experiences and program evaluation things. But I just really wanted to call that out.

But taking this deeper dive learning experience where we were able to do that and, like Tosin was describing, when we got to the root cause, again, it was a thing that was already there. When we pushed even farther in a proactive way, there were ways that we could address that. It was something new; it was just something that was lifted up. So I just wanted to underscore that as well, that this was a theme throughout the pandemic that I think was really perfectly encapsulated in the first part of the project.

[J.B. WOGAN]

I don't know if this is what you're getting at, but I know in some of our past episodes guests have talked about during the pandemic some of the adaptations that had to take place in Human Services programs addressed issues that actually were there before in terms of needing to go – you were required to go in person to meet with a case manager for a certain program; or you needed to go to classes that were in-person and suddenly having the flexibility to do things online, not having to commute, was much better tailored to the other needs that that person might have in terms of family or work or education.

It seems like those are some changes that would make sense to maintain outside of a pandemic context.

[JULIE SANON]

Absolutely, and leveraging technology with online classes or as it regards virtual types of life skills classes that were conducted as opposed to families coming to a site.

[J.B. WOGAN]

Mm-hm.

[JULIE SANON]

Because that's a transportation issue. Well, you eliminate the transportation issue altogether when it's a virtual environment.

[J.B. WOGAN]

Yeah, okay, so this next question is for everybody. The guide equips local organizations with tools for continuous improvement that is grounded in equitable approaches. I'm curious, and I'm sure our listeners will be as well, after Agape adopted these strategies, what changed as a result of the equity-focused rapid-cycle learning? Are you seeing measurable improvements for the families that Agape is serving? Are there clear improvements in the way Agape's two-gen programs operate today?

Julie, I thought maybe we would start with you for this one.

[JULIE SANON]

Sure, well I can say it has changed the way in which Agape does business in a lot of ways. For instance, something as routine as staffing is different now because of the questions that are being asked when we talk about data and our performance quality improvement team and the questions that are asked about, as Tosin said, what are we collecting? Why are we collecting it? When are we collecting it? What are we doing with it? How are we getting it out to everybody that we collected it? How are we saying that this has moved us in a direction for the positive for families?

It has changed all of that. Agape is an organization – we serve on average about 3,200 individuals a year, about 1,300-1,400 families a year. So when we talk about that two-gen model, families come in and out of portals. So if I am surveyed in this portal and surveyed in that portal and surveyed in that portal, at some point I stop wanting to be a part of the survey. Well, sometimes we don't think about that. So it's just asking the question...have you been surveyed by Agape in the past 30 days?

All of those kinds of things – it's a quicker assessment. We're looking at it quicker as opposed to looking at it from the standpoint of doing this work and seeing the fruition or the results of the work way over here. I'm a hands person, so you can't see my hands. But it's in smaller bites. It's in smaller bites making impact with that family's voice and choice and collaborative partners and leveraging those partners for families.

So I would say it's been a part of everything. We have weekly staffing. It's asking the question of the staff about...what are you getting out of this? Why do we do it? Do we do it virtual, or do we do it in-person? It's just a part of everything Agape is doing now to be able to get real-time, quick assessments on what we're doing. Not to say that there are not some matters that need a longer-term assessment. But we now think about it with a different lens is what I would add.

[J.B. WOGAN]

Okay, yeah, the omission of hands -- not being able to see hands as you're speaking, is a downside of the podcast. But I've got to tell you, I find people are more animated. You need to hear the animation, the energy, in your voice when you're using your hands. So people can detect that passion.

Tosin and Allison, I want to put the question to you too. What would you say are the measurable differences? How do you think about what's changed as a result of this work for Agape and for the families that it serves?

[TOSIN SHENBANJO]

I will go ahead and jump in.

Julie, what you were saying about how this is helping Agape ask the questions, I also think that this has allowed Agape staff to learn how to answer these questions in new ways. So I think that this rapid-cycle learning process and then the equity-infused rapid-cycle learning process has really allowed staff to think about new ways to ask questions about program challenges and opportunities. It's taught them new ways to think through strategies for addressing those opportunities, and then collecting data and analyzing data to understand how those strategies work.

So with our work with Agape, we really relied on different human-centered design activities, which are just activities focused on collaborating to generate ideas and to brainstorm. For example, I think that when it comes to addressing program challenges or program opportunities, Agape has typically had its own way of doing that. But with this rapid-cycle learning process, we introduced staff members and families and partners to a rosebud/thorn activity where they could think about different ways to identify those challenges and then similarly including different activities that they could use to identify strategies to address those program opportunities and then collaborate together to understand data that showed how those strategies worked.

So I think that this work has really allowed staff to build their capacity around program improvement, around collecting and generating data and understanding how data work, in a way like Julie said that maybe happens more quickly than traditional research or evaluation. So that's one thing for me that I wanted to elevate.

Another thing that I'd add is related to our families, Agape's families. With the Stars and TeamWorks work, we had conversations with families to understand how the strategies worked for them and what their perspectives were about successes or opportunities. With TeamWorks, we were really focused on helping the staff members rethink how they or having conversations about adult participants' goals around work and around education. So in the conversations that we had with families afterward, we learned that the strategies that we developed with Teamwork staff did allow them to feel like the goal conversations that they were leading were really driven by participants themselves, which was one of our goals in developing that strategy.

So I think that the rapid-cycle learning work that we did has allowed Agape staff members to think of strategies that can help participants feel more engaged in programming. So that's one thing that I wanted to offer as well.

[J.B. WOGAN]

Tosin, could you clarify what a rosebud and thorn activity? What do you mean by rosebud/thorn activities?

[TOSIN SHENBANJO]

So rose thorn/bud is an opportunity for organizations or whoever is using the at this time to think through roses, so things that are going well. So for us, it was things that were going well with programming. Buds were opportunities, and thorns were things that weren't going well or challenging. So we used that way to more collaboratively engage the participants and the staff members and the partners in ways of thinking through program challenges.

So we then took what some of the buds and thorns were, and we did some additional human-centered design activities to help them collaboratively brainstorm on how to address those opportunities. So that's what rose/thorn/bud activities are.

[J.B. WOGAN]

Okay.

So, Allison, I think we haven't gotten to you yet about how you would define change or how you think about clear improvements as a result of this work for Agape and the families that it serves.

[ALLISON HOLMES]

Yeah, I think what Julie was describing is a really key part in this. I would just reflect back. I really feel like a reinforcement of some of those key characteristics that I listed up about Agape earlier around their equity-centered lens and their data-informed and their learning culture. So I really see a deepening of all of those things. I think what Julie was describing is sort of the embodiment of what that looks like in practice for them.

I think the thing that's really fascinating about this – and I think it's a really key takeaway from this work and from what both Julie and Tosin were describing – is we have evaluation findings. We have specific program change that's there. But what you're also hearing is this overall capacity building and a deepening in culture as a result of an evidence-building project. I think that's a really just key part to highlight within this, especially given organizations may have a really wide range of experiences with research and evaluation partners which can run a really wide range in terms of whether that was positive or negative or where they as an organization is actually seeing benefit for themselves.

So in this, the findings for the programs that you're hearing both Julie and Tosin describe, they're seeing that. Like that has impact for how TeamWorks is actually able to connect with their clients as well as some of the collaboration that we're seeing within Stars. We're seeing program things there. And there's an enhancement. There's a capacity building. There's a deepening of how the organization is able to operate and changes in their culture. I think that's just something to really underline within this for what evaluation and evidence-building efforts can be – that it can be an enhancing experience.

There may be really negative paradigms and frames or other experiences that people may have had, but there are other ways that it can be. So I just wanted to really underline that.

[J.B. WOGAN]

Okay, well that's – oh, go ahead, Tosin.

[TOSIN SHENBANJO]

I agree, Allison. I just wanted to quickly add, Allison, that I agree. I think to your last point about those enhancement conversations that the root cause conversations that we were having with staff members and families and partners were showing staff what that can look like in practice. Like with our facilitating those activities with them, staff were seeing in real time what it can look like to really dig into what root causes are challenge, to really dig into what it means to be more asset-focused with our understanding or perspectives around what families are experiencing rather than deficit focus. So I think that that's another important piece here too – that staffers seeing what that can look like in practice.

[JULIE SANON]

Yes, and I would just add to it that we saw and continued to see a lot of aha moments. You know, intentions are to do the right thing; but sometimes, as you talk about life, the asset-focused piece of it and even that language – when you invite someone to the table, language is language. Even if it's not offensive, it certainly is not one that communicates that we're equal around this roundtable. So there were a lot of aha moments for us throughout this whole process.

[J.B. WOGAN]

So you're already headed in a good direction, kind of where I wanted to go next, because you're talking about what can be learned – what other organizations can learn from the partnership and the experience here. This is something that I wanted to ask each of you about. We're talking about the experiences of one nonprofit in one place. What can other organizations apply in their practices based on Agape's using equity-focused rapid-cycle learning?

Allison, I thought maybe we would start with you and then go around the table.

[ALLISON HOLMES]

Sure, so it's really interesting reflecting again on the larger arc of this body of work and the partnerships throughout this and what an organization may take away, as you're hearing about this arc in the experience. Something that really came to mind for me was this opportunity was only really possible because – and, Julie, you named part of it, that there was a readiness component in being able to pivot into this. And then I had named before we had a relationship and a foundation there – and again, I'm talking with my hands. You all are missing all the goodness because I talk with my hands.

But the other part of it is that there was a ready bank of learning questions that was also available. So we were able to make that pivot because in that learning culture there was a set of learning questions that was available that we could build on. Without that, we would have missed an opportunity to kind of pivot in and develop this project and move anything forward. So that's pre digging into everybody, but it's a really key part because you can't start a project if you don't know what you're trying to learn.

So I would actually lift that up as something. Some organizations like to have formal learning agendas. Others just have – there's lots of different tools or templates or anything, and it doesn't have to be formal. In this case, it wasn't. It was implicit. It was just a collection of facts that we were able to use; but that was a really, really component for what was the success of this. That prework, again, before you're diving into a specific project is really important. So that element of the prework is really key.

So I would lift that up for any organization that's considering how they might be going into everything. There may not be an immediate opportunity, but think about what are you curious about. What are those things that you want to know? Where are some o f those gaps? It doesn't have to be super formal. But if you start writing it down -- and, again, you're collecting, getting all of those perspectives -- then you will be ready in multiple ways to take advantage of opportunities that may come.

[J.B. WOGAN]

Julie, I want to hear from you. I know you actually referenced earlier that having the data capacity, having the ability to sort of – that one of the benefits is others can learn from you, can learn from your data. So, yeah, talk a little bit about how other cans can learn from what you're doing.

[JULIE SANON]

This may not answer that question, but I have this in my heart to say; so I'll say it (laughing). When you initially asked your first question, I would say David as the CEO, myself as the COO, and a lot of the leadership within Agape's organization – for us a learning and a learning to be able to impart to other organizations would be to get out of the way.

We have a role and responsibility to be able to shore up the staff with whatever they need in order to be successful – be that information, training, tools, resources, or whatever – but then to get out of the way and allow the staff and all of those that are invited to the table to be able to have their conversations. That's not to say that we don't walk away from it, but we're there for the ask. I think oftentimes when leadership gets involved in it, everybody then defers to leadership. So we were very intentional about not being in those conversations so that it could be authentic. It could be honoring of the families and of the staff and the collaborative partners.

So I'm not sure, J.B., if that answered the question; but I wanted to say that because I think it's important for leadership to understand their role in this process. That's what I wanted to share.

[J.B. WOGAN]

No, that's great. That addresses the overall intent of the question. Also, we always advise when our staff are talking to journalists to answer the question you wished you'd been asked. So I would be a hypocrite if I didn't respect that approach here.

Tosin, I want to give you an opportunity to answer that question as well.

[TOSIN SHENBANJO]

Yeah, a couple of things that I think other organizations can take away from this – and these are a little bit more concrete. But I think first is really considering interviews or data parties as a concrete way to bring additional voices into the conversation and then to just make sure that you're program improvement work is data-driven.

Julie, you started off our time today really speaking about the importance of data for Agape. I think with rapid-cycle learning, we used data to understand what we should address through those conversations with the work group. We collected more data to get clarity on what some of those program challenges were. Then once we developed the strategies based on those challenges, we collected more data; and we hosted data debriefings with the Stars and TeamWorks staff to really get their expertise, their contextual understanding, about what the data were showing us and how we could use that information to inform our next round of testing – where we refined the strategy and tested again and collected more data.

So I think as organizations are thinking of possibly learning rapid-cycle learning and making it equity-infused, that having those conversations – formal interviews or not formal – with staff members, with parts, with families, and then bringing those same collaborators to the table to understand data is something that they can take away from this project and apply to their own practice.

I think another thing too that we've talked about is that other organizations can really use this as an example to rethink how they are identifying program challenges, how they're identifying program opportunities, and then how they're addressing them. So I think part of this whole conversation has been how we use rapid-cycle learning and equity-focused rapid-cycle learning to help Agape get at the same goals that they might with a traditional evaluation. It just happens on a different timeline and involves different people and uses different methods.

So I think that this is a lesson in the possibilities that rethinking eval and research can give to organizations, whether that's through rapid-cycle learning or another method. Then rethinking the ways that you can engage your families, rethinking how they can be involved in conversations around understanding challenges and potentially even helping to develop strategies to address them. So those are just two things that I wanted to lift up about that question.

[J.B. WOGAN]

Okay, I want to wrap up by zooming out of it and looking to the future – and here I am using my hands too. Setting aside the specifics of this collaboration in Memphis and even setting aside the more general context about advancing equity in two-generation programs, I know from some of our conversations from before today's recording that you feel this work has shifted the way you collaborative in other spaces and with other partners. I want to hear more about that. What was special about this partnership between your respective organizations and, to tie this back a little to the previous questions, are there components of this collaboration that could be replicated and applied in other contexts and other communities?

Tosin, maybe we can stick with you and have you take a first crack at this question.

[TOSIN SHENBANJO]

Yeah, I think that one thing that was different – so Allison and Julie were talking a little bit about this work started. Before we got evolved with this rapid-cycle learning work with Casey and with Agape, Mathematica actually helped Agape conduct an implementation study to understand how Agape's two-gen initiatives were implemented. So what did they look like on the ground? What were some successes associated with the programming, and what were challenges or just opportunities to just strengthen them?

So we had that relationship built with Agape, and Allison actually attended our presentation where we were presenting our findings for the implementation study. So some of those relationships were either built or were starting to build, which I think did help facilitate our work together citizen relationship building and trust is a key component of equity-focused work. So I think that that's something that stands out about this partnership.

Another thing that I also want to raise, and I'd be curious for Julie's perspectives about this too, is that Allison as a grant maker has been a very active thought partner in this work. I think that is something that is very special about this project and is not always the case for me as someone who is day-to-day doing research. So not just grant making and checking in, but also actively helping us understand potential opportunities to strengthen the work, how to address potential roadblocks. She's also helped us think through ways to further all the equity in our work, ways to think more about families and how to bring them to the table.

So that's really stood out to me as somebody who's worked with many different grant makers – is just Allison's active thought partnership, and it's been very helpful because she's in those conversations in the field with other grantees. So she's been able to bring that and that expertise and context into our work, which I think has been really helpful.

[JULIE SANON]

I agree, and I think it's very collaborative. All of the meetings that I've been a part of, it's been very, very collaborative. She has come in, and it's been a "come let us reason together" kind of thing for the good of what we're trying to accomplish. I have found that to be very, very empowering because like you say, Tosin, you're in some of these conversations and it's more directed. So it has been very, very collaborative.

And I thank you, Allison, for what you've for Agape.

Tosin, what you've done for Agape through your organizations.

Agape is the better. Our commitment is to serving families. We want to ensure that in the Memphis communities families can move from where they are to where they want to be, and we want to be a contributor towards eliminating poverty in the Memphis community. Those are big, big, audacious goals. But what we've learned throughout this process I think that it's important for listeners to hear is if you eat that elephant one bite at a time, you will eventually get there. It may not happen tomorrow; but if you systematically tackle those issues, you'll move forward towards it.

So that is what we're doing. That is what we're doing through the help of this collaboration. We have seen significant successes as it regards our two-gen philosophy and the work that we're doing throughout that whole paradigm. So this has been very, very helpful for our team, for our organization, even for our collaborative partners within the community. Huge gains, short-term innovations, long-term change and growth is what we're seeing.

[J.B. WOGAN]

Allison, I can't tell on the camera if you're blushing. I think the answer was what was special was you. I remember when we were having a pre call, Allison. You were really passionate about wanting to talk a little bit about this and about what people could learn about the nature of this novel partnership. So in addition to the secret sauce that you bring, what else would you want to call out or highlight about what was special about this partnership?

[ALLISON HOLMES]

I am so appreciative of all of the kinds words, and I have thoroughly enjoyed the partnership with our Mathematica team and the Agape team. You all are so generous.

I'll push back a little bit that I don't know that the secret sauce is me, but I reflect on this and also other examples that I have just in my role in the Foundation of what can funding look like when you lean into equity-centered grantmaking as a research funder. You know, what does that look like? It's slightly different than other types of funding. So what can that look like?

Bringing partners to the table if we're willing to explore that over a length of time and learning together within that has really just been a gift and such privilege to have folks who are willing to kind of walk along this and explore this different ways. So I think something that's kind of special about this is having partners really come together who are willing to try something new and to have this example that grantmaking and research and evaluation, funding and partnerships, doesn't have to look a specific way. It can adapt over time and meet people where they are. All of the people at the table, all of the different actors and organizations, we can adjust to that.

So just a little bit of maybe blowing up the box. Maybe the box isn't there. We can try something – we can try something new. So I just, again, I'm just so appreciative of having partners who are willing to come together and just explore in that space because that isn't always the case around all sides.

So something I really try to reflect on – just what is my role as a funder in this, right? I mean, there's lots of different parameters and factors in this. Sometimes funders are (whispering) a bit of a pain to work with. We create problems. So I tried to reflect a little bit on that. So what is my role in this partnership? What should I do? What should I not do? What are constraints I can release? What are parameters in my role as a funder that I can boost? And really, when do I need to get out of the way?

Because the work is on the ground. We have a really strong evaluation team and amazing programmatic partners. So what is my role in sort of navigating that? But in thinking about again if this doesn't have to look a specific way, what do I do? How can I foster a space for evaluation and program teams to really come together and blossom to do the work that enhances everyone coming together?

I don't know that I have that answer for any funder that's listening, but this is just something that I've been trying to explore. But for any organization, whether you're on a research and evaluation team or any programmatic community partners, the one thing I would just lift up for taking forward is just ask. As your funder. Ask if this is something that has to happen. Because a lot of times, there might be flexibility; but if you don't ask, you may not know.

So if your funder isn't leading that and structuring a relationship differently, you could just ask. That would really be a nice push. There's a lot of discussion in philanthropy for doing things differently, but come to the table and ask those questions. Do I really need to send you that report? Is that really required? Is that going to help? Do I need to have this? Could I bring my community partner as part of my RFP response? Could I have more time if I do that? Because that aligns with our mission and value. I see that in your mission and values as well. Is there an opportunity for that?

Just ask those questions and see where that takes you.

[J.B. WOGAN]

I noticed earlier in the conversation, it almost sounded like there's an analogy with dating – where you get to know each other, then you go on some dates and you decide whether you want to go steady and maybe go on to marriage. But it sounds to me like in those early stages where you're developing trust and learning about one another, that seems to be part of the secret sauce here.

[JULIE SANON]

Oh, yes, yes.

[ALLISON HOLMES]

Yes, I agree. And Mathematica and Agape have been dating for a while.

[J.B. WOGAN]

I think we have the preview clip now.

[Laughter]

[J.B. WOGAN]

I think this is a great note to end on. Julie, Allison, and Tosin, thanks again for a terrific conversation.

I also want to thank our audience for listening to another episode of On the Evidence, the Mathematica podcast. This episode was produced by the inimitable Rick Stoddard. You can catch future episodes of the podcast by following us on Twitter, Spotify, Apple podcasts; or you can visit us at www.Mathematica.org.