Field Tests

Across the field tests, audiences reacted positively to messages that felt unscripted and authentic, rather than overly polished. Many  leaders had success drawing on the real stories and experiences of their messengers to move their audiences.  

Community of Practice members found that it was particularly important for teachers to hear from other teachers, who can speak to their shared experiences as educators and help address some of the concerns teachers may have about being asked to try new approaches. Another powerful messenger for the teacher audience was students sharing their emotional experiences learning math, especially when they shared accounts of how teacher practices helped or hindered their math learning.  

Members of  the Community of Practice found that students reacted positively to messages that they felt were authentic and matched their experiences and emotions, especially when messages were delivered by peer and near-peer (or near-in-age) students. 

In the context of AI-enabled tools, students had more positive reactions when the tools weren’t inundating students with false optimism or clichéd encouragement but instead acknowledged that struggling when learning something new can feel hard but is a normal and often necessary part of math learning. AI tool designers learned in early testing that students might hesitate to ask for help or become discouraged when they don’t get something right, so they trained their AI tools to offer help at the moment students started to struggle, rather than waiting for students to ask. 

Key Examples from the Field Tests: 

• The Bob Moses Center for Math Literacy Through Public Education and The Young People’s Project used near-peer student messengers to successfully reach 6th and 7th graders in an after-school math learning program. 
• In their field tests, Goblins and The Emancipation Group both honed their AI tools to intervene early on when students struggled, rather than waiting for students to seek help. 
• OKO incorporated the messaging recommendations into their AI tool and worked to ensure they didn’t rely on clichés or empty praise, which can feel inauthentic for students. 
• PERTS and WestEd, who developed video messaging for teachers, noted that ensuring their messages authentically captured the classroom experiences of teachers was essential to connecting with teacher audiences.

Rather than creating new opportunities to engage with their messages, Community of Practice members found success in building narrative interventions that embedded messaging into the tools and products they were already using to engage their audiences. 

As with the other findings, this theme looked different across contexts. Teachers are busy and inundated with materials and advice, parents have countless demands on their time and are often influenced by their own, often negative, feelings and experiences when it comes to math learning. And students are often overwhelmed and experiencing strong emotions when it comes to math learning. Accordingly, the most effective interventions leveraged existing or low-friction points of contact with these audiences: a podcast about math learning that and then discussed at a professional learning session, messages embedded within a lesson-planning tool for teachers, and a host of messages reaching students while they were actively engaged in math learning through digital products or during after-school programming or activities. 

Key Examples from the Field Tests: 

• Professional learning providers at the Partnership for Los Angeles Schools incorporated video messages from students into their existing professional learning offerings for the teachers they serve. 
• OpenStax layered the messaging recommendations into their AI-powered lesson-planning tool for teachers.
• Impact Florida collected teacher voices as part of a podcast series, which they integrated into their ongoing professional learning cohorts in their state.
• MyVillage Project embedded recommendation-aligned messaging into Count Dash, a student-designed math learning game powered by AI, as well as other ongoing work with near-peers, caregivers, and students.

Many in the Community of Practice found that when incorporating the Math Narrative Project messaging recommendations into their services, products, and tools, it was helpful to build in opportunities for reflection to help guide audiences toward the desired changes in beliefs and behavior. For example, field test leaders facilitated guided discussions asking audiences to explore and examine their thinking, feelings, and behaviors, crafted written prompts that helped audiences explore the Math Narrative Project recommendations independently, or explicitly included a message calling for the audience to reflect on their own beliefs and behaviors.

The opportunity to reflect is often an important precursor to shifting beliefs. Messaging interventions that suggest different practices can be interpreted by audiences as implicit criticism of their current practices and trigger defensiveness, rather than a willingness to change. Across several field tests, leaders found these reflection practices were effective in helping reduce that risk, especially among teachers.  

Key Examples from the Field Tests: 

• BranchED incorporated reflection into their professional learning series for teachers and captured some teacher reflections on video. 
• TeachFX used the messaging recommendations to inform a professional learning series that was supplemented by automated feedback on teacher practices and found reflection to be essential in driving changes in beliefs and behaviors. 
• Bank Street used the messaging recommendations to refine their professional learning series for teachers including reflection prompts to help guide conversations with teachers.   

Across the Community of Practice, members found that repeated exposure to their narrative interventions was particularly impactful in driving changes in beliefs and behaviors. In other words, one message was rarely enough. Given that the interventions aim to change long-lasting and entrenched narratives about math learning and change attitudes and behaviors that may be years or even decades in the making, it follows that messaging efforts should also be sustained. This insight aligns with the finding that embedding messaging into existing efforts is helpful. By delivering messaging repeatedly, through products, tools, and programming, audiences can be consistently exposed to positive math narratives over the long term. 

For Community of Practice members using AI tools, they saw positive correlations between increased student interaction with their tools and a range of metrics measuring positive beliefs around math learning and increased engagement and persistence in solving math problems. In the professional learning context, multiple field tests found it was important to engage with teachers multiple times over the course of the semester, and planned for future engagements to span the full year or more. 

Key Examples from the Field Tests: 

• EdLight delivered messaging through their AI chatbot, Ember, and found that students with 6+ interactions with the chatbot showed improved feelings and beliefs about math learning as measured by survey questions while users who interacted with the AI chatbot 2-5 times remained flat. 
• In their respective professional learning work, ConnectED and Teaching Lab each saw measurable progress in teachers’ perceptions of students as capable, in part by ensuring teachers received their key messages multiple times across several engagements. 
• MathTrust found that repeated use of their student messenger videos over a four-month period of conducting professional learning trainings helped to measurably change teacher perceptions of Black students’ behaviors and teachers’ self-perceptions of their ability to engage their Black students in math learning.
• FHI 360 tested and refined their own messages for parents, and plan to implement their new messaging into multiple points of contact moving forward.