Monday, May 4, 2020

Finding the Right Direction: How Measuring Engagement Changed Everything


This post is written by CVU Principal Adam Bunting:

My mother spent her career as a family systems psychologist. I always admired her work, particularly the deep respect she held for her clients. Just as great educators know about their students, she understood that her clients needed to be the sense makers. She would say things like, “Ultimately, they are the ones who do the work, not me.” I remember asking her once if she ever got frustrated with people who repeatedly made the same mistakes, and she had this to say, “You know...people work really hard...it’s just sometimes they work hard in the wrong direction.”

As a principal, I return to her phrase now and again--especially when the work feels personal and complex. I ask myself, How am I working hard in the wrong direction? I held my mother’s words especially close in 2018 when two events dominated the educational landscape: the Parkland shootings and an increased focus on the devastation of opioid addiction in Vermont families. Our work was well intentioned, but much of it--focused on deficit models--pulled us in the wrong direction of reinforcing walls and policies instead of the relationships that undergird health, connection, and engagement. Current events demand a similar frame as we boil away the superfluous and distill the healthiest educational experience we can for our young people. At CVU, we have landed on this mantra to guide us through this pandemic: connection first, engagement second, and academic learning third.

Connection is more readily described, but defining engagement is no small feat. Like many educators, my understanding of engagement evolved as my respect for my students deepened. I began my career seeing engagement as an individual choice a student makes in our classrooms. Later, I saw engagement as an emotional state where our learners find themselves as receptive, open, and curious. Then I saw engagement more as a product of the systems around us and the conditions of our lives and learning spaces. 

Our school has chosen to define engagement just as we might define happiness or flow--a state of being when humans are both more receptive to and more motivated to seek new learning. We cannot force engagement, but we can ensure the soil from which it grows has the proper nutrients: physical and emotional safety, meaningful adult relationships, meaningful peer relationships, balanced amounts of stress, connection to a purpose larger than self, small successes and appropriate challenges, and rich and relevant content. More pragmatically, engagement may be a school’s most important indicator for wellness and successful learning, and it is something we can measure by examining the conditions that create it.

So it was two years ago we launched the Engagement Survey at CVU--an attempt to map the engagement of all our students. For those who were disconnected, we wanted to build connections, and for those who were thriving, we wanted to understand and grow those conditions. We created simple likert scales asking students to strongly agree, agree, disagree or strongly disagree to prompts like…
  • I have friends at school.
  • The adults at school care about me.
  • I believe school is preparing me well for the future.
  • Life outside of school feels calm and manageable.
  • There is at least one adult I could talk to if I needed help.
  • I feel like my input is valued and voice is heard at school.
  • I experience the right amount of challenge.
Unlike any survey we had ever given, this one was non-anonymous, and it needed to be. In fact, the more times we administered these questions, the more we came to view anonymous surveys with a certain derision. What does it mean when we ask students highly personal questions and never follow up? With some serious sweat on the part of our technology integrationist and IT staff, we administered the survey through our advisory system. In addition, we sent the survey to parents and advisors--asking them to predict what their student might say to a given prompt.

What we got back changed the way I saw our school--a place where I’ve been a student, teacher, and administrator since 1990.

The immediate responses to the survey were powerfully positive and underscored a belief that has been long held at CVU: authentic relationships are the backbone of a learning organization. The subtext was poignant unto itself. Giving the survey alone says we care about your real feelings, experiences, and thoughts enough to ask and act upon them. We determined our approach would be one of curiosity and an assets orientation, knowing the data was only as good as the dialogue it inspired.

One of the best things we did was ask students to list at least one of their “go-to” adults by name. Within a few hours of viewing the responses, our advisors began sending emails to one another and to our staff. I got one that read, “You may not know this, but you are [student name’s] trusted adult.” Our staff were often the ones who had the most mentions of being the trusted adults. Anyone who doesn’t value administrative assistants, campus supervisors, or tutors needs only to see the data! I will admit it - I used the control-find function to see who indicated I was their go-to. I was surprised about how good it felt to know kids really counted on me. As one teacher said to me in the hall: “I had no idea I was that important to [student name]. It’s changing how I interact with him.”

Of course, as happy as I was that the survey fostered relationships between faculty and students (and helped teachers see the holistic student--not just the science or English learner), I was equally disturbed by the group of students who indicated they didn’t have any trusted adult. The percentage was less than 1%. Normally, I would have pointed to that statistic and exclaimed, ”Look at what a great job we’re doing!” Now, however, the percentages were connected to names. I can overlook the significance of numbers--but not the names of individual students. We implemented a few interventions, but my favorite was also the most basic: a group of our faculty members took it upon themselves to smile and say hello to our kids who expressed disconnection. As someone said to me recently, the biggest problems don’t always require big solutions; sometimes simple solutions work best.

Another problem was the reams of data created by our modest survey--about which I need to make a guilty admission. Because I was busy and because we simply had too much information on spreadsheets, it took me nearly a month to crudely sort the responses. When I did I was struck by a cascade of questions: What if the students who didn’t have an adult connection also didn’t have a supportive friend group? Wait...what if those same students also didn’t find their classes interesting? It’s not difficult to see where this line of questioning led me, and by the time I sorted the “strongly disagree” responses, a pattern emerged: the data had become predictive of the symptoms of disengagement and disconnection. In the month since we had given the survey, the students who expressed the most disconnection had suffered: one student was failing all of his classes; one student had been suspended for substance use issues; one student had been evaluated for self harm. It hurt to see, but what hurt worse was the surprise I felt when I saw one name on the list: a student who had dropped out prior to us administering the survey. Like most schools we aren’t exactly fast at pulling students or retired faculty from our email system. Despite no longer being enrolled, the young man had gotten the link and responded to the entire survey. Did he just want to be known? Did he just want someone to see him?

As with any important learning I’ve earned in my career, I was faced with a few uncomfortable questions: Why haven’t I been mapping this my entire career? How could I have been leading without this information? What could a proactive approach have meant to my past students? Why don’t all schools do this? And...of course...now what?

The “now what” was obvious. We needed help, and we needed help with something that should be a staple before they let English teachers like me become principals. I reached out to my brother, Matt Bunting, and to Brian Lloyd Newberry, a friend who has the coolest job title ever--Data Architect. In addition to being very systems-smart, both are of the most caring and socially conscious people I know. Brian invested hundreds of hours to build software to ensure we could get the data and correlations we needed instantaneously, and a company, Engage, was created to support these efforts. He crafted a heatmap (pictured below) of the school so we could see the macro data, and he asked me some really tough questions. I won’t forget the phone call when he said, “I thought you said this was for all students.” I responded, “Yeah...of course it is.” His reply: “Not if you think 87% means all.” In sorting through 1100 responses I had missed the most obvious data point: the 200 students who didn’t respond! Who were they? Why hadn’t they taken the survey? What was their experience like?



Stan Williams, co author of this blog, often reminds me that the questions we ask are more important than the answers we seek, and the heatmap above sparked many questions. In particular, it appeared the data skewed to the left (more negative) in questions that drive at personalization, student voice, and choice. Stan and Emily Rinkema tossed the question to their students in their Think Tank class who had this to say: people ask us our opinions...it’s just we never see the outcome of our thoughts. Student voice led us to student agency, and the same student who made the observation recommended we start a student congress instead of a student council to engage a much higher rate of community involvement.

We noticed other correlations supporting what can only be described as educational truisms, especially when triangulating grades and standardized tests with the survey:
Rigor is a vital factor in student engagement.
Intrinsic motivation influences outcomes.
Socio-economics matter.
A match between a student’s vision of the future and the schools influences engagement.
We’ve only begun to scratch the surface of the implications of our Engagement data, and like everyone else, our most recent survey was interrupted by Covid-19. As we have with so much of our curricula in the past few weeks, we scrambled, pulled together a team of caring faculty members, and ditched many of our normal questions in favor of understanding wellness in a new context. Here are a few...
  • I would like it if an adult from school would reach out to me directly to check in.
  • I am aware of mental health support services that are available.
  • I know how to ask for and access mental health support if I need/want it.
  • I am able to get some fresh air and/or exercise regularly.
  • I feel good about my sleep patterns.
  • My home environment allows me to engage in my work.
  • When this is all over, what are you looking forward to?
  • I am worried about my own health and wellbeing related to Covid-19.
  • I am worried about the health and wellbeing of close friends and family related to Covid-19.
We sent the restructured questions to our community and have received 57% of the responses thus far. As usual, I am both inspired and burdened by the data, and I am reminded of another of my mother’s sayings: we are all expressions of the systems from which we grow. And while I have more questions than answers, one data point juts out above the rest: students are much more worried about their loved ones than they are for themselves. My mother probably had a more clinical term, but I am choosing to see that selflessness as goodness and the connectedness as strength.

*For those who would like to run the Engage tool for free at their school during this time period, contact Matt Bunting (matt@engaged.school) and Brian Lloyd Newberry at bln@engaged.school.