Finding the Right Direction: How Measuring Engagement Changed Everything

This post is written by CVU Principal Adam Bunting:

My mother spent her career as a family systems psychologist. I always admired her work, particularly the deep respect she held for her clients. Just as great educators know about their students, she understood that her clients needed to be the sense makers. She would say things like, “Ultimately, they are the ones who do the work, not me.” I remember asking her once if she ever got frustrated with people who repeatedly made the same mistakes, and she had this to say, “You know...people work really hard...it’s just sometimes they work hard in the wrong direction.”

As a principal, I return to her phrase now and again--especially when the work feels personal and complex. I ask myself, How am I working hard in the wrong direction? I held my mother’s words especially close in 2018 when two events dominated the educational landscape: the Parkland shootings and an increased focus on the devastation of opioid addiction in Vermont families. Our work was well intentioned, but much of it--focused on deficit models--pulled us in the wrong direction of reinforcing walls and policies instead of the relationships that undergird health, connection, and engagement. Current events demand a similar frame as we boil away the superfluous and distill the healthiest educational experience we can for our young people. At CVU, we have landed on this mantra to guide us through this pandemic: connection first, engagement second, and academic learning third.

Connection is more readily described, but defining engagement is no small feat. Like many educators, my understanding of engagement evolved as my respect for my students deepened. I began my career seeing engagement as an individual choice a student makes in our classrooms. Later, I saw engagement as an emotional state where our learners find themselves as receptive, open, and curious. Then I saw engagement more as a product of the systems around us and the conditions of our lives and learning spaces. 

Our school has chosen to define engagement just as we might define happiness or flow--a state of being when humans are both more receptive to and more motivated to seek new learning. We cannot force engagement, but we can ensure the soil from which it grows has the proper nutrients: physical and emotional safety, meaningful adult relationships, meaningful peer relationships, balanced amounts of stress, connection to a purpose larger than self, small successes and appropriate challenges, and rich and relevant content. More pragmatically, engagement may be a school’s most important indicator for wellness and successful learning, and it is something we can measure by examining the conditions that create it.

So it was two years ago we launched the Engagement Survey at CVU--an attempt to map the engagement of all our students. For those who were disconnected, we wanted to build connections, and for those who were thriving, we wanted to understand and grow those conditions. We created simple likert scales asking students to strongly agree, agree, disagree or strongly disagree to prompts like…

• I have friends at school.
• The adults at school care about me.
• I believe school is preparing me well for the future.
• Life outside of school feels calm and manageable.
• There is at least one adult I could talk to if I needed help.
• I feel like my input is valued and voice is heard at school.
• I experience the right amount of challenge.

Unlike any survey we had ever given, this one was non-anonymous, and it needed to be. In fact, the more times we administered these questions, the more we came to view anonymous surveys with a certain derision. What does it mean when we ask students highly personal questions and never follow up? With some serious sweat on the part of our technology integrationist and IT staff, we administered the survey through our advisory system. In addition, we sent the survey to parents and advisors--asking them to predict what their student might say to a given prompt.

What we got back changed the way I saw our school--a place where I’ve been a student, teacher, and administrator since 1990.

The immediate responses to the survey were powerfully positive and underscored a belief that has been long held at CVU: authentic relationships are the backbone of a learning organization. The subtext was poignant unto itself. Giving the survey alone says we care about your real feelings, experiences, and thoughts enough to ask and act upon them. We determined our approach would be one of curiosity and an assets orientation, knowing the data was only as good as the dialogue it inspired.

One of the best things we did was ask students to list at least one of their “go-to” adults by name. Within a few hours of viewing the responses, our advisors began sending emails to one another and to our staff. I got one that read, “You may not know this, but you are [student name’s] trusted adult.” Our staff were often the ones who had the most mentions of being the trusted adults. Anyone who doesn’t value administrative assistants, campus supervisors, or tutors needs only to see the data! I will admit it - I used the control-find function to see who indicated I was their go-to. I was surprised about how good it felt to know kids really counted on me. As one teacher said to me in the hall: “I had no idea I was that important to [student name]. It’s changing how I interact with him.”

Of course, as happy as I was that the survey fostered relationships between faculty and students (and helped teachers see the holistic student--not just the science or English learner), I was equally disturbed by the group of students who indicated they didn’t have any trusted adult. The percentage was less than 1%. Normally, I would have pointed to that statistic and exclaimed, ”Look at what a great job we’re doing!” Now, however, the percentages were connected to names. I can overlook the significance of numbers--but not the names of individual students. We implemented a few interventions, but my favorite was also the most basic: a group of our faculty members took it upon themselves to smile and say hello to our kids who expressed disconnection. As someone said to me recently, the biggest problems don’t always require big solutions; sometimes simple solutions work best.

Another problem was the reams of data created by our modest survey--about which I need to make a guilty admission. Because I was busy and because we simply had too much information on spreadsheets, it took me nearly a month to crudely sort the responses. When I did I was struck by a cascade of questions: What if the students who didn’t have an adult connection also didn’t have a supportive friend group? Wait...what if those same students also didn’t find their classes interesting? It’s not difficult to see where this line of questioning led me, and by the time I sorted the “strongly disagree” responses, a pattern emerged: the data had become predictive of the symptoms of disengagement and disconnection. In the month since we had given the survey, the students who expressed the most disconnection had suffered: one student was failing all of his classes; one student had been suspended for substance use issues; one student had been evaluated for self harm. It hurt to see, but what hurt worse was the surprise I felt when I saw one name on the list: a student who had dropped out prior to us administering the survey. Like most schools we aren’t exactly fast at pulling students or retired faculty from our email system. Despite no longer being enrolled, the young man had gotten the link and responded to the entire survey. Did he just want to be known? Did he just want someone to see him?

As with any important learning I’ve earned in my career, I was faced with a few uncomfortable questions: Why haven’t I been mapping this my entire career? How could I have been leading without this information? What could a proactive approach have meant to my past students? Why don’t all schools do this? And...of course...now what?

The “now what” was obvious. We needed help, and we needed help with something that should be a staple before they let English teachers like me become principals. I reached out to my brother, Matt Bunting, and to Brian Lloyd Newberry, a friend who has the coolest job title ever--Data Architect. In addition to being very systems-smart, both are of the most caring and socially conscious people I know. Brian invested hundreds of hours to build software to ensure we could get the data and correlations we needed instantaneously, and a company, Engage, was created to support these efforts. He crafted a heatmap (pictured below) of the school so we could see the macro data, and he asked me some really tough questions. I won’t forget the phone call when he said, “I thought you said this was for all students.” I responded, “Yeah...of course it is.” His reply: “Not if you think 87% means all.” In sorting through 1100 responses I had missed the most obvious data point: the 200 students who didn’t respond! Who were they? Why hadn’t they taken the survey? What was their experience like?



Stan Williams, co author of this blog, often reminds me that the questions we ask are more important than the answers we seek, and the heatmap above sparked many questions. In particular, it appeared the data skewed to the left (more negative) in questions that drive at personalization, student voice, and choice. Stan and Emily Rinkema tossed the question to their students in their Think Tank class who had this to say: people ask us our opinions...it’s just we never see the outcome of our thoughts. Student voice led us to student agency, and the same student who made the observation recommended we start a student congress instead of a student council to engage a much higher rate of community involvement.

We noticed other correlations supporting what can only be described as educational truisms, especially when triangulating grades and standardized tests with the survey:

Rigor is a vital factor in student engagement.
Intrinsic motivation influences outcomes.
Socio-economics matter.
A match between a student’s vision of the future and the schools influences engagement.

We’ve only begun to scratch the surface of the implications of our Engagement data, and like everyone else, our most recent survey was interrupted by Covid-19. As we have with so much of our curricula in the past few weeks, we scrambled, pulled together a team of caring faculty members, and ditched many of our normal questions in favor of understanding wellness in a new context. Here are a few...

• I would like it if an adult from school would reach out to me directly to check in.
• I am aware of mental health support services that are available.
• I know how to ask for and access mental health support if I need/want it.
• I am able to get some fresh air and/or exercise regularly.
• I feel good about my sleep patterns.
• My home environment allows me to engage in my work.
• When this is all over, what are you looking forward to?
• I am worried about my own health and wellbeing related to Covid-19.
• I am worried about the health and wellbeing of close friends and family related to Covid-19.

We sent the restructured questions to our community and have received 57% of the responses thus far. As usual, I am both inspired and burdened by the data, and I am reminded of another of my mother’s sayings: we are all expressions of the systems from which we grow. And while I have more questions than answers, one data point juts out above the rest: students are much more worried about their loved ones than they are for themselves. My mother probably had a more clinical term, but I am choosing to see that selflessness as goodness and the connectedness as strength.

*For those who would like to run the Engage tool for free at their school during this time period, contact Matt Bunting (matt@engaged.school) and Brian Lloyd Newberry at bln@engaged.school.

Nothing Normal about this New: One School’s Approach to Connecting, Engaging, and Learning

There are so many things we took for granted during what we are now calling “regular school.” We were so comfortable that we didn’t stop to question things--not only tangible things like photocopiers and hallway fist-bumps, but things like our definitions of engagement and learning. That’s all changed now.

Schools all over the world are scrambling to piece together plans and systems and structures to maintain or replicate or recreate a school experience for our students. We are all doing the best we can with the resources we have in the contexts we woke up in a few weeks or months ago (is that all it’s been?!). But one thing I think we’ve all discovered is that there is no way that what we’re doing (or trying to do) now is anything like school as we knew it.

In Vermont, we had two stages to this new normal: Maintenance of Learning and Remote Learning. These were dictated and defined by the state. The Maintenance stage could not include new learning, assessments, or mandatory work. It was put in place to provide families and schools time to develop systems to maximize equity, including ways to provide essential services and access to education for all students. During this stage, which was three weeks long, schools worked to create systems, structures, and strategies for stage two (while still providing food and essential services to all of their students). When we entered stage two, which started officially this week, the goal was to add accountability and learning to the mix. Schools were required to create learning plans and send these to the state, and our goals had to include how we were going to deliver and assess learning for all of our students.

In an effort to document this experience more than anything else (but maybe to provide something that someone else can take and revise and improve), we thought we would share what our school system is doing and what we are doing as teachers within our system.

Defining a Purpose: Our principal, Adam Bunting, declared within the first week of the first stage that the driving values of our school throughout the entire remote experience would be Connection, Engagement, and Learning, in that order. All systems designs and all decisions made were filtered through these lenses--and if the plans didn’t work to maximize these values in this order, then it was back to the drawing board. The clarity and conviction behind this statement gave all of us--school leaders, teachers, students, and community members--a solid foundation and shared purpose.

The Student Schedule: Once we had a shared purpose, we could begin to develop systems and structures. Our student schedule for Maintenance Learning (stage one) provided a balance of synchronous and asynchronous opportunities, all optional. We asked for feedback from teachers, families, and students after two weeks, and then designed our Remote Learning (stage two) schedule to better support connection, engagement, and learning. See the explanations in italics beneath the schedule for details about each part. (Note: The teacher schedule includes optional PD from 9-10 each morning (specific topics posted weekly with links to hangouts), and optional faculty meetings two afternoons a week.)

Agenda, Task, and Materials: Teachers will post agendas, tasks, and materials for the day’s learning by 9 a.m. Students should plan for approximately 90 minutes of work per week for each class; this includes class meeting times as well as the time it takes to make sense of the task, access input, think, and demonstrate learning through output. Teaches will use the scheduling function in Google Classroom so that these are posted at the correct time and day and that students are not overwhelmed by posts at all times of the day/week. *AP courses will follow this expectation as well, unless the class has not completed the abbreviated curriculum published by the College Board.

Class Meeting Times: To maintain our strong classroom communities, all class meetings should ideally begin with a live Hangout for the class during which the teacher explains the week’s assignments and provides some direct instruction. If this is not possible, teachers should post a short video/screencast to meet this purpose. Following the live Hangout, the remainder of class meeting time could be used for quick check in for small groups or individual students, discussion, or work time.

Connect Time: Teachers will be available every day to answer questions from any of their students, through email or Hangout from 2:30-3:00. We understand that there may be days when this is not possible, but having a consistent time when students can access teachers is important.

Exploration Menu: Each week, a menu of options categorized by Think, Feel, Act are posted. Students who want to do any of these activities can document them and send photos to Seth, Jamie, and Tim (This is not mandatory; it’s a way to build community and share with each other).



Grading and Reporting: One of the most complicated decisions our school had to make was what to do about grades and transcripts. We are a standards-based school that provides end of course letter grades, and these grades inform the student GPAs on the transcript. When physical school closed for us, we had just wrapped up quarter 3, so all students had what we call Grade Snapshots that represented their current achievement of the course learning targets. Our quarters are cumulative, not averaged together, so the Q3 snapshot was an accurate communication of evidence students had provided up to that point; however, in most classes, had we continued as normal, students would have had many more opportunities to relearn, improve skills, and ultimately increase their target scores by the end of the year. This was particularly true in many of our semester courses, where the snapshot grades represented a small (and potentially misleading) sample size of evidence.

We spent a lot of time researching what other schools and colleges were doing, what states were recommending, and what leading experts on grading, equity, and assessment were suggesting. While there is no perfect model, we finally decided on the following, which was presented to faculty first, and after some revision, to students and families:

For year-long courses:

• End of year letter grade: Because students had the opportunity to provide a significant body of evidence of learning throughout the year, teachers can provide accurate scores for the course learning targets that were instructed, practiced, and assessed. Students will receive a letter grade on their transcripts based on these targets. In order to maximize equity and opportunity, the letter grade that students had at the last snapshot cannot go down during this new phase of learning, but can be raised through work they do the rest of the year. In order to receive a grade on the transcript, students must participate in Remote Learning; students who do not participate will get an Incomplete on their transcript.

For semester courses:

• Pass with Distinction or Incomplete: Students may not have had the opportunity to provide a sufficient body of evidence of learning for second semester courses. Teachers may not be able to provide accurate scores for learning targets in second semester courses at this time, so students will receive one of the following scores on their transcript: Incomplete or Pass with Distinction. 
• Incomplete (Inc): The following situations will result in an Inc on the student’s transcript, and students will have an opportunity to change this after the school year. Students who had a composite score below a 2 at the Q3 Snapshot and do not participate during Remote Learning to improve these scores, OR students who showed sufficient evidence of achievement at the Q3 Snapshot and did not participate during Remote Learning.
• Pass with Distinction (PD): Students who show sufficient evidence of achievement (either at the time of the Q3 snapshot or by the end of the year) and continue to participate during Remote Learning will receive a PD on their transcript.

Note about Grading and Transcripts: There may be individual circumstances where the above will need to be amended. For students with specific situations that may require other options, please contact your House Counselor and we will work together to ensure that no harm is done.

Originally, we had gone with 3 levels for semester courses: Inc, Pass, and Pass with Distinction. After many days of intense conversations about equity, access, and the extraordinary circumstances that a global pandemic brings, the decision was to simplify to 2 levels. Our leadership team felt strongly that keeping Pass with Distinction (rather than Pass) was an important symbolic recognition of how challenging these times are for so many in our community.

Assessment, Tracking, and Feedback

Once we had decided how to grade and report, we needed to develop some guidelines for assessment, tracking, and feedback that supported the purpose (connect, engage, and learn) as well as the new grading and reporting decisions. Here is the document we developed to help guide teacher choices.

Classes

We use KUDs as our curriculum documents at CVU, simple backwards-design templates that outline what students will Know, Understand, and be able to Do at the end of a learning experience. Teachers were asked to revise their existing KUDs or develop new ones for this remote period of learning. Our leadership team gave us all the option to either significantly pare down and modify what we had planned to do, or to completely veer from the existing KUD and develop a plan that we felt might be more engaging, relevant, or accessible. Because we needed to document curricular plans for the state, these KUDs provided us a way to be accountable to Vermont while also using what we know about learning design to prioritize our outcomes. As an example, here’s the KUD and revised plan for the course we teach: Think Tank: Remote Learning 2020.

Next Steps?

Our district is about to go on our spring vacation, which seems really strange. CVU created a menu of opportunities for students, faculty, staff, and families to stay connected and engaged over this coming week of break, as we know that cutting off contact may not be the best thing for some members of our community. As for what happens when we get back...who knows. There are so many things that change from day to day--for all of us collectively and for each of us individually. Planning seems virtually impossible, whether at the class level or the school level. What we do know is that we will continue to get feedback from all members of the community and revise and iterate as much as we can. We will continue to design opportunities that maximize and support connection, engagement, and learning (in that order!). And we will continue to do what CVU does best: take care of ourselves, take care of each other, and take care of the place.

Setting Clear Destinations: What Teachers Say

We’re doing a book study with a group of educators from around our district who chose to read The Standards-Based Classroom: Make Learning the Goal, written by us a few years ago. Some of these teachers were in the original pilot program in our district, so have almost a decade of experience in a standards-based system; others are newer to our district and to the practices central to SBL. What they all have in common is a desire to continue their learning and reflect on their practices, For each of the four sections of the book, participating teachers will share their experiences, their reflections, and their questions with each other and with us, and they have given us permission to share excerpts publicly. 

The first section of the book is all about setting clear destinations for learning. Our district uses the K-U-D to help with curriculum planning and communication. This is a backwards design template that articulates what students will Know, Understand, and be able to Do at the end of a period of learning. We use skill-based Learning Targets, which are the Ds in our K-U-Ds, and these targets are the level 3 in our Learning Scales. Thanks to Carol Tomlinson, Grant Wiggins, and Jay McTighe for all of their work in this area that inspired our systems and structures.

Humanities K-U-D

All of the courses at our high school have K-U-Ds that are common for common courses and made public each year. Here is the most recent version of the public document that links to all High School K-U-Ds. We have four middle schools (grades 5-8), and all disciplines across the district share common learning targets and scales. Here are the links to Grades 5-6 Targets & Scales and Grades 7-8 Targets & Scales.

Target and Scale

In our own experiences teaching in standards-based classrooms over the past decade, we have found that clear destinations defined and articulated through K-U-Ds, Learning Targets, and Scales have changed our focus from what we teach to what students learn. Our planning, instruction, and assessment practices have become so much more intentional and responsive, and as a result, learning and engagement have increased. Over time, we have developed some strategies and structures that help improve effectiveness. The teachers in the book study tried a few of these strategies, and here is their thinking:

Geoff Glaspie: High School Math Teacher

One of the [strategies] from the first section (chapter 3 specifically) that resonated with me was the idea of “lose the numbers” on the Learning Scale, because I can see how it has the potential for students to treat a formative more as communication rather than compensation or judgment. Up until now, I have used learning scales with numbers on them and circled or written what the student’s level was — both for formative and summative assessments. This followed from what I observed with my mentor teacher and from other teachers with whom I have taught collaboratively or in parallel. The most common reaction I see when returning formatives is that they absolutely narrow their focus to the # on the paper. They share out loud with others what their level is, ask questions about the score, e.g. “if I had done [x], would I have gotten a 3?”...I see them laser-focused on the number and more often than not, quickly file the returned quiz in their folder or in some cases, throw them out. While I take the time to give very specific feedback on the student’s work within the body of the assessment, my feeling is that they do not absorb the information and insight from my comments or feedback because they are stuck on the number they achieved.

I decided to give [replacing numbers with an arrow] a try on my next formative in Geometry. Instead of circling one of the boxes or writing a number on the scale, I underlined the words in green that the student was secure in, red what they did not show evidence of, and both red and green if they were starting to show proficiency, but were not yet consistent. Before I handed them back, I told them what they would see, why I was doing it, and what their next steps were. Below is an example of a marked scale on a quiz.

What I observed right away was that students were more focused on looking at what they did, my comments, and some immediately asked, “can we go over adding radicals today?” This type of question is not something that I have typically seen in reaction to getting a quiz returned. I was also asked “do you have some practice sheets for solving special right triangles I can have to work on?” This was enough for me to say this is a practice I need to continue.

Leanne Morton: High School Latin Teacher

The simple strategy of removing numbers from scales during the practice and learning process struck me as we reach the end of a marking period where numbers matter. Too often I think students worry about the grade, but with the transition to standards-based classroom, I have noticed a shift. By providing the language in the “I can” statements, students now know how to articulate what they might need to do in order to reach the next step. I have seen my conversations shift and powerfully so because there is actual language to use in helping students understand what they need to do in order to meet the next part of the scale. I have never thought about removing the numbers from the top of my scales on formatives/practice. Sometimes depending on how my copying and pasting goes, they do not appear, but I love the idea of encouraging the continuum of learning by adding a simple arrow. I have seen the arrow used in all the work Emily and Stan have produced for us at CVU, but never once thought about adding it to my work. I too have been trained by the grade/number machine and it is liberating to think about focusing our practice/learning time around the process. I like thinking about how we go from the first part of the scale to the last and what learning do we need to do rather than focusing on “how do I get the four?” I just started a unit on conjugating and translating verbs in the imperfect and perfect tenses. I am moving students from the present tense into the imperfect and decided to remind them about the targets involved. I changed the table to include the arrow and like the next part of my reflection, am employing part of the unit KUD into the work. You can see the slide show here.

Katie Kuntz: High School Humanities Teacher

Using KUDs and targets make so much sense to both teachers and students. When we were first asked to use KUD’s at CVU so many teachers said this was just another fad… that “the pendulum would swing another direction in a few short years.” To be honest, I can’t imagine teaching any other way.

Our basic format is at the very beginning of each unit we give students a packet and the very first page of that packet has the KUD, learning targets, formative and summatives, as well as a tentative calendar. When we hand it out we go over any new targets, explain the formatives and summatives and then ask students to look at the “Understands”. This section usually takes me the longest to write up as I tend not to be a “big picture” kind of gal but I have come to realize that this may be one of the most important parts. It allows students to see the connections and what learning they will do to make these connections. I have to say that I hate writing this section but feel amazing about them once I go through the process. It’s kind of like eating a salad. I don’t really want one for lunch, I’d rather eat a burger and fries, and I’m kind of grumpy during lunch, but then feel much happier in the afternoon about my food choice! Here’s an example of our KUD/scales/calendar format.

Tim Buckingham: Middle School Music: K-U-D's can be a template for all learning episodes -- bigger units, smaller lessons, even daily class agendas, etc...in order to intentionally tap knowledge, have a performance goal, and understand all of it within the context of learning over time. Could it be that easy?! Much like Understanding by Design (UbD), K-U-D's increase not only our organization as teachers but continue to have us focus on performance tasks at the heart of the education -- this keeps intention of designing learning based on the skill, incorporating content knowledge and the understanding of "why" we do it all.

Abby Granoff: Middle School Para-educator and Licensed Teacher: KUDs are the cornerstone of teaching. They help us to plan instruction, design assessments, and let students know what we want them to get out of it. If we don't know where we want our students to end up, we will be much less successful in getting them there. Once we have a KUD developed, we can share it with our students at the beginning of a unit, and ideally hang it up somewhere in the classroom and reference it every day. Also, when designing instruction, we can write what part of the KUD the lesson relates to on the board, so that students know where we're trying to go. KUDs give us a really clear destination, and allows us to be more intentional about the instruction we plan. If we think of the KUD as the destination, and planning is the road to get there, we can refer back to our KUD when planning to make sure that our instruction or lesson will actually get us to our destination and not take us on a scenic route or down a dead end.

Peg Rosenau: Middle School Para-educator and Licensed Teacher: The various standards that guide instruction in different disciplines provide a framework for instruction but also a huge amount of autonomy. Content in the digital age is ubiquitous, if not overwhelming. These combined can create a “drinking from a firehose” situation when determining what is most important to present to students, especially for a prescribed scope of time. K-U-Ds can bring some intentionality to this process by focusing ultimately on the desired skills that one wants students to get out of a unit- as it is the skills that ultimately demonstrate the knowledge and understanding of the experience.


If you have any thoughts or examples you would like to share, please feel free to comment. The more we share with each other, the better it will be for students.