Issues with research questionnaires

My interest in the Cynefin Framework grew out of my work as an instructional coach first, where I recognized the complexity of teaching reading and writing to diverse learners in classroom settings; and district technology coordinator second, where I saw the complexity of integrating educational technology in different school settings. It wasn’t until I worked at Rangeview High School in APS when Snowden’s criticism of surveys made sense to me. Our school principal, Ron Fay, would discuss with me the futility of trying to interpret results from the state of Colorado’s lengthy statistical Teaching and Learning Conditions Survey (TLCC).

When Snowden, in the clip above excerpted from his address, “How leaders change culture through small actions,” for Academi Wales, the public center in Wales for leadership and management excellence, picks apart all of the possible angles to answering one of IBM’s survey questions, I hear my principal’s frustration with the questions his teachers are asked to answer on the TLCC to assess the climate and culture of Rangeview High.

Snowden responds to the question, “Does your manager consult with you on a regular basis?” by pointing out the complex reality of the job he held at IBM.

“I’ve got several managers. Sometimes they consult me, sometimes they don’t. Sometimes they should, sometimes they shouldn’t,” he says.

Similarly, teachers in Colorado are asked whether they agree with the statement, “Teachers are provided with informal feedback to improve their instruction.”

Screen Shot 2018-07-27 at 7.27.20 PM
Screenshot snapped in the 17-18 TLCC results for Rangeview High School.

At my school, this feedback might come from students, other teachers, instructional coaches, district support people, or administrators. While teachers are likely to assume the question is about administrative feedback, teachers work with administrators in a variety of settings and dialogue with them in ways that lead to feedback about their instruction. As with the question asked about IBM management, the reality of the school system renders the data generated by the question ambiguous if not meaningless. To make sense of why 65% of respondents answered positively to this question at my school, leaders first have to determine how teachers interpreted a vague question. When you consider that the survey has roughly 140 questions, many of which are similarly ambiguous, you realize that the data collected to measure teaching and learning conditions, far from being actionable, is just the beginning of a fact-finding mission for school leaders who want to develop actions steps to improve culture.

Dos and donts TELL

It is telling that along with the data, the state provides schools with multiple pages of do’s and don’ts so they won’t be tempted to treat this data as informative or actionable. My favorite cautions are pictured below.

Screen Shot 2018-07-08 at 2.36.15 PM
This screenshot comes from the article, “How to Read Your Report,” which appears on the Teaching and Learning Conditions section of the Colorado Department of Education’s website.

 

I’ve circled the “Do” statement that encourages school leaders to have conversations about the statistics the survey generates in order to develop action steps, as well as the “Don’t” statement that says that the data isn’t really meant to be used at the school level. It is clear that this data system isn’t designed to be actionable or have meaning that can be readily determined by stakeholders. Instead, it is a survey designed by data scientists which is best used for system wide analysis by those experts.

In my experience, this type of data is presented to classroom teachers and school leaders in contexts where they are likely to misinterpret the statistics within.

The benefit of gathering stories is that our skill at making meaning from them is developed over a lifetime of trading in stories. The data we collect using Cognitive Edge’s Sensemaker software generates statistics that add depth to stories rather than statistics in need of stories.

Story collections as culture data

In his address, “How leaders change culture through small actions,” for Academi Wales, the public center in Wales for leadership and management excellence, Dave Snowden of Cognitive Edge describes how he gathers narrative data from patients and nurses in hospitals to develop what he calls a human sensor network. He refers to two sets of stories that emerge as The Patient’s Journey and The Nurse’s Journey.

I’ve excerpted the clip from his talk above because it might help school leaders who are beginning to gather stories envision potential collections that could emerge at their schools to promote deeper understanding of different stakeholders’ experiences. In the 2017-18 school year, I began to pilot Cognitive Edge’s Sensemaker software at Rangeview High School, where I collected a few hundred stories from students and teachers using the Culture SCAN survey. Since I was piloting the work, I would inevitably focus on how to introduce the survey to different groups of respondents, which involved thinking through how I would support them as they coded their stories using triangles and a sliding digital ball (pictured below).

triangle gif
After writing and giving the story a title, a respondent interprets the story by moving a ball inside a triangle to illustrate qualities evident in their story.

A few times in the 2017-18 school year I had a chance to put stories together in collections to share them with colleagues and students. To do so, I would read through the data and look for multiple stories about the same general topics, then curate small collections that included an even balance of both negative and positive experiences. My cleanest examples were two sets of stories about broad categories: student conduct and school leadership. With these small collections in hand, I facilitated the next part of the process, which was to put the stories back in front of teachers, students, deans, and school leaders. As they read, I asked them (while pointing at the stories coded as positive by respondents), “How do we get more like this? And, (pointing at the negative stories),  as determined by respondents), “How do we get fewer like this?” On each occasion, because I was piloting a process, I was probably more attuned to how I was facilitating than I was to how people were responding. Still, despite being a little self-conscious about introducing this novel process, I was able to gauge how all of the different groups became engrossed in reading the stories I presented to them. Just as engrossing as reading the stories were the conversations about how we could create conditions that would lead to more positive and fewer negative stories at our school.

Hearing Snowden talk about the work he’s done in hospital systems, I’m struck by his use of the word “empowerment.” I think the engagement I see when teachers read the stories written by students, and by their colleagues, is a sign that this process of collecting, curating, and redistributing stories might lead to a greater sense of empowerment across our system. Certainly, when I challenge teachers with the task of imagining a few ways they, in their current roles, might take actions that would amplify the positive and dampen the negative in their workplace, the room buzzes and they immediately have ideas they can act on.

The notion of empowerment also reminds me that an important next step in this work for me is to invite others into to process of creating story collections. To help others see the potential in this process without burying them in spreadsheets full of stories I’ve done this on my own to this point, but I think there it will empower other leaders to see a “human sensor network” in its raw .csv file-view, and in the populated triangles that the associated Sensemaker Explorer app produces. While I’ve been delivering short collections of stories in an easy to read format, I want to empower others with the abstract nature of pages and pages of stories, despite their messiness and occasional mundanity. (Among some moving stories and occasional rough ones, our budding human sensor has also revealed less powerful stories like this: “I came to work last week and there was no paper in the copy room.”)

It is my hope that by inviting others into the spreadsheets of data this work generates, and into the process of creating collections, we can think together about the collections we want to stitch together that can support a better understanding of a school. In the way Snowden puts together The Patient’s Journey and The Nurse’s Journey, I can imagine a freshman’s journey, a new teacher’s journey, or department chair’s journey. I’m anxious to see where this work will go next but my hunch is that empowerment will come from inviting others into this process and asking them what potential they see.

Governing constraints, enabling constraints

In an address to Academi Wales, the public center in Wales for leadership and management excellence, Dave Snowden of Cognitive Edge speaks about constraints in systems that force people to break rules, contrasted with constraints in systems that enable people to work responsively and develop novel practices.

The clip above is excerpted from that address. He explains how nurses in hospitals have to break rules in order to provide empathetic care. The nurses’ expertise in treating patients allows them to know which rules must be broken for the patient’s sake, while their familiarity with the hospital’s system allows them to know the real, presumably minor, consequences of this rule breaking.

This kind of story – over-controlling but out-of-touch boss duped by a savvy veteran employee- is told in organizations of all shapes and sizes and might be gratifying for workers to tell, however, Snowden explains that the hospital management isn’t necessarily incompetent, or mean spirited at all, just predisposed to leading complex systems with methods that leverage ordered processes.

He goes on to explain the difference between choatic systems and complex systems, both of which fall on the left side of Snowden’s Cynefin framework. He talks about his consulting work with governments, in which he asks the question, “How can we create enabling constraints which allow locally valid solutions to emerge?”

People use stories to envision action

When groups of stakeholders review stories written by their peers, we ask them how we can get more stories like the positive ones and fewer like the negative ones. I’ve shared story collections with teams of counselors, deans, early career teachers and more veteran teachers who serve as department chairs. It is never a surprise that all are inclined to identify the ways other groups can change their practices to improve the school system. Deans naturally suggest changes for teachers the same way teachers naturally suggest changes to administrative policies and practices. When, as a facilitative move, I ask each group to focus on how their group can impact stories, it is a kind of enabling constraint. Yes, people require a few reminders to think about how their groups can impact our system, but the suggestions they come up with soften and sound more reflective when they stop suggesting changes for others and begin envisioning actions steps they might take. Pressed to identify ways they can influence the stories we collect about student conduct, deans envision ways they can communicate with teachers differently, or act as mentors to new teachers. Similarly, teachers acknowledge that they can work with colleagues to make commitments about changes, and ask clarifying questions of administrators when they arise. All of this is easier when these stakeholders know that administrators, too, are reading the same set of stories and trying to identify ways they impact our system to amplify successes and dampen failures.

By asking them to focus on their own potential impact, groups are reminded of the importance of their roles. The facilitation relies on the positive presupposition that each group can contribute to improved school culture.

Knowledge management and enabling constraints

At Rangeview High School, I asked two different groups to reflect on two different school systems: student scheduling and college admissions testing, that are complex in nature with ordered and unordered properties. In informal conversations during which I took notes, I asked project leaders what went well and what needed to improve. With the counseling team, I asked them to review their own reflections about the arena scheduling process in order to determine how they might build on what was a successful first year with a student-centered approach to schedule building. With two stakeholders deeply involved in the school’s administration of the SAT, a large endeavor that intersects with all aspects of school leadership, they identified potential experiments from their own reflections on the past year’s work.

In both cases, the experts’ analysis can be presented back to them visualized across the domains of the Cynefin Framework. A dynamic aspect of knowledge management going forward using this approach is the way we can represent the complexity of these systems to involve more stakeholders, or to support the learning of newcomers to our school. Data that initially fit in a two column chart to identify what went well and what needed improvement, now better visualizes the complexity of the two projects. The local experts in the school help to identify which aspects of these projects are unordered. Instead of trying to bring order to them, we engage in inquiry using safe-to-fail experiments.

Now that both groups have thought critically about each project, it seems appropriate for school leaders to also engage in this process. Their vantage points in the system are surely different and they may see opportunity to experiment where a counselor or testing coordinator hasn’t. The Cynefin Framework school administrators build will be different based on those vantage points.

Snowden’s address challenges leaders to look at unordered aspects of their systems as potential experiments. How can we help stakeholders create enabling constraints that bring coherence to experiments while aligning organizational work to larger vision of supporting all students to navigating a path to fulfilling post-secondary work and learning?

Coyote Country

coyote signThere are trails all around my Denver neighborhood here in the the southeast corner of Denver, the Highline Canal Trail and the Cherry Creek Trail, and along them there are all kinds of signs about the dangers of coyotes. When my oldest daughter was young she’d ask us to read each one whenever we passed one of these signs on a walk. As we explained to her why there were so many cautions posted, a look of worry crept across her face as she sat in her stroller. We tried not to stoke her fear but the facts we provided under questioning clearly caused coyotes to grow bigger and scarier in her imagination. Each time we read a sign she’d say, “We gotta watch out for ky-woe-tees!”

She’d ask about coyotes at bedtime. We would answer her questions patiently while reassuring her that the coyotes didn’t pose any threat to children in their beds. “They hunt rabbits and mice, Baby,” we’d say.

“That’s not what the signs say,” her expression said.

One day Hailey and I took a walk during a tranquil snowstorm and saw a coyote tramping through new snow at the bottom of a canal that runs off of Cherry Creek. We saw her from our perch on a bridge over the canal so we had something of an arial view of her thick winter coat, a mosaic of greys, browns and white. While she stared back at us. I whispered, “THERE’s a coyote.” Hailey was frozen in awe of the wild creature. It trotted a dozen yards or so before it looked back again watching us watch her go. Her paws left a perfect track at the bottom of the canal.

The rest of that walk we had a different kind of talk about coyotes. In the wake of such a peaceful sighting, Hailey needed to understand what that coyote was doing. We talked about how she had a den, probably in the canal, and how she hunted for rabbits, which are everywhere here. My daughter asked a blizzard of questions the way only a child under the age of 8 can while under the trance of a beautiful wild animal.

Years later, Hailey is almost a teenager and too cool to be awed by much of anything. Occasionally, when the subject of wild animals comes up, she’ll ask from the back seat if I remember the time we saw that coyote in the snow. We always share our recollection of that encounter and we exchange new memories of the moment. “Isn’t it strange,” I ask, “how it can sometimes feel so warm in a snowstorm, when there is no wind, just the slow descent of flakes all around?”

The coyote might be my spirit animal in that I, too, stalk the trails around here. Usually when I’m traipsing along them, I’m just hunting a little exercise and the piece of mind I can find along the creek. I’ve lived in this neighborhood for years now.  Whenever I bring up the subject of moving to a different part of town, my whole family erupts in revolt at the idea, which means these trails will be my territory for years to come. My own den, familiar and safe, will stay where it is not far from where the local coyotes bed. From time to time when I’m on a walk alone I’ll spot a solitary coyote, camouflaged and alert. They’re so beautiful, these subjects of nightmares and years of curiosity.

On Father’s Day, a note about immigrant boys and their fathers

“This is a hard life, Mister.”

He was holding a paper attendance tracker form in his hands, which he shook at me while he gave me the rebuttal he had wisely kept to himself in the dean’s office. He explained how he was 14, a middle schooler, when his father had been deported, which also marked the time he began working to help his mother pay rent. Three years later, his despondent mother wouldn’t talk to him anymore and his house was a place where he and his siblings fended for themselves to survive economically and emotionally in Aurora, Colorado, their adopted home, now that no one expected their father to return.

I stood and nodded, knowing that the relationship building I try to do with struggling students had earned me this story.

He went on to tell me how hard he had to work since that time to help pay rent, to buy a truck, and to afford the insurance and maintenance costs that went along with the transportation he needed. Growing angry as he talked, he told me how he had worked jobs in fast food restaurants for years and how the managers would only allow him to clock a fraction of his hours. When he complained, they would invite him to quit. They knew he was undocumented, without any rights to stand on, so he wasn’t entitled to the minimum wage guarantees afforded to citizens, to documented workers.  S_____ knew at an early age what many American citizens don’t: the profits of fast food franchises in America are bolstered by the cheap labor costs of teens with fake social security cards. The fast food restaurants my students flood to every day at lunch cheat the kids they know they can cheat.

“This is a hard life, Mister.”

On that day, in that hallway, I told him that he was going to finish high school. Scared and angry as he was about the dean’s firm message about attendance, he also knew that his coaches and teachers cared for him. As a teacher of immigrants, I know that the trust we’d earned as a school with this immigrant boy missing his father was that much harder to earn because of the way our culture views and treats the immigrants we entice across our borders with jobs we’ve always counted on them filling.


 

The story didn’t pour out of F_____, I had to drag it out of him while his mother listened. I’d spoken to both of them about attendance a few times by phone before. Meeting face to face in the main office, I asked why he hadn’t transferred schools when he moved out of our attendance area like he and his mother had told me they were planning. He explained that his father had not gone to his new school to complete the transfer paperwork, so he had mostly stayed home while he was in academic limbo- too far away to drive to my class for our 8:30 AM start time but not enrolled in the high school right behind their new home. On that day F_____ promised to do better as an assistant principal and I laid out the potential consequences if he kept skipping school.

Weeks later, when F______ missed class for the first time since that conference, he told a coach his father had been deported. Now attending class daily, he sat in the first row right in front of where I started class each morning and he looked sadder by the day.

With his father gone, he chose to read My Life As an Undocumented Immigrant, by Jose Antonio Vargas in my class. The reading response he turned in during this traumatic time read:

Being an undocumented person is hard and comes with many challenges. Jose says, “I am still an undocumented immigrant. And that means living a different kind of reality.” This means people like him don’t grow up living a life like people that are documented and were born in the United States.

F____’s paper reminded me about the experiential cultural differences between me and so many of my students. My privilege contrasts with their constant worry.

These two boys missing their fathers and confronting trouble for missing school are in my thoughts on this Father’s Day. With both of them, I know I earned their trust in a cultural setting that treats them and their families like criminals while our Colorado economy booms with them as important cogs. When we said goodbye for summer, S_____ looked forward to working on a construction crew because the real estate market in the Denver metropolitan area is thriving and he found a boss who treated him fairly. The wages for a strong young laborer make it easy to afford rent, gas and insurance.

F____’s summer started on a more somber note. In past summers he would work long hours with his dad to renovate the houses of “people that are documented.” Will he be looking for work using the skills he learned working with his dad? Those skills and the stamina of strong young men are in high demand, as immigrant laborers know well. I like to think his dad’s friends will help take care of him. Unlike the hard fought trust teachers like me earn from immigrant students in our question to help them develop academic skills, the trust immigrant families have with employers is colder and more transactional. The risk and reward of employment with a labor crew is something immigrant students are all fluent in. Less personal than the trust grown between teachers and students, immigrants to my city trust they can find work and make money despite the harsh reception they receive on so many fronts.

Families know they risk being torn apart when they come here but they come here still. For wages. For jobs. American dreams are realistic dreams of families who expect full well to confront racism and corruption. While the horrifying new policy of separating children from parents, boys from fathers, is in the headlines and shocking to many American citizens on this Father’s Day, undocumented fathers and sons know our immigration system has been tearing families apart for a while.