Issues with research questionnaires

My interest in the Cynefin Framework grew out of my work as an instructional coach first, where I recognized the complexity of teaching reading and writing to diverse learners in classroom settings; and district technology coordinator second, where I saw the complexity of integrating educational technology in different school settings. It wasn’t until I worked at Rangeview High School in APS when Snowden’s criticism of surveys made sense to me. Our school principal, Ron Fay, would discuss with me the futility of trying to interpret results from the state of Colorado’s lengthy statistical Teaching and Learning Conditions Survey (TLCC).

When Snowden, in the clip above excerpted from his address, “How leaders change culture through small actions,” for Academi Wales, the public center in Wales for leadership and management excellence, picks apart all of the possible angles to answering one of IBM’s survey questions, I hear my principal’s frustration with the questions his teachers are asked to answer on the TLCC to assess the climate and culture of Rangeview High.

Snowden responds to the question, “Does your manager consult with you on a regular basis?” by pointing out the complex reality of the job he held at IBM.

“I’ve got several managers. Sometimes they consult me, sometimes they don’t. Sometimes they should, sometimes they shouldn’t,” he says.

Similarly, teachers in Colorado are asked whether they agree with the statement, “Teachers are provided with informal feedback to improve their instruction.”

Screen Shot 2018-07-27 at 7.27.20 PM
Screenshot snapped in the 17-18 TLCC results for Rangeview High School.

At my school, this feedback might come from students, other teachers, instructional coaches, district support people, or administrators. While teachers are likely to assume the question is about administrative feedback, teachers work with administrators in a variety of settings and dialogue with them in ways that lead to feedback about their instruction. As with the question asked about IBM management, the reality of the school system renders the data generated by the question ambiguous if not meaningless. To make sense of why 65% of respondents answered positively to this question at my school, leaders first have to determine how teachers interpreted a vague question. When you consider that the survey has roughly 140 questions, many of which are similarly ambiguous, you realize that the data collected to measure teaching and learning conditions, far from being actionable, is just the beginning of a fact-finding mission for school leaders who want to develop actions steps to improve culture.

Dos and donts TELL

It is telling that along with the data, the state provides schools with multiple pages of do’s and don’ts so they won’t be tempted to treat this data as informative or actionable. My favorite cautions are pictured below.

Screen Shot 2018-07-08 at 2.36.15 PM
This screenshot comes from the article, “How to Read Your Report,” which appears on the Teaching and Learning Conditions section of the Colorado Department of Education’s website.

 

I’ve circled the “Do” statement that encourages school leaders to have conversations about the statistics the survey generates in order to develop action steps, as well as the “Don’t” statement that says that the data isn’t really meant to be used at the school level. It is clear that this data system isn’t designed to be actionable or have meaning that can be readily determined by stakeholders. Instead, it is a survey designed by data scientists which is best used for system wide analysis by those experts.

In my experience, this type of data is presented to classroom teachers and school leaders in contexts where they are likely to misinterpret the statistics within.

The benefit of gathering stories is that our skill at making meaning from them is developed over a lifetime of trading in stories. The data we collect using Cognitive Edge’s Sensemaker software generates statistics that add depth to stories rather than statistics in need of stories.

Governing constraints, enabling constraints

In an address to Academi Wales, the public center in Wales for leadership and management excellence, Dave Snowden of Cognitive Edge speaks about constraints in systems that force people to break rules, contrasted with constraints in systems that enable people to work responsively and develop novel practices.

The clip above is excerpted from that address. He explains how nurses in hospitals have to break rules in order to provide empathetic care. The nurses’ expertise in treating patients allows them to know which rules must be broken for the patient’s sake, while their familiarity with the hospital’s system allows them to know the real, presumably minor, consequences of this rule breaking.

This kind of story – over-controlling but out-of-touch boss duped by a savvy veteran employee- is told in organizations of all shapes and sizes and might be gratifying for workers to tell, however, Snowden explains that the hospital management isn’t necessarily incompetent, or mean spirited at all, just predisposed to leading complex systems with methods that leverage ordered processes.

He goes on to explain the difference between choatic systems and complex systems, both of which fall on the left side of Snowden’s Cynefin framework. He talks about his consulting work with governments, in which he asks the question, “How can we create enabling constraints which allow locally valid solutions to emerge?”

People use stories to envision action

When groups of stakeholders review stories written by their peers, we ask them how we can get more stories like the positive ones and fewer like the negative ones. I’ve shared story collections with teams of counselors, deans, early career teachers and more veteran teachers who serve as department chairs. It is never a surprise that all are inclined to identify the ways other groups can change their practices to improve the school system. Deans naturally suggest changes for teachers the same way teachers naturally suggest changes to administrative policies and practices. When, as a facilitative move, I ask each group to focus on how their group can impact stories, it is a kind of enabling constraint. Yes, people require a few reminders to think about how their groups can impact our system, but the suggestions they come up with soften and sound more reflective when they stop suggesting changes for others and begin envisioning actions steps they might take. Pressed to identify ways they can influence the stories we collect about student conduct, deans envision ways they can communicate with teachers differently, or act as mentors to new teachers. Similarly, teachers acknowledge that they can work with colleagues to make commitments about changes, and ask clarifying questions of administrators when they arise. All of this is easier when these stakeholders know that administrators, too, are reading the same set of stories and trying to identify ways they impact our system to amplify successes and dampen failures.

By asking them to focus on their own potential impact, groups are reminded of the importance of their roles. The facilitation relies on the positive presupposition that each group can contribute to improved school culture.

Knowledge management and enabling constraints

At Rangeview High School, I asked two different groups to reflect on two different school systems: student scheduling and college admissions testing, that are complex in nature with ordered and unordered properties. In informal conversations during which I took notes, I asked project leaders what went well and what needed to improve. With the counseling team, I asked them to review their own reflections about the arena scheduling process in order to determine how they might build on what was a successful first year with a student-centered approach to schedule building. With two stakeholders deeply involved in the school’s administration of the SAT, a large endeavor that intersects with all aspects of school leadership, they identified potential experiments from their own reflections on the past year’s work.

In both cases, the experts’ analysis can be presented back to them visualized across the domains of the Cynefin Framework. A dynamic aspect of knowledge management going forward using this approach is the way we can represent the complexity of these systems to involve more stakeholders, or to support the learning of newcomers to our school. Data that initially fit in a two column chart to identify what went well and what needed improvement, now better visualizes the complexity of the two projects. The local experts in the school help to identify which aspects of these projects are unordered. Instead of trying to bring order to them, we engage in inquiry using safe-to-fail experiments.

Now that both groups have thought critically about each project, it seems appropriate for school leaders to also engage in this process. Their vantage points in the system are surely different and they may see opportunity to experiment where a counselor or testing coordinator hasn’t. The Cynefin Framework school administrators build will be different based on those vantage points.

Snowden’s address challenges leaders to look at unordered aspects of their systems as potential experiments. How can we help stakeholders create enabling constraints that bring coherence to experiments while aligning organizational work to larger vision of supporting all students to navigating a path to fulfilling post-secondary work and learning?