Governance happens in foggy weather

We need to reclaim human judgement, subjectivity and the primacy of direct experience as vital skills for leadership

17 years ago, I started dxw. I ran it for more than a decade, trying to help government build better services online. Over that time I helped to build a lot of things. Many of them were unremarkable: the routine stuff of government. Others I'm proud of, or remain frustrated by. I think anyone in government will probably relate. But now – having had a few years off since I sold the company – I think I've got a new lens through which to examine that experience.

I think we need to reconsider the balance we strike between thinking about the challenges we face, and feeling our way through them. I think the imbalance we have now – strongly towards thinking about them – leads directly to a lot of the waste, inefficiency and failure we see in our public services. It also explains why interacting with these services often feels frustrating and demoralising: like they can't share our understanding of the world, and have lost their humanity. This post is a bit long. Probably one to read with a cup of tea.

A few months ago, I was in France, at an event organised by Life Itself: an excellent and interesting bunch trying to work out how to live a practical, change-making, meaningful life in the midst of the polycrisis. We were there, in part, to talk about what it really means to live a meaningful life, so the conversation got philosophical pretty quickly.

In one of the sessions I was in, a conversation got going about the limits of truth. How can we know we know what we think we know? Is there really any such thing as objective truth? What even is knowledge in the first place? An impassioned argument was made in that session that truth, and consequently, "facts", are considerably slipperier things to define than we'd (read: I'd) generally like to admit. That in considering what things are "true", one must lean on things like felt sense, curiosity about conscious experience, intuition and first-hand observation. And that in so doing, what one can reliably reach is an idea of what is true for you or true in a particular context, or for a particular purpose.

Being the materialist, rationalist, Tim-Minchin-swigging middle-class white-collar technocrat that I am (was?), I found this line of argument quite uncomfortable. And partly with good reason. Taken to its extreme, this kind of very loose, experiential approach to life can cause people's feelings to so contaminate their thinking that they’re unable to see how things actually (probably) are, or understand with clarity what is actually (probably) possible. (lol philosophy)

I said as much, but more importantly, spent the next couple of days mulling it over. Because I could see their point. The other extreme – the very conceptual, hyper-rational one – isn't better. I've seen plenty of damage caused by woolly thinking. But I've also seen plenty of perverse situations where people's desire for predictable orderliness causes them to set up processes and systems that are equally damaging: because they, too, cannot see how things actually (probably) are, or understand with clarity what is actually (probably) possible.

Neither of these approaches is good. The hyper-experiential people can't think things through. But the hyper-rational/conceptual people can't see how things actually are, or can't imagine how the world, as seen by other people, might be different. From this position, the most extraordinary amount of organisational and cultural heartache and dysfunction has flowed. This isn't to say we should give up on truth – far from it – but we should acknowledge how it's grasped: not just through reason, but also through encounter, judgement, and grappling with the messy reality of experiencing the world.

Thoughts without content are empty; intuitions without concepts are blind

Kant, in the Critique of Pure Reason

Around this time, and with a prototype of these thoughts floating around in my head, I had the good fortune to meet an actual, honest-to-god philosopher thinking about almost exactly this problem. How often does that happen?

Bryan Kam is currently working on a paper and a book called Neither/Nor. In this work, Bryan and his co-author Isabela Granic argue that conceptual and experiential approaches needn't be opposing, incompatible camps: that it's better to think about them as overlapping and complementary skillsets. As different but equally valuable and important ways to understand the world. And, with an almost physical "woosh", the realisation landed that these ideas explain a lot of the frustration I have felt about governance, and the way big organisations do things – and, more specifically, the damaging assumptions that tend to be made about how software projects should be managed. These ideas are not mere philosophical niceties. They are deeply practical and applicable.

Over the last thirty years, this divide, in disguise, has been the main theme of the ongoing argument about how software development should be done. And if both the conceptual and the experiential have their place, and the right way is the middle way, I think we technologists have done ok. That's what most software development in sensible places now is. We don't do linear, rigid waterfall-style delivery anymore. We also don't just let the developers loose to go with the flow. We have planning, but we hold those plans loosely. We have task lists and backlogs, but we talk about them a lot. We have ideas, but we test them with users. We embrace trial and error, but we're mindful of the cost of failure, so we make progress in small steps, continously testing our ideas against the world. These are all examples of a pragmatic balancing of rational/conceptual thinking-things-through, and experiential/practical feeling-things-through.

And it's not surprising that that's happened. Software development is an almost perfect place for this middle-way to have been established. We take computers – literal concept machines that can't do anything else – and we create things that we throw into the world, exposing them to the full spectrum of human variety, interaction, expectation, peculiarity and experience. We've learned the hard way that a hard-conceptual approach doesn't work very well. The humans are too complicated. The specification for your new software is always wrong, no matter how much it is thought through. The Agile Manifesto is an almost perfect expression of this: you could rephrase the entire thing as "Through this work we have come to value the experiential over the conceptual".

Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

The Agile Manifesto

Unfortunately, outside of software development, this view is not widely shared. In most big-organisation settings, the dial is still firmly set to "conceptual", and software development has to sit in that. The boundary between software development teams and their wider, usually more traditional organisations is famously challenging. Entire classes of job exist just to manage it. You know you're in one when teams talk about "top cover" and product managers view their literal job – I have heard this more than once, explicitly – as being a "shit umbrella" for their teams. I recently read a very well-meaning post written by a designer in an attempt to reframe the terrible organisational dysfunction they’d had to endure in their role as a forcing function for good ideas to survive. And I've seen people give in to this way of thinking in order to be promoted, or otherwise progress their careers. This is plainly not a good place to be. This kind of oppositional culture – where conceptual and experiential modes of knowing become fossilised into adversarial camps – is anathema to creativity and, arguably, to human flourishing generally.

I think you have to lose a bit of your soul to go from Director to Director General...

Anon senior civil servant

But governance ends up this way for a reason. The real world is chaotic, messy, unpredictable and full of risk. Building these conceptual frameworks is necessary because it renders the chaos of the world manageable. In order to make sensible decisions about what to do, we need to be able to think about the world clearly. And to do that, we have to make simplifying assumptions about it. If we didn't we'd be constantly paralysed by the complexity of it all. But we make a dire error if we mistake the conceptual world we create in order to get things done for the actual world that we live in.

The map is not the territory

Alfred Korzybski

All models are wrong, but some are useful

George Box

I think I risk cliché in quoting these – they are mentioned so often – but we’re living the problem they describe. Our thinking about how to do things, make decisions and measure what we've achieved is often so abstract that a terrible rift has grown: between the senior responsible people and the business people on one side, and the front-line implementers and providers of services on the other.

The senior responsible people set strategies and policies, measure things, and work in a very abstract, very conceptual world. They mostly make decisions that are, in their own terms, plainly rational and reasonable. But this is often not the impression of the implementers and deliverers, whose work is much less conceptual, because they personally experience what the organisation does.

They have to confront the chaotic, messy reality of the world. They have to decide how to fit the policy to the situation when the two don't align. They have to look the users of the organisation's services in the eye. And when they do, they experience some of the emotional journey of that person. It's much more embodied. And from that position, much of the conceptual world of their leaders seems completely detached from reality. Because it is. But, in its own way, so is the world of the implementers and providers: for the most part, they don't really understand the constant, high-stakes balancing act of risk and opportunity that is the primary responsibility of their leaders, nor the sense of duty and obligation that usually go along with it.

And thus the status quo is built in many large organisations. With the leaders and the doers staring at each other across the void, neither really understanding the other, and both compelled by their incentives to move further apart – the leaders towards the conceptual, because it makes things manageable and measurable, and the doers towards the experiential, because it makes things practical and achievable – or often, as in policing, medicine or social work, because it forces itself upon you.

Over time, as organisations and systems have scaled – and in so doing, made abstraction more necessary – there has been an inexorable drag towards the conceptual across society. And I think that's a problem. If we're going to run big systems (which we must) then we have to work in abstractions. It's the only way to make running big things manageable: simplifying things, so that we can reason clearly. But I'd like to argue that the incorporation of experiential knowledge and understanding is an absolutely fundamental component of a rational, conceptual approach, not something that stands opposed to it.

If you're trying to do something effective in the world, you have to start with an understanding of the world as it is, as best you can. And experiential practice is part of how human beings come to understand things. If we neglect this, we cannot reach that understanding. And from that position, reason is a busted flush: you'll be making good decisions based on a flawed understanding, and producing bad outcomes as a result.

On two occasions I have been asked, "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" [...] I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.

Charles Babbage, Passages from the Life of a Philosopher

This isn't just theoretical. The consequences show up clearly in the daily reality of working in or interacting with these organisations. Across government and society, we have processes for things like procurement, recruitment, employment and planning that have evolved over a long time. We want them to be fair and predictable. We've designed them explicitly to be orderly, so that we can run processes that can generate evidence, be measured, and be used to hold people to account. Naturally enough, this has led them to become dominated by conceptual thinking. We've created abstract rules, measurable criteria and frameworks for people to follow. But we've done this to such an extent that many of them have become detached from the messy, complex reality of the world.

In so doing, we end up prioritising process, accountability, measurability and evidence-gathering over actual outcomes. They therefore fail in a fundamental way: not because the process itself falls over, but because it doesn’t deliver a good outcome. Sometimes, we even define the quality of the outcome in terms of the process. We say "if the process was followed, the outcome is fair/good". As if it's possible to declare as such by fiat.

I think there's a good argument that no process that delivers the wrong outcome can really be considered "fair", no matter how clever it is. Yet that's just what we do, across so many areas of professional life. We've collectively fallen into a trap, where we conflate process itself with fairness; where forcing things into categories that allow them to be numerically compared is an unqualified good; where good process and good governance mean minimising subjective judgement from the work of all but the most senior professionals, rather than encouraging and celebrating it as a quintessential part of what professional practice is for everybody.

These hyper-conceptual cultures of work have lost sight of something terribly important: that it's in fact irrational to try to eliminate human judgement and subjectivity from systems. In so doing we might create systems that, with a narrow view, seem fair, transparent and effective. But just a tiny widening of that perspective reveals it to be a façade.

The cultural assumption that a rational, professional, mature approach to governance means excluding or minimising the experiential and working in progressively deeper abstractions is one we need to challenge. We need a rebalancing: a recognition that human judgement is not a failure mode but a vital part of how complex decisions are made well. We need leaders who recognise the limits imposed on them by the necessity of working in a conceptual bubble. And we need to pierce the bubble where we can: more management-by-walking-around, more co-design, more agile practice, more trust, more decentralised decision-making. More "people and interactions over processes and tools".

We need to find space in our organisations for spontaneity, exception, serendipity and discretion. We do need good process, because a hard-experiential approach would definitely not be better – but we also need professional cultures that empower people to set process aside when it's justified, without fearing for their credibility (or their careers). We need to empower teams to make good decisions and then trust them to do the work. We need to practice describing what we know experientially in terms that others will understand, so that more options for transparency and accountability exist than the rendering of everything into numbers.

Good governance doesn't subsist in a collection of policies and processes. It's a habit of attention. Thinking clearly is important to almost everything we do, but whatever clarity we find is never absolute: even on our best days, the search for it happens in foggy weather.