Curiosity’s Wager 4: Deference: am I learning to say yes?
Part 4 of 10 in Curiosity’s Wager.
Deference concerns whose judgment determines what happens. A long-standing concern with computing systems is automation bias. Accepting the system's recommendation without adequately exercising our own judgment. I want to describe how I'm trying to avoid becoming a rubber stamp and why that approach still needs questioning.
Bainbridge examined how automation could leave people with crucial responsibilities while undermining their preparation to carry them out.
With coding assistants, an early interaction was accepting a suggested completion by pressing tab. Now agents can propose much larger changes and decisions. And we often find ourselves or studies show people often simply accept what the AI model is proposing. The question becomes, am I judging what's proposed or learning to say yes? It's am I taking responsibility for the decisions that are being made? In my own coding arrangement, I'm trying to make consequential decisions clear while establishing boundaries within which routine work can proceed. I don't want to approve every small step, but I've found I often don't need to, and doing so is costly. Partially because it's then not clear which steps are not small and which ones to attend to. So I want the decisions that reach me to deserve my attention and give me what I need to judge them. For example, an agent can build a prototype within agreed boundaries. I can then interact with it and question what it does rather than approve every step towards something I hadn't yet encountered. But that creates another risk, of course. A working prototype can make one direction feel compelling before I've considered what else we could have built. Making that result inspectable doesn't necessarily make the alternatives also visible. So the goal is oversight where it matters. That includes questioning how I decided where it matters. One thing I examine is my own pattern of responses. Where am I providing corrections? Where am I repeatedly saying yes? Did I understand the choice? Was I given enough information? Or did I just want the work to continue? you know, in my system, I can even see how quickly I'm making a decision. So I, you know, I get a briefing in my system and I review, you know, visuals and graphics or where I can see, actually, look, this is a decision type that I'm just rubber stamping and I need to think about whether or not it's one that I need to put measures in place, mechanisms in place to not allow that to happen. Shared curiosity matters here. I can make these patterns available for others to question. I can learn how they approach decisions and let those encounters change my arrangement. I haven't figured out the best approach. I'm trying to build a way of working in which my judgment stays active and which I can recognize when it isn't.
Series contents · Previous: Credulity: what grounds this belief? · Next: Overreliance: what disappears in the handoff?