Curiosity’s Wager 9: Recklessness: what must risk change?
Part 9 of 10 in Curiosity’s Wager.
Recklessness concerns how we proceed when our actions could cause harm. That is different from attempting to cause harm. We have a responsibility to anticipate risks and take reasonable precautions. I want to describe how recognizing risk has changed who we involve and how we build. Despite the sophistication of these models and despite what my co-founder and I investigate ourselves, we identified a point where we should bring someone else on board. We brought in a staff engineer with years of experience to help with our continuous integration pipeline and how we can confidently deliver the software we're developing. That's alongside, of course, working with our design partner, someone who knows the practice, who can reveal the consequences we might not anticipate alone. The engineer brings different experience and ways of questioning the technical work. One thing expertise provides is knowing what questions to ask. In my research on search, access to answers wasn't the whole story. Someone first needed to recognize a reason to ask and find words that could help them investigate. These tools, these AI tools today, raise a similar question. How do we recognize what we need to ask when we're operating in an area that is new to us? So we're trying to address risk through different kinds of testing, state machines, simulations, attention to recovery. We work to establish clear boundaries around where people retain authority and where systems may operate independently. We can also create separation between a proposed change and its consequential use, leaving opportunities to inspect and reconsider. These precautions need to work in practice, acknowledging a risk doesn't itself address it. A shared curiosity is part of recognizing risks, developing new questions and changing what we do in response. It doesn't replace technical safeguards or responsibility for acting.
I would contend that it can be reckless to assume we needn't investigate these tools or the arrangements taking shape around them. This doesn't mean everyone must adopt the tools and try them out. It means developing enough understanding with others to judge when to refuse, when to refine, and what we might want to make possible instead.
Series contents · Previous: Hubris: ordinary confidence, scaled up? · Next: What should we become able to do together?