Why AI Patching Fails

Jacob DePriest is the CISO and CIO of 1Password, and before that he was Deputy Chief Security Officer at GitHub. He's the rare guest who was a customer of the company he now protects — he'd been using 1Password since 2014 before joining — and he now sits over both security and IT at a company holding roughly a billion and a half credentials in its vaults. We caught him during Black Hat week in Vegas, and the conversation splits into two threads that turn out to be the same thread: who agents actually are from an identity standpoint, and whether the industry's favorite answer to AI-discovered vulnerabilities — AI-generated patches — actually works.

The short version: it mostly doesn't yet. 1Password's new research team, Off-by-one (with work from two-time Zero Signal guest Keith Hoodlet), ran frontier models against recently patched vulnerabilities that were too new to appear in training data. Less than half the time the model produced anything that looked like a fix, and only around 17% of patches held up as durable. The rest broke behavior, introduced new bugs, or claimed success while making things worse. Around that finding, Jacob lays out how he thinks about agent identity (start with the human, not the machine), what to do about the SailPoint stat that 92% of CISOs lack visibility into AI identities, and why credentials are piling up on laptops again after a decade of pushing everything to the cloud. This one is for security leaders deciding right now how much access to grant agents — and anyone tempted to let a model patch what a model found.

RISC Point is security & compliance consulting tailored to your business. Talk to RISC Point.

Hampton North is the premier US based cybersecurity search firm. Start building your security team with Hampton North.

Sysdig is the leader in AI-powered real-time cloud defense; stop watching and start defending.

The industry spent the last two years building a non-human identity narrative, and Jacob's argument is that reality ran right past it. The NHI model assumed machine identities would be provisioned by engineers who think in architectures. What actually happened is that everyone — finance, HR, the people team — started running Claude Code, Codex, and Cursor to get their jobs done.

That reframing has consequences. If an agent exists because a human delegated work to it, its identity story starts with that human — which is why Jacob describes what's happening as "human delegated identity at that point." The immediate operational problem is that "security teams don't know if it's the human or the agent doing the work," and that's before anything goes wrong. It's also why 1Password's acquisitions of Kolide and Apono weren't opportunistic — they were the output of a first-principles read on where agent identity is headed: toward privileged access, with a human in the loop, scaled up from five requests an hour to five a second.

Jacob also draws a persona line worth stealing: the technical engineer building product-grade agents and the "casual builder" are different users with different identity needs — and "now almost our entire company is made up of casual builders." Conor's three-tier framing — delegated, semi-autonomous, autonomous — maps onto this, and Jacob confirms that despite all the autonomous-agent talk on the show floor, the vast majority of what's actually running today is delegated, human-driven, and mostly local.

Conor puts the SailPoint numbers on the table: 92% of surveyed CISOs lack full visibility into their AI identities, and 86% don't enforce access policies on the ones they can see. Jacob's reaction is refreshingly unpanicked — of course those numbers look bad, everything is changing monthly — but he doesn't leave it there. If you can't get every agent expressed in your identity provider and your logs, segment.

The move is granularity over completeness. Business-oriented agents touching Notion, Slack, or G Suite carry a different risk profile than anything near production, so treat them differently, and use the segmentation to figure out where you need higher-signal alerting. Then comes the question Jacob thinks defines the next six months of this problem: "did a human kick this off on purpose, or was this an agent they asked to do something else and it was just trying to be helpful?" Intent is the thing detection has to resolve, and it's early days everywhere — including at 1Password, where the security team was evaluating vendors for exactly this problem while the corp dev team was doing acquisition diligence, and both landed on the same answer.

Jacob holds both the CISO and CIO titles, and Stuart pushes on the obvious tension: when AI enablement and risk posture conflict, who wins? Jacob's answer is that the conflict is the feature. His security team hears him ask about productivity; his IT team hears him raise risk. Both functions share offsites, strategic planning, and a leadership Slack channel — "we're all on the same team together" — which surfaces incentive clashes early instead of letting them calcify into org-chart warfare. He also thinks the combined title is a trend that's already here and just not reflected in titles yet.

The operating model underneath it is what 1Password calls the velocity weighted risk approach. Instead of a sprawling risk register that goes stale in weeks, they defined core principles: the P0 territory — anything that could expose customer data or touch production vault storage — is non-negotiable. "That's a sacred area where the bar is really, really high." Everything else gets an honest conversation instead of a reflexive no: if we connect this set of documents to that set, what's the worst that can happen?

The practical result is inverted gating. Most companies put the big security gate up front and let experimentation happen only after clearance. 1Password lets teams explore on test data and lower-risk internal systems fast — with a short list of non-negotiables — and applies the heavy review when something moves toward production. It's let them onboard early-stage AI partners quickly where the risk warranted it. The next step, already underway: codifying that risk profile into Markdown and code so agents can answer the third-party-risk question in Slack without a human in every loop.

Here's a reversal nobody predicted. After a decade of pushing everything to the cloud and treating the laptop as a dumb terminal, agentic tooling has dragged work — and secrets — back to the endpoint. "The things that are happening on employees' laptops now would freak out a CISO from like 3 years ago." Local agents need to connect to things, which means credentials on disk again: "It's environment variables that the agents need to be able to connect to things. It's API keys. It's session keys."

Jacob's conclusion is that vaulted credentials matter more than ever, and the interesting design problem is the relationship between the vault and the agent. 1Password published principles for its integrations, and the first one is the load-bearing one:

The 1Password for Claude integration is the concrete example. When Claude needs to log in somewhere, it negotiates with the 1Password browser extension, then hands over control — the extension does the filling, the credential never enters the model's context, and Claude picks back up afterward. The Codex extension works the same way. Jacob is candid that passkeys are still the right direction and that OAuth will be part of the story, but the near-term fight is getting keys off disk and into vaults — even if rotation comes later.

This is the research that gives the episode its title. The prevailing logic on the show floor goes: models are getting scary-good at finding vulnerabilities, so obviously we'll use models to patch them. Off-by-one, 1Password's new research team, tested that assumption with rigor — multiple frontier models, a harness, and a set of vulnerabilities recent enough that the real patches couldn't be in training data. This isolates the generative question: can the model fix something it hasn't seen?

The four-category breakdown, as Conor restates it on the recording: a durable patch; a brittle patch that folds under light fuzzing; a "fix" that breaks the application; or a patch that thinks it worked while introducing new vulnerabilities. Roughly 43% of attempts land in the first two categories combined. Durable patches alone: about 17%. Or as Jacob puts it when Conor reads the numbers back: "It's low." The failure modes matter as much as the rate — a model that "said it fixed it and kind of made it worse" is a worse outcome than one that admits defeat.

Jacob's read is neither doomer nor dismissive. "You can't go fix the problem if you don't identify the problem" — and the team shared results with the frontier labs ahead of publication and is releasing the full harness, software, and test results so any security team can replicate the runs against their own stack. "The numbers tell us we have to do better," and he expects the models themselves to improve. But the company-side answer is context: 1Password's product security team built Sage — "the security analysis guidance engine that we've built" — a harness trained on the team's past decisions and grounded in the 1Password codebase, currently finding vulnerabilities and bringing a human in for the fix. Who owns this problem? Jacob's answer: all three — the labs, the vendors, and CISOs who have to understand where reality actually is rather than where the marketing says it is.

Asked what the security team looks like in three years, Jacob doesn't predict headcount — he predicts a mindset. The teams that thrive will be builders: engineers with security minds, or security people with engineering minds, who can look at an AI-generated detection and say how the harness should work, how the context should be trained, and whether the output is trustworthy. Not to replace engineers — to scale the expert in one area across many engineering teams.

His parting counsel to the leader leaving Black Hat overwhelmed by vendor booths: skip the six-month agent-architecture design project. "Take 4 hours on a Monday morning. Pull up Claude, pull up Codex, see how far you can get." That applies to the GRC person, the privacy person, the SOC analyst — the iteration is what produces a formed opinion about what's next. "You just have to start." He practices what he preaches, spending weekends running experiments himself: "Sometimes I'll have them go at the same time and see who wins."

Guests — Jacob DePriest, CISO and CIO at 1Password; former Deputy Chief Security Officer at GitHub; 1Password customer since 2014 before joining the company.

Books mentioned — None named in the conversation.

Frameworks / models / tools named — Velocity weighted risk approach; non-human identity (NHI) model; Off-by-one (1Password security research team); Sage (security analysis guidance engine); 1Password; 1Password for Claude; Kolide; Apono; Claude / Claude Code; Codex; Cursor; Notion; Slack; G Suite; passkeys; OAuth.

Other people / shows / resources referenced — Keith Hoodlet (two-time prior Zero Signal guest); SailPoint CISO survey (92% visibility / 86% enforcement stats); Silicon Valley (TV show); Black Hat; Anthropic; OpenAI; GitHub; 1password.com/research (Off-by-one AI patching research, harness, and test results).