What Three Years of Our Own SOC Data Taught Us About Trusting AI

What Three Years of Our Own SOC Data Taught Us About Trusting AI

We didn’t trust AI at first, and honestly, we shouldn’t have

One of the things my team and I  saw maybe two or three years ago, when we were starting out with some smaller, bespoke, self-hosted models, was that accuracy wasn’t where we wanted it to be. A byproduct of that was we didn’t really trust it. We were like, “Well, humans are way better, right?”

So we set an arbitrary goal: once AI gets to 90%, or on par with what our analysts, security operations, and IT operations teams do from an accuracy perspective, then we can go ahead and let it rip in production. And as we were hovering at 65%, 75%, 85%, we were like, oh, we’re getting really close.

The number that actually changed things

Here’s the part that changed everything. My team started using statistical analysis to look at what we’d done historically, three years of our own cases. We found that, as humans, we were on average probably about 80 to 85% consistent with each other to begin with. So even as humans, we do things differently. Expecting our agents to do even better than we do ourselves was never realistic.

Once we got there, the trust had already increased, because now we had the statistical data to back it up. We weren’t chasing perfection against a human bar that didn’t actually exist.

That doesn’t mean we walked away. We act as an oversight committee now. We watch what the agents do, and we use other models and tools to make sure we’re not deviating. If something changes, say we change the model,  we go back and rework the prompts, the skills, the tasks.

Building versus buying: the VIP insider-threat agent

A great example of build vs buy is that over the last couple of months, we built a VIP insider-threat and risk-management agent. We could certainly have gone out and bought a UEBA with advanced AI tech and tooling. But instead, the team looked at what we already had. It turned out we already had all the data across four different tools: our identity provider, our EDR tool, our zero-trust network agents, and our SSPM.

Instead of buying something new, we used Hero AI, through the agent’s tools, to do multi-turn analysis, grab the data from all those sources, orchestrate it, correlate it, aggregate it, and then look at anomalies from a behavioral perspective. By doing so, we saved $100K. We did that in about a week.

That’s a strong use case for build vs. buy.

The numbers back it up

Through Turbine, we brought that 40-minute mean time to respond down to about 15 minutes. Now, with Hero AI-enabled investigations, we’re closing over 90% of our investigations autonomously. In July, our mean time to respond and remediate dropped to 6.7 minutes.

Speed is important, but accuracy matters more. When you pair accuracy with speed and that velocity, that’s truly probably the biggest gain we’ve seen.

A boring day is a good day

I know a lot of security practitioners will agree that in security, a boring day is a good day. 

But when things are on fire, that’s often when my team and I have the most fun, because we’re doing the deep investigations and figuring things out without feeling like we’ve drowned, since we now have an agent handling a lot of the tier-one and tier-two work. That’s been key.