Reasoning Meets Reliability: Why AI Stalls in the Modern SOC

The real AI question in the SOC is authority

Every security leader has heard the pitch: AI will triage your alerts, investigate your cases and close the gap between alert volume and analyst headcount. Most have already seen AI show up in their tools, often without asking for it. Far fewer are letting it make decisions that matter.

A new Cyberfuturists report, Reasoning Meets Reliability, explains why. Written by Oliver Rochford, founder of Cyberfuturists and a former Gartner analyst, the report moves the conversation past whether teams should use AI. As the report puts it: “AI is already present in the tools security teams use. The decision facing the CISO is how much authority to give it.”

That distinction is the heart of the research. A summary that an analyst reads is one thing. An automated action that changes your environment is another. In the report’s words: “Producing a summary carries limited risk. Suppressing an alert, escalating a case, disabling an account, or containing a device changes the environment and requires a higher standard of control.”

An adoption gap that depends on what you count

The research draws on two extended interviews with practitioners running standalone AI SOC products in production, more than 30 vendor briefings conducted over 18 months, public practitioner commentary and Swimlane telemetry. The two interviewees were chosen to contrast very different teams: the CISO of a European mobility company spanning more than 60 locations, and the sole security practitioner at a US conservation NGO supporting about 1,000 users.

The first finding is that AI SOC adoption numbers rarely measure the same thing. Gartner estimates adoption of AI SOC agents at just 1–5%. Yet many reported adoption figures count pilots, enabled features or analysts who simply have access to an assistant. The report is blunt about the threshold that matters: “Do not count a summary or drafted investigation as operational adoption. The meaningful threshold is whether the SOC relies on AI to make or carry out a defined security decision.”

Swimlane’s own telemetry tells a different story among organizations with established security automation: 94% of customers run agentic workloads in production within the Swimlane Turbine platform. One regional MSSP averages 1.48 million actions per day, 6,000 of them AI actions, with AI and deterministic automation running side by side.

The report is careful not to overstate this. It notes the two figures “do not use the same population or denominator and should not be treated as a direct statistical comparison.” But it argues they raise a question worth testing: “does AI reach production more readily when integrations and operational controls already exist?”

Why AI stalls between the pilot and production

A pilot runs on prepared data, one reliable integration and a manageable case volume. Production brings incomplete telemetry, permission differences, connector failures, manipulated inputs and peak loads. According to the research, the obstacle is not a lack of interest. Across the interviews and briefings, the recurring constraints were “trust, governance, integration, cost and the ability to verify outcomes.”

Trust. Security teams hand over specific decisions only after watching AI perform in their own environment. The report stresses that the system needs an honest way to say it doesn’t know. Forcing every case into a benign-or-malicious verdict “hides uncertainty and creates two operational risks: unnecessary escalations that analysts learn to ignore, and incorrect suppression that may not be discovered until an incident is reviewed.”

Governance. An AI that can take action needs a named identity, least-privilege access and a defined set of tools. Its prompts, models and context sources are production configuration too. They need testing, approval, version history and rollback.

Integration. AI has to work through the tools the SOC already uses, and those inputs are an attack surface. One interviewee, a former detection engineer, flagged prompt injection through logs and attack artifacts as a reason to separate reading from acting.

Cost. Running routine enrichment and ticketing through a model adds expense and variability without improving the decision. The report recommends measuring “cost per validated outcome at normal and peak volumes rather than relying on license price or model-call cost.”

Verification. This may be the most overlooked barrier. As the report states, “A successful command does not prove that the intended security outcome occurred.” The workflow must confirm the account was actually disabled or the indicator actually blocked, then retry, reverse or escalate if it wasn’t.

Crucially, these barriers don’t offset each other. “Integration without verification can automate an error,” the authors write. A workflow earns more AI authority only when all five hold together under production conditions.

What the practitioners did

Both interviewees used AI for detection and investigation and kept response decisions with people. Their concern was practical: they needed to see how the system reached a conclusion, how it showed uncertainty and how it would fail.

The results were still meaningful. At the mobility company, AI correlated HR, public profile and authentication data to uncover credential sharing across franchise locations, a low-severity pattern the team said it would not have investigated at scale. At the NGO, the AI service cost half of the $60,000 annual MDR it supplemented, and “inconclusive” was treated as a valid answer when context ran thin.

Redefining “AI-native” for security operations

There’s a widespread assumption that AI-native security operations demand a brand-new architecture built around models and agents. The research pushes back. “For SecOps, AI-native is not an architectural claim. It means the system’s decisions, actions and outcomes can be tested.”

That reframing leads to a clear division of labor. AI is valuable when evidence is incomplete, contradictory or unfamiliar. It can assemble context, compare explanations and recommend a response. Deterministic automation is better at execution: applying permissions, routing approvals, invoking tools, recording changes and checking results. The report’s verdict: “A production AI SOC needs both.”

Loop engineering: design, execute, validate

The paper borrows a concept from software development and vulnerability research to describe how the two work together. AI interprets the evidence, security automation executes the approved action, and an independent check confirms the result. The authors call this loop engineering, and they draw a firm line: “A workflow that cannot verify its outcome is unfinished.”

Known work and unknown work

The operating model sorts work by uncertainty. Known, repeatable work belongs in deterministic playbooks. Unfamiliar or ambiguous cases go to AI under supervision. When a pattern recurs and the response proves reliable, its steps graduate into a tested, versioned playbook while AI stays available for exceptions. The process also runs in reverse, because “Failed checks and material drift return the workflow to supervision.”

Practical implications for CISOs

The research offers concrete guidance for building an AI roadmap:

• Measure outcomes, not activity. Alerts touched and hours saved show utilization, not correctness. Track verdict quality against a human-reviewed baseline, override rates, time to validated resolution and cost per validated outcome.
• Expand authority one workflow at a time. It’s reasonable to automate enrichment, require approval for endpoint containment and keep identity compromise in recommendation-only mode.
• Invest in controls that outlast the model. Identity, access, integration, validation and audit should let you swap models without rebuilding the authority structure.
• Plan for workforce change early. One interviewed CISO said automating phishing triage displaced existing work within weeks, and the team retrained an analyst into cloud threat investigation.
• Don’t outsource accountability. An MDR or MSSP can supply capability, but contracts should cover model changes, audit access, approval boundaries and rollback.

Automation as the harness for AI

For organizations that already run security automation, the research points to a meaningful head start. Platforms like Swimlane Turbine already connect to security tools, retain case state, apply identities and permissions, route approvals and record actions. In the report’s framing, “These capabilities provide the harness required to move from a model’s recommendation to a governed production workflow.”

The research is careful about what that does and doesn’t prove. Existing automation doesn’t make an AI model perform well. What it does is reduce how much integration and control infrastructure a team must build before testing AI in live operations. A standalone AI SOC product has to prove its reasoning and its surrounding controls at the same time. On an established automation platform, the evaluation narrows to a single question: “does AI improve a defined security decision without weakening the controls already governing it?”

That is the approach behind Swimlane AI SOC, where Hero AI operates inside the deterministic workflows, connectors and approval gates security teams already trust. As the report concludes, Swimlane’s proposition is an AI SOC “built within an established automation platform rather than as a separate decision and control environment.”

The harness test

The research doesn’t ask readers to take any vendor at its word, Swimlane included. “A product claim is not a control,” the authors write. Instead, they offer a harness test of seven questions every CISO should ask before AI is allowed to act, each demonstrated on a representative workflow:

1. Can an AI recommendation become a controlled action?
2. Is the AI governed as an operational identity?
3. What happens when the AI is wrong or uncertain?
4. Can repeatable AI work graduate to automation?
5. Is performance measured against human-reviewed outcomes?
6. Does the workflow verify the security result?
7. Are the economics predictable at production scale?

The full report pairs each question with the proof to request, the red flags to watch for and how Swimlane addresses it. Its standard is simple: “A solution passes the harness test one workflow at a time.”

The goal isn’t maximum autonomy

The strategic question for security teams is no longer whether they will use AI. It’s how they will implement it without weakening the controls their operations depend on. Reasoning Meets Reliability gives CISOs a practical way to answer that question: pair AI’s reasoning with the reliability of deterministic automation, verify every outcome and expand authority only where the evidence holds.

The report’s closing line captures the point:

“The objective is not maximum autonomy. It is a measurable improvement in security outcomes, with sufficient evidence to determine when AI should act, when a person should decide and when the workflow should stop.”

Download the full report to read the complete research, including the seven-question harness test scorecard and the eight decisions for your SOC roadmap. Ready to see how Swimlane Turbine combines deterministic playbooks with Hero AI reasoning? Get an AI SOC demo.