DevOpsDays Cairo 2026 · Track 1

Incident
Commander

Evidence decides how much autonomy an AI agent gets.

What should we investigate next?

Nourseen Tarek Senior year student Alryada University Cyber Security Former observability intern, Ejada

Shahesta Salama Senior year student Alryada University Artificial Intelligence Big Data intern, Samsung Innovation Campus

Phase 1 · The problem

Finding the cause is not enough.
The real challenge is deciding what to investigate next.

Production incident

↓

Multiple plausible root causes

First-screen evidence still fits more than one story.

H1 48%H2 41%

↓

The gap

What should we investigate next?

13.8%

of SRE scenarios resolved autonomously.

63.7% → 17.9%

Accuracy collapse on failures never seen before.

73%

Accuracy reached in about 10 steps, by exhaustive re-checking, not by choosing well.

Systems emphasise the quality of the final diagnosis. We focused on the quality of the path that leads to it.

Phase 1 · What we built

An investigation loop — not a single answer.

01Incident

02Competing hypotheses

03Current beliefs

04Candidate checks, scored

05Run the most informative one

06New evidence → beliefs update

H1predicts connection_wait rises first

H2predicts connection_wait rises later

They disagree → this check separates them. Skip the ones both hypotheses would survive.

Next check = argmax( expected information gain ÷ cost )

Continue while uncertainty remains and a discriminating check exists.

Phase 1 · How we proved it

We proved the policy the only honest way: we turned it off.

12 recorded incidents · 24 runs · checks on vs off

Off

Accuracy on the hardest cases

100%

False-positive rollbacks

33%

Why off scores 0% and not 50%: the discriminating signal is reachable only through a check. Turn checks off and it is genuinely gone — so the ablation measures the policy, not the model.

Phase 2 · The problem

A conclusion is not the end.
Acting on it raises a different class of problem.

SafetyA wrong diagnosis cost a wasted investigation. A wrong action costs a second outage.

Blast radiusRolling back a release is recoverable. Failing over a shared database is not. Nothing in a confidence score tells the agent which is which.

Verification“The fix ran” is not “the fix worked”. Without a check, a failed remediation looks like success.

ExplainabilityAfterwards, who proves what the agent was permitted to do? Logs can be edited by whoever owns them.

AutonomyApprove everything and the value is gone. Approve nothing and it is reckless. Where is the line?

Where do we draw the line between a decision the system can make and an action that needs clear conditions?

Config and deploy changes are the single largest source of outages. The actions we would be automating are the risky ones.

Uniform governance across AI agents will lead to enterprise AI agent failure.

Phase 2 · Our answer

Five problems. One rule, and five mechanisms.

Stop

when unsure

Prove

it was allowed

Undo

when wrong

Notice what is not on this list: confidence.

A system that acts because it is confident is exactly the system that causes the second outage.

01 Safety	Abstain when evidence cannot separate causes. Refusing is a designed output.
02 Blast radius	Actions classified by reversibility and reach. Irreversible ones are forbidden.
03 Verification	Recovery judged only from telemetry collected after the action.
04 Explainability	Every decision sealed. Merkle + Ed25519, verifiable offline by a third party.
05 Autonomy	The line is reversibility, not confidence.

Architecture

Six layers. The one that decides is not the one that acts.

LiveService

OTLP

01 · ObservabilityTelemetry

02 · SREDetection

03 · Decision theoryInvestigation

04 · Policy-as-code Policy · OPA Outside the agent. The agent cannot widen its own permissions.

ALLOW

Only after allowExecuteOn the service

05 · Control loopVerificationFresh telemetry. Not recovered → auto-revert.

06 · CryptographyAuditDecision, evidence and actionMerkle tree · Ed25519 · offline verifier

The loop we already hadWhat acting required

10 engine modules15 unit tests14 policy tests 63 live trials24 benchmark runs3 strategies

Found in our own system

A transient latency blip vetoed a correct diagnosis. We fixed it and re-ran 19 trials. The fix fired in 4 of 10 runs.

Live demo

We’ll break the service now and watch it recover.

No clicks. No approvals. The system decides.

detect → investigate → policy → act → verify → seal

Evidence

One demo proves nothing. We ran 63 trials.

LIVE-0073Resolved in 17.6 s · beliefs 0.25 → 0.99 · 15 leaves sealed

Outcome	Trials
Confirmed → acted → verified recovered	10 / 10 ✓
Acted on exclusion → verification caught it → reverted	5 / 5 ✓
Evidence insufficient → abstained, no action	5 / 5 ✓
Irreversible action proposed → policy denied	5 / 5 ✓

	Ours	Exhaustive	No investigation
Resolved, reversible fault	80%	100%	0%
Harmful actions	0	0	0
Unreverted changes	0	0	0
Probes per incident	2.25	2.75	0
Canary traffic exposure	40 %·s	120 %·s	0

Comparison ran before the noise fix. The 10 / 10 above is the current code. Not tested: a real production environment, more than four failure shapes, a measured human baseline.

In DORA terms

Enterprise-grade metrics, delivered.

DORA · Four key metrics

MTTR

≈ 30 s

Mean Time to Restore. Elite tier is under 1 hour. From SLO breach to verified recovery, sealed.

Change failure rate

33% → 0%

False-positive rollbacks eliminated. Verification + auto-revert catch what a confidence score cannot.

Failed-deploy blast

Unreverted changes across 63 live trials. Failed deploys self-heal instead of escalating.

Deploy freq / lead time

Unblocked

Offline-verifiable audit chain removes the manual post-mortem gate on faster deploys.

Audit

Every decision is sealed. Change one, and the chain says so.

$ python verify.py LIVE-0073.postmortem.json

incident LIVE-0073
leaves 15
signature OK

VERIFIED · exit 0

$ python verify.py LIVE-0073.tampered.json

TAMPERED leaf
opa://…/02-rollback_deploy
ALLOW → DENY

TAMPERED · exit 1

Not logging. Evidence a third party can verify offline — without trusting us.

Worth

What this is worth depends on what one wrong action costs you.

Measured by us · 63 live trials

Wrong actions caught and reverted, 5 / 5.
Wrong rollbacks, 33% → 0%.
Harmful actions, 0 / 63.
Every decision provable, 100%.

Supplied by you

Incidents per month____

Revenue per minute____

Minutes the outage was extended____

Internal tool

Customer-facing

Revenue path

Incidents / month

200

One wrong action

$300

$600

$50,000

Value / year

$14,442

$108,208

$40,077,319

Before any of this: without a provable record of what the agent was allowed to do, the answer to “can we let it act” stays no.

Sustainability

We sustain the system by avoiding unnecessary work.

Cost & environment

9 / 24

checks avoided. Less unnecessary telemetry and query work.

Security & reliability

5 / 5

wrong actions reverted. Verification limits the cost of a wrong decision.

People & governance

Policy + evidence chain

Actions require authorization and remain independently verifiable.

Sustainability is disciplined use of compute, infrastructure, operational risk, and human attention.

Future work

From Proof-of-Concept to Real-World Evidence.

Test outside our lab

Evaluate Incident Commander on real incident workflows using existing observability infrastructure.

Run in the shadow

Observe real incidents without taking control. Compare our decisions with what SRE teams actually did.

Earn the right to act

Progress from human-approved actions to bounded autonomy, limited by reversibility, scope, and policy.

Learn from operations

Learn from human interventions, verified operational outcomes, evidence selection, and the autonomy policy.

Prove the value

Measure real-world impact: investigation effort, unnecessary actions, recovery cost, and ultimately ROI.

Finding the cause is not enough.The real challenge is deciding what to investigate next.

An investigation loop — not a single answer.

We proved the policy the only honest way: we turned it off.

A conclusion is not the end.Acting on it raises a different class of problem.

Five problems. One rule, and five mechanisms.

Six layers. The one that decides is not the one that acts.

Live demo

One demo proves nothing. We ran 63 trials.

Enterprise-grade metrics, delivered.

Every decision is sealed. Change one, and the chain says so.

What this is worth depends on what one wrong action costs you.

Measured by us · 63 live trials

Supplied by you

We sustain the system by avoiding unnecessary work.

From Proof-of-Concept to Real-World Evidence.

Test outside our lab

Run in the shadow

Earn the right to act

Learn from operations

Prove the value

Finding the cause is not enough.
The real challenge is deciding what to investigate next.

A conclusion is not the end.
Acting on it raises a different class of problem.