Engineering Investigator

The problem

finding an explanation ≠ establishing a cause

Ask a capable agent why the app is slow and you get an explanation. That is the problem. It reads the code, finds something genuinely suspicious, and tells you about it:

“I found an N+1 query in the order service — that’s likely your slowdown.”

Likely. Nothing was measured, no alternative was tested, the query may have been there for two years, and the affected customer may be on hotel wifi. You now have a plausible story, a code change to review, and no more knowledge about the incident than before.

What it does instead

report  →  normalize  →  scope  →  observe  →  hypotheses  →  evidence
   →  discriminating experiment  →  eliminate  →  root cause
   →  verify  →  a short answer  +  a client-ready one

When it activates

No slash command needed. Installed, it is loaded when a request needs it and stays out of the way otherwise — and when it does shape the work, one ⚡ line says so. How the skills work together →

What you actually see

The whole response, after an investigation across 218,000 logged requests:

## Result

The application is responding normally for this customer — the time is going
into transferring data over their connection, not into our processing.

Cause        Low network throughput on the affected connection (~2 Mbit/s, where
             comparable customers complete the same requests in about a second)
Confidence   High

Why          Server processing for this account matches every other tenant (p95
             210 ms vs 228 ms), response sizes are comparable, and the customer’s
             own network capture shows 6.4 s downloading a 1.7 MB response with
             only 310 ms of browser work.

Action       Ask the customer to test the same page on a different network. In
             parallel, our dashboard response is 1.7 MB — worth reducing,
             because it is what makes a weak connection unusable rather than
             merely slow.

### Client response

“We’ve looked into the slowness and our systems are responding normally for
your account — as quickly as they do for everyone else. What we can see is
that the data is taking much longer than usual to travel to your computers…”

Every hypothesis carries a kill condition

Written when the hypothesis is created, before any evidence arrives. A claim that nothing could disprove is a hunch, and hunches do not enter the ledger.

H2  Database regression on the checkout path

Plausible     Checkout p95 rose 6× with flat traffic; the endpoint is DB-heavy
              and Tuesday’s deploy touched the cart serializer.
Kill          DB time flat across the window, OR the slow requests never execute
              the suspected query, OR the previous version is equally slow on
              the same input.
Status        Investigating          Confidence  Medium
Next          Compare query count for POST /checkout across v1.8 and v1.9

Writing the kill condition first is what stops the investigation from drifting into collecting whatever evidence happens to be convenient. And the uncomfortable hypotheses — our own last deploy did this, nothing is wrong with the application — go in the ledger from the start, or the investigation can only ever convict the code.

Experiments, not more reading

When several explanations are live, the next action is not “read more code”. It is the cheapest safe observation that eliminates the most of them — and its outcome is predicted per hypothesis before it runs, which is what makes the result an elimination rather than an interpretation.

Experiment 2 — where is the checkout time spent?

Method     Time one affected request end to end: server processing, transfer,
           render. Same request, unaffected user, as comparison.
Predicts   H1 backend      → TTFB high, transfer normal
           H2 database     → TTFB high, and DB time dominates the trace
           H3 payload/net  → TTFB normal, transfer high
           H4 rendering    → TTFB and transfer normal, paint late
Observed   TTFB 148 ms · transfer 4.8 s · payload 1.9 MB · paint +180 ms
Eliminates H1, H2, H4 disproven. H3 supported.

Four hypotheses, one observation, three eliminations. An experiment whose every outcome would leave the ledger unchanged is not worth running.

Confidence that means something

A suspicious piece of code is POSSIBLE. Timing correlation with a deploy is LIKELY at best — something always changed near the start of any incident. What promotes it is a revert, a version comparison, or a trace that attributes the time.

It can conclude that it is not your application

Client networks, devices, browsers, CDNs, regional connectivity, third-party degradation — all of it is in scope, and the skill is comfortable saying the code is fine. But “not our fault” is a finding that needs proof like any other. Three things are required:

• Our side healthy in the same window the failures occupy
• An unaffected comparison — other users, regions, or requests on the same path
• The external factor measured: throughput, a provider error code, an edge header

With all three, the answer is “our servers responded normally while the affected connection transferred data far more slowly than the others we measured”. With only the first, it is “we have found no evidence of an application-side problem, but we cannot yet say whether the connection is responsible” — a different sentence, and the skill writes that one instead. What it never writes is “it’s your internet”.

Cost proportional to the question

It stops when the cause is established and verified, when two experiments in a row eliminate nothing, or when the remaining question needs access this environment does not have. The last two end in “here is what we know and the smallest thing that would settle it” — a result, not a failure.

A case you can resume

For anything past a quick check, the investigation is persisted:

.agent-investigation/
├── incident.md      report, normalized symptom, scope, capabilities, status
├── hypotheses.md    the ledger — kill conditions, evidence, statuses
├── evidence.md      numbered observations, typed and sourced
├── experiments.md   question, method, prediction, observation, elimination
├── timeline.md      only when sequence is itself evidence
└── conclusion.md    cause, confidence, verification, recommendation

Only the files that carry state get created. Say “continue the investigation” days later and it reads the case first: disproven hypotheses stay disproven, completed experiments are not re-run, and it picks up the open question — then tells you what changed this session, not the whole story again.

Install

Any Agent Skills-compatible agent

Claude Code

Using it

• “Users are reporting that the application is very slow.”
• “Checkout is randomly failing for some customers — investigate.”
• “This API returns incorrect data intermittently.”
• “It works locally but is extremely slow in production.”
• “Continue the investigation.”
• “Show the evidence.” · “What did you rule out?” · “What did you not check?”

Safety

Read-only by default. Every action is classified OBSERVE, REPRODUCE, or MUTATE, and investigation authority is read authority: production data, configuration, infrastructure, deployments, migrations, and security controls are never touched without explicit authorization for that specific action.

When the decisive experiment is a mutation — reverting a release to see whether the symptom disappears — it is presented with its blast radius, how to undo it, and what each outcome would prove, and then it stops and asks. Secrets, tokens, and personal data never enter the workspace or the answer.

Pairs with the other skills

incident  →  engineering-investigator  →  cause  →  proof-driven-dev  →  production-guard  →  ship

Engineering Investigator establishes what is actually happening. Impact Map maps what a fix would touch, ProofBuild proves the fix does what it claims, and Production Guard decides whether it is safe to ship. None requires the others.

Limitations

• It cannot see what it cannot reach. With no logs, metrics, or traces it establishes candidate mechanisms and the evidence that would discriminate between them — not a cause, and it says so
• Confidence above LIKELY usually needs reproduction or a controlled comparison; in a repository-only environment most conclusions honestly stop below that
• It investigates one symptom at a time — two unrelated incidents in one report should be split
• It does not monitor, alert, or page. It answers about a window that has already happened
• It will not mutate production to find out, which means some questions stay open until a human authorizes the experiment
• Debugging assistants and incident-response tooling already exist. The contribution here is the method: kill conditions, discriminating experiments, and a conclusion that stops at the confidence its evidence supports