What can another person verify?
“Submitting an empty form sends a POST request and the server returns HTTP 500” is an observation. It names an action and an outcome that tools can capture.
01 / MENTAL MODEL
A symptom is evidence, not the cause.
The visible failure is often several operations away from the defect that produced it. Start by separating what you observed from the explanation you currently believe.
What story might explain it?
“Validation is broken” is an interpretation. It may be useful, but it competes with other explanations: the browser sent an unexpected shape, the route skipped validation, or the error handler failed.
What mechanism is demonstrated?
A cause connects input and prior state through a specific operation to the failure. It predicts what will happen under a discriminating test and survives that test.
What restores the intended invariant?
A repair changes the earliest responsible boundary, preserves nearby correct behavior, and is protected by a test that failed before the change and passes after it.
02 / THE METHOD
Use the same seven steps at every scale.
A syntax error and a production outage differ in cost and tooling, but the reasoning loop is the same. Do not skip forward because a later step feels more active.
- 01
Define the mismatch.
Write the goal, exact input and starting state, expected outcome, and actual outcome. Avoid “does not work.” Name the first user-visible or machine-visible difference. If the expectation came from a requirement, protocol, test, or document, cite it.
Exit evidence:Two people can agree whether the mismatch occurred. - 02
Reproduce and preserve evidence.
Record the exact command or interaction, relevant versions, smallest safe input, frequency, timestamp, and full first diagnostic. Save logs before rerunning if a retry can destroy evidence. Remove secrets and private data without rewriting the behavior.
Exit evidence:You can make the same failure happen again, or you can clearly state the conditions under which it is intermittent. - 03
Read the first useful diagnostic.
Start at the earliest message produced by your code or the closest owned boundary. Read the exception type, message, file, line, call stack, status, error body, and any nested cause. Later errors may be consequences of the first one.
Exit evidence:You can point to the operation that detected the problem, even if you do not yet know why it received bad state. - 04
Reduce the failing case.
Remove half the input, code, state, dependencies, or requests while preserving the failure. Repeat. Replace remote resources with fixed local values when the boundary itself is not under investigation. Reduction reveals which facts are necessary.
Exit evidence:Every remaining part changes the failure when removed or altered. - 05
Inspect one boundary.
Choose where responsibility changes: caller to function, text to parser, browser to server, application to database, process to operating system, client to network. Record value, type, shape, units, identity, encoding, timing, authority, and ownership immediately before and after it.
Exit evidence:You know whether the boundary received correct state and whether it returned or committed correct state. - 06
Test one falsifiable hypothesis.
Write a sentence that could be wrong: “The second request overwrites the first because both updates use the same stale version.” Design one change whose result differs between that explanation and the strongest alternative. Do not change three settings at once.
Exit evidence:The result eliminates at least one plausible explanation. - 07
Repair, verify, and prevent regression.
Fix the responsible mechanism, not only the visible symptom. Run the smallest failing case, nearby boundary cases, and the wider relevant suite. Add a regression test or operational guard. Document why the repair works and what the evidence does not prove.
Exit evidence:The old case fails before the repair, passes after it, and related behavior remains correct.
03 / FIRST FIVE MINUTES
Stabilize the investigation before changing code.
The first minutes determine whether later observations are trustworthy. Capture the scene, then simplify it.
Write this incident line
Given [starting state and input], when [exact action], I expected [observable outcome], but observed [actual outcome] at [time/version/environment].
Example: “Given an empty cart in app version 1.8.2, when I add product 42 once, I expect quantity 1, but the UI shows 2 after one POST request at 10:14 UTC.” The sentence does not yet blame the UI, API, or database.
- Stop random edits.Return to a known state or record every existing change.
- Protect evidence.Copy the first error, request ID, failing input, and relevant logs.
- Confirm the target.Check working directory, branch, process, URL, database, and version.
- Reproduce once.Use one exact command or interaction and record the result.
- Change one uncertainty.Choose the smallest observation that can separate two explanations.
| Ambiguous statement | Useful observation |
|---|---|
| “The API is broken.” | POST /items returned 409 with request ID r-182; GET /health returned 200. |
| “The variable is empty.” | Immediately before parse_price, repr(raw) was ' ' and its length was 1. |
| “The query is slow.” | The plan scanned 8.2 million rows, returned 24, and spent 3.7 seconds in the sort step. |
| “It is a network issue.” | DNS returned one address; TCP connected in 18 ms; TLS failed because the certificate name did not match. |
04 / ERROR MESSAGES
Read an error as a structured report.
An error message is produced by a detector. Ask what operation detected the mismatch, what information it had, and what earlier state could have reached it.
What category was detected?
TypeError, constraint violation, timeout, name-resolution failure, and permission denied indicate different detectors and recovery choices.
What local expectation failed?
Read every word. “Expected integer, received string” describes a boundary mismatch; it does not yet prove which component supplied the string.
Where was it noticed?
A line number identifies the detecting operation. The cause can be earlier. Inspect the values that entered that line and the calls that produced them.
How did execution arrive there?
Read from your earliest relevant call toward the failure. Ignore framework frames only after understanding which boundary they represent.
PYTHON TRACEBACK
Traceback (most recent call last):
File "report.py", line 18, in <module>
total = price_for(row)
File "report.py", line 11, in price_for
return int(row["quantity"]) * row["unit_price"]
TypeError: unsupported operand type(s) for *: 'int' and 'str'Reason from the detector backward
- The multiplication operator received an integer on the left and a string on the right.
int(row["quantity"])converted quantity, so quantity is not the immediate mismatch.row["unit_price"]remained text. Inspect where the row was decoded and what the data contract says about price representation and units.- Do not “fix” this by wrapping the whole expression in a broad exception handler. Normalize and validate the boundary, then test invalid and decimal input.
05 / REDUCE
A minimal reproduction is a causal instrument.
Reduction is not tidying. Each removal is an experiment asking whether a piece of state is necessary for the failure.
1Whole systemUser, browser, API, worker, database
2One boundaryCaptured request replayed directly to API
3One operationHandler called with a fixed validated object
4Smallest stateTwo fields and one failing value
Remove data dimensions
Use one record, one user, one request, one file, one time window, or one character. Then add dimensions back until the result changes.
Remove execution dimensions
Replace concurrency with serial execution, retries with one attempt, cache with a fixed value, or network calls with local fixtures—unless that removed dimension is the suspected cause.
Freeze change
Record runtime, dependency lockfile, configuration keys, database schema version, time zone, locale, random seed where supported, and external response fixture.
Keep the failure signature
The reduced case must fail for the same reason, not merely produce another error. Compare error category, message, location, state transition, or invariant violation.
Worked reduction: duplicate form submission
The full page sometimes creates two orders. First, capture the network panel: one click produced two POST requests with different request IDs. Database and queue behavior are downstream, so replay is unnecessary yet. Remove analytics, styling, and unrelated components; the duplicate remains. Replace the real request with a counter; the counter still reaches two. Remove keyboard handlers; it remains. Remove the form’s submit listener and it reaches one. The button had both a click handler that called createOrder() and a form submit handler that called the same function. The smallest reproduction is one form, one submit button, and those two listeners.
The repair gives the form submission one owner, prevents default navigation where appropriate, and adds a test that one activation produces one logical command. Server idempotency remains valuable protection, but it is not a substitute for fixing the double dispatch.
06 / SYSTEM LAYERS
Locate the failure before changing configuration.
A request can cross document, program, process, network, service, and storage boundaries. Check each layer with evidence that belongs to that layer.
| Layer | Inspect | Useful evidence | Common false conclusion |
|---|---|---|---|
| Input and interface | Raw value, event, focus, field name, encoding, units | Captured bytes, DOM state, command arguments | “The function is wrong” before proving what it received |
| Program logic | Types, branches, loop bounds, state transitions, exceptions | Debugger values, focused logs, unit tests | “The language ignored my code” when another branch ran |
| Process and runtime | Executable, working directory, environment, permissions, descriptors, memory | Process identity, version output, exit status, resource counters | “The file is missing” when the process uses another directory |
| Name and route | DNS answer, address family, route, proxy, listener | Name lookup, route lookup, socket tuple | “The server is down” when the client reached another address |
| Transport and protocol | Connect, TLS, HTTP status, headers, framing, deadlines | Handshake error, request trace, response headers | “The network is slow” when the server returned 429 quickly |
| Service boundary | Validation, identity, authorization, transaction, idempotency | Request ID, structured error, audit event, trace span | “Authentication passed, so access is allowed” |
| Storage | Schema, constraint, isolation, query plan, replication, migration | Constraint name, plan rows, transaction trace, schema version | “The database lost data” before checking commit and read authority |
| Asynchronous work | Message identity, ordering, retry, lease, deadline, cancellation | Attempt number, queue timestamps, deduplication key | “The worker ran twice” without deciding whether delivery may repeat |
07 / PROGRAM STATE
Trace values and control flow before rewriting logic.
Most beginner defects become clearer when you write down the value, type, owner, and next operation at one precise point.
Values and types
Inspect representations that look alike: 0, "0", false, an empty collection, a missing field, and null can drive different branches. Print representations that expose whitespace and escapes. State units beside numbers.
Branches
Write the predicate and substitute the actual values. If a branch did not run, do not add logging inside it and conclude nothing happened. Observe immediately before the decision and record which alternative ran.
Loops
Test zero, one, two, ordinary, and boundary-size collections. Record starting index, stopping condition, update, and collection length. An off-by-one defect is a disagreement about which endpoints are included.
Mutation and ownership
Ask which reference owns state, whether another reference aliases it, and when mutation becomes visible. A function may receive the correct object and still observe unexpected later changes because another component retained the same mutable value.
PLAUSIBLE BUT WRONG
def average(values):
total = 0
for value in values:
total += value
return total / len(values)TRACE THE CONTROL FLOW
With [4, 8, 12], the loop enters once, adds 4, and immediately returns 4 / 3. The formula is not the earliest defect; the return statement ends the function before the remaining values run.
def average(values):
if not values:
raise ValueError("values must not be empty")
total = 0
for value in values:
total += value
return total / len(values)08 / WEB & BROWSER
Separate document, style, script, network, and server evidence.
A blank or incorrect page is not automatically a JavaScript problem. Start at the earliest layer that disagrees with the intended result.
- 1
Document
Is the expected element present in the parsed DOM? Check source, parser corrections, identifiers, native form behavior, and accessibility tree before assuming CSS hid it.
- 2
Style
Inspect the winning declaration, specificity, inheritance, computed value, box size, overflow, and stacking context. Toggle one declaration rather than adding
!important. - 3
Script
Check whether the module loaded, whether an earlier exception stopped execution, which listener owns the event, and which state transition should follow.
- 4
Request
Inspect URL, method, headers, credentials policy, request body, timing, status, response headers, and response body. Console messages cannot replace the network record.
- 5
Server
Use the request ID to connect the client observation to validation, authorization, transaction, dependency, and response evidence without exposing private data.
09 / DATA & DATABASES
Debug meaning before optimizing the query.
A syntactically valid query can return the wrong facts. State grain, identities, time meaning, and invariants before reading the plan.
What does one row mean?
Write one sentence for every input and output relation. A join between orders and items changes one-row-per-order into one-row-per-item unless you aggregate back deliberately.
Which key names the fact?
Check whether the key is stable, unique, scoped by tenant, and valid over time. Similar display names are not identities.
Is missing different from zero?
SQL uses three-valued logic. Test null explicitly, preserve the difference between unknown and zero, and verify outer joins before filtering their nullable side.
What committed together?
Record transaction boundaries, isolation level, constraints, retries, and the state left after rejection. Application logs that say “saved” do not prove a commit.
Where is the work?
Compare estimated and actual rows, scan and join choices, sorting, spills, lock waits, and returned rows. Optimize only after correctness and realistic parameters are fixed.
Which copy can answer?
Identify primary, replica, cache, snapshot, event log, and derived table. A stale read can be correct for its declared authority and wrong for the product requirement.
-- Wrong question: "Why is this total too large?"
-- Better evidence: state row grain before and after each join.
SELECT o.id, COUNT(*) AS rows_after_join, SUM(o.total) AS repeated_total
FROM orders AS o
JOIN order_items AS i ON i.order_id = o.id
GROUP BY o.id;
-- If o.total is an order-level fact, joining one-to-many repeats it.
-- Aggregate item facts separately or select the order fact once.10 / ASYNC & CONCURRENCY
Record event order, identity, and ownership.
“Sometimes” often means the result depends on an order that the design did not make explicit. A timeline turns timing folklore into state evidence.
Request A reads version 7Owner: editor A
Request B reads version 7Owner: editor B
A writes version 8Expected predecessor: 7
B overwrites version 8Stale predecessor: 7
The symptom is “A’s edit disappeared.” The discriminating evidence is not a longer delay; it is whether both writes carried the version they read and whether the storage boundary rejected B’s stale precondition. Optimistic concurrency turns the silent lost update into an observable conflict.
Race
Two operations access shared state and the outcome depends on order. Capture a timeline and enforce ownership, serialization, atomicity, or version checks.
Duplicate delivery
Queues and clients may retry. Use a logical operation key and record whether the effect already committed. “Exactly once” claims require clearly bounded authority.
Cancellation leak
Cancellation must reach child tasks, timers, response bodies, locks, and sockets. Count live resources before and after repeated cancellation.
Timeout confusion
Distinguish DNS, connect, TLS, first-byte, body, dependency, queue, and total deadline. One generic timeout hides the failed stage.
11 / LINUX & NETWORKS
Trace names to addresses, sockets, processes, and files.
Do not restart services or change firewall rules until you know which layer failed. Each step has a different observable contract.
- Artifact
Which executable or script, hash, version, configuration, and working directory should run?
- Process
Does the exact process exist, under which identity, with what arguments, environment, limits, and exit status?
- Listener
Which address and port does it own? Loopback, wildcard, IPv4, and IPv6 listeners have different reachability.
- Name and route
What address did the client resolve, and which route and interface will carry it?
- Connection and TLS
Did TCP connect? Did certificate name, time, trust chain, and protocol negotiation succeed?
- HTTP and application
Which method, host, path, headers, status, body, request ID, and application log correspond?
- Resource and recovery
Are descriptors, memory, CPU, queues, disk, or dependency budgets saturated, and can the known artifact restore service?
Safe command discipline
Before running a command, state what it reads or changes, its exact target, required authority, expected output, and rollback. Prefer read-only inspection first. Never paste an unexplained destructive command, recursive target, credential-bearing URL, or production data dump into a debugging session.
# Evidence questions, not a universal copy-and-run recipe:
pwd # Which working directory am I in?
python3 --version # Which runtime will this shell launch?
curl -v --max-time 5 URL # Which protocol stage and response fail?
# Choose operating-system-specific process and socket tools deliberately.
# Sanitize hostnames, tokens, cookies, and private response bodies before sharing.12 / TESTS & PREVENTION
A regression test preserves the causal lesson.
A test should fail for the original reason before the repair. A test written only after the fix can pass without ever exercising the defect.
Capture the smallest failure.
Name the behavior, construct only necessary state, perform the operation, and assert the observable outcome or invariant. Confirm it fails on the defective version.
Change the responsible boundary.
Make the smallest complete correction. Avoid unrelated refactoring until the causal claim is verified.
Run the focused and nearby tests.
Confirm the regression passes, then test empty, boundary, ordinary, error, and interaction cases that share the repaired mechanism.
Ask what remains unproved.
A unit test may not prove database isolation, browser semantics, file permissions, network deadlines, or deployment configuration. Add the narrowest higher-level evidence needed.
13 / WORKED CASES
Follow the evidence through six complete diagnoses.
Each case separates symptom, hypothesis, evidence, cause, fix, and prevention. Read the evidence before opening the diagnosis.
A report total is too small
Symptom
For prices [4, 8, 12], the function returns approximately 1.33 instead of 8.
Hypothesis
The division formula is wrong, or the loop returns before it sees every item.
Evidence
A trace records one iteration: total changes from 0 to 4, then the function returns. Inputs and division length are correct.
Cause
The return statement is indented inside the loop, so the first iteration ends the function.
Fix
Move return after the loop and define the empty-input policy explicitly.
Prevention
Test empty, one-item, two-item, and ordinary lists; trace control flow when a collection operation sees only its first value.
A card overflows only on narrow screens
Symptom
At 320 CSS pixels, the page scrolls horizontally by 76 pixels. Wider layouts appear correct.
Hypothesis
The grid columns are too wide, or a child has an unbreakable minimum width.
Evidence
Layout inspection shows the grid fits. A code sample has white-space: pre, an intrinsic width of 382 pixels, and no overflow container.
Cause
The preformatted child forces its grid track beyond the viewport’s available width.
Fix
Allow the grid item to shrink with min-width: 0 and make the code block an intentional keyboard-focusable horizontal scroller.
Prevention
Test at 320 pixels, zoom, long tokens, and code samples; audit both document overflow and intended internal scroll containers.
One click creates two orders
Symptom
One visible button activation sometimes creates two order records.
Hypothesis
The client dispatches twice, a retry repeats the effect, or the server lacks idempotency.
Evidence
The browser records two POST requests before any timeout. A reduced form still has both a click handler and a submit handler calling the same command.
Cause
One activation triggers two client event paths. The API also accepts both as distinct operations.
Fix
Give form submission one owner and use a logical idempotency key at the server boundary.
Prevention
Test keyboard and pointer activation, assert one logical command per submission, and test repeated requests with the same operation key.
Revenue doubles after a join
Symptom
A monthly revenue query returns exactly twice the verified ledger total for some orders.
Hypothesis
Payments are duplicated, or joining orders to two item rows repeats an order-level total.
Evidence
Before the join there is one row per order. After the join there is one row per item. Orders with two items contribute their total twice.
Cause
The query aggregates an order-grain measure after changing the relation to item grain.
Fix
Aggregate item facts at the intended grain or join a separately aggregated result; do not sum repeated order totals.
Prevention
State grain beside every relation, reconcile row counts and totals at each stage, and test orders with zero, one, and multiple items.
A slower search replaces a newer result
Symptom
A user types “cat” and then “caterpillar,” but the final list sometimes shows results for “cat.”
Hypothesis
The old request finishes after the new request and both completions may own the same result state.
Evidence
A timeline shows request 2 starts later and completes first. Request 1 then completes and writes without checking whether it is still current.
Cause
Completion order, rather than request identity, determines which response owns visible state.
Fix
Cancel superseded work where supported and commit a response only if its request generation still matches current state.
Prevention
Test deliberately reversed response order, cancellation, empty queries, errors, and rapid input changes with a deterministic fake transport.
The service works locally but not through its hostname
Symptom
curl http://127.0.0.1:8000/health returns 200 on the host, while the published HTTPS hostname fails.
Hypothesis
The process, listener scope, DNS, proxy route, TLS name, or upstream configuration may differ.
Evidence
The process listens on loopback only. DNS and the public proxy are healthy, but the proxy cannot reach a loopback-only listener in another network namespace.
Cause
The application bound to an address visible only inside its own host or container boundary.
Fix
Bind to the explicitly intended internal interface, keep external exposure controlled by the proxy and policy, then verify the full HTTPS path.
Prevention
Test listener address, proxy-to-upstream reachability, health, TLS, request host, and recovery during deployment—not only loopback access.
14 / ASKING FOR HELP
Make the problem reproducible for another mind.
A good help request is a compact evidence packet, not an apology and not a complete project dump.
Include
- the intended outcome and relevant requirement;
- the smallest code, data, or configuration that still fails;
- exact commands or interaction steps;
- expected and actual observable results;
- the complete first useful error and call stack;
- runtime, dependency, browser, operating-system, and schema versions that matter;
- hypotheses already tested and what each result eliminated.
Remove or replace
- tokens, keys, cookies, passwords, and connection strings;
- private names, records, prompts, documents, and logs;
- unrelated source files and generated artifacts;
- screenshots when searchable text is available;
- claims such as “nothing changed” without the command and observed output.
Copyable structure
Goal:
Smallest reproduction:
Exact environment and versions:
Steps or command:
Expected result:
Actual result:
First useful diagnostic:
Boundary already inspected:
Hypothesis tested:
Result of that test:
Sanitization performed:15 / INCIDENT LEARNING
After recovery, explain the system—not the person.
A defect becomes institutional knowledge when the review connects conditions, controls, detection, response, and prevention.
- Impact
Which users, operations, data, security properties, and time window were affected? Separate known impact from possible exposure.
- Timeline
Record deploys, config changes, first failure, detection, actions, misleading signals, recovery, and verification using one clock.
- Trigger and contributing conditions
The trigger began the event. Contributing conditions allowed it to escape, spread, remain hidden, or resist recovery.
- Control behavior
Which tests, limits, alerts, isolation, rollback, backup, and review controls worked, failed, or were absent?
- Corrective actions
Assign owners and verification evidence. Prefer changes that remove ambiguity, narrow authority, detect earlier, limit impact, and rehearse recovery.
16 / WORKSHEET
Complete one evidence sheet before making the third change.
Print this section or copy its headings into an issue. Short factual answers are better than a polished story written too early.
- Goal and source of truth
What should happen, and which requirement, test, protocol, or invariant defines it?
- Starting state and exact input
Which version, environment, identity, data, command, event, and time condition are involved?
- Expected and actual observation
What is the first measurable difference? Include status, value, type, bytes, rows, time, or state transition.
- Reproduction
List exact steps and frequency. Does a clean restart, fixture, or isolated environment preserve the same failure signature?
- First useful diagnostic
Copy the sanitized error, location, stack, constraint, request ID, exit status, or protocol outcome.
- Smallest preserved case
What did you remove? Which remaining facts are necessary for the failure?
- Boundary observation
Immediately before and after one boundary, record value, type, shape, units, identity, encoding, time, authority, and owner.
- Competing hypotheses
Write at least two explanations. Which single test produces different results under them?
- Test result and causal explanation
What did the test eliminate? Connect the demonstrated cause to the observed symptom step by step.
- Repair and verification
Which responsible boundary changed? Which focused, boundary, regression, and wider checks passed?
- Prevention and limits
Which test, constraint, guard, observation, or recovery step prevents the class? What remains unproved?