Turn an incident lesson into the next task’s test cases

Recover a sequence, not only a label.

“Retry bug” is not enough information to design a test. You need the sequence of events, the boundary at which the system made a commitment, what the client observed, and the behavior that was wrong. Without that, the next test may exercise retries while missing the original failure.

Use the fictional order-creation example in this guide. A server accepts an order. Its response is lost. The client retries. The system should recover the accepted result under the same request ID rather than create another order. This is a demonstration contract, not a report of a Shifu customer incident.

Separate observation, policy, and uncertainty.

An incident can establish that an event happened without establishing the correct product policy. Record both. The fact that a response was lost does not by itself define how long request identifiers should remain valid or how a reused ID with a different body should behave.

Before writing code, classify the material. A reported symptom may need confirmation, an accepted requirement can define an invariant, and an unanswered policy needs a decision. Keep the categories visible in the task brief.

Write the behavior before choosing the test level.

Describe the key check in plain language. That lets a reviewer catch a wrong assumption before the test is embedded in fixtures and helper code. The test level follows the boundary that owns the behavior.

A client unit test can show that the identifier is reused. It cannot, by itself, establish that two server requests cannot both create orders. That guarantee may depend on persistence or another shared boundary. Use the narrowest real integration that can exercise the relevant behavior, and distinguish it from mocked orchestration.

Given a request has been accepted and its response is lost,
when the client retries with the same request ID,
then the client receives the original accepted order,
and the system has created only one order for that ID.

Challenge the check before trusting it.

Ask what defect would make the test fail. If removing duplicate prevention still leaves it green, the test does not prove the important invariant. If the test only asserts that a function was called twice, it may establish retry orchestration while saying nothing about the accepted result.

Make the failure sequence explicit. A timeout injected before acceptance and a dropped response after acceptance represent different states. For concurrency, coordinate the overlap deliberately instead of relying on sleep timing. A flaky test may hide the very boundary it is meant to protect.

Inspect the assertions with the same care as production code. Do they identify the original order? Do they observe the relevant side effect? Is cleanup isolating runs? A passing test whose setup never reaches the failure state offers little reassurance.

Ask an agent for a failure-to-check map.

Give the coding agent the incident sequence, the accepted contract, current implementation, and unresolved questions. Ask it to propose a small map from each failure mode to an observable assertion and the boundary that owns it. Review that map before requesting implementation.

Require it to mark hypothetical cases separately from the incident that was actually observed. A broad list of possible problems can be useful for exploration, but it should not rewrite the incident history or imply that every risk has occurred.

After implementing the checks, run the relevant suite and inspect the results. Record execution separately from proposed tests. A test plan is not a test result.

Leave a reason for the next change.

Keep a short explanation of why the regression check exists, linked to the accepted contract or permitted incident record. Future maintainers should be able to decide whether a changed requirement supersedes the test, instead of removing an awkward assertion without understanding the failure it prevents.

For a team pilot, count useful prevented misunderstandings and remaining misses qualitatively before trying to estimate a productivity gain. The immediate goal is simpler: the next change should not have to rediscover why this behavior matters.