Measure the cost of rebuilding context before measuring AI productivity

Start with a delay you can observe.

A team can produce code quickly and still wait a long time to finish a change. The missing time may be spent finding earlier decisions, asking which behavior matters, explaining an exception, waiting for a reviewer, or correcting the wrong interpretation. Those are different problems and need different interventions.

Choose a recent change with a concrete example of reconstruction: somebody had to find out why a retry was restricted, who could see a field, or which consumer depended on an old response. Start there. “AI productivity” is too broad to diagnose before the team can describe the work and where it became difficult.

Define the task and the finish line.

Pick one service and a small set of comparable changes. State the problem, owner, expected behavior, relevant risk, and the evidence required for acceptance. Keep task size and uncertainty visible; two changes with the same ticket label are not necessarily comparable.

For a fictional retry feature, completion might require the accepted request to survive a lost response, the retry to return the same order, and the relevant tests to pass. A generated patch is an intermediate output. It is not completion if the contract remains unresolved or the reviewer must reconstruct it.

Use a worksheet that follows the work.

Record intervals as they happen where practical, with a short note explaining the work. Estimates made afterwards should be labeled estimates. Track waiting separately from active effort so a slow approval queue is not mistaken for slow reasoning.

Test one intervention, not an entire tool stack.

For an initial pilot, give one set of comparable tasks a checked context brief: problem, current behavior, constraints, source links, and open questions. Preserve the normal engineering checks. Record whether a person helped prepare or interpret the brief; founder-assisted setup and independent use are different experiences.

Keep the coding tool, review expectations, and definition of done as stable as practical. If the model, staffing, task type, and deployment process all change, a difference in time has several plausible causes. A tiny pilot can reveal friction and failure modes, but it cannot reliably estimate a company-wide percentage gain.

Review individual tasks before averaging them. One critical misunderstanding can matter more than several short preparation wins. Include cases where the brief was irrelevant, incomplete, or slower to prepare than the context it replaced.

Write the decision rules before seeing a result.

The thresholds below are deliberately qualitative. Replace them with expectations appropriate to your team before the pilot starts. A team handling high-risk changes may put greater weight on missed constraints than on preparation effort.

• Continue when engineers can prepare and apply relevant context with acceptable effort, and completed changes preserve the important constraints.
• Revise when the idea helps only after substantial interpretation, when source ownership is unclear, or when old decisions are repeatedly treated as current.
• Stop or narrow the use when critical constraints are missed, privacy boundaries cannot be met, or maintaining the context costs more than the demonstrated benefit.
• Record unknowns instead of turning an uninstrumented step into zero time or a missing event into non-use.

Read the result at the level it supports.

A shorter prompt, fewer questions, more generated code, and a confident review are activity signals. The result you want is a correctly scoped change accepted with less avoidable reconstruction or rework. Those observations may move together, but they are not interchangeable.

Our own published planning benchmark concerns model-rated pre-coding plans. It does not establish faster delivery or production correctness. That distinction is why a team pilot should follow real work through review and verification instead of borrowing a planning score as an ROI estimate.

Take the worksheet into one engineering discussion. Ask the owner of a recent change to identify a single episode of reconstructed context and its cause. That gives the team a testable starting point before committing to a larger rollout.