Engineering context · Pilot study
Better engineering plans with remembered context.
Five engineering scenarios tested whether prior decisions, rationale and constraints help AI prepare the next task. The same model answered each question with and without Shifu.
Without Shifu
5.01Average score / 10With Shifu
7.46Average score / 10Observed improvement
+2.44Points across five primary tasksThe engineering scenarios
These scores measure pre-coding engineering plan quality: applying the relevant history before implementation. They do not measure executed code, task completion time or developer productivity.
| Scenario | Without Shifu | With Shifu | Change |
|---|---|---|---|
| Retry recoveryPlan safe recovery after an accepted upload loses its response and the client restarts, including compatibility with an older service. | 3.36 | 5.46 | +2.11 |
| Identity isolationPlan a credential transition that preserves distinct private identities and handles pending work safely. | 5.14 | 6.33 | +1.19 |
| Empty-result handlingInvestigate why meaningful source content produces no usable evidence and the same work repeatedly becomes eligible. | 6.08 | 8.04 | +1.96 |
| Correction persistencePlan reprocessing that preserves explicit user corrections while handling concurrent edits and later updates. | 4.84 | 8.47 | +3.63 |
| Historical backfillRecover older missed work after an extraction upgrade without replaying the entire history. | 5.66 | 8.99 | +3.33 |
| Primary task average | 5.01 | 7.46 | +2.44 |
Retry recovery · dimension scores
| Dimension | Without Shifu | With Shifu | Change |
|---|---|---|---|
| Expected usefulness | 2.60 | 4.60 | +2.00 |
| Constraint coverage | 1.53 | 3.78 | +2.25 |
| Rationale application | 2.30 | 5.32 | +3.02 |
| Source faithfulness | 6.80 | 8.00 | +1.20 |
| Implementation readiness | 3.55 | 5.60 | +2.05 |
Identity isolation · dimension scores
| Dimension | Without Shifu | With Shifu | Change |
|---|---|---|---|
| Expected usefulness | 4.70 | 5.50 | +0.80 |
| Constraint coverage | 4.18 | 5.00 | +0.83 |
| Rationale application | 4.65 | 5.93 | +1.28 |
| Source faithfulness | 7.13 | 8.43 | +1.30 |
| Implementation readiness | 5.05 | 6.80 | +1.75 |
Empty-result handling · dimension scores
| Dimension | Without Shifu | With Shifu | Change |
|---|---|---|---|
| Expected usefulness | 5.27 | 7.48 | +2.20 |
| Constraint coverage | 4.63 | 7.45 | +2.83 |
| Rationale application | 4.88 | 7.95 | +3.08 |
| Source faithfulness | 8.55 | 9.23 | +0.67 |
| Implementation readiness | 7.05 | 8.08 | +1.02 |
Correction persistence · dimension scores
| Dimension | Without Shifu | With Shifu | Change |
|---|---|---|---|
| Expected usefulness | 4.25 | 8.23 | +3.97 |
| Constraint coverage | 3.55 | 7.60 | +4.05 |
| Rationale application | 3.68 | 8.88 | +5.20 |
| Source faithfulness | 7.18 | 8.80 | +1.63 |
| Implementation readiness | 5.53 | 8.83 | +3.30 |
Historical backfill · dimension scores
| Dimension | Without Shifu | With Shifu | Change |
|---|---|---|---|
| Expected usefulness | 5.07 | 8.18 | +3.10 |
| Constraint coverage | 4.53 | 8.90 | +4.38 |
| Rationale application | 4.20 | 9.25 | +5.05 |
| Source faithfulness | 7.45 | 9.45 | +2.00 |
| Implementation readiness | 7.05 | 9.18 | +2.13 |
What improved
Each answer was graded on the same five dimensions, weighted equally. The values below average the five primary tasks.
| Dimension | Without Shifu | With Shifu | Change |
|---|---|---|---|
| Expected usefulness | 4.38 | 6.80 | +2.42 |
| Constraint coverage | 3.68 | 6.55 | +2.86 |
| Rationale application | 3.94 | 7.47 | +3.53 |
| Source faithfulness | 7.42 | 8.78 | +1.36 |
| Implementation readiness | 5.65 | 7.70 | +2.05 |
Controls, reported separately
Three additional checks examined setup knowledge, following a new instruction and answering with the required facts already supplied. They are excluded from the headline average. The supplied-facts check reuses a primary scenario and is not an independent sample.
| Scenario | Without Shifu | With Shifu | Change |
|---|---|---|---|
| Setup procedureConnect a context integration while retaining manual capture settings and excluding historical imports. | 4.11 | 7.95 | +3.84 |
| Current instruction priorityPlan onboarding under an explicit manual-only, no-schedule, no-history requirement despite older automatic defaults. | 6.87 | 8.28 | +1.41 |
| Full-context controlRepeat the retry-recovery scenario with all critical constraints already supplied to both conditions. | 8.86 | 9.22 | +0.37 |
How we ran it
- Five history-dependent engineering tasks were selected from real work before answer generation. Three controls are reported separately and excluded from the headline.
- Both conditions used fresh isolated Codex sessions, GPT-6.1 Sol at low reasoning, identical task inputs and one answer per condition. The with-Shifu condition also received the current integration skill and read-only production context retrieval.
- Project-specific expected constraints and source references were held out from the answering models, except in the full-context control. Explicit records from earlier benchmark activities were filtered from retrieved context; the filter is heuristic.
- Jev 1.13.0 scored each answer on five anchored dimensions, normalized to 0–10. Each answer score is the equal-weight mean of those dimensions; the headline gives each of the five primary tasks equal weight.
- Condition labels were hidden from the judge. Both answers were judged using the same task-specific reference packet, retrieved evidence and integration instructions. Those materials verify assertions; expected requirements and current user instructions remain authoritative.
What this tells us
- This internal pilot covers one work history and five selected primary tasks, with one answer per condition. It does not establish statistical significance or represent all engineers, repositories or tasks.
- Scores estimate the quality of pre-coding engineering plans. Generated implementations and tests were not executed; productivity, production safety and customer outcomes were not measured.
- The with-Shifu condition combines integration instructions and retrieval, so the result does not isolate memory alone. Production retrieval is bounded and the live corpus was not frozen.
- Reference documents can contain local, uncommitted contracts or proposals. Their presence in production memory and their deployed behavior were not independently established. A retrieval gap may reflect ingestion or selection.
- The full-context control repeats one primary scenario and is not an independent task domain. Setup gains may reflect instructions supplied by the integration skill.
- Scenario descriptions are anonymized paraphrases, not verbatim benchmark prompts. Private prompts, generated answers, source records, account identifiers and implementation details are not published. Scores are unchanged.