Deep Dive · Knowledge Outlives the Mission

The Evidence Map for Institutional Memory in Long-Lived Programs

The action layer behind the core verdict: how to fund an AI retrieval layer as part of a continuity loop without claiming it transfers judgment.

Source Storm Research Verification 12 citation clusters checked Prepared for Leader discussion

How to use this

Use this before approving a knowledge platform or a retiring-expert capture push. It keeps the conversation on the continuity loop: which decision gate, who owns answer acceptance, what the cited packet contains, and what evidence shows a lesson changed practice.

First moves before scaling a corpus assistant

Choose one high-consequence handoff, not an all-knowledge ambition.

Pick a role or lifecycle decision with high attrition risk and a concrete decision product, then name the source types and experts it requires.

Build a cited decision packet before building a chat experience.

Every answer used at the gate carries source links, version and access labels, known limits, and an accountable reviewer.

Make application observable.

At the gate, a subject-matter reviewer assesses applicability, records the action or non-action, and carries any change into a checklist, standard, training item, or corrective-action record.

Treat retiring-expert capture as an interactive transfer program.

Combine structured interviews with cases, mentoring, peer discussion, and a live question route. A recorded archive is an input, not the completion condition.

Evaluate against real engineering questions and downstream decisions.

Measure citation coverage, answer correctness, escalation rate, time to useful evidence, and documented decision changes. Ingestion volume and chatbot satisfaction are not value measures.

Owner, briefing, proof

Owner

A named continuity owner for the gate plus a subject-matter reviewer who accepts or escalates answers, not only a platform or data owner.

Briefing

The cited decision packet: source links, version and access labels, known limits, flagged conflicts, and the human escalation route.

Proof

Application evidence at the gate: documented decision changes, citation coverage and correctness, escalation rate, and time to useful evidence.

Where to start

Start with one recurring review gate. Convert its recurring question into a cited decision packet with a named reviewer, and log every use for a quarter. If the team cannot name the gate, the owner, or the escalation route, map retiring-expert attrition risk first and let that map pick the gate. Scale the corpus assistant only after the gate shows documented applied lessons.

Claim ledger

12/12

Checked

citation clusters independently traced to public primary sources

Fabricated

invented or unsupported source clusters removed from the public claim set

Corrected

claim wording narrowed after source review

Demoted

useful signal kept out of the headline

ConfirmedNASA Lessons Learned (APPEL): Lifecycle verified as collect, record, disseminate, apply; the repository is the sharing mechanism, not the finish line.nasa.gov

ConfirmedNASA LLIS Lesson 6856: Closed-loop infusion documented; milestone SME reviews and compliance matrices tie lessons to procedures and training.llis.nasa.gov

ConfirmedNASA knowledge-transition report (2022): Capture differs from transfer; recorded interviews alone were insufficient; 60% could not identify a process.ntrs.nasa.gov

CorrectedMayfield et al., SIGIR 2024: A perspective paper on evaluating machine-generated reports, not an empirical NIST study; proposes claim-to-source citation mapping.nist.gov

ConfirmedLost in the Middle (Liu et al.): Relevant evidence placed mid-context gets missed; a longer context window does not guarantee access.aclanthology.org

ConfirmedGAO-02-195: No assurance captured lessons would be applied; cultural and time barriers named in 2002.gao.gov

ConfirmedNASA OIG IG-12-012: 16 of 28 project managers had used the system; users called it outdated; funding did not track contributions.oig.nasa.gov

CorrectedNASA NPR 7120.6: Historical evidence for closed-loop controls; the directive is obsolete, not current mandatory policy.nasa.gov

ConfirmedNIST AI RMF 1.0: Voluntary framework; continuous governance, documented roles, and human oversight support the loop framing.nist.gov

ConfirmedGAO-25-107653: Federal generative-AI uptake documented alongside policy, resource, and appropriate-use barriers.gao.gov

DemotedSpacesuit knowledge-capture case: Sustained capture investment documented; use measurement still open, so it does not prove continuity outcomes.ntrs.nasa.gov

ConfirmedIAEA TECDOC-1675: Nuclear R&D knowledge timeframes, retention risk, and capture techniques for long-horizon organizations.iaea.org

Where the evidence stops

No source shows AI preserving tacit expertise or making a program immune to expert turnover.
No public study establishes a general, independently proven ROI for AI knowledge assistants in long-lived engineering programs.
The audit statistics describe the 2002 and 2012 systems, not current performance.
Long-context research cuts against the claim that more documents or a bigger window delivers the right lesson at the right time.

What would change our mind

A controlled or multi-year field study measures AI-assisted retrieval against mentoring or conventional search on decision quality, rework, safety, or ramp time.
A public engineering organization publishes audited evidence that a source-cited AI knowledge system changed lifecycle decisions or mission outcomes.
A major documented AI retrieval failure in a safety-critical engineering setting resets assurance and accountability expectations.
NIST or another authoritative body issues evaluation or provenance controls for retrieval over high-consequence technical corpora.
A long-running capture program publishes use, transfer, or outcome data that supports or weakens the interactive-transfer finding.

Deep Dive staged from verified Storm Research · nothing here asserts above the registry calibration