Candidate preservation implementation report (Phase 2)

Branch: development Date: 2026-06-14 Status: Implemented — discovery / preservation / provenance only Related: Repository-wide refactor governance · Repository-wide refactor audit · Source Packet Workflow · Source Evaluation and Evidence Policy

Purpose

Repository-wide refactor governance requires every source packet to be assessed against the entire institutional knowledge repository, and requires that signals which cannot yet be promoted are preserved as candidates rather than silently discarded.

Phase 1 established that governance in policy, templates, and prompts. Phase 2 brings the intake pipeline into alignment by making candidate preservation automatic and shared across intake paths, through one common mechanism.

This phase is discovery, preservation, and provenance only. It does not create resources, relationships, graph edges, or hyperlinks, and it does not modify corpus content. Human review remains required for any promotion.

Superseded framing (2026-06-14): This report describes the automatic intake mechanism, which deliberately stops at packet-level candidate capture. It must be read alongside the Promotion Discovery Rule and the Promotion review governance update: automatic capture is the first step, but promotion review (targeted evidence discovery → re-evaluate → promote/defer) is now mandatory for every refactor. Where this report says promotion happens "only during repository-wide refactors," read that as "during the mandatory promotion review that every refactor must now perform."

1. Scripts modified

File	Change
`scripts/candidate_extraction.py`	New. Shared candidate-extraction library: registry-backed matching, confidence rubric, dataclasses, idempotent dedup/merge, packet + repo-level writers.
`scripts/extract_candidates.py`	New. Generic CLI used by all non-structured intake paths (RFUMS / DPU-RFU packets, URL captures, faculty profiles, future packet types). HTML→text reduction; optional `--promote`.
`scripts/intake_ors_annual_reports.py`	Wired to the shared library via `emit_candidates()`; every ORS intake run now writes packet-level candidates automatically.
`scripts/intake_agif_reports.py`	Wired to the shared library via `emit_candidates()`; every AGIF intake run now writes packet-level candidates automatically.
`tests/test_candidate_extraction.py`	New. 10 unit tests covering matching, rubric, dedup, detection, idempotency, and legacy-curation preservation.
`mkdocs.yml`, `docs/reports/README.md`	Navigation + index entries for this report.

No corpus files were modified: resources/, data/grants.yaml, data/people.yaml, data/publications.yaml, data/outputs.yaml, link YAMLs, and graph exports are untouched.

2. Candidate classes supported

Resource candidates (scripts/candidate_extraction.py:RESOURCE_CANDIDATE_TYPES):

facility
laboratory
center
institute
equipment_collection
capability
program
course
person
external_partner

Relationship candidates (resource_relationship): preserved with source entity, target entity, relationship type, evidence, and provenance — no graph edge or hyperlink is created.

Resource candidate schema

candidate_name · candidate_type · source_packet · source_file ·
evidence_snippet · surrounding_context · confidence ·
possible_existing_match · recommended_next_action
(+ mention_count, source_files for provenance/dedup)

Relationship candidate schema

source_entity · target_entity · relationship_type · source_packet ·
source_file · evidence_snippet · surrounding_context · confidence ·
possible_existing_match · recommended_next_action
(+ mention_count, source_files)

3. Intake paths covered

Intake path	Mechanism	Status
ORS annual reports	`intake_ors_annual_reports.py` → shared library	Automatic on every run
AGIF reports	`intake_agif_reports.py` → shared library	Automatic on every run
RFUMS / DPU-RFU packets	`extract_candidates.py` (PDF + HTML capture)	Generic CLI
URL captures	`extract_candidates.py` (snapshot HTML→text)	Generic CLI
Faculty profiles	`extract_candidates.py` (uploads / snapshot)	Generic CLI
Future packet types	`extract_candidates.py` (any uploads/snapshot/extracted)	Generic CLI

All paths share the same candidate_extraction library — no duplicated extraction logic.

Demonstration run (2026-06-14-rfums-dpu-grants-from-2024-and-2023, dry, no promotion): 4 resource candidates + 2 relationship candidates preserved, including Argonne National Laboratory matched at high confidence to EXT-003, plus National Cancer Institute, Resuscitation Institute, and Tobacco Use Research Lab preserved for review.

4. Confidence rubric

Confidence is assigned by assign_confidence() and documented in the module:

Confidence	Assigned when
high	Exact/containment match to an existing registry entity name; OR an explicit named facility/lab/center/institute reference; OR an explicit relationship statement (e.g. "directs", "houses", "in collaboration with").
medium	Strong contextual evidence (named entity) OR repeated references across ≥2 sources.
low	Single mention, ambiguous terminology, or title-only reference.

Repeated mentions raise confidence (medium) and a registry match raises it to high, reflecting accumulated evidence.

5. Deduplication approach

Extraction is idempotent. Identity keys:

Resource candidate: candidate_type | normalize_name(candidate_name)
Relationship candidate: relationship_type | normalize_name(source) | normalize_name(target)

normalize_name() lowercases and collapses non-alphanumerics, so casing, punctuation, and spacing differences collapse to one record.

Within a run, the CandidateCollector merges by key — bumping mention_count, unioning source_files, keeping the richest evidence snippet, and raising confidence.
Across runs, write_packet_candidates() and promote_to_review() load the existing file, index by key, and update matched records in place rather than appending duplicates. Re-processing the same source, overlapping reports, or an annual-report series therefore converges to a stable set.
Legacy curated packet files (hand-authored candidates: lists from Phase 1) are preserved verbatim under curated_candidates: and never overwritten.

6. Output locations

Level	File	When written	By
Packet discovery	`<packet>/extracted/candidates.yaml`	At intake (every run)	intake scripts + generic CLI
Repository registry	`data/review/resource_candidates.yaml`	During repository-wide refactors (human-reviewed)	`--promote` step
Repository registry	`data/review/relationship_candidates.yaml`	During repository-wide refactors (human-reviewed)	`--promote` step

How records move between levels: intake automatically populates the packet-level extracted/candidates.yaml. Promotion into the repository-level data/review/*.yaml registries happens only during a repository-wide refactor, after human review, via extract_candidates.py --promote (or candidate_extraction.promote_to_review()). This keeps automatic intake free of corpus-adjacent writes while preserving the curated repository registries from Phase 1.

7. Remaining gaps between governance and implementation

Relationship entity resolution. Free-text relationship detection records the statement and provenance but leaves source_entity / target_entity as "(resolve from context)". Entity resolution within the sentence window is not yet automated.
Programs / courses / capabilities / equipment are supported as candidate types, but detection currently keys on facility/lab/center/institute surface forms plus structured award titles. Dedicated detectors for course catalog references and equipment manifests are not yet implemented.
People candidates are modeled in the schema and matched via the people registry, but unmatched-PI preservation still flows through the grant intake path (person_identity_status: unresolved) rather than the candidate file.
Legacy curated packets (ORS, AGIF) keep their Phase-1 candidates.yaml; re-running their intake will preserve curation under curated_candidates: and layer auto-discovery beside it, but the two schemas are not yet unified.
Pre-existing test failure (unrelated to this phase): tests/test_ikac_snapshots.py (4 tests) fails on the committed data/grant_resource_links.yaml evidence text ("... Lab: NexGeN"; PI ...) which breaks the metrics-snapshot YAML parser. This predates Phase 2 and was not modified here.

8. Recommended Phase 3 work

Reciprocity & hyperlink suggestion (explicitly deferred from this phase): derive reciprocal-link suggestions from preserved relationship candidates.
Entity resolution for relationship candidates (resolve source/target from the sentence window against the registry).
Dedicated detectors for programs, courses, capabilities, and equipment collections.
Unify candidate schemas so legacy curated packet files migrate into the generated resource_candidates / relationship_candidates structure.
Promotion review UI surfacing data/review/*.yaml candidates with one-click "promote / reject / keep as candidate" decisions. (Promotion review itself is no longer optional — see the Promotion Discovery Rule; this item is only about tooling ergonomics.)
Fix the pre-existing grant_resource_links.yaml evidence-text quoting that breaks the IKaC metrics snapshot parser.

Validation

pytest tests/test_candidate_extraction.py — 10 passed.
Full suite: 40 passed, 4 pre-existing failures (test_ikac_snapshots.py, unrelated — see §7.5).
mkdocs build --strict — clean.