Skip to content

Grant ingestion filter audit — registry PI gate

Date: 2026-06-14
Branch: development
Trigger: ORS annual report bulk intake (sources/inbox/2026-06-14-ors-annual-reports-2020-2025/)
Policy: Grant ingestion governance
Status: Audit complete — Phase 2 implemented 2026-06-14 (unresolved PI apply)

Finding

Current bulk grant incorporation requires a resolved person_id in people.yaml before writing to data/grants.yaml. Evidence-backed awards whose PI is not in the registry are parsed and indexed in the packet but not merged into the corpus.

This violates the intended IKaC principle: unresolved knowledge is still knowledge.

ORS packet impact (2026-06-14 run)

Stage Count
Parsed (raw) 1,000
Deduped ~865
Registry PI (deduped) 144
Applied to grants.yaml (required person_id) 133
Tier C / non-registry (not applied) ~723

Within tier C, ~36 rows carry collaborative-signal titles with unresolved PIs — previously treated as discard-only.

Where filtering happens

1. scripts/intake_ors_annual_reports.py (primary)

Function / block Behavior
match_person() Sets person_id or None from people.yaml last-name index.
classify_tier() Returns C when person_id is falsy (lines ~178–193).
run_intake() apply loop to_apply = [a for a in deduped if a.person_id and a.tier in {"A", "B"}] — hard registry gate (~499–503).
award_to_grant() Builds people: [] when no person_id; apply never calls this for tier C (~379–381).
dump_tier("tier_c_unmatched_or_weak.yaml", tier_c) Preserves non-applied rows in packet only (~473).

Effect: Tier C rows never reach data/grants.yaml on --apply.

2. scripts/build_grants_layer.py (seed / CSV path)

Block Behavior
Main loop Iterates people.yaml only; seeds grants from grant_pi + Internal Grants CSV (~509–532).
grant_from_csv_row() Always requires person_id + person_name (~120–145).

Effect: Cannot seed grants for PIs not already in people.yaml. By design for CSV sync; not the ORS bulk path.

3. Cursor refactor prompts (prior practice)

Location Behavior
sources/inbox/_template/refactor_prompt.md Required people.yaml associations; no explicit “add grant with unresolved PI” rule (updated 2026-06-14).
ORS refactor_output.md (prior) Framed tier C as “rejected” / “not in scope” rather than unresolved corpus candidates.

4. scripts/export_graph_tables.py (downstream)

Check Behavior
person_grant_links validation Warns if person_id missing from people.yaml (~155–159).
Graph export PI_ON edges come from PGL — grants with unresolved PI on grants.yaml alone would not export PI edges until PGL exists.

Effect: Unresolved grants in grants.yaml are safe for corpus storage if PGL is omitted; graph export remains conservative.

5. Not a filter (already supports unresolved names)

Location Behavior
data/grants.yaml (RFUMS sub-awards) people: entries with person_name + role + notes, no person_id, for external PIs — precedent for unresolved investigators on grant records.

What is not filtering (clarification)

  • Facility naming rules — correctly limit grant_resource_links; not the cause of 723 missing grants.
  • Dedupe — removes duplicate rows, not unresolved PIs.
  • Tier A/B split — affects facility link confidence, not registry gate (B still requires person_id for apply).

Do not run until explicitly requested. Order:

Phase 1 — Schema and policy (done 2026-06-14)

  • [x] grant-ingestion-governance.md
  • [x] Update intake template, source evaluation policy, refactor prompts
  • [ ] Optional: add grant_evidence_status, person_identity_status, resource_links_status to schemas/grant.schema.yaml when schema dir is tracked

Phase 2 — Intake script (done 2026-06-14)

File: scripts/intake_ors_annual_reports.py

  1. ~~Add tier D or rename tier C subtypes~~ — tier C = unresolved PI; grant_evidence_status filters weak rows at apply time.
  2. On --apply, second apply bucket:
  3. to_apply_unresolved: not a.person_id and grant_evidence_status != needs_review
  4. award_to_grant() with person_name, role: pi, unresolved notes; no person_id
  5. Do not append person_grant_links.yaml rows for unresolved.
  6. intake_summary.yaml: applied_resolved, applied_unresolved_pi, applied_pgl, applied_grl, skipped_duplicates.
  7. Dedupe via is_duplicate_grant() on normalized PI identity + title (+ amount/funder fuzzy match for resolved).

ORS Phase 2 run: +692 unresolved grants, +0 PGL, +1 GRL, 173 skipped duplicates.

Tests: tests/test_ors_grant_intake.py

Phase 3 — Graph export

File: scripts/export_graph_tables.py

  • Grants with unresolved PI: include in nodes_grants.csv; omit PI_ON until PGL exists.
  • Optional: include_in_graph: false on grant node metadata until PI resolved.

Phase 4 — Refactor workflow

  • Regenerate ORS unresolved merge in a separate commit after user approval (not part of policy commit).
  • Resolve PIs via faculty profile packets → add PGL in follow-up commits.

Phase 5 — Generalize

  • Extract shared scripts/grant_intake_common.py for --apply rules used by future bulk sources (ORS CSV export from wishlist, internal grants CSV reconciliation).

Risk notes

Risk Mitigation
Duplicate grants (name-only PI) Dedupe on normalized pi_raw + title + funder; merge evidence arrays
Homonym PIs Set person_identity_status: ambiguous; do not auto-match
Graph noise No PGL until resolved; include_in_graph flags
Corpus bloat ~700 grant rows — acceptable for institutional record; display policy can hide unresolved from public site

Acceptance criteria (future implementation PR)

  1. ORS --apply adds unresolved-PI grants to grants.yaml with evidence paths.
  2. Zero new person_id inventions; zero new people.yaml rows from script alone.
  3. PGL count unchanged for unresolved apply pass.
  4. pytest + mkdocs build --strict pass.
  5. refactor_output.md reports resolved vs unresolved counts separately.