Source Packet Workflow

Branch: development Status: Design + scaffold (no automated extraction, no LLM API code, no scraping) Related: Development Branch Charter · Source Evaluation and Evidence Policy · Repository-wide refactor governance · Main Evidence & Verification Audit

Purpose

This document defines the source packet workflow: the practical, human-first intake process for updating the DePaul ORS Resource Map.

It restores the original successful pattern that built the working map:

rich source material
  → Cursor refactors the Resource Map (AI-assisted synthesis)
  → evidence-backed diffs
  → human review (Git)
  → ./sync_docs.sh
  → mkdocs build --strict
  → push to main only after testing

It is the operational expression of Institutional Knowledge as Code (IKaC):

Every source packet is a potential update to the entire repository — not a grant-only, people-only, or publication-only pipeline. See Repository-wide refactor governance.
The Resource Map is the product.
Markdown / YAML / Git are the knowledge corpus.
Cursor-style refactoring is the AI engine (for now).
MkDocs is the public publication layer.
Graphs / search are derived views, downstream.

What a source packet is

A source packet is a self-contained folder in the repository that gathers everything a human or AI needs to propose an evidence-backed update to the Resource Map — before any page or data file is touched.

Canonical location:

sources/inbox/<YYYY-MM-DD>-<short-slug>/

Example: sources/inbox/2026-06-13-grice-lab/

A packet contains:

File	Role
`intake.yaml`	Structured metadata: intake channel, curator identity, sources (with provenance), target resources, confidence
`metadata.yaml`	Same role when the packet came from the intake web form (includes `intake_form_source` + `curator_added_sources`)
`curator_notes.md`	Human context, judgment, caveats, and open questions
`evidence_checklist.md`	Per-claim evidence tracking (claim → source → status)
`refactor_prompt.md`	A ready-to-copy Cursor prompt that drives the refactor
`source documents / links`	The raw material itself (PDFs, saved HTML, notes, URLs)
`README.md`	One-line summary and packet status

A packet is working material, not published content. It lives under sources/, which is outside the MkDocs docs_dir, so packets never appear on the public site (see Publication boundary).

Why source packets replace regex scraping

The v2 experiment tried regex-based faculty-profile extraction. It is preserved on archive/resource-map-v2-20260613 as a lesson, not a base. Source packets are the deliberate alternative:

Regex scraping (rejected)	Source packet (adopted)
Brittle pattern matching against changing HTML	Curator captures the source once, in context
Loses surrounding meaning and nuance	Preserves full rich context for synthesis
Produces low-confidence structured guesses	Produces human-judged, evidence-backed claims
No human in the loop until cleanup	Human in the loop from intake to merge
Hard to audit provenance	Provenance is the packet; every claim cites a source
Treats the map as a scraping target	Treats institutional knowledge as a curated corpus

The packet keeps rich source material intact so that the AI synthesis step has the same context a knowledgeable human would have — which is exactly what made the original map good.

How source packets preserve rich context

Capture, don't distill, at intake. Save the actual source (PDF, saved page, email text, meeting notes) into the packet folder. Links are allowed, but a captured copy is preferred because external pages change.
Record curator judgment separately. curator_notes.md holds the human reasoning — what matters, what is uncertain, what should not be inferred.
Keep claims and evidence paired. evidence_checklist.md forces every proposed change to point at a specific source, mirroring the map's existing evidence_text / source_urls conventions (see the audit's §3 and §7).
Name targets, don't assume them. intake.yaml lists target resource IDs only when known; unknown targets stay blank for the reviewer to resolve.

Curator-added sources

Sources enter the repository through two intake channels:

Channel	When	Metadata location
`intake_form`	Submitter uses the local intake web app	`metadata.yaml` → `intake_form_source`
`curator_manual`	Maintainer copies `_template/` and assembles a packet by hand	`intake.yaml` → `intake_channel: curator_manual`
`curator_added`	Maintainer adds files or URL captures after form intake, or any source in a manual packet	`metadata.yaml` → `curator_added_sources[]` or per-source fields in `intake.yaml`

Rule: All sources added directly by repository maintainers outside the intake form must be recorded as curator_added, with these fields preserved in packet metadata:

Field	Required	Meaning
`intake_channel`	yes	Always `curator_added` for maintainer-added sources
`added_by`	yes	Curator name (maintainer who added the source)
`added_at`	yes	ISO date (`YYYY-MM-DD`) the source entered the packet
`added_rationale`	yes	Why the curator captured or uploaded this source

For manual packets (intake_channel: curator_manual), also record packet-level curator, created, and curator_rationale.

For form packets, the primary URL or upload is intake_form (automatic in intake_form_source). Any later maintainer capture — candidate-link follow-ups, PDFs dropped into uploads/, saved HTML — belongs in curator_added_sources with the three provenance fields above. Do not add maintainer captures silently; provenance is part of the audit record.

Refactor output section A must list intake channel and curator provenance for every captured file.

See also Source Evaluation and Evidence Policy § Curator-added provenance.

Source evaluation and evidence policy

Full policy: Source Evaluation and Evidence Policy.

Captured source vs source used as evidence

These are different concepts and must not be collapsed:

Captured source — saved in the packet (uploads/, source_snapshot/, source_snapshot/captured/) with provenance (timestamp, SHA-256, HTTP metadata, capture_status).
Source used as evidence — a capture (or upload) that supports a specific claim in evidence_checklist.md and a proposed corpus edit.

Every capture is preserved for audit whether or not it is used as evidence. Refactors must report both sets explicitly (see Refactor output expectations).

Capture status

Status	When
`success`	Intended content retrieved and stored (e.g. HTTP 200 faculty profile).
`partial`	Body stored but incomplete or blocked (e.g. HTTP 403 publisher page — audit only, not full article).
`failed`	No usable snapshot (network error, blocked URL, oversize response).

HTTP 403 is always partial, never success. Do not treat a 403 snapshot as confirmation of article metadata unless corroborated by a high-confidence source in the same packet.

High-confidence sources

University faculty profiles, lab/department/center sites, grants, journal articles, DOI pages, Google Scholar, ORCID, Web of Science, Scopus, ResearchGate (when clearly the scholar), professional society profiles, official project sites, and faculty-maintained research websites may support people, publications, collaborations, memberships, service, expertise, and research themes.

Scholarly indexes (Google Scholar, ORCID, Web of Science, Scopus, ResearchGate) are strong for publications, coauthor networks, and expertise; they are not sufficient alone for facility ownership, equipment ownership, access policy, or operational control — those need institutional or project evidence.

Conservative claims

Require stronger institutional or project sources for: facility/equipment ownership, facility access, organizational authority, administrative responsibility, laboratory affiliation beyond official statements, and operational control.

Lower-confidence sources

News, press releases, blogs, and marketing materials — corroborate before use. Social media, forums, and anonymous sites — not primary evidence.

Publications and service (resource pages)

Register publications by verified DOI with title, year, venue, abstract, and authorship in publications.yaml — do not store full-text articles in packets.
On resource pages, omit service/editorial/society blocks unless the role involves facilities or hardware.

Repository-wide refactor (required)

Intake channel and source type describe how material arrived, not which corpus layers may change.

Every refactor must:

Assess full impact on people, facilities, equipment, laboratories, centers, institutes, programs, courses, grants, publications, outputs, external partners, expertise, capabilities, relationships, hyperlinks, reciprocity, and evidence quality.
Classify every discovery as: evidence-backed update · candidate knowledge · rejected claim (see Repository-wide refactor governance).
Preserve candidates — facility, equipment, lab, center, capability, and relationship signals must not be silently discarded because the packet came through a grant, faculty, publication, or admin workflow.
Run promotion review (mandatory) — for every preserved candidate, perform targeted evidence discovery, re-evaluate, and promote what becomes evidenced (see Promotion review below).
Record sections L (impact assessment), M (candidate registry), and N (promotion review summary) in refactor_output.md.

Layer-specific rules (grants, publications, outputs) apply when that layer is implicated; they do not limit assessment of other layers.

Promotion review (required)

Candidate preservation is the beginning of processing, not the end. Per the Promotion Discovery Rule, whenever a source produces new candidates (facilities, labs, centers, institutes, clinics, equipment, capabilities, external facilities, programs, courses, partners, people, or relationships) the refactor must:

Preserve each candidate with full provenance.
Perform targeted evidence discovery from high-confidence sources permitted by Source Evaluation and Evidence Policy (institutional sites, faculty/lab/center/department/project/grant pages, ORCID, Google Scholar, ResearchGate, Web of Science, Scopus, DOI-linked publications, official partner sites).
Capture discovered evidence as sources with provenance (packet capture — never an unsourced claim).
Re-evaluate every candidate.
Promote candidates that meet promotion criteria (institutional/project evidence — the bar is unchanged).
Create/update all required entities and evidence-backed typed relationships.
Add reciprocal hyperlinks and evaluate relationship visibility.
Preserve unresolved candidates with reason_not_promoted.
Record section N (sources reviewed/imported, candidates promoted/deferred, relationships + reciprocal links added, remaining gaps).

A source is fully processed only when promotion review is completed, or explicitly deferred with justification recorded in section N.

Automatic candidate preservation (Phase 2)

Candidate preservation is automated by one shared mechanism so no signal is silently discarded at intake:

Shared library: scripts/candidate_extraction.py (registry-backed matching, confidence rubric, idempotent dedup).
Generic CLI: python scripts/extract_candidates.py sources/inbox/<packet> — used by RFUMS / DPU-RFU packets, URL captures, faculty profiles, and future packet types.
Structured intake: the ORS and AGIF intake scripts call the same library automatically on every run.

Intake writes packet-level discoveries to <packet>/extracted/candidates.yaml. This automatic capture guarantees no signal is silently discarded — but capture is only the first step. Per the Promotion Discovery Rule, every packet refactor must then run promotion review: targeted evidence discovery, re-evaluation, and promotion of candidates that become evidenced. Promotion into the repository-level registries (data/review/resource_candidates.yaml, data/review/relationship_candidates.yaml) and into the corpus happens as part of each refactor's promotion review, under human review (e.g. extract_candidates.py --promote for the registries; manual page/link creation for promotions). Promotion review is mandatory, not deferred — see Promotion review above and the Candidate preservation implementation report.

How Cursor refactoring should use them

The AI engine is Cursor-style refactoring of the corpus, not an extraction API. The flow:

A maintainer opens the packet and reads curator_notes.md / submitter_notes.md and evidence_checklist.md.
They copy refactor_prompt.md into Cursor.
Cursor proposes edits to the existing corpus files following repository conventions:
resources/<PREFIX>/<ID>.md prose sections (Description, Documented Relationships, Notes)
data/resource_access.yaml, data/resource_verification.yaml, data/resource_equipment.yaml
data/people.yaml (with evidence_file + evidence_text per association)
data/grants.yaml, data/person_grant_links.yaml, data/grant_resource_links.yaml (when evidenced)
documented-link YAML (data/course_resource_links.yaml, future resource_resource_links.yaml)
Every proposed claim must cite a source already present in the packet. No claim without evidence.
Cursor does not run sync_docs.sh or graph export as part of the proposal — those are deterministic build steps run after human approval.

Hard constraints for the refactor step (encoded in every refactor_prompt.md):

Edit existing source-of-truth files; do not invent new schemas ad hoc.
Preserve existing IDs; never reuse or renumber (see resource-map-index.md governance).
Keep access vs verification evidence in their respective YAML files.
Do not fabricate URLs, emails, dates, or confidence scores.
Conservative status: when evidence is incomplete, use needs_review, not inflated confidence.
Leave docs/ untouched — it is a generated mirror.
Behave as a repository editor, not a chatbot — the objective is evidence-backed refactoring, not extraction.

Hypertext and reciprocity

The Resource Map is a hypertext knowledge system. Refactors should strengthen the network of evidence-backed relationships, not simply append isolated facts.

For every proposed relationship involving people, facilities, laboratories, equipment, grants, publications, outputs, departments, colleges, external partners, or organizations:

Create human-readable hyperlinks wherever appropriate.
Prefer reciprocal links — if page A links to B, evaluate whether B should link back to A.
Report all new links and reciprocal links created.
Report links deliberately not created, with reasons.
Associate every relationship with explicit source(s) and preserve provenance.
Do not create relationships based solely on topic similarity.
Do not infer facilities, equipment, ownership, collaboration, or organizational ties without evidence.
If a source mentions an entity not yet in the repository: do not discard it silently and do not invent a page — list it as a candidate addition with supporting evidence.
Prefer improving existing content over creating new pages.
Preserve narrative readability, existing evidence, and repository-wide consistency.

Refactor output expectations

Every refactor should produce a structured report (sections A–J in refactor_prompt.md, plus expanded source-evaluation detail in section A) before or alongside the Git diff:

Section	Content
A	Source evaluation summary — (1) all captured sources with `intake_channel` (`intake_form` vs `curator_added`), `capture_status`, and for curator-added sources `added_by` / `added_at` / `added_rationale`; (2) sources used as evidence mapped to claims; (3) sources reviewed but not used; (4) reason not used for each unused capture. Also summarize what was read.
B	Proposed repository changes
C	New relationships identified
D	New hyperlinks created
E	Reciprocal hyperlinks created
F	Candidate entities not yet represented
G	Relationships considered but rejected
H	Evidence supporting every accepted relationship
I	Provenance concerns
J	Conflicts with existing repository content
L	Full Resource Map impact assessment — per-layer reviewed / updates / candidates / rejections
M	Candidate knowledge registry — preserved signals insufficient for promotion (provenance + snippet)
N	Promotion review summary — sources reviewed/imported, candidates promoted/deferred, relationships + reciprocal links added, remaining gaps (or explicit deferral with justification)

See Source Evaluation and Evidence Policy for capture-status rules and scholarly-source guidance.

When the source documents grants or awards: follow Grant ingestion governance for grant-layer writes. Report grant existence, PI identity resolution, and resource link status separately in sections B, F, and M — do not omit evidence-backed awards because the PI is not in people.yaml. The same source may also yield facility, people, or publication candidates; assess all layers.

The human reviewer uses this report together with evidence_checklist.md when inspecting the diff.

Knowledge integrity

Refactors should strengthen discoverability, provenance, hyperlink structure, reciprocal relationships, evidence visibility, and consistency — while avoiding hallucinated entities, unsupported relationships, and speculative facility, collaboration, or equipment assignments.

Audit-driven refactors

When acting on repository audits (hypertext-reciprocity-audit.md and future reports under docs/reports/):

Follow Audit-driven refactor policy.
Prefer high-value, low-risk surfacing and hyperlink improvements over new claims.
Audit findings alone are not evidence — every change must trace to sources already in the repo or to a source packet.
Distinguish knowledge surfacing, hyperlink/reciprocal improvements, provenance improvements, and new discovery (packet-only).
Add an audit implementation summary to refactor_output.md (relationships surfaced, links added, evidence used, recommendations deferred).
Prefer incremental diffs; preserve narrative readability; stop before commit unless explicitly instructed.

How evidence is tracked

Evidence tracking is layered so provenance survives from source to public page to (eventual) graph export:

source document in packet
   └─ evidence_checklist.md   (claim ↔ source ↔ status, in the packet)
        └─ data/*.yaml         (evidence_text, source_urls, evidence_file)
             └─ resources/*.md  (rendered Access / Verification / Relationships)
                  └─ docs/resources/*.md  (published mirror via sync_docs.sh)
                       └─ data/graph_exports/*.csv  (derived view, later)

The packet is the upstream root of this chain. After merge, the packet remains in sources/ as the durable provenance record for that update.

How human review happens through Git diffs

Review is a Git diff review, not a web app:

The refactor produces changes to tracked files plus the new packet folder.
The reviewer inspects the diff: every YAML/markdown change should be traceable to a packet evidence item.
The reviewer checks the refactor output report (sections A–J): new links, reciprocity, rejected relationships, and candidate entities.
The reviewer can request changes, edit directly, or reject.
Approval = a clean, evidence-backed diff on development.

This keeps the existing strengths the audit identified (conservative graph rules, include_in_public_site / include_in_graph flags, documented-link patterns) while making provenance explicit and reviewable.

How MkDocs is rebuilt

After approval, the deterministic build runs (unchanged from today):

./sync_docs.sh            # apply_* scripts inject sections; copy resources/ → docs/
.venv/bin/mkdocs build --strict

sync_docs.sh regenerates access/verification/equipment/people sections and the indexes, then mirrors resources/ into docs/resources/. Only after a clean --strict build and review does work move toward main.

Publication boundary

mkdocs.yml sets docs_dir: docs. The sources/ tree is outside docs/, so:

Source packets are version-controlled but never published.
They add provenance to the repo without bloating the public site.
No mkdocs.yml change is required to adopt this workflow.

How downstream graph/search views may later be derived

Graphs and search are derived views, built after the corpus is correct — never the source of truth (per the charter). Once structured relationship records exist (see the audit's §12–13 recommendation for resource_resource_links.yaml), scripts/export_graph_tables.py can extend its CSV export. Because every edge traces back to packet evidence, derived views inherit provenance automatically. Neo4j, GraphRAG, and search remain optional consumers of the corpus, not inputs to it.

Lifecycle summary

sources/inbox/_template/         ← copy this to start
        │  cp -R
        ▼
sources/inbox/2026-06-13-grice-lab/   ← fill intake.yaml, notes, checklist, sources
        │  open refactor_prompt.md in Cursor
        ▼
proposed diffs to resources/ + data/*.yaml   ← AI synthesis, evidence-backed
        │  promotion review (REQUIRED): targeted evidence discovery →
        │  re-evaluate candidates → promote what is evidenced (section N)
        ▼
candidates resolved (promoted or deferred-with-justification)
        │  git diff review
        ▼
human approval on development
        │  ./sync_docs.sh && mkdocs build --strict
        ▼
tested build on development
        │  (only after testing)
        ▼
merge to main  → Raspberry Pi hourly pull → public demo

The packet stays in sources/ afterward as the permanent provenance record.

A packet is "processed" only after promotion review. Reaching "candidates preserved" is not done; the refactor must complete the Promotion Discovery Rule (targeted evidence discovery → re-evaluate → promote/defer) and record section N, or explicitly defer promotion review with justification.

What this workflow is not

Not a scraper (no regex extraction, no HTML parsing pipeline).
Not an LLM API integration (Cursor is the engine for now; no API keys, no autonomous calls).
Not a web app (review is via Git diffs; the archived v2 Flask app is not revived).
Not a new source of truth (Neo4j/search stay derived).
Not a MkDocs replacement.

It is deliberately small: structured folders, human judgment, evidence checklists, and a copy-paste Cursor prompt.