living_review package

Submodules

living_review.classifier module

classifier.py

This module provides semantic filtering and classification utilities for scientific papers in the context of machine-learning applications to particle accelerator physics.

It uses a pre-trained sentence-transformers model (MiniLM-L6-v2) to compute embeddings for papers, accelerator/ML reference queries, and category descriptions. Papers are first filtered for relevance (accelerators ∧ ML, excluding domain noise), and then assigned to categories using semantic similarity and keyword heuristics.

Key Features

• Device selection (CPU/GPU/MPS) for embedding computation

• On-demand lazy loading of the semantic model

• Semantic relevance filtering using accelerator/ML/noise queries

• Category classification with thresholds, keyword overrides, and deduplication

Typical Usage

>>> from living_review.data_model import Paper
>>> from living_review.classifier import classify_papers
>>> papers = load_accepted_papers()   # relevance is decided by relevance.py
>>> classify_papers(papers)

living_review.classifier.classify_papers(papers, threshold=0.25, max_cats=2)

Assign semantic categories to each paper.

Uses a combination of: - semantic similarity with predefined category descriptions, - special handling for review papers, - keyword overrides (e.g. “surrogate model” → Surrogate Models).

Should be applied only to papers accepted by the relevance funnel.

Parameters:

• papers (list of Paper) – Papers to classify in-place (field categories updated).

• threshold (float, optional) – Minimum similarity required to assign a category (default=0.25).

• max_cats (int, optional) – Maximum number of categories to keep per paper (default=2).

Returns:

Papers are modified in place. Each .categories becomes a list of dicts with fields: {“label”: str, “score”: float}.

Return type:

None

Notes

• If no category passes the thresholds, a default Others category with score 0.0 is assigned.

• Deduplication ensures the highest score per label is kept.

living_review.classifier.device_str()

Select the most appropriate device for embedding computation.

Returns:

“mps” if Apple Metal backend is available, “cuda” if NVIDIA GPU CUDA backend is available, otherwise “cpu”.

Return type:

str

living_review.classifier.dual_semantic_scores(texts)

Compute semantic relevance scores of input texts with respect to accelerator physics, machine learning, and noise queries.

Parameters:

texts (list of str) – List of textual inputs (title + abstract concatenated).

Returns:

(scores_accel, scores_ml, scores_noise), each a list of floats aligned with the input order.

Return type:

tuple of lists

living_review.classifier.load_sem_model()

Lazy-load the sentence transformer model used for semantic similarity.

Loads all-MiniLM-L6-v2 from HuggingFace Hub on the first call and caches it globally. Subsequent calls return the cached model.

Returns:

The loaded MiniLM model, bound to the appropriate device.

Return type:

SentenceTransformer

living_review.gates module

gates.py

Stage B of the relevance funnel: deterministic accept/reject rules.

Papers arriving from the fetchers are first checked against cheap, auditable rules before any model runs:

AUTO-ACCEPT

• primary arXiv category is physics.acc-ph; or

• the venue matches the accelerator-venue whitelist (PRAB, JACoW conferences, NIM-A, …) AND any ML keyword appears in title+abstract.

AUTO-REJECT (conjunctions only — never venue or domain alone; medical accelerator papers such as proton-therapy beam delivery are in scope)

• “accelerator/acceleration” appears only in compute-hardware context (DNN/FPGA/ASIC/…) AND the text has zero accelerator-system vocabulary hits; or

• zero accelerator-system vocabulary AND a clear foreign-domain signal (education, civil engineering, medical imaging, finance, …).

Everything else is GRAY and goes to the Stage C adjudicator. Papers with empty abstracts that do not auto-accept are gray with rule empty_abstract — the funnel routes them straight to the pending queue because title-only adjudication is not trusted (see SCOPE.md).

All patterns are word-boundary regexes (see config.py) — the previous substring matching rejected any text containing “jet”, “chip”, or “soc”.

class living_review.gates.GateResult(decision, rule)

Bases: object

decision: str

rule: str

living_review.gates.apply_gates(paper)

Apply Stage B deterministic rules to one paper.

Returns:

decision in {“accept”, “reject”, “gray”} plus the rule that fired.

Return type:

GateResult

living_review.gates.has_machine_vocab(text)

True if the text carries explicit machine-subsystem vocabulary (“particle accelerator”, “emittance”, “RF cavity”, …).

Used by the funnel’s false-negative guard: the 2026-07 model benchmark found the NLI under-scores genuine accelerator papers with ML-venue phrasing (scores 0.03-0.10 on e.g. latent-space accelerator tuning). A paper that explicitly names accelerator machinery is never auto-rejected on an NLI score alone — it goes to the pending queue.

Return type:

bool

living_review.gates.has_ml_vocab(text)

True if the text carries any ML/AI vocabulary (ML_CONTENT_TERMS). A paper with none cannot belong in the review; the NLI reject-guard only protects papers that have some.

Return type:

bool

living_review.gates.venue_is_whitelisted(venue)

True if the venue matches the accelerator-venue whitelist.

Return type:

bool

living_review.adjudicator module

adjudicator.py

Stage C of the relevance funnel: model-based adjudication of the gray zone.

Papers that neither auto-accept nor auto-reject in Stage B (gates.py) are scored by a zero-shot NLI cross-encoder against the scope hypothesis from SCOPE.md. Three-way outcome:

• score >= NLI_THRESHOLDS[“accept”] -> accepted

• score <= NLI_THRESHOLDS[“reject”] -> rejected

• otherwise -> pending (human review queue)

The adjudicator is a pluggable protocol so an instruct-LLM backend can be added later (see TODO.md) without touching the funnel. Any failure inside an adjudicator marks the batch pending rather than aborting the pipeline run — papers are simply retried the next night.

class living_review.adjudicator.AdjudicationResult(decision, score, model, revision, rule=None)

Bases: object

decision: str

model: str

revision: str | None

rule: str | None = None

score: float | None

class living_review.adjudicator.Adjudicator(*args, **kwargs)

Bases: Protocol

Anything that can score papers against the SCOPE.md criterion.

adjudicate(papers)

Return type:

List[AdjudicationResult]

class living_review.adjudicator.LLMAdjudicator

Bases: object

Instruct-LLM adjudicator applying the full SCOPE.md rubric.

Deferred — see TODO.md. Kept here so the funnel interface is already shaped for it.

adjudicate(papers)

Return type:

List[AdjudicationResult]

class living_review.adjudicator.NLIAdjudicator(model='MoritzLaurer/deberta-v3-base-zeroshot-v2.0', revision=None, thresholds=None, hypothesis='This paper applies machine learning or artificial intelligence to a particle accelerator, beamline, or particle beam.')

Bases: object

Zero-shot NLI cross-encoder adjudicator.

Uses a HuggingFace zero-shot-classification pipeline with the pinned model/revision from config. The model is loaded lazily on first use and cached for the process lifetime.

adjudicate(papers)

Return type:

List[AdjudicationResult]

score(papers)

Entailment scores in [0, 1] for each paper (title + abstract).

Return type:

List[float]

living_review.relevance module

relevance.py

Funnel orchestration: which papers enter the published review.

Nightly flow (see SCOPE.md for the editorial criterion):

1. Select undecided papers — no terminal accepted/rejected decision. Pending papers ARE retried (abstract backfill may have progressed, thresholds may have moved); accepted/rejected are never touched.

2. Backfill missing abstracts (enrich.py).

3. Stage B deterministic gates (gates.py): auto-accept / auto-reject / gray. Empty-abstract non-accepts go straight to pending.

4. Stage C adjudicator (adjudicator.py) on the gray zone.

Every decision is recorded in Paper.review with full provenance. Accepted/rejected decisions are terminal: nights after a decision, the paper is skipped entirely — this is what makes the pipeline incremental and keeps a model/threshold change from silently rewriting the archive.

living_review.relevance.accepted_papers(db)

Return type:

List[Paper]

living_review.relevance.demote_others_only(papers)

Split classified papers into publishable ones and Others-only strays.

A paper whose only category is Others at score 0.0 fit nothing in the taxonomy — treated as a triage signal, not a published category: it is demoted to pending (human queue) and withheld from the export.

Returns:

The publishable subset (papers not demoted).

Return type:

list of Paper

living_review.relevance.export_pending_queue(db, path)

Write the ranked pending queue to a JSON file for human review.

Return type:

int

living_review.relevance.pending_papers(db)

Return type:

List[Paper]

living_review.relevance.rank_pending(papers)

Sort the pending queue most-relevant-first for human review, using the MiniLM similarity to the accelerator+ML reference queries (its only remaining scoring role). Falls back to unranked order on any failure.

Return type:

List[Paper]

living_review.relevance.run_funnel(db, adjudicator)

Run backfill → gates → adjudicator over all undecided papers in the DB.

Returns:

Outcome counts: undecided, enriched, gate_accepted, gate_rejected, pending_empty_abstract, adjudicated, nli_accepted, nli_rejected, nli_pending.

Return type:

dict

living_review.relevance.set_review(paper, decision, stage, rule=None, score=None, model=None, revision=None)

Record a relevance decision with provenance on a paper.

Return type:

None

living_review.relevance.undecided_papers(db)

Papers without a terminal decision (pending ones are retried).

Return type:

List[Paper]

living_review.dedup module

dedup.py

Two-pass deduplication of Paper records.

Pass 1 — identifier graph: papers sharing any normalized identifier (doi:, arxiv:, inspire:, see utils.canonical_ids) are the same work; groups are found with a union-find over those identifiers. This also links DataCite arXiv DOIs (10.48550/arXiv.XXXX) to their arXiv record.

Pass 2 — fuzzy titles: remaining records with no shared identifier are compared within year buckets (year ± 1, to tolerate preprint → journal transitions) using utils.similar_title; pairs at or above config.FUZZY_TITLE_THRESHOLD merge.

Merging delegates to Paper.merge_with, so curated fields and terminal review decisions survive; the primary record of a group is chosen to be the one carrying a decision (then curated, then richest metadata).

living_review.dedup.dedup_papers(papers, fuzzy_threshold=0.93, tie_breaker=None)

Deduplicate a list of papers (identifier pass, then fuzzy-title pass).

Parameters:

• papers (list of Paper)

• fuzzy_threshold (float) – similar_title score at or above which two id-disjoint records merge.

• tie_breaker (callable, optional) – Extra similarity function (e.g. embedding cosine) consulted for pairs scoring in [fuzzy_threshold - 0.08, fuzzy_threshold); if it returns >= 0.9 the pair merges anyway.

Returns:

Deduplicated records, first-seen order preserved.

Return type:

list of Paper

living_review.dedup.merge_group(group)

Merge a group of records for the same work into its primary record.

Return type:

Paper

living_review.enrich module

enrich.py

Metadata backfill for papers with missing abstracts.

A third of the historical DB entries carry empty abstracts (one root cause: OpenAlex returns abstract_inverted_index, not abstract). Since both the relevance funnel and category classification score title + abstract, papers without abstracts are systematically mis-scored. This module fills the gap before any scoring runs:

1. Crossref by DOI (/works/{doi}, JATS markup stripped),

2. OpenAlex by DOI (/works/doi:{doi}, inverted index reconstructed),

3. arXiv by id (batched id_list queries, <= 100 ids per call).

Existing non-empty abstracts are never overwritten. All network failures are warnings — enrichment must never break a pipeline run.

living_review.enrich.backfill_abstracts(papers, session=None, arxiv_lookup=<function _arxiv_abstracts>)

Fill empty abstracts in place from Crossref, OpenAlex, and arXiv.

Parameters:

• papers (iterable of Paper) – Papers to enrich; only those with empty abstracts are touched.

• session (requests.Session, optional) – HTTP session (injectable for tests); defaults to the shared one.

• arxiv_lookup (callable, optional) – list[str] -> dict[arxiv_id, abstract] (injectable for tests).

Returns:

Number of abstracts filled.

Return type:

int

living_review.enrich.fetch_arxiv_metadata(ids)

Fetch abstract + subject categories for arXiv ids (batched, best-effort).

Returns:

arxiv_id -> {“abstract”: str, “arxiv_categories”: [primary, …]}.

Return type:

dict

living_review.enrich.reconstruct_openalex_abstract(inv)

Rebuild plain text from an OpenAlex abstract_inverted_index.

Return type:

str

living_review.enrich.strip_jats(text)

Remove JATS/XML tags and collapse whitespace in a Crossref abstract.

Return type:

str

living_review.migrate module

migrate.py

One-off migration of the legacy published DB into the canonical data/db.json, cleaning it through the relevance funnel.

Procedure: 1. Load the legacy site/data/livingreview.json (papers only). 2. Deduplicate (identifier graph + fuzzy titles). 3. Backfill arXiv subject categories + missing abstracts (network). 4. Apply Stage B gates; emit the easy-slice eval sets

(data/eval/positives.json = gate-auto-accepts, data/eval/negatives.json = gate-auto-rejects).

5. Unless gates_only, adjudicate the gray zone with the NLI model.

6. Write: data/db.json (every paper + decision provenance), the regenerated accepted-only published JSON + BibTeX + PDF, and data/migration_dropped.md — a human-readable table of every dropped paper, to be eyeballed BEFORE the branch is merged. Rescuing a paper = set review.decision: accepted, review.stage: human, curated: true in data/db.json (and remove it from the negatives eval file).

Decisions carry stage: “migration:gate” / “migration:nli” so this run stays distinguishable from nightly ones.

living_review.migrate.migrate(source='site/data/livingreview.json', db_path='data/db.json', report_path='data/migration_dropped.md', eval_dir='data/eval', dry_run=False, gates_only=False, adjudicator=None, output_dir='.')

Run the one-off migration. See module docstring.

living_review.cli module

cli.py

Command-line interface (CLI) for the Living Review pipeline.

Subcommands

• run : nightly pipeline (fetch → dedup → funnel → classify → export)

• review : print the pending human-review queue

• migrateone-off migration of the legacy published DB into the

  canonical data/db.json (see migrate.py)

Backward compatibility: invoking without a subcommand behaves like run (the CI workflow predates subcommands).

Typical Usage

Run a full scan of the last 30 days from all sources:

$ python -m living_review.cli run –days 30 –sources all

Disable PDF export but keep BibTeX:

$ python -m living_review.cli run –no-pdf

Show the pending queue:

$ python -m living_review.cli review

living_review.cli.cmd_history(args)

living_review.cli.cmd_migrate(args)

living_review.cli.cmd_review(args)

living_review.cli.cmd_run(args)

living_review.cli.main(argv=None)

Entry point for the Living Review CLI.

living_review.config module

config.py

Central configuration module for the Living Review project.

This file collects all constants, keywords, category descriptions, semantic queries, and thresholds used across the pipeline. Keeping them centralized ensures consistency between different modules (fetchers, classifier, pipeline, etc.).

Contents

• Accelerator / ML keywords

• Negative keywords (to filter out noise domains)

• Reference semantic queries (used for similarity scoring)

• Category descriptions (used for classification)

• Default thresholds and constants (date window, API page sizes)

Typical Usage

>>> from living_review import config
>>> config.ACCEL_KEYWORDS[:5]
['accelerator', 'linac', 'synchrotron', 'collider', 'storage ring']

living_review.config.ACCEL_SYSTEM_VOCAB = ['particle accelerators?', '(proton|electron|ion|linear) accelerators?', 'accelerator facilit(y|ies)', '\\blinacs?\\b', '\\bcyclotrons?\\b', '\\bsynchrotrons?\\b', 'storage rings?', '\\bcolliders?\\b', 'beam ?lines?', '\\bbeam\\b', '\\bbeams\\b', 'rf cavit(y|ies)', '\\bcavit(y|ies)\\b', '\\bcryomodules?\\b', '\\bklystrons?\\b', '\\bundulators?\\b', '\\bwigglers?\\b', '\\bemittance\\b', '\\bwakefields?\\b', '\\bbetatron\\b', '\\bquadrupoles?\\b', '\\bsextupoles?\\b', '\\bdipole magnets?\\b', '\\bmagnets?\\b', '\\bseptum\\b', '\\bkickers?\\b', '\\bcollimators?\\b', '\\binjectors?\\b', '\\bgantry\\b', '\\bgantries\\b', '\\bdosimetry\\b', '\\bdosimetric\\b', 'proton therapy', 'ion therapy', '\\bradiotherapy\\b', '\\bBPMs?\\b', 'beam position monitors?', 'free[- ]electron lasers?', '\\bFEL\\b', 'light sources?', 'synchrotron radiation', '\\bSRF\\b', '\\bLHC\\b', '\\bCERN\\b', '\\bFermilab\\b', '\\bDESY\\b', '\\bXFEL\\b', '\\bSLAC\\b', '\\bLCLS\\b', '\\bGANIL\\b', '\\bFRIB\\b', '\\bCEBAF\\b', '\\bJ-PARC\\b', '\\bBNL\\b', '\\bluminosity\\b', 'beam dynamics', 'beam loss', 'beam halo', '\\bbunch(es)?\\b', 'charged particles?']

Word-boundary patterns whose presence indicates the paper talks about an accelerator/beam system at all. Zero hits is a necessary condition for auto-rejection (never sufficient alone).

living_review.config.ARXIV_PAGE_SIZE = 100

Maximum number of results per page in arXiv API queries.

Type:

int

living_review.config.DATE_WINDOW_DAYS = 7

Default sliding window (in days) for fetching new papers.

Type:

int

living_review.config.DETECTOR_ANALYSIS_TERMS = ['track (reconstruction|finding|fitting)', 'particle (identification|tracking)', 'jet tagging', '\\bjet(s)? (classification|reconstruction)\\b', 'event (reconstruction|selection|classification)', 'trigger (system|rate|menu|decision|algorithms?)', '\\bHLT\\b', '\\bcalorimeters?\\b', 'detector (data|design|response|simulation|performance)', 'tracking (system|detector)', 'vertex reconstruction', 'particle-?flow', 'neutrino (selection|identification|oscillation)', '\\bhits?\\b.*\\btracks?\\b', 'physics analysis']

HEP detector/analysis context. ML on detector products at a collider is out of scope (SCOPE.md) but scores 0.92-0.99 with the NLI — the 2026-07 model benchmark found this to be the funnel’s dominant false-positive class. Detector-context papers without machine-subsystem vocabulary route to the pending queue instead of the adjudicator.

living_review.config.FOREIGN_DOMAIN_TERMS = ['\\bearthquakes?\\b', '\\btsunamis?\\b', '\\bclassrooms?\\b', '\\bcurricul(um|a)\\b', '\\bstudents?\\b', '\\bpedagog\\w+\\b', '\\be-?learning\\b', 'learning outcomes?', '\\bteaching\\b', '\\bcustomer churn\\b', '\\bmarketing\\b', '\\be-?commerce\\b', '\\bblockchain\\b', '\\bcryptocurrenc\\w+\\b', '\\bgenomes?\\b', '\\bgenomic\\w*\\b', '\\bprotein\\w*\\b', '\\bcrops?\\b', '\\bagricultur\\w+\\b', '\\blivestock\\b', '\\bconcrete\\b', '\\bmasonry\\b', '\\bpavements?\\b', '\\bgeotechnical\\b', '\\brailways?\\b', '\\btraffic\\b', '\\bvehicles?\\b', '\\bdrones?\\b', '\\bUAVs?\\b', '\\bwireless networks?\\b', '\\bbeamforming\\b', '\\bantennas?\\b', '\\bradar\\b', '\\b5G\\b', '\\b6G\\b', '\\btumou?rs?\\b', '\\bcancer\\b', '\\bpatients?\\b', '\\bclinical\\b', '\\bradiograph\\w+\\b', '\\bMRI\\b', '\\bultrasound\\b', '\\bdental\\b', '\\bstock market\\b', '\\bfinancial\\b', '\\bsentiment analysis\\b', '\\btokamaks?\\b', '\\bstellarators?\\b', '\\baltadefinizione\\b', '\\bcb01\\b', 's?tr?eaming[- ]ita\\b', '\\bfilm completo\\b', '\\bguardare? film\\b', '\\bblasting\\b', '\\bmining\\b', '\\bexcavation\\b', '\\bquarry\\b']

Clear foreign-domain signals. Auto-reject requires one of these AND zero ACCEL_SYSTEM_VOCAB hits — a proton-therapy paper mentioning ‘patients’ and ‘gantry’ is protected by its accelerator vocabulary.

living_review.config.FUZZY_TITLE_THRESHOLD = 0.93

similar_title score at or above which two id-disjoint records are considered the same work (see dedup.py).

Type:

float

living_review.config.HARDWARE_CONTEXT_TERMS = ['\\bDNNs?\\b', '\\bCNNs?\\b', '\\binference engines?\\b', '\\bFPGAs?\\b', '\\bASICs?\\b', '\\bVLSI\\b', '\\bTPUs?\\b', '\\bGPUs?\\b', '\\bsystolic arrays?\\b', '\\bquantization\\b', '\\bRISC-V\\b', 'edge (computing|devices?|AI)', '\\bin-memory computing\\b', '\\bDRAM\\b', '\\bSRAM\\b', 'compute-in-memory', '\\bCIM\\b', 'energy[- ]efficien(t|cy)', '\\bthroughput\\b', '\\blow[- ]power\\b', 'hardware[- ](accelerat\\w+|architectures?|design)', 'neural network accelerat\\w+', '\\bchip\\b', '\\bSoCs?\\b', '\\bmicrocontrollers?\\b', '\\bembedded systems?\\b']

Compute-hardware context. ‘Accelerator’ collocated only with these and zero ACCEL_SYSTEM_VOCAB hits means a DNN-hardware paper (auto-reject).

living_review.config.MACHINE_SUBSYSTEM_VOCAB = ['\\blinacs?\\b', '\\bcyclotrons?\\b', '\\bsynchrotrons?\\b', 'storage rings?', 'beam ?lines?', 'rf cavit(y|ies)', '\\bcavit(y|ies)\\b', '\\bcryomodules?\\b', '\\bklystrons?\\b', '\\bundulators?\\b', '\\bwigglers?\\b', '\\bemittance\\b', '\\bwakefields?\\b', '\\bbetatron\\b', '\\bquadrupoles?\\b', '\\bsextupoles?\\b', '\\bdipole magnets?\\b', '\\bseptum\\b', '\\bkickers?\\b', '\\bcollimators?\\b', '\\binjectors?\\b', '\\bgantry\\b', '\\bgantries\\b', '\\bdosimetry\\b', 'proton therapy', '\\bBPMs?\\b', 'beam position monitors?', 'beam dynamics', 'beam loss', 'beam halo', 'beam diagnostics', 'beam tuning', 'beam control', 'beam optics', 'machine protection', 'beam intensit(y|ies)', 'superconducting magnets?', '\\bquench(es)?\\b', '\\bSRF\\b', 'orbit correction', '\\bmagnet (design|control|tuning)\\b', 'particle accelerators?', '(proton|electron|ion|linear) accelerators?', 'accelerator (tuning|control|operation|physics)']

ACCEL_SYSTEM_VOCAB minus facility names and bare ‘beam’. Used by the detector-context gate — a paper about ML on detector data at a facility mentions the facility but not the machine.

Type:

Machine-subsystem vocabulary

living_review.config.ML_CONTENT_TERMS = ['machine learning', 'deep learning', 'neural network', 'reinforcement learning', 'bayesian optimization', 'anomaly detection', 'autoencoder', 'GAN', 'diffusion model', 'graph neural network', 'surrogate model', 'surrogate', 'physics-informed', 'PINN', 'transformer', 'foundation model', 'agentic AI', 'autonomous agent', 'LLM', 'policy gradient', 'policy learning', 'policy optimization', 'RL', 'artificial intelligence', 'AI', 'data-driven', 'gaussian process', 'gaussian processes', 'random forest', 'gradient boosting', 'boosted trees', 'generative model', 'generative models', 'large language model', 'convolutional', 'LSTM', 'classifier', 'deep generative', 'GPT']

Gate-side ML vocabulary (word-boundary matched). A paper with NONE of these has no ML content and cannot belong in the review regardless of its arXiv category (auto-accept requires one; the NLI reject-guard only protects papers that have one). Distinct from the arXiv query keywords.

living_review.config.NLI_HYPOTHESIS = 'This paper applies machine learning or artificial intelligence to a particle accelerator, beamline, or particle beam.'

Scope hypothesis, derived from SCOPE.md.

Type:

str

living_review.config.NLI_MODEL = 'MoritzLaurer/deberta-v3-base-zeroshot-v2.0'

Zero-shot NLI cross-encoder used by the Stage C adjudicator.

Type:

str

living_review.config.NLI_MODEL_REVISION = None

Pinned HF revision hash for reproducibility (set after first calibration; None = latest).

Type:

str or None

living_review.config.NLI_THRESHOLDS = {'accept': 0.9, 'reject': 0.15}

Entailment-score cutoffs. score >= accept -> accepted; score <= reject -> rejected; in between -> pending (human queue). Calibrated 2026-07 on the gate-derived easy slices (142 positives / 96 negatives): accept=0.90 admits 2/96 junk (both genuinely borderline accelerator-shielding papers), reject=0.15 loses 8/142 easy positives — all of which auto-accept at the gates in production and never reach the NLI. Re-run scripts/calibrate_thresholds.py after any model change.

Type:

dict

living_review.config.VENUE_WHITELIST_PATTERNS = ['phys\\w*\\.?\\s*rev\\w*\\.?\\s*accel\\w*\\.?\\s*(and|&)?\\s*beams', 'special\\s*topics\\W*accelerators\\s*(and|&)\\s*beams', '\\bPRAB\\b', '\\bPR-?STAB\\b', '\\bIPAC\\b', '\\bLINAC\\s*\\d*\\b', '\\bICALEPCS\\b', '\\bNAPAC\\b', '\\bHB\\s?20\\d\\d\\b', '\\bIBIC\\b', '\\bFEL\\s?20\\d\\d\\b', '\\bJACoW\\b', '\\bEPAC\\b', "\\bPAC\\s?[' ]?\\d\\d\\b", '\\bCOOL\\b', '\\bDIPAC\\b', 'nucl(ear|\\.?)\\s*instrum(ents|\\.?)\\s*(and|&)?\\s*methods.*\\bA\\b', '\\bNIM[- ]?A\\b', 'journal of instrumentation', '\\bJINST\\b']

Venues that publish accelerator work; whitelist-venue AND any ML keyword auto-accepts (Stage B). Matching is case-insensitive regex on the venue string.

living_review.data_model module

data_model.py

Data model definitions for the Living Review project.

This module defines the Paper dataclass, the central representation of a scientific paper throughout the pipeline. It ensures consistent handling of metadata, provenance, and status progression, and provides helpers for deduplication and serialization.

Contents

• Paper: dataclass representing a paper with metadata, provenance, and audit trail.

• status_rank: helper to order publication statuses.

• _canonical_key: helper to generate fallback IDs for deduplication.

Canonical use

• Every paper is represented internally as a Paper object.

• Papers are serialized into the canonical JSON DB (site/data/livingreview.json) via Paper.to_dict().

• Papers can be reconstructed from the DB via Paper.from_dict(), guaranteeing a stable round-trip between memory and storage.

Typical Usage

>>> from living_review.data_model import Paper
>>> raw = {"title": "AI for Beam Dynamics", "authors": ["A. Researcher"],
...        "arxiv_id": "1234.5678", "source": "arxiv"}
>>> p = Paper.from_source(raw)
>>> p.key_for_dedup()
('1234.5678', '', 'ai for beam dynamics')
>>> d = p.to_dict()
>>> Paper.from_dict(d).id
'arxiv:1234.5678'

class living_review.data_model.Paper(id, doi=None, arxiv_id=None, inspire_id=None, title='', authors=<factory>, abstract=None, date=None, year=None, venue=None, status=None, categories=<factory>, keywords=<factory>, arxiv_categories=<factory>, curated=False, notes=None, review=<factory>, links=<factory>, sources=<factory>, history=<factory>, last_updated=None)

Bases: object

Representation of a scientific paper.

id

Canonical identifier (e.g. “doi:…”, “arxiv:…”, or “hash:…”).

Type:

str

doi

Digital Object Identifier if available.

Type:

str, optional

arxiv_id

arXiv identifier if available.

Type:

str, optional

inspire_id

INSPIRE identifier if available.

Type:

str, optional

title

Title of the paper.

Type:

str

authors

List of author names.

Type:

list of str

abstract

Abstract or summary of the paper.

Type:

str, optional

date

ISO date string (YYYY-MM-DD).

Type:

str, optional

year

Publication year.

Type:

int, optional

venue

Journal or conference venue.

Type:

str, optional

status

Publication status (pending, preprint, published…).

Type:

str, optional

categories

Classification categories assigned to the paper, as {"label": str, "score": float} entries.

Type:

list of dict

keywords

List of keywords associated with the paper.

Type:

list of str

arxiv_categories

arXiv subject categories, primary first (e.g. [“physics.acc-ph”, “cs.LG”]).

Type:

list of str

curated

Whether this entry has been manually curated (protected from overwrite).

Type:

bool

review

Relevance-decision provenance set by the funnel; accepted/rejected decisions are terminal and survive merges.

Type:

dict

notes

Free-text notes by curators.

Type:

str, optional

links

Dictionary of useful links (arXiv, DOI, PDF, publisher).

Type:

dict

sources

Provenance info (which fetcher, when).

Type:

list of dict

history

Change history (merges, status updates).

Type:

list of dict

last_updated

Timestamp of last update in ISO format.

Type:

str, optional

abstract: str | None = None

arxiv_categories: List[str]

arxiv_id: str | None = None

authors: List[str]

categories: List[Dict]

curated: bool = False

date: str | None = None

doi: str | None = None

static from_dict(d)

Reconstruct a Paper object from its dictionary representation.

Used when loading from the canonical DB (site/data/livingreview.json).

Parameters:

d (dict) – Dictionary with Paper fields, as produced by to_dict().

Returns:

A Paper instance with all fields populated.

Return type:

Paper

static from_source(raw)

Build a Paper from raw metadata (dict).

Normalizes identifiers, title, and authors, and ensures provenance. Will assign a canonical id of form: - doi:… (if DOI available), - arxiv:… (if arXiv available), - hash:… (fallback hash if no DOI/arXiv).

Parameters:

raw (dict) – Raw metadata from a fetcher.

Returns:

A new Paper object.

Return type:

Paper

history: List[Dict[str, str]]

id: str

inspire_id: str | None = None

key_for_dedup()

Generate a key for deduplication.

Returns:

(arxiv_id, doi, simplified_title)

Return type:

tuple of str

keywords: List[str]

last_updated: str | None = None

links: Dict[str, str]

merge_with(other)

Merge another record for the same work into this one, in place.

self is the record already in the DB, other the incoming one. Rules: - Identifiers: fill any missing doi/arxiv_id/inspire_id. - Abstract: prefer the longer one. - Venue: prefer a real venue over None/”arXiv”/”Unknown Venue”. - Status: higher status_rank wins (stored value preserved). - links / sources / keywords / arxiv_categories: union. - curated=True protects the human-editable fields (title, abstract,

notes, categories, keywords, venue, status) from any overwrite.

• An existing review decision of accepted/rejected is terminal and never replaced; otherwise a decided incoming review is adopted.

Returns:

True if any field changed.

Return type:

bool

notes: str | None = None

review: Dict

sources: List[Dict[str, str]]

status: str | None = None

title: str = ''

to_dict()

Serialize Paper to a JSON-safe dict.

This is the method used when writing to the canonical DB (site/data/livingreview.json). It ensures: - Normalized identifiers, - Always includes categories/keywords as lists, - Timestamps in ISO format.

Returns:

Dictionary representation of the Paper.

Return type:

dict

venue: str | None = None

year: int | None = None

living_review.data_model.status_rank(status)

Return integer rank of a status (higher = more advanced).

Parameters:

status (str or None) – Status string (pending, preprint, published…). Legacy values (proceeding, report, phd, internship, unknown) are ranked via STATUS_ALIASES without being rewritten.

Returns:

Position in STATUS_ORDER, or -1 if unknown.

Return type:

int

living_review.exporters module

exporters.py

Output/export utilities for the Living Review project.

This module provides functions to export processed papers and statistics into formats directly consumable by the Hugo site and citation managers.

Export Targets

• JSON → site/data/livingreview.json + site/data/statistics.json

  The canonical JSON database, containing papers + statistics. Used by Hugo templates, Decap CMS, and downstream visualisations.

• BibTeX → site/static/downloads/livingreview.bib

  A citation file containing all papers in BibTeX format.

• PDF → site/static/downloads/livingreview.pdf

  A printable PDF summary of the review.

Typical Usage

>>> from living_review.exporters import export_json, export_bibtex, export_pdf
>>> export_json(papers, stats, outdir=".")
>>> export_bibtex(papers, outdir=".")
>>> export_pdf(papers, stats, outdir=".")

living_review.exporters.export_bibtex(papers, outdir)

Export papers into a BibTeX file for citation management.

Output file

• site/static/downloads/livingreview.bib

living_review.exporters.export_json(papers, stats, outdir, chunking=None)

Export the canonical JSON database for Hugo and Decap CMS.

Output files

• site/data/livingreview.json : Full DB (stats + papers)

• site/data/statistics.json : Simplified global stats

Metadata

• Adds last_updated as UTC ISO timestamp.

• Adds next_update based on environment variable UPDATE_INTERVAL_HOURS (default: 24h).

living_review.exporters.export_pdf(papers, stats, outdir)

Export a printable PDF summary of the review.

Output file

• site/static/downloads/livingreview.pdf

living_review.fetchers module

fetchers.py

Data-source fetchers for the Living Review project.

This module provides functions to query multiple bibliographic APIs and return lists of Paper objects (via Paper.from_source).

Supported sources: - arXiv: via arxiv Python client. - InspireHEP: via REST API. - HAL (Hyper Articles en Ligne). - OpenAlex. - Crossref.

Each fetcher:

• Retrieves results within a given date window.

• Normalizes metadata into the canonical schema expected by Paper.

• Populates links, status, and provenance (source).

A shared requests.Session with retry logic is used for robustness.

living_review.fetchers.arxiv_query_for_window(start=None, end=None)

Build arXiv queries targeting accelerator physics and ML categories.

Parameters:

• start (datetime.date, optional) – If given, a submittedDate:[… TO …] range is appended so the API bounds the result set (otherwise each query is truncated at max_results with no guarantee of covering the window).

• end (datetime.date, optional) – If given, a submittedDate:[… TO …] range is appended so the API bounds the result set (otherwise each query is truncated at max_results with no guarantee of covering the window).

Returns:

Query strings to be passed to the arxiv client.

Return type:

list of str

living_review.fetchers.fetch_arxiv(start, end)

Fetch papers from arXiv within the given date range.

Parameters:

• start (datetime.date) – Start date.

• end (datetime.date) – End date.

Returns:

Papers retrieved from arXiv.

Return type:

list of Paper

living_review.fetchers.fetch_crossref(start, end)

Fetch papers from Crossref API across PRAB, JACoW, and general accelerator+ML topics.

Combines three categories: - PRAB (prefix:10.1103 PhysRevAccelBeams) - JACoW / IPAC / ICALEPCS / LINAC conference papers - Generic ‘accelerator machine learning’ search

Parameters:

• start (datetime.date)

• end (datetime.date)

Return type:

list of Paper

living_review.fetchers.fetch_hal(start, end)

Fetch papers from HAL API (filtered to ML + accelerator physics).

Return type:

List[Paper]

living_review.fetchers.fetch_inspire(start, end, rows=50, max_pages=5)

Fetch papers from InspireHEP API (AI/ML applied to accelerators).

Return type:

List[Paper]

living_review.fetchers.fetch_openalex(start, end)

Fetch papers from OpenAlex API (60-day windows etc.), and set venue to the actual journal/conference (not the source name).

Return type:

List[Paper]

living_review.fetchers.fetch_pubmed(start, end, rows=50)

Fetch papers from Europe PMC (PubMed interface).

Parameters:

• start (datetime.date) – Start date.

• end (datetime.date) – End date.

• rows (int, optional) – Number of results per page (default=50).

Returns:

Papers retrieved from Europe PMC / PubMed.

Return type:

list of Paper

living_review.fetchers.fetch_semanticscholar(start, end, limit=100)

Fetch papers from the Semantic Scholar Graph API related to machine learning and accelerator physics.

Parameters:

• start (datetime.date) – Start date.

• end (datetime.date) – End date.

• limit (int, optional) – Max number of results to fetch (default=100).

Returns:

Papers retrieved from Semantic Scholar.

Return type:

list of Paper

living_review.fetchers.fetch_springer(start, end, rows=20)

Fetch papers from Springer Nature API (PAM v2).

Parameters:

• start (datetime.date) – Start date.

• end (datetime.date) – End date.

• rows (int, optional) – Number of results to retrieve (default=20).

Returns:

Papers retrieved from Springer.

Return type:

list of Paper

living_review.fetchers.make_session()

Create a requests.Session with retry strategy.

Retries on server errors (500, 502, 503, 504) up to 3 times with exponential backoff.

Returns:

Configured session with retry-enabled adapters.

Return type:

requests.Session

living_review.logs module

logs.py

Logging utilities for the Living Review project.

This module manages: - Persistent scan logs (scan_log.json) storing metadata about each run. - Error logs (errors.log) with stack traces. - Retrieval of the last scanned date range.

Contents

• append_scan_log: record metadata about a scan (papers, chunks, status).

• log_error: record exceptions and stack traces in a log file.

• get_last_scan: retrieve the last recorded scan range.

Typical Usage

>>> from living_review import logs
>>> logs.append_scan_log("logs", start, end, npapers=42)
>>> logs.log_error("logs", Exception("Something went wrong"))
>>> last = logs.get_last_scan("logs")

living_review.logs.append_scan_log(logdir, start, end, npapers, nchunks=1, status='ok', error_msg=None)

Append an entry to the scan log (scan_log.json).

Parameters:

• logdir (str or Path) – Directory where the log files are stored.

• start (datetime.date or str) – Start date of the scan.

• end (datetime.date or str) – End date of the scan.

• npapers (int) – Number of papers processed.

• nchunks (int, optional) – Number of chunks processed (default=1).

• status (str, optional) – Status string for the run (default=”ok”).

• error_msg (str, optional) – Error message if the scan encountered an issue.

Returns:

Updates scan_log.json with a new entry.

Return type:

None

living_review.logs.get_last_scan(logdir)

Retrieve the last scan range from scan_log.json.

Parameters:

logdir (str or Path) – Directory containing the scan log file.

Returns:

Dictionary with keys {“start”: str, “end”: str} if available, otherwise None.

Return type:

dict or None

living_review.logs.log_error(logdir, exc)

Append an error entry with stack trace to errors.log.

Parameters:

• logdir (str or Path) – Directory where the error log is stored.

• exc (Exception) – Exception object to log.

Returns:

Writes a timestamped error entry to errors.log.

Return type:

None

living_review.pipeline module

pipeline.py

Main orchestration pipeline for the Living Review project.

The LivingReviewPipeline class coordinates the entire workflow: 1. Load the canonical DB (data/db.json; falls back to the legacy

published file on first run).

2. Fetch papers from multiple bibliographic sources (arXiv, InspireHEP, HAL, OpenAlex, Crossref) and merge/deduplicate them into the DB.

3. Optionally ingest CMS-approved manual submissions.

4. Run the relevance funnel over undecided papers only (enrich → gates → NLI adjudicator, see relevance.py / SCOPE.md). Accepted/rejected decisions are terminal.

5. Classify accepted papers into categories; Others-only strays are demoted to the pending queue.

6. Compute statistics and export.

Data files

• data/db.jsoncanonical DB — every paper ever seen,

  with decision provenance. Committed.

• data/pending_review.json : ranked human-review queue (regenerated).

• site/data/livingreview.json: published, accepted-only, derived — what

  Hugo renders. Never hand-edited.

• BibTeX → site/static/downloads/livingreview.bib

• PDF → site/static/downloads/livingreview.pdf

class living_review.pipeline.LivingReviewPipeline(start, end, sources=None, output_dir='.', chunking=None, db_path='data/db.json', promote_manual=False, adjudicator=None)

Bases: object

Orchestrates the end-to-end Living Review workflow.

run()

Execute the pipeline end-to-end.

living_review.stats module

stats.py

Computation of summary statistics for the Living Review project.

This module aggregates counts of papers by year, category, venue, keyword, and monthly trends. These statistics are used for reporting and visualizations in the exported JSON/HTML outputs.

Contents

• KEYWORDS: predefined list of relevant keywords to track.

• compute_stats: aggregate statistics from a list of papers.

Typical Usage

>>> from living_review.stats import compute_stats
>>> stats = compute_stats(papers)
>>> stats["per_year"]
{'2024': 15, '2025': 7}

living_review.stats.compute_stats(papers)

Compute aggregated statistics from a list of papers.

Parameters:

papers (list of Paper) – Papers to analyze. Each must have attributes .year, .categories, .venue, .title, .abstract, and .date (string ISO or datetime).

Returns:

Dictionary with the following keys: - “per_year”: counts of papers per publication year. - “per_category”: counts of papers per semantic category. - “per_venue/journal”: counts of papers per venue/journal. - “per_keyword”: counts of predefined keywords matched in titles/abstracts. - “monthly_trends”: counts of papers per month (YYYY-MM).

Return type:

dict

living_review.utils module

utils.py

Utility functions for the Living Review project.

This module provides helper functions for: - Deduplicating papers based on unique keys. - Normalizing identifiers (DOI, arXiv ID). - Cleaning up LaTeX markup and titles for comparison. - Fuzzy similarity scoring between titles. - Checking if a date lies within a given range.

Contents

• deduplicate: remove duplicate papers by (arxiv_id, doi, normalized_title).

• within_range: test whether a date falls within [start, end].

• norm_doi: normalize DOI strings to a canonical form.

• norm_arxiv_id: normalize arXiv identifiers to a canonical form.

• simplify_title: lowercase, strip LaTeX and punctuation for fuzzy matching.

• first_author_key: heuristic to extract first author surname.

• similar_title: fuzzy similarity score between two titles.

Typical Usage

>>> from living_review.utils import norm_doi, simplify_title, within_range
>>> norm_doi("https://doi.org/10.1103/PhysRevLett.123.456")
'10.1103/physrevlett.123.456'
>>> simplify_title("A {LaTeX} Example: On $\alpha$-decay")
'a latex example on alpha decay'
>>> within_range(dt.date(2025, 1, 10), start, end)
True

living_review.utils.canonical_ids(paper)

Set of normalized identifier strings for a paper, used for identifier-graph deduplication. Two papers sharing any element of this set refer to the same work.

Parameters:

paper (Paper) – Any object exposing .doi, .arxiv_id, .inspire_id.

Returns:

Elements of the form doi:..., arxiv:..., inspire:....

Return type:

set of str

living_review.utils.deduplicate(papers)

Remove duplicate papers based on their deduplication key.

Each Paper must implement .key_for_dedup() which returns a tuple (arxiv_id, doi, normalized_title). Duplicates are detected when this key repeats.

Parameters:

papers (list of Paper) – Papers to deduplicate.

Returns:

Deduplicated list of papers (order preserved: first occurrence kept).

Return type:

list of Paper

living_review.utils.first_author_key(authors)

Heuristic key for first author: uses last token of first author’s name. Returns lowercase surname or None if unavailable.

Return type:

Optional[str]

living_review.utils.make_session()

Create a shared requests.Session with retry strategy. Retries on server errors (500, 502, 503, 504) up to 3 times with exponential backoff.

living_review.utils.norm_arxiv_id(ax)

Normalize arXiv identifiers (remove prefix and version).

Return type:

Optional[str]

living_review.utils.norm_doi(doi)

Normalize DOI to lowercase without URL prefixes.

Return type:

Optional[str]

living_review.utils.norm_space(s)

Collapse multiple spaces and trim a string.

Return type:

Optional[str]

living_review.utils.similar_title(a, b)

Compute fuzzy similarity ratio between two titles.

Parameters:

• a (str) – Titles to compare.

• b (str) – Titles to compare.

Returns:

Similarity ratio in [0, 1], where 1 = identical.

Return type:

float

living_review.utils.simplify_title(t)

Lowercase, strip LaTeX, punctuation, and extra spaces from title.

Return type:

Optional[str]

living_review.utils.within_range(d, start, end)

Check whether a date lies within a given range [start, end].

Parameters:

• d (datetime.date) – Date to test.

• start (datetime.date) – Start of the range.

• end (datetime.date) – End of the range.

Returns:

True if start <= d <= end, otherwise False.

Return type:

bool

Module contents

living_review

A Python package for managing and analyzing Living Reviews, with a focus on applications in particle accelerators and machine learning.

This package provides: - Data model (Paper class) to represent scientific papers. - Fetchers for multiple bibliographic APIs (arXiv, InspireHEP, HAL,

OpenAlex, Crossref).

• Semantic filtering and classification of papers using sentence-transformers.

• Statistics computation for bibliometrics and trends.

• Export utilities to JSON and HTML.

• Logging of scans and errors.

• A pipeline (LivingReviewPipeline) to orchestrate the entire workflow.

• A CLI (living_review.cli) for running scans from the terminal.

living_review.__version__

Current version of the package.

Type:

str