SPIRAL2 LLRF fault-diagnosis pipeline

Every event starts as a raw postmortem buffer captured by the LLRF system's trigger logic across 27 monitored signals. Labels come from Classify_PostMortemFile, a deterministic classifier built on the first hardware fault-register bit to trip chronologically — not the older, less reliable summary ALM header field this pipeline used to rely on. Four of the seven categories get a further physics-threshold refinement on top of that hardware decode: a cavity-voltage fall of at least 50% within a bounded time window separates a genuine quench from a false alarm; the reflected-power/cavity-voltage ratio Ucav/Ucat > 8.5 marks an RF safety threshold exceedance; oscillation, gain, and beam-anomaly subtypes of RF regulation out of tolerance come from Z-score+FFT analysis of the drive and reflected signals. The remaining three categories are the hardware decode alone.

Of 14,232 raw files scanned (2019–2025), 102 fail to parse and 20 are zero-byte, leaving 14,109 ground-truth rows. Before any of the seven fault categories or Normal is even considered, each file must pass six validity checks — in order, so a file failing an earlier check is never evaluated against a later one. Chart below reflects the corrected classifier (re-run 2026-09-09, see the callout further down) — the check order and the four reliability/state checks are unchanged, only the feed-forward check's scope changed:

Fixing the scope recovers real events — not an estimate this time, the classifier has actually been re-run against the full corpus and the before/after is a direct diff:

Is the header trustworthy, or could a stuck bit be discarding files that were actually running? — the coherence check from before the correction, still valid as a read of what the header field itself behaves like (the sample below was drawn from the old, unscoped Beam NON grouping, which mixes pre-2021 defaulted-NON files with genuine post-2021 feed-forward-off files — useful for checking the flag's physical meaning, not a current rejection-count figure; see the corrected counts above for that). For a random sample of files in each rejected group (n=100, plus a 150-file passing baseline mixing Normal and every fault category), we read the real cavity-voltage (Ucav) and drive (Uci) waveform amplitude via PyPostMortem, independent of the header flags that got the file rejected.

One more thing tested along the way: is there an independent beam-presence signal in this data at all, now that the header field turned out not to be one? The classifier's own code carries a field literally called Presence faisceau, but it's built by renaming a per-sample état bit labeled Temp RF int on the way out — a mismatch between the variable name and its documented source. Tested against 250 of the same sampled files: even in the passing baseline, this bit reads "on" for essentially none of the trace (median 0.09% of samples). That's not what a real, sustained beam-presence signal should look like, so this field looks mislabeled or misassigned rather than trustworthy — worth a bug report to whoever owns Classify_PostMortemFile, separately from the feed-forward correction above. courant pickup (pickup-probe current) was the other candidate; it also failed, pinning at a near-constant ~33 µA for the large majority of files in every group. Net: there is currently no independently-verified beam-presence signal in this dataset at all.

Status: V7 extracted and validated. The classifier fix is applied, re-run, and the recovery confirmed (+756 events, entirely 2019–2020, everything else untouched). prepare_data_cluster_v7.py (forked from V6, with the beam_present→feed_forward_enabled feature rename applied) has run against the corrected ground truth: 2,865 events (2,866 ground-truth-passing minus the same single feature-extraction failure V6 also hit), 1,478 Normal / 1,387 fault, 810 features (up from 777 — expected: the same 0.95 correlation-dedup threshold keeps a different subset once the dataset's composition changes, not a bug), 246 PCA components. Validated directly: NaN rate in the newly-recovered 2019–2020 events (0.20% / 0.31%) is lower than the full corpus's own average (0.45%), and their fault-category mix (majority Normal, RF regulation out of tolerance the largest fault category) matches the rest of the corpus — no sign the recovered events are degenerate. Steps 01–08 below still describe V6 — re-running them against V7 (Phase 2) hasn't started yet.

Three operations, in order, applied to every one of the 27 signals independently: a 4th-order Butterworth high-pass filter (cutoff 0.01, normalized to each file's own Nyquist frequency — i.e. relative to that file's own sample rate, since decimation varies ~250× across the corpus) removes DC offset and slow drift while keeping fast transients; each event is aligned to its own true trigger sample, read directly off the acquisition's real time axis (zero_idx = argmin(|time|)) rather than a fixed offset; then the filtered signal is Z-score normalized over its own event window, with the filtered-but-unnormalized version kept alongside it (the 2026-09-07 fix from step 02, so real-amplitude statistics have real amplitude to work from).

The old bug, with a real number attached: for a representative NDEC=200 event (2022 10 12 CMA01_0001, an Oscillation Ucav subtype of RF regulation out of tolerance), the true trigger sits at sample 50,047 of 100,096. The pipeline's old hardcoded trigger_idx=3000 would have pointed 534 ms early — 47,047 samples off, 47% of the entire file — treating what was actually mid-acquisition steady-state as if it were the pre-trigger precursor window. Every "pre-trigger" and "post-trigger" feature in the old pipeline was computed from the wrong slice of the trace for a file like this one.

Two more things worth being precise about, since they're easy to gloss over. First, the precursor window used for early-fault-onset detection isn't "the first 3,000 samples" or "34 ms" in some fixed sense — it's scale_samples(3000, dt_us), which converts a sample count tuned at the dominant NDEC=200 spacing (dt ≈ 11.4 µs) into whatever sample count gives the same physical duration for this file's own dt_us. A file with NDEC=100 (like several of the recovered 2019 events above) needs about twice as many samples for the same 34 ms. Second, one fault category — RF regulation out of tolerance — gets a further onset refinement on top of the informatic trigger: Alexandre's AMPT-based physical onset detection (find_true_t0_with_ampt), applied only to that category since it's the one his validation actually covers, per the earlier user decision documented in the methodology.

Three families, computed for every event: 17 statistics × 27 signals over the full window (459 candidates); seven physics-informed families targeting specific LLRF dynamics (39 candidates: effective decay, detuning, microphonics, RF mismatch, modulator command, phase stability, RF power flow); and temporal/derivative features — per-segment slopes, finite differences, CUSUM change points, early/late-window ratios — the largest group by far. Before correlation filtering that's 1,200+ raw candidates; after the >0.95 dedup pass, 810 features survive in V7 (up from 777 in V6). Verified directly on V7 with the same family classifier the paper's SHAP analysis uses (100% coverage, zero unclassified features, both versions):

One family worth a close look because it's the most physically interpretive: effective system decay. Post-trigger, the pipeline fits cavity voltage amplitude (|U_cav|, in MV/m) to V(t) = V₀·e^−t/τ_eff over a fixed 500-sample-equivalent window (≈5.7 ms), by linear regression on ln|U_cav|. The report is explicit that τ_eff is not the cavity's intrinsic Q_L — with the LLRF loop still active and protection logic responding, what's measured is the whole system's response. First pass at this (below, corrected) picked one clean-fitting and one poor-fitting quench event and left it at "an active loop makes the decay non-exponential" — too glib, and the log-scale plot hid the actual units and made a genuinely important detail invisible. Redone properly: measured how long each of the 31 real quench events takes to drop below 20% of its pre-trigger amplitude, across the whole category, not two cherry-picked examples.

Checked whether the pipeline's own subtype-clustering step (09, k=2 search per category) already found this: it gives 19/9 for this category on V6's 28 events, and that smaller cluster (n=9) is disproportionately the group with no valid decay window (56% vs 26% in the larger cluster) — correlated with the fast/slow split, but not identical to it. Checked why: the real top-differentiating features between those two clusters (same Cohen's-d method the report's own subtype figure uses) turn out to be modulator-command and reference-frequency statistics, not the decay features directly — the existing clustering runs over the full 810-feature space and lands on whatever axis separates best overall, which isn't necessarily collapse-timescale. The collapse-time gap shown above is a cleaner, more directly physical signal than what the current full-feature clustering surfaces — worth flagging as a concrete, real methodology improvement: either add collapse-timescale as an explicit feature, or run the subtype search on the physics-informed decay features specifically rather than the full feature space, before trusting the current quench subtype split as physically meaningful.

Last check: the whole-signal-statistic normalization-order bug (Sec. 00/01, fixed 2026-09-07) reduced near-constant features from 65 to 15 on V6. Re-ran the same check on V7: 16 near-constant features (std<1e-6 of 810) — same story, one more. The two previously-disclosed non-functional features (reflected_saturation_fraction, control_saturation_fraction, still-miscalibrated legacy thresholds) are still in the list, unchanged. The one addition, llrf_loop_closed, is easily explained: V7's recovered 2019–2020 events skew the corpus slightly more toward closed-loop operation, pushing that flag's variance just under the threshold — a composition effect, not a new bug.

Early fault-onset detection, restricted to the pre-trigger-only 386-feature subset so the task can't cheat off the post-trigger transient. Five unsupervised baselines (Isolation Forest, LOF, Mahalanobis distance on a PCA-reduced Ledoit–Wolf covariance, PCA reconstruction error, DBSCAN) establish how much separability exists without labels, then a class-weighted Random Forest is trained supervised on the same subset. An ensemble score combines all of the unsupervised methods with the RF's own probability output.

The large supervised/unsupervised gap (0.999 vs 0.58–0.79) is a real, learnable decision boundary, not a window-length leak: a follow-up check restricting to only the already-fixed-window columns (382 of 766, same conclusion as the 386-feature production subset above) still gets RF AUC 0.9998 while Isolation Forest/LOF on the identical columns stay at 0.65/0.59 — unsupervised methods can't see what the label-trained model sees, which window length alone can't explain.

A model really keying on an approaching fault should look uncertain far from the trigger and grow confident as it nears — instead, confidence is already saturated at the furthest offset tested and never moves. Taken alone, that pattern would suggest the model is picking up a stable per-capture characteristic unrelated to the fault itself, rather than genuine fault-onset proximity — the open caveat the report already carries (Sec. 6/7/8).

Fault-vs-Normal on the full event window (post-trigger included), using all 777 features, compared across Logistic Regression, Random Forest, XGBoost, and an RBF-kernel SVM. This task is expected to sit near ceiling performance — the full window includes the interlock-trip transient itself, which is close to trivially separable on amplitude alone — a materially easier task than precursor detection above by design, not evidence of a leak on its own.

Four competing approaches to the same seven-category task, each class-weighted appropriately for its own structure: Binary Relevance, Classifier Chains, and Label Powerset (all with RF or XGBoost base estimators), plus four deep multi-head architectures (CNN, Transformer, CNN-LSTM, MLP) sharing a per-label weighted BCE loss. Every fault event in this dataset carries exactly one active label by construction of the ground-truth classifier, so the genuinely multi-label case (several simultaneous faults) hasn't been exercised on real data yet. Label Powerset is the best-performing approach on the current class-weighted, leakage-safe rerun; all four deep architectures underperformed the classical baselines on the comparison run of record.

A class-weighted Random Forest predicts the primary fault category among the seven, on the full 777-feature set. The training target is argmax over the multi-label vector — a deterministic first-flagged-category rule, not an independently labeled physical root cause — and because the current dataset has zero genuinely multi-fault events, that argmax currently has one unambiguous answer per event rather than exercising any real tie-breaking. A real limitation of the task's current formulation, disclosed as such rather than presented as a solved comparison.

Within each fault category, K-Means, Agglomerative, and Gaussian Mixture clustering are compared by silhouette score to look for subtypes. Two corrections were needed before this was trustworthy: a near-zero-denominator division bug that let a handful of events take extreme outlier values, and a subtler bias where unconstrained silhouette maximization reliably chose whichever split isolated the single tightest minority subgroup — as small as one event — over a genuine population split. Fixed by winsorizing features at the 5th/95th percentile per category and requiring every cluster hold at least 15% of that category's events. Every one of the five analyzable categories converges to a balanced k=2 split under that constraint.

SHAP explains the classification models per physics-informed feature family, reported as per-feature-mean importance rather than a naive per-group sum — the statistical and temporal families outnumber the seven physics families combined roughly 20:1, so a group sum would structurally favor them regardless of how informative any single physics feature is. Every number in the paper, including every figure and table above, traces to a versioned, git-tracked results manifest rather than being hand-typed: each pipeline script writes its own manifest, a generator converts it into LaTeX macros, and a checker flags any bare numeric literal in the report text that isn't backed by one.