Losses and criteria¶

NormMSE is the regression training loss; the functions below compute the criteria that score candidates. See Criteria.

NormMSE ¶

NormMSE(eps: float = 1e-08, centered: bool = True, scale: float | None = None, censor_at: float | None = None)

Bases: Module

Sum-of-squares error normalized by the spread of the targets.

loss = Σ(y - ŷ)² / (n · scale)          when `scale` is set
loss = Σ(y - ŷ)² / Σ(y - ȳ)²           centered, per call
loss = Σ(y - ŷ)² / Σ y²                 centered=False (gmdhpy parity)

scale is the per-sample target variance measured once over the whole training split (Trainer._fit_target_scale) and is the form the trainer uses. Fixing the denominator rather than recomputing it per batch keeps the loss batch-size-independent and safe on a degenerate batch — with the schema default batch_size: 1 a per-batch centered denominator would be exactly zero on every batch — and puts the training loss on the same scale as the dev-set regularity criterion, which is what makes the absolute early_stop_patience / divergence_threshold thresholds mean the same thing across datasets.

The per-call fallback (scale=None) applies when the loss is used outside a Trainer.train, e.g. a standalone train_out_proj after load_model_checkpoint.

scale is a plain attribute rather than a registered buffer on purpose: loss_fn is a SONN submodule, so a buffer would add a state_dict key and invalidate existing checkpoints. It is a property of the data, not of the fitted model, and is re-measured on every train() including resume.

censor_at handles a right-censored target (e.g. California housing prices recorded as 5.0 for every house worth 5.0 or more). A target at or above the cap only says "at least this much", so for those rows the prediction is clipped to the cap before the squared error: predicting above the cap costs nothing and pushes no gradient, predicting below it is penalized as usual. Uncensored rows are untouched. This is censored least squares - the Tobit likelihood with the noise scale taken to zero - and it keeps the objective a convex piecewise quadratic. Predictions should be clipped to the cap at inference time as well; the model is now free to predict above it. The cap is in the units the loss sees (log price when the target is log-transformed). None (default) disables it.

forward ¶

forward(y_pred: Tensor, y_true: Tensor) -> torch.Tensor

Normalized squared error of y_pred against y_true.

Parameters:

Name	Type	Description	Default
`y_pred`	`tensor`	Predictions. Clipped to `censor_at` on the censored rows when it is set.	required
`y_true`	`tensor`	Targets, same shape as `y_pred`.	required

Returns:

Name	Type	Description
`loss`	`scalar tensor`	sum((y - y_hat)^2) divided by n*`scale` when `scale` is set, otherwise by the (centered) sum of squares of `y_true`; `eps` is added to the denominator.

errors ¶

regularity_error ¶

regularity_error(y_hat_B: Tensor, y_B: Tensor, eps: float = 1e-12, centered: bool = True) -> torch.Tensor

GMDH regularity criterion (regression) — squared error on subset B, normalized by the spread of the targets on B.

Δ²(B) = Σ_i (y_i - ŷ_i)² / Σ_i (y_i - ȳ)²      (centered=True)
Δ²(B) = Σ_i (y_i - ŷ_i)² / Σ_i y_i²            (centered=False)

ŷ comes from a candidate whose coefficients were fit on subset A and evaluated on subset B. Lower is better; Δ² → 0 for a perfect fit, Δ² > 1 means the model is worse than the baseline.

The centered form is the fraction of variance unexplained (1 - R²): Δ² ≈ 1 for predictions no better than the mean predictor, and the value is invariant to a constant offset on y — which matters because a linear-in-coefficients model is itself translation-invariant, so its measured quality must not move when the target is shifted.

centered=False is Ivakhnenko's original Σy², kept for parity with gmdhpy and other classical implementations. Its baseline is the zero predictor, which is only a meaningful reference when the targets are already centered: for non-centered targets Σy² ≈ N(ȳ² + var) is dominated by the mean (on a target with mean 454 and sd 17 the denominator is ~700x the variance), so Δ² collapses into a razor-thin band near zero and stops being comparable across datasets or against absolute thresholds.

Candidate ranking is unaffected by the choice — the denominator is a constant within one evaluation — so this is about what the number means, not about which neuron wins.

Parameters:

Name	Type	Description	Default
`y_hat_B`	`(..., N) tensor`	Predictions on subset B from each candidate. Leading dims carry the candidate batch.	required
`y_B`	`(N,) tensor`	Ground-truth targets on subset B.	required
`eps`	`float`	Floor on the denominator, guarding against degenerate targets (constant under `centered`, all-zero otherwise).	`1e-12`
`centered`	`bool`	Normalize by the centered sum of squares. Default True.	`True`

Returns:

Name	Type	Description
`delta_sq`	`(...) tensor`	Regularity error per candidate. Lower is better.

bias_error ¶

bias_error(y_hat_A: Tensor, y_hat_B: Tensor, y: Tensor, eps: float = 1e-12, centered: bool = True) -> torch.Tensor

GMDH bias criterion (regression) — squared difference between predictions from the candidate fit on A and on B, evaluated on a common set (typically A ∪ B), normalized by the spread of the targets on that set:

n_B = Σ_i (ŷ_iA - ŷ_iB)² / Σ_i (y_i - ȳ)²     (centered=True)
n_B = Σ_i (ŷ_iA - ŷ_iB)² / Σ_i y_i²           (centered=False)

See regularity_error for why the centered form is the default; the two criteria must use the same denominator, since cmpComb_validate_bias combines them as (1-alpha)*bias + alpha*regularity and a mismatch would silently re-weight error_alpha.

Measures parameter stability: small n_B means the model's coefficients are insensitive to which half of the data they were fit on (= captures real structure); large n_B means the model is fitting noise.

Equivalent in ranking (and cheaper) for linear-in-coefficients models: n_B = ||a_A - a_B||² up to a constant scale.

Sample alignment: y_hat_A[..., i], y_hat_B[..., i], and y[i] all refer to the same row of input. Both forward passes must be on a fixed evaluation set in the same order — never via a shuffled dataloader.

Parameters:

Name	Type	Description	Default
`y_hat_A`	`(..., N) tensors`	Predictions on the common evaluation set from candidate coefficients fit on subset A and B respectively. Leading dims carry the candidate batch.	required
`y_hat_B`	`(..., N) tensors`	Predictions on the common evaluation set from candidate coefficients fit on subset A and B respectively. Leading dims carry the candidate batch.	required
`y`	`(N,) tensor`	Ground-truth targets on the common evaluation set, used only for the denominator's normalization.	required
`eps`	`float`	Floor on the denominator.	`1e-12`
`centered`	`bool`	Normalize by the centered sum of squares. Default True.	`True`

Returns:

Name	Type	Description
`n_B`	`(...) tensor`	Bias error per candidate. Lower is better; identical fits give 0.

regularity_error_ce ¶

regularity_error_ce(logits_B: Tensor, y_B: Tensor, eps: float = 1e-12) -> torch.Tensor

CE-GMDH regularity criterion — Normalized Cross-Entropy on subset B.

NCE = mean_i [ -log p_model(y_i | x_i) ] / H(Y_B)

H(Y_B) is the Shannon entropy of the empirical class frequencies on B, which equals the cross-entropy of the constant-marginal predictor and serves as the principled "no-skill" baseline. NCE → 0 for a perfect model; NCE ≈ 1 means no better than predicting marginal frequencies; NCE > 1 means actively worse than the trivial classifier.

Parameters:

Name	Type	Description	Default
`logits_B`	`(..., N, K) float tensor`	Projection-layer outputs (W · z) on subset B for each candidate. Leading dims are treated as candidate-batch dimensions.	required
`y_B`	`(N,) long tensor`	Integer class labels on B. For one-hot labels, pass y_B.argmax(-1).	required
`eps`	`float`	Floor for H(Y_B), guarding against absent classes on near-degenerate validation splits.	`1e-12`

Returns:

Name	Type	Description
`nce`	`(...) tensor`	Regularity error per candidate. Lower is better.

bias_error_l2 ¶

bias_error_l2(z_A: Tensor, z_B: Tensor, y: Tensor = None, logits: bool = False) -> torch.Tensor

CE-GMDH bias criterion — L2 disagreement on neuron outputs, normalized by label energy. Direct port of Ivakhnenko's formula:

n_B = Σ_i ||z_A(x_i) - z_B(x_i)||^2  /  Σ_i ||y_i||^2

For one-hot labels, Σ_i ||y_i||^2 = N regardless of N and K, so the default (y = None) uses N as the denominator — the structurally correct value for standard classification encoding. Pass y explicitly for ordinal encodings or soft/weighted labels.

logits=True selects the multi-class form, where each sample carries a length-K vector and ||·||² sums over the class axis. The default scalar form treats the last axis as the sample axis. The flag is explicit rather than inferred from z_A.dim() because (ensemble, N) and (N, K) are both 2-D and mean different things — inferring would silently reduce over the wrong axis for one of them.

Sample alignment: z_A[..., i] and z_B[..., i] (scalar form), or z_A[..., i, :] and z_B[..., i, :] (logits form), MUST refer to the same input x_i.

Parameters:

Name	Type	Description	Default
`z_A`	(..., N) tensors, or (..., N, K) when `logits`	Neuron outputs on the common evaluation set, from the same neuron structure with parameters fit on subset A vs B. Leading dims carry the candidate batch.	required
`z_B`	(..., N) tensors, or (..., N, K) when `logits`	Neuron outputs on the common evaluation set, from the same neuron structure with parameters fit on subset A vs B. Leading dims carry the candidate batch.	required
`y`	`tensor`	Labels for explicit denominator. (N,) integers for ordinal encoding, or (N, K) one-hot / soft labels. If None, denominator defaults to N — the one-hot case.	`None`
`logits`	`bool`	Treat the last axis as classes rather than samples. Default False.	`False`

Returns:

Name	Type	Description
`bias`	`(...) tensor`	Lower is better; identical fits give 0.

bias_error_js ¶

bias_error_js(logits_A: Tensor, logits_B: Tensor) -> torch.Tensor

CE-GMDH bias criterion — Normalized Jensen–Shannon divergence in predicted-probability space.

n_B = Σ_i JS(p_A(x_i) || p_B(x_i))  /  (N · ln 2)

The denominator is the analog of Ivakhnenko's Σ_i ||y_i||²: the maximum total disagreement attainable on N samples, which for JS is N · ln 2 (since JS is bounded by ln 2 per sample regardless of K). Result is in [0, 1]: 0 = identical predictions, 1 = disjoint-support disagreement on every sample.

Use when you want bounded, portable thresholds across layers / K / datasets, or when behavioral agreement matters more than parameter stability. JS underweights minority-class disagreements on imbalanced data — prefer bias_error_l2 there.

Sample alignment: logits_A[..., i, :] and logits_B[..., i, :] MUST refer to the same input x_i.

Parameters:

Name	Type	Description	Default
`logits_A`	`(..., N, K) tensors of identical shape`	Projection-layer outputs (W · z) from the two candidate fits on the common evaluation set. Leading dims carry the candidate batch.	required
`logits_B`	`(..., N, K) tensors of identical shape`	Projection-layer outputs (W · z) from the two candidate fits on the common evaluation set. Leading dims carry the candidate batch.	required

Returns:

Name	Type	Description
`bias`	`(...) tensor in [0, 1]`