Automated detection of missing interaction terms in UK personal lines GLMs.

A Poisson frequency GLM for motor insurance with 12 rating factors has 66 possible pairwise interactions. Manually searching them - fitting, testing, reviewing 2D actual-vs-expected plots - takes days and is driven by intuition rather than data. You will miss interactions that are not obvious from marginal plots, and you will spend time testing pairs that are irrelevant.

The standard manual process:

1. Fit a GBM to get a benchmark prediction
2. Loop over pairs of factors; produce 2D A/E plots
3. Identify where the multiplicative GLM assumption breaks down
4. Test candidate interactions via likelihood-ratio test
5. Repeat

This library automates steps 2–4.

The pipeline has three stages:

Stage 1 - CANN: Train a Combined Actuarial Neural Network (Schelldorfer & Wüthrich 2019) on the residuals of your existing GLM. The CANN uses a skip connection so it starts from the GLM prediction and only learns what the GLM is missing. After training, any deviation of the CANN from zero encodes structure the GLM cannot express - interactions.

Stage 2 - NID: Apply Neural Interaction Detection (Tsang et al. 2018) to the trained CANN weights. The algorithm reads the interaction structure directly from the weight matrices: two features can only interact if they both contribute to the same first-layer hidden unit. The NID score for a pair (i, j) is:

d(i,j) = Σ_s  z_s · min(|W1[s,i]|, |W1[s,j]|)

where z_s is how much first-layer unit s influences the output (the product of absolute weight matrices from layer 2 to the output). This gives a ranked list of candidate interactions in milliseconds after training.

Stage 3 - GLM testing: For each top-K candidate pair, refit the GLM with the interaction added and compute a likelihood-ratio test statistic. The output table includes deviance improvement, AIC/BIC, p-values (Bonferroni corrected), and n_cells - the parameter cost of adding each interaction.

Both Poisson (frequency) and Gamma (severity) families are supported.

import polars as pl
import numpy as np
from insurance_interactions import InteractionDetector, build_glm_with_interactions

# You have: X (Polars DataFrame), y (claim counts), exposure,
# and mu_glm (fitted values from your existing Poisson GLM)

detector = InteractionDetector(family="poisson")
detector.fit(
    X=X_train,
    y=y_train,
    glm_predictions=mu_glm_train,
    exposure=exposure_train,
)

# Ranked interaction table with deviance gains and LR test results
print(detector.interaction_table())

# Top recommended interactions (significant after Bonferroni correction)
suggested = detector.suggest_interactions(top_k=5)
# [("age_band", "vehicle_group"), ("age_band", "ncd"), ...]

# Refit GLM with approved interactions
final_model, comparison = build_glm_with_interactions(
    X=X_train,
    y=y_train,
    exposure=exposure_train,
    interaction_pairs=suggested,
    family="poisson",
)
print(comparison)

The interaction table contains one row per candidate pair:

The n_cells column is important for credibility decisions: a strong interaction requiring 200 new parameters may be less useful than a moderate one requiring 4.

uv add insurance-interactions

With SHAP interaction validation (requires CatBoost):

uv add "insurance-interactions[shap]"

Training is controlled via DetectorConfig:

from insurance_interactions import DetectorConfig, InteractionDetector

cfg = DetectorConfig(
    cann_hidden_dims=[32, 16],   # MLP architecture
    cann_n_epochs=300,
    cann_n_ensemble=5,           # Average over 5 training runs for stable NID
    cann_patience=30,            # Early stopping patience
    top_k_nid=20,                # NID pairs to forward to GLM testing
    top_k_final=10,              # Interactions in final suggest_interactions()
    mlp_m=True,                  # MLP-M variant: reduces false positive interactions
    nid_max_order=2,             # 2 = pairwise; 3 = also compute three-way
    alpha_bonferroni=0.05,       # Significance level after Bonferroni correction
)
detector = InteractionDetector(family="poisson", config=cfg)

Setting mlp_m=True activates the MLP-M architecture (Tsang et al. 2018): each feature gets its own small univariate network to absorb the main effect, forcing the main MLP to model only interactions. This reduces false positive interactions at the cost of more training parameters. Recommended for datasets with strongly correlated features (e.g. age and NCD).

cann_n_ensemble=3 (or more) trains multiple CANN runs with different random seeds and averages the NID scores. CANN training is stochastic; a single run may produce unstable weight matrices. Three runs is a reasonable default; five is better for production use.

Run the detector separately for frequency and severity:

freq_detector = InteractionDetector(family="poisson")
freq_detector.fit(X=X, y=claim_counts, glm_predictions=mu_freq_glm, exposure=exposure)

sev_detector = InteractionDetector(family="gamma")
sev_detector.fit(X=X_claims, y=claim_amounts, glm_predictions=mu_sev_glm, exposure=claim_counts)

In practice, frequency and severity interactions differ. Young driver × sports car interactions are typically stronger in frequency. Severity interactions are noisier due to the higher variance in claim amounts.

The CANN is from Schelldorfer & Wüthrich (2019), "Nesting Classical Actuarial Models into Neural Networks" (SSRN 3320525). NID is from Tsang, Cheng & Liu (2018), "Detecting Statistical Interactions from Neural Network Weights" (ICLR 2018). The direct application of this pipeline to insurance GLMs is in Lindström & Palmquist (2023), "Detection of Interacting Variables for GLMs via Neural Networks" (European Actuarial Journal).

The CANN architecture:

μ_CANN(x) = μ_GLM(x) * exp(NN(x; θ))

The GLM prediction enters as a fixed log-space offset. The output layer of the neural network is zero-initialised so the CANN equals the GLM exactly at the start of training. The network then learns only the residual structure - which, in a well-specified GLM missing interactions, corresponds to those interaction terms.

• NID depends on the CANN having converged. Poor training (small dataset, high learning rate, too few epochs) produces unreliable weight matrices. Use n_ensemble ≥ 3 and check cann.val_deviance_history.
• Very small datasets (< 5,000 policies) may not provide enough signal for the CANN to learn stable residual structure. The LR tests still work but the NID ranking may be noisy.
• NID is not a statistical test - it produces a ranking, not p-values. The LR test in Stage 3 provides the statistical rigour.
• Correlated features (age and NCD in UK motor) can spread interaction signal across spurious pairs. The MLP-M variant with L1 sparsity partially mitigates this.
• The GLM refit step uses glum. If your rating engine uses a different GLM package, the n_cells, delta_deviance, and LR statistics are still valid; just refit your own model with the suggested interaction pairs.

UK actuaries working under PRA SS1/23 model risk governance and FCA Consumer Duty pricing rules need interaction decisions to be auditable. This library is designed to support that: it produces a ranked table with test statistics, not a black-box model. The actuary decides which interactions to add; the library provides the shortlist and the evidence.

Model building

Uncertainty quantification

Deployment and optimisation

Governance

Spatial

All libraries →