Framework for Regime-Switching State-Space Models
Can you learn the rules from the data alone?
Enterprise-grade unsupervised pre-training diagnostic system for identifying practical identifiability boundaries in 2-regime switching state-space models. Uses Observable Moment Matrix + SVD to determine whether regime parameters are identifiable from observations alone.
>85% test coverage. Full type hints. Production-ready.
"The circularity trap: You need to know the true model to train a model that learns the true model."
In regime-switching systems, you observe transitions but never directly observe which regime you're in. This creates a fundamental question: Given your data and interventions, can you actually identify the regime parameters? Or are you chasing a ghost? Identifiability Diagnostic answers this before you waste compute.
The framework implements a rigorous mathematical pipeline to determine identifiability:
[1] Compute First-Order Innovations
dy_t = y_t - y_{t-1}
[2] Lift to Feature Space
v_t = [dy_t, dy_{t-1}]^T
[3] Align Interventional Context
Match intervention levels u to innovation timeline
[4] Construct Moment Matrix
M โ โ^{|U| ร 4} with conditional covariances
M_ij = Cov(v_t | u_t = u_i)
[5] Execute SVD
M = U ฮฃ V^T (Singular Value Decomposition)
[6] Compute Metrics
ฯ_2 (second singular value)
Effective rank
Condition number
[7] Deploy Gate
if ฯ_2 > ฯ (5.12e-17): IDENTIFIABLE โ
else: NON-IDENTIFIABLE โ
For regime-switching model:
where s_t โ {0, 1} denotes regime, and u_t is exogenous intervention. The Observable Moment Matrix captures the covariance structure stratified by intervention level.
Key Decision Rule: The second singular value ฯโ is the identifiability gating metric. If ฯโ exceeds threshold, regime parameters are identifiable from data alone.
| Approach | Strength | Weakness |
|---|---|---|
| Train & Pray | No pre-checks needed | 100 hours of compute wasted on unidentifiable problem |
| Theoretical Analysis | Mathematically rigorous | Requires hand-solving for each problem (weeks of work) |
| Observable Moment Matrix | Automatic, <130ms per check | Limited to SSM regime switching (but that's the target domain) |
Automatically detects parameter identifiability via singular value spectrum of Observable Moment Matrix.
Full support for multi-regime autoregressive processes with exogenous interventions.
Effective rank, condition numbers, singular values, and deployment decisions all computed automatically.
Full type hints, error handling, logging, and >85% test coverage. Ready for enterprise deployment.
GitHub Actions workflows, automated testing across Python 3.8โ3.11, multi-platform (Linux, macOS, Windows).
Easy installation, full dependency management, semantic versioning, and regular updates.
Benchmark results on Intel i7 @ 3.6GHz:
| Task | Data Size | Time |
|---|---|---|
| Data Generation (T=100k) | 100,000 observations | ~50 ms |
| Pipeline Execution | 100,000 observations | ~80 ms |
| Full Diagnostic Cycle | 100,000 observations | ~130 ms |
| Parametric Sweep (5 ฮต values) | 500,000 observations | ~650 ms |
Test Coverage
>85%
Type Hints
100%
Python Versions
3.8โ3.11
Platforms
3/3 โ
pip install identifiability-diagnosticgit clone https://github.com/prakulhiremath/identifiability-diagnostic.git
cd identifiability-diagnostic
pip install -e .pip install -e ".[dev]"from identifiability_diagnostic import (
RegimeSwitchingGenerator,
PretrainingDiagnosticPipeline
)
from identifiability_diagnostic.utils.config import load_config
# Load configuration
config = load_config('config/parameters.yaml')
# Generate synthetic data
generator = RegimeSwitchingGenerator(config)
y, u = generator.generate(epsilon=1e-3, symmetric=False)
# Run diagnostic pipeline
pipeline = PretrainingDiagnosticPipeline(config)
metrics = pipeline.run(y, u)
# Interpret results
print(f"ฯ_2 = {metrics['sigma_2']:.4e}")
print(f"Decision: {metrics['deploy_decision']}")
print(f"Identifiable: {metrics['identifiable']}")# Run parametric sweep
python main.py --config config/parameters.yaml --log-level INFO
# Run with custom epsilon values
python main.py --epsilon 1e-5 1e-4 1e-3 1e-2 1e-1
# Test mode (smaller dataset)
python main.py --test --verbose
# Symmetric control only
python main.py --symmetric-onlyDefault configuration in config/parameters.yaml:
simulation:
T: 100000 # Time series length
seed: 42 # Random seed
sigma_n: 0.5 # Process noise std dev
regime_0:
a: 0.90 # AR coefficient regime 0
b: 1.0 # Intervention coupling
c: 0.0 # Bilinear term
regime_1:
a: 0.90 # AR coefficient regime 1
b: 1.0
c: 0.2 # Asymmetric bilinear!
diagnostic:
u_levels: [-2, -1, 0, 1, 2] # Intervention grid
threshold_tau: 5.12e-17 # Gating thresholdpytest tests/ -v --cov=src/identifiability_diagnosticidentifiability-diagnostic/ โโโ .github/workflows/ # CI/CD configurations โโโ src/ โ โโโ identifiability_diagnostic/ โ โโโ __init__.py โ โโโ core/ # Core algorithms โ โ โโโ data_generator.py โ โ โโโ pipeline.py โ โ โโโ metrics.py โ โโโ utils/ # Utilities โ โ โโโ config.py โ โ โโโ logging.py โ โโโ exceptions.py โโโ tests/ โ โโโ unit/ # Unit tests (20+) โ โโโ integration/ # Integration tests (10+) โโโ examples/ # Usage examples โโโ config/ โ โโโ parameters.yaml # Default config โโโ main.py # CLI entry point โโโ setup.py # Package setup โโโ pyproject.toml # Modern packaging โโโ requirements.txt # Dependencies โโโ README.md
@software{identifiability_diagnostic_2024,
title={Identifiability Diagnostic Framework for
Regime-Switching Models},
author={Prakul S. Hiremath},
year={2026},
url={https://github.com/prakulhiremath/
identifiability-diagnostic}
}Identifiability Diagnostic Framework โ Regime-Switching State-Space Model Analysis
MIT License ยท Python 3.8+ ยท SVD-Based Analysis ยท Open Source
"Know before you train."