Coverage for src/basanos/math/optimizer.py: 100%
60 statements
« prev ^ index » next coverage.py v7.14.1, created at 2026-09-25 12:05 +0000
1"""Correlation-aware risk position optimizer (Basanos).
3This module provides utilities to compute correlation-adjusted risk positions
4from price data and expected-return signals. It relies on volatility-adjusted
5returns to estimate a dynamic correlation matrix (via EWM), applies shrinkage
6towards identity, and solves a normalized linear system per timestamp to
7obtain stable positions.
9Performance characteristics
10---------------------------
11Let *N* be the number of assets and *T* the number of timestamps.
13**Computational complexity**
15+----------------------------------+------------------+--------------------------------------+
16| Operation | Complexity | Bottleneck |
17+==================================+==================+======================================+
18| EWM volatility (``ret_adj``, | O(T·N) | Linear in both T and N; negligible |
19| ``vola``) | | |
20+----------------------------------+------------------+--------------------------------------+
21| EWM correlation (``cor``) | O(T·N²) | ``ewm_covariance`` from |
22| | | ``cvx.linalg`` over all N² pairs |
23+----------------------------------+------------------+--------------------------------------+
24| Linear solve per timestamp | O(N³) | Cholesky / LU per row in |
25| (``cash_position``) | * T solves | ``cash_position`` |
26+----------------------------------+------------------+--------------------------------------+
28**Memory usage** (peak, approximate)
30``ewm_covariance`` from ``cvx.linalg`` processes the input Polars DataFrame
31and returns a dict of covariance matrices. Peak RAM ≈ **O(T · N²)** bytes.
32Typical working sizes on a 16 GB machine:
34+--------+--------------------------+------------------------------------+
35| N | T (daily rows) | Peak memory (approx.) |
36+========+==========================+====================================+
37| 50 | 252 (~1 yr) | ~70 MB |
38+--------+--------------------------+------------------------------------+
39| 100 | 252 (~1 yr) | ~280 MB |
40+--------+--------------------------+------------------------------------+
41| 100 | 2 520 (~10 yr) | ~2.8 GB |
42+--------+--------------------------+------------------------------------+
43| 200 | 2 520 (~10 yr) | ~11 GB |
44+--------+--------------------------+------------------------------------+
45| 500 | 2 520 (~10 yr) | ~70 GB ⚠ exceeds typical RAM |
46+--------+--------------------------+------------------------------------+
48**Practical limits (daily data)**
50* **≤ 150 assets, ≤ 5 years** — well within reach on an 8 GB laptop.
51* **≤ 250 assets, ≤ 10 years** — requires ~11-12 GB; feasible on a 16 GB
52 workstation.
53* **> 500 assets with multi-year history** — peak memory exceeds 16 GB;
54 reduce the time range or switch to a chunked / streaming approach.
55* **> 1 000 assets** — the O(N³) per-solve cost alone makes real-time
56 optimization impractical even with adequate RAM.
58See ``BENCHMARKS.md`` for measured wall-clock timings across representative
59dataset sizes.
61Internal structure
62------------------
63The implementation is split across focused private modules to keep each file
64readable and independently testable:
66* `_config` — `BasanosConfig` and all
67 covariance-mode configuration classes.
68* `_engine_base` — `_BatchCore`, the validated ``prices`` / ``mu`` /
69 ``cfg`` dataclass composing `_CoreDataMixin`, `_SolveMixin` and
70 `_SignalEvaluatorMixin`.
71 `BasanosEngine` subclasses it; the streaming warmup builds it directly.
72* `_engine_validation` — free functions that validate the
73 ``prices`` / ``mu`` / ``cfg`` inputs (re-exported here).
74* `_engine_core` — the `_CoreDataMixin` providing the
75 core data-access properties (``assets``, ``ret_adj``, ``vola``, ``cor``,
76 ``cor_tensor``).
77* `_engine_solve` — private helpers providing the
78 ``_iter_matrices`` and ``_iter_solve`` generators (per-timestamp solve
79 logic).
80* `_engine_diagnostics` — private helpers providing
81 matrix-quality diagnostics (condition number, effective rank, solver
82 residual, signal utilisation).
83* `_engine_performance` — the `_PerformanceMixin` providing the
84 portfolio-Sharpe sweep helpers (``sharpe_at_shrink``,
85 ``sharpe_at_window_factors``, ``naive_sharpe``).
86* `_engine_ic` — private helpers providing signal
87 evaluation metrics (IC, Rank IC, ICIR, and summary statistics).
88* This module — `BasanosEngine`, a single flat class that wires
89 every method together in clearly delimited sections.
90"""
92import dataclasses
94import numpy as np
95import polars as pl
96from jquantstats import Portfolio
98from ._config import (
99 BasanosConfig,
100 CovarianceConfig,
101 CovarianceMode,
102 EwmaShrinkConfig,
103 SlidingWindowConfig,
104)
105from ._config_report import ConfigReport
106from ._engine_base import _BatchCore as _BatchCore
107from ._engine_core import _CoreDataMixin as _CoreDataMixin
108from ._engine_diagnostics import _DiagnosticsMixin as _DiagnosticsMixin
109from ._engine_ic import _SignalEvaluatorMixin as _SignalEvaluatorMixin
110from ._engine_performance import _PerformanceMixin as _PerformanceMixin
111from ._engine_solve import _SolveMixin as _SolveMixin
112from ._engine_validation import _numeric_assets as _numeric_assets
113from ._engine_validation import _validate_inputs as _validate_inputs
114from ._engine_validation import _validate_non_monotonic_prices as _validate_non_monotonic_prices
115from ._engine_validation import _validate_null_fraction as _validate_null_fraction
116from ._engine_validation import _validate_positive_prices as _validate_positive_prices
117from ._engine_validation import _validate_required_date_columns as _validate_required_date_columns
118from ._engine_validation import _validate_shape_and_column_sets as _validate_shape_and_column_sets
119from ._engine_validation import _warn_short_sliding_window_data as _warn_short_sliding_window_data
121# ---------------------------------------------------------------------------
122# Re-export config symbols so ``from basanos.math.optimizer import …`` keeps
123# working for existing callers.
124# ---------------------------------------------------------------------------
125__all__ = [
126 "BasanosConfig",
127 "BasanosEngine",
128 "CovarianceConfig",
129 "CovarianceMode",
130 "EwmaShrinkConfig",
131 "SlidingWindowConfig",
132]
135@dataclasses.dataclass(frozen=True)
136class BasanosEngine(_BatchCore, _DiagnosticsMixin, _PerformanceMixin):
137 """Engine to compute correlation matrices and optimize risk positions.
139 Encapsulates price data and configuration to build EWM-based
140 correlations, apply shrinkage, and solve for normalized positions.
142 Public methods are organised into clearly delimited sections (some
143 inherited from the private mixin classes):
145 * **Core data access** — `assets`, `ret_adj`, `vola`, `cor`, `cor_tensor`
146 * **Solve / position logic** — `cash_position`, `position_status`,
147 `risk_position`, `position_leverage`, `warmup_state`
148 * **Portfolio and performance** — `portfolio`, `naive_sharpe`,
149 `sharpe_at_shrink`, `sharpe_at_window_factors`
150 * **Matrix diagnostics** — `condition_number`, `effective_rank`,
151 `solver_residual`, `signal_utilisation`
152 * **Signal evaluation** — `ic(h)`, `rank_ic(h)`, `ic_mean(h)`, `ic_std(h)`,
153 `icir(h)`, `rank_ic_mean(h)`, `rank_ic_std(h)` (``h`` defaults to 1)
154 * **Reporting** — `config_report`
156 Data-flow diagram
157 -----------------
159 .. code-block:: text
161 prices (pl.DataFrame)
162 │
163 ├─ vol_adj ──► ret_adj (volatility-adjusted log returns)
164 │ │
165 │ ├─ ewm_covariance ──► cor / cor_tensor
166 │ │ │
167 │ │ └─ shrink2id / FactorModel
168 │ │ │
169 │ vola covariance matrix
170 │ │ │
171 └── mu ──────────┴── _iter_solve ──────────┘
172 │
173 cash_position
174 │
175 ┌────────┴────────┐
176 portfolio diagnostics
177 (Portfolio) (condition_number,
178 effective_rank,
179 solver_residual,
180 signal_utilisation,
181 ic, rank_ic, …)
183 Attributes:
184 prices: Polars DataFrame of price levels per asset over time. Must
185 contain a ``'date'`` column and at least one numeric asset column
186 with strictly positive values that are not monotonically
187 non-decreasing or non-increasing (i.e. they must vary in sign).
188 mu: Polars DataFrame of expected-return signals aligned with *prices*.
189 Must share the same shape and column names as *prices*.
190 cfg: Immutable `BasanosConfig` controlling EWMA half-lives,
191 clipping, shrinkage intensity, and AUM.
193 Examples:
194 Build an engine with two synthetic assets over 30 days and inspect the
195 optimized positions and diagnostic properties.
197 >>> import numpy as np
198 >>> import polars as pl
199 >>> from basanos.math import BasanosConfig, BasanosEngine
200 >>> dates = list(range(30))
201 >>> rng = np.random.default_rng(42)
202 >>> prices = pl.DataFrame({
203 ... "date": dates,
204 ... "A": np.cumprod(1 + rng.normal(0.001, 0.02, 30)) * 100.0,
205 ... "B": np.cumprod(1 + rng.normal(0.001, 0.02, 30)) * 150.0,
206 ... })
207 >>> mu = pl.DataFrame({
208 ... "date": dates,
209 ... "A": rng.normal(0.0, 0.5, 30),
210 ... "B": rng.normal(0.0, 0.5, 30),
211 ... })
212 >>> cfg = BasanosConfig(vola=5, corr=10, clip=2.0, shrink=0.5, aum=1_000_000)
213 >>> engine = BasanosEngine(prices=prices, mu=mu, cfg=cfg)
214 >>> engine.assets
215 ['A', 'B']
216 >>> engine.cash_position.shape
217 (30, 3)
218 >>> engine.position_leverage.columns
219 ['date', 'leverage']
220 """
222 # ------------------------------------------------------------------
223 # Fields and validation — inherited from _BatchCore
224 # ------------------------------------------------------------------
225 # (prices, mu, cfg, __post_init__ -> _validate_inputs)
226 # Defined in _engine_base.py, which BasanosStream also builds on.
228 # ------------------------------------------------------------------
229 # Core data-access properties — inherited from _CoreDataMixin
230 # ------------------------------------------------------------------
231 # (assets, ret_adj, vola, cor, cor_tensor)
232 # Implementations live in _engine_core.py.
234 # ------------------------------------------------------------------
235 # Internal solve helpers — inherited from _SolveMixin
236 # ------------------------------------------------------------------
237 # (_compute_mask, _check_signal, _scale_to_cash, _row_early_check,
238 # _denom_guard_yield, _compute_position, _replay_positions,
239 # _iter_matrices, _iter_solve, warmup_state)
240 # Implementations live in _engine_solve.py; patch targets remain in that
241 # module's namespace, e.g. ``patch("basanos.math._engine_solve.solve")``.
243 # ------------------------------------------------------------------
244 # Position properties
245 # ------------------------------------------------------------------
247 @property
248 def cash_position(self) -> pl.DataFrame:
249 r"""Optimize correlation-aware risk positions for each timestamp.
251 Supports two covariance modes controlled by ``cfg.covariance_config``:
253 * `EwmaShrinkConfig` (default): Computes EWMA correlations, applies
254 linear shrinkage toward the identity, and solves a normalised linear
255 system $C\,x = \mu$ per timestamp via Cholesky / LU.
257 * `SlidingWindowConfig`: At each timestamp uses the
258 ``cfg.covariance_config.window`` most recent vol-adjusted returns to fit a
259 rank-``cfg.covariance_config.n_factors`` factor model via truncated SVD and
260 solves the system via the Woodbury identity at $O(k^3 + kn)$ rather
261 than $O(n^3)$ per step.
263 Non-finite or ill-posed cases yield zero positions for safety.
265 Returns:
266 pl.DataFrame: DataFrame with columns ['date'] + asset names containing
267 the per-timestamp cash positions (risk divided by EWMA volatility).
269 Performance:
270 For ``ewma_shrink``: dominant cost is ``self.cor`` (O(T·N²) time,
271 O(T·N²) memory). The per-timestamp
272 linear solve adds O(N³) per row.
274 For ``sliding_window``: O(T·W·N·k) for sliding SVDs plus
275 O(T·(k³ + kN)) for Woodbury solves. Memory is O(W·N) per step,
276 independent of T.
277 """
278 assets = self.assets
280 # Compute risk positions row-by-row using _replay_positions.
281 prices_num = self.prices.select(assets).to_numpy()
283 risk_pos_np = np.full_like(prices_num, fill_value=np.nan, dtype=float)
284 cash_pos_np = np.full_like(prices_num, fill_value=np.nan, dtype=float)
285 vola_np = self.vola.select(assets).to_numpy()
287 self._replay_positions(risk_pos_np, cash_pos_np, vola_np)
289 # Build Polars DataFrame for cash positions (numeric columns only)
290 cash_position = self.prices.with_columns(
291 [(pl.lit(cash_pos_np[:, i]).alias(asset)) for i, asset in enumerate(assets)]
292 )
294 return cash_position
296 @property
297 def position_status(self) -> pl.DataFrame:
298 """Per-timestamp reason code explaining each `cash_position` row.
300 Labels every row with exactly one of four `SolveStatus`
301 codes (which compare equal to their string equivalents):
303 * ``'warmup'``: Insufficient history for the sliding-window
304 covariance mode (``i + 1 < cfg.covariance_config.window``).
305 Positions are ``NaN`` for all assets at this timestamp.
306 * ``'zero_signal'``: The expected-return vector ``mu`` was
307 all-zeros (or all-NaN) at this timestamp; the optimizer
308 short-circuited and returned zero positions without solving.
309 * ``'degenerate'``: The normalisation denominator was non-finite
310 or below ``cfg.denom_tol``, the Cholesky / Woodbury solve
311 failed, or no asset had a finite price; positions were zeroed
312 for safety.
313 * ``'valid'``: The linear system was solved successfully and
314 positions are non-trivially non-zero.
316 The codes map one-to-one onto the three NaN / zero cases
317 described in the issue and allow downstream consumers (backtests,
318 risk monitors) to distinguish data gaps from signal silence from
319 numerical ill-conditioning without re-inspecting ``mu`` or the
320 engine configuration.
322 Returns:
323 pl.DataFrame: Two-column DataFrame ``{'date': ..., 'status': ...}``
324 with one row per timestamp. The ``status`` column has
325 ``Polars`` dtype ``String``.
326 """
327 statuses = [status for _i, _t, _mask, _pos, status in self._iter_solve()]
328 return pl.DataFrame({"date": self.prices["date"], "status": pl.Series(statuses, dtype=pl.String)})
330 @property
331 def risk_position(self) -> pl.DataFrame:
332 """Risk positions (before EWMA-volatility scaling) at each timestamp.
334 Derives the un-volatility-scaled position by multiplying the cash
335 position by the per-asset EWMA volatility. Equivalently, this is
336 the quantity solved by the correlation-adjusted linear system before
337 dividing by ``vola``.
339 Relationship to other properties::
341 cash_position = risk_position / vola
342 risk_position = cash_position * vola
344 Returns:
345 pl.DataFrame: DataFrame with columns ``['date'] + assets`` where
346 each value is ``cash_position_i * vola_i`` at the given timestamp.
347 """
348 assets = self.assets
349 cp_np = self.cash_position.select(assets).to_numpy()
350 vola_np = self.vola.select(assets).to_numpy()
351 with np.errstate(invalid="ignore"):
352 risk_pos = cp_np * vola_np
353 return self.prices.with_columns([pl.lit(risk_pos[:, i]).alias(asset) for i, asset in enumerate(assets)])
355 @property
356 def position_leverage(self) -> pl.DataFrame:
357 """L1 norm of cash positions (gross leverage) at each timestamp.
359 Sums the absolute values of all asset cash positions at each row.
360 NaN positions are treated as zero (they contribute nothing to gross
361 leverage).
363 Returns:
364 pl.DataFrame: Two-column DataFrame ``{'date': ..., 'leverage': ...}``
365 where ``leverage`` is the L1 norm of the cash-position vector.
366 """
367 assets = self.assets
368 cp_np = self.cash_position.select(assets).to_numpy()
369 leverage = np.nansum(np.abs(cp_np), axis=1)
370 return pl.DataFrame({"date": self.prices["date"], "leverage": pl.Series(leverage, dtype=pl.Float64)})
372 # ------------------------------------------------------------------
373 # Portfolio and performance
374 # ------------------------------------------------------------------
376 @property
377 def portfolio(self) -> Portfolio:
378 """Construct a Portfolio from the optimized cash positions.
380 Converts the computed cash positions into a Portfolio using the
381 configured AUM. The ``cost_per_unit`` from `cfg` is forwarded
382 so that `net_cost_nav` and
383 `position_delta_costs` work out
384 of the box without any further configuration.
386 Returns:
387 Portfolio: Instance built from cash positions with AUM scaling.
388 """
389 cp = self.cash_position
390 assets = [c for c in cp.columns if c != "date" and cp[c].dtype.is_numeric()]
391 scaled = cp.with_columns(pl.col(a) * self.cfg.position_scale for a in assets)
392 return Portfolio.from_cash_position(self.prices, scaled, aum=self.cfg.aum, cost_per_unit=self.cfg.cost_per_unit)
394 # ------------------------------------------------------------------
395 # Performance sweeps — inherited from _PerformanceMixin
396 # ------------------------------------------------------------------
397 # (sharpe_at_shrink, sharpe_at_window_factors, naive_sharpe)
398 # Implementations live in _engine_performance.py.
400 # ------------------------------------------------------------------
401 # Reporting
402 # ------------------------------------------------------------------
404 @property
405 def config_report(self) -> "ConfigReport":
406 """Return a `ConfigReport` facade for this engine.
408 Returns a `ConfigReport` that
409 includes the full **lambda-sweep chart** — an interactive plot of the
410 annualised Sharpe ratio as `shrink` (λ) is swept
411 across [0, 1] — in addition to the parameter table, shrinkage-guidance
412 table, and theory section available from
413 `report`.
415 Returns:
416 basanos.math._config_report.ConfigReport: Report facade with
417 ``to_html()`` and ``save()`` methods.
419 Examples:
420 >>> import numpy as np
421 >>> import polars as pl
422 >>> from basanos.math.optimizer import BasanosConfig, BasanosEngine
423 >>> dates = pl.Series("date", list(range(200)))
424 >>> rng = np.random.default_rng(0)
425 >>> prices = pl.DataFrame({"date": dates, "A": rng.lognormal(size=200), "B": rng.lognormal(size=200)})
426 >>> mu = pl.DataFrame({"date": dates, "A": rng.normal(size=200), "B": rng.normal(size=200)})
427 >>> cfg = BasanosConfig(vola=10, corr=20, clip=3.0, shrink=0.5, aum=1e6)
428 >>> engine = BasanosEngine(prices=prices, mu=mu, cfg=cfg)
429 >>> report = engine.config_report
430 >>> html = report.to_html()
431 >>> "Lambda" in html
432 True
433 """
434 return ConfigReport(config=self.cfg, engine=self)
436 # ------------------------------------------------------------------
437 # Matrix diagnostics — inherited from _DiagnosticsMixin
438 # ------------------------------------------------------------------
439 # (condition_number, effective_rank, solver_residual, signal_utilisation)
440 # Implementations live in _engine_diagnostics.py; patch targets remain in
441 # that module's namespace, e.g.
442 # ``patch("basanos.math._engine_diagnostics.solve")``.
444 # ------------------------------------------------------------------
445 # Signal evaluation — inherited from _SignalEvaluatorMixin
446 # ------------------------------------------------------------------
447 # (_ic_series, ic, rank_ic, ic_mean, ic_std, icir,
448 # rank_ic_mean, rank_ic_std)
449 # Implementations live in _engine_ic.py.