Coverage for src/basanos/math/optimizer.py: 100%
64 statements
« prev ^ index » next coverage.py v7.14.1, created at 2026-08-04 07:53 +0000
1"""Correlation-aware risk position optimizer (Basanos).
3This module provides utilities to compute correlation-adjusted risk positions
4from price data and expected-return signals. It relies on volatility-adjusted
5returns to estimate a dynamic correlation matrix (via EWM), applies shrinkage
6towards identity, and solves a normalized linear system per timestamp to
7obtain stable positions.
9Performance characteristics
10---------------------------
11Let *N* be the number of assets and *T* the number of timestamps.
13**Computational complexity**
15+----------------------------------+------------------+--------------------------------------+
16| Operation | Complexity | Bottleneck |
17+==================================+==================+======================================+
18| EWM volatility (``ret_adj``, | O(T·N) | Linear in both T and N; negligible |
19| ``vola``) | | |
20+----------------------------------+------------------+--------------------------------------+
21| EWM correlation (``cor``) | O(T·N²) | ``ewm_covariance`` from |
22| | | ``cvx.linalg`` over all N² pairs |
23+----------------------------------+------------------+--------------------------------------+
24| Linear solve per timestamp | O(N³) | Cholesky / LU per row in |
25| (``cash_position``) | * T solves | ``cash_position`` |
26+----------------------------------+------------------+--------------------------------------+
28**Memory usage** (peak, approximate)
30``ewm_covariance`` from ``cvx.linalg`` processes the input Polars DataFrame
31and returns a dict of covariance matrices. Peak RAM ≈ **O(T · N²)** bytes.
32Typical working sizes on a 16 GB machine:
34+--------+--------------------------+------------------------------------+
35| N | T (daily rows) | Peak memory (approx.) |
36+========+==========================+====================================+
37| 50 | 252 (~1 yr) | ~70 MB |
38+--------+--------------------------+------------------------------------+
39| 100 | 252 (~1 yr) | ~280 MB |
40+--------+--------------------------+------------------------------------+
41| 100 | 2 520 (~10 yr) | ~2.8 GB |
42+--------+--------------------------+------------------------------------+
43| 200 | 2 520 (~10 yr) | ~11 GB |
44+--------+--------------------------+------------------------------------+
45| 500 | 2 520 (~10 yr) | ~70 GB ⚠ exceeds typical RAM |
46+--------+--------------------------+------------------------------------+
48**Practical limits (daily data)**
50* **≤ 150 assets, ≤ 5 years** — well within reach on an 8 GB laptop.
51* **≤ 250 assets, ≤ 10 years** — requires ~11-12 GB; feasible on a 16 GB
52 workstation.
53* **> 500 assets with multi-year history** — peak memory exceeds 16 GB;
54 reduce the time range or switch to a chunked / streaming approach.
55* **> 1 000 assets** — the O(N³) per-solve cost alone makes real-time
56 optimization impractical even with adequate RAM.
58See ``BENCHMARKS.md`` for measured wall-clock timings across representative
59dataset sizes.
61Internal structure
62------------------
63The implementation is split across focused private modules to keep each file
64readable and independently testable:
66* `_config` — `BasanosConfig` and all
67 covariance-mode configuration classes.
68* `_engine_validation` — free functions that validate the
69 ``prices`` / ``mu`` / ``cfg`` inputs (re-exported here).
70* `_engine_core` — the `_CoreDataMixin` providing the
71 core data-access properties (``assets``, ``ret_adj``, ``vola``, ``cor``,
72 ``cor_tensor``).
73* `_engine_solve` — private helpers providing the
74 ``_iter_matrices`` and ``_iter_solve`` generators (per-timestamp solve
75 logic).
76* `_engine_diagnostics` — private helpers providing
77 matrix-quality diagnostics (condition number, effective rank, solver
78 residual, signal utilisation).
79* `_engine_performance` — the `_PerformanceMixin` providing the
80 portfolio-Sharpe sweep helpers (``sharpe_at_shrink``,
81 ``sharpe_at_window_factors``, ``naive_sharpe``).
82* `_engine_ic` — private helpers providing signal
83 evaluation metrics (IC, Rank IC, ICIR, and summary statistics).
84* This module — `BasanosEngine`, a single flat class that wires
85 every method together in clearly delimited sections.
86"""
88import dataclasses
90import numpy as np
91import polars as pl
92from jquantstats import Portfolio
94from ._config import (
95 BasanosConfig,
96 CovarianceConfig,
97 CovarianceMode,
98 EwmaShrinkConfig,
99 SlidingWindowConfig,
100)
101from ._config_report import ConfigReport
102from ._engine_core import _CoreDataMixin as _CoreDataMixin
103from ._engine_diagnostics import _DiagnosticsMixin as _DiagnosticsMixin
104from ._engine_ic import _SignalEvaluatorMixin as _SignalEvaluatorMixin
105from ._engine_performance import _PerformanceMixin as _PerformanceMixin
106from ._engine_solve import _SolveMixin as _SolveMixin
107from ._engine_validation import _numeric_assets as _numeric_assets
108from ._engine_validation import _validate_inputs as _validate_inputs
109from ._engine_validation import _validate_non_monotonic_prices as _validate_non_monotonic_prices
110from ._engine_validation import _validate_null_fraction as _validate_null_fraction
111from ._engine_validation import _validate_positive_prices as _validate_positive_prices
112from ._engine_validation import _validate_required_date_columns as _validate_required_date_columns
113from ._engine_validation import _validate_shape_and_column_sets as _validate_shape_and_column_sets
114from ._engine_validation import _warn_short_sliding_window_data as _warn_short_sliding_window_data
116# ---------------------------------------------------------------------------
117# Re-export config symbols so ``from basanos.math.optimizer import …`` keeps
118# working for existing callers.
119# ---------------------------------------------------------------------------
120__all__ = [
121 "BasanosConfig",
122 "BasanosEngine",
123 "CovarianceConfig",
124 "CovarianceMode",
125 "EwmaShrinkConfig",
126 "SlidingWindowConfig",
127]
130@dataclasses.dataclass(frozen=True)
131class BasanosEngine(_CoreDataMixin, _DiagnosticsMixin, _PerformanceMixin, _SignalEvaluatorMixin, _SolveMixin):
132 """Engine to compute correlation matrices and optimize risk positions.
134 Encapsulates price data and configuration to build EWM-based
135 correlations, apply shrinkage, and solve for normalized positions.
137 Public methods are organised into clearly delimited sections (some
138 inherited from the private mixin classes):
140 * **Core data access** — `assets`, `ret_adj`, `vola`, `cor`, `cor_tensor`
141 * **Solve / position logic** — `cash_position`, `position_status`,
142 `risk_position`, `position_leverage`, `warmup_state`
143 * **Portfolio and performance** — `portfolio`, `naive_sharpe`,
144 `sharpe_at_shrink`, `sharpe_at_window_factors`
145 * **Matrix diagnostics** — `condition_number`, `effective_rank`,
146 `solver_residual`, `signal_utilisation`
147 * **Signal evaluation** — `ic(h)`, `rank_ic(h)`, `ic_mean(h)`, `ic_std(h)`,
148 `icir(h)`, `rank_ic_mean(h)`, `rank_ic_std(h)` (``h`` defaults to 1)
149 * **Reporting** — `config_report`
151 Data-flow diagram
152 -----------------
154 .. code-block:: text
156 prices (pl.DataFrame)
157 │
158 ├─ vol_adj ──► ret_adj (volatility-adjusted log returns)
159 │ │
160 │ ├─ ewm_covariance ──► cor / cor_tensor
161 │ │ │
162 │ │ └─ shrink2id / FactorModel
163 │ │ │
164 │ vola covariance matrix
165 │ │ │
166 └── mu ──────────┴── _iter_solve ──────────┘
167 │
168 cash_position
169 │
170 ┌────────┴────────┐
171 portfolio diagnostics
172 (Portfolio) (condition_number,
173 effective_rank,
174 solver_residual,
175 signal_utilisation,
176 ic, rank_ic, …)
178 Attributes:
179 prices: Polars DataFrame of price levels per asset over time. Must
180 contain a ``'date'`` column and at least one numeric asset column
181 with strictly positive values that are not monotonically
182 non-decreasing or non-increasing (i.e. they must vary in sign).
183 mu: Polars DataFrame of expected-return signals aligned with *prices*.
184 Must share the same shape and column names as *prices*.
185 cfg: Immutable `BasanosConfig` controlling EWMA half-lives,
186 clipping, shrinkage intensity, and AUM.
188 Examples:
189 Build an engine with two synthetic assets over 30 days and inspect the
190 optimized positions and diagnostic properties.
192 >>> import numpy as np
193 >>> import polars as pl
194 >>> from basanos.math import BasanosConfig, BasanosEngine
195 >>> dates = list(range(30))
196 >>> rng = np.random.default_rng(42)
197 >>> prices = pl.DataFrame({
198 ... "date": dates,
199 ... "A": np.cumprod(1 + rng.normal(0.001, 0.02, 30)) * 100.0,
200 ... "B": np.cumprod(1 + rng.normal(0.001, 0.02, 30)) * 150.0,
201 ... })
202 >>> mu = pl.DataFrame({
203 ... "date": dates,
204 ... "A": rng.normal(0.0, 0.5, 30),
205 ... "B": rng.normal(0.0, 0.5, 30),
206 ... })
207 >>> cfg = BasanosConfig(vola=5, corr=10, clip=2.0, shrink=0.5, aum=1_000_000)
208 >>> engine = BasanosEngine(prices=prices, mu=mu, cfg=cfg)
209 >>> engine.assets
210 ['A', 'B']
211 >>> engine.cash_position.shape
212 (30, 3)
213 >>> engine.position_leverage.columns
214 ['date', 'leverage']
215 """
217 prices: pl.DataFrame
218 mu: pl.DataFrame
219 cfg: BasanosConfig
221 def __post_init__(self) -> None:
222 """Validate inputs by delegating to `_validate_inputs`."""
223 _validate_inputs(self.prices, self.mu, self.cfg)
225 # ------------------------------------------------------------------
226 # Core data-access properties — inherited from _CoreDataMixin
227 # ------------------------------------------------------------------
228 # (assets, ret_adj, vola, cor, cor_tensor)
229 # Implementations live in _engine_core.py.
231 # ------------------------------------------------------------------
232 # Internal solve helpers — inherited from _SolveMixin
233 # ------------------------------------------------------------------
234 # (_compute_mask, _check_signal, _scale_to_cash, _row_early_check,
235 # _denom_guard_yield, _compute_position, _replay_positions,
236 # _iter_matrices, _iter_solve, warmup_state)
237 # Implementations live in _engine_solve.py; patch targets remain in that
238 # module's namespace, e.g. ``patch("basanos.math._engine_solve.solve")``.
240 # ------------------------------------------------------------------
241 # Position properties
242 # ------------------------------------------------------------------
244 @property
245 def cash_position(self) -> pl.DataFrame:
246 r"""Optimize correlation-aware risk positions for each timestamp.
248 Supports two covariance modes controlled by ``cfg.covariance_config``:
250 * `EwmaShrinkConfig` (default): Computes EWMA correlations, applies
251 linear shrinkage toward the identity, and solves a normalised linear
252 system $C\,x = \mu$ per timestamp via Cholesky / LU.
254 * `SlidingWindowConfig`: At each timestamp uses the
255 ``cfg.covariance_config.window`` most recent vol-adjusted returns to fit a
256 rank-``cfg.covariance_config.n_factors`` factor model via truncated SVD and
257 solves the system via the Woodbury identity at $O(k^3 + kn)$ rather
258 than $O(n^3)$ per step.
260 Non-finite or ill-posed cases yield zero positions for safety.
262 Returns:
263 pl.DataFrame: DataFrame with columns ['date'] + asset names containing
264 the per-timestamp cash positions (risk divided by EWMA volatility).
266 Performance:
267 For ``ewma_shrink``: dominant cost is ``self.cor`` (O(T·N²) time,
268 O(T·N²) memory). The per-timestamp
269 linear solve adds O(N³) per row.
271 For ``sliding_window``: O(T·W·N·k) for sliding SVDs plus
272 O(T·(k³ + kN)) for Woodbury solves. Memory is O(W·N) per step,
273 independent of T.
274 """
275 assets = self.assets
277 # Compute risk positions row-by-row using _replay_positions.
278 prices_num = self.prices.select(assets).to_numpy()
280 risk_pos_np = np.full_like(prices_num, fill_value=np.nan, dtype=float)
281 cash_pos_np = np.full_like(prices_num, fill_value=np.nan, dtype=float)
282 vola_np = self.vola.select(assets).to_numpy()
284 self._replay_positions(risk_pos_np, cash_pos_np, vola_np)
286 # Build Polars DataFrame for cash positions (numeric columns only)
287 cash_position = self.prices.with_columns(
288 [(pl.lit(cash_pos_np[:, i]).alias(asset)) for i, asset in enumerate(assets)]
289 )
291 return cash_position
293 @property
294 def position_status(self) -> pl.DataFrame:
295 """Per-timestamp reason code explaining each `cash_position` row.
297 Labels every row with exactly one of four `SolveStatus`
298 codes (which compare equal to their string equivalents):
300 * ``'warmup'``: Insufficient history for the sliding-window
301 covariance mode (``i + 1 < cfg.covariance_config.window``).
302 Positions are ``NaN`` for all assets at this timestamp.
303 * ``'zero_signal'``: The expected-return vector ``mu`` was
304 all-zeros (or all-NaN) at this timestamp; the optimizer
305 short-circuited and returned zero positions without solving.
306 * ``'degenerate'``: The normalisation denominator was non-finite
307 or below ``cfg.denom_tol``, the Cholesky / Woodbury solve
308 failed, or no asset had a finite price; positions were zeroed
309 for safety.
310 * ``'valid'``: The linear system was solved successfully and
311 positions are non-trivially non-zero.
313 The codes map one-to-one onto the three NaN / zero cases
314 described in the issue and allow downstream consumers (backtests,
315 risk monitors) to distinguish data gaps from signal silence from
316 numerical ill-conditioning without re-inspecting ``mu`` or the
317 engine configuration.
319 Returns:
320 pl.DataFrame: Two-column DataFrame ``{'date': ..., 'status': ...}``
321 with one row per timestamp. The ``status`` column has
322 ``Polars`` dtype ``String``.
323 """
324 statuses = [status for _i, _t, _mask, _pos, status in self._iter_solve()]
325 return pl.DataFrame({"date": self.prices["date"], "status": pl.Series(statuses, dtype=pl.String)})
327 @property
328 def risk_position(self) -> pl.DataFrame:
329 """Risk positions (before EWMA-volatility scaling) at each timestamp.
331 Derives the un-volatility-scaled position by multiplying the cash
332 position by the per-asset EWMA volatility. Equivalently, this is
333 the quantity solved by the correlation-adjusted linear system before
334 dividing by ``vola``.
336 Relationship to other properties::
338 cash_position = risk_position / vola
339 risk_position = cash_position * vola
341 Returns:
342 pl.DataFrame: DataFrame with columns ``['date'] + assets`` where
343 each value is ``cash_position_i * vola_i`` at the given timestamp.
344 """
345 assets = self.assets
346 cp_np = self.cash_position.select(assets).to_numpy()
347 vola_np = self.vola.select(assets).to_numpy()
348 with np.errstate(invalid="ignore"):
349 risk_pos = cp_np * vola_np
350 return self.prices.with_columns([pl.lit(risk_pos[:, i]).alias(asset) for i, asset in enumerate(assets)])
352 @property
353 def position_leverage(self) -> pl.DataFrame:
354 """L1 norm of cash positions (gross leverage) at each timestamp.
356 Sums the absolute values of all asset cash positions at each row.
357 NaN positions are treated as zero (they contribute nothing to gross
358 leverage).
360 Returns:
361 pl.DataFrame: Two-column DataFrame ``{'date': ..., 'leverage': ...}``
362 where ``leverage`` is the L1 norm of the cash-position vector.
363 """
364 assets = self.assets
365 cp_np = self.cash_position.select(assets).to_numpy()
366 leverage = np.nansum(np.abs(cp_np), axis=1)
367 return pl.DataFrame({"date": self.prices["date"], "leverage": pl.Series(leverage, dtype=pl.Float64)})
369 # ------------------------------------------------------------------
370 # Portfolio and performance
371 # ------------------------------------------------------------------
373 @property
374 def portfolio(self) -> Portfolio:
375 """Construct a Portfolio from the optimized cash positions.
377 Converts the computed cash positions into a Portfolio using the
378 configured AUM. The ``cost_per_unit`` from `cfg` is forwarded
379 so that `net_cost_nav` and
380 `position_delta_costs` work out
381 of the box without any further configuration.
383 Returns:
384 Portfolio: Instance built from cash positions with AUM scaling.
385 """
386 cp = self.cash_position
387 assets = [c for c in cp.columns if c != "date" and cp[c].dtype.is_numeric()]
388 scaled = cp.with_columns(pl.col(a) * self.cfg.position_scale for a in assets)
389 return Portfolio.from_cash_position(self.prices, scaled, aum=self.cfg.aum, cost_per_unit=self.cfg.cost_per_unit)
391 # ------------------------------------------------------------------
392 # Performance sweeps — inherited from _PerformanceMixin
393 # ------------------------------------------------------------------
394 # (sharpe_at_shrink, sharpe_at_window_factors, naive_sharpe)
395 # Implementations live in _engine_performance.py.
397 # ------------------------------------------------------------------
398 # Reporting
399 # ------------------------------------------------------------------
401 @property
402 def config_report(self) -> "ConfigReport":
403 """Return a `ConfigReport` facade for this engine.
405 Returns a `ConfigReport` that
406 includes the full **lambda-sweep chart** — an interactive plot of the
407 annualised Sharpe ratio as `shrink` (λ) is swept
408 across [0, 1] — in addition to the parameter table, shrinkage-guidance
409 table, and theory section available from
410 `report`.
412 Returns:
413 basanos.math._config_report.ConfigReport: Report facade with
414 ``to_html()`` and ``save()`` methods.
416 Examples:
417 >>> import numpy as np
418 >>> import polars as pl
419 >>> from basanos.math.optimizer import BasanosConfig, BasanosEngine
420 >>> dates = pl.Series("date", list(range(200)))
421 >>> rng = np.random.default_rng(0)
422 >>> prices = pl.DataFrame({"date": dates, "A": rng.lognormal(size=200), "B": rng.lognormal(size=200)})
423 >>> mu = pl.DataFrame({"date": dates, "A": rng.normal(size=200), "B": rng.normal(size=200)})
424 >>> cfg = BasanosConfig(vola=10, corr=20, clip=3.0, shrink=0.5, aum=1e6)
425 >>> engine = BasanosEngine(prices=prices, mu=mu, cfg=cfg)
426 >>> report = engine.config_report
427 >>> html = report.to_html()
428 >>> "Lambda" in html
429 True
430 """
431 return ConfigReport(config=self.cfg, engine=self)
433 # ------------------------------------------------------------------
434 # Matrix diagnostics — inherited from _DiagnosticsMixin
435 # ------------------------------------------------------------------
436 # (condition_number, effective_rank, solver_residual, signal_utilisation)
437 # Implementations live in _engine_diagnostics.py; patch targets remain in
438 # that module's namespace, e.g.
439 # ``patch("basanos.math._engine_diagnostics.solve")``.
441 # ------------------------------------------------------------------
442 # Signal evaluation — inherited from _SignalEvaluatorMixin
443 # ------------------------------------------------------------------
444 # (_ic_series, ic, rank_ic, ic_mean, ic_std, icir,
445 # rank_ic_mean, rank_ic_std)
446 # Implementations live in _engine_ic.py.