Use AdaptGrow, a GPU-accelerated matrix factorization algorithm, to show rolling correlation and tail-dependence matrices into exhausting clusters, delicate issue loadings, and structural-break indicators at single-GPU and multi-node scale
Quant methods routinely group devices for portfolio building, danger aggregation, statistical arbitrage, and commerce surveillance. Incorrect groupings could make concentrated positions seem diversified, obscure danger shared throughout nominal boundaries, and choose statistical-arbitrage pairs whose relationships fail beneath stress.
The sensible problem is that the proper groupings are neither immediately observable nor secure. Issue exposures drift, devices change classifications, and dependencies can change sharply throughout market stress. A clustering pipeline should due to this fact separate routine variation from structural change and be cheap sufficient to rerun as new returns arrive.
There are two frequent methods to group devices from a dependence matrix. Onerous clustering strategies are computationally low-cost however assign each instrument to precisely one group, which breaks down at sector boundaries and masks the graded exposures that matter for danger budgeting. Comfortable factorization strategies like SymNMF deal with boundary devices and produce usable issue loadings, however their dense matrix aims have traditionally restricted sensible use to reasonable instrument counts slightly than the size at which this downside truly lives.
This publish covers a workflow that addresses each limitations. The workflow begins with rolling return home windows and constructs two complementary inputs: absolute Pearson correlation for broad co-movement and the tail pairwise dependence matrix (TPDM) for joint habits throughout excessive observations. SymNMF represents every instrument by a row of nonnegative issue loadings. Retaining the row gives a delicate illustration; taking its argmax produces a tough label.
A memory-efficient SymNMF formulation reduces peak storage from ~20n2 to ~4n2 bytes, which is what makes ~100,000 devices match on a single NVIDIA GB200. For bigger issues, a distributed implementation row-shards the dependence matrix and reduces communication to O(nk) slightly than O(n2), enabling factorization of 1 million devices throughout 16 nodes. A single adaptive solver, AdaptGrow, handles each correlation and tail-dependence inputs by studying the eigenspectrum to decide on between full-batch and block-stochastic gradients, eliminating the necessity to choose or tune separate solvers for various enter buildings.
The result’s a clustering pipeline that produces exhausting labels, delicate issue loadings, and structural-break indicators, reruns cheaply as new returns arrive, and scales from a single GPU to multi-node infrastructure with out altering the solver interface.
A companion pocket book, linked beneath, implements the total pipeline and reproduces all outcomes on this publish.
Factorization at scale
Scale is constrained first by reminiscence. A dense FP32 dependence matrix requires ~40 GB for 100,000 devices and ~4 TB for 1 million devices. A naive SymNMF implementation additionally materializes a number of extra n x n intermediates. The trace-based formulation used right here eliminates these intermediates, decreasing estimated peak storage from roughly 20n2 bytes to 4n2 bytes plus smaller issue buffers. This variation is what makes roughly 100,000 devices match on one high-memory GPU.
NVIDIA acceleration enters at every stage of the pipeline. PyTorch dispatches the dominant SH matrix multiplications to cuBLAS. cuSOLVER performs the spectral probe used for rank and solver choice. cuDF retains non-obligatory Parquet ingestion and preprocessing on the GPU. For scale-out, PyTorch Distributed row-shards S whereas preserving a reproduction of H on every employee. NCCL all-gathers the row-sharded S H merchandise and all-reduces the gradients, so communication operates on O(nk) knowledge slightly than the total O(n2) matrix. The setting is packaged with an NVIDIA NGC PyTorch container and cudf-cu13.
Within the companion paper, the 100,000-instrument matrix was distributed throughout 4 NVIDIA GB200 GPUs for quicker execution, though its 40 GB enter suits on one GB200. Throughout three seeds in FP32, AdaptGrow converged in 13.0 seconds on correlation and 12.4 seconds on TPDM. At 1 million devices, the 4 TB matrix was row-sharded throughout 64 GB200 GPUs on 16 nodes; full-batch AdaGrad accomplished the correlation factorization in roughly 2 minutes, whereas AdaptGrow accomplished the TPDM factorization in roughly 4 minutes. These are particular person factorization measurements, not end-to-end timings for all 250 temporal home windows.
The temporal setup
The workflow evaluates 250 rolling home windows, approximating each day re-clustering over one buying and selling yr. The artificial return stream comprises two managed occasions: devices altering their planted group membership and a number of other teams experiencing a joint tail-stress episode with out altering membership.
This managed setup verifies two completely different behaviors. Adjusted Rand index (ARI) ought to establish the membership change, whereas TPDM ought to expose the co-crash that unusual correlation largely misses. For manufacturing use, change the artificial generator with a returns desk whereas preserving the identical windowing, dependence-estimation, factorization, and monitoring levels.
The million-instrument outcomes are separate distributed scale assessments and require infrastructure akin to the printed 16-node configuration.
Selecting the rank ok
Begin by inspecting the main eigenvalues from an preliminary consultant window. Select ok on the clearest separation between sign eigenvalues and the noise ground, then hold ok fastened throughout subsequent home windows in order that stability scores stay comparable. The artificial knowledge used right here has a planted rank of 24. Manufacturing knowledge might not comprise a pointy hole, so rank choice must also be checked in opposition to cluster interpretability and stability.
Factorizing with SymNMF
For every rolling window, the workflow passes the dependence matrix S and chosen rank ok to AdaptGrow. The solver returns H, the place every row of H comprises an instrument’s delicate issue loadings, and taking the row-wise argmax produces a tough cluster label.
Every window makes use of the identical fixed-seed initialization and is fitted independently slightly than warm-started, stopping earlier labels from masking a real reclassification. AdaptGrow is a single adaptive solver that auto-configures from the matrix’s eigenvalue spectrum. Each regimes use the identical per-coordinate AdaGrad preconditioner and differ solely in how the gradient is computed. AdaptGrow seeds its batch fraction from the post-rank eigenvalue hole described in additional element beneath.
A clear hole selects the total gradient. A flatter post-rank spectrum begins with a less expensive block-sampled gradient corrected utilizing Stochastic Variance Lowered Gradient (SVRG), then expands the pattern towards the total gradient if progress stalls. The identical solver runs unchanged from one GPU to many.
Mathematically, the diagonal AdaGrad replace is outlined as:
(G leftarrow G + g odot g)
(H leftarrow max left(H – eta cdot g / (sqrt{G} + varepsilon), ; 0right))
the place g is both the total gradient ∇f(H) (full-batch department) or a block-sampled estimate of it (stochastic department), with all operations above being element-wise.
Why this solver
After deciding on the factorization rank ok, AdaptGrow examines the post-rank hole ratio (gamma_{ok+1} = lambda_{ok+1} / |lambda_{ok+2}|). The additional eigenvalue permits for a standard issue, resembling a market issue alongside the group-level elements. Within the companion experiments, a big post-rank hole corresponded to shorter correlation-matrix runs, for which full-batch AdaGrad was most effective. A flatter spectrum corresponded to longer TPDM runs, the place lower-cost block-sampled gradients have been extra helpful. AdaptGrow due to this fact begins with sampled SVRG updates when the hole is small and will increase the sampled fraction towards the total matrix when progress stalls. The brink of 5 used right here is an empirical setting from these experiments, not a common statistical cutoff. AdaptGrow algorithm sketch:
# gamma_{ok+1} >= 5 selects the total gradient; in any other case begin sampled
def adaptgrow(S, ok, lr, phi=None, steps=2000, eps=1e-8):
phi = phi if phi will not be None else seed_from_eigenspectrum(S, ok)
# fixed-seed init, impartial per window (no warm-start)
H = scale_matched_init(S, ok)
# diagonal (per-coordinate) AdaGrad accumulator
G = torch.zeros_like(H)
for t in vary(steps):
# clear post-rank hole
if phi >= 1.0:
g = 4 * (H @ (H.T @ H) – S @ H)
# flat post-rank spectrum
else:
g = block_svrg_grad(S, H, phi)
# similar diagonal AdaGrad replace both means
G += g * g
# projected step
H = (H – lr * g / (G.sqrt() + eps)).clamp_min(0)
# develop sampled fraction towards full gradient
if stagnating() and phi < 1.0:
phi = min(2 * phi, 1.0)
return H
Spherical k-means for direct exhausting clustering
When just one label per instrument is required, spherical k-means gives a lower-cost baseline. It clusters the L2-normalized rows of S utilizing cosine similarity, making it the suitable comparability for SymNMF on this dependence geometry. The companion paper establishes the formal relationship between the 2 aims. In follow, use spherical k-means for well-separated exhausting clusters and SymNMF when delicate issue loadings or boundary devices matter.
Monitoring stability by time with the Rand index
The identical pipeline runs on two dependence matrices that share the latent construction however measure various things:
Correlation captures co-movement throughout the total return distribution, pushed by the physique of the info.
TPDM captures co-movement conditional on excessive occasions, which is what drives drawdowns and joint tail danger.
With ok fastened and every window factored independently, the examine is straightforward: issue each St, hard-label by argmax on H, and ask how the labels change because the window slides.
The metric
Since cluster labels are arbitrary throughout runs, pairs of devices have been in contrast as an alternative. The ARI scores how usually two clusterings put the identical pair collectively or aside, giving 1 for an identical clusterings and about 0 for unrelated ones.
Stability and break detection
ARI(t-Δ, t) is tracked, evaluating every window with the one a full window width earlier, the place Δ = 50 steps is that width. Consecutive home windows overlap by all however one stride, so a reclassification enters progressively and barely strikes the step-to-step ARI(t-1, t); spacing the comparability a full width aside lets the gathered change register as an actual dip.
On this artificial experiment, each curves sit excessive throughout the calm interval (baseline ARI ≈ 0.93 for correlation and ≈ 0.80 for the noisier, tail-sensitive TPDM), then drop sharply when the window crosses the reclassification occasion earlier than recovering.
To show that dip into an alert, the workflow overlays a self-calibrating 3σ management restrict: it calibrates on the calm pre-break home windows and flags any drop beneath the restrict, with none pre-set threshold.
Why have completely different tail and physique estimators? A co-crash adjustments co-movement, not group membership, so the relabeling metric doesn’t register it. The exhausting labels and the correlation curve keep unmoved, and by correlation the confused sectors nonetheless look diversified.
To detect it, the cross-sector dependence is measured amongst these sectors immediately, in every matrix, over time. On the disaster peak, correlation reads simply ≈ 0.04 whereas the TPDM reads ≈ 0.13, a number of occasions larger, as a result of the identical sectors share tail dependence that correlation by no means registers. Off the diagonal, the correlation block stays darkish whereas the TPDM’s stressed-sector block lights up. On this artificial episode, the co-crash is muted in correlation however seen within the TPDM’s cross-sector dependence, although the exhausting cluster labels don’t change.
Between spherical k-means and SymNMF
On this artificial experiment, the 2 strategies get better the identical broad construction however differ on a small, vital subset of devices. On the well-separated correlation matrix they attain a cross-method ARI of about 0.83, shut sufficient that spherical k-means is an affordable approximation however not interchangeable with SymNMF. About 9% of devices sit at a boundary between clusters, which exhausting argmax should assign to a single group whereas the delicate factorization retains them cut up throughout each.
The 2 strategies diverge additional because the eigenvalue spectrum collapses towards a single dominant issue (the near-rank-1 TPDM regime the paper research at scale). There each row of S aligns with almost the identical main path, so devices are now not separable by angle and a tough spherical partition is unstable.
SymNMF retains graded issue loadings in H, which is the helpful output when exhausting cluster labels are now not nicely outlined. Spherical k-means is due to this fact the computationally extra environment friendly alternative when sectors are angularly well-separated and exhausting clusters are sufficient, whereas SymNMF gives outcomes interpretable each as delicate and exhausting clustering.
Get began with GPU-accelerated instrument clustering
SymNMF’s dense goal beforehand restricted sensible implementations to reasonable matrix sizes. The memory-efficient GPU implementation extends that capability to roughly 100,000 devices on one NVIDIA GB200 GPU and 1 million devices throughout a number of nodes. AdaptGrow makes use of the eigenspectrum to pick out both full-batch AdaGrad or lower-cost block-stochastic updates, enabling the identical workflow to adapt to clean-gap and flat-spectrum inputs.
The workflow applies this pipeline throughout 250 artificial home windows utilizing each SymNMF and its matched spherical k-means baseline. The ensuing delicate loadings, exhausting labels, and stability diagnostics can assist statistical arbitrage, momentum indicators, market-neutral portfolio building, publicity management, and danger budgeting whereas figuring out structural breaks.
Pocket book
The companion pocket book reproduces all outcomes on this publish, together with:
Producing the artificial return stream
Establishing rolling correlation and TPDM matrices
Choosing the factorization rank
Operating SymNMF and spherical k-means
Deriving exhausting and delicate cluster outputs
Calculating adjusted Rand index stability scores
Detecting the planted structural break
Reproducing the three figures on this publish
Run clustering_through_time.ipynb finish to finish—rolling St, impartial SymNMF suits, argmax labels, and the soundness, tail-risk, and k-means figures. Deploy on construct.nvidia.com with NVIDIA Brev, or by yourself GPU from the repository.
Stack
PyTorch + cuDF: cuBLAS for the S·H GEMMs, cuSOLVER for the spectral probe, NCCL for the distributed runner; cuDF handles on-GPU Parquet ingest (non-obligatory, with a pandas fallback). One container: an NGC PyTorch picture plus cudf-cu13.
Scale-out
Distributed PyTorch and NCCL implementations of each algorithms function on a row-sharded S. AdaptGrow has been validated on as much as 64 NVIDIA GB200 GPUs throughout 16 nodes. The repository’s scripts/run_distributed.py configures multi-node execution with torchrun or Slurm; deployment directions are offered in scripts/README.md.
Study extra
Companion technical paper (SymNMF derivation, spectral rank choice, and GPU speed-up ladder): Low-Rank Dependence Decomposition by way of Accelerated Symmetric Non-negative Matrix Factorization
Supply repository and clustering_through_time.ipynb pocket book: NVIDIA/SymNMF-factors
