Skip to content

On-Node Threading and Multi-Core Scaling

MTL5 parallelizes its hot kernels across CPU cores with a small persistent thread pool (no OpenMP/TBB — just the C++ standard concurrency runtime). Threading is off by default and opt-in per process.

Set the environment variable before running:

Terminal window
MTL5_NUM_THREADS=8 ./your_program

MTL5_NUM_THREADS is read once (clamped to the hardware concurrency) and sizes a single process-wide pool. Unset or =1 means fully serial: no worker threads are created and each kernel runs its original serial path (a cheap pool-size check is all the threading layer adds), so the default build behaves as before.

Or size it from code, before anything uses the pool:

#include <mtl/parallel/thread_pool.hpp>
int main(int argc, char** argv) {
mtl::parallel::configure(8); // overrides MTL5_NUM_THREADS
// ...
}

configure(n) is clamped to the hardware concurrency, as the variable is, and returns false without changing anything once the pool exists: after the first mtl::parallel::pool() call or the first kernel that runs threaded. pool().size() is the size in use.

The pool is public API, mtl::parallel::thread_pool, reached through mtl::parallel::pool() (#630). Code that runs its parallel work there shares one set of workers with MTL5’s kernels instead of oversubscribing the cores. The older spelling, mtl::detail::thread_pool::instance() from <mtl/detail/thread_pool.hpp>, still works and names the same pool, unless MTL5_POOL_SPIN selects the spin pool for the kernels (see below).

run(count, task) is the primitive, and its contract is:

  • task(tid) runs once for each tid in [0, count); count is clamped to size().
  • The calling thread runs tid 0; the pool’s workers run 1..count-1.
  • run blocks until every tid has completed, and every task’s writes happen-before the caller’s next action after it returns.
  • A region started while another is in progress, nested inside a task or concurrently from another thread, runs serially on its caller, every tid in order. It never deadlocks or oversubscribes.
  • The first exception from any tid is rethrown after all tids join.
  • run does not allocate. It holds the task as a pointer to your callable plus a function pointer, so this holds whatever the callable captures.

parallel_for does not allocate either. parallel_reduce and parallel_accumulate do: a threaded call keeps its per-chunk partials in a std::vector.

Each of the three loops also takes a team cap, max_threads, before the body: parallel_for(n, grain, max_threads, body), and likewise for parallel_reduce<T> and parallel_accumulate<Acc>. 0 means the whole pool. Use it to run at a thread count of your own below the pool’s. The grain still applies under the cap. A capped reduction groups its sum by the capped team, so it gives the same bits as an uncapped pool of that size.

The pool is a static inside an inline function in a header-only library. A program that includes MTL5 from several binaries can end up with one pool per binary:

  • every shared object built with -fvisibility=hidden keeps its own copy, and Python extension modules built with nanobind or pybind11 are built that way;
  • every Windows DLL that includes the header has its own copy;
  • on macOS, a single copy can’t be relied on.

Each copy is a correct pool, so nothing deadlocks. But each one is sized from MTL5_NUM_THREADS, so two such modules start twice the threads asked for, and configure(n) sizes only the copy in the binary that calls it. On Linux with default visibility, and in a statically linked program, there is one pool already, and none of this is needed.

To give every binary the same pool (#639), let one shared library hold it and have each module that uses MTL5 link that library:

Terminal window
cmake -B build -DMTL5_SHARED_POOL_LIBRARY=ON # builds and installs mtl5_pool
find_package(MTL5 REQUIRED)
target_link_libraries(my_extension PRIVATE MTL5::shared_pool)

Linking MTL5::shared_pool defines MTL5_SHARED_POOL for the module. Its pool() and configure() then refer to the pool in mtl5_pool. The shared library is loaded once per process, by every module that depends on it, so they share the same pool and the same configuration. That holds even for Python extensions loaded with RTLD_LOCAL. Each module looks the pool up once, so run() stays inline, with no extra call per region.

Without MTL5’s CMake, compile one translation unit of one shared library with MTL5_SHARED_POOL and MTL5_SHARED_POOL_BUILD defined, and include <mtl/parallel/shared_pool_definition.hpp> from it. Compile every other module with MTL5_SHARED_POOL defined, and link that library.

Rules:

  • Define MTL5_SHARED_POOL for a whole module, or not at all. Two translation units of one binary that disagree about it violate the one-definition rule.
  • Every module must be built against the same MTL5 pool header. The exported functions carry a layout version (shared_pool_v2 since #805). It changes when the pool’s layout does, so a module built against an incompatible header fails to link rather than misbehaving at run time.
  • The default, header-only build is unchanged. A module that does not link MTL5::shared_pool keeps its own pool.

thread_pool hands each region to its workers under a mutex and wakes them through a condition variable. A waiting worker, and the caller in its join, first spin for up to thread_pool::spin_us (1000 µs by default; set it with MTL5_THREAD_POOL_SPIN_US, the same in every translation unit) before they sleep (#805). Without that spin, regions that follow each other closely put the cores to sleep between them. On the Arduino Ventuno Q, whose A78C idle states take 1.45–1.76 ms to leave, every handoff then cost about a millisecond: a 512×512 LU, which opens one region per column, ran 7× slower on four threads than on one, and QR, LQ and Hessenberg 1.5–2.2× slower. With the spin, LU on four threads matches one thread and the other three run 1.5–1.6× faster. The cost is up to spin_us of a core per worker after the last region of a burst.

That is cheap for throughput work, but a caller in a hard real-time loop gets three problems from it:

  • a worker preempted while it holds the mutex blocks the caller (priority inversion);
  • a wake-up after the spin goes through the scheduler;
  • nothing states what one run() costs at worst.

mtl::parallel::spin_pool, in <mtl/parallel/spin_pool.hpp> (#636), has the same members and the same contract as thread_pool: size(), run, parallel_for, parallel_reduce and parallel_accumulate. Only the handoff differs:

  • Dispatch is one store to an atomic word that holds a generation and the region’s team size.
  • A waiting worker spins on that word, spin iterations of a CPU pause with a yield every 64th, and then sleeps in std::atomic::wait: on Linux a futex (for this 8-byte word, one in libstdc++‘s or libc++‘s shared table), and WaitOnAddress on Windows.
  • The join is an atomic countdown. The caller spins on it, then sleeps on it.
  • No notify system call while everyone is spinning. Each side keeps a count of sleepers, so the side that publishes calls notify only if a thread is actually asleep.
  • Nothing in the pool takes a lock. The busy flag and the first-exception slot are atomics too, so a preempted worker holds nothing the caller needs.
#include <mtl/parallel/spin_pool.hpp>
mtl::parallel::spin_pool rt(4, /*spin=*/40000); // 4 threads, spin ~1.4 ms
rt.run(4, [&](unsigned tid) { step(tid); }); // same contract as thread_pool

Define MTL5_POOL_SPIN in every translation unit alike, as with MTL5_HAS_BLAS. mtl::parallel::pool() then returns the process-wide spin_pool, and mtl::parallel::default_pool names its type. The pool is sized by configure() or MTL5_NUM_THREADS, as the default pool is, and its spin is read from MTL5_POOL_SPIN (default spin_pool::default_spin, 4096; 0 sleeps at once). Kernels reach their pool through pool(), a direct call with no virtual dispatch.

The selection is part of pool()’s mangled name, so if two modules disagree on it, the link fails rather than mixing the two at run time. Under MTL5_SHARED_POOL the spin pool is shared across modules the same way the thread pool is.

Under MTL5_POOL_SPIN, a direct call to thread_pool::instance() starts a second pool beside the kernels’ pool, and the two oversubscribe the cores. Reach the pool through pool().

The chunked loops are one set of templates used by both pool types, so a kernel partitions its work identically on either. GEMM’s and LU’s results are bit-identical across the two pools as well as across thread counts.

Per run() on a pool of n:

  • What can block: a worker, in std::atomic::wait, once its spin runs out; and the caller, in its join, once its own spin runs out, until the slowest tid finishes. Nothing else blocks. While every worker is spinning, the common path makes no system call.
  • Dispatch while every worker is spinning: one store, then one cache-line transfer of the word to each worker. Measured at 0.1–0.2 µs to the last worker at p50, and 0.1–0.3 µs at p99, for 2–8 threads on an i7-12700K.
  • A sleeping worker is woken through the kernel’s scheduler, and MTL5 cannot bound that. On the same machine under powersave, it measured about 10 µs at p50 and about 100 µs at p99, with rare millisecond outliers. To stay off this path, make the spin longer than the gap between regions.
  • A worker preempted mid-region delays the join by as long as the scheduler keeps it off a core. The caller spins, then sleeps, and that worker wakes it when it finishes. The pool cannot bound this, but the deployment can:
    • pin the pool’s threads to cores reserved for them;
    • give them a real-time scheduling class;
    • keep the pool no larger than the reserved cores, leaving the OS a core of its own.
  • Cost: after every region, each worker spins a full core for the spin. The default spin is about 140 µs on an Alder Lake P-core. A spin longer than the gap means the cores never idle.

Measured region latency, back to back and in a 1 kHz loop:

PoolBack to back, p50 / p991 kHz loop, p50 / p99
thread_pool3.0–6.1 / 3.6–8.0 µs20–41 / 58–118 µs
spin_pool, default spin0.23–0.48 / 0.25–0.57 µs11–16 / 61–110 µs (workers asleep)
spin_pool, spin outlasting the gap0.24–0.51 / 0.25–0.60 µs0.7–1.1 / 1.0–1.4 µs

Ranges are over T = 2, 4 and 8 on an i7-12700K’s P-cores. The method, maxima and caveats are in the i7-12700K results.

With MTL5_WITH_BLAS/MTL5_WITH_LAPACK, dense2D<float/double> operations call the vendor library, and with the SuiteSparse options CHOLMOD and SPQR are called. These libraries run thread teams of their own. A call made from a tid of an MTL5 region would start one team per tid: a region over 8 workers, each calling an 8-thread vendor GEMM, puts 64 runnable threads on 8 cores. MTL5’s own kernels avoid this by running a nested region serially. Vendor calls follow the same rule (#640):

  • Inside a region, a vendor call runs on one thread where the vendor offers a control scoped to the calling thread. MTL5 sets it for the duration of the call and then restores the previous value.
  • Outside a region, nothing changes. A direct mult(A, B, C) on BLAS types never enters the pool, so the vendor’s team is the only parallelism there, and capping it would make the call serial.

“Inside a region” is mtl::parallel::detail::in_region(), a thread-local flag. It is true on every tid of a region that the pool runs on more than one thread, and on every tid of a region that runs serially because another region is in progress. The serial fast path (a pool of one, or a region of one tid) leaves the flag unchanged: such a region has no MTL5 parallelism for the vendor to compete with. Under MTL5_SHARED_POOL the flag is exported with the pool, so a task started in one module sees it in another.

BackendControlScopeInside a region
MKLmkl_set_num_threads_localthe calling threadone thread¹
BLIS ≥ 1.0bli_thread_set_num_threadsthe calling thread; a new thread starts from BLIS_NUM_THREADSone thread¹
BLIS 0.9bli_thread_set_num_threadsthe whole processunchanged, as OpenBLAS
OpenBLASopenblas_set_num_threads, OPENBLAS_NUM_THREADSthe whole processunchanged; see below
CHOLMOD, SPQRcholmod_common::nthreads_max (their own OpenMP), and the BLAS they callthat callone thread, and the BLAS cap above

¹ The vendor is set to one thread at the first vendor call of a region and stays at one for the rest of the region. Later calls then cost a thread-local check rather than two writes, and with BLIS a write costs more than a small ddot. The region’s calling thread (tid 0) gets its previous setting back when the region ends, including when the region ends with an exception. A pool worker runs nothing but region tids, so its setting stays at one. A vendor setting that a task changes during a region is not capped again.

CMake works out the vendor from the symbol that vendor’s thread control exports in BLAS_LIBRARIES/LAPACK_LIBRARIES, and reports it as BLAS/LAPACK vendor: mkl|blis|openblas. For BLIS it also runs a small program that measures whether the setting is per-thread: BLIS 1.0 made it thread-local, unless BLIS was built with BLIS_DISABLE_TLS. BLIS is capped only when that program confirms it. When cross-compiling the program cannot run, so BLIS is left uncapped. A generic libblas.so.3 that forwards to a vendor without exporting its control is reported as not identified. The Debian/Ubuntu BLIS alternative is one such library. In that case vendor calls are not capped; naming the vendor (-DBLA_VENDOR=FLAME for BLIS, -DBLA_VENDOR=Intel10_64lp for threaded MKL) links the library that has the control. mtl::interface::blas_vendor() and blas_capped_in_region() report the result to code.

Measured on the i7-12700K: 64 dgemms in one 8-tid region, each vendor set to 8 threads (i7-12700K results). Uncapped, BLIS ran at 0.1–3 GFLOP/s; capped it ran at 405–507. MKL went from 328–381 GFLOP/s to 468–513. OpenBLAS cannot be capped per thread: uncapped it varied between rounds from 102 to 407 GFLOP/s, and set to one thread for the whole process it held 430–486.

OpenBLAS: the application decides which runtime owns the cores

Section titled “OpenBLAS: the application decides which runtime owns the cores”

OpenBLAS’s only control is process-wide, so MTL5 never changes it: doing so would also make the application’s direct calls serial. Choose one of two configurations:

  • A threaded MTL5, a serial vendor. Set OPENBLAS_NUM_THREADS=1, or call openblas_set_num_threads(1) once at startup. Vendor calls made inside regions then run serially, as they do under MKL and BLIS. Vendor calls made outside regions also run serially.
  • A threaded vendor, a serial MTL5. Leave MTL5_NUM_THREADS unset (1), and give the cores to OpenBLAS’s team.

The same choice applies to BLIS 0.9, to BLIS’s process-wide settings, and to any vendor that MTL5 cannot identify. The variables are OPENBLAS_NUM_THREADS, BLIS_NUM_THREADS, MKL_NUM_THREADS and OMP_NUM_THREADS, the last for OpenMP builds of any of them. MTL5 has no switch that sets these for you: each is one line of the vendor’s own API, and the application is the one that knows which configuration it wants.

dot, two_norm, axpy and scale on dense float/double vectors take MTL5’s own SIMD loops in a BLAS build too (#754); interface::blas:: keeps the wrappers for a direct call. Level 2 and Level 3 still dispatch.

A Level 1 routine is bandwidth-bound, so a vendor has nothing to win over a SIMD loop that reads each operand once. What it can add is the cost of its thread team. OpenBLAS splits ddot and daxpy across its team above a few thousand elements, and on a machine whose cores are busy each call waits for the team to be scheduled. Per call, OpenBLAS 0.3.32 on the i7-12700K, best of 5 × 200 calls:

nddot, 20 threads, idleddot, 20 threads, 3 cores busyddot, 1 threaddaxpy, 20 threads, 3 cores busydaxpy, 1 thread
8,1921.6 µs1.1 µs1.6 µs1.4 µs2.1 µs
16,3841.8 µs6,245 µs1.7 µs6,025 µs4.6 µs
1,048,57654 µs3,150 µs252 µs6,095 µs341 µs

dnrm2 and dscal showed no such cost. A shared workstation or a CI runner always has busy cores, and a Krylov iteration makes several of these calls on vectors of exactly that size: with OpenBLAS at its default threads, a BiCGSTAB/ILU(0) solve at n = 16,384 took 448 ms against 15 ms without BLAS.

On an idle machine OpenBLAS’s team does make a large ddot faster, 54 µs against 208 µs at n = 10⁶. MTL5’s loops thread through MTL5’s own pool for that: with MTL5_NUM_THREADS > 1, dot and two_norm split above 65,536 elements and axpy and scale above the same grain (see the table below).

mtl5_bench records the vendor and the thread count it would use outside a region, probes.blas_vendor and probes.blas_vendor_threads, next to MTL5’s own probes.threads. A BLAS-build Level 2 or Level 3 result read without them can’t be told from a result measured at a different thread count.

KernelWherePartitionBit-identical to serial?
GEMM (mult, native blocked)detail/gemm_blocked.hppic-blocks of CYes
GEMV (row- and col-major)operation/mult.hppoutput rowsYes
axpy, scaloperation/{axpy,scale}.hppelement rangeYes
dot, two_normoperation/{dot,norms}.hppchunked reductionNo¹
dot, dot_real, two_norm, frobenius_norm with an Accumulatoroperation/{dot,norms}.hpp, detail/accumulate.hppchunked reduction, partials combined by accumulator_traits::mergeYes for an exact accumulator, otherwise No¹ ²
SpMV (compressed2D * vector)mat/operators.hppoutput rowsYes
GEMM, GEMV, SpMV with an Accumulatoroperation/mult.hppoutput rows, one accumulator per output entryYes
LU (native right-looking trailing update; accumulator path: column k and row k)operation/lu.hpprows of the trailing matrix / entries of the column and rowYes
Cholesky (native and accumulator paths)operation/cholesky.hpprows below the diagonalYes
Householder application (QR, tridiagonalization; void path)operation/householder.hppcolumns (left) / rows (right)Yes
Bunch–Kaufman LDLᵀ, accumulator pathoperation/ldlt_bk.hpprows of the on-demand Schur columnYes
Sparse triangular solves (level-scheduled, #535)sparse/factorization/level_schedule.hpprows within a levelYes

¹ The reduction kernels sum per-chunk partials, so the result is deterministic for a fixed thread count but not bit-identical to the serial summation (the grouping/associativity differs). The library’s tolerances (WithinRel) account for this. Every other kernel produces each output element from exactly one chunk, so its result is bit-identical regardless of MTL5_NUM_THREADS.

² The accumulator-typed reductions (#582) split only when the operands are plain dense storage (ThreadableDenseVector/ThreadableDenseMatrix) and the accumulator satisfies mergeable_accumulator. An accumulator without a merge runs serially, as before. With an exact accumulator, such as a Universal quire, every partial and the merge are exact, so the result is the same bits for any thread count: on 2²⁰ posit<32,2> elements, dot<quire<P>, P> took 830 ms on 1 thread, 437 ms on 2, 228 ms on 4 and 125 ms on 8, with identical results. A rounding accumulator (a plain wider type, fma_accumulator) regroups like the other reductions: a double accumulator over the same data gave a different last few bits at each of 1, 2, 4 and 8 threads.

Because the iterative solvers (mtl::itl — CG, GMRES, BiCGSTAB, MINRES, …) and the eigensolvers are written against A * x, dot, and axpy, they inherit the SpMV/L1 threading with no solver-code changes.

  • parallel::thread_pool — a persistent pool of worker threads. run(count, task) runs task(tid) for tid ∈ [0,count) (the caller runs tid 0), blocking until all finish. One condition-variable handoff per parallel region — no thread is spawned per call.
  • parallel_for(n, grain, body) — contiguous chunking of [0,n) for the element-wise/per-row kernels.
  • parallel_reduce<T>(n, grain, map) — chunked reduction combined with operator+ in chunk order, for dot/nrm2.
  • parallel_accumulate<Acc>(n, grain, init, map, combine) — the same chunking, but each chunk accumulates into its own partial and the partials are folded in chunk order by the caller’s combine, for accumulators that have no +. One thread, or too little work to split, is exactly acc = init(); map(acc, 0, n). detail::accumulate drives it with accumulator_traits::merge.
  • A grain threshold keeps small problems serial so the handoff never costs more than the work; nested/concurrent regions fall back to serial (no oversubscription or deadlock).

Measurement methodology (pin to distinct physical cores)

Section titled “Measurement methodology (pin to distinct physical cores)”

Getting an honest scaling curve requires confining the run to distinct physical cores — not to SMT siblings and not to slower efficiency cores. taskset sets the process affinity mask (it does not hard-pin individual threads); but when the mask is exactly N logical CPUs, one per physical core, the OS keeps the N worker threads one-per-core in steady state. This matters especially on hybrid CPUs. On the i7-12700K used below, logical CPUs pair up as SMT siblings on 8 P-cores and the 4 E-cores sit at the top:

lscpu -e=CPU,CORE,MAXMHZ
# P-cores: logical 0,1 -> core0 ; 2,3 -> core1 ; ... 14,15 -> core7 (~4.9-5.0 GHz)
# E-cores: logical 16..19 -> cores 8..11 (3.8 GHz)

So the one-logical-CPU-per-physical-P-core set is 0,2,4,6,8,10,12,14. A naive taskset -c 0-7 instead spans only four physical cores (0–3) doubled by SMT — an easy way to mismeasure “8 threads” as 4 hyperthreaded cores. Pin explicitly:

Terminal window
MTL5_NUM_THREADS=8 taskset -c 0,2,4,6,8,10,12,14 ./bench # 8 distinct P-cores

Indicative only. Single-run on a shared Intel i7-12700K, fp64, the process affinity-masked to N distinct physical P-cores (0,2,4,6,8,10,12,14, one logical CPU each — no SMT siblings, no E-cores). Directional (±10–15%), not an authoritative benchmark.

fp64 throughput (GFLOP/s) and speedup vs 1 thread:

Kernel1T2T4T8Tspeedup @4T@8T
GEMM (N=2048)57.4110.3205.1362.53.57×6.32×
GEMM (N=4096)57.4——393.7—6.85×
GEMV (N=8192)9.413.217.120.61.82×2.19×
dot (N=8M)4.67.68.611.11.89×2.43×
axpy (N=8M)3.45.36.46.91.87×2.02×
scal (N=8M)2.64.25.66.62.12×2.48×

SpMV (490k-row 2D Laplacian) is measured in ms/matvec (lower is better), not GFLOP/s:

threads1T2T4T8Tspeedup @8T
ms/matvec2.181.641.211.131.92×
  • GEMM is compute-bound and scales with cores — 6.32× at 8 cores (N=2048), rising to 6.85× at N=4096 (~86% parallel efficiency). Scaling improves with N, which rules out a memory-bandwidth ceiling: larger tiles do more arithmetic per parallel region, so the fixed per-region synchronization amortizes better. At small N (e.g. N=1024, ~5.1×) that per-region handoff is the main limiter; the remaining few percent is the all-core turbo drop.
  • The Level-1/2 kernels and SpMV are memory-bandwidth-bound. They reach ~2–2.5× and then plateau: a couple of P-cores already saturate the (two DDR5 channel) memory bandwidth, so more threads cannot help. This is the expected roofline behavior, not a defect — the value is the ~2× a few cores extract over one, and (for the solvers) overlapping it with the compute-bound GEMM/preconditioner work.

Measurement caveat learned the hard way. An earlier draft of this table reported GEMM at only 4.3× on “8 threads” — that run used taskset -c 0-7, which is four physical cores hyperthreaded, not eight cores. Pinning to distinct physical cores (above) shows the real 6.3–6.9×. Always verify the affinity mask against the core topology before drawing a scaling conclusion. The full investigation — hypotheses, controls, and attribution — is written up as a performance-engineering case study: Multi-Core Scaling of MTL5’s Threaded Kernels.

Every row added above is owner-computes: each output entry is produced by exactly one thread, so the accumulator paths are bit-identical to serial for every accumulation configuration. Only the four reductions in the merge row split a single sum across threads. The Accuracy Architecture explains the two kinds, and Accumulation Configurations gives the measured determinism results.

  • Transposed SpMV (scatters into y, which would race across rows). Its accumulator path keeps one accumulator per output entry and stays serial, since privatizing them per thread would change the summation grouping.
  • Sparse numeric factorizations (LU, Cholesky, LDLᵀ, supernodal, QR), spgemm, the preconditioners and the smoothers run serially, with or without an accumulator. Their solves use the level-scheduled triangular solves above.
  • Dense factorizations are threaded within each step (above) but not blocked over the L3 building blocks, and ldlt_factor is serial.

For reproducible-to-the-bit results across runs, keep MTL5_NUM_THREADS fixed (or =1): every kernel except the reductions (dot/nrm2, and the accumulator-typed ones with a rounding accumulator) is bit-identical across thread counts, and even those are deterministic for a fixed thread count. If you need serial-exact reductions, run those with MTL5_NUM_THREADS=1, or use an exact accumulator, whose reductions give the same bits at any thread count.