Skip to content

Interleave width across the benchmark machines

The generic row kernels (gemv, trmv and the forward substitution on a row-major matrix) sum W rows per pass, one accumulator per row (#749, #772). These sweeps were built with the earlier default, W = 6 for float and double, measured only on an i7-12700K (#775). This page measures W on every machine of the #737 platform campaign (#780). The per-ISA defaults below come from it, and so does the table run-time selection uses (#778).

  • Tool: bench_interleave_rows --reps 21, n = 4096. It’s built in its own build directory, the same generic build as the platform session’s native variant: no BLAS, no MTL5_NATIVE_FAST_GEMM, no -march. That means SSE2 on x86-64, NEON on AArch64, and /O2 under MSVC.
  • Where: it runs on the session’s first CPU, under the session’s clock policy, after its capability card’s runs (INTERLEAVE=1 in the machine profiles), and only on a machine that passed the session’s preflight.
  • Repeats: three runs per compiler in each session, and two sessions per machine, the second after a cold boot. Those are the same sessions as the machines’ capability cards (benchmark systems). Each cell below is the median of the 6 runs. Each run’s value is already a median of 21.
  • Compilers: GCC and Clang on Linux, MSVC and clang-cl on Windows. The Jetson has no Clang.
  • Why the median over runs: each run’s fastest width is listed too, but the median is what counts, because the first run of a sweep often favours a narrower width than the runs after it.
  • x86-64: W = 8 is the better default. Over every x86-64 machine, compiler and kernel, W = 8 averages 1.5% slower than each kernel’s fastest width, 10% at worst (Clang on the i7-12700K, f32 gemv). W = 6 averages 4.1% slower, and 31% at worst: clang-cl’s f64 trmv, slower at W = 6 than at W = 4 on both Ryzens and in both sessions. On the two Zen 4 machines, under both compilers, W = 8 is the fastest width of every kernel or within 0.2% of it. The Ivy Bridge Xeon is the exception: it prefers 4 for f64 and loses at most 6% at W = 8.

  • AArch64: W = 6 is not safe; W = 4 is the better default. With Clang (on the Arduino Ventuno Q), W = 4 is the fastest width in every kernel, and W = 6 is 18-24% slower than it. With GCC, gemv agrees: f64 and f32 are fastest at 4 on both AArch64 machines. On the Jetson, GCC’s f64 trmv and forward substitution are fastest at 2, and lose 12% at 4.

  • GCC on AArch64 had a separate problem, now fixed (#795). In these sessions row-major f32 trmv and the forward substitution take 5.6-7.2x longer at W = 4 than at W = 2, and 1.4-4.5x longer at W = 6 to 12, on both machines (GCC 11 and 13). Clang on the same machine does not. The cause was the keep_scalar barrier from #776, which was on for GCC on AArch64 although it had been measured only on x86-64. An A/B on both boards (median of 3 runs, --reps 21):

    GCC, f32W = 4, with the barrierW = 4, without itW = 2, before
    Jetson Orin, trmv50.04 ms6.31 ms7.29 ms
    Jetson Orin, forward substitution39.45 ms5.58 ms7.08 ms
    Ventuno Q, trmv31.66 ms3.12 ms4.75 ms
    Ventuno Q, forward substitution26.22 ms2.99 ms4.70 ms
    Jetson Orin, gemv8.23 ms8.34 ms
    Ventuno Q, gemv5.51 ms9.14 ms

    So on AArch64 the triangular kernels no longer take the barrier, and W = 4 is their fastest width, 13-36% faster than the 2 they were held at. gemv keeps it: it is neutral on the Jetson and 1.66x faster with it on the Ventuno. The tables below are from the 2026-11-11 sessions, with the barrier.

These are the built defaults (include/mtl/interface/interleave_rows.hpp). Every target caps the width at half its vector registers. Targets other than x86-64 and AArch64 are assumed to have 8, so they get 4: the measured 6 that was the default before #772 is capped there, as it is on 32-bit x86.

ISA classfloatdoubleworst loss against the fastest width
x86-64 (SSE2, generic build)8810% (i7-12700K, Clang, f32 gemv); 6% on the Xeon
AArch64 (NEON, generic build)440% under Clang; 12% under GCC (Jetson, f64 trmv and forward substitution)
  • x86-64 with AVX-512: none of these machines has it. The Zen 4 machines support AVX-512, but the generic build doesn’t use it, so that class is still unmeasured.
  • Run-time selection (#778): it recovers the rest, for the CPUs it knows; see the next section.

With MTL5_INTERLEAVE_RUNTIME, float and double compile each of the three kernels at W = 2, 4, 6, 8 and 12. On first use they take the width a table lists for the CPU and the compiler, and the compile-time default where it lists nothing (include/mtl/interface/interleave_runtime.hpp). It is off by default: a file that calls all three kernels for both types took 0.5-0.9 s longer to compile on the i7 (1.2 s without), and their code grew from 24-29 KB to 62-78 KB.

The table is each kernel’s fastest width in the sweeps above, for the cores those machines have:

coreidentified byGCCClangMSVC, clang-cl
Alder Lake (i7-12700K)Intel family 6, model 0x97, 0x9Af64 12, 12, 12; f32 6, 8, 8f64 8; f32 6
Ivy Bridge (Xeon E5-2420 v2)Intel family 6, model 0x3A, 0x3Ef64 4, 6, 4; f32 8, 4, 4f64 4; f32 6
Zen 4 (both Ryzens)AMD family 0x19, model 0x60-0x7F8
Cortex-A78AE (Jetson Orin)Arm part 0xd42f64 4, 2, 2; f32 4
Cortex-A78C (Ventuno Q)Arm part 0xd4b44

The three numbers are gemv, trmv and the forward substitution. On AArch64 the part comes from /proc/cpuinfo (Linux only). On the Ventuno that is the A78C, not the A55, which is where the sweeps ran. The Jetson’s f32 trmv and forward substitution are 4 since #795. The sweeps were a generic build, so an x86-64 build with AVX uses no entry.

How close it comes, measured with the table in place (bench_interleave_rows, same method as above, one session, median of the runs): the worst kernel, as time at the chosen width over time at the fastest, against today’s fixed default.

machine, compilerrunsrun-time choice, worstfixed default, worst
i7-12700K, GCC61.007 (f64 gemv)1.044 (f64 trmv)
i7-12700K, Clang31.007 (f64 trmv)1.094 (f32 gemv)
Jetson Orin, GCC31.0001.128 (f64 trmv)
Ventuno Q, GCC31.0001.000
Ventuno Q, Clang31.0001.000

The Xeon and the two Ryzens weren’t reachable for this. Their entries are the sweeps’ fastest widths, so in those sweeps the choice is the fastest by construction. Against the fixed default that is up to 6% (the Xeon) and 0.2% (the Ryzens). They are still to be measured with the table in place.

f64 gemv on the i7 under GCC is bandwidth-bound at n = 4096. Widths 6 to 16 are within about 5% of each other there, and run to run the fastest moves among 8, 12 and 16.

The dispatch is a switch on a cached width per call. bench_interleave_dispatch times the generic gemv built without and with run-time selection, at the same width, out of line as a caller’s call is. At n = 64 the cost is under 1% on the i7 (GCC and Clang), the Jetson and the Ventuno under Clang, and 1-3.4% on the Ventuno under GCC (f64 the most). At n = 256 it is at most 1.6% (Clang on the i7). At n = 16 it is 1-2 ns on the i7, and inside the run-to-run spread on the boards.

The table’s widths are for n = 4096. At small n they need not be the fastest: on the i7 under GCC, the chosen 12 (f64) and 6 (f32) are 3-4% slower than 8 at n = 64 and 2-3% at n = 256. Under Clang the chosen f32 6 is 8% faster than 8 at n = 64 and 256, and level with it at n = 16.

Per machine and compiler: the fastest width of each kernel, and the worst cost of a fixed width against the fastest (time at that width / time at the fastest; 1.000 is no loss).

machineISAcompilerfastest (f64 gemv, trmv, trisolve / f32 …)W=4W=6W=8W=6 / W=4
arduino-ventuno-qAArch64clang++4, 4, 4 / 4, 4, 41.0001.2371.4721.237
arduino-ventuno-qAArch64g++4, 4, 4 / 4, 2, 26.6544.4083.1641.252
i7-12700kx86-64clang++8, 8, 8 / 6, 6, 61.1241.0521.1000.957
i7-12700kx86-64g++12, 12, 12 / 6, 8, 81.0881.0411.0290.984
jetson-orinAArch64g++4, 2, 2 / 4, 2, 27.1863.9764.5321.582
ryzen-7-8845hsx86-64clang-cl8, 8, 6 / 8, 8, 81.1891.3101.0021.149
ryzen-7-8845hsx86-64msvc8, 8, 8 / 8, 8, 81.3231.0781.0000.976
ryzen-9-8945hsx86-64clang-cl8, 8, 8 / 8, 8, 81.2161.3151.0001.137
ryzen-9-8945hsx86-64msvc8, 8, 8 / 8, 8, 81.3301.0591.0000.988
xeon-e5-2420v2x86-64clang++4, 4, 4 / 6, 6, 61.0921.0391.0631.039
xeon-e5-2420v2x86-64g++4, 6, 4 / 8, 4, 41.0381.0261.0461.019

Time at a fixed width / time at the fastest width, over every machine, compiler, kernel and type of the class: geometric mean / worst case.

ISAW=2W=4W=6W=8W=12
x86-641.628 / 2.6831.102 / 1.3301.041 / 1.3151.015 / 1.1001.518 / 3.119
AArch641.242 / 1.7081.523 / 7.1861.500 / 4.4081.640 / 4.5322.053 / 3.799

0xd4b; Ubuntu clang version 18.1.3 (1ubuntu1); vector registers 32; built default W = 6 (f64), 6 (f32); median of 6 runs (2 sessions), each the median of 21. Data: benchmarks/data/arduino-ventuno-q/interleave_rows/.

kernelW=1W=2W=4W=6W=8W=12W=16fastestW=6 / fastestW=6 / W=4per-run fastest
gemv f6417.74910.6298.2549.84010.42512.23814.22841.1921.1924: 6
trmv f6410.2255.8004.5235.3815.7686.62010.53541.1901.1904: 6
trisolve f6410.2825.8774.4835.3005.6208.89310.45941.1821.1824: 6
gemv f3216.7309.5636.3897.5449.40611.40814.05441.1811.1814: 6
trmv f328.8525.0423.2904.0594.4295.4269.39541.2341.2344: 6
trisolve f328.7294.9623.1793.9344.3138.6019.31841.2371.2374: 6

0xd4b; g++ (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0; vector registers 32; built default W = 6 (f64), 6 (f32); median of 6 runs (2 sessions), each the median of 21. Data: benchmarks/data/arduino-ventuno-q/interleave_rows/.

kernelW=1W=2W=4W=6W=8W=12W=16fastestW=6 / fastestW=6 / W=4per-run fastest
gemv f6418.84410.4318.1049.5879.99212.10814.02241.1831.1834: 6
trmv f6410.4795.7104.4495.1245.3596.2666.87741.1521.1524: 6
trisolve f6410.4425.7774.3935.1185.2926.1066.70841.1651.1654: 6
gemv f3216.6599.0055.5566.9587.7779.21311.01141.2521.2524: 6
trmv f328.9764.76131.6756.7877.5727.6758.17521.4260.2142: 6
trisolve f328.7554.71226.19720.76814.90915.14517.94024.4080.7932: 6

12th Gen Intel(R) Core(TM) i7-12700K; Ubuntu clang version 21.1.8 (6ubuntu1); vector registers 16; built default W = 6 (f64), 6 (f32); median of 6 runs (2 sessions), each the median of 21. Data: benchmarks/data/i7-12700k/interleave_rows/.

kernelW=1W=2W=4W=6W=8W=12W=16fastestW=6 / fastestW=6 / W=4per-run fastest
gemv f648.3385.1154.0023.8173.7053.7513.84581.0300.9548: 6
trmv f644.3462.7252.1251.9931.9211.9512.69481.0370.9388: 4, 12: 2
trisolve f644.3042.7102.0851.9961.8972.2422.74981.0520.9578: 6
gemv f326.9903.9402.9102.6252.8872.8762.97061.0000.9026: 6
trmv f323.7592.1101.5891.4161.5181.5082.29261.0000.8916: 6
trisolve f323.7282.0591.5421.3721.4741.9062.35961.0000.8896: 6

12th Gen Intel(R) Core(TM) i7-12700K; g++ (Ubuntu 15.2.0-16ubuntu1) 15.2.0; vector registers 16; built default W = 6 (f64), 6 (f32); median of 6 runs (2 sessions), each the median of 21. Data: benchmarks/data/i7-12700k/interleave_rows/.

kernelW=1W=2W=4W=6W=8W=12W=16fastestW=6 / fastestW=6 / W=4per-run fastest
gemv f648.3735.0963.8173.7083.6333.5623.647121.0410.97112: 5, 16: 1
trmv f644.3312.6652.1072.0102.0101.9532.054121.0290.95412: 6
trisolve f644.3092.6452.0791.9761.9331.9102.045121.0340.95012: 6
gemv f326.9687.3332.7792.7342.7452.7693.51061.0000.9846: 3, 8: 1, 12: 2
trmv f323.7342.0521.5441.4721.4411.4581.84081.0220.9538: 6
trisolve f323.6942.0161.4811.4321.4131.4211.76881.0130.9678: 5, 12: 1

ARMv8 Processor rev 1 (v8l); g++ (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0; vector registers 32; built default W = 6 (f64), 6 (f32); median of 6 runs (2 sessions), each the median of 21. Data: benchmarks/data/jetson-orin/interleave_rows/.

kernelW=1W=2W=4W=6W=8W=12W=16fastestW=6 / fastestW=6 / W=4per-run fastest
gemv f6425.13413.59912.88819.94224.79641.00640.27041.5471.5474: 6
trmv f6413.8537.9768.97612.55214.89521.99421.15721.5741.3982: 5, 4: 1
trisolve f6413.5497.8008.71012.43115.08222.45721.26021.5941.4272: 5, 4: 1
gemv f3224.91213.2137.73512.23614.14318.18123.28641.5821.5824: 6
trmv f3213.1886.92649.77511.04612.96514.92117.17921.5950.2222: 6
trisolve f3213.0276.80039.42727.03530.81625.83328.99823.9760.6862: 6

AMD Ryzen 7 8845HS w/ Radeon 780M Graphics; Clang 22.1.3 with MSVC-like command-line; vector registers 16; built default W = 6 (f64), 6 (f32); median of 6 runs (2 sessions), each the median of 21. Data: benchmarks/data/ryzen-7-8845hs/interleave_rows/.

kernelW=1W=2W=4W=6W=8W=12W=16fastestW=6 / fastestW=6 / W=4per-run fastest
gemv f6410.7045.8413.4893.2063.1196.9697.73981.0280.9196: 2, 8: 4
trmv f645.7343.2121.9712.2651.7283.7383.98681.3101.1498: 6
trisolve f645.7453.2361.9251.7621.7653.7153.92961.0000.9156: 2, 8: 4
gemv f3210.0955.4532.9972.7382.5207.8249.23681.0870.9146: 2, 8: 4
trmv f325.4282.8941.6141.4481.4393.9754.82581.0060.8976: 2, 8: 4
trisolve f325.4082.9111.7001.4811.4313.9254.71381.0350.8716: 2, 8: 4

AMD Ryzen 7 8845HS w/ Radeon 780M Graphics; MSVC 19.52.37029.0; vector registers 16; built default W = 6 (f64), 6 (f32); median of 6 runs (2 sessions), each the median of 21. Data: benchmarks/data/ryzen-7-8845hs/interleave_rows/.

kernelW=1W=2W=4W=6W=8W=12W=16fastestW=6 / fastestW=6 / W=4per-run fastest
gemv f6410.6985.8363.6193.1723.0464.1768.70481.0410.8778: 6
trmv f645.7533.1801.9291.8041.7102.5754.57481.0550.9356: 1, 8: 5
trisolve f645.8143.2241.9331.7851.6972.5494.50481.0520.9238: 6
gemv f3210.0875.3913.0462.4822.3024.3459.02981.0780.8156: 1, 8: 5
trmv f325.4322.8321.5581.5201.4802.4894.73181.0270.9766: 1, 8: 5
trisolve f325.3892.8641.6211.5671.4932.4744.66381.0490.9666: 1, 8: 5

AMD Ryzen 9 8945HS w/ Radeon 780M Graphics; Clang 22.1.3 with MSVC-like command-line; vector registers 16; built default W = 6 (f64), 6 (f32); median of 6 runs (2 sessions), each the median of 21. Data: benchmarks/data/ryzen-9-8945hs/interleave_rows/.

kernelW=1W=2W=4W=6W=8W=12W=16fastestW=6 / fastestW=6 / W=4per-run fastest
gemv f6410.5865.9283.5953.2473.1196.8777.69481.0410.9038: 6
trmv f645.6943.2292.0022.2771.7313.7523.99481.3151.1378: 6
trisolve f645.6973.2241.9391.7761.7413.6653.96381.0200.9168: 6
gemv f329.9515.4252.9882.6652.4577.6639.10181.0850.8928: 6
trmv f326.7043.5501.8951.7161.6474.9516.05681.0430.9068: 6
trisolve f326.6743.5471.9961.7721.6444.9315.92781.0780.8888: 6

AMD Ryzen 9 8945HS w/ Radeon 780M Graphics; MSVC 19.52.37029.0; vector registers 16; built default W = 6 (f64), 6 (f32); median of 6 runs (2 sessions), each the median of 21. Data: benchmarks/data/ryzen-9-8945hs/interleave_rows/.

kernelW=1W=2W=4W=6W=8W=12W=16fastestW=6 / fastestW=6 / W=4per-run fastest
gemv f6410.5965.8643.6063.1893.0664.1478.61781.0400.8848: 6
trmv f645.7523.2011.9721.7971.7212.5834.56181.0440.9118: 6
trisolve f645.7753.2171.9661.7851.6962.5524.45781.0530.9088: 6
gemv f3210.0435.3002.9862.3792.2454.2908.87881.0590.7978: 6
trmv f325.3512.8081.8801.8571.7693.1075.93381.0490.9884: 1, 6: 1, 8: 4
trisolve f326.7003.5211.9401.8691.7923.0825.85681.0430.9648: 6

Intel(R) Xeon(R) CPU E5-2420 v2 @ 2.20GHz; Ubuntu clang version 21.1.8 (6ubuntu1); vector registers 16; built default W = 6 (f64), 6 (f32); median of 6 runs (2 sessions), each the median of 21. Data: benchmarks/data/xeon-e5-2420v2/interleave_rows/.

kernelW=1W=2W=4W=6W=8W=12W=16fastestW=6 / fastestW=6 / W=4per-run fastest
gemv f6419.77013.85113.12513.49613.70115.12717.51641.0281.0284: 6
trmv f6410.5117.5547.1717.2887.3618.21215.05241.0161.0164: 4, 6: 2
trisolve f6410.1987.3127.0497.3217.27410.13615.04141.0391.0394: 6
gemv f3219.43910.7239.0858.5218.69810.03711.50261.0000.9386: 5, 8: 1
trmv f3210.1945.6274.7504.4634.6645.30710.86661.0000.9406: 5, 8: 1
trisolve f329.9575.4424.6374.2484.5158.65211.11761.0000.9166: 6

Intel(R) Xeon(R) CPU E5-2420 v2 @ 2.20GHz; g++ (Ubuntu 15.2.0-16ubuntu1) 15.2.0; vector registers 16; built default W = 6 (f64), 6 (f32); median of 6 runs (2 sessions), each the median of 21. Data: benchmarks/data/xeon-e5-2420v2/interleave_rows/.

kernelW=1W=2W=4W=6W=8W=12W=16fastestW=6 / fastestW=6 / W=4per-run fastest
gemv f6420.67513.61813.17713.29613.67613.98115.34641.0091.0094: 5, 6: 1
trmv f6410.3447.3337.0677.0507.2707.6749.01461.0000.9984: 3, 6: 3
trisolve f6410.0917.2106.8546.9317.1697.3638.80341.0111.0114: 5, 6: 1
gemv f3219.09319.7558.7528.6488.43110.62314.85481.0260.9884: 1, 8: 5
trmv f329.9985.4714.4714.5234.6275.4837.48441.0121.0124: 4, 6: 2
trisolve f3210.0295.3534.2834.3644.4065.2377.02441.0191.0194: 6

Run a platform session with INTERLEAVE=1: --interleave in benchmarks/remote_session.sh, or $env:INTERLEAVE = '1' for a Windows profile. Publish it with benchmarks/publish_platform.cmake. Then:

python3 benchmarks/interleave_table.py import <machine> <session stamp> <results dir>
python3 benchmarks/interleave_table.py table

The first command copies the sweeps to benchmarks/data/<machine>/interleave_rows/<stamp>/: one CSV per compiler, with its .sysinfo sidecar, and the raw outputs. The second rewrites the generated part of this page.