Pre-release / continuous build: this is an experimental build, not a stable release.
CUDA: Add OCP FP8 support (#28413) * Initial CUDA WIP version * Use fp8 intrinsics * Do W8A16 for FP8 in GEMV case (base) osimons@ub-osimons:~/lcpp_fp8.git$ ./build-x64-linux-gcc-reldbg/bin/llama-bench -m /mnt/share/gguf/nvidia/Qwen3.6-27B-NVFP4/Qwen3.6-27B-NVFP4_fp8.gguf -p 8196 -ub 512,2048 -d 0,32000 -n 256 --load-mode dio ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32165 MiB): Device 0: NVIDIA RTX PRO 4500 Blackwell, compute capability 12.0, VMM: yes, VRAM: 32165 MiB | model | size | params | backend | ngl | n_ubatch | lm | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | ---------: | --------------: | -------------------: | | qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | pp8196 | 2781.53 ± 8.48 | | qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | tg256 | 37.56 ± 0.02 | | qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | pp8196 @ d32000 | 1616.31 ± 2.61 | | qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | tg256 @ d32000 | 34.11 ± 0.03 | | qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | pp8196 | 2542.00 ± 0.51 | | qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | tg256 | 37.36 ± 0.02 | | qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | pp8196 @ d32000 | 1762.86 ± 1.33 | | qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | tg256 @ d32000 | 34.09 ± 0.02 | * Optimize GEMV kernel for FP8 * Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.2 Fixes https://github.com/ggml-org/llama.cpp/pull/28413#discussion_r4044500099 * Remove trailing whitespace * Add supported for strided copies to/from FP8 in CUDA * Prepare Q_reg for FP8 FlashAttention in fattn-vec.cuh * Fix odd-rows for convert/getrows, possible with FP8 * Fix FP8 fallback in cuBlas * Gate mmvf_f8x2_e4m3_to_bf162 on FP8 availability, i.e. CTK >= 11.8 * Restrict (fused) MMVF to 2*ts alignment, and cublas FP8 fo 16-byte * Add todo about accumulator/precision honoring This is beyond the scope of the stacked PR * Avoid races in fp8_fallback * Allow Hopper and other BW SKUs for FP8 * Use intrinsic over explicit manual conversions * Resolve TODO As per spec, we don't have to preserve signs * Add commend on cublaslt workspaces * Use convert_unary_cont_cuda over block dequant in convert.cu * Use get_rows_cuda_float for fp8 kv cache * Make FP8 copies work on all IHV's * More cleanup * Simplify by falling back to shared cuBLAS helper * Fuse weight_scale multiplication for FP8 also * More code-cleanup * Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.4 Only cvt.rn.bf16x2.e4m3x2 is exposed from 13.2 onwards, so 93e98b9203dfacb85f9ac665f5f0482a8c6b4192 was incorrect fix * Use F32 activations in fallback mmvf path * Explicitly link against cublaslt * Move FP8 conversion intrinsics into ggml_cuda_cast Cleaner this way * WIP src1/prec for fp8 in MMVF * Remove comment * Keep only f8 x f8 KV cache for now * Just do FP8 x FP32 x FP32 in MMVF as its fast enough * Update ggml/src/ggml-cuda/ggml-cuda.cu Co-authored-by: Aman Gupta * Add batched FP8 GEMMS in cublas as suggested by code review https://github.com/ggml-org/llama.cpp/pull/28413#issuecomment-6676015644 * Extract fp8x2 -> float2 load into helper * Searapte copy checks to/from fp8 * Compiler massaging to avoid repeated address offsets --------- Co-authored-by: Aman GuptaWebsite: - https://llama.app
Attestations: - https://github.com/ggml-org/llama.cpp/attestations/66285066
macOS/iOS: - macOS Apple Silicon (arm64) - macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED - macOS Intel (x64) - iOS XCFramework
Linux: - Ubuntu x64 (CPU) - Ubuntu arm64 (CPU) - Ubuntu 24.04 s390x (CPU) - Ubuntu 26.04 s390x (CPU) - Ubuntu 26.04 s390x (zDNN) - Ubuntu x64 (Vulkan) - Ubuntu arm64 (Vulkan) - Ubuntu x64 (CUDA 12) - CUDA 12.8 libraries - Ubuntu x64 (CUDA 13) - CUDA 13.5 libraries - Ubuntu arm64 (CUDA 13) - CUDA 13.5 libraries - Ubuntu x64 (ROCm 10.0) - Ubuntu x64 (OpenVINO) - Ubuntu x64 (SYCL FP32) - Ubuntu x64 (SYCL FP16) - Linux arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU) - setup guide
Android: - Android arm64 (CPU) - Android arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU) - setup guide
Windows: - Windows x64 (CPU) - Windows arm64 (CPU) - Windows arm64 (OpenCL Adreno) - Windows x64 (CUDA 12) - CUDA 12.4 DLLs - Windows x64 (CUDA 13) - CUDA 13.5 DLLs - Windows arm64 (CUDA 13) - CUDA 13.5 DLLs - Windows x64 (Vulkan) - Windows arm64 (Vulkan) - Windows x64 (OpenVINO) - Windows x64 (SYCL) - Windows x64 (ROCm 10.0)
openEuler: - DISABLED - openEuler x86 (310p) - openEuler x86 (910b, ACL Graph) - openEuler aarch64 (310p) - openEuler aarch64 (910b, ACL Graph)
UI: - UI