cuda: restore q8 fp16 cache on large-memory devices

Use a smaller default Q8->F16 cache reserve on 112+ GiB CUDA devices so GB10 keeps the fast cached weight path by default. Measured on README.md with the DS4 Flash GGUF: prefill 316.85 t/s, generation 13.83 t/s after the cleanup.
This commit is contained in:
cghart
2026-05-12 22:18:48 +02:00
committed by antirez
parent a5f4f761b7
commit f8b4ed635d
+4
View File
@@ -359,6 +359,10 @@ static uint64_t cuda_q8_f16_cache_reserve_bytes(uint64_t total_bytes) {
const uint64_t reserve = cuda_parse_mib_env("DS4_CUDA_Q8_F16_CACHE_RESERVE_MB", &present);
if (present) return reserve;
if (total_bytes >= 112ull * 1024ull * 1024ull * 1024ull) {
return 512ull * 1048576ull;
}
/* The expanded Q8->F16 cache is only an acceleration path. Keep enough
* device memory free for cuBLAS workspaces, transient graph buffers, and
* driver bookkeeping instead of letting optional cached weights consume the