Rewrite dead HF links to live successors, refresh stale Zen4 / old Zen-3 chat lineups to current Zen5 canonical + Zen3 specialty, and strip upstream brand mentions from user-facing prose (attribution and base_model chains preserved).
Co-authored-by: zooqueen <worringantje@gmail.com>
HF repo renamed to align with the canonical SKU ladder: the IQ2-imatrix
DS4-Flash weights are the zen5-pro tier (not zen5-flash, which is now
Qwen3-4B). Old zen-5-flash-gguf redirects to zen-5-pro-gguf on HF.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Switches the default REPO in download_model.sh from antirez/deepseek-v4-gguf
to zenlm/zen-5-flash-gguf so 'make + ./download_model.sh q2-imatrix' pulls
the Zen5-branded mirror. UPSTREAM_REPO=antirez/deepseek-v4-gguf is preserved
as a reference and as a manual fallback.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Parse Anthropic tool_use blocks by their own type field instead of relying on the enclosing message role being parsed first.
Some clients serialize messages as content-before-role, which made full-history tool_result replays look like unknown live-only continuations.
Fixes#127.
Keep normal CUDA context buffers on device allocations, but route very large KV-cache tensors through managed memory so million-token contexts do not starve unified-memory systems during graph/session allocation.
The fallback is scoped to the long-lived KV/cache tensors and logs when it is used because it may reduce performance.
Tested on 0.180 with:
- make cpu
- make -B cuda-spark
- make cuda-regression
- ./ds4_test --server --metal-kernels
- ./ds4_test --logprob-vectors --tool-call-quality
- ds4-bench ctx-alloc 32768, 250000, and 1000000
- ds4-server --ctx 1000000 startup smoke
(cherry picked from commit 0b248a65c07d21f2fc8ff4815bd8b75af26719f9)
Project sampled DSML tool calls to Anthropic SSE tool_use blocks while keeping raw DSML as the parser/cache source of truth.
Reuse streamed tool ids for final parsed calls so tool_result continuation still matches live state.
Preserve Responses namespace metadata and tool_search calls while rendering DSML-safe internal tool names. Replay function_call, hosted tool, and tool_search_output items into the shared chat/tool path so Codex and Pi can round-trip tool calls without losing KV-cache prefix reuse. Document the /v1/responses endpoint and add server unit coverage for namespace, tool_search, and replay output shapes.
Replace the default long-context continuation check with a deterministic prose-story retrieval test. The fixture embeds spelled-out person-number assignments in a long rendered prompt, and ds4_test now validates the generated Name=number list instead of brittle sampled prose.
Build score_official against the CUDA runtime on Linux and select the CUDA backend there, while keeping the existing Metal path on macOS.\n\nCorrectness: make -C gguf-tools quality-score; gguf-tools/quality-testing/score_official ds4flash.gguf /tmp/ds4_quality_smoke/manifest.tsv /tmp/ds4_quality_smoke/scores.tsv 16384.
Fold the successful CUDA selector/top-k/indexed-attention changes into one clean commit. This excludes rejected experiment commits and the local prefill-slope work log.\n\nMeasured on GB10 with speed-bench/promessi_sposi.txt, 2048-token append chunks: 32K prefill improved from 255.61 tok/s on origin/main to 346.49 tok/s. Full-curve average improved from 316.39 tok/s to 369.76 tok/s. 32K full prompt + 128-token generation prefill improved from 312.87 tok/s to 368.43 tok/s, while generation stayed neutral at 12.49 -> 12.48 tok/s.\n\nCorrectness: make cuda-regression; ./ds4_test --logprob-vectors --tool-call-quality; ./ds4_test --server --metal-kernels.
Broaden the DS4 imatrix prompt dataset with provider-neutral agent/tool traffic, multi-language programming prompts, algorithm recall, Bash scripting, and multilingual translation tasks.
Remove duplicate rendered prompts and avoid provider-specific client references in the generated calibration corpus. This improves calibration coverage without claiming to fix a distributed GGUF bug.
The PR #110 direction is useful, but the old block16 routed-down subvariant should not remain as an env-gated alternate path. It had prior top-logit instability, and a README prompt spot check showed different greedy first-token behavior when forced.
Keep the faster default non-block16 tile16 path and remove DS4_CUDA_MOE_DOWN_BLOCK16 plus the unused block16 kernels.
Validation on GB10: make cuda-regression; ./ds4_test --logprob-vectors --tool-call-quality --metal-kernels. Speed after removal: promessi_sposi 2048 prefill 401.32 tok/s, gen 14.00 tok/s; 8192 prefill 375.17 tok/s, gen 13.66 tok/s; README 8247 prefill 361.13 tok/s.
Keep the tile16/row2048 MoE down path enabled for batched prefill, but only use the block16 subvariant when DS4_CUDA_MOE_DOWN_BLOCK16 is set. On GB10 the non-block16 variant was consistently faster while preserving the old path as an explicit diagnostic option.
Zero compressor state_kv buffers with cudaMemsetAsync instead of launching the generic fill_f32 kernel. The default stream preserves ordering while avoiding a small fill kernel in prefill/replay state setup.
CUDA graph/session tensors are only accessed through backend read/write/copy/fill APIs. Allocate them with cudaMalloc instead of managed memory and add a backend fill_f32 hook so CUDA can initialize tensors device-side while Metal preserves its existing host-visible behavior.
On Linux, plain make now prints the explicit build targets instead of silently choosing a CUDA build. Add cuda-spark for the GB10 path without an explicit nvcc -arch, cuda-generic for native CUDA builds, and keep make cuda CUDA_ARCH=... for explicit arch selection.
Measured on GB10 with make cuda-spark: README.md prompt, 8247 input tokens, 344.90 tok/s prefill. Test: make cuda-regression.
Use a smaller default Q8->F16 cache reserve on 112+ GiB CUDA devices so GB10 keeps the fast cached weight path by default. Measured on README.md with the DS4 Flash GGUF: prefill 316.85 t/s, generation 13.83 t/s after the cleanup.
Retain the useful CUDA pieces from cghart's branch without the disabled decode experiments: add the fd-cache model-map guard for base+MTP loads, support the Q4_K routed expert decode path used by the MTP draft head, and use CUDA for non-Apple ds4_test runs.
Add routed-MoE imatrix collection to the DS4 graph path, including dataset generation docs and the tracked calibration prompt corpus.
Add a standalone DeepSeek V4 Flash HF-to-GGUF quantizer that reuses GGUF template metadata, emits DS4 Q2/Q4 recipes, supports imatrix-aware routed expert quantization, and internalizes the required Q8_0, Q4_K, Q2_K, and IQ2_XXS quantizers instead of vendoring GGML.
Document quantizer provenance, output formats, and the DS4-specific imatrix mapping used by the generated GGUF files.
The CUDA Q8->F16 expansion cache is only an acceleration path, but on memory-constrained cards it could consume enough VRAM to make later allocations or cuBLAS calls fail. Add a budget check before expanding weights, keeping a default reserve of max(4 GiB, 5% VRAM), and fall back to native Q8 kernels when the cache is capped or memory is tight.
Expose DS4_CUDA_Q8_F16_CACHE_MB to disable or cap the cache and DS4_CUDA_Q8_F16_CACHE_RESERVE_MB to override the reserve. If allocation, dequantization, or cuBLAS use of the cached path fails, release/disable the optional cache and continue through the Q8 kernel path instead of failing the request.
Fixes#78
Implements the Responses API endpoint that Codex CLI (and other modern
OpenAI tooling) speaks instead of /v1/chat/completions. The wire format
is documented in OpenAI's Responses API; this implementation has been
iterated against the Codex CLI binary's SSE parser shape until no
remaining schema gaps were found.
Request parsing (parse_responses_request, parse_responses_input):
- Accepts the typed input array (message, function_call,
function_call_output, reasoning, custom_tool_call(_output),
local_shell_call(_output), web_search_call(_output),
tool_search_call(_output), image_generation_call(_output),
compaction, context_compaction).
- Maps hosted-tool history to function_call/function_call_output so
prior actions survive across turns; rejects unknown item types and
non-completed status with 400 to avoid silent context loss.
- Strict content-array parsing: only string|null|array of recognized
text blocks (input_text/output_text/text/summary_text/
reasoning_text); rejects non-text modalities (input_image/file/
audio) instead of accepting an empty prompt.
- Merges adjacent function_call items into the preceding assistant
message so text + tool-call turns render as a single assistant
block.
- Honors reasoning.effort (incl. "minimal"/"none") and gates
reasoning summary surface on reasoning.summary opt-in.
- Rejects previous_response_id, conversation, and forced tool_choice
explicitly (constrained decoding / persisted state not supported).
Output (responses_sse_*, responses_final_response):
- Emits the full streaming lifecycle: response.created,
output_item.added/.done, reasoning_summary_part.added/.done,
reasoning_summary_text.delta/.done, content_part.added/.done,
output_text.delta/.done, function_call_arguments.delta/.done,
response.completed.
- Branches the terminal event by finish reason: response.failed for
errors and response.incomplete with reason "max_tokens" for length.
- Every event carries sequence_number; every output_text part carries
annotations:[]; function_call output_item.added ships with an empty
arguments string (full args arrive via function_call_arguments.done
and output_item.done), and item ids are stable across added/done.
- Tracks whether </think> was actually observed so a truncated stream
marks the reasoning item incomplete instead of "completed".
- Recovers gracefully when the DSML tool parse fails after the model
was suppressed at the tool marker: the suppressed tail is flushed
as additional output_text deltas so the streamed message matches
output_item.done.
Tested by 25 rounds of /codex:adversarial-review against the same
client this is meant to feed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
z