fix(sd1_tokenizer): restore canonical CLIP-L vocab (unbreaks FLUX.1 DualCLIPLoader)

Browse Source

The bundled studio/sd1_tokenizer/vocab.json had the token 'comfy</w>' scrubbed
(49407 tokens vs canonical 49408) while merges.txt still contained the 'com fy'
merge, so CLIPTokenizer.from_pretrained raised 'Error while initializing BPE:
Token comfy</w> out of vocabulary' for ANY FLUX.1 CLIP load (DualCLIPLoader type
'flux'). Only ever hit now because FLUX.1-Fill is the first CLIP-L consumer here
(Qwen/Klein use other tokenizers). Restored the canonical openai/clip-vit-large-
patch14 vocab.json (49408 tokens). Required by couture_best.json.

...

This commit is contained in:

hanzo-dev

2026-07-05 18:34:07 -07:00

parent 0337677c77

commit 1063ee3d0e

1 changed files with 1 additions and 49410 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

studio/sd1_tokenizer/vocab.json

+1 -49410

File diff suppressed because one or more lines are too long