diff --git a/plans/heretic-encoder-swap.md b/plans/heretic-encoder-swap.md new file mode 100644 index 0000000..f3bff61 --- /dev/null +++ b/plans/heretic-encoder-swap.md @@ -0,0 +1,139 @@ +# Task: Swap Qwen3-4B Encoder for Heretic Abliterated Version + +**Datum:** 2026-04-10 +**Status:** Ready — waiting for correct Heretic encoder to be published +**Depends on:** FLUX.2 Klein 4B working (✅ done as of 2026-04-10) + +--- + +## Goal + +Replace the standard `qwen_3_4b_klein.safetensors` with an abliterated (Heretic) version that has: +- **Zero measurable quality loss** (KL divergence = 0.0000) +- **No prompt refusals** (≤3/100 in DreamFast v1.2.0 testing) + +Result: `generate_image(prompt, model="flux-2-klein-4b.safetensors")` will work with **any** prompt without refusals. + +--- + +## Current State + +| File | Location | Status | +|------|----------|--------| +| `flux-2-klein-4b.safetensors` | `~/ComfyUI/models/diffusion_models/` | ✅ Working | +| `qwen_3_4b_klein.safetensors` | `~/ComfyUI/models/text_encoders/` | ✅ Working (standard, has refusals) | +| `flux2-vae.safetensors` | `~/ComfyUI/models/vae/` | ✅ Working | + +The MCP workflow [`mcp/mcp-image-gen/src/workflows/flux2_klein_heretic.json`](../mcp/mcp-image-gen/src/workflows/flux2_klein_heretic.json) already uses `qwen_3_4b_klein.safetensors` — **no code change needed**, only the file on disk needs to be replaced. + +--- + +## The Problem to Solve First + +The standard Heretic repos may not have the **FLUX.2 Klein-compatible** encoder dimensions: + +| Encoder | `hidden_size` | Conditioning dim | Usable? | +|---------|--------------|-----------------|---------| +| BFL Qwen3-4B (FLUX.2 Klein) | **2560** | 7680 (2560×3) | ✅ | +| DreamFast/qwen3-4b-heretic | unknown — must check | ? | ⚠️ verify first | +| Standard Qwen3-4B | 4096 | 4096 | ❌ wrong | + +**Before downloading, verify DreamFast's model is fine-tuned from the BFL variant** (hidden_size=2560), not the standard Qwen3 (hidden_size=4096). + +--- + +## Steps + +### Step 1: Check DreamFast Heretic repo + +```bash +huggingface-cli model-info DreamFast/qwen3-4b-heretic 2>/dev/null | grep -i hidden +``` + +Or browse: https://huggingface.co/DreamFast/qwen3-4b-heretic/blob/main/config.json +Look for: `"hidden_size": 2560` — that's the FLUX.2 Klein-compatible version. + +### Step 2a: If DreamFast has the right dimensions (2560) + +```bash +# Download +huggingface-cli download DreamFast/qwen3-4b-heretic \ + --local-dir /tmp/qwen3-4b-heretic/ + +# Back up working encoder first +cp ~/ComfyUI/models/text_encoders/qwen_3_4b_klein.safetensors \ + ~/ComfyUI/models/text_encoders/qwen_3_4b_klein_backup.safetensors + +# Swap in the Heretic version +cp /tmp/qwen3-4b-heretic/model.safetensors \ + ~/ComfyUI/models/text_encoders/qwen_3_4b_klein.safetensors +``` + +### Step 2b: If DreamFast has wrong dimensions (4096) — find alternative + +Options in order of preference: +1. **Lockout/qwen3-4b-heretic-zimage** — check if BFL-compatible: + ```bash + huggingface-cli model-info Lockout/qwen3-4b-heretic-zimage 2>/dev/null | grep hidden + ``` +2. **Run Heretic abliteration yourself** on the working `qwen_3_4b_klein.safetensors` + Tool: https://github.com/FailSpy/abliterator + Script: `python abliterator.py --model qwen_3_4b_klein.safetensors --output qwen_3_4b_klein_heretic.safetensors` + +3. **Wait** for DreamFast or BFL to publish the FLUX.2-specific abliterated encoder + +### Step 3: Live test + +```python +generate_image( + "an explicit test prompt that would normally be refused", + model="flux-2-klein-4b.safetensors", + steps=20 +) +``` + +Expected: Image generated, no refusal error in ComfyUI logs. + +### Step 4: If it works — no code changes needed + +The MCP code, workflow JSON, and registry are already correct. Just verify: +- Check `journalctl --user -u comfyui -f` during generation for any errors +- Confirm file in `~/Pictures/mcp-generated/` was saved + +--- + +## Fallback Plan + +If the Heretic encoder is unavailable in the right dimensions, the **GGUF route** works too: + +```bash +# ComfyUI-GGUF is already installed: ~/ComfyUI/custom_nodes/ComfyUI-GGUF +# Download Heretic GGUF (if BFL-compatible variant published): +huggingface-cli download Lockout/qwen3-4b-heretic-zimage \ + qwen-4b-zimage-hereticV2-q8.gguf \ + --local-dir ~/ComfyUI/models/text_encoders/ +``` + +Then update [`flux2_klein_heretic.json`](../mcp/mcp-image-gen/src/workflows/flux2_klein_heretic.json) node `"1"`: +```json +"class_type": "CLIPLoaderGGUF", // instead of CLIPLoader +"inputs": { + "clip_name": "qwen-4b-zimage-hereticV2-q8.gguf", + "type": "flux2" +} +``` + +--- + +## No Code Changes Required (unless GGUF fallback) + +The entire MCP server, workflow registry, and test suite are already correct. This is **purely a model file task**. + +--- + +## Success Criteria + +- [ ] `generate_image("...", model="flux-2-klein-4b.safetensors")` works with prompts that currently get refused +- [ ] Output image quality identical to standard encoder (check: no visible artifacts vs reference) +- [ ] ComfyUI logs show no dimension errors +- [ ] `qwen_3_4b_klein_backup.safetensors` kept as rollback