Photomaker error: "GGML_ASSERT(cgraph->n_nodes < cgraph->size) failed"

[Disonantemus]

When trying to use Photomaker, sd fails and abort.

Steps to reproduce

Download models (URLs found in this same repo)

• sdxlUnstableDiffusers_v11.safetensors (to generate)
• sdxl_vae.safetensors (to fix black output)
• photomaker-v1.safetensors (to use photomaker)

Create reference image "in.png"

magick logo: -crop 512x512+50+0 -gravity Center -background white -extent 512x512 in.png

Convert model to gguf (too large for my hardware)

sd -M convert -m sdxlUnstableDiffusers_v11.safetensors --type q4_0 -o sdxlUnstableDiffusers_v11-q4_0.gguf

Generation Command (failed and aborted)

sd -m sdxlUnstableDiffusers_v11-q4_0.gguf -p "man img, man with futuristic clothes" --steps 10 --cfg-scale 5.0 --vae sdxl_vae.safetensors --stacked-id-embd-dir photomaker-v1.safetensors --input-id-images-dir in.png
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce GTX 1660 SUPER (NVIDIA) | uma: 0 | fp16: 1 | warp size: 32 | shared memory: 49152 | matrix cores: none
[INFO ] stable-diffusion.cpp:197  - loading model from 'sdxlUnstableDiffusers_v11-q4_0.gguf'
[INFO ] model.cpp:905  - load sdxlUnstableDiffusers_v11-q4_0.gguf using gguf format
[INFO ] stable-diffusion.cpp:232  - loading vae from 'sdxl_vae.safetensors'
[INFO ] model.cpp:908  - load sdxl_vae.safetensors using safetensors format
[INFO ] stable-diffusion.cpp:244  - Version: SDXL
[INFO ] stable-diffusion.cpp:277  - Weight type:                 q4_0
[INFO ] stable-diffusion.cpp:278  - Conditioner weight type:     q4_0
[INFO ] stable-diffusion.cpp:279  - Diffusion model weight type: q4_0
[INFO ] stable-diffusion.cpp:280  - VAE weight type:             f32
[INFO ] model.cpp:908  - load photomaker-v1.safetensors using safetensors format
[INFO ] lora.hpp:117  - loading LoRA from 'photomaker-v1.safetensors'
  |=====================================>            | 1134/1527 - 0.00it/s[INFO ] stable-diffusion.cpp:392  - loading stacked ID embedding (PHOTOMAKER) model file from 'photomaker-v1.safetensors'
[INFO ] model.cpp:908  - load photomaker-v1.safetensors using safetensors format
  |==================================================| 4168/4168 - 250.00it/s
[INFO ] stable-diffusion.cpp:503  - total params memory size = 3998.25MB (VRAM 3998.25MB, RAM 0.00MB): clip 778.42MB(VRAM), unet 1881.88MB(VRAM), vae 94.47MB(VRAM), controlnet 0.00MB(VRAM), pmid 1243.48MB(VRAM)
[INFO ] stable-diffusion.cpp:522  - loading model from 'sdxlUnstableDiffusers_v11-q4_0.gguf' completed, taking 5.05s
[INFO ] stable-diffusion.cpp:556  - running in eps-prediction mode
[INFO ] stable-diffusion.cpp:690  - Attempting to apply 0 LoRAs
[INFO ] stable-diffusion.cpp:1246 - apply_loras completed, taking 0.00s
/usr/src/debug/stable-diffusion.cpp-vulkan-git/stable-diffusion.cpp/ggml/src/ggml.c:5764: GGML_ASSERT(cgraph->n_nodes < cgraph->size) failed
ptrace: Operation not permitted.
No stack.
The program is not being run.
Aborted (core dumped)

Converted model was tested ok (with same file & vae)

sd -m sdxlUnstableDiffusers_v11-q4_0.gguf --vae sdxl_vae.safetensors -p "man img, man with futuristic clothes" --steps 10

Questions

• Am I doing something wrong?
• Where can I find more information/community using sd.cpp? besides this repo.
• Is possible use ip adapter with sd.cpp?

---

System:

OS: Arch Linux x86_64
Kernel: Linux 6.12.24-1-lts
Shell: bash 5.2.37
WM: dwm (X11)
Terminal: tmux 3.5a
CPU: Intel(R) Core(TM) i7-4790 (8) @ 3.60 GHz
GPU: NVIDIA GeForce GTX 1660 SUPER [Discrete] (6GB)
Memory: 2.47 GiB / 15.56 GiB (16%)
Locale: en_US.UTF-8

[stduhpf]

Photomaker includes a LoRA, and SDXL LoRAs are kinda broken right now. The workaround is to increase LORA_GRAPH_SIZE to a bigger value (see #658).

Am I doing something wrong?

No. this is a known bug.

Where can I find more information/community using sd.cpp? besides this repo.

I think this is pretty much the only place

Is possible use ip adapter with sd.cpp?

No(t yet).

[Disonantemus]

Photomaker includes a LoRA, and SDXL LoRAs are kinda broken right now. The workaround is to increase LORA_GRAPH_SIZE to a bigger value (see #679).

• I didn't know that Photomaker includes a LoRA!
• Yeah ... I did see some Issues/PR about LORA_GRAPH_SIZE.
• Then, it's not possible 4 me to use Photomaker, because I'm not a programmer.
• I'll wait to the commit.

Thanks!

[Disonantemus]

Photomaker includes a LoRA, and SDXL LoRAs are kinda broken right now. The workaround is to increase LORA_GRAPH_SIZE to a bigger value (see #679).

• I did compile your fork
• Now I can generate without errors!
• Thanks again!

Small problem

If you use just one image, you can't put it directly, like this:

--input-id-images-dir reference_id.png

I did thatn and didn't work, because sd.cpp didn't find it:

[WARN ] stable-diffusion.cpp:1559 - Provided PhotoMaker model file, but NO input ID images
[WARN ] stable-diffusion.cpp:1560 - Turn off PhotoMaker

You need to put the reference image(s) in a folder, like this:

tree
.
  ├─ ID_images
  │  ├─ reference_id.png

And just put the folder in the command:

--input-id-images-dir ID_images/

I didn't get that from the example here, but now is working as expected.

Preparation

• Convert realDream_sdxl6 to q8_0
• Convert photomaker-v1 to q8_0

Command

./sd -m realDream_sdxl6-q8_0.gguf -p "man img, man with futuristic clothes" --cfg-scale 5.0 --stacked-id-embd-dir ../pm/photomaker-v1-q8_0.gguf --input-id-images-dir ID_images/ --steps 10 -b 3 --style-ratio 10 --vae-on-cpu --diffusion-fa

Reference image:

Generated images:

1	2	3

[stduhpf]

I just realized I linked to the wrong PR 😅 I meant #658, not #679 .