Multideploy: Deploy to multiple hooks of the same type #6241

tomo2403 · 2025-02-21T18:28:55Z

This hook allows the user to deploy certificates to multiple services at once. It can store configurations for numerous services, even for the same hook.

Example

You have three Docker containers and a Synology NAS (DSM). However, using the docker and synology_dsm hooks, you can only deploy to one Docker container with renewals. This problem is solved with Multideploy.

Sample config file

The file can be named multideploy.yml or multideploy.yaml. It is stored in the domain folder. $DOMAIN_DIR is a variable that allows deploying certificated to a dir named after the certificate's domain to make changes easier.

version: 1.0

configs:
  - name: "default"
    services:
      - "webserver"
      - "webserver2"

services:
  - name: "webserver"
    hook: "docker"
    environment:
      - DEPLOY_DOCKER_CONTAINER_LABEL: "sh.acme.autoload.domain=example.com"
      - DEPLOY_DOCKER_CONTAINER_KEY_FILE: "/etc/nginx/ssl/$DOMAIN_DIR/key.pem"
      - DEPLOY_DOCKER_CONTAINER_CERT_FILE: "/etc/nginx/ssl/$DOMAIN_DIR/cert.pem"
      - DEPLOY_DOCKER_CONTAINER_CA_FILE: "/etc/nginx/ssl/$DOMAIN_DIR/ca.pem"
      - DEPLOY_DOCKER_CONTAINER_FULLCHAIN_FILE: "/etc/nginx/ssl/$DOMAIN_DIR/full.pem"
      - DEPLOY_DOCKER_CONTAINER_RELOAD_CMD: "service nginx force-reload"
  - name: "webserver2"
    hook: "docker"
    environment:
      - DEPLOY_DOCKER_CONTAINER_LABEL: "sh.acme.autoload.domain=example.com"
      - DEPLOY_DOCKER_CONTAINER_KEY_FILE: "/etc/nginx/ssl/$DOMAIN_DIR/key.pem"
      - DEPLOY_DOCKER_CONTAINER_CERT_FILE: "/etc/nginx/ssl/$DOMAIN_DIR/cert.pem"
      - DEPLOY_DOCKER_CONTAINER_CA_FILE: "/etc/nginx/ssl/$DOMAIN_DIR/ca.pem"
      - DEPLOY_DOCKER_CONTAINER_FULLCHAIN_FILE: "/etc/nginx/ssl/$DOMAIN_DIR/full.pem"
      - DEPLOY_DOCKER_CONTAINER_RELOAD_CMD: "service nginx force-reload"

Wiki

Please tell me if you will merge this first before I start writing a wiki entry for this. thx
https://github.com/acmesh-official/acme.sh/wiki/deployhooks#36-deploying-to-multiple-services-with-the-same-hooks

Deployhook docker

gilman88 · 2025-02-27T05:25:04Z

+1 on this kind of capability at least for wildcard certs. I was recently looking at a situation with multiple mikrotik routers where this would have been helpful.

Neilpang · 2025-02-28T18:40:23Z

please update the wiki page first.

tomo2403 · 2025-03-01T12:25:26Z

@Neilpang done

Neilpang · 2025-03-02T12:19:54Z

let's remove the configs part:

configs:
  - name: "default"
    services:
      - "webserver"
      - "webserver2"

It's not necessary.

In the yaml example, please add some other hooks, not just docker hook. because it should work with any hooks.

Don't use a hardcoded 'multideploy.yml" file, let's make it a env variable, just like the others:

export  DEPLOY_YAML="/path/to/my/multideploy.yaml"
acme.sh --deploy -d xxxx.com  --deploy-hook  multideploy

You can just copy the "$DEPLOY_YAML" file to the domain folder, it will be easier for the user to use.

tomo2403 · 2025-03-02T13:35:43Z

The configurations (configs) are intended to simplify the testing/staging process. They allow the user to quickly select or deselect the services the certificate should be deployed to without having to comment out every line. Soon, I also want to implement an overriding functionality similar to Docker (compose.override.yaml). Configurations will then be even more necessary to enhance testing and deployment further when multiple certificates are deployed to the same services. This would allow the user to maintain a minimal, non-redundant configuration. That is also the reason for the hardcoded filepath.

Do you agree with this @Neilpang?

Neilpang · 2025-03-04T19:49:59Z

The configurations (configs) are intended to simplify the testing/staging process.

no, this is too complicated.

That is also the reason for the hardcoded filepath.

no, use the env variable to pass value. it's the same way as others.

tomo2403 · 2025-03-27T18:33:15Z

@Neilpang, I removed configs and introduced a variable deploy file name. The wiki is now up to date.

zeocax

tested and working fine for me

Neilpang · 2025-04-13T12:11:12Z

deploy/multideploy.sh

+  _debug _cfullchain "$_cfullchain"
+  _debug _cpfx "$_cpfx"
+
+  DOMAIN_DIR=$_cdomain


why do you need this DOMAIN_DIR ?

This can be used in the deploy file to specify the path for a cert if you are going to have multiple certs on the same service. This makes it easier to copy the deploy file to another domain dir.

which deploy file uses it ?

have a look at the sample config above

which one ?

first comment after pr opening. Both services use DOMAIN_DIR. I use it for all my domains because they are deployed to the same service. If I change something in one deploy file I can just copy the file to all other domains without having to change the path for each domain in my docker container

deploy/multideploy.sh

Neilpang · 2025-04-13T12:15:50Z

deploy/multideploy.sh

+#   $1 - A YAML formatted string containing environment variable key-value pairs.
+# Usage:
+#   _clear_envs "<yaml_string>"
+_clear_envs() {


no need to clear

If a hook has optional envs and you have two services that use the same hook, but one service uses the optional envs and the other doesn't, both services will use the optional envs because they are saved by the first service. Clearing them ensures that multideploy only uses the envs specified in the deploy file

deploy/multideploy.sh

…oy-yaml

Gerporgl · 2025-05-21T01:22:29Z

I tested this and I'm looking forward to have this merged.
I needed this to auto renew and publish my wildcard certificate to my synology and nginx servers.
Very useful.

invario · 2025-05-27T14:36:17Z

@tomo2403 First, thanks so much for putting this together!

I grabbed the file and am attempting to use it within a docker container after I added yq in. My multideploy.yml looks like this (very simple for testing purposes)

# check your acme.sh version in wiki!
version: 1.0

services:
  - name: "npm"
    hook: "docker"
    environment:
      - DEPLOY_DOCKER_CONTAINER_LABEL: "sh.acme.autoload.service=example.com"
      - DEPLOY_DOCKER_CONTAINER_KEY_FILE: "/data/tls/custom/npm-1/privkey.pem"
      - DEPLOY_DOCKER_CONTAINER_FULLCHAIN_FILE: "/data/tls/custom/npm-1/fullchain.pem"
      - DEPLOY_DOCKER_CONTAINER_RELOAD_CMD: "nginx -s reload"

I then executed the following:
acme.sh --deploy -d example.com --deploy-hook multideploy

But unfortunately, it is returning this:

[Tue May 27 10:30:50 EDT 2025] The domain 'example.com' seems to already have an ECC cert, let's use it.
[Tue May 27 10:30:50 EDT 2025] MULTIDEPLOY_FILENAME is not set, so I will use 'multideploy.yml'.
[Tue May 27 10:30:50 EDT 2025] Deploying to 'npm' using 'docker'
[Tue May 27 10:30:50 EDT 2025] The DEPLOY_DOCKER_CONTAINER_LABEL variable is not defined, we use this label to find the container.
[Tue May 27 10:30:50 EDT 2025] See: https://github.com/acmesh-official/acme.sh/wiki/deploy-to-docker-containers
[Tue May 27 10:30:50 EDT 2025] Container id:
[Tue May 27 10:30:50 EDT 2025] can not find container id
[Tue May 27 10:30:50 EDT 2025] Error deploying for domain: example.com
[Tue May 27 10:30:50 EDT 2025] Error encountered while deploying.
[Tue May 27 10:30:50 EDT 2025] Success

So it seems like the environment variable isn't being set for some reason, very odd.

tomo2403 · 2025-05-27T14:46:54Z

Hello @invario,

So it seems like the environment variable isn't being set for some reason, very odd.

How many times have you tried this?
Could you please share the contents of your example.com.conf file?
Also, please post the output of your command with debug enabled:

acme.sh --deploy -d example.com --deploy-hook multideploy --debug 3

invario · 2025-05-27T14:55:08Z

Hello @invario,

So it seems like the environment variable isn't being set for some reason, very odd.

How many times have you tried this?

At least 5 times.. Even completely wiped the multideploy.yml file and copy and pasted from your example and manually typed it in just in case there were something off.

Could you please share the contents of your example.com.conf file?

Le_Domain='example.com'
Le_Alt='*.example.com'
Le_Webroot='dns_cf'
Le_PreHook=''
Le_PostHook=''
Le_RenewHook=''
Le_API='https://acme-v02.api.letsencrypt.org/directory'
Le_Keylength='ec-256'
Le_OrderFinalize='https://acme-v02.api.letsencrypt.org/acme/finalize/**BLOCKED**'
CF_Token='**BLOCKED**'
CF_Account_ID=''
CF_Zone_ID='**BLOCKED**'
Le_LinkOrder='https://acme-v02.api.letsencrypt.org/acme/order/2421948647/387838364687'
Le_LinkCert='https://acme-v02.api.letsencrypt.org/acme/cert/065ca48054c135c1fc336ed5367345528824'
Le_CertCreateTime='1748228453'
Le_CertCreateTimeStr='2025-05-26T03:00:53Z'
Le_NextRenewTimeStr='2025-07-24T03:00:53Z'
Le_NextRenewTime='1753326053'
Le_DeployHook='multideploy'

The above is what the example.com.conf looks like prior to running the deploy command. Aftewards, an additional 3 line are added:

DEPLOY_DOCKER_CONTAINER_KEY_FILE='/data/tls/custom/npm-1/privkey.pem'
DEPLOY_DOCKER_CONTAINER_FULLCHAIN_FILE='/data/tls/custom/npm-1/fullchain.pem'
DEPLOY_DOCKER_CONTAINER_RELOAD_CMD='nginx -s reload'

Also, please post the output of your command with debug enabled:
acme.sh --deploy -d example.com --deploy-hook multideploy --debug 3

Debug log as follows:

[Tue May 27 10:45:33 EDT 2025] Let's find the script directory.
[Tue May 27 10:45:33 EDT 2025] _SCRIPT_='/usr/local/bin/acme.sh'
[Tue May 27 10:45:33 EDT 2025] _script='/root/.acme.sh/acme.sh'
[Tue May 27 10:45:33 EDT 2025] _script_home='/root/.acme.sh'
[Tue May 27 10:45:33 EDT 2025] Using default home: /root/.acme.sh
[Tue May 27 10:45:33 EDT 2025] Using config home: /acme.sh
[Tue May 27 10:45:33 EDT 2025] LE_WORKING_DIR='/root/.acme.sh'
https://github.com/acmesh-official/acme.sh
v3.1.2
[Tue May 27 10:45:33 EDT 2025] Running cmd: deploy
[Tue May 27 10:45:33 EDT 2025] Using config home: /acme.sh
[Tue May 27 10:45:33 EDT 2025] default_acme_server
[Tue May 27 10:45:33 EDT 2025] ACME_DIRECTORY='https://acme.zerossl.com/v2/DV90'
[Tue May 27 10:45:33 EDT 2025] _ACME_SERVER_HOST='acme.zerossl.com'
[Tue May 27 10:45:33 EDT 2025] _ACME_SERVER_PATH='v2/DV90'
[Tue May 27 10:45:33 EDT 2025] The domain 'example.com' seems to already have an ECC cert, let's use it.
[Tue May 27 10:45:33 EDT 2025] DOMAIN_PATH='/acme.sh/example.com_ecc'
[Tue May 27 10:45:33 EDT 2025] DOMAIN_CONF='/acme.sh/example.com_ecc/example.com.conf'
[Tue May 27 10:45:33 EDT 2025] _deployApi='/root/.acme.sh/deploy/multideploy.sh'
[Tue May 27 10:45:33 EDT 2025] _cdomain='example.com'
[Tue May 27 10:45:33 EDT 2025] _ckey='/acme.sh/example.com_ecc/example.com.key'
[Tue May 27 10:45:33 EDT 2025] _ccert='/acme.sh/example.com_ecc/example.com.cer'
[Tue May 27 10:45:33 EDT 2025] _cca='/acme.sh/example.com_ecc/ca.cer'
[Tue May 27 10:45:33 EDT 2025] _cfullchain='/acme.sh/cexample.com_ecc/fullchain.cer'
[Tue May 27 10:45:33 EDT 2025] _cpfx='/acme.sh/example.com_ecc/example.com.pfx'
[Tue May 27 10:45:33 EDT 2025] DOMAIN_DIR='example.com_ecc'
[Tue May 27 10:45:33 EDT 2025] MULTIDEPLOY_FILENAME is not set, so I will use 'multideploy.yml'.
[Tue May 27 10:45:33 EDT 2025] Deploy file='/acme.sh/example.com_ecc/multideploy.yml'
[Tue May 27 10:45:33 EDT 2025] Deploy file version is compatible: 1.0
[Tue May 27 10:45:33 EDT 2025] Services='npm'
[Tue May 27 10:45:33 EDT 2025] Config has services.
[Tue May 27 10:45:33 EDT 2025] Checking service='npm'
[Tue May 27 10:45:33 EDT 2025] Service='npm'
[Tue May 27 10:45:33 EDT 2025] SERVICE='npm'
[Tue May 27 10:45:34 EDT 2025] HOOK='docker'
[Tue May 27 10:45:34 EDT 2025] Deploying to 'npm' using 'docker'
[Tue May 27 10:45:34 EDT 2025] _deployApi='/root/.acme.sh/deploy/docker.sh'
[Tue May 27 10:45:34 EDT 2025] _cdomain='example.com'
[Tue May 27 10:45:34 EDT 2025] DEPLOY_DOCKER_CONTAINER_LABEL
[Tue May 27 10:45:34 EDT 2025] The DEPLOY_DOCKER_CONTAINER_LABEL variable is not defined, we use this label to find the container.
[Tue May 27 10:45:34 EDT 2025] See: https://github.com/acmesh-official/acme.sh/wiki/deploy-to-docker-containers
[Tue May 27 10:45:34 EDT 2025] Try use /var/run/docker.sock
[Tue May 27 10:45:34 EDT 2025] _cversion='8.12.1'
[Tue May 27 10:45:34 EDT 2025] _major='8'
[Tue May 27 10:45:34 EDT 2025] _minor='12'
[Tue May 27 10:45:34 EDT 2025] DEPLOY_DOCKER_CONTAINER_KEY_FILE
[Tue May 27 10:45:34 EDT 2025] DEPLOY_DOCKER_CONTAINER_CERT_FILE
[Tue May 27 10:45:34 EDT 2025] DEPLOY_DOCKER_CONTAINER_CA_FILE
[Tue May 27 10:45:34 EDT 2025] DEPLOY_DOCKER_CONTAINER_FULLCHAIN_FILE
[Tue May 27 10:45:34 EDT 2025] DEPLOY_DOCKER_CONTAINER_PFX_FILE
[Tue May 27 10:45:34 EDT 2025] DEPLOY_DOCKER_CONTAINER_RELOAD_CMD
[Tue May 27 10:45:34 EDT 2025] _req='{"label":[""]}'
[Tue May 27 10:45:34 EDT 2025] _req='%7b%22label%22%3a%5b%22%22%5d%7d'
[Tue May 27 10:45:34 EDT 2025] _data
[Tue May 27 10:45:34 EDT 2025] url='http://localhost/containers/json?filters=%7b%22label%22%3a%5b%22%22%5d%7d'
10:45:34.067005 [0-x] == Info: [READ] client_reset, clear readers
10:45:34.067143 [0-0] == Info: [SETUP] added
10:45:34.067222 [0-0] == Info:   Trying /var/run/docker.sock:0...
10:45:34.067332 [0-0] == Info: Connected to localhost (/var/run/docker.sock) port 0
10:45:34.067438 [0-0] == Info: using HTTP/1.x
10:45:34.067505 [0-0] => Send header, 180 bytes (0xb4)
0000: GET /containers/json?filters=%7b%22label%22%3a%5b%22%22%5d%7d HT
0040: TP/1.1
0048: Host: localhost
0059: User-Agent: curl/8.12.1
0072: Accept: */*
007f: Content-Type: application/json
009f: Content-Length: 0
00b2:
10:45:34.067904 [0-0] == Info: Request completely sent off
10:45:34.068005 [0-0] <= Recv header, 17 bytes (0x11)
0000: HTTP/1.1 200 OK
10:45:34.068115 [0-0] == Info: [WRITE] cw_out, wrote 17 header bytes -> 17
10:45:34.068227 [0-0] == Info: [WRITE] download_write header(type=c, blen=17) -> 0
10:45:34.068322 [0-0] == Info: [WRITE] client_write(type=c, len=17) -> 0
10:45:34.068409 [0-0] <= Recv header, 19 bytes (0x13)
0000: Api-Version: 1.47
10:45:34.068520 [0-0] == Info: [WRITE] header_collect pushed(type=1, len=19) -> 0
10:45:34.068617 [0-0] == Info: [WRITE] cw_out, wrote 19 header bytes -> 19
10:45:34.068724 [0-0] == Info: [WRITE] download_write header(type=4, blen=19) -> 0
10:45:34.068828 [0-0] == Info: [WRITE] client_write(type=4, len=19) -> 0
10:45:34.068917 [0-0] <= Recv header, 32 bytes (0x20)
0000: Content-Type: application/json
10:45:34.069036 [0-0] == Info: [WRITE] header_collect pushed(type=1, len=32) -> 0
10:45:34.069134 [0-0] == Info: [WRITE] cw_out, wrote 32 header bytes -> 32
10:45:34.069233 [0-0] == Info: [WRITE] download_write header(type=4, blen=32) -> 0
10:45:34.069347 [0-0] == Info: [WRITE] client_write(type=4, len=32) -> 0
10:45:34.069443 [0-0] <= Recv header, 28 bytes (0x1c)
0000: Docker-Experimental: false
10:45:34.069585 [0-0] == Info: [WRITE] header_collect pushed(type=1, len=28) -> 0
10:45:34.069658 [0-0] == Info: [WRITE] cw_out, wrote 28 header bytes -> 28
10:45:34.069755 [0-0] == Info: [WRITE] download_write header(type=4, blen=28) -> 0
10:45:34.069874 [0-0] == Info: [WRITE] client_write(type=4, len=28) -> 0
10:45:34.069973 [0-0] <= Recv header, 15 bytes (0xf)
0000: Ostype: linux
10:45:34.070058 [0-0] == Info: [WRITE] header_collect pushed(type=1, len=15) -> 0
10:45:34.070160 [0-0] == Info: [WRITE] cw_out, wrote 15 header bytes -> 15
10:45:34.070253 [0-0] == Info: [WRITE] download_write header(type=4, blen=15) -> 0
10:45:34.070352 [0-0] == Info: [WRITE] client_write(type=4, len=15) -> 0
10:45:34.070457 [0-0] <= Recv header, 31 bytes (0x1f)
0000: Server: Docker/27.5.0 (linux)
10:45:34.070567 [0-0] == Info: [WRITE] header_collect pushed(type=1, len=31) -> 0
10:45:34.070684 [0-0] == Info: [WRITE] cw_out, wrote 31 header bytes -> 31
10:45:34.070778 [0-0] == Info: [WRITE] download_write header(type=4, blen=31) -> 0
10:45:34.070873 [0-0] == Info: [WRITE] client_write(type=4, len=31) -> 0
10:45:34.070968 [0-0] <= Recv header, 37 bytes (0x25)
0000: Date: Tue, 27 May 2025 14:45:34 GMT
10:45:34.071078 [0-0] == Info: [WRITE] header_collect pushed(type=1, len=37) -> 0
10:45:34.071182 [0-0] == Info: [WRITE] cw_out, wrote 37 header bytes -> 37
10:45:34.071275 [0-0] == Info: [WRITE] download_write header(type=4, blen=37) -> 0
10:45:34.071369 [0-0] == Info: [WRITE] client_write(type=4, len=37) -> 0
10:45:34.071462 [0-0] <= Recv header, 19 bytes (0x13)
0000: Content-Length: 3
10:45:34.071566 [0-0] == Info: [WRITE] header_collect pushed(type=1, len=19) -> 0
10:45:34.071666 [0-0] == Info: [WRITE] cw_out, wrote 19 header bytes -> 19
10:45:34.071757 [0-0] == Info: [WRITE] download_write header(type=4, blen=19) -> 0
10:45:34.071858 [0-0] == Info: [WRITE] client_write(type=4, len=19) -> 0
10:45:34.071940 [0-0] <= Recv header, 2 bytes (0x2)
0000:
10:45:34.072005 [0-0] == Info: [WRITE] header_collect pushed(type=1, len=2) -> 0
10:45:34.072094 [0-0] == Info: [WRITE] cw_out, wrote 2 header bytes -> 2
10:45:34.072173 [0-0] == Info: [WRITE] download_write header(type=4, blen=2) -> 0
10:45:34.072262 [0-0] == Info: [WRITE] client_write(type=4, len=2) -> 0
10:45:34.072342 [0-0] <= Recv data, 3 bytes (0x3)
0000: [].
10:45:34.072442 [0-0] == Info: [WRITE] cw_out, wrote 3 body bytes -> 3
10:45:34.072517 [0-0] == Info: [WRITE] download_write body(type=1, blen=3) -> 0
10:45:34.072627 [0-0] == Info: [WRITE] client_write(type=1, len=3) -> 0
10:45:34.072709 [0-0] == Info: [WRITE] xfer_write_resp(len=203, eos=0) -> 0
10:45:34.072809 [0-0] == Info: [WRITE] cw-out is notpaused
10:45:34.072887 [0-0] == Info: [WRITE] cw-out done
10:45:34.072942 [0-0] == Info: [READ] client_reset, clear readers
10:45:34.073040 [0-0] == Info: Connection #0 to host localhost left intact
[Tue May 27 10:45:34 EDT 2025] listjson='[]'
[Tue May 27 10:45:34 EDT 2025] Container id:
[Tue May 27 10:45:34 EDT 2025] can not find container id
[Tue May 27 10:45:34 EDT 2025] Error deploying for domain: example.com
[Tue May 27 10:45:34 EDT 2025] Error encountered while deploying.
[Tue May 27 10:45:34 EDT 2025] Setting Le_DeployHook
[Tue May 27 10:45:34 EDT 2025] Success

Gerporgl · 2025-05-27T15:26:00Z

Maybe I can also add something weird I have observed and that I did not mention in my previous comment.

I am also getting errors, but not all the time. I am using SSH deploy and Synology deploy.

Sometimes the SSH deploy is giving me "DEPLOY_SSH_USER not defined." then "Error deploying for domain: ..." and then "Error encountered while deploying.", but then proceed successfully to the next deploy (Synology).
When I retry, it usually works. In fact, it seems to be really 50/50% work/fail. And I mean now that I am testing this more, it works once, and seems to be guaranteed to not work next time.
I am obviously worried that my next automated renewal will not work.

I also see this line most of the time, not sure if this is an error: "./acme.sh: 241: [: unexpected operator".

I've assumed there was still some race condition issues that hopefully would get resolved at some point...

Perhaps this is a different problem.

invario · 2025-05-27T15:44:29Z

Hang on, here's a new debug log with `--output-insecure` included which might be more helpful:

[Tue May 27 11:42:03 EDT 2025] readlink exists=0
[Tue May 27 11:42:03 EDT 2025] dirname exists=0
[Tue May 27 11:42:03 EDT 2025] Let's find the script directory.
[Tue May 27 11:42:03 EDT 2025] _SCRIPT_='/usr/local/bin/acme.sh'
[Tue May 27 11:42:03 EDT 2025] _script='/root/.acme.sh/acme.sh'
[Tue May 27 11:42:03 EDT 2025] _script_home='/root/.acme.sh'
[Tue May 27 11:42:03 EDT 2025] Using default home: /root/.acme.sh
[Tue May 27 11:42:03 EDT 2025] Using config home: /acme.sh
[Tue May 27 11:42:03 EDT 2025] ACCOUNT_CONF_PATH='/acme.sh/account.conf'
[Tue May 27 11:42:03 EDT 2025] OK
[Tue May 27 11:42:03 EDT 2025] 1:AUTO_UPGRADE='1'
[Tue May 27 11:42:03 EDT 2025] LE_WORKING_DIR='/root/.acme.sh'
https://github.com/acmesh-official/acme.sh
v3.1.2
[Tue May 27 11:42:03 EDT 2025] Running cmd: deploy
[Tue May 27 11:42:03 EDT 2025] Using config home: /acme.sh
[Tue May 27 11:42:03 EDT 2025] ACCOUNT_CONF_PATH='/acme.sh/account.conf'
[Tue May 27 11:42:03 EDT 2025] default_acme_server
[Tue May 27 11:42:03 EDT 2025] ACME_DIRECTORY='https://acme.zerossl.com/v2/DV90'
[Tue May 27 11:42:03 EDT 2025] _ACME_SERVER_HOST='acme.zerossl.com'
[Tue May 27 11:42:03 EDT 2025] _ACME_SERVER_PATH='v2/DV90'
[Tue May 27 11:42:03 EDT 2025] CA_CONF='/acme.sh/ca/acme.zerossl.com/v2/DV90/ca.conf'
[Tue May 27 11:42:03 EDT 2025] The domain 'example.com' seems to already have an ECC cert, let's use it.
[Tue May 27 11:42:03 EDT 2025] DOMAIN_PATH='/acme.sh/example.com_ecc'
[Tue May 27 11:42:03 EDT 2025] DOMAIN_CONF='/acme.sh/example.com_ecc/example.com.conf'
[Tue May 27 11:42:03 EDT 2025] APP
[Tue May 27 11:42:03 EDT 2025] 19:Le_DeployHook='multideploy,'
[Tue May 27 11:42:03 EDT 2025] _deployApi='/root/.acme.sh/deploy/multideploy.sh'
[Tue May 27 11:42:03 EDT 2025] multideploy_deploy exists=0
[Tue May 27 11:42:03 EDT 2025] _cdomain='example.com'
[Tue May 27 11:42:03 EDT 2025] _ckey='/acme.sh/example.com_ecc/example.com.key'
[Tue May 27 11:42:03 EDT 2025] _ccert='/acme.sh/example.com_ecc/example.com.cer'
[Tue May 27 11:42:03 EDT 2025] _cca='/acme.sh/example.com_ecc/ca.cer'
[Tue May 27 11:42:03 EDT 2025] _cfullchain='/acme.sh/example.com_ecc/fullchain.cer'
[Tue May 27 11:42:03 EDT 2025] _cpfx='/acme.sh/example.com_ecc/example.com.pfx'
[Tue May 27 11:42:03 EDT 2025] DOMAIN_DIR='example.com_ecc'
[Tue May 27 11:42:03 EDT 2025] MULTIDEPLOY_FILENAME is not set, so I will use 'multideploy.yml'.
[Tue May 27 11:42:03 EDT 2025] yq is installed.
[Tue May 27 11:42:03 EDT 2025] Checking file='/acme.sh/example.com_ecc/multideploy.yml'
[Tue May 27 11:42:03 EDT 2025] File found
[Tue May 27 11:42:03 EDT 2025] Deploy file='/acme.sh/example.com_ecc/multideploy.yml'
[Tue May 27 11:42:03 EDT 2025] Deploy file version is compatible: 1.0
[Tue May 27 11:42:03 EDT 2025] Services='npm'
[Tue May 27 11:42:03 EDT 2025] Config has services.
[Tue May 27 11:42:03 EDT 2025] Checking service='npm'
[Tue May 27 11:42:03 EDT 2025] File='/acme.sh/example.com_ecc/multideploy.yml'
[Tue May 27 11:42:03 EDT 2025] Deploy file='/acme.sh/example.com_ecc/multideploy.yml'
[Tue May 27 11:42:03 EDT 2025] Services='npm'
[Tue May 27 11:42:03 EDT 2025] Service='npm'
[Tue May 27 11:42:03 EDT 2025] Exporting envs='DEPLOY_DOCKER_CONTAINER_LABEL: "sh.acme.autoload.service=example.com"
DEPLOY_DOCKER_CONTAINER_KEY_FILE: "/data/tls/custom/npm-1/privkey.pem"
DEPLOY_DOCKER_CONTAINER_FULLCHAIN_FILE: "/data/tls/custom/npm-1/fullchain.pem"
DEPLOY_DOCKER_CONTAINER_RELOAD_CMD: "nginx -s reload"'
[Tue May 27 11:42:03 EDT 2025] APP
[Tue May 27 11:42:03 EDT 2025] 20:DEPLOY_DOCKER_CONTAINER_LABEL='sh.acme.autoload.service=example.com'
[Tue May 27 11:42:03 EDT 2025] Saved DEPLOY_DOCKER_CONTAINER_LABEL='sh.acme.autoload.service=example.com'
[Tue May 27 11:42:03 EDT 2025] APP
[Tue May 27 11:42:03 EDT 2025] 21:DEPLOY_DOCKER_CONTAINER_KEY_FILE='/data/tls/custom/npm-1/privkey.pem'
[Tue May 27 11:42:03 EDT 2025] Saved DEPLOY_DOCKER_CONTAINER_KEY_FILE='/data/tls/custom/npm-1/privkey.pem'
[Tue May 27 11:42:03 EDT 2025] APP
[Tue May 27 11:42:03 EDT 2025] 22:DEPLOY_DOCKER_CONTAINER_FULLCHAIN_FILE='/data/tls/custom/npm-1/fullchain.pem'
[Tue May 27 11:42:03 EDT 2025] Saved DEPLOY_DOCKER_CONTAINER_FULLCHAIN_FILE='/data/tls/custom/npm-1/fullchain.pem'
[Tue May 27 11:42:03 EDT 2025] APP
[Tue May 27 11:42:03 EDT 2025] 23:DEPLOY_DOCKER_CONTAINER_RELOAD_CMD='nginx -s reload'
[Tue May 27 11:42:03 EDT 2025] Saved DEPLOY_DOCKER_CONTAINER_RELOAD_CMD='nginx -s reload'
[Tue May 27 11:42:03 EDT 2025] SERVICE='npm'
[Tue May 27 11:42:03 EDT 2025] HOOK='docker'
[Tue May 27 11:42:03 EDT 2025] Deploying to 'npm' using 'docker'
[Tue May 27 11:42:03 EDT 2025] _deployApi='/root/.acme.sh/deploy/docker.sh'
[Tue May 27 11:42:03 EDT 2025] docker_deploy exists=0
[Tue May 27 11:42:03 EDT 2025] _cdomain='example.com'
[Tue May 27 11:42:03 EDT 2025] DEPLOY_DOCKER_CONTAINER_LABEL
[Tue May 27 11:42:03 EDT 2025] The DEPLOY_DOCKER_CONTAINER_LABEL variable is not defined, we use this label to find the container.
[Tue May 27 11:42:03 EDT 2025] See: https://github.com/acmesh-official/acme.sh/wiki/deploy-to-docker-containers
[Tue May 27 11:42:03 EDT 2025] APP
[Tue May 27 11:42:03 EDT 2025] 24:SAVED_DEPLOY_DOCKER_CONTAINER_LABEL=''
[Tue May 27 11:42:03 EDT 2025] docker exists=127
[Tue May 27 11:42:03 EDT 2025] Try use /var/run/docker.sock
[Tue May 27 11:42:03 EDT 2025] curl exists=0
[Tue May 27 11:42:03 EDT 2025] _cversion='8.12.1'
[Tue May 27 11:42:03 EDT 2025] _major='8'
[Tue May 27 11:42:03 EDT 2025] _minor='12'
[Tue May 27 11:42:03 EDT 2025] DEPLOY_DOCKER_CONTAINER_KEY_FILE
[Tue May 27 11:42:03 EDT 2025] DEPLOY_DOCKER_CONTAINER_CERT_FILE
[Tue May 27 11:42:03 EDT 2025] DEPLOY_DOCKER_CONTAINER_CA_FILE
[Tue May 27 11:42:03 EDT 2025] DEPLOY_DOCKER_CONTAINER_FULLCHAIN_FILE
[Tue May 27 11:42:03 EDT 2025] DEPLOY_DOCKER_CONTAINER_PFX_FILE
[Tue May 27 11:42:03 EDT 2025] DEPLOY_DOCKER_CONTAINER_RELOAD_CMD
[Tue May 27 11:42:03 EDT 2025] _req='{"label":[""]}'
[Tue May 27 11:42:03 EDT 2025] od exists=0
[Tue May 27 11:42:03 EDT 2025] _url_encode
[Tue May 27 11:42:03 EDT 2025] _hex_str=' 7b 22 6c 61 62 65 6c 22 3a 5b 22 22 5d 7d'
[Tue May 27 11:42:03 EDT 2025] _req='%7b%22label%22%3a%5b%22%22%5d%7d'
[Tue May 27 11:42:03 EDT 2025] _data
[Tue May 27 11:42:03 EDT 2025] url='http://localhost/containers/json?filters=%7b%22label%22%3a%5b%22%22%5d%7d'
11:42:03.848065 [0-x] == Info: [READ] client_reset, clear readers
11:42:03.848336 [0-0] == Info: [SETUP] added
11:42:03.848407 [0-0] == Info:   Trying /var/run/docker.sock:0...
11:42:03.848507 [0-0] == Info: Connected to localhost (/var/run/docker.sock) port 0
11:42:03.848615 [0-0] == Info: using HTTP/1.x
11:42:03.848693 [0-0] => Send header, 180 bytes (0xb4)
0000: GET /containers/json?filters=%7b%22label%22%3a%5b%22%22%5d%7d HT
0040: TP/1.1
0048: Host: localhost
0059: User-Agent: curl/8.12.1
0072: Accept: */*
007f: Content-Type: application/json
009f: Content-Length: 0
00b2:
11:42:03.849070 [0-0] <= Recv header, 17 bytes (0x11)
0000: HTTP/1.1 200 OK
11:42:03.849176 [0-0] == Info: [WRITE] cw_out, wrote 17 header bytes -> 17
11:42:03.849264 [0-0] == Info: [WRITE] download_write header(type=c, blen=17) -> 0
11:42:03.849358 [0-0] == Info: [WRITE] client_write(type=c, len=17) -> 0
11:42:03.849444 [0-0] <= Recv header, 19 bytes (0x13)
0000: Api-Version: 1.47
11:42:03.849652 [0-0] == Info: [WRITE] header_collect pushed(type=1, len=19) -> 0
11:42:03.849758 [0-0] == Info: [WRITE] cw_out, wrote 19 header bytes -> 19
11:42:03.849826 [0-0] == Info: [WRITE] download_write header(type=4, blen=19) -> 0
11:42:03.849901 [0-0] == Info: [WRITE] client_write(type=4, len=19) -> 0
11:42:03.849969 [0-0] <= Recv header, 32 bytes (0x20)
0000: Content-Type: application/json
11:42:03.850053 [0-0] == Info: [WRITE] header_collect pushed(type=1, len=32) -> 0
11:42:03.850127 [0-0] == Info: [WRITE] cw_out, wrote 32 header bytes -> 32
11:42:03.850195 [0-0] == Info: [WRITE] download_write header(type=4, blen=32) -> 0
11:42:03.850270 [0-0] == Info: [WRITE] client_write(type=4, len=32) -> 0
11:42:03.850336 [0-0] <= Recv header, 28 bytes (0x1c)
0000: Docker-Experimental: false
11:42:03.850416 [0-0] == Info: [WRITE] header_collect pushed(type=1, len=28) -> 0
11:42:03.850490 [0-0] == Info: [WRITE] cw_out, wrote 28 header bytes -> 28
11:42:03.850687 [0-0] == Info: [WRITE] download_write header(type=4, blen=28) -> 0
11:42:03.850851 [0-0] == Info: [WRITE] client_write(type=4, len=28) -> 0
11:42:03.850949 [0-0] <= Recv header, 15 bytes (0xf)
0000: Ostype: linux
11:42:03.851018 [0-0] == Info: [WRITE] header_collect pushed(type=1, len=15) -> 0
11:42:03.851093 [0-0] == Info: [WRITE] cw_out, wrote 15 header bytes -> 15
11:42:03.851181 [0-0] == Info: [WRITE] download_write header(type=4, blen=15) -> 0
11:42:03.851255 [0-0] == Info: [WRITE] client_write(type=4, len=15) -> 0
11:42:03.851328 [0-0] <= Recv header, 31 bytes (0x1f)
0000: Server: Docker/27.5.0 (linux)
11:42:03.851416 [0-0] == Info: [WRITE] header_collect pushed(type=1, len=31) -> 0
11:42:03.851527 [0-0] == Info: [WRITE] cw_out, wrote 31 header bytes -> 31
11:42:03.851640 [0-0] == Info: [WRITE] download_write header(type=4, blen=31) -> 0
11:42:03.851757 [0-0] == Info: [WRITE] client_write(type=4, len=31) -> 0
11:42:03.851849 [0-0] <= Recv header, 37 bytes (0x25)
0000: Date: Tue, 27 May 2025 15:42:03 GMT
11:42:03.851995 [0-0] == Info: [WRITE] header_collect pushed(type=1, len=37) -> 0
11:42:03.852115 [0-0] == Info: [WRITE] cw_out, wrote 37 header bytes -> 37
11:42:03.852223 [0-0] == Info: [WRITE] download_write header(type=4, blen=37) -> 0
11:42:03.852319 [0-0] == Info: [WRITE] client_write(type=4, len=37) -> 0
11:42:03.852424 [0-0] <= Recv header, 19 bytes (0x13)
0000: Content-Length: 3
11:42:03.852544 [0-0] == Info: [WRITE] header_collect pushed(type=1, len=19) -> 0
11:42:03.852655 [0-0] == Info: [WRITE] cw_out, wrote 19 header bytes -> 19
11:42:03.852772 [0-0] == Info: [WRITE] download_write header(type=4, blen=19) -> 0
11:42:03.852878 [0-0] == Info: [WRITE] client_write(type=4, len=19) -> 0
11:42:03.852976 [0-0] <= Recv header, 2 bytes (0x2)
0000:
11:42:03.853055 [0-0] == Info: [WRITE] header_collect pushed(type=1, len=2) -> 0
11:42:03.853163 [0-0] == Info: [WRITE] cw_out, wrote 2 header bytes -> 2
11:42:03.853250 [0-0] == Info: [WRITE] download_write header(type=4, blen=2) -> 0
11:42:03.853358 [0-0] == Info: [WRITE] client_write(type=4, len=2) -> 0
11:42:03.853455 [0-0] <= Recv data, 3 bytes (0x3)
0000: [].
11:42:03.853650 [0-0] == Info: [WRITE] cw_out, wrote 3 body bytes -> 3
11:42:03.853739 [0-0] == Info: [WRITE] download_write body(type=1, blen=3) -> 0
11:42:03.853840 [0-0] == Info: [WRITE] client_write(type=1, len=3) -> 0
11:42:03.853935 [0-0] == Info: [WRITE] xfer_write_resp(len=203, eos=0) -> 0
11:42:03.854046 [0-0] == Info: [WRITE] cw-out is notpaused
11:42:03.854135 [0-0] == Info: [WRITE] cw-out done
11:42:03.854202 [0-0] == Info: [READ] client_reset, clear readers
11:42:03.854280 [0-0] == Info: Connection #0 to host localhost left intact
[Tue May 27 11:42:03 EDT 2025] listjson='[]'
[Tue May 27 11:42:03 EDT 2025] Container id:
[Tue May 27 11:42:03 EDT 2025] can not find container id
[Tue May 27 11:42:03 EDT 2025] Error deploying for domain: example.com
[Tue May 27 11:42:03 EDT 2025] Error encountered while deploying.
[Tue May 27 11:42:03 EDT 2025] Clearing envs='DEPLOY_DOCKER_CONTAINER_LABEL: "sh.acme.autoload.service=example.com"
DEPLOY_DOCKER_CONTAINER_KEY_FILE: "/data/tls/custom/npm-1/privkey.pem"
DEPLOY_DOCKER_CONTAINER_FULLCHAIN_FILE: "/data/tls/custom/npm-1/fullchain.pem"
DEPLOY_DOCKER_CONTAINER_RELOAD_CMD: "nginx -s reload"'
[Tue May 27 11:42:03 EDT 2025] Deleting key='DEPLOY_DOCKER_CONTAINER_LABEL'
[Tue May 27 11:42:03 EDT 2025] Deleting key='DEPLOY_DOCKER_CONTAINER_KEY_FILE'
[Tue May 27 11:42:03 EDT 2025] Deleting key='DEPLOY_DOCKER_CONTAINER_FULLCHAIN_FILE'
[Tue May 27 11:42:03 EDT 2025] Deleting key='DEPLOY_DOCKER_CONTAINER_RELOAD_CMD'
[Tue May 27 11:42:03 EDT 2025] Setting Le_DeployHook
[Tue May 27 11:42:03 EDT 2025] OK
[Tue May 27 11:42:03 EDT 2025] 19:Le_DeployHook='multideploy'
[Tue May 27 11:42:03 EDT 2025] Success

Which looks like your deploy script is reading the DEPLOY_DOCKER_CONTAINER_LABEL .

HMM....

deploy/multideploy.sh

tomo2403 · 2025-05-27T20:08:10Z

@egarzadev

When testing this out myself I had strange issues with environment variables not being applied consistently. I found that reverting this commit in my local multideploy.sh file seemed to resolve the issues.

Interesting, this commit should not be related to the problem with the environment variables.

tomo2403 · 2025-05-27T20:14:00Z

@Gerporgl @invario @egarzadev Could you please pull the latest change and test it again? I can't reproduce this error with my setup.

invario · 2025-05-27T20:15:17Z

@Gerporgl @invario @egarzadev Could you please pull the latest change and test it again? I can't reproduce this error with my setup.

Pulled, working on my end. Thanks for the quick turnaround.

invario · 2025-05-28T01:00:02Z

I'd like to make a suggestion that if a service fails to deploy properly, a non-zero return code should be returned upon completing deployment of all other services. Perhaps even something simple like increment the return code by 1 for each service that fails.

3792274New · 2025-05-28T06:53:16Z

dhparams

tomo2403 · 2025-05-28T18:24:13Z

@invario

I'd like to make a suggestion that if a service fails to deploy properly, a non-zero return code should be returned upon completing deployment of all other services. Perhaps even something simple like increment the return code by 1 for each service that fails.

I implemented your feature request, but unfortunately, the _deploy() method always returns 1 when an error occurs. At least you will receive an error if something fails. If someone fixes this problem, the exit code will be the number of failed deployments. This could be done very fast, but I don't know if changes to this will break something.

acme.sh/acme.sh

Lines 5833 to 5839 in 42bbd1b

    
             if ! $d_command "$_d" "$CERT_KEY_PATH" "$CERT_PATH" "$CA_CERT_PATH" "$CERT_FULLCHAIN_PATH" "$CERT_PFX_PATH"; then 
        
               _err "Error deploying for domain: $_d" 
        
               return 1 
        
             fi 
        
           ); then 
        
             _err "Error encountered while deploying." 
        
             return 1

invario · 2025-05-28T19:28:21Z

@invario

I'd like to make a suggestion that if a service fails to deploy properly, a non-zero return code should be returned upon completing deployment of all other services. Perhaps even something simple like increment the return code by 1 for each service that fails.

I implemented your feature request, but unfortunately, the _deploy() method always returns 1 when an error occurs. At least you will receive an error if something fails. If someone fixes this problem, the exit code will be the number of failed deployments. This could be done very fast, but I don't know if changes to this will break something.

acme.sh/acme.sh

Lines 5833 to 5839 in 42bbd1b

if ! $d_command "$_d" "$CERT_KEY_PATH" "$CERT_PATH" "$CA_CERT_PATH" "$CERT_FULLCHAIN_PATH" "$CERT_PFX_PATH"; then

_err "Error deploying for domain: $_d"

return 1

fi

); then

_err "Error encountered while deploying."

return 1

It'd be nice to have but completely unnecessary and probably breaks some sort of standard for error reporting. Anyone that has a problem deploying services should just use the debug switch to figure it out. Thank you!

Gerporgl · 2025-05-29T00:31:44Z

@Gerporgl @invario @egarzadev Could you please pull the latest change and test it again? I can't reproduce this error with my setup.

I tried it and it seems to work fine, I am not get the error I was having anymore.

Thanks!

tomo2403 added 18 commits February 15, 2025 20:12

Merge pull request #1 from tomo2403/deployhook-docker

cd9c764

Deployhook docker

implemented checking deploy file

0ce2830

fixed indents

cfe3226

refactored getting services

f1d214a

fixed formatting and private var names

18575b1

implemented handling envs

a6060f9

implemented deploying to services

4fbade3

added yq to dockerfile

f275f3c

implemented checking for different kinds of deploy file

06e75de

added debug messages

6189aa5

improved preprocessing and fixed bug with wrong param of services

ccd9d9a

fixed IFS problems

8d77bef

added docs

a9c2435

added docs and enhanced log messages

7a35d68

added header doc

b6a6e67

allowed using varaibles in deploy file

98aaff6

fixed missing wiki link

f26d404

Merge branch 'dev' into multideploy-yaml

488d147

tomo2403 added 2 commits March 1, 2025 13:26

Update links in multideploy.sh

69858fb

Merge branch 'dev' into multideploy-yaml

e9b5bbf

tomo2403 added 3 commits March 27, 2025 19:11

removed configs and implemented specification of deploy file name

67b5176

Merge branch 'dev' into multideploy-yaml

2e6065c

fixed formatting

bc23e8c

zeocax reviewed Apr 12, 2025

View reviewed changes

Neilpang reviewed Apr 13, 2025

View reviewed changes

deploy/multideploy.sh Show resolved Hide resolved

Neilpang reviewed Apr 13, 2025

View reviewed changes

deploy/multideploy.sh Show resolved Hide resolved

Neilpang reviewed Apr 13, 2025

View reviewed changes

deploy/multideploy.sh Outdated Show resolved Hide resolved

Neilpang reviewed Apr 13, 2025

View reviewed changes

deploy/multideploy.sh Outdated Show resolved Hide resolved

tomo2403 added 3 commits April 13, 2025 16:38

simplified deploy method

5189a31

Merge branch 'dev' into multideploy-yaml

ccf11cc

Merge remote-tracking branch 'origin/multideploy-yaml' into multidepl…

d3a8c7f

…oy-yaml

tomo2403 requested a review from Neilpang April 13, 2025 14:40

invario suggested changes May 27, 2025

View reviewed changes

deploy/multideploy.sh Outdated Show resolved Hide resolved

fixed bug with envs due to the use of a wrong function

eca6e69

invario mentioned this pull request May 28, 2025

Add 'localcopy' deploy-hook #6379

Open

tomo2403 added 2 commits May 28, 2025 20:02

implemented exiting with 1 if at least one deployment fails

1dc597b

implemented exiting with the number of failed deployments

3551e4a

Uh oh!

Multideploy: Deploy to multiple hooks of the same type #6241

Are you sure you want to change the base?

Multideploy: Deploy to multiple hooks of the same type #6241

Uh oh!

Conversation

tomo2403 commented Feb 21, 2025

Example

Sample config file

Wiki

Uh oh!

gilman88 commented Feb 27, 2025

Uh oh!

Neilpang commented Feb 28, 2025

Uh oh!

tomo2403 commented Mar 1, 2025

Uh oh!

Neilpang commented Mar 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tomo2403 commented Mar 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Neilpang commented Mar 4, 2025

Uh oh!

tomo2403 commented Mar 27, 2025

Uh oh!

zeocax left a comment

Choose a reason for hiding this comment

Uh oh!

Neilpang Apr 13, 2025

Choose a reason for hiding this comment

Uh oh!

tomo2403 Apr 13, 2025

Choose a reason for hiding this comment

Uh oh!

Neilpang Apr 13, 2025

Choose a reason for hiding this comment

Uh oh!

tomo2403 Apr 13, 2025

Choose a reason for hiding this comment

Uh oh!

Neilpang Apr 13, 2025

Choose a reason for hiding this comment

Uh oh!

tomo2403 Apr 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Neilpang Apr 13, 2025

Choose a reason for hiding this comment

Uh oh!

tomo2403 Apr 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Gerporgl commented May 21, 2025

Uh oh!

invario commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tomo2403 commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

invario commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Gerporgl commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

invario commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

tomo2403 commented May 27, 2025

Uh oh!

tomo2403 commented May 27, 2025

Neilpang commented Mar 2, 2025 •

edited

Loading

tomo2403 commented Mar 2, 2025 •

edited

Loading

tomo2403 Apr 13, 2025 •

edited

Loading

invario commented May 27, 2025 •

edited

Loading

tomo2403 commented May 27, 2025 •

edited

Loading

invario commented May 27, 2025 •

edited

Loading

Gerporgl commented May 27, 2025 •

edited

Loading

invario commented May 27, 2025 •

edited

Loading