OKD 4.x mirroring to Artifactory JCR fails after uploading layer blobs #2201
Unanswered
debugnetiq1
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Same behavior with OKD 4.18 and 4.19. Uploading the layers (phase 0) ends with a number of errors for layers that could not be uploaded. The uploaded layers remain in _uploads and no manifests are mirrored to allow JCR assembling the layers into images
export LOCAL_REGISTRY=artifactory-jcr.local
export LOCAL_REPOSITORY=okd-v4.19/okd
export PRODUCT_REPO=okd
export RELEASE_NAME=scos-release
export LOCAL_SECRET_JSON=/root/downloads/docker_reg_creds_4.19.json
export GODEBUG=x509ignoreCN=0
export DOCKER_OPTS+=" --insecure-registry artifactory-jcr"
root@artifactory-jcs:~# GODEBUG=x509ignoreCN=1 oc adm release mirror --insecure-skip-tls-verify=true -a ${LOCAL_SECRET_JSON} --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE} --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE} 2>&1 | tee -a oc_adm_mirror_okd.log
. . .
info: Mirroring 191 images to artifactory-jcr.home.lab/okd-v4.19/okd ...
artifactory-jcr.home.lab/
okd-v4.19/okd
blobs:
quay.io/okd/scos-content sha256:bd9ddc54bea929a22b334e73e026d4136e5b73f5cc29942896c72e4ece69b13d 34B
quay.io/okd/scos-content sha256:2520ea6d60196c4d358dbba026bdfa051b346b8ff72683e10ad0000e9b726eb4 339B
. . .
quay.io/okd/scos-release sha256:6b2459a9848db17f96b6d5d14130987b4490b5fac3f85aaac247d7b3f200b0e8 58.13MiB
manifests:
sha256:02ddfc0a36035d1dedc24367412c12b96919fb0e573236e2b0709fd2ba5528c1 -> 4.19.0-okd-scos.0-x86_64-olm-catalogd
sha256:06f19fa113f5b57be450171df79a9cc6382a136b81e20814baaee7bfa095dc91 -> 4.19.0-okd-scos.0-x86_64-ibmcloud-machine-controllers
sha256:08365e506fde263691a2c4907f5f7ffe2a338ec53611b6d50264258f27472133 -> 4.19.0-okd-scos.0-x86_64-cluster-control-plane-machine-set-operator
. . .
stats: shared=4 unique=446 size=18.63GiB ratio=0.99
phase 0:
artifactory-jcr.home.lab okd-v4.19/okd blobs=450 mounts=0 manifests=191 shared=4
info: Planning completed in 1m15.31s
uploading: artifactory-jcr.home.lab/okd-v4.19/okd sha256:7be6248f58c6e6964995271f72c8d5ede6d3006bbbd4787e93714aeeef3e3145 42.96MiB
uploading: artifactory-jcr.home.lab/okd-v4.19/okd sha256:e41382123509bcda6eb4fd1b89128f5b1f2ed16dc5a1b8165be8e927e0b22654 32.68MiB
uploading: artifactory-jcr.home.lab/okd-v4.19/okd sha256:646e15b4e1f84c84a27bec3feb7139eecf216ceea28bd6405e66c8fce64fff79 31.06MiB
uploading: artifactory-jcr.home.lab/okd-v4.19/okd sha256:db99071539179373a3adb32c4ea3c55e19e966054c8b84bce44d36faee90b5af 2.177MiB
. . .
uploading: artifactory-jcr.home.lab/okd-v4.19/okd sha256:b052e1c609320a90cbf360ae468d8357eec4bdc1483c7f5364f8a84726e36108 19.8MiB
uploading: artifactory-jcr.home.lab/okd-v4.19/okd sha256:4750a5430ebc09640993a817d26995f9d96936f4ae537b8209adb25c28dab20f 22.61MiB
uploading: artifactory-jcr.home.lab/okd-v4.19/okd sha256:07f8ff59126201dfd2c76bd24a633139281b408cc404329ea109500abd182fca 12.14MiB
error: unable to copy layer sha256:b142af5894825609e66fcfec82408405920ee574834b14438ad1332dd5eb4ea3 to artifactory-jcr.home.lab/okd-v4.19/okd: Patch "https://artifactory-jcr.home.lab/v2/okd-v4.19/okd/blobs/uploads/80c63c67-e6ca-4c4b-866d-8a8a494b0c19.patch": unexpected EOF
error: unable to copy layer sha256:cb4698b07025c749fab087e081492e9b32ca819b4750f3da084f5799b6ddbd8c to artifactory-jcr.home.lab/okd-v4.19/okd: Patch "https://artifactory-jcr.home.lab/v2/okd-v4.19/okd/blobs/uploads/7fc6c90a-fe1d-4fb3-838c-5b7c9aa2340a.patch": read tcp 192.168.2.42:52742->184.150.154.66:443: read: connection reset by peer
error: unable to copy layer sha256:388f6186c22dafe4cc1b243f97273034100c1cc2a7d4b969c2f17e9e37d87cb9 to artifactory-jcr.home.lab/okd-v4.19/okd: Patch "https://artifactory-jcr.home.lab/v2/okd-v4.19/okd/blobs/uploads/5dc88734-c7d4-4e29-b60c-2669ecc004cf.patch": unexpected EOF
. . . etc (32 layers in error)
error: unable to copy layer sha256:6c1716093f599787e6ae80265e4ffd0278101c1c20c2961c540a8cf9e043221e to artifactory-jcr.home.lab/okd-v4.19/okd: Patch "https://artifactory-jcr.home.lab/v2/okd-v4.19/okd/blobs/uploads/da72fc55-c7c0-4641-97b7-0a2649212b58.patch": unexpected EOF
error: unable to copy layer sha256:0c8c50799877bd4cf5955c27c6b1e32453e7b22d97de1e86e5288e3203b70a68 to artifactory-jcr.home.lab/okd-v4.19/okd: Patch "https://artifactory-jcr.home.lab/v2/okd-v4.19/okd/blobs/uploads/0b6d1ae6-40e6-4cae-99c3-49b5e3ed458a.patch": unexpected EOF
info: Mirroring completed in 3h28m45.18s (645.2kB/s)
error: one or more errors occurred while uploading images
root@artifactory-jcs:~#
And this fails - no manifest present
root@artifactory-jcs:~ # oc adm release extract -a ${LOCAL_SECRET_JSON} --command=openshift-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}"
error: image "artifactory-jcr.home.lab/okd-v4.19/okd:4.19.0-okd-scos.0" not found: manifest unknown: The named manifest is not known to the registry.
root@artifactory-jcs:~#
Artifactory JCR is however listed as supported registry

My assumption is that subsequent mirroring phases are aborted due to the layers' upload issue. Help much appreciated on how to overcome this issue starting with the upload errors.
Beta Was this translation helpful? Give feedback.
All reactions