Does not correctly retry requests to Backblaze B2

[metadaddy]

Describe the bug
When uploading lots of folders and files to a Backblaze B2 bucket, Cyberduck shows the 1000 calls per PT1M allowed for API_LIST_FILE_VERSIONS_CALLED error shown in the screenshot below.

Looking at the code, executeHead(), executeGet() and executePost() in BaseB2Request.java look for the Retry-After HTTP header in the response. The Backblaze B2 documentation (rate limits and integration checklist) say that the error response may include Retry-After (I think this guidance may have changed over time). For 429 and 503 errors where Retry-After is not included, the client code should retry using an exponential backoff starting with 1 second.

To Reproduce
Steps to reproduce the behavior:

• Try to upload lots of folders and files to a Backblaze B2 bucket.

Expected behavior
The folders and files are uploaded correctly.

Screenshots

Desktop (please complete the following information):

• Cyberduck Version 9.1.3 (42945)
• MacOS Ventura 13.6.6

Log Files
Please include the log file. You can reach the logging output in Preferences → Connection.

• Not available

Additional context
This error was reported to me by a mutual user of Cyberduck and Backblaze B2. I'll point them at this issue so they can add more detail if necessary.

[rlevene]

Cyberduck Version 9.1.3 (42945)
MacOS Ventura 13.6.6

Log file was not with debug enabled and does not show anything related to this error (timestamp) so not sharing it

[metadaddy]

Adding this here as it's closely related, but I can spin it out into a separate issue if necessary...

When uploading files, Cyberduck does not follow the documented procedure for retries:

To upload a single file, first you call b2_get_upload_url to get a URL. Then call b2_upload_file using that URL.

The upload URL that is returned is targeted at a single storage pod in the Backblaze data center. This makes uploads efficient because you send the data directly to the place where it will be stored. But, it means that if that storage pod is unable to take your data at the time, you will have to get a new upload URL and try again.

Backblaze recommends that you write your code to try five different upload URLs before you report an error. Two attempts are almost always sufficient, and five failures is a sign that something is wrong with your request or that you are having problems connecting to the Backblaze B2 service.

(A similar process applies to large files with b2_get_upload_part_url and b2_upload_part)

Looking at the code, the fix would be in backblaze-b2-java-api, but the README there says that that project is no longer active. Are there plans to migrate Cyberduck to the the official Backblaze Java SDK?

[dkocher]

We are maintaining a fork ¹ of the B2 Java API.

Footnotes

1. https://github.com/iterate-ch/backblaze-b2-java-api ↩

[dkocher]

The implementation uploads many files to the same upload_url until that URL gives an error ¹.

Footnotes

1. https://github.com/iterate-ch/cyberduck/blob/167927f045277b3f29444518cebc2020f60360dd/backblaze/src/main/java/ch/cyberduck/core/b2/B2WriteFeature.java#L99 ↩

[dkocher]

The Retry-After response header should be honoured ¹ and retries being delayed in the generic RetryTransferCallable.

Footnotes

1. https://github.com/iterate-ch/cyberduck/blob/167927f045277b3f29444518cebc2020f60360dd/backblaze/src/main/java/ch/cyberduck/core/b2/B2ExceptionMappingService.java#L112 ↩