NX No such file or directory (os error 2) - when using the new remote cache from 20.8 #30733

ianldgs · 2025-04-15T13:25:19Z

Current Behavior

Sometimes, when running tasks (happened with test and build), it fails with:

 NX   No such file or directory (os error 2)

Expected Behavior

Target runs normally

GitHub Repo

No response

Steps to Reproduce

Set NX_SELF_HOSTED_REMOTE_CACHE_SERVER to a remote cache server. E.g. this one
Run targets like test/build

Got this failure locally (macos) semi-consistently with a next.js app and a large .next/cache folder in dist.

It went away after I cleaned up all the cache:

find apps -type d -path "*\.next/cache*" | xargs rm -rf
rm -rf node_modules/.cache
rm -rf .nx
nx reset

But it's now happening on CI.

Nx Report

NX   Report complete - copy this into the issue template

Node           : 22.14.0
OS             : darwin-arm64
Native Target  : aarch64-macos
pnpm           : 10.4.0

nx                     : 20.8.0
@nx/js                 : 20.8.0
@nx/jest               : 20.8.0
@nx/eslint             : 20.8.0
@nx/workspace          : 20.8.0
@nx/cypress            : 20.8.0
@nx/devkit             : 20.8.0
@nx/esbuild            : 20.8.0
@nx/eslint-plugin      : 20.8.0
@nx/module-federation  : 20.8.0
@nx/next               : 20.8.0
@nx/node               : 20.8.0
@nx/plugin             : 20.8.0
@nx/react              : 20.8.0
@nx/rollup             : 20.8.0
@nx/storybook          : 20.8.0
@nx/vite               : 20.8.0
@nx/web                : 20.8.0
@nx/webpack            : 20.8.0
typescript             : 5.7.3
---------------------------------------
Community plugins:
@jscutlery/semver      : 5.3.1
@nx-tools/nx-container : 6.1.0
@nxkit/playwright      : 3.0.2
ngx-deploy-npm         : 8.2.0
---------------------------------------
Local workspace plugins:
         @imc-trading/webgui
         @imc-trading/nx-web
---------------------------------------
Cache Usage: 272.04 MB / 92.64 GB

Failure Logs

NX   Successfully ran target test for 38 projects and 5 tasks they depend on

Nx read the output from the cache instead of running the command for 5 out of 43 tasks.

 NX   No such file or directory (os error 2)
Pass --verbose to see the stacktrace.

Time for 'loading dotenv files' 1.3593339999999898
Time for 'code-loading' 53.17854199999999
Time for 'create-project-graph-async' 10.137083000000018
Time for 'total for sendMessageToDaemon()' 8.409125000000017
Time for 'deserialize daemon response' 3.732250000000022
TRACE nx::native::db: process{id=45749}: Creating connection to "/Users/iserpa/code/frontend/.nx/workspace-data/46B517E3-35EA-597B-BF52-B92D070073D4.db"
TRACE nx::native::db::initialize: process{id=45749}: Getting lock on db lock file
TRACE nx::native::db::initialize: process{id=45749}: Got lock on db lock file
TRACE nx::native::db::initialize: process{id=45749}: Checking if current existing database is compatible with Nx 20.8.0
TRACE nx::native::db::initialize: process{id=45749}: Database is compatible with Nx 20.8.0
Time for 'total for sendMessageToDaemon()' 1.2373329999999783
Time for 'deserialize daemon response' 0.006832999999971889
TRACE nx::native::tasks::details: Recording task details
Time for 'hashMultipleTasks' 13.501750000000015
Time for 'total for sendMessageToDaemon()' 10.54725000000002
Time for 'deserialize daemon response' 0.5096249999999714
TRACE nx::native::cache::cache: GET 1385638370963531659
TRACE nx::native::cache::cache: GET 1385638370963531659 70.041µs
TRACE hyper_util::client::legacy::pool: retrieve{hash=1385638370963531659}: checkout waiting for idle connection: ("http", nx-cache.intra)
DEBUG log: retrieve{hash=1385638370963531659}: starting new connection: http://nx-cache.intra/    
TRACE hyper_util::client::legacy::connect::http: retrieve{hash=1385638370963531659}: Http::connect; scheme=Some("http"), host=Some("nx-cache.intra"), port=None
Time for 'total for sendMessageToDaemon()' 1.5515829999999369
Time for 'deserialize daemon response' 0.00483299999996234
DEBUG hyper_util::client::legacy::connect::http: retrieve{hash=1385638370963531659}: connecting to 10.0.0.1:80
TRACE log: retrieve{hash=1385638370963531659}: registering event source with poller: token=Token(5115243008), interests=READABLE | WRITABLE    
DEBUG hyper_util::client::legacy::connect::http: connected to 10.0.0.1:80
TRACE hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
TRACE hyper_util::client::legacy::pool: checkout dropped for ("http", nx-cache.intra)
TRACE hyper_util::client::legacy::pool: put; add idle connection for ("http", nx-cache.intra)
DEBUG hyper_util::client::legacy::pool: pooling idle connection for ("http", nx-cache.intra)
TRACE nx::native::cache::http_remote_cache: HTTP response status: 200 OK
TRACE nx::native::cache::http_remote_cache: Downloaded 134 bytes from remote cache
TRACE nx::native::cache::http_remote_cache: Retrieved terminal output from cache: 45 bytes
TRACE nx::native::cache::http_remote_cache: Retrieved exit code from cache: 0
TRACE nx::native::cache::http_remote_cache: Extracted tarball to /Users/iserpa/code/frontend/.nx/cache/1385638370963531659
TRACE nx::native::cache::cache: applying remote cache results: "1385638370963531659" (/Users/iserpa/code/frontend/.nx/cache/1385638370963531659)
TRACE nx::native::cache::expand_outputs: No glob patterns found, checking if entries exist
TRACE nx::native::cache::cache: Removing expanded outputs: []
TRACE nx::native::cache::cache: Copying Files from Cache "/Users/iserpa/code/frontend/.nx/cache/1385638370963531659" -> "/Users/iserpa/code/frontend"
TRACE hyper_util::client::legacy::pool: idle interval checking for expired
TRACE nx::native::cache::expand_outputs: No glob patterns found, checking if entries exist
TRACE nx::native::cache::cache: Removing expanded outputs: []
TRACE nx::native::cache::cache: Copying Files from Cache "/Users/iserpa/code/frontend/.nx/cache/1385638370963531659" -> "/Users/iserpa/code/frontend"
TRACE nx::native::cache::expand_outputs: No glob patterns found, checking if entries exist
TRACE nx::native::cache::cache: Removing expanded outputs: []
TRACE nx::native::cache::cache: Copying Files from Cache "/Users/iserpa/code/frontend/.nx/cache/1385638370963531659" -> "/Users/iserpa/code/frontend"
TRACE nx::native::cache::expand_outputs: No glob patterns found, checking if entries exist
TRACE nx::native::cache::cache: Removing expanded outputs: []
TRACE nx::native::cache::cache: Copying Files from Cache "/Users/iserpa/code/frontend/.nx/cache/1385638370963531659" -> "/Users/iserpa/code/frontend"
TRACE nx::native::cache::expand_outputs: No glob patterns found, checking if entries exist
TRACE nx::native::cache::cache: Removing expanded outputs: []
TRACE nx::native::cache::cache: Copying Files from Cache "/Users/iserpa/code/frontend/.nx/cache/1385638370963531659" -> "/Users/iserpa/code/frontend"
TRACE nx::native::cache::expand_outputs: No glob patterns found, checking if entries exist
TRACE nx::native::cache::cache: Removing expanded outputs: []
TRACE nx::native::cache::cache: Copying Files from Cache "/Users/iserpa/code/frontend/.nx/cache/1385638370963531659" -> "/Users/iserpa/code/frontend"

—————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 NX   Successfully ran target test for project xxx


Task Execution Timings:
{
  "xxx:test": null
}
Time for 'total for sendMessageToDaemon()' 11.31420799999978
Time for 'deserialize daemon response' 0.03341700000237324

 NX   No such file or directory (os error 2)

Error: No such file or directory (os error 2)
    at DbCache.applyRemoteCacheResults (/Users/iserpa/code/frontend/node_modules/.pnpm/[email protected]_@[email protected]_@[email protected]_@[email protected]__@[email protected]_b4fae504402af20c09600375d5c44547/node_modules/nx/src/tasks-runner/cache.js:119:27)
    at DbCache.get (/Users/iserpa/code/frontend/node_modules/.pnpm/[email protected]_@[email protected]_@[email protected]_@[email protected]__@[email protected]_b4fae504402af20c09600375d5c44547/node_modules/nx/src/tasks-runner/cache.js:100:22)
    at async TaskOrchestrator.applyCachedResult (/Users/iserpa/code/frontend/node_modules/.pnpm/[email protected]_@[email protected]_@[email protected]_@[email protected]__@[email protected]_b4fae504402af20c09600375d5c44547/node_modules/nx/src/tasks-runner/task-orchestrator.js:130:30)
    at async Promise.all (index 0)
    at async TaskOrchestrator.applyCachedResults (/Users/iserpa/code/frontend/node_modules/.pnpm/[email protected]_@[email protected]_@[email protected]_@[email protected]__@[email protected]_b4fae504402af20c09600375d5c44547/node_modules/nx/src/tasks-runner/task-orchestrator.js:126:21)
    at async TaskOrchestrator.applyFromCacheOrRunTask (/Users/iserpa/code/frontend/node_modules/.pnpm/[email protected]_@[email protected]_@[email protected]_@[email protected]__@[email protected]_b4fae504402af20c09600375d5c44547/node_modules/nx/src/tasks-runner/task-orchestrator.js:223:40)
    at async TaskOrchestrator.executeNextBatchOfTasksUsingTaskSchedule (/Users/iserpa/code/frontend/node_modules/.pnpm/[email protected]_@[email protected]_@[email protected]_@[email protected]__@[email protected]_b4fae504402af20c09600375d5c44547/node_modules/nx/src/tasks-runner/task-orchestrator.js:85:13)
    at async Promise.all (index 0)
    at async TaskOrchestrator.run (/Users/iserpa/code/frontend/node_modules/.pnpm/[email protected]_@[email protected]_@[email protected]_@[email protected]__@[email protected]_b4fae504402af20c09600375d5c44547/node_modules/nx/src/tasks-runner/task-orchestrator.js:61:9)
    at async defaultTasksRunner (/Users/iserpa/code/frontend/node_modules/.pnpm/[email protected]_@[email protected]_@[email protected]_@[email protected]__@[email protected]_b4fae504402af20c09600375d5c44547/node_modules/nx/src/tasks-runner/default-tasks-runner.js:69:16)

TRACE nx::native::db::connection: Closing database connection

Package Manager Version

[email protected]

Operating System

macOS
Linux
Windows
Other (Please specify)

Additional Information

No response

The text was updated successfully, but these errors were encountered:

ianldgs · 2025-04-16T10:31:27Z

@FrozenPandaz / @AgentEnder I'm happy to work with you guys to debug this.
Let me know if there's any other flag or pre-release that we could get more details on the problem, like a stack trace or smth

ianldgs · 2025-04-16T11:04:26Z

Ok, I got it consistently working again, locally. Tracked it to the attached cache tarball.

Running the test target of one of our projects produced it, and it gets used as the remote cache for following runs.
The project uses nextjs and jest, and there are actually no tests in that project.

ls -lah cache-1385638370963531659
total 16
drwx------@   4 iserpa  staff   128B Apr 16 12:47 .
drwx------@ 205 iserpa  staff   6.4K Apr 16 12:47 ..
-rwxrwxrwx    1 iserpa  staff     4B Jan  1  1970 code
-rwxrwxrwx    1 iserpa  staff    45B Jan  1  1970 terminalOutput

cat cache-1385638370963531659/code # (empty output, nothing inside this file)
cat cache-1385638370963531659/terminalOutput 
No tests found, exiting with code 0

cache-1385638370963531659.tgz

ianldgs · 2025-04-16T11:30:43Z

Update:
found some stack trace by running the failing task in isolation with --verbose. Added it to the description.
found trace logs by enabling NX_NATIVE_LOGGING=trace, also added to the description.

ianldgs · 2025-04-16T13:13:35Z

Seems to be related to tasks that declare outputs, but none of those outputs are in the cache.

For example, jest and vitest declare outputs (coverage directory), but if you run tests without coverage, it won't produce those outputs.

    "test": {
      "executor": "@nx/jest:jest",
      "outputs": ["{workspaceRoot}/coverage/{projectRoot}"],
...
    "test": {
      "executor": "@nx/vite:test",
      "outputs": ["{options.reportsDirectory}"],

Then, when the cache results are being applied, and it looks for those outputs, which don't exist, it throws that "No such file or directory" error.

ianldgs · 2025-04-29T07:42:55Z

@FrozenPandaz / @AgentEnder / @vsavkin can we please get an update on this? Is there a plan to include a fix, at least before the official v21 launch?

AgentEnder · 2025-04-30T16:58:11Z

Hi, yes this needs to be fixed. Its on my list, just tending to some other stuff still too.

drewandre · 2025-05-06T15:45:26Z

Is this possibly related to #27724 (next.js output: standalone fails with "File exists (os error 17)"), which is still an issue in v21.0.0?

Edit: corrected referenced nx version

ianldgs · 2025-05-06T15:58:49Z

@drewandre hard to tell. I've never experienced the problem described in #27724, which seems old.
But FYI, I just opened #31085, which is yet another issue with remote cache and standalone output, that happened before when stuff was rewritten in rust: #19502.
So, unfortunately we're still blocked from using remote cache for Next.js regardless of #27724.

ianldgs added the type: bug label Apr 15, 2025

FrozenPandaz assigned AgentEnder Apr 15, 2025

FrozenPandaz added scope: core core nx functionality priority: high High Priority (important issues which affect many people severely) labels Apr 15, 2025

AgentEnder mentioned this issue Apr 30, 2025

fix(core): error when restoring http cache with no outputs #30961

Merged

AgentEnder closed this as completed in #30961 May 1, 2025

drewandre mentioned this issue May 6, 2025

It failed when building a next application with config output:standalone. #27724

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

NX No such file or directory (os error 2) - when using the new remote cache from 20.8 #30733

NX No such file or directory (os error 2) - when using the new remote cache from 20.8 #30733

ianldgs commented Apr 15, 2025 •

edited

Loading

ianldgs commented Apr 16, 2025

Uh oh!

ianldgs commented Apr 16, 2025 •

edited

Loading

Uh oh!

ianldgs commented Apr 16, 2025 •

edited

Loading

Uh oh!

ianldgs commented Apr 16, 2025

Uh oh!

ianldgs commented Apr 29, 2025

Uh oh!

AgentEnder commented Apr 30, 2025

Uh oh!

drewandre commented May 6, 2025 •

edited

Loading

Uh oh!

ianldgs commented May 6, 2025 •

edited

Loading

Uh oh!

NX No such file or directory (os error 2) - when using the new remote cache from 20.8 #30733

NX No such file or directory (os error 2) - when using the new remote cache from 20.8 #30733

Comments

ianldgs commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Current Behavior

Expected Behavior

GitHub Repo

Steps to Reproduce

Nx Report

Failure Logs

Package Manager Version

Operating System

Additional Information

ianldgs commented Apr 16, 2025

Uh oh!

ianldgs commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ianldgs commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ianldgs commented Apr 16, 2025

Uh oh!

ianldgs commented Apr 29, 2025

Uh oh!

AgentEnder commented Apr 30, 2025

Uh oh!

drewandre commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ianldgs commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ianldgs commented Apr 15, 2025 •

edited

Loading

ianldgs commented Apr 16, 2025 •

edited

Loading

ianldgs commented Apr 16, 2025 •

edited

Loading

drewandre commented May 6, 2025 •

edited

Loading

ianldgs commented May 6, 2025 •

edited

Loading