-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Issue with native modules (.node) resolving incorrectly in VSCode extension built with esbuild #4154
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
By default, attempting to bundle a You asked about marking as external and best practice. Indeed, marking dependencies as external is best practice when bundling for node (which is the case here as VSCode runs Electron which runs node). This is covered in the getting started instructions for node:
Not all packages are bundler-friendly, so the most robust and general way to handle dependencies is to leave them unbundled, and to make sure they are present alongside your code at run-time (just like writing normal JavaScript code for node without using esbuild). Some libraries may support being bundled and it may be ok to bundle them, but that's something you have to evaluate on a case-by-case basis. I recommend starting off by excluding all dependencies using |
@evanw thanks for taking time to respond. I will mark the native modules as external and give it a try. This is my esbuild configuration for reference
I have a question about the bundled code and wanted to understand whether it's a bug or expected behavior. My out folder has
The |
@aspire-shatheesh The |
@hyrious thankyou. The node libraries such as lancedb provided prebuilt binaries in .node format. So to make it work, do i need to remove file loader in esbuild config and instead mark it as external ? Just confirming if that is the way forward |
Yes, you should mark it as external. The The So neither the |
In VS Code you can publish platform specific extensions. Simply adds |
@evanw and @hyrious thanks for the input and guidance. I’ve now marked the native modules as external. Below is my updated esbuild config and now I am half way through it.
The current node_modules directory is around 200+ MB. Since I'm marking certain packages as externals, I now need to manually copy these node_modules into the /out directory when packaging the VSCode extension using vsce. However, I'm encountering errors related to missing dependencies of @lancedb—for example, the apache-arrow module, which is a dependency of @lancedb. To work around this, I had to write a script to recursively traverse all of @lancedb's dependencies and copy them into node_modules as well. I have pasted the script below incase if required. I have few questions:
Once again, thank you so much for taking the time to help with my questions and challenges—I really appreciate your support. |
Before answering your questions, it is totally fine to have a VS Code extension that big. For example
You can have
AFAIK there's no such usage in esbuild. You may be able to run a full bundle (including everything) to help you find out used files in node_modules, but not all packages can be bundled by esbuild, especially those with native node extensions. So simply no.
VS Code extensions, when activated, run in Node.js environment (it is slightly different than bare Node.js since it is simulated by electron). That means you can do everything in a VS Code extension like in Node.js, including downloading something from the network and importing them in the runtime. You can even ask your users to have |
Hi,
I am developing a VSCode extension that uses a private NPM package. This private package, in turn, depends on the @xenova/transformers and @lancedb packages.
lancedb includes a platform-specific binary. When the project is built using esbuild, it generates a file named lancedb.linux-x64-gnu-V6OX6RLO.node in the /out directory. However, when I run the extension, I encounter an error stating that native_js_1.Connection.new() is undefined.
From the log and code snippet below, we can see that nativeBinding is just a string ("./lancedb.linux-x64-gnu-V6OX6RLO.node") rather than an actual module object. As a result, the call to native_js_1.Connection.new(uri, opts) fails.
If I rename the file from lancedb.linux-x64-gnu-V6OX6RLO.node to lancedb.linux-x64-gnu.node, the module loads correctly.
Error:
TypeError: Cannot read properties of undefined (reading 'new') {stack: 'TypeError: Cannot read properties of undefine…e/ide/vscode-extn/out/extension.js:177487:22)', message: "Cannot read properties of undefined (reading 'new')"}
Code from extension.js
const nativeConn = await native_js_1.Connection.new(uri, opts);
log of native_js_1
{Connection: undefined, Index: undefined, RecordBatchIterator: undefined, NativeMergeInsertBuilder: undefined, Query: undefined, …}
Excerpts from extension.js to demonstrate that lancedb resolves to a string instead of a module object:
Log response
loading native bindings....
extensionHostProcess.js:179
type---- string
extensionHostProcess.js:179
🔍 Platform: linux
extensionHostProcess.js:179
🔍 Arch: x64
extensionHostProcess.js:179
📁 Local native file existed: false
extensionHostProcess.js:179
📦 Native binding object: ./lancedb.linux-x64-gnu-V6OX6RLO.node
extensionHostProcess.js:179`
Similar Issue with Sharp package
I'm encountering a similar problem with the sharp package as well . In the /out directory, a native binary file named sharp-linux-x64-DQNVFNDI.node is present after the esbuild process.
However, at runtime, the require_sharp call resolves to a string path instead of loading the native module. As a result, any function calls on require_sharp fail because it is not the expected module object. I am not clear how to get this working in vscode extension. Any help would be appreciated
My esbuild setting
Queries:
This may be happening because esbuild is treating .node files as static assets and not bundling or resolving them correctly as native modules.
The text was updated successfully, but these errors were encountered: