[BUG] PEP 420 namespace packages via package_dir don't work with editable installs

[mgorny]

setuptools version

78.1.0

Python version

3.13.3

OS

Gentoo Linux amd64

Additional environment information

No response

Description

I'm trying to port Triton's setup.py to stop using symlinks to cross-link Python packages and use package_dir instead. Unfortunately, it seems that if a package referenced through package_dir does not have a __init__.py (i.e. is effectively a PEP 420 namespace package), it cannot be imported from an editable install.

Expected behavior

All packages being importable from an editable install.

How to Reproduce

mkdir -p python/test third_party/{a,b}
> python/test/__init__.py
> third_party/a/__init__.py
> third_party/a/foo.py
> third_party/b/foo.py
cat > setup.py <<EOF
from setuptools import setup
setup(name="test",
      packages=["test", "test.a", "test.b"],
      package_dir={"": "python",
                   "test.a": "third_party/a",
                   "test.b": "third_party/b"})
EOF
pip install -e .
python -c 'import test.a.foo; import test.b.foo'

Output

$ pip install -e .
Obtaining file:///tmp/repro
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: test
  Building editable for test (pyproject.toml) ... done
  Created wheel for test: filename=test-0.0.0-0.editable-py3-none-any.whl size=2570 sha256=ede19ca9e89c0f19de29833a8d0466abf67ab9b99dd30d04237612a997803cd0
  Stored in directory: /tmp/pip-ephem-wheel-cache-ijmh_p0o/wheels/0e/11/03/c16968b602979cd363b907ce63c485bbbc0baada53c46b3cc5
Successfully built test
Installing collected packages: test
Successfully installed test-0.0.0
$ python -c 'import test.a.foo; import test.b.foo'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import test.a.foo; import test.b.foo
                       ^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'test.b'

Note that wheel has a working file structure, by comparison:

$ unzip -l dist/test-0.0.0-py3-none-any.whl 
Archive:  dist/test-0.0.0-py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  04-11-2025 16:14   test/__init__.py
        0  04-11-2025 16:14   test/a/__init__.py
        0  04-11-2025 16:14   test/a/foo.py
        0  04-11-2025 16:14   test/b/foo.py
       48  04-11-2025 16:15   test-0.0.0.dist-info/METADATA
       91  04-11-2025 16:15   test-0.0.0.dist-info/WHEEL
        5  04-11-2025 16:15   test-0.0.0.dist-info/top_level.txt
      559  04-11-2025 16:15   test-0.0.0.dist-info/RECORD
---------                     -------
      703                     8 files

[abravalheri]

Hi @mgorny , I believe that with a complex package_dir like this one, the package is being installed via a meta path finder.

The problem is that meta path finders are tricky, and the import machinery has some quirks when it comes to namespaces. I believe this is described in the docs as part of the limitations: https://setuptools.pypa.io/en/latest/userguide/development_mode.html#limitations.

So the TL;DR is that I played around with this use case before, but I did not manage to find a full blown solution given the state of the ecosystem.

Things could be a bit different different if wheels allowed symlinks.

---

Let me try to run the reproducer locally to see if I find something different than what I had investigated before

[abravalheri]

Let me try to run the reproducer locally to see if I find something different than what I had investigated before

So I run the following:

> docker run --rm -it python:3.13-bookworm /bin/bash

python3 -m venv /tmp/venv

mkdir /tmp/pkg
cd /tmp/pkg

mkdir -p python/test third_party/{a,b}
touch python/test/__init__.py
touch third_party/a/__init__.py
touch third_party/a/foo.py
touch third_party/b/foo.py

cat > setup.py <<EOF
from setuptools import setup
setup(name="test",
      packages=["test", "test.a", "test.b"],
      package_dir={"": "python",
                   "test.a": "third_party/a",
                   "test.b": "third_party/b"})
EOF

cat << EOF > pyproject.toml
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
EOF

/tmp/venv/bin/python -m pip install -e .

As you mentioned, if I run python -c 'import test.a.foo; import test.b.foo', the command will fail.

So I opened the REPL to investigate the following:

>>> import sys
>>> editable = sys.meta_path[-1]
>>> editable.find_spec('test.b.foo')
ModuleSpec(name='test.b.foo', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7fea76ead010>, origin='/tmp/pkg/third_party/b/foo.py')

So we can see here that, if it would come to the meta path finder itself, it would have found the module, but I suspect that the import machinery is not reaching that stage and it halts before reaching the meta path finder.

[abravalheri]

There is an open question, discussion in python/cpython#92054 related to this.

I believe we would need some input from the cpython team about how to proceed.

But if any member of the community finds a way to support this use case that does not depend on inputs from the cpython team and would like to contribute, it would be great.

[jaraco]

I encountered a related problem when working out the editable strategy for coherent.build, which has a similar problem that the layout of the project doesn't match the runtime expectation. In the implementation, I explored using the editables package, but found it inadequate. Ultimately, however, I ended up using a proxy module to make these packages appear virtually in the requisite namespace.

To illustrate how this works with the coherent.build project itself (which builds itself):

• The code for the coherent.build Python package is located in the root of the repository, so ./__init__.py is the implementation of the coherent.build package, and ./backends.py is the implementation of coherent.build.backends.
• Normally, . is ../coherent.build, but that's not guaranteed; it could be anything.
• Since there's nothing on the file system that could represent a naturally importable coherent namespace package, the editable wheel needs to create such a thing (similar to test in the example above).
• Moreover, the build subpackage of coherent isn't even manifest on the file system as build, so something needs to be created to represent that package as well. This same problem applies to non-namespace packages (since the containing directory could be named anything).
• Therefore, the backend builds an editable wheel which contains coherent/build/__init__.py.
  • The coherent/ naturally represents the PEP 420 namespace package,
  • build/ represents the subpackage, and
  • __init__.py is a proxy module that when imported will replace itself with the functionality from ./__init__.py and also adjust its own __path__ to represent os.getcwd() aka ., e.g. ../coherent.build.

While the indirection is a little disconcerting, the implementation is tight (just 12 lines of code plus a template) and flexible (works for namespace and simple packages alike) and has proven reliable (after working out encoding flaws) and compatible across editable and non-editable installs (across namespaces in the same installation).

I imagine Setuptools could do something similar, taking the package_dir declaration (mapping package -> dir) and creating an explicit proxy from that package (possibly in a namespace) to its directory.

[jaraco]

Applying that concept to the above example, I realize the problem is more complicated, because:

• Setuptools is proxying every package from python/.
• One of the packages in python/ is test, but that package is not a PEP 420 namespace package, therefore, it's not composable with other packages containing test.*.
• The other third-party packages are subpackages of test.

Although Python does support PEP 420 namespace packages as subpackages of simple packages, I'm not sure that's a well-supported scenario, and that's what's happening here.

[jaraco]

So we can see here that, if it would come to the meta path finder itself, it would have found the module

Right, but the reason it never comes to that meta path finder is because the import machinery sees import test.b and first resolves test and uses its __path__ or submodule_search_locations to resolve test.b. I'm slightly surprised that test.a loads successfully.

[jaraco]

Interesting.

So we can see here that, if it would come to the meta path finder itself, it would have found the module, but I suspect that the import machinery is not reaching that stage and it halts before reaching the meta path finder.

That's true, but it fails to find test.b:

>>> editable.find_spec('test.b') is None
True

[jaraco]

Here's where the editable finder requires that the package be a simple package:

setuptools/setuptools/command/editable_wheel.py

Lines 834 to 838 in 56962ec

	init = candidate_path / "__init__.py"
	candidates = (candidate_path.with_suffix(x) for x in module_suffixes())
	for candidate in chain([init], candidates):
	if candidate.exists():
	return spec_from_file_location(fullname, candidate)

[jaraco]

It looks like adding support for nested namespace packages may work around the issue (#4954). I created this Dockerfile:

FROM jaraco/multipy-tox

# disable pip-run
ENV PYTHONPATH=

WORKDIR /tmp/pkg

RUN py -m venv .venv

# get rid of the test module for clarity
RUN rm -r /usr/lib/python3.13/test

RUN mkdir -p python/test third_party/a third_party/b

RUN touch python/test/__init__.py \
    third_party/a/__init__.py \
    third_party/a/foo.py \
    third_party/b/foo.py

RUN echo 'from setuptools import setup\n\
setup(\n\
    name="test",\n\
    packages=["test", "test.a", "test.b"],\n\
    package_dir={\n\
        "": "python",\n\
        "test.a": "third_party/a",\n\
        "test.b": "third_party/b"\n\
    }\n\
)' > setup.py

RUN echo '[build-system]\n\
requires = ["setuptools>=42"]\n\
build-backend = "setuptools.build_meta"' > pyproject.toml

RUN py -m pip install -q -U pip setuptools@git+https://github.com/pypa/setuptools@feature/4943-namespace-child-simple

RUN py -m pip install --no-build-isolation -e .

When running it, it emits:

 🐚 docker run -it @$(docker build -q .) py -c "import test.a; import test.b; print('done')"
done

[jaraco]

@abravalheri Would you be willing to examine that draft PR and decide if and how you'd like to employ that change? Thanks.