Why I wrote PEP 832 -- virtual environment discovery

While I decide what to do with PEP 832 after polling folks on their opinion, I thought I would write out why I'm even bothering with any of this.

I'm going to talk from the perspective of VS Code and its Python extensions, but you could just as easily substitute "VS Code" for your editor of choice or even "AI agent" and it wouldn't change the problem: it isn't necessarily easy for tools like VS Code to know what workflow (tool) you're using and thus where you're putting your (virtual) environment(s) (I'm going to say "environment" as a stand-in for virtual environments, conda environments, etc.). Knowing where the environment lives is important in order to know how to run your code (as the environment will have a Python interpreter that you can use), analyze your dependencies (so linting, auto-complete, etc. do the right thing), etc. So having a way to communicate to VS Code where to find the environment is important.

The problems

First time seeing a project

When you first open a project in VS Code, you may have just done a git checkout, so nothing is set up yet. How is VS Code to know what workflow tool you prefer for creating an environment? Do you prefer Hatch? Poetry? uv? No preference? Some custom solution you have just for you (my-tool)? VS Code could guess maybe based on some tool table in pyproject.toml, but that requires having a known list of tools to look for. And while I know some people will say, "just assume uv", I've been doing this long enough to know not everyone uses that tool and it isn't guaranteed to be the tool of choice forever (I remember when pipenv was what everyone recommended). Right now there is no way to know what tool a project wants to see used. Same goes for if you have a personal preference when a project has no reason to care. VS Code could have you specify such a preference in your settings or in .vscode/settings.json for a project, but that would then be VS Code-specific and would only work for supported workflow tools that were hard-coded (sorry, my-tool).

What if you ran git checkout , ran your workflow tool to create the environment, and then opened the project in VS Code? There might be an extra hint about which tool was used if VS Code is able to find the environment on its own, but that assumes VS Code even knew where to look for an environment for that hard-coded list of tools. This also has the same issue of needing VS Code to know where various tools put their environments as well as tie the environment back to the project. And that sort of thing is typically an implementation detail, and thus prone to change unexpectedly.

Finding all the environments

Tying into finding the environments, how do you know which environments are meant for the project? Some tools keep environments locally with the project files, but some keep them in a global location. And when that happens you need specific knowledge to map the environments back or hope the tool has a CLI call that returns a list of the environments in a way that never changes.

And then there's the added wrinkle of workflow tools that never tie an environment to one project. Tools like virtualenvwrapper and conda let you name an environment so they can be shared across projects. But then that means VS Code can't know what environment to use without asking. And when someone has 700 or more environments (and that number is from actual experience; it isn't an exaggeration), it doesn't make is easy to know what environments to suggest.

Possible solutions

Let's work backwards from already having an environment to figuring out what workflow tool to run.

The simplest thing is when there's a virtual environment in a .venv directory. It's easy to discover and can thus be quickly used. It doesn't help you modify the environment as you don't know what workflow tool was used and whether it cares that it is used to make any changes to the environment, but at least you can run code and analyze the environment.

Solving for when an environment is kept outside of a project or if there is more than one can be done with a file that records such details. Right now I'm proposing a .python-envs file which is just a newline-delimited list of paths to environments, with the last one being the one to use by default if the user doesn't want to have to choose (FYI what's in the PEP as of this writing is out-of-date).

Knowing what workflow tool a project wants you to use is currently suggested to be done by a [workflow] table in pyproject.toml. That would specify how to launch a workflow tool in a server mode implementing the workflow server protocol (WSP, which has a nice connotation for whitespace in grammar definitions). That would let the tool communicate via JSON-RPC over stdin/stdout with VS Code about what environments there are, creating environments, etc. VS Code would provide a way to specify a fallback tool for when a project doesn't have a preference, but there currently isn't any thinking about how to make specifying a fallback a standard.

An alternative I have proposed in the past is having some naming convention for CLI tools and then they return the necessary details. This could still use WSP or come up with some CLI standard that tools implement, but it would help solve the issue of not knowing what fallback to use.

Why I care about any of this

VS Code is used across the experience spectrum: from first-time programmers to senior programmers with decades of experience. One common theme across experience levels is no one likes having to set something up. Another theme is no one likes dealing with environments. As such, I'm trying to find as much of a solution as people will agree to that makes getting started as easy as possible and hides environments as much as possible. And I want to do all of this in a way that isn't specific to VS Code if it doesn't have to be.

On top of that, people often don't want to leave VS Code to do any workflow stuff. That makes VS Code a user of the workflow tool rather than a peer. Thus VS Code uses the workflow tools as middleware which they aren't usually designed to do. With no tool-to-tool communication standard you end up with bespoke support like the Hatch extension which uses public APIs from the Python environments extension, which means it doesn't easily scale to help other tools. It's great for VS Code users that extension exists, but other editors don't get that benefit. As well, the extension takes effort and that takes away from what could have been a general solution for any editor.