Web Development

Devenv, python, and packaging Lambdas

One of the first things I did with Nix flakes and then Devenv.sh was standardize the Node environment for my AWS CDK deployments. A related issue that I let slide at the time was Python dependencies for when a project has Lambda functions with Python runtimes.

My comfort level with Python isn’t very high, and the variety of Python dependency tooling (pip, Poetry, pyenv, venv) leaves me bewildered on a good day. The Lambda functions that I ship aren’t complicated and don’t need many dependencies, although they need to be compatible with the given runtime. A further complexity specific to Lambda functions is that AWS expects the function and its dependencies to be zipped up together.

My preferred way to handle that pre-devenv was to use pip with the -t flag, which lets me choose the installation directory. This will install the requested package and its dependencies in that directory, which is perfect for my packaging needs. Devenv has excellent support for Python, and getting Python 3.12 is easy:

devenv.nix:

1
2
3
4
languages.python = {
  version = "3.12";
  enable = true;
};

devenv.yaml:

1
2
3
4
5
6
inputs:
  nixpkgs-python:
    url: github:cachix/nixpkgs-python
    inputs:
      nixpkgs:
        follows: nixpkgs

Trouble is, that configuration leave me with working python (good) but no pip (not so good). After some unhelpful googling I started playing around with Poetry, but it felt like overkill for what I was looking to accomplish. Ultimately I decided to try venv:

1
2
3
4
5
languages.python = {
  version = "3.12";
  enable = true;
  venv.enable = true;
};

Without venv, this is what my python environment for this project looked like:

1
2
3
4
5
(nix:devenv-shell-env) (devenv) $ which python
/nix/store/6z1qpvp2pyafhkvzsgfh8ivsy8p27c7m-python3-3.12.7-env/bin/python
(nix:devenv-shell-env) (devenv) $ python --version
Python 3.12.7
(nix:devenv-shell-env) (devenv) $ which pip

With venv, it looks like this:

1
2
3
4
5
6
7
8
(nix:devenv-shell-env) (devenv) $ which python
/path/to/project/.devenv/state/venv/bin/python
(nix:devenv-shell-env) (devenv) $ python --version
Python 3.12.7
(nix:devenv-shell-env) (devenv) $ which pip
/path/to/project/.devenv/state/venv/bin/pip
(nix:devenv-shell-env) (devenv) $ pip --version
pip 24.3.1 from /path/to/project/.devenv/state/venv/lib/python3.12/site-packages/pip (python 3.12)

The good news is here that I have pip now, so the immediate problem is solved. I did find myself a little curious about what’s happening in the root .devenv directory:

1
2
3
4
5
6
-rwxr-xr-x 1 user group  271 Nov 12 22:35 pip
-rwxr-xr-x 1 user group  271 Nov 12 22:35 pip3
-rwxr-xr-x 1 user group  271 Nov 12 22:35 pip3.12
lrwxr-xr-x 1 user group   10 Nov 12 22:35 python -> python3.12
lrwxr-xr-x 1 user group   10 Nov 12 22:35 python3 -> python3.12
lrwxr-xr-x 1 user group   77 Nov 12 22:35 python3.12 -> /nix/store/6z1qpvp2pyafhkvzsgfh8ivsy8p27c7m-python3-3.12.7-env/bin/python3.12

That’s about what I expected. A symlink back to the same immutable python binary, and the usual pip module (module?) in the project directory. This gets around problems I’d had earlier with enabling pip, which tried to put it in the immutable /nix/ store, which didn’t work. This may have been the obvious solution for someone experienced with Python, but I’m not.