Quick Deep Learning Python Development Environment with Poetry

Trying to find a smoother developer experience compared to virtualenvs

Created: Sep 22, 2023 by Pradeep Gowda Updated: Nov 04, 2023 Tagged: python · poetry · xtts · deeplearning

Let’s say you want to start playing with a new python library and a framework ¹, and you don’t want it to mess up anything else you have installed on your system.

The common steps involved in trying out some new library/framework are

1. Install the package (via pip)
2. read the documentation
3. follow the “hello world” equivalent example
4. if you like what you see, include the library in your own “full sized” projects.

Often step 1 trips up people because they do pip install foo and that specific package+version combo might conflict with a dependency baz installed by yet another package bar; this is a fairly common occurrence ² in all kinds of software, not just Python.

What you need is a self-contained, isolated environment, often called a virtualenv.

The typical virtualenv steps to achieve the same things as above are:

$ mkdir newstuff
$ cd newstuff
$ python3 -m venv venv
# the second venv is the directory name. can be anything of your choice, but many IDEs recognize .venv and venv and pick it up use it with your code.

$ source venv/bin/activate
$ pip install TTS

But, even if one becomes familiar with virtualenv, there remains a small user experience irritation.

1. separate venv directory hanging around along with the code
2. having to source venv/bin/activate. not exactly “intuitive”
3. having to remember what’s installed in this venv (you can query it with pip list or pip freeze)
4. etc

We can improve the UX with Python Poetry. You can read why poetry beats pip setup for some rationale, I’ll focus on how the typical workflow changes with it. Let’s look at how I used it today - to play with TTS.

First install OS lib dependencies ³ and then:

➜  sandbox mkdir ttslearn
➜  sandbox cd ttslearn
➜  ttslearn poetry init

This command will guide you through creating your pyproject.toml config.

Package name [ttslearn]:
Version [0.1.0]:
Description []:
Author [Pradeep Gowda <me@btbytes.com>, n to skip]:
License []:
Compatible Python versions [^3.11]:

Would you like to define your main dependencies interactively? (yes/no) [yes]
You can specify a package in the following forms:
  - A single name (requests): this will search for matches on PyPI
  - A name and a constraint (requests@^2.23.0)
  - A git url (git+https://github.com/python-poetry/poetry.git)
  - A git url with a revision (git+https://github.com/python-poetry/poetry.git#develop)
  - A file path (../my-package/my-package.whl)
  - A directory (../my-package/)
  - A url (https://example.com/packages/my-package-0.1.0.tar.gz)

Package to add or search for (leave blank to skip): tts
Found 20 packages matching tts
Showing the first 10 matches

Enter package # to add, or the complete package name if it is not listed []:
 [ 0] TTS
 [ 1] tts369
 [ 2] TTS2
 [ 3] tts-arranger
 [ 4] xfyun-tts
 [ 5] macosx-tts
 [ 6] vnpy-tts
 [ 7] google-tts
 [ 8] mPython-tts
 [ 9] sanskrit-tts
 [ 10]
 > 0
Enter the version constraint to require (or leave blank to use the latest version):
Using version ^0.17.4 for TTS

Add a package (leave blank to skip):

Would you like to define your development dependencies interactively? (yes/no) [yes] no
Generated file

[tool.poetry]
name = "ttslearn"
version = "0.1.0"
description = ""
authors = ["Pradeep Gowda <pradeep@btbytes.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.11"
TTS = "^0.17.4"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"


Do you confirm generation? (yes/no) [yes] yes

The above chioces were written to pyproject.toml file. ⁴

I ran poetry install to install the dependencies I mentioned above. I activate the “shell” which has all the dependencies installed with poetry shell.

poetry shell
Spawning shell within /Users/pradeep/Library/Caches/pypoetry/virtualenvs/ttslearn-MgJDGpNz-py3.11
➜  ttslearn emulate bash -c '. /Users/pradeep/Library/Caches/pypoetry/virtualenvs/ttslearn-MgJDGpNz-py3.11/bin/activate'

You can see the python interpret is provided by the virtual environment specified by the pyproject.toml configuration.

Immediately a few things stand out:

1. the menu driven approach.
2. poetry -h and poetry list
3. The dependencies (and exact version number) are captured in the pyproject.toml; you always know what version of the software you had installed to work “here”. (You can add more dependencies with poetry add torch
4. The shell is where I do all my work, whether its the REPL with all the deps installed or run a script in the ttslearn directory

I can now take the new library for a spin:

(ttslearn-py3.11) ➜  ttslearn python3
Python 3.11.2 (main, Feb 26 2023, 17:21:36) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> from TTS.api import TTS
>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> model_name = TTS().list_models()[0]
No API token found for 🐸Coqui Studio voices - https://coqui.ai
Visit 🔗https://app.coqui.ai/account to get one.
Set it as an environment variable `export COQUI_STUDIO_TOKEN=<token>`

That looks good enough to know that the library is installed correctly.

Advanced usage: If you decide to convert your “script” into an actual application, it is very easy to do so because Poetry handles many things for you including packaging your app for distribution with poetry build or add custom entry points.

Conclusion: I set out to see if Poetry can indeed be better than virtualenv based setup. So far it shows clear upgrades over that approach.

fin.