What is pyproject.toml in Python
Dependency management in Python is tricky, and sometimes frustrating work. Newcomers, are usually tempted to install any dependency (i.e. package) they may find useful, even in a single virtual enviroment. Therefore, this approach increases the chances of having conflicting package dependencies and ending up in the so-called dependency hell.
In a couple of my previous articles, we covered a few different methods for handling dependencies in Python projects, using setup.py, setup.cfg and requirements.txt files. However, as of Python 3.6, a new standard configuration file called pyproject.toml was introduced, and aims to simplify the way users manage dependencies and metadata definitions.
Over the last few years, the pyproject.toml file has become the standard (and most popular) way for managing depdendeices in Python projects. In the next few sections we will explore how dependency management can be achieved using this file. Furthermore, we will also demonstrate how to install a Project with pyproject.toml specification in editable mode.
Subscribe to Data Pipeline, a newsletter dedicated to Data Engineering
Dependency management prior to pyproject.toml
When Python was first released, the defacto package used for building distributions was distutils. Over time, setuptools made its appearence aiming to build additional features on top of distutils. Both tools made use of a setup.py file where users could specify dependencies and metadata used as part of the package build distribution.
This however, has created an issue given that any project that chooses to use setuptools must be import the package in setup.py file. Therefore, setup.py cannot be executed without knowing its dependencies but at the same time, the purpose of the file itself is to determine these dependencies. And this is how we ended up with the so-called chicken and egg problem in Python dependency management.
I hope this information is enough to understand why a new approach was required. If you are interested in learning more about a more detailed explanation of the chicken-and-egg problem with setuptools and pip make sure to read PEP-518.
The new propasal, which is part of PEP-518, aimed to specify a new way for Python projects to list their dependencies upfront, so that tools like pip can make sure they are installed prior to the project build.
The pyproject.toml
The pyproject.tom file was introduced as part of the Python Enhancement Proposal (PEP) 518, that specifies how Python projects must specify build dependencies.
These build dependeencies will be stored in the file that is located at the root directory of the project and follows the TOML (Tom's Obvious, Minimal Language) syntax.
It contains metadata information such as the project name, version, description, author, license, and various other details.
One of the key features of the pyproject.toml file is the ability to define project dependencies. This allows developers to specify the packages and their versions required for the project to run properly. This helps in maintaining the consistency of the project and ensures that the project can be easily reproduced by other developers.
The pyproject.toml file also supports the concept of extras which allows developers to define optional dependencies for a project. This allows users to install only the necessary dependencies in order to run the project. Usually, in the extras section one could specify additional requirements that will be used as part of testing (e.g. pytest).
In addition to the standard metadata and dependencies, pyproject.toml file also supports custom fields that can be used by third-party tools. As an example, you can consider linters, formatters and checkers such as black and mypy. This allows developers to extend the functionality of the file and add custom fields as per their requirements.
Managing dependencies in pyproject.toml
pyprojet.toml can be used with package dependency management tools, such as setuptools and poetry.
Here's an example file for a project using poetry:
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
[tool.poetry]
name = "my-project"
version = "1.0.0"
description = "My Python project"
authors = ["John Doe <[email protected]>"]
license = "MIT"
[tool.poetry.dependencies]
python = "^3.6"
[tool.poetry.dev-dependencies]
pytest = "^4.6"
[tool.poetry.extras]
docs = ["sphinx"]And here's an example with setuptools:
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
[project]
name = "my_package"
description = "My package description"
readme = "README.rst"
requires-python = ">=3.7"
keywords = ["one", "two"]
license = {text = "BSD 3-Clause License"}
classifiers = [
"Framework :: Django",
"Programming Language :: Python :: 3",
]
dependencies = [
"requests",
'importlib-metadata; python_version<"3.8"',
]
dynamic = ["version"]
[project.optional-dependencies]
pdf = ["ReportLab>=1.2", "RXP"]
rest = ["docutils>=0.3", "pack ==1.1, ==1.3"]
[project.scripts]
my-script = "my_package.module:function"Installing project in editable mode from pyproject.toml
If you are actively developing a project, the chances are you may want to install the project locally in editable mode. When installing a package in editable mode from a specific location, any changes made to the source code will be reflected immediately in the environment (without you having to re-install the "new" version).
Assuming you are using poetry to manage your Python dependencies and in order to install a Python project in editable mode, you need to have the following content in your pyproject.toml file
[build-system]
requires = ["poetry-core>=1.0.8"]
build-backend = "poetry.core.masonry.api"and from the project's root directory, simply run
$ pip install -e .Alternatively, a poetry install will also result in editable install. You can find out more on how to manage your Python projects' dependencies with Poetry, in one of my latest articles:
Final Thoughts
In today's article we discussed about the usage of pyproject.toml in Python when it comes to managing dependencies and distributing projects across the community.
Overall pyproject.toml provides a standard and easy-to-use configuration for Python projects. It simplifies the process of defining metadata and dependencies, and ensures that the project can be easily reproduced by other developers.
Subscribe to Data Pipeline, a newsletter dedicated to Data Engineering
Related articles you may also like
