Python Type Hinting in Data Science Projects: A Must, a Maybe, or a No-No?
PYTHON PROGRAMMING
Should we use type hints in data-science projects realized in Python?
Want a disclaimer? Here you are: It depends. In Proof-of-Concept types of projects, it's often unnecessary. In production projects, at least in 2023, it rather is. But again, that depends.
I will try to be as concise as possible, and get to the point as fast as possible. I don't want to spend hours on considering all pros and cons, for the simple reason that the data-science market has quite clear expectations of our work. My goal is to present these expectations to you, not to discuss them in detail.
Let's start by stating the obvious. First of all, type hints in Python are optional. Optional like in you don't have to use type hints in Python. If so, there's only one answer to our main question: You can but no, you don't have to use type hints in Data Science projects!
So… that's it? Are we done here?
Wait a second. We did state the obvious, but we did not touch upon anything beyond the obvious.
Should we use type hints in data-science projects realized in Python? It depends. In Proof-of-Concept types of projects, it's not necessary. In production projects, at least in 2023, it rather is.
An example. Imagine you're a Python developer working for a private company. The company has its own rules and recommendations for Python development. One of the rules is: Use type hints. That's it – irrespective of what you prefer, you have to use them. Had this been a recommendation, you wouldn't have had to use them. However, since this is a rule, you have to use optional type hints.
Okay, good point. But we're talking about type hints in data science projects in general, not in a particular company. So, optional, right? You don't have to use them?
Before answering, let me tell you how, when and why I use type hints in Python.
I wrote what I think about type hints here:
To put it short, I try to use type hints in a way that makes the code more readable. Also, thanks to type hints, static checkers can help keep code correct.
We should remember that in the heart of Python lies dynamic typing.
At the same time, we should remember that in the heart of Python lies dynamic typing. And when we put much effort on implementing type hints, we're sort of taking out this dynamic typing from Python. And what's left? What's left is Python without its heart.
When we put much effort on implementing type hints, we're sort of taking out this dynamic typing from Python. And what's left? What's left is Python without its heart.
The way I see this, there are situations in which type hints should not be used. Like quick prototyping. I do that very often: in order to see how something could work, I implement a simple prototype. Sometimes I may use some type hints, just to show the required types of arguments, especially if they are to be some custom types. Remember, I am talking about prototyping, and if it's important to state that a particular function returns an object of dict[str, tuple[int, str]], then I do state it in a type hint. Not make static checkers happy, but to show what needs to be shown.
But more often than not, I don't care during prototyping about type hints. The time will come for them. What's important for now is that the code would work dynamically. This is when static types are not that useful.
But when I write a data-science software product, these days I always use type hints. I will be honest with you. Yes, they can (dynamically) help catch some static errors – but they can be quite a hindrance. Sometimes I feel implementing good type hints takes more time than everything else taken together. The code is much longer and much more complicated. In advanced projects, good type hints can be difficult to implement, mainly due to the complexity of the code. What makes things worse is that many type checkers are still far from optimal, showing errors where type hints are correct. Sure, you can keep type hints overly simplistic (x: dict), but usually you shouldn't do that. In a production project, you should go into more detail, so instead of doing this:
from typing import Optional
def foo(x: dict) -> Optional[dict]:
...you may need to do, for instance, something like this:
def foo(x: dict[str, dict[str, float]]) -> Optional[dict[str, str]]:
...or, equivalently:
from typing import Optional
Params = dict[str, dict[str, float]]
Descriptions = dict[str, str]
def foo(x: params) -> Optional[Descriptions]:
...Without type hints, the code becomes:
def foo(x):
...Which is better? You won't be surprised to see my answer: Well, that depends.
Let's summarize the options provided above:
- No type hints: fast coding based on Duck Typing.
- Overly simplified type hints: only a little slower coding, but the advantages of such type hinting are rather minor.
- More detailed type hints: much help from static checkers at the cost of much slower coding, but also a much bigger risk for static checkers to fail; usually, duck typing becomes hidden if not forgotten; less readable code.
- More detailed type hints based on type aliases: like above, but the code is much more readable.
- Extremely detailed type hints, to the very deepest level of detail: for me, definitely too fanatical approach without advantages and plenty of disadvantages, the same as for detailed type hinting – but in an exaggerated form.
- Dedicated tools such as
[pylance](https://pypi.org/project/pylance/)or[typeguard](https://pypi.org/project/typeguard/). Do remember, however, that such tools may lead to runtime efficiency costs, if you decide to use them in for runtime type checking – while type hinting itself is a static checking tool that has no effect on runtime.
As often, the best choice lies somewhere in between those options, in a place you would call a golden choice. But where this choice lies depends on several aspects of the projects, such as expectations from the client, the company, the project leader, and – most of all – the type of the project.
While prototyping, you will often choose not to use type hints, especially when you're not a devoted fan of type hinting. When you are, however, more often than not you will find yourself using them – unless the deadline is too close to spend even an additional unnecessary minute. Is it unnecessary? Well, you already know the answer… It depends.
The above approach is what I follow in my daily data science work. Some details vary from project to project, but the generalities are almost always the same. I suppose these are the rules that most companies conducting data-science projects follow, and so you may wish to follow these rules as well – unless the company or project rules are strict. Then you simply don't have options to choose from.
Whether or not you're a happy type-hinting user, you should know how it works and how to use it. These days, a Python developer who does not know how to use type hints… is not a Python developer anymore. Accept that: you must know type hinting, and you must know it well.
Do remember, however, that the Python type-hinting system is helpful only to some point of complexity. Even if you, as the code's author, understand all the nuances of complex type hints you implemented, other developers will likely need to spend much time to understand the code. If I was forced to cross this line, I would've thought that maybe it was time to change the language. Being forced to go that far, I'd likely think a static-typing language like Go would do better.
Thanks for reading. If you enjoyed this article, you may also enjoy other articles I wrote; you will see them here. And if you want to join Medium, please use my referral link below:
