Python Types: Optional Can Mean Mandatory

PYTHON PROGRAMMING

According to the Python documentation, typing.Optional is a handy way to indicate that an object can be None. It's a concise and elegant way to express this concept, but is it also crystal clear?

Let me rephrase this question: When you see the word "optional" in a Python context, what do you think it means? Imagine you see an argument called x that has the type of Optional[int]. The int part is rather clear, as most likely indicates an integer, but what does Optional mean? What's your first thought?

Let's consider the following two options:

1. I don't have to provide a value of x because it's optional.
2. x value can be either int or None.

If you know Python type hinting well enough, you know option 2 is correct. But when you don't… Maybe I'm wrong, but I cannot imagine any person who doesn't know Python choosing option 2. It is option 1 that seems to make most sense. When I see information that something is optional, I think that… well, that it's optional…

This issue leads to a frequent misuse of the Typing.Optional type. This article aims to shed light on this misuse and guide you towards the correct understanding of this type.

The meaning of typing.Optional

These three type hints are equivalent:

from typing import Optional, Union

x: Union[str, None]
x: Optional[str]
x: str | None

Each of them conveys the same information: that x can be either a string or None. While perfectly valid, the first one (Union[str, None]) represents the early stages of type hinting in Python: it was the initial approach, but it's not necessarily the preferred method nowadays. Then, Optional was added to the typing module, providing a more concise and straightforward way to express this concept. According to the mypy documentation:

You can use the [Optional](https://docs.python.org/3/library/typing.html) type modifier to define a type variant that allows None, such as Optional[int] (Optional[X] is the preferred shorthand for Union[X, None]).

Eventually, in Python 3.10, the | operator was introduced in type hinting. As the mypy documentation says,

PEP 604 introduced an alternative way for spelling union types. In Python 3.10 and later, you can write Union[int, str] as int | str.

As you see, this is a general operator for union types, not specifically designed to indicate that a variable can be None.

While each of the three versions is valid, the choice should depend on several factors. First, if you're using an older Python version than 3.10, the | operator is unavailable. Even with the __future__ import:

from __future__ import annotations

it may still fail in some cases. You can read about this in the mypy documentation.

Nevertheless, I suggest not using the Union type to indicate the possibility of being None, as it's unnecessarily verbose and, as the documentation quoted above says, it's not the preferred option anymore. Mypy recommends typing.Optional (quote: "Optional[X] is the preferred shorthand for Union[X, None]"). I vote for the same, for the simple reasons that the Optional type was created exactly for such use cases, and that it will work also for older Python versions, unlike the | operator.

Here are several examples of correct type hints that utilize Optional:

As you can see, Optional can be used in simple and complex type hints. Let's analyze only the middle one. The dict[str, Optional[int]] type indicates that a variable should be a dictionary, with keys being strings and values being either integers or None.

This article is not about making this choice, however. I'd like to discuss a frequent misuse of the typing.Optional type – and to show how to avoid it. In so doing, I will explain where I see the source of this misuse, how to correct it, and how to understand the typing.Optional type.

The Optional misunderstanding

Consider the following function signature:

from typing import Optional

def foo(s: str, n: Optional[int] = 1) -> list[str]:
    ...

Let's analyze the type hints in this function signature. However, don't get too attached to this analysis! And this is because these type hints can (though don't have to) be wrong. Here we are:

• s is a string (str) argument; it can be both positional and keyword argument, and it is required;
• n is an optional integer (int) argument, either positional or keyword, with the default value of 1;
• the foo() function returns a list of strings (list[str]).

Now a question: What's wrong with the above analysis?

It is the second bullet point that is wrong. It says that n is an optional integer. On the one hand, this is a perfectly valid English sentence. The n argument is indeed optional, because you don't have to provide it's value; when you omit it, the default value of 1 will be used.

On the other hand, this is not necessarily a valid typing sentence. What I mean is, this is an incorrect meaning of typing.Optional. Above, we used Optional[int] to mean the following: you don't have to provide the value of n. This meaning is incorrect. The correct meaning of Optional[int] in the typing syntax is as follows: n can be either int or None.

The image below summarizes the two understandings:

Let's improve the function signature. We have four options to choose from, each representing a different situations. Choose the one that suits your specific scenario.

As you will see, there will be option 0, in which the signature will remain unchanged. Yes, this type hint can be correct – but its meaning is rarely what you need.

Option 0: Leave as is

from typing import Optional

def foo(s: str, n: Optional[int] = 1) -> list[str]:
    ...

The type hint n: Optional[int] = 1 is fully correct. The point is, it means something else than many think because it means that

• n can be either int or None, and
• the default value of n is 1.

So, the default is 1, but the user can still provide None.

While it's technically correct, I'd use such a type hint only in very specific – and rare – circumstances in which this would be what is really needed. They are so rare that I have never found myself in a situation that would require this type hint. It simply sounds unnatural to me.

I am so harsh on this option because in my eyes, it is this type hint that makes typing.Optional so frequently misused: It suggests that n is optional because it has a default value, so you simply don't have to provide a value for this variable or argument.

I included this option because it's technically correct – but it's really something you should almost never consider. At the very least, remember that many less experienced Python users will most likely misunderstand this type hint.

Option 1: Use Optional and None as the default value

from typing import Optional

def foo(s: str, n: Optional[int] = None) -> list[str]:
    if n is None:
        ...
    ...

This is the most frequent scenario when you need None for an integer – or any other type. A default value (None) is used here as a sort of sentiment that triggers a particular action. So, if the user provides an integer, some integer-based processing is done. But when n is None, this processing can be switched off whatsoever. Certainly, it's an example scenario, but it's a very frequent one .

Note the if block, which aims to conduct an explicit None check. Maybe it isn't needed in all such scenarios, but according to the mypy documentation:

Most operations will not be allowed on unguarded None or [Optional](https://docs.python.org/3/library/typing.html) values […] Instead, an explicit None check is required. Mypy has powerful type inference that lets you use regular Python idioms to guard against None values.

Option 2: Don't use Optional

def foo(s: str, n: int = 1) -> list[str]:
    ...

In this option, what you really need is a default value of n, but n cannot be None. Here, you simply don't have to provide the value of n, because, with a default value, you don't have to provide it's value in a call to foo(). So, this is the correct meaning of optional in English (the argument is optional because you don't have to provide it's value) but incorrect in the typing syntax (the argument is not Optional because n cannot be None).

Option 3. Use Optional for n but require its value

This use case shows why using typing.Optional does not make the argument optional. As discussed above, the typing.Optional type means that a variable can be None, but it does not mean that when it's used for a argument, you don't have to provide its value. So, this is perfectly valid code:

def foo(s: str, n: Optional[int]) -> list[str]:
    if n is None:
        ...
    ...

Even though n is Optional, it is not optional – you have to provide its value. Hence, you have to provide n, but it still can be None. Here, optional means that n is optionally int as it can also be None.

Like in option 1, more often than not you should use the explicit None check for n, as what I wrote about using None applies here as well.

Conclusion

We have discussed a frequent mistake related to the typing.Optional type. The mistake stems from the fact that while the name suggests that typing.Optional deals with optional arguments, it actually refers to whether a variable can be None – and has nothing to do with whether or not you have to provide the argument's value in a function call or not.

In my opinion, the term "optional" doesn't accurately convey the meaning of typing.Optional. However, it's a well-established Python term that's been around for a while, so I don't anticipate a change anytime soon – if ever whatsoever. Therefore, it's important to be aware of this potential misunderstanding. Let's hope that over time, the Python codebase will become less susceptible to typing.Optional misuse and misinterpretation.

---

Thanks for reading. If you enjoyed this article, you may also enjoy other articles I wrote; you will see them here. And if you want to join Medium, please use my referral link below:

Join Medium with my referral link – Marcin Kozak

Tags: Data Science Programming Python Tips And Tricks Typing

Comment