lists of tuples seem to assume Literal too eagerly, causes errors with concatenate + operator

Pylance v2022.1.4-pre.1
Pylance language server 2022.1.4-pre.1 (pyright 98868f2) starting
Fedora 34
Python 3.10.0

if you have a list of tuples where one of the lists has the same tuple multiple times, it types them all as Literal and then doesnt allow you to concatenate another kind of list

# these all pass

x = [1, 2, 3] + [4, 5, 6]

y = [(1, "a"), (2, "b"), (3, "c")] + [(4, "d"), (5, "e"), (6, "f")]

# these dont fail even though True / False is repeated, not sure if it's assuming
# Literal on those 
z = [(1, True, "a"), (2, True, "b"), (3, True, "c")] + [
    (4, False, "d"),
    (5, False, "e"),
    (6, False, "f"),
]

q = [(1, True, 5), (2, True, 6), (3, True, 7)] + [
    (4, False, 8),
    (5, False, 9),
    (6, False, 10),
]

# this one fails, because it assumes Literal for repeated values
p = [(10, True, 15), (10, True, 15), (10, True, 15)] + [
    (4, False, 8),
    (5, False, 9),
    (6, False, 10),
]


# real world case uses a list comprehension (also fails)
r = [(10, True, i) for i in range(3)] + [(4, False, j) for j in range(3)]

/home/classic/dev/sqlalchemy/test4.py
  /home/classic/dev/sqlalchemy/test4.py:20:5 - error: Operator "+" not supported for types "list[tuple[Literal[10], Literal[True], Literal[15]]]" and "list[tuple[Literal[4], Literal[False], Literal[8]] | tuple[Literal[5], Literal[False], Literal[9]] | tuple[Literal[6], Literal[False], Literal[10]]]"
    Operator "+" not supported for types "list[tuple[Literal[10], Literal[True], Literal[15]]]" and "list[tuple[Literal[4], Literal[False], Literal[8]] | tuple[Literal[5], Literal[False], Literal[9]] | tuple[Literal[6], Literal[False], Literal[10]]]" (reportGeneralTypeIssues)
  /home/classic/dev/sqlalchemy/test4.py:28:5 - error: Operator "+" not supported for types "list[tuple[Literal[10], Literal[True], int]]" and "list[tuple[Literal[4], Literal[False], int]]"
    Operator "+" not supported for types "list[tuple[Literal[10], Literal[True], int]]" and "list[tuple[Literal[4], Literal[False], int]]" (reportGeneralTypeIssues)
2 errors, 0 warnings, 0 informations

mypy will emit errors if the lists of tuples dont have the same format of tuple, but it doesn't assume repeated values are Literal, this seems like a bug.

Yes, this is expected behavior in this case. There are tradeoffs for different inference strategies. We've tuned the inference rules for pyright over the past couple of years based on real-world code, and for tuples (which are immutable), we've found that inferring the narrowest type possible works best in most situations. There are cases where it doesn't work as well, like the one you've highlighted above. We're unlikely to change the behavior because it will produce more false positives than it will solve, and many pyright users rely on the current behavior.

For full information about the inference rules used in pyright, please refer to this documentation.

I've seen an example of this too. It's arguably a typeshed issue, where we should allow list[T] + list[U] == list[T | U], but attempts to change the signature of list.__add__ have been tricky (e.g., python/typeshed#5260).

OK, here's my actual code. how do i type this, do i have to use cast ?

    @testing.combinations(
        *([
            (
                filename,
                True,
                testing.exclusions.closed()
                if not _whitelist.search(filename)
                else testing.exclusions.open(),
            )
            for filename in _file_combinations("plugin_files")
        ] + [
            (
                filename,
                False,
                testing.exclusions.open()
            )
            for filename in _file_combinations("plain_files")
        ]),
        argnames="path,use_plugin",
    )
    def test_files(self, mypy_runner, path, use_plugin):
       ...

like this. this works. im not yet enthusiastic about this behavior but I guess I dont have too many places with a concatenation like this.

    @testing.combinations(
        *(
            cast(
                List[Tuple[Any, ...]],
                [
                    (
                        filename,
                        True,
                        testing.exclusions.closed()
                        if not _whitelist.search(filename)
                        else testing.exclusions.open(),
                    )
                    for filename in _file_combinations("plugin_files")
                ],
            )
            + cast(
                List[Tuple[Any, ...]],
                [
                    (filename, False, testing.exclusions.open())
                    for filename in _file_combinations("plain_files")
                ],
            )
        ),
        argnames="path,use_plugin",
    )

Michael Bayer (@zzzeek)

Michael Bayer (@zzzeek) I'd suggest to factor out the tuple into a function:

def _get_args(filename: str, use_plugin: bool) -> Tuple[str, bool, ???]:
    if use_plugin and not _whitelist.search(filename):
        exclusion = testing.exclusions.closed()
    else:
        exclusion = testing.exclusions.open()
    return (filename, use_plugin, exclusion)

@testing.combinations(*([_get_args(filename,True) for filename in _file_combinations("plugin_files")] + [_get_args(filename,False) for filename in _file_combinations("plain_files"0]

I use a list literal with unpacking in these cases. Here is an example to illustrate:

def F(xs: list[int], ys: list[str], b: bool) -> list[str | int]:
    if b:
        return xs + ys      # type error
    else:
        return [*xs, *ys]   # no type error

After much deliberation, I've decided to revisit this issue and change the way that pyright infers tuple expressions. In particular, when a tuple expression is embedded within another container (list, dict, set, or tuple) expression, the type checker will no longer retain literal values for the tuple. This change will be included in the next release of pyright.

This is included in pyright 1.1.365.

Navigation Menu

Repository navigation

lists of tuples seem to assume Literal too eagerly, causes errors with concatenate + operator #2940

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

lists of tuples seem to assume Literal too eagerly, causes errors with concatenate + operator #2940

Description

Activity

erictraut commented on Jan 26, 2022

JelleZijlstra commented on Jan 26, 2022

zzzeek commented on Jan 26, 2022

zzzeek commented on Jan 26, 2022

JelleZijlstra commented on Jan 26, 2022

danr commented on Jan 27, 2022

erictraut commented on May 22, 2024

erictraut commented on May 29, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions