[RFC] Add optional automatic validation and conversion

@hynek

Validation and conversion of attribute values is currently only done if the user passes the corresponding arguments to attrib().

In contrast, libs like Pydantic automatically validate and convert all values (in __init__() as well as __setattr__()). It also allows the user to easily pass the most common validators like ge=0 or maxlength=50.

This lets you do fancy stuff like this, which is very useful, e.g., for REST API server models and clients, or generally for loading nested JSON data. Settings/config management is another usecase:

>>> from dateatime import datetime                                                    
>>> from pydantic import BaseModel                                                    
>>>                                                                                   
>>> class Child(BaseModel):                                                           
...     x: int                                                                        
...     y: int                                                                        
...     d: datetime                                                                   
...                                                                                   
>>> class Parent(BaseModel):                                                          
...     name: str                                                                     
...     child: Child                                                                  
...                                                                                   
>>> data = {                                                                          
...     'name': 'spam',                                                               
...     'child': {                                                                    
...         'x': 23,                                                                  
...         'y': '42',  # sic!                                                        
...         'd': '2020-05-04T13:37:00',                                               
...     },                                                                            
... }                                                                                 
>>> Parent(**data)                                                                    
Parent(name='spam', child=Child(x=23, y=42, d=datetime.datetime(2020, 5, 4, 13, 37)))

While pydantic works quite nicely, it is way slower than attrs and also uses inheritance which is not always what users may want.

I tried implementing a similar functionality but failed. IMHO, there are some changes needed directly within attr’s class-buildng facilities, as the following example illustrates:

from datetime import datetime
from functools import partial

import attr


# Two examples for pre-defined validators for later use
def check_ge(inst, attr, value, bound):
    if value < bound:
        raise ValueError(f'"{attr.name}" is not >= {bound}: {value}')

def check_le(inst, attr, value, bound):
    if value > bound:
        raise ValueError(f'"{attr.name}" is not <= {bound}: {value}')


def validating_class(auto_converters=True, **kwargs):
    # Extension wrapper for "attrs()"
    def wrap(cls):
        # if auto_converters:
        #   Here, I'd have to manually iterate/evaluate all attribs.
        #   This duplicates work in "attr.s()".
        cls = attr.s(auto_attribs=True, frozen=True, slots=True, **kwargs)(cls)
        # if auto_converters:
        #   It is very hard (impossible?) to update "cls" here and add (optional) 
        #   converters to all attributes
        return cls
    return wrap


def field(default, ge=None, le=None, **kwargs):
    # Extension wrapper for "field()"
    v = []
    # This does not work, b/c I cannot access type information here
    # if attr_type is int:
    #     if ge is not None:
    #         v.append(partial(check_ge, bound=ge))
    #     if le is not None:
    #         v.append(partial(check_le, bound=le))

    if 'validator' in kwargs:
        v.append(validator)
    if v:
        kwargs['validator'] = v
    return attr.ib(converter=int, validator=v)


@validating_class()
class Child:
    x: int = field(ge=0, le=100)
    y: int = field(ge=0, le=100)
    d: datetime  # auto-add "field(converter=datertime.fromisoformat)"


@validating_class()
class Parent:
    name: str
    child: Child = # auto-add "field(converter=lambda d: Child(**d))"


data = {'name': 'spam', 'child': {'x': 23, 'y': '42', 'd': '2020-05-04T13:37:00'}}
p = Parent(**data)
print(p)
# Parent(name='spam', child=Child(x=-1, y=0, datetime(2020, 5, 4, 13, 37)))

I already talked about this with @hynek and he asked me create this issue.

I also created an issue for orjson asking to add support for attrs.

Summary (what’s needed)

Optional conversions (and validation) of all attributes. Conversion should only happen when necessary (e.g., do not list(value) if value is already a list). This is the most important part.
Shortcuts for commonly used validators: attrib(ge=0, le=10).
An asjson() function that uses stdlib and provides an encoder that can handle attrs classes and datetimes and similar common types.
Explicit orjson support might not be needed when https://github.com/ijl/orjson/issues/92 gets implemented

Thanks for writing this up

I added a summary to the issue.

If you have no genereal objections against this issue, I can start working on it and see how far I get.

Hi,

in my personal opinion, structuring and conversion should not be part of the class but of something else. So I've created a library called cattrs that implements the something else (called a converter).

For example, in your writeup you're using something that looks like ISO 8601 for the datetime instance, but I think you'd be hard pressed to come to a consensus on this (why not unix timestamps?). Also if you want to use the pendulum DateTime instead, where does that logic live?

So pydantic is not equivalent to attrs, it's equivalent to attrs+cattrs. I'm not saying everyone should use cattrs (and I'm struggling to maintain it myself, if you look at the issues page...), I'm saying if we want this functionality in a blessed way for attrs, it should be another package under the attrs umbrella or another module in the attrs repository, but it should definitely be another component and not part of attrs as it exists now.

Super busy right now, but just to put a rough stake in the ground: I'm 100% interested in making it possible to write a package like pydantic using attrs (aka provide hooks and tools). But structuring and conversion remains out of scope for attrs proper.

I have tried cattrs and it is even faster than pydantic (except if you use dateutil’s parse()), but attrs+cattrs is imho lacking in user experience. You just have to learn and type too much in order to get basic conversion and evaluation done.

I want something, that is super easy to use for most use cases (and that makes the uncommon use cases possible, of course ;-)) and that is backed by attrs.

I don't necessarily want to create a new package. I’d be perfectly happy if attrs implemented the required hooks and if cattrs added another layer of abstraction for a UX like in the example above. :)

@sscherfke

I have tried cattrs and it is even faster than pydantic (except if you use dateutil’s parse()),

@sscherfke That got me interested, so I replaced dateutil with ciso8601 in pydantics benchmarks. You can see the results here: pydantic/pydantic#1568.

@sscherfke

@sscherfke So cattrs grew out of a set of converters doing what you suggest, actually. I just found that approach ultimately too limiting.

The simplest solution would be to just include a converter like:

def ensure_cls(cls):
    """If the attribute is an instance of cls, pass, else try constructing."""
    def converter(val):
        if isinstance(val, cls):
            return val
        else:
            return cls(**val)
    return converter

in attrs proper.

Your converter (converter=lambda d: Child(**d)) wouldn't work for passing in an existing instance of Child, though. The real type of the field then becomes a Union[Child, Dict[str, Any]] and this complicates Mypy integration. People will want an ensure_enum later too...

@fbergroth

@fbergroth

That got me interested, so I replaced dateutil with ciso8601 in pydantics benchmarks. You can see the results here: pydantic/pydantic#1568.

I discovered this while writing an article about attrs, data classes and pydantic which will be published later this week. I just replaced datetuil’s parse with datetime.fromisoformat() :)

@Tinche

So cattrs grew out of a set of converters doing what you suggest, actually. I just found that approach ultimately too limiting.

My idea is to offer a limited set of the most commonly used functionality via a very simple (and possibly limited) API but also provide a more powerful but also more complex API.

The main idea (and problem) is that you do not have to pass a converter to attrib() and that attrs automatically chooses one for you based on the attributes type:

# Just a rough example :)
for attrib in attribs:
    if has(attrib.type):
        c = lambda d: attrib.type(**d)
    elif issubclass(attrib.type, datetime):
        c = datetime.fromisoformat
    else:
        c = attrib.type
    
    # only apply converter if instance is not already of correct type:
    attrib.converter = lambda i: i if isinstance(i, attr.type) else c(i)

The real type of the field then becomes a Union[Child, Dict[str, Any]] and this complicates Mypy integration.

🤔 Yes, this may indeed be a problem. However, I just tested this in pydantic and mypy has no problems with this example:

from pydantic import BaseModel


class Child(BaseModel):
    x: int
    y: int


class Parent(BaseModel):
    name: str
    child: Child


x = Parent(name='spam', child=Child(x=1, y=2))
y = Parent(name='spam', child=Child(x=1, y='2'))  # y is a str!
d = {'name': 'spam', 'child': {'x': 1, 'y': 2}}
z = Parent(**d)  # As when loaded from json
ä = Parent(name=d['name'], child=d['child'])  # Just for fun :)

assert x == y
assert x == z
assert x == ä

I need to point out that datetime.fromisoformat() is absolutely not an adequate parser for ISO strings, as the Stdlib docs clearly say. It’s merely the inverse of datetime.isoformat().

I need to point out that datetime.fromisoformat() is absolutely not an adequate parser for ISO strings, as the Stdlib docs clearly say. It’s merely the inverse of datetime.isoformat().

That is (unfortunately) correct. A library like arrow can: arrow.parser.DateTimeParser().parse_iso(text)

I’m aware that fromisoformat() is very limited and that there are other/better parsers (like the ones arrow or pendulum provide). But if the use case is that you only transmit iso-dates generated and parsed by the stdlib, fromisoformat() works perfectly fine. Furhtermore, it is not yet time to discuss a proper parser for my suggested changes. ;-)

This RFC is the result of an article that I have been writing and that I now finally published: https://stefan.sofa-rockers.org/2020/05/29/attrs-dataclasses-pydantic/

Stefan mentioned to me in private he might get it to work using cattrs so this will wait for now.

I’d like to point out one thing tho: pydantic has a great DX but it comes at the price of correctness and flexibility. That is a very valid tradeoff to make and we can see people loving it for that.

However it’s not a trade-off I am willing to make. Correctness, explicitness, and flexibility are more important to me than typing extra characters. We already have a pydantic, so let’s try to be an actual alternative and not try to imitate them.

(JFTR, I’m not saying anyone wanted that; I’m just writing it out explicitly for clarity so please don’t feel anyone attacked :))

I really hope cattrs can rise to the task because it really seems to have its fans!

I needed a quick solution at work that works "good enough" and does not require changes within (c)attrs, so I just created another layer of abstraction above (c)attrs. 🙃

Here is an example:

from datetime import datetime

from mattr import (  # magic attrs – the only thing that mattrs :D
    frozen,  # attrs(auto_attrib, slots, frozen, kw_only)
    field,  # Wrapper around attrib() w/ validation shortcuts
    evolve,  # attrs.evolve()
    fromdict,  # cattrs.structure() with custom converter
    asdict,   # attrs.asdict()
    fromjson,   # cattrs.structure(json.loads()) with custom converter
    asjson,  #  json.dumps(cattrs.unstructure()) with custom converter
)


@frozen()
class Child:
    x: int = field(ge=0)
    y: int = field(ge=0)
    d: datetime


@frozen()
class Parent:
    name: str = field(re='^[a-c]+$', max_length=3)
    child: Child



DATA = {
    'name': 'abc',
    'child': {
        'x': 23,
        'y': '42',  # sic!
        'd': '2020-05-04T13:37:00',
    },
}


p1 = fromdict(DATA, Parent)
print(asdict(p1))

p_json = asjson(p1)
print(p_json)
p2 = fromjson(p_json, Parent)
assert p1 == p2

try:
    p3 = evolve(p2, name='spam')
except ValueError as e:
    print(e)  # ValueError: "name" exceeds the maximum length of 3: 4

The UX is okay, I think. I don’t know how much I (dis)like fromXYZ(data, cls) style. It feels a bit clunky, but looks quite functional and involves less magic than cls(**data). 🤷

Validation is done by attrs, conversion is done by cattrs. I still think that it would maybe be nice if we could teach attrs some basic auto-conversion rules based on the attribute type.

Performance for structuring + validation is a bit slower than just attrs+cattrs but I did not optimizie this yet. Dumping is as fast a attrs+cattrs.

Add an auto-converter, converters and validators

@hynek

@hynek would like my changes proposed in #701 to mature for a while in another project.

I’ll integrate them into typed-settings and will use them from there for my other projects.

@hynek What would be your definition of “the auto-converters and validators are mature enough to be added to attrs”? :)

That is an excellent question! I guess “you’ve built something useful and learned/fixed any rough edges”? I think I’ll try to get 20.3 out in the next two weeks so you can start tinkering.

Excelent! Can’t wait for this to happen. 😀

It's helpful for frontend developer to check params if an optional of type validator could raise some exception for the following code:

from attr import attrs

In [2]: @attrs(auto_attribs=True, kw_only=True)
   ...: class T:
   ...:     a: int
   ...:     b: str
   ...:

In [3]: t=T(a='aa', b=2)

In [4]: t
Out[4]: T(a='aa', b=2)

By now, "kw_only" can show missing key, type annotation is not involved in type checking. May be "type_validation" can be added to check wrong field type. I'm not digging so deep into type annotation runtime checking, and Pydantic can do some auto type validation as below:

In [1]: from pydantic import BaseModel

In [2]: class T(BaseModel):
    ...:     a: int
    ...:     b: str
    ...: 

In [3]: t=T(a='aaa', b=1)
---------------------------------------------------------------------------
ValidationError: 1 validation error for T
a
  value is not a valid integer (type=type_error.integer)

Something like this may be added to attrs in the future. A prototype for that feature currently evolves in Typed Settings.

Yea, please check it out and provide feedback, so we can get it right!

@Tinche

I am no longer convinced my proposal is the right approach.

Cattrs is a bit harder to use and you need a few lines more code, but it is a lot more mature, more flexible, and (i gues) faster.

@Tinche @hynek Do you think anything in these modules may still be woth to be added to attrs?

And what do you think of addign cattrs to the attrs organization? This might give it more visibility.

And what do you think of addign cattrs to the attrs organization? This might give it more visibility.

I talked to Hynek privately and we agreed on doing just this (but we're both on holiday atm, and we agreed to move cattrs away from Poetry first). So it's going to happen :)

We also discussed another, as of yet unannounced project to be added to the org ;)

I’ve updated Typed Setting's auto-convert hook to use a cattrs converter. This may for some users combine Pydantic's ease of use and (c)attrs flexibility and customizablitly. Do you think, this is a valid use case and maybe worth to be added to (c)attrs?

From Typed Settings:

def make_auto_converter(converter: cattr.Converter) -> "_FieldTransformer":
    """
    Creates and returns an auto-converter `field transformer`_.

    .. _field transformer: https://www.attrs.org/en/stable/extending.html
                        #automatic-field-transformation-and-modification

    Args:
        converters: A cattrs :class:`~cattrs.Converter` that can handle the
            types of all fields.

    Returns:
        A function that can be passed as *field_transformer* to
        :func:`attr.define()`.

    Example:

        .. code-block:: python

            >>> from datetime import datetime
            >>> from pathlib import Path
            >>>
            >>> import attr
            >>> import cattr
            >>>
            >>> converter = cattr.GenConverter()
            >>> converter.register_structure_hook(
            ...     datetime, lambda v, _t: datetime.fromisoformat(v)
            ... )
            >>> converter.register_structure_hook(Path, lambda v, t: t(v))
            >>>
            >>> auto_convert = make_auto_converter(converter)
            >>>
            >>> @attr.define(field_transformer=auto_convert)
            ... class C:
            ...     a: Path
            ...     b: datetime
            ...
            >>> inst = C(a="spam.md", b="2020-05-04")
            >>> inst
            C(a=PosixPath('spam.md'), b=datetime.datetime(2020, 5, 4, 0, 0))
            >>> inst.b = "2022-01-01"
            >>> inst
            C(a=PosixPath('spam.md'), b=datetime.datetime(2022, 1, 1, 0, 0))

    """

    def auto_convert(cls, attribs):
        """
        A field transformer that tries to convert all attribs of a class to
        their annotated type.
        """
        attr.resolve_types(cls, attribs=attribs)
        results = []
        for attrib in attribs:
            # Do not override explicitly defined converters!
            if attrib.converter is None:
                c = partial(converter.structure, cl=attrib.type)
                attrib = attrib.evolve(converter=c)
            results.append(attrib)

        return results

    return auto_convert

@hynek

@hynek How are you feeling about closing this issue? Or is this something you want?

Navigation Menu

Repository navigation

[RFC] Add optional automatic validation and conversion #649

Summary (what’s needed)

10 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[RFC] Add optional automatic validation and conversion #649

Description

Summary (what’s needed)

Activity

wsanchez commented on May 26, 2020

sscherfke commented on May 27, 2020

Tinche commented on May 27, 2020

hynek commented on May 27, 2020

sscherfke commented on May 27, 2020

fbergroth commented on May 27, 2020

Tinche commented on May 27, 2020

sscherfke commented on May 27, 2020

hynek commented on May 28, 2020

wsanchez commented on May 28, 2020

sscherfke commented on May 28, 2020

sscherfke commented on May 29, 2020

hynek commented on May 30, 2020

sscherfke commented on May 30, 2020

10 remaining items

sscherfke commented on Oct 18, 2020

hynek commented on Oct 19, 2020

sscherfke commented on Oct 19, 2020

Bryant-Yang commented on Dec 28, 2020

sscherfke commented on Dec 28, 2020

hynek commented on Dec 30, 2020

sscherfke commented on Sep 22, 2021

Tinche commented on Sep 22, 2021

sscherfke commented on Dec 31, 2021

sscherfke commented on Jul 9, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions