Storing embedded entity, it's ignoring exclude_from_indexes from embedded

I'm trying to have a embedded entity with a field that is larger than 1500bytes. I add that field to the exclude_from_indexes key of the embedded entity. When I try and save the parent Entity it tells me the field is bigger than 1500bytes. If I save the embedded entity independently, it works.

Is exclude_from_indexes ignored on embedded entity?

client = datastore.Client(dataset_id=projectID)
record_key = client.key('Record', my_id)
record_entity = datastore.Entity(record_key)

embedded_key = client.key('Data', another_id)
embedded_entity = datastore.Entity(key=embedded_key,exclude_from_indexes=('big_field',))
embedded_entity['field1']='1234'
embedded_entity['big_field']='large string bigger than 1500bytes'

record_entity['RandomFieldName']=embedded_entity

client.put(record_entity)
#Error: gcloud.exceptions.BadRequest: 400 The value of property &quot;big_field&quot; is longer than 1500 bytes.

client.put(embedded_entity)
#No Error

@pcostell

@pcostell What's the expected behavior here?

This should succeed. I am able to get this to work successfully directly from the API (in a commit):

{
 "mode": "NON_TRANSACTIONAL",
 "mutation": {
  "insertAutoId": [
   {
    "properties": {
     "embedded": {
      "entityValue": {
       "properties": {
        "long_prop": {
         "stringValue": "a string longer than 1500 bytes", # I just used 'a'*1501
         "indexed": false
        }
       }
      }
     }
    },
    "key": {
     "path": [
      {
       "kind": "ModelWithEmbedded"
      }
     ]
    }
   }
  ]
 }
}

Gives me a 200.

It looks like information about exclude_from_indexes is not propagated in embedded entities: https://github.com/GoogleCloudPlatform/gcloud-python/blob/c8d231ba7462e2f35d689b526a70cef4712c565a/gcloud/datastore/batch.py#L282

@pcostell

Thanks @pcostell!

@pcostell

@pcostell thanks for debugging this.

Properly handling entity meanings in datastore.

@igama

@igama This has been fixed:

>>> from gcloud import datastore
>>> from gcloud.environment_vars import TESTS_DATASET
>>> from gcloud.datastore import client
>>> client.DATASET = TESTS_DATASET
>>> CLIENT = datastore.Client()
>>> record_key = CLIENT.key('Record', 1234)
>>> record_entity = datastore.Entity(record_key)
>>> embedded_key = CLIENT.key('Data', 5678)
>>> embedded_entity = datastore.Entity(key=embedded_key, exclude_from_indexes=('big_field',))
>>> embedded_entity['field1'] = '1234'
>>> embedded_entity['big_field'] = 'large string bigger than 1500bytes ' * 50
>>> len(embedded_entity['big_field'])
1750
>>> record_entity['RandomFieldName'] = embedded_entity
>>> CLIENT.put(record_entity)
>>> CLIENT.put(embedded_entity)
>>> 
>>> CLIENT.delete(record_entity.key)
>>> CLIENT.delete(embedded_entity.key)

Note -- This should probably use . to qualify the name (it matches a proposed field mask design):

exclude_from_indexes=('RandomFieldName.big_field')

Otherwise, there is no way to exclude nested field vs non-nested fields.

@pcostell

@pcostell I'm not sure what you mean RE: "there is no way" --- see the above. The way is to exclude them on the nested Entity and then we propagate that into the protobuf created. Maybe you mean it would be useful to differentiate between properties in queries?

As for the name used, see the reported snippet at the top of this issue.

Ah sorry definitely read recorded_entity instead of embedded_entity at embedded creation. Please ignore my comment :-)

No worries. Thanks for taking the time to read the notification

@dhermes

@dhermes thank you, Will try it in the next days.

feat: add basic interceptor to client (#1206)

fix: fix docstring code block formatting (#1206)

chore: Update CODEOWNERS file to enforce AION SDK team reviews on sha…

feat: add basic interceptor to client (#1206)

chore: librarian release pull request: 20251216T130719Z (#1251)

chore(deps): update dependency google-auth to v2.6.3 (#1206)

chore(deps): update all dependencies (#1206)

docs: migrate code from googleapis/python-bigquery (#18303)

Navigation Menu

Repository navigation

Storing embedded entity, it's ignoring exclude_from_indexes from embedded #1206

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Storing embedded entity, it's ignoring exclude_from_indexes from embedded #1206

Description

Activity

dhermes commented on Oct 28, 2015

pcostell commented on Oct 28, 2015

pcostell commented on Oct 28, 2015

dhermes commented on Oct 28, 2015

igama commented on Oct 29, 2015

dhermes commented on Dec 21, 2015

pcostell commented on Dec 21, 2015

dhermes commented on Dec 21, 2015

pcostell commented on Dec 21, 2015

dhermes commented on Dec 21, 2015

igama commented on Dec 30, 2015

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions