Datastore Query with Key filter failing on GCE

Since I upgraded to 0.11.0, I'm seeing a problem with my production environment on GCE, but not when I run it locally.

I am trying to query the datastore, using a filter on a property that is a Key. I have example code below - the basic problem is that when using the filter, I get no results back, even when querying using a key I know exists.

Looking at the print statements, the difference between my local and GCE environments appears to be that the key I create myself has project=project_id whereas the query returns a key with project=s~project_id.

Environment: GCE, Python 2.7.9, gcloud 0.11.0 [following code asserts with no results returned from 2nd query]
Local: Mac OS X, Python 2.7.10, gcloud 0.11.0 [following code works]

from gcloud import datastore

def main():
    client = datastore.Client()

    last_result = None
    query = client.query(kind='Foo')
    results = query.fetch(1)
    for result in results:
        last_result = result
    assert last_result is not None

    query2 = client.query(kind='Foo')
    key = client.key('Bar', last_result['bar'].id)
    print key
    print last_result['bar']
    query2.add_filter('bar', '=', key)
    results = query2.fetch(1)
    assert len(list(results)) > 0

if __name__ == '__main__':
    main()

Thanks for the report! I'm in an airport but will try to dig around

I can't reproduce. I just ran this on GCE and it passed:

daniel_j_hermes@instance-1:~$ python test.py
<Key[{'kind': 'Bar', 'id': 1234L}], project=PROJ-NAME>
<Key[{'kind': u'Bar', 'id': 1234L}], project=PROJ-NAME>
daniel_j_hermes@instance-1:~$ pip show gcloud
---
Name: gcloud
Version: 0.11.0
Location: /usr/local/lib/python2.7/dist-packages
Requires: googleapis-common-protos, oauth2client, pyOpenSSL, httplib2, six, protobuf

Maybe you're having issues with eventual consistency?

Sorry for the delay in replying - I don't think it's a consistency issue - the data I'm reading hasn't changed recently. I'm getting a repro 100% of the time here.
This is an old project in App Engine terms - over a couple of years - there seems to be some inconsistency between the project id (e.g. "PROJ-NAME") and the datastore id (e.g. "s~PROJ-NAME") and which one to use when running on GCE.

@pcostell

@pcostell Any ideas?

This likely is a key normalization issue (which gets fixed substantially with the next version of the API).

In Datastore v1beta2, we only normalize the datasetId if you leave it blank. I can't remember how much gcloud-python does in order to try to generate the "correct" dataset -- does client.Key(...) have a dataset == "s~PROJ" or dataset == "PROJ"?

Can you try writing your query without using gcloud-python's key? Instead of using the client, create a key directly:

key = datastore.Key('Bar', last_result['bar'].id, project=None)

@pcostell

@pcostell gcloud-python doesn't make any effort to get it correct, but wipes the dataset from a key before sending in a request.

Ah ok. But you don't wipe the dataset when someone uses a key as a query argument unless they query by __key__. So in this case, the query is for 'my-project-id', but the actual data is stored in Datastore as 's~my-project-id'.

Note though that you can't necessarily wipe the project in this case, as someone could also query for 'my-other-project-id'.

I'm not sure this is worth fixing now -- in v1beta3 you should always populate the project id and should never have to deal with dataset ids.

Gotcher. Let me see if I can repro.

Just to add, I made a temp fix by, when creating the Datastore client, using datastore.Client(project="s~project_id") as the constructor instead of leaving it blank.

@ndenny

@ndenny That is the best move for now. You could also use find_true_project but that all is moot in v1beta3.

@pcostell I was able to reproduce and it was definitely the project in the key:

>>> from gcloud import datastore
>>> client = datastore.Client(project='explicit-project-name')
>>> # Make the keys.
>>> k1 = client.key('Foo', 1234)
>>> k1a = k1._clone()
>>> k1a._project = 's~' + k1a._project
>>> k2 = client.key('Bar', 1234)
>>> # Store the data to be queried.
>>> e = datastore.Entity(key=k2)
>>> e['foo'] = k1
>>> client.put(e)
>>> # Query without s~.
>>> q = client.query(kind='Bar')
>>> q.add_filter('foo', '=', k1)
>>> q_iter = q.fetch()
>>> entities, more_results, cursor = q_iter.next_page()
>>> print(len(entities))
1
>>> # Query with s~.
>>> q = client.query(kind='Bar')
>>> q.add_filter('foo', '=', k1a)
>>> q_iter = q.fetch()
>>> entities, more_results, cursor = q_iter.next_page()
>>> print(len(entities))
0

@ndenny

I don't think we normalize projectId => applicationId in v1beta2. So when
you store e['foo'] = k1, we'll store the key with the project Id instead
of the application Id. If you had stored this in GAE or left the
datasetId/projectId blank, we would store it with the application Id (s~...)

On Mon, Mar 28, 2016 at 11:46 AM Danny Hermes notifications@github.com
wrote:

@ndenny https://github.com/ndenny That is the best move for now. You
could also use find_true_project
https://github.com/GoogleCloudPlatform/gcloud-python/blob/4ccb7a10829075192f844479b315df612136e05c/gcloud/datastore/helpers.py#L33
but that all is moot in v1beta3.

@pcostell https://github.com/pcostell I was able to reproduce and it
was definitely the project in the key:

from gcloud import datastore>>> client = datastore.Client(project='explicit-project-name')>>> # Make the keys.>>> k1 = client.key('Foo', 1234)>>> k1a = k1._clone()>>> k1a._project = 's~' + k1a._project>>> k2 = client.key('Bar', 1234)>>> # Store the data to be queried.>>> e = datastore.Entity(key=k2)>>> e['foo'] = k1>>> client.put(e)>>> # Query without s~.>>> q = client.query(kind='Bar')>>> q.add_filter('foo', '=', k1)>>> q_iter = q.fetch()>>> entities, more_results, cursor = q_iter.next_page()>>> print(len(entities))1>>> # Query with s~.>>> q = client.query(kind='Bar')>>> q.add_filter('foo', '=', k1a)>>> q_iter = q.fetch()>>> entities, more_results, cursor = q_iter.next_page()>>> print(len(entities))0

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#1639 (comment)

That makes sense - I realize the key bit of information here is that the data I'm querying was created in GAE.

@pcostell

@pcostell I'm closing since v1beta3 will fix this eventually and the amount of code required to fix would be quite a lot (i.e. emulating an OR query every time a key is used).

Video Intelligence region tag update [(#1639)](GoogleCloudPlatform/py…

docs: add samples from video/cloud-client/labels (#49)

Video Intelligence region tag update [(#1639)](GoogleCloudPlatform/py…

docs: add samples from video/cloud-client/shotchange (#72)

feat!: use microgenerator (#76)

chore: setup system tests for zonal buckets in cloud build (#1639)

fix: DataFrame accessors is not pupulated (#1639)

Navigation Menu

Repository navigation

Datastore Query with Key filter failing on GCE #1639

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Datastore Query with Key filter failing on GCE #1639

Description

Activity

dhermes commented on Mar 18, 2016

dhermes commented on Mar 22, 2016

ndenny commented on Mar 27, 2016

dhermes commented on Mar 28, 2016

pcostell commented on Mar 28, 2016

dhermes commented on Mar 28, 2016

pcostell commented on Mar 28, 2016

dhermes commented on Mar 28, 2016

ndenny commented on Mar 28, 2016

dhermes commented on Mar 28, 2016

pcostell commented on Mar 28, 2016

ndenny commented on Mar 28, 2016

dhermes commented on Mar 29, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions