Repository navigation
Conversation
Why: the Cloud Storage service cannot attribute usage of purely
client-side features (such as the multi-range downloader) because they
are not visible on the wire. This change lets the Python SDK report
which client-side features are in use on each request so that the
service can account for them.
What:
- Add `google/cloud/storage/_tracked_features.py`, the single source of
truth for feature bit assignments (`TrackedFeature` IntEnum). Each
member is on its own `NAME = <bit>` line so tooling can parse it.
Bit 0 is `MULTI_RANGE_DOWNLOADER`; bit 1 is reserved for parallel
composite upload (PCU).
- Add `google/cloud/storage/_feature_tracking.py` with
`FEATURES_HEADER`, `feature_mask()`, `encode_feature_mask()` and
`features_metadata()` helpers.
- Attach the header as gRPC metadata on every BidiReadObject stream
opened by `AsyncMultiRangeDownloader` (initial open, re-open on
retry/redirect, and the stream factory used by the read retry
manager).
Encoding contract:
- Header / metadata key: `x-goog-storage-python-features` (lowercase).
- Value: 4-byte big-endian uint32 bitmask, URL-safe base64 with padding
(`base64.urlsafe_b64encode(struct.pack(">I", mask))`). Mask 1 encodes
as `AAAAAQ==`. The header is omitted entirely when the mask is 0.
The service rejects the standard `+`/`/` alphabet.
This mirrors `x-goog-storage-go-features` in google-cloud-go
(googleapis/google-cloud-go#20684). Note that, unlike Go where bit 0
means "multistream in MRD", the Python `AsyncMultiRangeDownloader`
currently opens a single stream per object, so bit 0 means "MRD used";
it will come to mean multistream usage once that is implemented.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces client-side feature tracking for the Google Cloud Storage Python SDK by encoding a 32-bit feature mask into a URL-safe base64 string and attaching it as the x-goog-storage-python-features header. Specifically, the AsyncMultiRangeDownloader is updated to include this header to track its usage on BidiReadObject streams, accompanied by comprehensive unit tests. The review feedback suggests a performance optimization to pre-compute the static feature-tracking metadata as a module-level constant (_MRD_FEATURES_METADATA) instead of re-evaluating and encoding it dynamically on every stream open or re-open.
| def _mrd_features_metadata() -> List[Tuple[str, str]]: | ||
| """Return the feature-tracking metadata attached to every MRD stream. | ||
|
|
||
| ``AsyncMultiRangeDownloader`` currently opens a single BidiReadObject | ||
| stream per object, so this bit indicates that the multi-range downloader | ||
| was used (rather than multistream usage as in other SDKs). | ||
| """ | ||
| return features_metadata(TrackedFeature.MULTI_RANGE_DOWNLOADER) |
There was a problem hiding this comment.
Since the feature-tracking metadata for the multi-range downloader is static and does not change at runtime, we can pre-compute it as a module-level constant instead of re-evaluating the feature mask, packing, and base64 encoding on every stream open/re-open. This avoids unnecessary CPU overhead during stream initialization.
# Feature-tracking metadata attached to every MRD stream.
# AsyncMultiRangeDownloader currently opens a single BidiReadObject
# stream per object, so this bit indicates that the multi-range downloader
# was used (rather than multistream usage as in other SDKs).
_MRD_FEATURES_METADATA = features_metadata(TrackedFeature.MULTI_RANGE_DOWNLOADER)|
|
||
| async def _do_open(): | ||
| current_metadata = list(metadata) if metadata else [] | ||
| current_metadata.extend(_mrd_features_metadata()) |
There was a problem hiding this comment.
Use the pre-computed _MRD_FEATURES_METADATA constant instead of calling the helper function.
| current_metadata.extend(_mrd_features_metadata()) | |
| current_metadata.extend(_MRD_FEATURES_METADATA) |
| ) | ||
|
|
||
| current_metadata = list(metadata) if metadata else [] | ||
| current_metadata.extend(_mrd_features_metadata()) |
There was a problem hiding this comment.
Use the pre-computed _MRD_FEATURES_METADATA constant instead of calling the helper function.
| current_metadata.extend(_mrd_features_metadata()) | |
| current_metadata.extend(_MRD_FEATURES_METADATA) |
|
Closing: Python MRD is single-stream today; MRD usage is already server-visible. Will revisit when multistream lands. |
Summary
Adds a client feature-tracking header to the Python Cloud Storage SDK so the service can attribute usage of purely client-side features that are otherwise invisible on the wire.
This is the Python counterpart of
x-goog-storage-go-featuresin google-cloud-go (googleapis/google-cloud-go#20684).Changes
google/cloud/storage/_tracked_features.py(new) – single source of truth for feature bit assignments:TrackedFeature(enum.IntEnum). Each member is on its ownNAME = <bit>line so tooling can parse it.MULTI_RANGE_DOWNLOADER = 0google/cloud/storage/_feature_tracking.py(new) –FEATURES_HEADER,feature_mask(),encode_feature_mask(),features_metadata().asyncio/async_multi_range_downloader.py– attaches the header as gRPC metadata on everyBidiReadObjectstream opened byAsyncMultiRangeDownloader: initialopen(), re-open on retry/redirect, and the stream factory used by the read retry manager.Encoding contract
x-goog-storage-python-features(lowercase).uint32bitmask → URL-safe base64 with padding (base64.urlsafe_b64encode(struct.pack(">I", mask))). E.g. mask1→AAAAAQ==,0xF8→AAAA-A==.0.+//alphabet, hence URL-safe.Note on bit 0 semantics
In Go, bit 0 means "multistream used in MRD". Python's
AsyncMultiRangeDownloadercurrently opens a single stream per object, so here bit 0 means "MRD used". It will come to mean multistream usage once that is implemented.Tests
tests/unit/test__tracked_features.py(new): exact encodings (1 → AAAAAQ==,4 → AAAABA==,0xF8 → AAAA-A==), URL-safety, omission on mask 0, enum uniqueness/range, and that the registry file matches the tooling regex^\s*([A-Z0-9_]+)\s*=\s*(\d+).tests/unit/asyncio/test_async_multi_range_downloader.py: asserts("x-goog-storage-python-features", "AAAAAQ==")is passed to_AsyncReadObjectStream.open()on initial open, alongside user metadata, on redirect re-open (together with the routing token), and on the replacement stream opened by the read retry path.tests/unit/asyncio/test_async_read_object_stream.py: asserts the metadata is forwarded verbatim toAsyncBidiRpcnext to the mergedx-goog-request-params.pytest tests/unit/test__tracked_features.py tests/unit/asyncio→ 256 passed.blackandflake8clean on touched files.