Skip to content

[spanner-sqlalchemy]: support spanner_storing in Alembic op.create_index() and autogenerate compare_indexes() #18835

Open

Description

@rocksnow1942

Determine this is the right repository

  • I determined this is the correct repository in which to report this bug.

Summary of the issue

Environment details

  • OS: Linux
  • Python version: 3.13
  • google-cloud-sqlalchemy-spanner version: 1.17.2 (code unchanged on main)
  • sqlalchemy version: 2.0.52
  • alembic version: 1.16.2

Problem description

Using spanner_storing on sa.Index with Alembic hits two issues:

1. op.create_index(..., spanner_storing=["col"]) raises KeyError on string column names

In SpannerDDLCompiler.visit_create_index (google/cloud/sqlalchemy_spanner/sqlalchemy_spanner.py, lines 712–720):

if "storing" in options:
    storing = options["storing"]
    storing_columns = [
        index.table.c[col] if isinstance(col, str) else col
        for col in storing
    ]
    text += " STORING (%s)" % ", ".join(
        [self.preparer.quote(c.name) for c in storing_columns]
    )

When op.create_index("ix_tbl_c1", "tbl", ["c1"], spanner_storing=["c2"]) runs inside an Alembic migration, alembic.operations.schemaobj.SchemaObjects.index constructs an ad-hoc sa.Table("tbl") containing only the indexed key columns (["c1"]). Looking up index.table.c["c2"] fails with KeyError: 'c2'.

This also breaks Alembic's autogenerated downgrade() when dropping or replacing a reflected index: SpannerDialect.get_multi_indexes populates dialect_options["spanner_storing"] as a list[str], so Alembic renders op.create_index(..., spanner_storing=['c2']), which crashes with KeyError: 'c2' when downgrade() is executed.

(Note: Upstream Alembic's PostgresqlImpl.create_index and MSSQLImpl.create_index work around the same pattern for postgresql_include / mssql_include by appending Column(col, sqltypes.NullType) to index.table when col not in index.table.c, or visit_create_index can extract the column name directly via col if isinstance(col, str) else col.name.)

2. Alembic --autogenerate ignores changes to spanner_storing

SpannerDialect.get_multi_indexes reflects spanner_storing from INFORMATION_SCHEMA.INDEX_COLUMNS, but google-cloud-sqlalchemy-spanner does not register an alembic.ddl.impl.DefaultImpl subclass (__dialect__ = "spanner+spanner") overriding compare_indexes. Because DefaultImpl.compare_indexes only checks unique and key column_names, adding, removing, or modifying spanner_storing on an existing sa.Index produces an empty migration in alembic revision --autogenerate.

(Note: On Spanner / Spanner Emulator, STORING columns have no semantic order and INFORMATION_SCHEMA.INDEX_COLUMNS may return them in table column order rather than DDL declaration order, so compare_indexes should compare storing column names as sets: set(meta_storing) != set(reflected_storing).)

(Bonus: SpannerDialect.get_multi_indexes also currently omits i.is_null_filtered and i.parent_table_name from INFORMATION_SCHEMA.INDEXES, so spanner_null_filtered and spanner_interleave_in are not populated in reflected dialect_options.)

Steps to reproduce (no database required)

import io
import alembic.operations
from alembic.ddl.impl import DefaultImpl
from alembic.migration import MigrationContext
from google.cloud.sqlalchemy_spanner.sqlalchemy_spanner import SpannerDialect
import sqlalchemy as sa


class SpannerDefaultImpl(DefaultImpl):
    __dialect__ = "spanner+spanner"


# 1. Reproduce KeyError in op.create_index with spanner_storing=["c2"]
buf = io.StringIO()
ctx = MigrationContext.configure(
    dialect=SpannerDialect(),
    opts={"as_sql": True, "output_buffer": buf},
)
op = alembic.operations.Operations(ctx)

try:
    op.create_index("ix_tbl_c1", "tbl", ["c1"], spanner_storing=["c2"])
except KeyError as e:
    print(f"Issue 1 reproduced: KeyError: {e!r}")

# 2. Reproduce compare_indexes ignoring spanner_storing changes
t1 = sa.Table("tbl", sa.MetaData(), sa.Column("c1", sa.Integer), sa.Column("c2", sa.String))
idx_meta = sa.Index("ix_tbl_c1", t1.c.c1, spanner_storing=["c2"])

t2 = sa.Table("tbl", sa.MetaData(), sa.Column("c1", sa.Integer), sa.Column("c2", sa.String))
idx_refl = sa.Index("ix_tbl_c1", t2.c.c1)

print("Issue 2 compare_indexes result:", ctx.impl.compare_indexes(idx_meta, idx_refl))

Actual output

Issue 1 reproduced: KeyError: 'c2'
Issue 2 compare_indexes result: ComparisonResult(status='equal', message=None)

Expected output

  1. op.create_index("ix_tbl_c1", "tbl", ["c1"], spanner_storing=["c2"]) compiles CREATE INDEX ix_tbl_c1 ON tbl (c1) STORING (c2) without raising KeyError.
  2. compare_indexes(idx_meta, idx_refl) returns ComparisonResult.Different(...) when spanner_storing columns differ.

API client name and version

No response

Reproduction steps: code

file: main.py

"""MCVE reproducing two issues in google-cloud-sqlalchemy-spanner and alembic:

Issue 1: SpannerDDLCompiler.visit_create_index fails with KeyError when op.create_index
         is called with spanner_storing=[<str>, ...] because Alembic creates an ad-hoc Table
         containing only index key columns, so index.table.c[col] fails.
         Location: google/cloud/sqlalchemy_spanner/sqlalchemy_spanner.py:1203

Issue 2: Alembic's DefaultImpl (used by SpannerDialect) ignores spanner_storing in
         compare_indexes, failing to detect when storing columns are added, removed, or changed.
         google-cloud-sqlalchemy-spanner currently does not provide an Alembic Impl subclass
         to compare dialect-specific index options.
"""

from __future__ import annotations

import io
import sys
import alembic.operations
from alembic.ddl.impl import DefaultImpl
from alembic.migration import MigrationContext
from google.cloud.sqlalchemy_spanner.sqlalchemy_spanner import SpannerDialect
import sqlalchemy as sa


# Alembic requires a DefaultImpl subclass registered for the dialect name.
# google-cloud-sqlalchemy-spanner does not provide one out of the box.
class SpannerDefaultImpl(DefaultImpl):
    __dialect__ = "spanner+spanner"


def reproduce_issue_1() -> None:
    """Demonstrate KeyError when op.create_index is called with spanner_storing."""
    print("=" * 70)
    print("Issue 1: op.create_index(..., spanner_storing=['c2']) raises KeyError")
    print("=" * 70)

    buf = io.StringIO()
    ctx = MigrationContext.configure(
        dialect=SpannerDialect(),
        opts={"as_sql": True, "output_buffer": buf},
    )
    op = alembic.operations.Operations(ctx)

    try:
        # In an Alembic migration downgrade() or upgrade(), op.create_index creates an Index
        # on an ad-hoc Table containing only the indexed column ['c1']. Column 'c2' is not in table.c.
        op.create_index("ix_tbl_c1", "tbl", ["c1"], spanner_storing=["c2"])
        print("UNEXPECTED: op.create_index succeeded!")
    except KeyError as e:
        print(f"REPRODUCED: KeyError: {e!r}")
        print("Root cause:")
        print("  In google.cloud.sqlalchemy_spanner.SpannerDDLCompiler.visit_create_index:")
        print("    storing_columns = [")
        print("        index.table.c[col] if isinstance(col, str) else col")
        print("        for col in storing")
        print("    ]")
        print("  index.table.c['c2'] fails because 'c2' is only in STORING, not an indexed column.")


def reproduce_issue_2() -> None:
    """Demonstrate compare_indexes ignoring spanner_storing changes."""
    print("\n" + "=" * 70)
    print("Issue 2: DefaultImpl.compare_indexes ignores spanner_storing changes")
    print("=" * 70)

    dialect = SpannerDialect()
    buf = io.StringIO()
    ctx = MigrationContext.configure(
        dialect=dialect,
        opts={"as_sql": True, "output_buffer": buf},
    )
    impl = ctx.impl

    # Metadata index has storing=['c2']
    t1 = sa.Table("tbl", sa.MetaData(), sa.Column("c1", sa.Integer), sa.Column("c2", sa.String))
    idx_meta = sa.Index("ix_tbl_c1", t1.c.c1, spanner_storing=["c2"])

    # Reflected database index has NO storing columns
    t2 = sa.Table("tbl", sa.MetaData(), sa.Column("c1", sa.Integer), sa.Column("c2", sa.String))
    idx_refl = sa.Index("ix_tbl_c1", t2.c.c1)

    result = impl.compare_indexes(idx_meta, idx_refl)
    print(f"Result of compare_indexes: {result!r}")
    if getattr(result, "status", None) == "equal" or result == 0:
        print("REPRODUCED: Indexes are reported EQUAL despite different spanner_storing!")
        print("Root cause:")
        print("  DefaultImpl.compare_indexes only compares unique constraint and indexed columns.")
        print("  Spanner-specific storing columns require an override in a SpannerImpl class.")


if __name__ == "__main__":
    reproduce_issue_1()
    reproduce_issue_2()

Reproduction steps: supporting files

file: mydata.csv

alpha,1,3
beta,2,5

Reproduction steps: actual results

file: output.txtmydata.csv

Calculated: foo

Reproduction steps: expected results

file: output.txtmydata.csv

Calculated: bar

OS & version + platform

No response

Python environment

No response

Python dependencies

No response

Additional context

No response

Activity

  1. added
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    triage meI really want to be triaged.
    on Oct 23, 2026
  2. added
    priority: p2Moderately-important priority. Fix may not be included in next release.
    api: spannerIssues related to the Spanner API.
    and removed
    triage meI really want to be triaged.
    on Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    api: spannerIssues related to the Spanner API.priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions