Repository navigation
feat(migrate): plan-based migrations: apso migrate asks, then writes the migration (TypeScript, Python, Go) - #164
Merged
Merged
Conversation
…ed checks rows survive TypeORM's migration:generate changes a column type by DROP + ADD, casts enums without remapping removed values, backfills NOT NULL columns with their default, and builds a renamed entity as a new empty table. migrate generate and verify now flag those statements, and verify --seed runs the migration against seeded rows and fails if rows or values go missing (following RENAME TO / RENAME COLUMN). Fixtures are TypeORM's real output for five changes.
…, per-column --accept-loss A dropped column whose values all turn up in a non-id column of a table the migration creates counts as moved, not lost (field to its own entity, foreign key to a join table). --check runs SELECTs returning ok/label after the migration, to prove transforms the data check can't follow (a split field). --accept-loss now names the table or column whose loss is intended. Review: DROPS_COLUMN names the tables the migration creates and suggests INSERT ... SELECT; a drop or SET NOT NULL after an UPDATE/INSERT that names the column is treated as handled.
…a changes Compares the deployed .apsorc with the current one. Each change whose intent the schema can't settle becomes a question with content-derived id and fixed options: - removed field: rename to a same-type new field, split into several, move to a field on a new related entity, or drop - removed entity: rename to an added entity with the same fields, or drop - removed enum value: which remaining value replaces it - required field on existing rows: backfill value or SQL - type change: plain cast or SQL - ManyToOne becoming ManyToMany or removed: copy or drop the links A rename is only suggested when it is the single reading. Answers live in .apso/migration-plan.json (or --answer id=option) and carry over when the plan is rebuilt. Exits 2 while questions are open, 1 on a bad answer. Field-rename detection follows fix/147-migrate-rename-detection (#147).
schemaDdl() writes the CREATE statements for every enum type, table, index and foreign key. Column rules follow the TypeScript entity templates (computed via getFieldForTemplate) and names follow TypeORM's DefaultNamingStrategy, so the database it builds matches what TypeORM builds for the generated entities and deployed services see no drift. readCatalog() reads a database back from Postgres's catalog (columns, constraints, indexes, enums) so two databases can be compared; it is the start of the language-neutral drift check. Tests: - project-parity (opt-in, APSO_PARITY_PROJECT=<scaffolded service>): scaffolds each fixture into a real service, runs TypeORM schema:sync on PGlite, and requires an identical catalog. 19/19 pass: the 8 migration eval cases before and after, the 9-entity inventory schema, the decimal example, and a coverage fixture. - ddl.test.ts (always): every fixture's DDL runs on PGlite, statements are snapshotted, and constraint names match TypeORM's. Unsupported field types (spatial, range) fail with a clear error.
The writer copied the old template, which turned a missing or "false" boolean default into true. #161 fixed the template; the writer now writes DEFAULT true only for a default of true or "true". Real-project parity still passes 19/19 against the fixed templates.
changeDdl(before, after, plan) writes the statements that move a database from the deployed .apsorc to the new one. - Renames: the answered renames are applied to the deployed schema and its model rebuilt; objects pair up by position, so every derived name that changes becomes a RENAME (table, column, id sequence, constraint, index, enum type). - The renamed model is diffed against the new one by identity. Statements run in a fixed order: drop old keys and indexes, create types and tables, add columns (nullable first when they need data), retype columns (enum swaps through <name>_old with the answered value mapping, casts with the answered SQL), move data (split UPDATEs, move INSERTs, copied many-to-many links, backfills), SET NOT NULL, drop columns/tables/types, then create keys, indexes and foreign keys. schemaDdl now builds a Model (tables, columns, constraints, indexes, enum types) and renders it; its output is unchanged (snapshots and real-project parity 19/19). Tests: each of the 8 migration eval cases is built with schemaDdl, seeded, migrated with the rendered change, and must match a fresh build of the new schema with every checks.sql row passing. Answering drop instead of rename fails the checks; a plain text-to-integer cast fails on 'n/a'.
…swered plan generate now reads .apso/migration-plan.json (re-planning against the current .apsorc), exits 2 listing any open questions, and otherwise writes src/migrations/<ts>-<Name>-migration.ts with up() from changeDdl and down() from downDdl, plus the plan saved beside it as <ts>-<Name>-migration.plan.json. It no longer builds the project or runs TypeORM. --typeorm keeps the old TypeORM migration:generate path, which a render error points to (for field types the DDL writer does not support yet). downDdl renders the reverse plan answered from the forward answers: renames invert, added fields drop, enum values map back, a split rejoins with concat_ws, and data moved to a new table or into many-to-many links is copied back. Anything it cannot answer makes down() throw, naming the question. Tests: up then down returns each eval case to the deployed schema, with the seeded rows unchanged except where the forward change is lossy by design (split whitespace, 'n/a' score). End to end through the real commands (plan --answer, generate, verify --seed --check) all 8 eval cases pass, including TypeORM's drift check on the built service.
…their own apso migrate plan needs no arguments. The baseline is, in order: - --baseline, when given - the before schema of the plan in progress (.apso/migration-plan.json) - the after schema saved with the newest planned migration - the .apsorc from the last commit that changed the migrations folder, for services whose migrations predate the plan and otherwise an error that asks for --baseline once. generate saves the before and after schemas in the plan beside each migration and clears the working plan once the migration is written, so the next change starts from what the last migration produced. verify defaults to the newest planned migration and the schema it starts from. plan prints JSON when its output is not a terminal, so an agent gets the open questions without --json. End to end on all 8 eval cases with argument-free commands (only the first plan on each service passes --baseline, since the eval projects are not git repos), plus a second change on rename-field that finds its baseline from the first migration.
…ify); old flow moves to apso migrate sandbox In a terminal it asks each open question; for an agent (or --json) it prints the open questions as JSON and exits 2, and the agent reruns with --answer. planWithAnswers moves into lib/plan so plan and migrate share it.
…t defaults
apso migrate and apso migrate plan print each open question with labels
("Renamed to mobilePhone (keeps the data)"), a SQL hint for sql options and a
starting rule per two-way split, so the editor and agents show the same words
as the terminal prompts. Lint errors in the new command fixed.
…ython quoting (#148) apso migrate and apso migrate generate now write the planned migration in the service's own wrapper: a TypeORM class (TypeScript), an Alembic revision in alembic/versions/ (Python, down_revision chained to the current head), or a goose SQL file in migrations/ (Go, YYYYMMDDHHMMSS_<name>.sql so it sorts with goose's own files). The plan JSON is saved beside it as before, and latestPlannedMigration finds it in each language's folder. Verify for Python and Go runs without TypeORM: Apso's DDL writer builds the deployed schema in PGlite, the seed loads, the migration's SQL runs as written, the data-loss and --check steps run as for TypeScript, and drift is a catalogDiff against a fresh build of the new .apsorc. down then runs and must give back the deployed schema. TypeScript verify is unchanged. migrateUnsupported is gone; --typeorm still needs a TypeScript service. literal) instead of """...""", which broke on SQL ending in a quoted identifier. The old sandbox generators get the same fix, and goose output fences any statement with its own ";" in StatementBegin/End. Tests: Alembic and goose snapshots for all 8 plan fixtures, py_compile of each Alembic file, and verify passing for python and go on rename-field, split-field and extract-entity with their seed and checks.
feat(migrate): Alembic and goose output for planned migrations; fix Python quoting (#148)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #147, #148, #155.
apso migratenow compares the deployed .apsorc with the current one and turns every change it can't read from the schema into a question in .apso/migration-plan.json:In a terminal it asks. For agents and the editor it prints the questions as JSON and exits 2; JSON questions carry option labels, SQL hints and split defaults. Once every question is answered it writes the migration from the answers:
The answered plan is saved beside the migration, and the migration is verified on PGlite with seeded rows, a data-loss check, drift, and down() on Python and Go. The same answers always give the same SQL. The old snapshot preview moves to
apso migrate sandbox.Also fixes #148 (Python SQL quoting, and the same class of bug in goose files).
Tests: 486 passed (19 opt-in skipped), 0 lint errors. End to end, all 8 migration eval cases (rename field and entity, enum replace, required field, split, extract entity, many-to-many, type change) pass on seeded rows from answered plans, and a wrong answer fails.
Not verified: goose itself never parsed the Go files (not installed), and nothing ran on real Postgres, only PGlite.