Port asg017's rescore index

ARCHITECTURE.md

@@ -28,6 +28,22 @@ Very much a WIP.
 
 #### `xyz_vector_chunksNN`
 
+- `rowid INTEGER`
+- `vectors BLOB`
+
+#### `xyz_rescore_chunksNN`
+
+The vectors of a column indexed by `rescore()`, quantized, in the layout of
+`xyz_vector_chunksNN`.
+
+- `rowid INTEGER`
+- `vectors BLOB`
+
+#### `xyz_rescore_vectorsNN`
+
+The `float32` vectors of a column indexed by `rescore()`, one row per row of
+the table.
+
 - `rowid INTEGER`
 - `vector BLOB`
 
@@ -52,7 +68,10 @@ The supported way to tell which tables belong to a `vec0` table is by name. A
 `vec0` table `xyz` owns these tables and no others (`NN` is two digits):
 
 - `xyz_info`, `xyz_chunks`, `xyz_rowids`: always
-- `xyz_vector_chunksNN`: one per vector column
+- `xyz_vector_chunksNN`: one per vector column, except those indexed by
+  `rescore()`
+- `xyz_rescore_chunksNN`, `xyz_rescore_vectorsNN`: one each per vector column
+  indexed by `rescore()`
 - `xyz_auxiliary`: only if there are auxiliary columns
 - `xyz_metadatachunksNN`: one per metadata column
 - `xyz_metadatatextNN`: one per text metadata column
@@ -69,6 +88,8 @@ join sqlite_master as s
  and substr(s.name, 1, length(v.name) + 1) = v.name || '_'
  and (substr(s.name, length(v.name) + 2) in ('rowids', 'chunks', 'info', 'auxiliary')
    or substr(s.name, length(v.name) + 2) glob 'vector_chunks[0-9][0-9]'
+   or substr(s.name, length(v.name) + 2) glob 'rescore_chunks[0-9][0-9]'
+   or substr(s.name, length(v.name) + 2) glob 'rescore_vectors[0-9][0-9]'
    or substr(s.name, length(v.name) + 2) glob 'metadatachunks[0-9][0-9]'
    or substr(s.name, length(v.name) + 2) glob 'metadatatext[0-9][0-9]')
 where v.type = 'table' and v.rootpage = 0
@@ -84,24 +105,25 @@ relied on:
 - Up to SQLite 3.53, so does a connection that parsed the schema before loading
   the extension.
 - Before SQLite 3.37 there is no `PRAGMA table_list` and no prefix marking (see
-  below), so the vector chunk tables are never shadow tables.
+  below), so the tables whose suffix has an underscore (`vector_chunksNN`,
+  `rescore_chunksNN`, `rescore_vectorsNN`) are never shadow tables.
 - SQLite finds the virtual table that owns a newly created or newly parsed
   table by splitting its name at the **last** underscore
   ([documented](https://www.sqlite.org/vtab.html#the_xshadowname_method)). For
   `xyz_vector_chunks00` it looks for a virtual table called `xyz_vector`, so
   `xShadowName` is never asked about it on that path.
-- `xShadowName` does report `vector_chunksNN`, and SQLite also marks shadow
+- `xShadowName` does report those suffixes, and SQLite also marks shadow
   tables by name prefix when it parses the virtual table's own schema entry.
   That only sees tables parsed earlier, which in practice means after a
   `VACUUM` (it rewrites `sqlite_master` with virtual tables last). A `vec0`
   table created after that `VACUUM` is not covered until the next one.
-- SQLite 3.54 additionally marks by prefix after every `xConnect`. There the
-  vector chunk tables become `shadow` once the `vec0` table has been used in
-  the connection, except in the connection that created it. `PRAGMA
-  table_list` connects every virtual table first, so it always reports them as
-  `shadow`, but under `SQLITE_DBCONFIG_DEFENSIVE` a direct write to a chunk
-  table is still accepted until the `vec0` table is first touched. This prefix
-  matching is not part of the documented contract.
+- SQLite 3.54 additionally marks by prefix after every `xConnect`. There those
+  tables become `shadow` once the `vec0` table has been used in the
+  connection, except in the connection that created it. `PRAGMA table_list`
+  connects every virtual table first, so it always reports them as `shadow`,
+  but under `SQLITE_DBCONFIG_DEFENSIVE` a direct write to one of them is still
+  accepted until the `vec0` table is first touched. This prefix matching is not
+  part of the documented contract.
 
 Rules that follow from this:
 

CHANGELOG.md

@@ -2,6 +2,11 @@
 
 ## [Unreleased]
 
+### Added
+
+- A vector column definition accepts `indexed by flat()`, as in `embedding float[768] indexed by flat()`. `flat()` is the exhaustive search that a column without the clause gets. Ported from [asg017@0de765f](https://github.com/asg017/sqlite-vec/commit/0de765f) ([asg017#273](https://github.com/asg017/sqlite-vec/pull/273)).
+- A `float` vector column can be declared `indexed by rescore(quantizer=bit)` or `indexed by rescore(quantizer=int8)`, with an optional `oversample=N` from 1 to 128 (8 by default). It also keeps its vectors quantized, in the new `_rescore_chunksNN` shadow table, and its `float` vectors by rowid in `_rescore_vectorsNN`. A KNN query scans the quantized vectors for the `k * oversample` nearest rows, then ranks those by the exact distances of their `float` vectors, to which distance constraints apply. The `oversample=N` command, as in `insert into t(t) values ('oversample=16')`, sets the oversample of a connection's queries. Rowid filters, MMR, auxiliary columns, `UPDATE`, `optimize`, and `ALTER TABLE ... RENAME` work on rescore columns; a table with one cannot have metadata or partition key columns. Ported from [asg017#276](https://github.com/asg017/sqlite-vec/pull/276) ([asg017@ba0db0b](https://github.com/asg017/sqlite-vec/commit/ba0db0b)) with its fixes [asg017@5e4c557](https://github.com/asg017/sqlite-vec/commit/5e4c557), [@82f4eb0](https://github.com/asg017/sqlite-vec/commit/82f4eb0), [@5522e86](https://github.com/asg017/sqlite-vec/commit/5522e86), [@b7fc459](https://github.com/asg017/sqlite-vec/commit/b7fc459), [@6e2c4c6](https://github.com/asg017/sqlite-vec/commit/6e2c4c6), the int8 quantizer of [@575371d](https://github.com/asg017/sqlite-vec/commit/575371d), and the distance constraints of the unmerged [asg017#309](https://github.com/asg017/sqlite-vec/pull/309). Unlike upstream, a point query for a rowid that a rescore table does not have returns no row instead of failing with `Could not fetch vector data`.
+
 ### Changed
 
 - Reading a vector column without a KNN query, as in `select embedding from v`, reads each chunk's vectors through one blob handle instead of opening the chunk's blob again for every row: selecting every vector of a 10,000-row `float[768]` table took 25 ms instead of 2,080 ms, and `select vec_distance_cosine(embedding, ?) from v` 41 ms instead of 2,093 ms.
@@ -10,6 +15,7 @@
 - A KNN query picks each chunk's k nearest rows with the heap of upstream's `SQLITE_VEC_EXPERIMENTAL_MIN_IDX` build option ([asg017/sqlite-vec@0de765f](https://github.com/asg017/sqlite-vec/commit/0de765f4570c1d65a0699e8e38d6c5562da9e23e)) instead of one pass over the chunk for every row it keeps, ordered so that it returns the same rows in the same order as before: on a 10,000-row `float[768]` table, `k = 1000` and `k = 4096` take 9.3 ms instead of 17.8 and 18.0 ms, `k = 100` 8.8 ms instead of 9.2, and `k = 1` 8.7 ms either way. A NaN distance ranks after every number instead of where its position put it. A static blob can produce one, and so can a `vec0` table written before 2.1.0, which started rejecting NaN elements. On such a table a NaN row used to push nearer rows out of the result or behind farther ones; queries now return the nearest rows in order, which can leave out a NaN row that a query with `k` smaller than the row count used to return.
 - A `DELETE` that removes the last row of a chunk now deletes the chunk's rows from the `_chunks`, `_vector_chunksNN`, and `_metadatachunksNN` shadow tables, so their pages are free for reuse without `optimize`; the emptied chunk used to stay until `optimize` removed it. Ported from [asg017@cb147c8](https://github.com/asg017/sqlite-vec/commit/cb147c8) via [vlasky@3a64182](https://github.com/vlasky/sqlite-vec/commit/3a64182) ([asg017#268](https://github.com/asg017/sqlite-vec/pull/268)).
 - A `DELETE` of a row whose bit in its chunk's validity bitmap is clear now fails with `vec0 deletion error: validity bit is not set`; it used to delete the row when a later row in the same byte of the bitmap was set. Only a damaged `_chunks` table has such a row.
+- A command that a `vec0` table does not know, as in `insert into t(t) values ('compact')`, now fails with `unknown vec0 command: 'compact'` instead of `SQL logic error`, as upstream [asg017@6e2c4c6](https://github.com/asg017/sqlite-vec/commit/6e2c4c6) does.
 
 ### Fixed
 

README.md

@@ -295,8 +295,9 @@ VACUUM;
 `VACUUM` should not corrupt vec tables; a checkpoint first is recommended when
 using WAL so the rewrite starts from a clean state.
 
-`optimize` runs through a hidden column named after the table, so a `vec0`
-table cannot have another column with its name. `CREATE` rejects tables such as
+`optimize`, like the `oversample=N` command of a rescore index, runs through a
+hidden column named after the table, so a `vec0` table cannot have another
+column with its name. `CREATE` rejects tables such as
 `CREATE VIRTUAL TABLE emb USING vec0(emb float[4])`, and tables named
 `distance`, `k`, `mmr_lambda`, or (without a primary key column) `rowid`, which
 are names of columns `vec0` always declares. Older releases allowed these
@@ -308,6 +309,73 @@ ALTER TABLE emb RENAME TO emb_vectors;
 INSERT INTO emb_vectors(emb_vectors) VALUES('optimize');
 ```
 
+### Rescore Index
+
+A `float` vector column declared `indexed by rescore(quantizer=bit)` or
+`indexed by rescore(quantizer=int8)` also keeps each vector quantized, to 1 bit
+or 8 bits per dimension. A KNN query scans the quantized vectors for the
+`k * oversample` nearest rows, then ranks those candidates by the distances of
+their `float` vectors. The distances it returns are exact, and its rows are
+those of an exact search whenever the nearest rows are among the candidates:
+
+```sql
+create virtual table vec_rescore using vec0(
+  sample_embedding float[8] indexed by rescore(quantizer=bit)
+);
+insert into vec_rescore(rowid, sample_embedding)
+  select rowid, sample_embedding from vec_examples;
+
+select rowid, distance
+from vec_rescore
+where sample_embedding match '[0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]'
+  and k = 3;
+/*
+┌───────┬──────────────────┐
+│ rowid │     distance     │
+├───────┼──────────────────┤
+│ 5     │ 1.16368770599365 │
+│ 13    │ 1.75137972831726 │
+│ 11    │ 1.83941268920898 │
+└───────┴──────────────────┘
+*/
+```
+
+`oversample` is 8 unless the column sets it, as in
+`indexed by rescore(quantizer=bit, oversample=16)`, from 1 to 128. The
+`oversample=N` command sets it for every rescore column, for the queries of the
+connection that runs it. With an oversample of 1, the query above keeps only the
+3 rows nearest by bit distance, and row 11 is not among them:
+
+```sql
+insert into vec_rescore(vec_rescore) values ('oversample=1');
+
+select rowid, distance
+from vec_rescore
+where sample_embedding match '[0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]'
+  and k = 3;
+/*
+┌───────┬──────────────────┐
+│ rowid │     distance     │
+├───────┼──────────────────┤
+│ 5     │ 1.16368770599365 │
+│ 13    │ 1.75137972831726 │
+│ 16    │ 2.36842274665833 │
+└───────┴──────────────────┘
+*/
+```
+
+- `quantizer=bit` sets one bit for each value >= 0, and needs a number of
+  dimensions divisible by 8. `quantizer=int8` maps [-1, 1] onto the 256 values
+  of an int8, so it suits vectors whose values lie in that range, such as
+  normalized embeddings.
+- Distance constraints apply to the exact distances of the candidates, so a
+  query with `distance > x`, as in cursor-based pagination, returns fewer than
+  `k` rows when most of the candidates are nearer than `x`.
+- A table with a rescore column cannot have metadata or partition key columns.
+  Auxiliary columns work.
+- `indexed by flat()` declares the exhaustive search that a column without the
+  clause gets.
+
 ## Sponsors
 
 > [!NOTE]