TinyStore 0.5.1

TinyStore 0.5.1 came out on 2 November 2026. Search keeps its postings in segments, as Lucene and SQLite's FTS5 do, so a large import builds its index in minutes, and searches answer faster. A dropped index now gives its pages back to the disk. Nothing that a program calls changes.

• Search
  • Postings in segments: 8.8 million passages indexed in under five minutes.
  • Faster searches: the best ten of 8.8 million passages in 3.3 ms at the median.
  • A dropped index gives its space back: search.db shrinks after dropIndex.

#Postings in segments

An import sets documents a thousand at a time, and each setMany commits them together:

const passages = store.index<Passage>('passages', { fields: { text: 1 } })
for (const batch of batches) await passages.setMany(batch) // up to 1,000 documents each

passages = store.index("passages", Passage, fields={"text": 1})
for batch in batches:
    await passages.set_many(batch)  # up to 1,000 documents each

passages, err := search.OpenIndex[Passage](store, "passages", search.Fields{"text": 1})
for _, batch := range batches {
	err = passages.SetMany(ctx, batch) // up to 1,000 documents each
}

let passages = store.index::<Passage>("passages").field("text", 1.0).open()?;
for batch in &batches {
    passages.set_many(batch.iter().map(|(id, passage)| (id.as_str(), passage)))?;
}

Before 0.5.1, a batch added each word's postings to that word's list, where the list lay in the file. A thousand passages hold about 12,000 different words, so one batch wrote about 12,000 pages. On MS MARCO, a build of that layout got slower with every million passages, and it was stopped after three. Now each batch writes into the newest segment, and background work merges the segments:

const passages = store.index<Passage>('passages', { fields: { text: 1 } })
for (const batch of batches) await passages.setMany(batch)
const { hits } = await passages.search({ text: 'lisbon tram' })

A document is found by the next search as soon as its setMany returns (contract SRCH-04). A batch writes about as many pages as its postings fill, however many words it holds, and a posting is written again only once per segment size.

On MS MARCO's 8.8 million passages, set a thousand at a time from two threads, the index was built in 291 s: 267 s for the sets, and 24 s to merge what was left into one segment. In an earlier run on the same machine, FTS5 built the same passages in 3.4 minutes and Tantivy in 3.6. TinyStore also keeps each passage itself, 3.4 GB of them. The machine was a Ryzen 7 7700, each build in a WSL2 container with 2 CPUs, and the report has every number. How search finds draws the segments and their sizes.

#Faster searches

A search reads less to find the same ten:

const { hits } = await passages.search({ text: 'lisbon tram' }) // the best ten, exact

result = await passages.search("lisbon tram")  # the best ten, exact

result, err := passages.Search(ctx, "lisbon tram") // the best ten, exact

let found = passages.search("lisbon tram").run()?; // the best ten, exact

• A block's postings are unpacked eight bytes at a time.
• A document's length is read only for a posting that is scored. Most postings a search unpacks are never scored.
• A word that can't lift a document past the tenth best on its own is read only for the documents the other words found, and only while it can still change the ten.

On MS MARCO's 8.8 million passages, with 1,097 of its queries asked one at a time, warm:

The ranges cover four rounds in one run on a Ryzen 7 7700, each search in a WSL2 container with 2 CPUs. "Unmerged" is the index as background work left it after the import, in 22 segments. TinyStore also reads each hit's value, which the other engines don't. The report has every number and how to run it again. Every answer is still the exact best ten (contract SRCH-01).

#A dropped index gives its space back

A change to an index's fields means a new index under a new name: it is built beside the old one, so search keeps working, and then the old one is dropped:

await store.dropIndex('catalog-v2') // true: dropped, its rows removed in the background

await store.drop_index("catalog-v2")

dropped, err := search.Drop(ctx, store, "catalog-v2")

store.drop_index("catalog-v2")?;

Before 0.5.1, search.db kept every page the old index had, so the file stayed at its peak size for good (#46). Now search.db gives the pages back to the disk once they have stayed free for ten minutes, 4 MiB every ten seconds. A purge and a rebuild give their pages back the same way.

A search.db that a release before 0.5.1 made gives pages back only after one vacuum of the whole file. Maintenance runs that vacuum while the file holds at most 8 MiB of data. A larger file keeps its free pages and fills them with its next writes, as before. Rebuilds has the details.

#Install

bun add @tinyshed/tinystore@0.5.1
npm install @tinyshed/tinystore@0.5.1

pip install tinyshed-tinystore==0.5.1

go get tinyshed.org/tinystore@v0.5.1

cargo add tinystore@0.5.1
cargo install tinystore-cli@0.5.1

0.5.1 moves search.db to segments when it first opens the file, and 0.5.0 refuses the file after that. The wire protocol doesn't change, so a 0.5.0 server works with a 0.5.1 SDK, and a 0.5.1 server with a 0.5.0 SDK. The server image is ghcr.io/tinyshed/tinystore:0.5.1.

Every commit since 0.5.0 is in the GitHub release.