> Fetch the index of the TinyStore docs, every page as markdown, at https://tinyshed.org/tinystore/llms.txt

# TinyStore 0.5.1

TinyStore 0.5.1 came out on 2 November 2026. Search keeps its postings in
segments, as Lucene and SQLite's FTS5 do, so a large import builds its index
in minutes, and searches answer faster. A dropped index now gives its pages
back to the disk. Nothing that a program calls changes.

- **Search**
  - [Postings in segments](#postings-in-segments): 8.8 million passages indexed in under five minutes.
  - [Faster searches](#faster-searches): the best ten of 8.8 million passages in 3.3 ms at the median.
  - [A dropped index gives its space back](#a-dropped-index-gives-its-space-back): `search.db` shrinks after `dropIndex`.

> \[!WARNING]
> **0.5.0 can't open `search.db` after 0.5.1**
> 0.5.1 moves `search.db` to a new layout when it first opens the file. After
> that, 0.5.0 refuses it. An index that 0.5.0 wrote answers searches right
> away, and background work moves it into segments. The other files of a
> store don't change.

## Postings in segments

An import sets documents a thousand at a time, and each `setMany` commits
them together:

```ts
const passages = store.index<Passage>('passages', { fields: { text: 1 } })
for (const batch of batches) await passages.setMany(batch) // up to 1,000 documents each
```

```python
passages = store.index("passages", Passage, fields={"text": 1})
for batch in batches:
    await passages.set_many(batch)  # up to 1,000 documents each
```

```go
passages, err := search.OpenIndex[Passage](store, "passages", search.Fields{"text": 1})
for _, batch := range batches {
	err = passages.SetMany(ctx, batch) // up to 1,000 documents each
}
```

```rust
let passages = store.index::<Passage>("passages").field("text", 1.0).open()?;
for batch in &batches {
    passages.set_many(batch.iter().map(|(id, passage)| (id.as_str(), passage)))?;
}
```

Before 0.5.1, a batch added each word's postings to that word's list, where
the list lay in the file. A thousand passages hold about 12,000 different
words, so one batch wrote about 12,000 pages. On MS MARCO, a build of that
layout got slower with every million passages, and it was stopped after
three. Now each batch writes into the newest segment, and background work
merges the segments:

```scene
code ts
  const passages = store.index<Passage>('passages', { fields: { text: 1 } })
  for (const batch of batches) await passages.setMany(batch)
  const { hits } = await passages.search({ text: 'lisbon tram' })
end
actor import "Your import"           at 8,50  icon=terminal
actor newest "Newest segment"        at 38,25 icon=file
actor full   "Full segments"         at 38,75 icon=layers
actor merged "Segment of 10,000"     at 68,75 icon=database
actor query  "A search"              at 92,25 icon=search
link import newest
link newest full
link full merged
link newest query
link merged query
set newest "empty"
set full "9 × 1,000 passages"
set merged "none yet"

step "Your import sets a thousand passages. Their words go into the newest segment, one piece per word, side by side in the file." line=2
  send import -> newest "1,000 passages"
  set newest "1,000 passages"
step "So the batch writes about as many pages as its postings fill, not a page for each of its 12,000 words." line=2
  pulse newest
step "The next batch starts a new segment. Ten full segments of one size are merged into one of the next size in the background, a few thousand words per commit." line=2
  send newest -> full "full"
  set full "10 × 1,000 passages"
  send full -> merged "merge"
  set merged "10,000 passages"
  & set full "none"
  set newest "empty"
step "A search reads each word's piece in every segment as one list. A passage is found as soon as its setMany returns." line=3
  send query -> newest "lisbon tram"
  & send query -> merged "lisbon tram"
  send merged -> query "the best ten" color=ok
```

A document is found by the next search as soon as its `setMany` returns
(contract SRCH-04). A batch writes about as many pages as its postings fill,
however many words it holds, and a posting is written again only once per
segment size.

On MS MARCO's 8.8 million passages, set a thousand at a time from two
threads, the index was built in 291 s: 267 s for the sets, and 24 s to merge
what was left into one segment. In an earlier run on the same machine, FTS5
built the same passages in 3.4 minutes and Tantivy in 3.6. TinyStore also
keeps each passage itself, 3.4 GB of them. The machine was a Ryzen 7 7700,
each build in a WSL2 container with 2 CPUs, and the
[report](https://github.com/tinyshed/research/blob/main/tinystore/reports/search-tinystore-2026-11-02.md)
has every number. [How search finds](https://tinyshed.org/tinystore/docs/handbook/inside/how-search-finds.md)
draws the segments and their sizes.

## Faster searches

A search reads less to find the same ten:

```ts
const { hits } = await passages.search({ text: 'lisbon tram' }) // the best ten, exact
```

```python
result = await passages.search("lisbon tram")  # the best ten, exact
```

```go
result, err := passages.Search(ctx, "lisbon tram") // the best ten, exact
```

```rust
let found = passages.search("lisbon tram").run()?; // the best ten, exact
```

- A block's postings are unpacked eight bytes at a time.
- A document's length is read only for a posting that is scored. Most
  postings a search unpacks are never scored.
- A word that can't lift a document past the tenth best on its own is read
  only for the documents the other words found, and only while it can still
  change the ten.

On MS MARCO's 8.8 million passages, with 1,097 of its queries asked one at a
time, warm:

| Engine             | Median     | 99th percentile |
| ------------------ | ---------- | --------------- |
| TinyStore 0.5.1    | 3.3–3.5 ms | 21–23 ms        |
| the same, unmerged | 3.4–3.8 ms | 22–25 ms        |
| Tantivy            | 2.3–2.5 ms | 27–30 ms        |
| PISA               | 2.3–2.7 ms | 15–16 ms        |
| Lucene             | 1.7–1.9 ms | 12–13 ms        |

The ranges cover four rounds in one run on a Ryzen 7 7700, each search in a
WSL2 container with 2 CPUs. "Unmerged" is the index as background work left
it after the import, in 22 segments. TinyStore also reads each hit's value,
which the other engines don't. The
[report](https://github.com/tinyshed/research/blob/main/tinystore/reports/search-tinystore-queries-2026-11-02.md)
has every number and how to run it again. Every answer is still the exact
best ten (contract SRCH-01).

## A dropped index gives its space back

A change to an index's fields means a new index under a new name: it is
built beside the old one, so search keeps working, and then the old one is
dropped:

```ts
await store.dropIndex('catalog-v2') // true: dropped, its rows removed in the background
```

```python
await store.drop_index("catalog-v2")
```

```go
dropped, err := search.Drop(ctx, store, "catalog-v2")
```

```rust
store.drop_index("catalog-v2")?;
```

Before 0.5.1, `search.db` kept every page the old index had, so the file
stayed at its peak size for good ([#46](https://github.com/tinyshed/tinystore/issues/46)).
Now `search.db` gives the pages back to the disk once they have stayed free
for ten minutes, 4 MiB every ten seconds. A purge and a rebuild give their
pages back the same way.

A `search.db` that a release before 0.5.1 made gives pages back only after
one vacuum of the whole file. Maintenance runs that vacuum while the file
holds at most 8 MiB of data. A larger file keeps its free pages and fills
them with its next writes, as before. [Rebuilds](https://tinyshed.org/tinystore/docs/guides/search/rebuilds.md) has the details.

## Install

```sh for=bun
bun add @tinyshed/tinystore@0.5.1
npm install @tinyshed/tinystore@0.5.1
```

```sh for=python
pip install tinyshed-tinystore==0.5.1
```

```sh for=go
go get tinyshed.org/tinystore@v0.5.1
```

```sh for=rust
cargo add tinystore@0.5.1
cargo install tinystore-cli@0.5.1
```

0.5.1 moves `search.db` to segments when it first opens the file, and 0.5.0
refuses the file after that. The wire protocol doesn't change, so a 0.5.0
server works with a 0.5.1 SDK, and a 0.5.1 server with a 0.5.0 SDK. The
server image is `ghcr.io/tinyshed/tinystore:0.5.1`.

Every commit since 0.5.0 is in the
[GitHub release](https://github.com/tinyshed/tinystore/releases/tag/v0.5.1).