Your first store

This chapter installs TinyStore and runs a first program: it keeps a value, adds a background job, runs it and logs a line. Then it looks inside the store with the tinystore command. It takes about ten minutes, in TypeScript, Python, Go or Rust.

#Install

bun add @tinyshed/tinystore

pip install tinyshed-tinystore

go get tinyshed.org/tinystore

cargo add tinystore
cargo add serde --features derive
cargo add tracing tracing-subscriber

# the SQLite build TinyStore expects, which a dependency cannot set for you
[env]
LIBSQLITE3_FLAGS = "SQLITE_DQS=0 -USQLITE_ENABLE_MEMORY_MANAGEMENT SQLITE_ENABLE_DBPAGE_VTAB"

Rust programs compile the core in, so a Rust program is the store. It builds SQLite too, with the flags in .cargo/config.toml: without them readers wait on one another, which the store logs once when it opens. The other languages talk to the same core in a separate process by default, as How the store runs explains below.

#The whole program

import { open } from '@tinyshed/tinystore'

interface User {
	name: string
	visits: number
}

interface Email {
	to: string
	subject: string
}

await using store = await open('./data')

const users = store.bucket<User>('users')
await users.set('42', { name: 'Ada', visits: 1 })
console.log(await users.get('42')) // { name: 'Ada', visits: 1 }

const emails = store.queue<Email>('emails')
await emails.add({ to: 'ada@example.com', subject: 'Welcome' })

const log = store.logger('hello')
await emails.runDue(async email => {
	log.info('email sent', { to: email.to })
})

import asyncio
from dataclasses import dataclass

import tinystore


@dataclass
class User:
    name: str
    visits: int


@dataclass
class Email:
    to: str
    subject: str


async def main() -> None:
    async with await tinystore.open("data") as store:
        users = store.bucket("users", User)
        await users.set("42", User(name="Ada", visits=1))
        print(await users.get("42"))  # User(name='Ada', visits=1)

        emails = store.queue("emails", Email)
        await emails.add(Email(to="ada@example.com", subject="Welcome"))

        log = store.logger("hello")

        async def send(email: Email, run: tinystore.Run) -> None:
            log.info("email sent", to=email.to)

        await emails.run_due(send)


asyncio.run(main())

package main

import (
	"context"
	"fmt"
	"log/slog"

	"tinyshed.org/tinystore"
	"tinyshed.org/tinystore/jobs"
	"tinyshed.org/tinystore/kv"
	"tinyshed.org/tinystore/records"
)

type User struct {
	Name   string `json:"name"`
	Visits int    `json:"visits"`
}

type Email struct {
	To      string `json:"to"`
	Subject string `json:"subject"`
}

func main() {
	ctx := context.Background()
	store, err := tinystore.Open(ctx, "data")
	if err != nil {
		panic(err)
	}
	defer store.Close()

	// errors left out from here on
	users, err := kv.OpenBucket[User](store, "users")
	err = users.Set(ctx, "42", User{Name: "Ada", Visits: 1})
	user, found, err := users.Get(ctx, "42")
	fmt.Println(user, found) // {Ada 1} true

	emails, err := jobs.OpenQueue[Email](store, "emails")
	_, err = emails.Add(ctx, Email{To: "ada@example.com", Subject: "Welcome"})

	log := slog.New(records.NewHandler(store, "hello"))
	err = emails.RunDue(ctx, func(ctx context.Context, email Email, run *jobs.Run) error {
		log.Info("email sent", "to", email.To)
		return nil
	})
}

use serde::{Deserialize, Serialize};
use tinystore::{Options, Store};
use tracing_subscriber::prelude::*;

#[derive(Serialize, Deserialize, Debug)]
struct User {
    name: String,
    visits: u32,
}

#[derive(Serialize, Deserialize)]
struct Email {
    to: String,
    subject: String,
}

fn main() -> tinystore::Result<()> {
    let store = Store::open("data", Options::default())?;

    let users = store.bucket::<User>("users").open()?;
    users.set("42", &User { name: "Ada".into(), visits: 1 })?;
    println!("{:?}", users.get("42")?); // Some(User { name: "Ada", visits: 1 })

    let emails = store.queue::<Email>("emails").open()?;
    emails.add(&Email { to: "ada@example.com".into(), subject: "Welcome".into() })?;

    let log = store.logger("hello").layer()?;
    tracing_subscriber::registry().with(log).init();
    emails.run_due(|email: Email, _run| {
        tracing::info!(to = %email.to, "email sent");
        Ok::<_, std::convert::Infallible>(())
    })?;
    store.close()
}

Run it, and it prints the user and one log line:

15:39:52.006 INFO  hello  email sent  to=ada@example.com

The rest of this chapter goes through the program a piece at a time.

#Open the store

open takes a directory and creates it if it is missing. Everything the program keeps goes into that directory, and nothing goes anywhere else. When the program ends, await using in TypeScript, async with in Python, defer store.Close() in Go and store.close() in Rust close the store. A store that is closed has written everything it was given.

One process holds a directory at a time. If a second program opens the same directory, it does not get a second copy of the data: it joins the first store, as How the store runs shows.

#Keep a value

A bucket is a named collection of values of one type, by key. set writes a value and returns after it is on the disk. get reads it back, or returns nothing (undefined, None, found false, None) when the key is not there.

The type is the one you give the bucket: an interface in TypeScript, a dataclass in Python, a struct in Go and Rust. The value is stored as JSON, so a Python program reads what a Go program wrote. Buckets do much more: keys that expire, versions for safe updates, branches such as one user's sessions. Chapter 6, Sessions and limits, covers them.

#Do work later

A queue holds jobs: work to do now, after a delay, or at a given time. add puts a job in the queue and returns after the job is on the disk. A worker runs a handler on each job. If the handler returns, the job is done. If it throws or returns an error, the job runs again after a pause.

This program uses runDue (run_due, RunDue), which runs the jobs that are due and then returns. That suits a script or a test. A server calls work instead, which keeps running jobs as they come due until the store closes:

emails.work(async email => {
	await smtp.send(email)
})

async def deliver(email: Email, run: tinystore.Run) -> None:
    await smtp.send(email)


emails.work(deliver)

err = emails.Work(ctx, func(ctx context.Context, email Email, run *jobs.Run) error {
	return smtp.Send(ctx, email)
})

let worker = emails.work(|email: Email, _run| smtp.send(&email))?;

#What a crash does

A job is on the disk before add returns. So if the program dies right after it, the job is still there when the program starts again, and a worker runs it then:

await emails.add({ to: 'ada@example.com', subject: 'Welcome' })

emails.work(async email => {
  await smtp.send(email)
})

A write that returned success survives the process dying (contract DUR-01). The other side of this is that a job runs at least once, not exactly once. If the program dies after the email was sent but before the job was marked done, the email is sent again after the restart. Chapter 7, Replies in the background, shows how a handler makes a second run harmless.

#Log what happened

store.logger('hello') is a logger whose lines go to two places. It prints each line to the console, as any logger does. It also keeps the line in the store, in a stream named hello, so that you can read your logs back later by time, level, fields or text.

Go uses the standard slog package with a handler from TinyStore, and Rust uses tracing with a layer from TinyStore, so the rest of your program logs as it always did. A log call never waits for the disk. The logger keeps its lines in memory and writes them to the store within a second.

#Look inside

The tinystore command reads a store while your program runs. It never changes anything and never stops the program:

tinystore status ./data

● ./data  served by tinystore 0.1.0 · pid 15865 · up 4s

  kv        76 KiB
  jobs     125 KiB
  records   42 KiB
  ────────────────
  total    243 KiB

tinystore kv ./data list users    # 42  {"name":"Ada","visits":1}
tinystore logs ./data             # 15:39:52.006 INFO  hello  email sent  to=ada@example.com
tinystore logs ./data -f          # every new line as it arrives, until Ctrl+C

tinystore alone lists every command. Chapter 28, Running it, covers them all, along with tinystore mcp, which lets an AI agent read the store.

#How the store runs

The directory now contains a file for each engine the program used:

data/
├── LOCK               held by the process that serves the store
├── kv.db              the bucket
├── jobs.db            the queue
├── records.db         the log line
└── server/SERVE       how other programs find the sidecar

The engines open lazily: kv.db appears when the first bucket opens, and a program that never logs has no records.db. Next to each file you may see -wal and -shm files. Those are SQLite's write-ahead log, and they belong to the file beside them.

A Rust program runs the store inside its own process. A Bun, Node, Python or Go program does not, by default. Its SDK starts a small tinystore process next to it, called the sidecar, and talks to it over a local socket or a named pipe on Windows. The sidecar writes its address to server/SERVE, so a second program that opens the same directory finds it and joins it. When the last program has disconnected, the sidecar waits 30 seconds and exits.

#Why this design?

Why a sidecar by default. A directory can have only one owner at a time, because two processes writing the same SQLite file through separate caches would wait on each other's locks, and their background work would collide. But a web application often runs several processes: four workers behind a load balancer, and a cron script beside them. The sidecar lets all of them share one store with one owner. If your program is the only one that ever opens the directory, open it with embedded: true (embedded=True, tinystore.Embedded()), and the core runs inside your process with no sidecar at all.

Why a write waits for the disk. A program that was told "saved" must find the value after a crash or a power loss, so set and add return only after the disk has synced. A sync takes time, so TinyStore groups writes: while one sync runs, the writes that arrive wait together and share the next one. A hundred concurrent requests pay for a few syncs, not a hundred. Chapter 15, The journey of a write, follows one write all the way down.

Why logs do not wait. A log line is worth less than the request that writes it, so the logger never makes a request wait for the disk. Lines wait in a bounded buffer and are written every second. If the buffer is full, a line is dropped and counted, never blocking the program. Logs and metrics also default to a lighter durability than the other engines: they survive the program crashing, but a power loss may take the last second of them.

#See also

• One directory, many engines: every file in the directory, and every way to reach it.
• Why TinyStore: what TinyStore replaces, and what it is not.