Why TinyStore

TinyStore is the storage of one application in one directory: SQL databases, key-value state, background jobs, files, search, logs, metrics and more, each an engine of the same runtime. This chapter says what it replaces, what it is not, and why it is built the way it is.

#The stack it replaces

A web application on one server usually grows a stack of services around it:

Each of those services is a process to install, configure, secure, monitor, upgrade and back up. Each one fails in its own way, and the application has to handle every one of those failures. Most applications never need the scale these services were built for. They pay for it anyway.

TinyStore replaces the stack with a directory that belongs to the application:

data/
├── sql/chat.db    your SQL databases, one file each
├── kv.db          sessions, codes, counters, limits, settings
├── jobs.db        background jobs and schedules
├── blobs/         uploaded files
├── search.db      search indexes
├── records.db     logs and events
└── metrics.db     metrics

You open the whole directory with one call, and every engine opens from it:

import { open } from '@tinyshed/tinystore'

await using store = await open('./data')

const sessions = store.bucket<Session>('sessions', { idle: '30d' })
const emails = store.queue<Email>('emails')

import tinystore

async with await tinystore.open("data") as store:
    sessions = store.bucket("sessions", Session, idle="30d")
    emails = store.queue("emails", Email)

store, err := tinystore.Open(ctx, "data")
if err != nil {
	return err
}
defer store.Close()

sessions, err := kv.OpenBucket[Session](store, "sessions", kv.Idle(30*24*time.Hour))
emails, err := jobs.OpenQueue[Email](store, "emails")

let store = Store::open("data", Options::default())?;

let sessions = store.bucket::<Session>("sessions").idle(Duration::from_hours(24 * 30)).open()?;
let emails = store.queue::<Email>("emails").open()?;

There is no server to install. To back up the application, you back up the directory. To run a test, you open a store in a temporary directory and delete it afterwards.

#The engines

Every engine speaks the same small vocabulary. A bucket, a queue and a stream open from the store by their name. A call that writes returns after the write is on disk. An error has one of the same few kinds in every engine and every language: not found, conflict, limit, invalid and a few more.

#What TinyStore is not

• It is not a database server for many machines. One process holds a directory at a time. Other processes on the same machine share it through that process, and other machines reach it over the network. Several machines can keep copies of a store, but each file is written on one machine at a time. If many machines must write the same rows at once, use a database server.
• It is not faster than a specialised database at that database's job. PostgreSQL plans complex queries better, and Redis on its own answers a single key faster than a write that waits for the disk. TinyStore makes the whole workload of one application cheap, and it removes the network hops between the services.
• It is young. 0.1 is the first release. Until 1.0, a minor release may still change the API.

#Why this design?

Why one directory. The state of an application is more than its tables. A session, a job waiting to run, an uploaded avatar and yesterday's logs are all state. When each lives in a different service, a backup is several backups taken at different moments, and a restore never quite matches. With one directory, a backup is one zip of one moment, and moving the application to another server is copying a folder.

Why SQLite under every engine. SQLite is the most tested storage engine there is, and its file format survives crashes and power loss. It needs no server. TinyStore runs it in a specific way: one writer per file, writes from many callers grouped into one disk sync, and readers that never block the writer. Under the hood explains each choice.

Why a file per engine. SQLite lets one connection write a file at a time. If logs, jobs and sessions shared a file, a burst of log lines would make a sign-in wait. With a file per engine, no engine ever waits for another engine's writer. When you do want two engines to commit together, such as a job and the row it is about, you open the job's queue inside your SQL database, and the two share that database's file.

Why one core in Rust. The first versions of TinyStore were written in Go, and the Bun and Python SDKs had to repeat parts of it: the logger, the config, the console format. Each copy drifted. Now one core does everything, and every language reaches it through the same wire protocol. A feature is written once and appears in every SDK with the same behaviour.

#See also

• Your first store: install TinyStore and run a first program.
• One directory, many engines: what is in the directory, and how each language reaches it.