One directory, many engines

A store is one directory with a file per engine, held by one process at a time. This chapter shows what each file is, how engines share a file when you ask them to, and the four ways a program reaches a store: in its own process, through a sidecar, through a private child, or over the network.

#Four ways to open a store

import { connect, open } from '@tinyshed/tinystore'

const shared = await open('./data')                       // the directory's sidecar, shared with other processes
const alone = await open('./data', { private: true })     // a server for this process alone: tests and scripts
const inside = await open('./data', { embedded: true })   // the core inside this process: no sidecar
const remote = await connect('tls://db.internal:7443', { token })

shared = await tinystore.open("data")                   # the directory's sidecar, shared with other processes
alone = await tinystore.open("data", private=True)      # a server for this process alone: tests and scripts
inside = await tinystore.open("data", embedded=True)    # the core inside this process: no sidecar
remote = await tinystore.connect("tls://db.internal:7443", token=token)

shared, err := tinystore.Open(ctx, "data")                      // the directory's sidecar, shared with other processes
alone, err := tinystore.Open(ctx, "data", tinystore.Private())   // a server for this process alone: tests and scripts
inside, err := tinystore.Open(ctx, "data", tinystore.Embedded()) // the core inside this process: no sidecar
remote, err := tinystore.Connect(ctx, "tls://db.internal:7443", token)

// A Rust program always runs the core in its own process.
let store = Store::open("data", Options::default())?;

Each line above opens a different store, for the sake of the example. A real program picks one of them. The calls after open are the same in all four cases, because all four reach the same core:

#What is in the directory

data/
├── LOCK              held by the one process that owns the store
├── server/SERVE      the sidecar's address, so other programs find it
├── kv.db             buckets, counters, limits, `once`, live settings
├── jobs.db           queues and schedules
├── sql/
│   └── chat.db       a database the application named, one file each
├── blobs/            uploaded files
├── search.db         search indexes
├── records.db        logs and events
├── metrics.db        metrics
├── encryption.key    the key of encrypted buckets, unless you keep it elsewhere
└── log/              a node's log of each file's commits, when the store is a node

A file appears when its engine first opens. A program that only uses buckets has kv.db and nothing else. Every .db file is an SQLite database, and the -wal and -shm files next to one belong to it. Channels and locks keep nothing on the disk, so they have no file.

You never edit these files by hand, but you can read them. The tinystore command reads a store while it runs, and any SQLite tool opens a .db file.

#A file per engine

Every file has its own writer. A burst of a thousand log lines goes to records.db, while a sign-in writes its session to kv.db at the same time, and neither waits for the other (contract STORE-02).

Each file also has its own durability, which says how far a write goes before the call returns:

You can set one durability for the whole store when you open it, such as open('./data', { durability: 'os' }), and a database can set its own. The defaults suit most programs: what the application decides is synced, and what it observes about itself is fast.

#Two engines in one file

A write is atomic within one file, and never across two. So if a request inserts a message into sql/chat.db and adds a job to jobs.db, a crash between the two can leave the message without its job.

When two things must commit together, open one inside the other's file. A bucket or a queue opened from a database lives in that database's file, and commits with its rows:

const db = await store.database('chat', { migrations: './migrations' })
const replies = db.queue<Reply>('replies') // kept in sql/chat.db, not in jobs.db

db = await store.database("chat", migrations="migrations")
replies = db.queue("replies", Reply)  # kept in sql/chat.db, not in jobs.db

db, err := sqldb.Open(ctx, store, "chat", sqldb.Migrations(os.DirFS("migrations")))
replies, err := jobs.OpenQueue[Reply](db, "replies") // kept in sql/chat.db, not in jobs.db

let db = store.database("chat").migrations("migrations").open()?;
let replies = db.queue::<Reply>("replies").open()?; // kept in sql/chat.db, not in jobs.db

replies is a queue like any other, and a worker runs its jobs the same way. The difference is where it lives: a transaction of db can now insert a message and add its reply job in one commit (contract DUR-06). Chapter 7, Replies in the background, builds exactly that.

#Close the store

await store.close()

await store.close()

err := store.Close()

store.close()?;

close stops the workers, waits for the handlers that are running, writes what the loggers and metrics still hold, and closes the engines in the reverse order of opening. A store whose last handle is dropped closes the same way. A handle used after its store closed fails with a closed error (ClosedError in TypeScript and Python, tinystore.ErrClosed in Go, ErrorKind::Closed in Rust).

In TypeScript, await using store = await open(…) closes the store at the end of the block, and async with does the same in Python.

#Why this design?

Why one owner per directory. SQLite allows several processes to write one file. They then take turns through a lock on the file, and under load a process that waits too long gets database is locked. Writes from different processes also cannot share a disk sync, because no process sees the others' writes before they commit. With one owner, every write to a file goes through one queue in one process: waiting is a short, orderly line, and a hundred writes share a few syncs. The LOCK file makes the rule hold: a second process that tries to own the directory is refused (contract STORE-01).

Why the sidecar uses the same protocol as a remote server. A Bun, Python or Go call becomes a message of the wire protocol, whether it travels through memory to an embedded core, through a socket to a sidecar, or over TLS to another machine. Only the transport changes. So a program moves from one way to another by changing its open line, and a test that runs against a private child tests the same code paths as production. Chapter 22, One core, four languages, shows the protocol in detail.

Why a file per engine instead of one big database. Logs and metrics arrive in bursts and can lose their last second to a power loss. Sessions and jobs arrive steadily and must never lose anything. In one file, they would have to share one writer and one durability, so either logs would slow down sign-ins or sessions would be less safe. Separate files let each engine keep the right trade-off, and let you choose to join two of them when a commit must cover both.

#See also

• Your first store: a first program, and the files it makes.
• The chat we build: the application the rest of the book builds.