Readers in another region

The chat's users are in America and Europe. Every read from Europe crosses the ocean, about 100 ms each way, and the chat feels slow there. This chapter puts a copy of the store's files in Europe: reads become local, and writes still go to the one machine that writes each file. TinyStore calls every store a node; a node holds the files it asks for, writes some of them, and serves them to other nodes.

#One home per file

Every file has one home: the node that writes it. Every other node holding the file holds a copy, which only readers and the replication engine open. A copy receives the home's commits as pages, in order, and applies each one whole.

A copy of a file is the home's file page for page after each commit it applies, and a reader of the copy never sees half a commit (contract REPL-04). It may be behind its home by the time the pages take to arrive (contract REPL-05).

#Start two nodes

# America: the home, where the chat was deployed and its migrations run
tinystore serve ./us --node-listen 0.0.0.0:7444 --node-config node.toml

# Europe: copies every file from America, and serves the programs in Europe
tinystore serve ./eu --from us.internal:7444 --node-listen 0.0.0.0:7444 --node-config node.toml

token-file = "/etc/tinystore/node-token"   # the token that nodes show each other
advertise = "eu.internal:7444"             # where other nodes reach this one

[tls]
cert = "/etc/tinystore/node.pem"
key = "/etc/tinystore/node.key"
roots = "/etc/tinystore/ca.pem"

A link between two machines takes TLS. A node with no [tls] refuses a link beyond its own machine unless its config says insecure = true (contract REPL-11). The European chat processes open ./eu as they open any store, through its sidecar, so their code does not change.

A Bun program can be a node itself, with its store inside it:

const store = await open('./data', { embedded: true, node: { from: ['eu.internal:7444'], token } })

#Reads are local

const session = await sessions.under(userId).get(digest(token)) // read from Europe's copy
const page = await messages.where({ chat_id: chatId }).orderBy('id', 'desc').list({ limit: 50 })

Every read, a key, a page of messages, a search, a log query, runs on the local copy. Nothing crosses the ocean.

#Writes go home

A write on a copy is passed to the file's home. The home commits it, the pages come back, and the call answers once the local copy holds the commit:

await sessions.under(userId).set(digest(token), session)

A write on a copy passes to its home and answers once the copy holds it (contract REPL-06). The program reads its own writes: a get right after a set sees it. The cost is one round trip to the home.

A counter's add answers at once, because two adds give the same total in any order. The node keeps the add in a queue of its own and sends it to the home with an id, so an add sent twice after a lost link counts once (contract REPL-12).

#Writes that must not wait for the ocean

A write on a copy pays the round trip to its home. A file whose writers are in one region can have its home there instead: each node writes the files it makes. Give the chat's European tenants a database of their own, made by the European node, and its home is Europe:

const tenant = await store.database(`tenant-${tenantId}`, { migrations }) // made here, written here

A catalog, catalog.db, names the home of every file. One node a person chooses keeps it, catalog = true in its config; a node that makes a file claims its name there first, so two nodes never make the same file.

#When a home's disk must not lose a commit

By default a home answers a write after its own commit, and its copies trail. With synchronous = true in the home's config, a write answers only once a copy holds it on its disk. Losing the home then loses nothing it answered. A commit that no copy held within five seconds still stands in the home's file, and answers committed locally instead of plain success, because losing that home's disk would lose it (contract REPL-09).

#Why this design?

Why pages and not operations. A copy could receive the operations, "set key 42", and apply them itself. Then every engine would need replication of its own, and a copy whose engine version differs could compute another result. Pages are the result: the copy's file becomes the home's file, byte for byte, whatever engine wrote it, migrations and indexes included.

Why one home and no merging. Two homes of one file would have to merge two versions of a page, and SQLite's pages cannot be merged. Systems that accept writes on both sides of a cut cable pay with conflicts that the program must resolve. TinyStore keeps one writer per file and puts each file's home where its writers are.

Why the network is the core's. The links run on the store's own threads, so a store inside a Bun or Python program is the same node as a server. A small service can hold a copy of the one database it reads, with no sidecar and no extra process.

#See also

• When the home dies: moving a home, and promoting a copy.
• The server: serving a store to programs on other machines.