# The wire protocol, version 1 The bytes a TinyStore server and its clients exchange over any byte stream: a child's stdin and stdout, a Unix socket, a Windows named pipe, TCP or TLS. Read it to write a client in a language TinyStore has no SDK for, or to see what an SDK does on the wire; the Bun and Python SDKs are built to this page. Frames, the profile, the handshake, the errors and every engine's methods, kv, jobs, blobs, sql, records and metrics, are built in `server/wire`, and why the server behaves as it does is [its design](https://github.com/tinyshed/research/blob/main/tinystore/design/server.md). Every example on this page is a vector of [testdata/wire/vectors.json](https://github.com/tinyshed/tinystore/blob/f11772d3985c4e6daeafa51a62c67f1bb0b3c498/testdata/wire/vectors.json), which the server and every SDK are tested against: a value in a typed notation, the bytes it is, and the bytes a decoder refuses, each named after the rule it breaks. Every message is also a vector of [messages.json](https://github.com/tinyshed/tinystore/blob/f11772d3985c4e6daeafa51a62c67f1bb0b3c498/testdata/wire/messages.json), its bytes and its fields by the names the tables below give them, with the methods' numbers and the errors' codes, so that an SDK checks each field of each message in one loop; the Go types write it, through a schema of each message's keys that `TestMessageVectors` holds. ## Frames Everything is little-endian. A frame is a twelve-byte header and a body: ```text 0 4 5 6 8 12 │ body length │ kind │ flags │ method │ stream │ body … u32 u8 u8 u16 u32 ``` - **The length is the body's.** A frame is 12 + body length bytes; a body past the agreed maximum is refused before a byte of it is read. - **What is reserved is zero.** An unknown kind, a flag this kind does not define, a method outside a `REQUEST` or a stream where the kind names none is a protocol error, and the connection ends. | kind | | stream | method | body | flags | | ---- | ---------- | ----------------------------------- | ------------- | --------------------------------------- | ---------- | | 1 | `HELLO` | 0 | 0 | a message | — | | 2 | `WELCOME` | 0 | 0 | a message | — | | 3 | `REQUEST` | the new stream | the operation | a message | END | | 4 | `RESPONSE` | its stream | 0 | a message, or an error | END, ERROR | | 5 | `DATA` | its stream | 0 | bytes, or a message, as the method says | END, ERROR | | 6 | `CANCEL` | its stream | 0 | empty | — | | 7 | `CREDIT` | its stream, or 0 for the connection | 0 | u32: bytes granted | — | | 8 | `PING` | 0 | 0 | 8 bytes | — | | 9 | `PONG` | 0 | 0 | the `PING`'s 8 bytes | — | | 10 | `GOAWAY` | 0 | 0 | a message | — | END is 1: the sender's last frame on its stream. ERROR is 2, only beside END: the body is [an error](#errors). A method's high byte is its engine and its low byte the operation, so a new engine takes a new high byte and the frame stays as it is: | high byte | engine | | --------- | ------- | | `0x01` | kv | | `0x02` | jobs | | `0x03` | blobs | | `0x04` | sql | | `0x05` | records | | `0x06` | metrics | A `HELLO` from Bun: ```text 16 00 00 00 01 00 00 00 00 00 00 00 a body of 22 bytes, HELLO 82 a map of two 01 01 protocol: 1 02 b1 74 69 6e 79 73 74 6f 72 65 2d 62 75 6e 2f 30 2e 31 client: "tinystore-bun/0.1" ``` ## A connection 1. The client sends `HELLO`, and the server answers `WELCOME`, or `GOAWAY` and closes. Nothing comes before them, and the server waits five seconds for `HELLO`. The connection speaks the older of the client's newest protocol and the server's, which `WELCOME` names: a client newer than its server speaks the server's protocol, or closes when it no longer does, and only a client older than the oldest protocol the server speaks is refused. 2. The client opens streams with `REQUEST`, as many at once as `WELCOME` allows. 3. Either side may send `PING`; the other answers `PONG` with the same eight bytes. 4. The server ends the connection with `GOAWAY`. The client opens no stream after it; a `REQUEST` that crossed it on the way is answered `unavailable` without being run, so it may be sent again elsewhere; the streams already running finish, and the server closes. Stream numbers are reused, so no number could say which requests the server saw: each answer says it. `HELLO`: | key | field | type | | | --- | ------------- | ---- | ------------------------------------------------------------------------------------------- | | 1 | protocol | uint | the newest the client speaks, 1 | | 2 | client | str | a name and version, for logs | | 3 | token | str | required on TCP | | 4 | max body | uint | the largest body the client takes; the server's when absent | | 5 | stream credit | uint | the `DATA` the server may send on a stream before the client grants more; 2 MiB when absent | | 6 | challenge | bin | 16 random bytes, when the client found the server through `SERVE` | A body must fit the credit it is sent under, so the max body both sides agree is the smallest of the server's, the client's and the client's stream credit: a client granting 64 KiB a stream takes bodies of 64 KiB at most. `WELCOME`: | key | field | type | | | --- | ----------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------- | | 1 | protocol | uint | the one this connection speaks | | 2 | server | str | its version | | 3 | instance | bin | 16 random bytes a start | | 4 | capability | str | `admin`, `data` or `read` | | 5 | max body | uint | the largest body either side sends | | 6 | in flight | uint | the streams a client may have open at once | | 7 | connection credit | uint | the bytes of `REQUEST` and `DATA` bodies a client may send before credit comes back; at least the max body | | 8 | stream credit | uint | the bytes of `DATA` a client may send on a stream before credit comes back | | 9 | engines | array of str | what this server serves | | 10 | now | int | the store's clock, unix milliseconds | | 11 | proof | bin | on a local connection whose `HELLO` carried a challenge: the HMAC-SHA256 of the challenge, keyed with the 32 bytes of `SERVE`'s secret | | 12 | durability | str | `full` or `os`, when the store was opened with one: how far its files' commits go before they return | A client that found the server through `SERVE` sends no `REQUEST` before the proof checks, compared in constant time: only the store directory's owner can read `SERVE`, so a process that took the endpoint of a server gone cannot answer, and a client whose proof fails reads `SERVE` again, as a stale one. `HELLO` carries nothing a stranger could use, since a local connection takes no token. A challenge of another length is a `HELLO` the server cannot take; a remote connection's challenge gets no proof, its server being the one its TLS certificate names. The vectors hold a proof, `proofs`, and a `HELLO` with a challenge. A server that does not take the connection at all, past its connections or closing, still reads the `HELLO` and answers `GOAWAY` where the `WELCOME` belongs, with `limit` or `unavailable`. `GOAWAY`: | key | field | type | | | --- | ------- | ---- | ------------------------ | | 1 | code | str | as [an error's](#errors) | | 2 | message | str | | ## Finding a local server A store directory has at most one server, since its `LOCK` lets one process open it. That server publishes how to reach it in `/server/`, which only the directory's owner may enter: mode 0700 and, on Windows, which lets anyone traverse a directory to a file whose path they know, a protected DACL naming its owner alone, which every file made in it inherits. `SERVE` there is a hint; `LOCK` is the truth. ```json {"protocol": 1, "server": "0.4.0", "pid": 4212, "instance": "q1q0l3yZ3JvYw8g7oM2sYA", "secret": "Zm9vYmFyYmF6cXV4Zm9vYmFyYmF6cXV4Zm9vYmFyYmE", "endpoints": ["unix:///srv/app/data/server/tinystore.sock"], "sidecar": true} ``` | field | | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | protocol | the newest protocol the server speaks | | server | its version | | pid | its process, for a person reading the file; no client trusts it, since a pid is reused | | instance | 16 random bytes a start, base64url without padding, which `WELCOME` repeats | | secret | 32 random bytes a start, base64url without padding, the key of the proof | | endpoints | where it listens: `unix://` and a socket's path, or `pipe:` and a Windows named pipe's name | | sidecar | `true` when the server is a sidecar, which `tinystore serve --local` starts for its clients. A client of a newer release may replace a sidecar. The field is absent for a Go program's own server and for `tinystore serve `, which a person runs | - **Written whole, under the lock.** Only the store holding `LOCK` writes `server/`. A server removes a `SERVE` left behind before it listens, since the endpoint that file names may be another process's by now, writes its own once it listens, as a temporary file renamed into place, and removes it when it leaves, before it lets go of `LOCK`. - **Find or start.** A client reads `SERVE`, connects to an endpoint and checks the proof. Any failure, no file, a refused connection or a proof that does not check, starts \`tinystore serve --dir --local --log /server/serve.log`. That child takes `LOCK`, which means the old server is gone, and publishes itself; or it finds `LOCK` held and exits with code 3, and the client reads `SERVE` again until the winner answers, five seconds at most, starting one again when none does. - **The proof.** A client that found its server through `SERVE` sends a `HELLO` with 16 random bytes as its challenge, and sends no `REQUEST` until `WELCOME`'s proof equals the HMAC-SHA256 of the challenge keyed with the secret, compared in constant time. An endpoint's name is anyone's to take once its server is gone, a Windows pipe's above all, and only the directory's owner can read the secret. A client whose proof fails reads `SERVE` again, as a stale one. - **A reader delays a change and never fails it.** On Windows a client holding `SERVE` open refuses its rename or removal, and so may a scanner; the server tries again eight times, from 1 to 64 ms apart. How a client knows its server depends on the transport: | transport | how the client knows its server | | --------------------------------------------------- | ---------------------------------------------------------------------------- | | stdio | it started the server itself, as a private child | | a Unix socket or a named pipe found through `SERVE` | the proof | | TCP with TLS | the certificate, checked before its token leaves | | TCP | not at all: the token travels in the clear, on a network its operator trusts | Where a local server listens: - **A Unix socket**, `/server/tinystore.sock`, when that absolute path fits a `sockaddr_un`, 104 bytes on macOS and the BSDs and 108 on Linux; past it, the same name in a directory of the owner's own, `$XDG_RUNTIME_DIR/tinystore-`, or in the temporary directory without one. `` is the first eight bytes of the SHA-256 of the store's absolute path, in hex: eight, so that a socket under macOS's temporary directory still fits. - **A named pipe on Windows**, `pipe:tinystore-`, created as the first instance of its name with a DACL naming its owner alone, which refuses remote clients. No local endpoint takes a token: its permission is the file system's, and a local connection is `admin`. A remote server's tokens file gives each token one capability: `admin` may do anything, `data` reads and writes, and `read` reads only. A `read` connection may call only the methods the schema marks `read`, every other one being `permission`; its SQL runs on a reader and is a query, a statement that writes, attaches a file or sets a pragma being `permission` too; and its `sql.open` applies no migration. Its opens may still name a bucket, a queue or files the store did not have, as any first open does, and its reads of an `idle` key renew it, as any read does. ## Streams A client names a stream with any number from 1 not in use. A number is in use from its `REQUEST` to the server's final frame on it, and may be named again after. A `REQUEST` on a number in use, or past the streams in flight, ends the connection. | shape | the client sends | the server sends | | ---------- | ------------------------------ | ------------------------------- | | a call | `REQUEST`·END | `RESPONSE`·END | | an upload | `REQUEST`, `DATA`…, `DATA`·END | `RESPONSE`·END | | a download | `REQUEST`·END | `RESPONSE`, `DATA`…, `DATA`·END | | both ways | `REQUEST`, `DATA`…, `DATA`·END | `RESPONSE`, `DATA`…, `DATA`·END | - **The server ends every stream once**: `RESPONSE`·END or `DATA`·END, either with ERROR, a cancelled stream too. It takes the number out of use before that frame leaves, so a client may name it again the moment it arrives. - **A download's `RESPONSE` is its header**, a message of the method's, `{}` when it has nothing to say; its items follow as `DATA`, and its last `DATA`·END is its trailer, a message saying where the next page begins. - **`DATA` carries bytes or items.** A blob's bytes are raw; rows, jobs and series are a message each. The method says which. A `DATA` carries at least one byte, but the one that ends its stream. - **A transfer shares the connection.** A sender splits bytes into frames of 64 KiB at most, so that no stream holds the connection longer than one such frame takes to send. - **`CANCEL` asks the server to stop.** It answers with the final frame it would have sent, or with the error `cancelled`; a client may drop what arrives on a stream it cancelled. A client ends an upload early with `CANCEL`: ERROR is the server's alone. - **A frame for a stream no longer in use is dropped**: a `DATA` the client sent before the final frame reached it, a `CANCEL` or a `CREDIT` that crossed it. The dropped `DATA` still gives its connection credit back. - **A body that is not its method's message fails its stream** with `invalid`, naming the rule it broke; a frame that is not a frame ends the connection. ## Credit A sender never sends more than it was granted, and a receiver grants only room it has, so a connection holds no more than its grants and the server's reader never waits. - **Client to server.** Every `REQUEST` and `DATA` body counts against the connection's credit, and a `DATA` body against its stream's too. The server gives credit back with `CREDIT` as it lets go of bodies: on stream 0 for the connection, on a stream's number for that stream. - **Server to client.** `DATA` bodies count against the stream credit `HELLO` stated, and the client grants more with `CREDIT` on the stream as it consumes them. A `RESPONSE` body is not counted: it is one message within the agreed body, and every collection travels as `DATA` instead, so a client holds at most its streams in flight times the agreed body in answers. Nor is the `DATA`·END·ERROR that fails a download, so that a stream always has room for its end. A client grants nothing for the connection: its `CREDIT` names a stream. - **A grant waits for half the window.** A receiver grants what it let go of once half its window has gone, and not a frame for every frame it takes. - **A body past its credit** ends the connection with `GOAWAY` and `protocol`, the connection's last frame. 64 KiB of credit on stream 5: ```text 04 00 00 00 07 00 00 00 05 00 00 00 a body of 4 bytes, CREDIT, stream 5 00 00 01 00 65,536 ``` ## Messages: the MessagePack profile A message is one [MessagePack](https://github.com/msgpack/msgpack/blob/master/spec.md) value: a map from small unsigned integer keys to values, under these rules. - **Types.** Nil, bool, integers, float 64, str, bin, array and map. A float 32, an extension type, the timestamp among them, or any other type byte is invalid. - **Integers.** Each field says signed or unsigned and its range. A decoder takes any encoding of a value that fits, `ce 00 00 01 2c` for 300 as well as `cd 01 2c`, and refuses one that does not. - **A float keeps its value; bits that must survive are bin.** A float field is a float 64: `cb 80 00 00 00 00 00 00 00` is -0. The server keeps its bits, so `cb 7f f8 00 00 00 00 00 01`, a NaN whose payload is 1, comes back as it went; an SDK keeps the value, NaN, the infinities and -0 included, as far as its language does, and JavaScript promises no NaN's payload. A float whose bits are the data therefore travels as bin: a kv float value, a series' samples, an aggregate's values. A float field also takes an integer that a float 64 holds exactly, since a JavaScript encoder writes 3.0 as 3. - **Text is str, bytes are bin.** A str is valid UTF-8, and neither stands for the other. - **Keys.** A message's keys are unsigned integers; a field that is a map of names, an error's `what` or a series' labels, says so and has str keys. A key twice is invalid. A key the reader does not know is read past, its value still well formed and within bounds, and then it depends on the reader. A server refuses a request carrying one, `unimplemented` naming the field, since a field it skipped could change what the client asked, and the answer would be to another question. A client reading an answer, and a server reading `HELLO`, skip it. So a client leaves out a field at its zero value, and an older server takes everything a newer client sends that it would have sent anyway; and a server adds an answer field only where a client that skips it still reads the answer right, or else answers it only to a client that asked for it in a field of its request. An absent key is the field's default, and nil stands only where a field allows it. - **Bounds come before allocation.** A length or a count past the bytes left in the body is invalid, since every element takes at least a byte; nesting deeper than 8 is invalid; a body is at most the agreed maximum. - **Time is an integer**: unix milliseconds, unless a field says nanoseconds. Records keep nanoseconds, past what a JavaScript number holds, so an SDK reads such a field as a `BigInt`: ```text 1,790,000,000,123,456,789 ns → cf 18 d7 5b 84 2b 4e cd 15 ``` A duration is an integer of milliseconds. - **Columns of numbers are bin**: a series' times and values, little-endian int64 and float64 one after another. - **Encoders are canonical, decoders are not.** An encoder writes the shortest form of every integer, length and count, keys in ascending order, every float in eight bytes, so that a vector compares byte for byte; a decoder takes any valid form in any order. - **An SDK's codec is its own**: a general library writes plain maps and reads 64-bit integers without rounding when told, but takes a float 32, an extension, a key twice or a value nested nine deep, which this profile refuses. The SDKs' codecs refuse every `refused` vector. ## Errors An error is the final frame of its stream, with ERROR set: | key | field | type | | | --- | ------- | -------------- | ---------------------------------------------------------------------------------------------- | | 1 | code | str | below | | 2 | message | str | what failed, as the engine's error says it | | 3 | what | a map of names | the item it names: a bucket and key, a queue and key, a series' labels, a table and constraint | | code | means | sent again | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | | `invalid` | the request cannot be done as asked | no | | `limit` | a bound: memory, size or count; `what` names the `limit`, what the call `wanted` of it and the `bound`, where the engine knows them | later, or smaller | | `closed` | the store or the handle closed | after opening again | | `in_use` | a name is taken, or a lock's key stayed held past a hold's wait | later | | `conflict` | a condition or a version no longer holds | after reading again | | `corrupt` | stored bytes no longer read | no | | `too_old`, `too_new` | a time outside its engine's window | no | | `suspended` | a series in quarantine | after its repair | | `outcome_unknown` | a commit whose result is unknown | after reading what it wrote | | `committed_locally` | a synchronous commit its home made and no copy it waits for held in time: it stands, and losing that home's disk would lose it | no | | `lost` | a lease ran out or was revoked at its maxHold: another holder may have run beside it | no | | `permission` | the connection's capability does not allow it, as a `read` connection opening an encrypted bucket | no | | `unimplemented` | a method, or a field of a request, this server does not have: `what` names the field, the message the server's version | no | | `cancelled` | the client cancelled it | — | | `unavailable` | the server is closing | on another connection | | `internal` | a fault of the server's | — | The first nine are the root's sentinels, so that an SDK's error classes mean what `errors.Is` means in Go. A frame or a `HELLO` the server cannot take is a `GOAWAY` with `protocol` or `unauthenticated`, and the connection ends. A name's value in `what` is str, or bin where its bytes are not UTF-8, as a kv key of bytes may be. A conflict on stream 7: ```text 2c 00 00 00 04 03 00 00 07 00 00 00 a body of 44 bytes, RESPONSE, END and ERROR, stream 7 83 a map of three 01 a8 63 6f 6e 66 6c 69 63 74 code: "conflict" 02 ad 73 74 61 74 65 20 63 68 61 6e 67 65 64 message: "state changed" 03 81 a6 62 75 63 6b 65 74 a8 73 65 73 73 69 6f 6e 73 what: {"bucket": "sessions"} ``` ## Versions A server and its clients upgrade apart: a remote server deployed once while the applications' SDKs move on, or a sidecar still serving a directory after its application's SDK was updated. So each side promises what it does with an older or a newer one: - **A connection speaks the older protocol of its two sides.** `HELLO` says the newest the client speaks and `WELCOME` the one the connection speaks; a server refuses only a client older than the oldest protocol it still speaks. A protocol moves only for what a field cannot carry: frames, the handshake, credit. - **Within a protocol a message grows by fields, and a request is understood whole or refused.** A field the server does not know is `unimplemented`, naming the field and the server's version, so a newer client using something new learns which server to upgrade rather than reading an answer to another question. A client leaves a field out at its zero value, so one using nothing new talks to any server of its protocol. - **An answer grows only by what a client may skip.** A field whose meaning a client must know is answered only to a client that asked for it with a field of its request, which an older server refuses. - **A method is added, never repurposed.** An older server answers a newer method `unimplemented`, naming its version. From `v0.1.0` on, a message never changes meaning and a field's number is never reused. A client that finds a sidecar of an older release than its own replaces it: 1. The client sends `server.stop`. The sidecar answers, lets its running streams finish, and gives back `SERVE` and `LOCK`. 2. The client starts its own binary, as it does when no sidecar runs, and prints one line about it on stderr. 3. Every other client of the old sidecar gets `GOAWAY` and connects to the new sidecar when it connects again. A client doesn't replace a server that `SERVE` doesn't call a sidecar, such as a Go program's own server or one that a person runs. It also keeps a sidecar that refuses to stop. In both cases, it prints a warning once on stderr. ## Methods Each engine's messages are fixed on this page with its slice, before its code: numbers, keys, types, shapes and vectors. A call answers with one message; whatever returns a collection is a download, its items `DATA` under credit and its last `DATA` saying where the next page begins. What each will carry: | engine | operations | shape | | ---------- | ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | the server | stop | a call | | kv | open, get, has, set, delete, take, touch, add, max, clear, batch, view, allow, configure | calls | | | scan | a download, an entry a message | | | watch | a download that does not end: a config's kept fields, again after each change | | | run | both ways: the answer a key keeps, or the key's run handed over and its answer back | | | usage, refund | calls on a quota, which allow uses | | jobs | open, enqueue of many, update, cancel, get, claim, settle of many, step, keep | calls | | | scan | a download, an entry a message | | | watch | a download that ends with its job: its entry, again after each change | | | work | both ways: jobs out, their outcomes back | | blobs | open, stat, delete, copy, move, usage, clear | calls | | | scan | a download, an object a message | | | put | an upload | | | get, with an offset and a length | a download; a whole read whose bytes do not match its hash ends with ERROR and `corrupt` instead of END | | sql | open, with its migration files; exec; batch | calls | | | query | a download: the columns, then rows a message at a time | | records | append of many, drop | calls | | | read, follow | a download, records a message at a time | | | lines | an upload of another program's output | | metrics | ingest, drop | calls | | | read, aggregate | a download, a series a message, or several for a series longer than a body holds | | search | open, set, setMany, delete, get, search, status, dropIndex, list | calls | The values they carry: - the server's own call carries nothing, `{}` each way; - a kv value is nil, an integer or bin, as its row keeps it, a bin's first byte the kind that wrote it (1 text, 2 bytes, 3 JSON, 4 a float's eight bytes), and a version is an opaque bin compared only for equality; - a job's value is its JSON as str; - an SQL value is one of SQLite's five: nil, an integer, a float, str or bin; - a record's attribute keeps its JSON spelling as str; - a series' samples, and an aggregate's buckets and values, are bin columns. - a search's document is its JSON as str, and a vector bin, four bytes a little-endian float a dimension. ### the server The server's own methods take the range below the engines', `0x00xx`. | method | | request | answer | | -------- | ------ | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `0x0001` | stop | `{}` | `{}`, then the server stops as its closing does: the streams running finish, every connection is told `GOAWAY`, and a sidecar gives back `SERVE` and its directory | | `0x0002` | clock | a clock: a time to set it to, a while to move it forward by, or neither to read it | a clock: the time it reads once moved | | `0x0003` | backup | `{1: files}` | a download: `{}`, then the zip of the whole store in `DATA`, its bytes as they come, the last with END | A stop is an admin connection's alone, `permission` to a data connection, and a server whose program said nothing of stopping refuses it with `permission` too: a program serving its own store decides when that store's server stops. `tinystore serve` stops at one, as it does at Ctrl+C, which is how `tinystore stop` ends a sidecar and how an SDK newer than the sidecar it found replaces it with its own binary, as [Versions](#versions) says. Only an admin connection can ask for a backup. The server first opens each engine whose file is in the directory. It copies each database that no client has opened, without opening it. Then it sends what the `backup` package writes: a copy of every engine's file, and a manifest of their sizes and checksums, which a restore checks. `files`, an array of str, names files of the application's by their paths inside the store's directory, which the zip holds beside the engines' with the engine `host`; a path outside the directory, or an engine's own, is `invalid`. A clock is `{1: at, 2: advance}`. `at` is a time in unix milliseconds, and `advance` is a duration in milliseconds. Only a private server started with `tinystore serve --stdio --clock