Wire Protocol

Security model

Two independent layers, both worth understanding before writing a client:

Socket file permissions

reamer-stream sets the socket file to 0600 (owner read/write only) immediately after bind() — not left to whatever the process's umask happens to produce. This restricts access between users: only the account the server runs as (and root) can even open the socket.

Optional shared-secret token

File permissions alone don't restrict access between processes running as the same user. Setting the REAMER_STREAM_AUTH_TOKEN environment variable before starting reamer-stream closes that gap: every new connection must then send an Auth message with a matching token as its first frame — anything else sent first gets the connection closed immediately, no response. Unset (the default): no auth required. There is no --auth-token=<...> CLI flag deliberately — a secret value passed via argv is visible to any other process on the machine (/proc/<pid>/cmdline, ps), which would defeat the token's own purpose.

At rest. .bin files, .reamer result files, and audit.log are all stored unencrypted. Encrypting them — e.g. filesystem-level encryption at the deploying firm's own OS layer — is explicitly the deploying firm's responsibility, not something Reamer Stream does itself.

Transport & configuration

A Unix domain socket, same machine only — no TCP/remote option exists in v1. Default path: /tmp/reamer-stream.sock, overridable via reamer-stream --socket-path=<path>.

Precedence for the data directory: --data-dir flag, then REAMER_STREAM_DATA_DIR, then ./data.

The server also caps concurrent connections at 256 (not configurable) — a same-machine ceiling, not sized for internet-facing exposure. A connection opened past the cap is accepted then immediately closed with no data sent.

Framing

There is no pagination in either direction — a HistoricalResponse can be arbitrarily large, and an IngestCsv/IngestBars request is capped only at a generous 128 MB. Read exactly length bytes before attempting to parse the payload; a large message may arrive across multiple reads.

All multi-byte integers are big-endian. All floating-point fields are IEEE 754 binary64, also big-endian — exactly what struct.pack('>d', x) does in Python. Never treat the wire bytes as directly castable to a C struct, even on a little-endian machine. A string field is length-prefixed: 2 bytes unsigned big-endian length, then that many raw UTF-8 bytes, no null terminator.

QueryHistorical / HistoricalResponse

Client → server: QueryHistorical

has_start=0 means "from the first bar available"; has_end=0 means "through the last bar available." Both 0 returns the ticker's entire series.

Server → client: HistoricalResponse

Each bar record, in order (64 bytes total — an explicit field-by-field big-endian encoding, not a raw copy of the in-memory struct):

Bars are returned in ascending ts order.

IngestBars & IngestCsv

IngestBars is for a connector that already has structured bars — a Databento or Alpaca connector pulling from a vendor API that returns typed records directly. IngestCsv is for anything with a CSV file and no structured-bar step of its own — the server parses it using the same auto-detecting parser every other Reamer Stream ingestion path uses.

Server → client: IngestAck

Ingested data is immediately queryable via QueryHistorical — including on the same connection, right after the IngestAck.

Subscribe / Unsubscribe / Publish

A stream name is a plain string with no relationship to a ticker unless a caller chooses to reuse the same string for both.

Subscribing twice to the same stream on the same connection is harmless (idempotent). Publishing to a stream with zero current subscribers is not a failure — PublishAck{success=true} either way; the record is simply dropped. No persistence: a subscriber not listening at publish time never sees that record.

Server → client: PushedRecord (unsolicited)

Auth

A wrong token gets no AuthAck at all — the connection is simply closed, so a peer can't distinguish "wrong token" from "malformed frame" from the outside.

Concurrency semantics

reamer-stream is single-threaded: StreamServer::poll() is one epoll_wait loop, and every accepted event — including the blocking StreamEngine call inside it — runs to full completion before the loop returns to epoll_wait for the next one. There is no thread pool, no per-connection thread.

What this does not claim: no ordering guarantee between two connections' requests that both arrive before the server gets to either one — which one epoll_wait happens to report first is unspecified. The guarantee is about atomicity of each individual request, not about which one wins a race to be first.

This is a property of the single-threaded design, not a lock: StreamEngine itself has no internal synchronization and is not safe to call concurrently with no external locking — reamer-stream's own single-threaded dispatch is what makes the guarantee above hold.

Failure modes

A client can tell which of two categories a failed request falls into from the response shape alone: silence (connection closes, nothing sent back) always means the request itself couldn't be trusted enough to interpret; an explicit success=false/found=false field always means the request was understood and something at the data layer said no.

Connection closed, no response sent

Normal ack/response with a failure field, connection stays open

Worked example

Request: QueryHistorical for ticker "AAPL", no bounds:

Response: ticker found, one bar (ts=1700000000, all price/volume fields 0.0, tick_count=1):