Reclaim a connection once its write half to the client is shut down and the surviving client→upstream direction has carried nothing for this many ms. Minimum 1; 0 disables. See Dead-peer detection
TcpKeepaliveConfig
Field
Type
Default
Range
Description
enabled
boolean
true
—
false turns dead-peer detection off entirely
idleMs
number
300000
≥ 1000
Quiet time before the first probe
intervalMs
number
30000
≥ 1000
Gap between probes
retries
number
5
1–127
Unanswered probes before the connection is dropped
Out-of-range values are rejected by the constructor rather than clamped. The two timings reach
the kernel as whole seconds (TCP_KEEPIDLE / TCP_KEEPINTVL), so a sub-second value would be
installed as 0 and refused by setsockopt after SO_KEEPALIVE had already been enabled —
leaving a listener that reports a schedule it is not running. retries has the same problem
above TCP_KEEPCNT's ceiling of 127.
ListenerConfig
Field
Type
Default
Description
host
string
'0.0.0.0'
Bind address
port
number
required
Bind port
mode
'tls' | 'http'
'tls'
Listener protocol. 'http' serves plaintext for ACME HTTP-01 challenges; everything else gets a 301 to https://<host><uri>. See HTTP listener.
defaultCert
CertConfig
—
Fallback cert for routes without their own cert
mtls
MtlsConfig
—
Listener-level mTLS, used when a route doesn't override it
maxConnections
number
0 (unlimited)
Drop new connections when active count reaches this
idleTimeoutMs
number
60000
Close connections silent for this many ms
protection
ProtectionConfig
—
IP-level protection
failoverIngress
{ trustedSources: string[] }
—
Makes this listener the receiving end of peer failover. See Peer failover.
RouteConfig
Field
Type
Default
Description
sni
string
required
Hostname for exact match, or '*.suffix' for wildcard, or '' for default (maximum 255 UTF-8 bytes; no control characters; exact duplicates remain last-wins, later wildcard duplicates are skipped, even when the winning entry is rejected)
metricsGroup
string
—
Stable grouping key for aggregating several domain routes as one tenant (maximum 128 UTF-8 bytes; no control characters)
upstreams
Upstream[]
required
Destination(s); multiple UDS upstreams are load-balanced
terminateTls
boolean
required
true = decrypt TLS; false = TCP passthrough
cert
CertConfig
—
Per-route cert, overrides listener defaultCert
mtls
MtlsConfig
—
Per-route mTLS, overrides listener mtls
suspended
boolean
false
Hold connections and emit 'suspended' events
suspendTimeoutMs
number
30000
Drop held connections after this ms if not resolved
maxConnectionsPerSecond
number
—
Route-wide new-connection rate cap (token bucket). Connections are silently dropped when exhausted.
burst
number
maxConnectionsPerSecond
Token bucket burst ceiling for the route rate limit
http2
boolean
false
Advertise h2 in ALPN so clients negotiate HTTP/2. Raw H2 frames flow through to the upstream unchanged. Requires terminateTls: true.
The route's application protocol. Required to be 'http' before sourceAddressHeader: 'xForwardedFor' or a header-carried forwardFingerprint is accepted — see Source address forwarding.
allowedSources
string[]
— (any client)
Only clients whose TCP peer address is in one of these CIDRs reach the route. See Per-route source allowlist.
failover
FailoverConfig
—
Forward new connections to peer nodes while the local upstream is down, and optionally a share of them while the local Harper threads are much busier than a peer's. See Peer failover and Spill-over.
Upstream
// TCP upstream{ kind: 'tcp', host: string, port: number }// Unix Domain Socket upstream{ kind: 'uds', path: string, ipAffinity?: boolean, // route same-IP connections to same socket ipAffinityTtlMs?: number, // evict affinity entry after this ms idle (default 300000) pid?: number, // Linux PID of the worker process (enables CPU monitoring) tid?: number, // Linux TID of the worker thread (must be set with pid)}
requireClientCert defaults to true. Set to false to accept connections without a client cert while still validating those that do present one.
ProtectionConfig
Field
Type
Default
Description
rateLimit
{ connectionsPerSecond, burst? }
—
Per-second token bucket per source IP
sustained
{ connectionsPerMinute, burst? }
—
Per-minute token bucket per source IP (independent of rateLimit)
penaltyBox
{ durationMs? }
—
Block an IP for durationMs after any rate limit exhaustion
maxConcurrentPerIp
number
0 (unlimited)
Max simultaneous connections per source IP
allowlist
string[]
[]
CIDRs that bypass all checks. IPv4 clients of a dual-stack listener are matched as IPv4 (see below)
blocklist
string[]
[]
CIDRs that are always blocked. IPv4 clients of a dual-stack listener are matched as IPv4 (see below)
ja3Blocklist
string[]
[]
JA3 MD5 hex fingerprints to block (32 chars each)
ja4Blocklist
string[]
[]
JA4 TLS fingerprints to block (36-char strings; see JA4 blocking)
tlsHandshakeTimeoutMs
number
10000
Abort slow TLS handshakes
requireSni
boolean
false
Reject connections without an SNI extension
A dual-stack ([::]) listener reports an IPv4 client as ::ffff:a.b.c.d. Protection treats it as
a.b.c.d: only IPv4 allowlist/blocklist entries match it (::/0 does not; use 0.0.0.0/0
alongside it), it shares rate-limit, penalty and concurrency state with that IPv4 address, and
blockedIps() reports the IPv4 form. An IPv4-mapped entry (::ffff:10.0.0.0/104) is stored as its
IPv4 form (10.0.0.0/8); one with a prefix under 96 is rejected. The 'blocked' event still
reports the address as accepted.
TLS & mTLS
Per-route certificates
Each route can have its own certificate. Routes without a cert use the listener's defaultCert.
const proxy = new SymphonyProxy({ listeners: [{ port: 443, defaultCert: { certChain: wildcardCert, privateKey: wildcardKey }, }], routes: [ // Uses its own cert { sni: 'special.example.com', cert: { certChain: specialCert, privateKey: specialKey }, ... }, // Falls back to listener defaultCert { sni: '*.example.com', ... }, ],});
The default route serves a connection that sends no SNI (including a plain-TCP client on a
passthrough route, which sends no ClientHello at all) and one whose SNI matches no exact or
wildcard route. It is never used for:
a name whose own exact or wildcard route was rejected: that name resolves to nothing;
a failover ingress listener, which closes an unmatched name before the handshake so the forwarding node tries its next peer;
an ACME challenge on an HTTP listener: only a route configured for the Host can answer one, so the default route's upstream can't validate certificates for names it doesn't serve.
requireSni: true still blocks a connection with no SNI before any route is chosen. A failover
policy on the default route is ignored (logged and counted in failingRoutes): peers would refuse
every connection it forwarded.
Suspended routes
Use suspended routes to inspect or authorize connections before proxying them:
Connections not resolved within suspendTimeoutMs are dropped automatically. Calling resolveConnection with an unknown or already-expired ID is a no-op.
HTTP listener (ACME + HTTPS redirect)
Setting mode: 'http' on a listener switches it from the default SNI-routed TLS proxy to a plaintext HTTP/1.1 handler intended for port 80. It serves two purposes:
ACME HTTP-01 challenges — requests under /.well-known/acme-challenge/ are proxied to the route matched by the Host header (using the same wildcard rules as SNI matching). The upstream is the route's first upstream, so a single route table covers both :443 TLS routing and :80 ACME proxying.
HTTPS redirect — every other request gets HTTP/1.1 301 Moved Permanently with Location: https://<host><request-target>.
Requests with no Host header return 400 Bad Request. ACME requests for hosts with no matching route return 404 Not Found.
Active connections are the primary factor; CPU utilisation (0–1000, representing 0–100%) is a tiebreaker that steers new connections away from overloaded threads when connection counts are equal.
If the picked socket refuses the connection (not listening, file missing, accept queue full, or
connect timeout),
symphony tries each remaining socket once, best score first, within the same connect timeout. One
dead worker socket then costs a failed connect instead of the client's connection.
IP session affinity
Add ipAffinity: true to any UDS upstream entry to pin source IPs to the same socket:
The same ipAffinity / ipAffinityTtlMs values apply to all sockets in the set (values from the first entry are used for the shared balancer).
Thread CPU utilisation monitoring
When each UDS upstream serves a known worker thread, symphony can read its CPU utilisation from /proc/{pid}/task/{tid}/stat and incorporate it into socket selection:
Symphony samples /proc/{pid}/task/{tid}/stat every 250 ms and computes the thread's CPU utilisation over the interval. Sockets without pid/tid keep a CPU score of 0 and fall back to pure least-connections. A thread that can no longer be read (process exit, crash) scores 0 until it is found again; symphony never keeps a value for a thread that is gone.
pid/tid may be the ids the worker sees in its own PID namespace, as Harper in a container reports them (process.pid and gettid()). Before sampling a socket, symphony connects to it and closes at once, reads the listening process's host pid from the socket credentials (SO_PEERCRED), and, when that process's NSpid: shows the configured pid as its namespaced pid, maps the configured tid to the host thread the same way. Every 30 s it checks that the socket file is still the same one (its inode) and connects again only when it is not; an unresolved socket (around a restart) is retried every 5 s. All of this runs in the background, so sampling never waits on it. The thread's start time is recorded with it, so a reused id is detected on the next sample and the socket is looked up again. Configured ids that do not name the socket's owner at all are still sampled as given, for socket selection only; they never count toward a route's load (see spill-over).
Per-route rate limiting
Use maxConnectionsPerSecond on a route to cap the rate of new connections accepted for that route, independent of source IP. This prevents a single busy route from starving other routes under high load:
routes: [ { sni: 'api.example.com', maxConnectionsPerSecond: 500, // route-wide cap; burst defaults to this value burst: 1000, // allow short bursts up to 1000 conn/s upstreams: [{ kind: 'uds', path: '/run/app/api.sock' }], terminateTls: true, cert: { certChain, privateKey }, }, { sni: 'admin.example.com', maxConnectionsPerSecond: 20, upstreams: [{ kind: 'uds', path: '/run/app/admin.sock' }], terminateTls: true, cert: { certChain, privateKey }, },]
Connections that exceed the limit are silently dropped (TCP RST). This is a global token bucket per route — not per IP. For per-IP rate limiting use protection.rateLimit.
Per-route source allowlist
allowedSources restricts a route to a set of client addresses. Several tenants can share one
listener and still each have their own list, since the list belongs to the route the SNI selects:
Entries are IPv4 or IPv6 CIDRs; a bare IP is a single host. Absent or empty = any client.
The client address is the TCP peer symphony sees. Inbound PROXY headers and X-Forwarded-For
are never consulted. An IPv4 client on a dual-stack ([::]) listener (::ffff:a.b.c.d) is
matched against the IPv4 entries only, so ::/0 does not admit it; an IPv4-mapped entry
(::ffff:10.0.0.0/104) is treated as its IPv4 form (10.0.0.0/8).
A refused connection is closed (TCP RST) right after route selection: before the TLS handshake,
the upstream connect, the route's maxConnectionsPerSecond bucket and any 'suspended' event.
It is counted as source_not_allowed in the route's and the listener's blockedByReason, and
no 'blocked' event is emitted.
On a suspended route, the route's own list decides whether the connection is suspended at all.
A resolveConnection() route may carry its own allowedSources, checked in addition before the
handshake and upstream connect; an invalid entry there rejects the resolution with an 'error'
event.
updateConfig() changes apply to new connections. The list is checked once, when a connection
is admitted, so established connections are not dropped when their address is removed.
An invalid entry drops the route (it refuses every connection and is counted in failingRoutes)
instead of failing the whole new SymphonyProxy() / updateConfig() call for every other route.
A rejected route never falls through to a wildcard or default route.
The list applies on every listener that resolves the route, including an
HTTP listener's ACME challenge proxy. A list on a route
that answers ACME HTTP-01 challenges therefore also refuses the CA's validators, whose addresses
are not published.
It restricts a route, not the upstream behind it. The client picks the SNI, so a second route
reaching the same upstream (an alias, or a wildcard that also matches) is a way around a list
it doesn't carry — and symphony does not compare the HTTP Host header with the SNI. Give
every route to a restricted upstream the same list.
This is unrelated to protection.allowlist, which exempts listener-wide addresses from protection
checks rather than restricting anything.
An older symphony ignores the field, admitting every client. A config producer should not count
on enforcement until every process accepting on the port runs a version that supports it.
Peer failover
Every node of a cluster serves the same hostname with the same certificate. While a route's local
upstream is down (Harper restarting, crashed, deploying), symphony can send that route's new
connections to another node instead of failing them, so a restart needs no load-balancer change.
// On every node: the public listener, plus an ingress listener that only peers may reach.listeners: [ { port: 443 }, { port: 9443, failoverIngress: { trustedSources: ['10.0.1.0/24'] } },],routes: [{ sni: 'app.example.com', terminateTls: true, cert, upstreams: [{ kind: 'uds', path: '/run/harper/http.sock' }], failover: { peers: [{ host: '10.0.1.12', port: 9443 }, { host: '10.0.1.13', port: 9443 }], // drain: true, // forward every new connection, whatever the local health // probeIntervalMs: 1000, // peerCooldownMs: { min: 5000, max: 60000 }, // spill: {}, // also move load to less-busy peers (see Spill-over below) },}],
When a connection is forwarded. After the ClientHello peek, protection, route lookup,
allowedSources and the route rate limit, and before the TLS handshake: if the route has failover,
is not suspended, and either drain is set or every slot of its local upstream (or of its h2
upstream) is refusing connections. A slot is marked refusing when a connect to it is refused, finds
no socket file, finds its accept queue full, times out, or finds its host or network unreachable or
down (EHOSTUNREACH, ENETUNREACH, ENETDOWN, EHOSTDOWN; a TCP connect to a stopped container's
bridge address fails this way after a few seconds of unanswered ARP). A background probe connects
to each refusing slot every probeIntervalMs and closes at once; a successful connect clears the slot. A routeHealth event
({ route, metricsGroup, status: 'up' | 'down' }) reports each transition. A connect proves the
socket accepts, not that the application behind it is ready.
A passthrough route (terminateTls: false) also forwards a connection when its local connect
fails, on a client listener and unless the route is suspended. Nothing has been read from the client
at that point, so the connection that discovers a dead upstream is served by a peer instead of being
lost. A slot that fails in one of the ways above is marked, and the next local slot is tried first.
Any other connect error, such as EACCES, goes to the peers at once and marks nothing. Either way the
failure counts upstream_connect.
How. The connection is forwarded as raw TLS: the peer's symphony terminates it with its own copy
of the certificate, so client certificates and ALPN are negotiated with the peer. symphony sends a
PROXY v2 header (the real client address) and the ClientHello, then waits for the peer's first byte
before reading anything from the client — the ClientHello stays unread on the client socket until
then. A peer that refuses, times out, or closes without sending a byte received nothing a retry
could duplicate, so the same ClientHello is replayed to the next peer. The peer's first byte commits
the connection to it; nothing after that is retried. Peers are tried round-robin, skipping peers in
cooldown (per route and peer, starting at min, doubling per consecutive failure up to max, reset
by a success), at most once each.
Local upstream down: all attempts share the listener's TLS handshake timeout. If every peer is
cooling down, the one whose cooldown ends first is tried once. If no peer takes the connection it
is closed (exhausted).
Local connect failed (passthrough): the connect and the peer attempts share one TLS handshake
timeout. The connect gets at most half of it, and the peers get whatever it left, as above. After a
drain or spill attempt no peer took, the two split what is left of the timeout, and a peer tried
before may be tried again.
Draining (drain, local upstream up): peers in cooldown are skipped, the attempts share at most
half the handshake timeout (and at most 2 s), and if no peer takes the connection before answering
it is served by the local upstream with what is left of the handshake timeout (localFallbacks).
The ingress listener (failoverIngress) is the receiving end:
A connection from outside trustedSources is closed before any byte is read. An empty list
admits nobody.
A PROXY v2 header is required (v1, malformed or slow headers are refused). Its source address
replaces the TCP peer address for protection, allowedSources and the header sent to the
upstream. Its TLVs are ignored. A PROXY v2 LOCAL header is a peer's
load query when it names routes, and refused
(proxy_header_local) otherwise.
A connection is never forwarded again, and a name with no exact or wildcard route is closed rather
than served by a default route. If the route's local upstream is down or draining, the connection
is closed before the handshake, which the forwarding node treats as "try the next peer".
Limits and security.
Only new connections are forwarded; established ones close with their upstream.
A TLS-terminating route loses the connection that discovers a dead upstream: it connects after
the handshake, which has already read the ClientHello, so there is nothing to replay. Only a failed
connect marks a slot (the probe only clears marks), so on a quiet route that is the first connection
after every outage. A passthrough route forwards that connection instead.
At a peer, a failed local connect on the ingress listener closes with no bytes, which sends the
forwarding node on to its next peer. A suspended route's resolved upstream is not covered either.
On a passthrough route, a TCP upstream slower to accept than half the handshake timeout counts as
down: its connections go to peers and the slot is marked. A probe has 1 s to clear the mark, so the
route stays failed over while connects take longer than that.
A trusted source can assert any client address, which bypasses per-IP limits and
allowedSources. It grants no identity: the peer verifies client certificates itself. On hosts
where containers egress with the host's address, block container traffic to the ingress port.
Peers must be IP literals; symphony does not resolve names.
Roll out failoverIngress on every node before any node's routes gain failover: an older
listener given a forwarded connection reads the PROXY header as TLS and fails it.
An invalid failover serves the route without failover, counted in failingRoutes. An invalid
trustedSources CIDR fails construction or updateConfig().
Both failover (with the routes) and trustedSources (updateConfig({ failoverIngress: [{ port, trustedSources }] })) are hot-swappable; adding or removing failoverIngress on a listener needs a
restart (symphony-server recreates the proxy).
Metrics: each failover route's routeMetrics[].failover carries localUp, refusing/total slots,
replays, budget exhaustion (exhausted), localFallbacks, and per-peer forwarded counts and
failures by reason (refused, timeout, zero_byte_close, error; zero-valued reasons omitted).
Ingress refusals are listener block reasons (untrusted_source, proxy_header_invalid,
proxy_header_local, proxy_header_v1, proxy_header_timeout); a forwarded connection refused
for a down route is the route error locally_down.
On a passthrough route, upstream_connect counts failed local connects, including those a peer then
served. Compare it with forwarded and exhausted to see whether clients were affected. A connection
no peer took counts one upstream_connect, not two.
Spill-over to a less-loaded peer
DNS without load feedback (latency or geoproximity records with no health checks) keeps sending a
busy node its full share. With failover.spill, a route whose local Harper threads are much busier
than the same route's on a peer forwards a share of its new client connections to that peer,
through the same forwarding path as failover, while its own upstream stays up.
failover: { peers: [{ host: '10.0.1.12', port: 9443 }, { host: '10.0.1.13', port: 9443 }], spill: { // highWater: 0.75, // local load at which spilling may start // margin: 0.2, // how far below the local load a peer must be to raise the share // step: 0.05, // change of the share per second, up and down // maxFraction: 0.5, // largest share of new connections spilled // maxPeerRttMs: 20, // only peers this close (measured); default: no limit },},
Load. A route's load is the mean CPU (0–1 of one core) of its worker threads, smoothed over about
3 s, from the thread CPU monitor — so its UDS upstreams need
pid/tid, and only threads tied to their socket's owner count. A route with a TCP upstream, or
whose threads cannot be measured, never spills. With an h2 upstream set, the busier of the two
counts. Per-thread CPU is relative to one core: if an instance runs more threads than it has CPU
quota, it saturates below 1.0, so lower highWater to match.
Peer loads. About every probeIntervalMs (at least 100 ms), each node sends each peer one PROXY v2
LOCAL header on its failoverIngress port, naming the spill routes it shares (custom TLV 0xE3,
up to 64 per query). The peer answers SYMPHONY-LOAD/1 and one line per name: its load in permille
for the exact route configured for that name, or - when it would not take a spilled connection
(no exact route, draining, local upstream down, or load not measurable). The connect time of the
query is the peer's RTT. A report is usable for 3 × probeIntervalMs (at least 3 s); a failed
query leaves it to expire, while an answer of - clears it at once. A peer that does not answer —
including symphony 1.5.0 and earlier, which closes LOCAL headers — is never a target.
The rule. Every 250 ms, and after each answer, the controller takes the least-loaded peer with a
usable report that is not cooling down (and within maxPeerRttMs, when set), and with L the local
load and P that peer's:
condition
share of new connections
L ≥ highWater and P ≤ L − margin
rises by step/s, up to maxFraction
L ≥ highWater − margin/2 and P ≤ L − margin/2
held
otherwise, or L unknown
spilling stops at once; the share decays by step/s
Each new connection spills with that probability. Whatever the share, nothing spills once the local
or peer reading behind the decision is older than its freshness window, so a stalled poll stops
spilling by itself. Because spilling needs the peer at least margin/2 below, two nodes reading each
other's loads correctly cannot spill to each other at once; with delayed readings both can briefly,
and each share then moves at most step per second.
What a spilled connection costs. It is relayed through the busy node for its whole life: an extra
network hop, and cross-region egress if the peer is far. If the peer refuses, times out, or closes
before answering (within half the handshake timeout, at most 1 s), the connection is served locally
with what is left of the handshake timeout, and the peer cools down. After the peer's first
byte the connection is committed, as with failover: a peer that then fails (for example its upstream
went down after its last report) loses that connection. Connections that arrived on an ingress
listener never spill, and only new connections move. Where DNS steering with load feedback exists
(Akamai GTM), it remains the better tool; spill-over complements it for spikes of a few seconds.
Exact-SNI routes only: a wildcard's load is not each matching name's load on a peer, where an
exact route can override it. An invalid spill (or one on a wildcard route) disables only
spill-over, counted in failingRoutes; down and drain failover still apply.
The spilled share, its target, the local load, and each peer's last report and RTT are in
routeMetrics[].failover.spill and .peers[] (spilled, load, rttMs), and on the admin
endpoint as symphony_failover_spill_fraction, symphony_failover_local_load,
symphony_failover_spilled_total, symphony_failover_peer_load and
symphony_failover_peer_rtt_seconds. Load queries answered are the listener counter
loadQueries (symphony_listener_load_queries_total).
Source address forwarding
Use sourceAddressHeader on a route to control how the real client IP is communicated to the upstream. The PROXY protocol carriers work whether or not TLS is terminated (they prefix the connection); 'xForwardedFor' requires terminateTls: true and a plaintext HTTP/1 upstream, since it rewrites the HTTP request.
Value
Behaviour
'proxyProtocol'
Sends a PROXY protocol v1 (text) header (PROXY TCP4 <src-ip> <dst-ip> <src-port> 0\r\n) before any application data. Default for UDS upstreams.
'proxyProtocolV2'
Sends a PROXY protocol v2 (binary) header before any application data. v2 adds a TLV section — the carrier for forwardFingerprint below and for mTLS client cert forwarding. Keep it opt-in: the consumer must speak v2 (nginx/HAProxy do; Harper core's UDS reader parses v1 only before Harper 5.2).
'xForwardedFor'
Reads and rewrites every request on the connection (not just the first), inserting an X-Forwarded-For header after the request line — a pipelined or keep-alive request must be parsed too, or a later request could smuggle a spoofed header past the first-request-only rewrite. Default for TCP upstreams (disabled). Requires protocol: 'http' on the route (below).
'none'
Does not forward source address information. Default for TCP upstreams.
Declaring the route protocol
sourceAddressHeader: 'xForwardedFor', and forwardFingerprint under any mode other than 'proxyProtocolV2', rewrite an HTTP/1 request — they need a route that says it actually carries HTTP. Set protocol: 'http' to opt in:
{ sni: 'app.example.com', upstreams: [{ kind: 'uds', path: '/run/app/worker.sock' }], terminateTls: true, cert: { certChain, privateKey }, sourceAddressHeader: 'xForwardedFor', protocol: 'http', // required — omitting this is a config error, not a silent no-op}
protocol defaults to 'opaque' — a route for a non-HTTP application protocol (MQTT, or any other raw TCP/TLS protocol), limited to the PROXY-protocol carriers, which work on any byte stream. The declaration exists because ALPN can't stand in for it: a native protocol that negotiates no ALPN (MQTT does not) is indistinguishable at the TLS layer from an HTTPS client that simply didn't offer one. A route that requests a header-injection mode without declaring protocol: 'http' is rejected with a descriptive error — it never silently stops injecting the header, since a backend that silently sees the wrong (or no) client IP is worse than a route that refuses to serve traffic at all.
This declaration only helps when a header could actually be injected in the first place. A passthrough route (terminateTls: false) never decrypts the stream, so a header-carried mode has no carrier at all regardless of protocol — declaring 'http' on a passthrough route wouldn't make it work, and is rejected with a distinct "no carrier" error instead of steering you toward a declaration that can't help. Use sourceAddressHeader: 'proxyProtocolV2' there instead (it works on any byte stream, passthrough included).
Like a cert that fails to build, a route rejected for either reason is isolated to just that route: it's dropped (its SNI resolves to nothing) rather than failing new SymphonyProxy() or updateConfig() for every other route on the same port-set — one tenant's config mistake doesn't take the rest down with it. The rejection is still loud: it's logged (symphony: skipping route '<sni>': ...) and counted in failingRoutes. A rejected declaration is dropped on a hot-swap too, since it won't heal until the config is edited. A cert that fails to build on a hot-swap (typically a cert and key rotating as two separate files) instead keeps the route's last-good version serving until the pair is consistent again, but only while the route still asks for the client-certificate policy that version enforces (passthrough, no mtls, or the same clientCaCert bundle and requireClientCert). A route whose policy changed is dropped, so an old verifier never outlives the edit that replaced it.
This is a breaking change for a hand-written route that already uses sourceAddressHeader: 'xForwardedFor' (or a header-carried forwardFingerprint) without protocol: 'http' — add the declaration when upgrading, or that route will stop serving traffic (silently, aside from the log line) rather than failing loudly at startup.
PROXY protocol (default for UDS)
Most backends that consume PROXY protocol (nginx, HAProxy, HarperDB) read the header once per connection before parsing application data.
{ sni: 'api.example.com', upstreams: [{ kind: 'uds', path: '/run/app/worker.sock' }], terminateTls: true, cert: { certChain, privateKey }, // sourceAddressHeader: 'proxyProtocol', // this is already the default for UDS}
X-Forwarded-For (for Bun and other HTTP backends)
Bun's built-in HTTP server does not support PROXY protocol. Use 'xForwardedFor' instead — symphony injects the header into every HTTP request on the connection, not just the first:
symphony computes the client's JA3/JA4 fingerprint from the ClientHello (the same value used for ja3Blocklist/ja4Blocklist). Set forwardFingerprint to also hand it to the upstream, so a backend behind symphony can make its own bot/abuse decisions on it.
forwardFingerprint
Value forwarded
'ja3'
The JA3 MD5 hex (32 chars)
'ja4'
The JA4 fingerprint
['ja3', 'ja4']
Both — e.g. while an existing JA3-keyed blocklist or threat-intel feed is migrated to JA4. List order doesn't matter; JA3 is always emitted first.
'none' (default)
Nothing forwarded
The carrier depends on sourceAddressHeader, and every selected fingerprint uses the same one:
With 'proxyProtocolV2', each fingerprint rides a PROXY v2 TLV — type 0xE0 for JA3, 0xE1 for JA4 (in HAProxy's 0xE0–0xEF private range). This works even in passthrough (terminateTls: false), since the header prefixes the raw TLS bytes. No protocol declaration is needed — the TLV carries it regardless of the route's application protocol.
Otherwise, symphony injects an X-JA3 / X-JA4 HTTP header per selected fingerprint. This requires protocol: 'http' on the route (see Declaring the route protocol) and a plaintext HTTP/1 upstream (terminateTls: true and not http2); it is skipped for HTTP/2 upstreams (use 'proxyProtocolV2' there). For that HTTP/1 case, the client-supplied copy of each forwarded header is stripped so the injected value is authoritative and can't be spoofed — this guarantee does not extend to an h2-negotiated connection on an http2: true route: injection and stripping are both skipped there, so a client-supplied X-JA3/X-JA4 reaches the upstream unmodified. Use 'proxyProtocolV2' wherever h2 is possible.
A config that requests a header-carried forwardFingerprint without declaring protocol: 'http' is rejected (and isolated to just that route) — the same fail-loud rule as sourceAddressHeader: 'xForwardedFor'. A config that requests forwardFingerprint with no viable carrier at all — passthrough (terminateTls: false) without sourceAddressHeader: 'proxyProtocolV2', where there's neither an HTTP request to inject a header into nor a v2 TLV — is also rejected, with a distinct "no carrier" error: no protocol declaration could fix a passthrough route's inability to inject a header, so it isn't steered toward one. A route that could have carried the header but silently won't for some connections — http2: true, where ALPN negotiation is per-connection and some clients may still land on HTTP/1 — logs a startup warning instead, since that outcome isn't guaranteed.
// TLV carrier — works for any upstream that speaks PROXY v2, including passthrough{ sni: 'app.example.com', upstreams: [{ kind: 'tcp', host: '10.0.0.5', port: 8080 }], terminateTls: true, cert: { certChain, privateKey }, sourceAddressHeader: 'proxyProtocolV2', forwardFingerprint: ['ja3', 'ja4'], // 0xE0 and 0xE1 TLVs}// HTTP-header carrier — for HTTP/1 backends that read request headers{ sni: 'app.example.com', upstreams: [{ kind: 'uds', path: '/run/app/worker.sock' }], terminateTls: true, cert: { certChain, privateKey }, sourceAddressHeader: 'xForwardedFor', forwardFingerprint: 'ja3', // upstream reads X-JA3 alongside X-Forwarded-For protocol: 'http',}
Forwarding mTLS client certificates
When symphony terminates TLS on a route with mtls configured, the verified client
certificate would normally be invisible to the upstream. With
sourceAddressHeader: 'proxyProtocolV2' and terminateTls: true, the v2 TLV section
carries the connection's TLS facts, once per connection, before any application data:
TLV
Type
Content
ALPN
0x01
Negotiated ALPN protocol (e.g. h2)
Authority
0x02
SNI hostname from the ClientHello
SSL
0x20
client bit field (0x01 = TLS, 0x02 = client cert presented on this connection), verify (u32 BE, 0 only for a verified cert), and sub-TLVs 0x21 (TLS version) and 0x23 (cipher suite)
A certificate chain is only forwarded when the mtls verifier accepted it — rustls
aborts the handshake otherwise — so an upstream receiving a 0xE2 TLV over a trusted
(e.g. UDS) link can treat it as a verified client identity. A pathological chain that
cannot fit the v2 header's 16-bit length budget is omitted with a warning (the SSL TLV
still signals that a verified cert was presented). On passthrough routes
(terminateTls: false) the TLS facts are unavailable; the header carries the source
address, SNI authority, and any configured fingerprint TLV.
Protection
Recommended starting values for public-facing deployments
Use sustained to enforce a per-minute cap that is independent of the per-second rateLimit bucket. Both buckets are checked on every connection; exhausting either one blocks the connection. This lets you allow short bursts while still capping total volume over longer windows:
protection: { rateLimit: { connectionsPerSecond: 50, burst: 200 }, // allow short bursts sustained: { connectionsPerMinute: 600, burst: 600 }, // but cap at 10/s long-term}
Penalty box
When penaltyBox is configured, exhausting any rate limit places the source IP in a penalty box for durationMs (default 10 minutes). While penalized, all connections from that IP are rejected outright without touching the token buckets — protecting the proxy from re-assembling per-connection state under a sustained attack.
Extension semantics: while an IP is boxed, symphony still debits its token buckets on each connection attempt (to measure whether the attack is continuing). If a bucket is exhausted (the IP is still sending at the excess rate), the penalty deadline is reset to now + durationMs — effectively extending the penalty by a full durationMs from the moment of continued excess. If the IP stops attacking, the buckets refill and debits succeed; the deadline is not extended, so the IP is readmitted once the original deadline expires.
Blocked events from penalized IPs have reason: 'penalty_boxed'. They appear under penaltyBoxed in blockedIps():
const info = proxy.blockedIps();// info.penaltyBoxed — IPs currently in the penalty box
Penalty state is stored on per-IP runtime state and survives a configuration hot-swap. penaltyBox can be added or removed via updateConfig without restarting.
Hot-swap note on active deadlines: changing durationMs via updateConfig affects new penalty stamps immediately, but IPs already in the box retain their current deadline until it expires or is re-stamped by a continued rate-limit hit. There is no retroactive recalculation of existing deadlines.
JA3 blocking
JA3 fingerprints the TLS ClientHello by hashing a canonical string of the version, cipher suites, extensions, elliptic curves, and EC point formats using MD5. That MD5 is by specification, not a security claim — it allows lists of known-bad fingerprints to be compared cheaply. Collect JA3 fingerprints from your logs (available in the blocked event ja3 field) and add known-bad clients:
ja3Blocklist: [ 'e7d705a3286e19ea42f587b344ee6865', // example known-bad scanner]
Limitation: Chrome and other modern browsers randomize the order of ClientHello extensions on each connection, so a single browser can produce many different JA3 hashes. This makes per-browser JA3 blocking unreliable. Use ja4Blocklist where extension-order randomization is a concern.
Upgrade note: earlier versions filtered only one of the 16 GREASE values when computing JA3, so hashes for clients that send other GREASE values (e.g. Chrome) were nonstandard and unstable. JA3 values collected from earlier symphony versions may no longer match and should be re-collected.
JA4 blocking
JA4 is the randomization-resistant successor to JA3. It sorts the cipher and extension lists before hashing, so it produces a stable fingerprint regardless of ClientHello field ordering. Fingerprints are 36-char lowercase ASCII strings in the form t<ver><sni><cc><ec><alpn>_<sha256/12>_<sha256/12>.
Collect JA4 values from the blocked event ja4 field and configure them:
ja4Blocklist: [ 't13d1516h2_8daaf6152771_b186095e22b6', // example Chrome fingerprint]
Matching is case-insensitive. JA4 fingerprints are always emitted as lowercase. A blocklist
entry that isn't a structurally valid JA4 (or JA3) string is rejected at construction rather
than silently installed as an entry that could never match.
Fail-closed reassembly: when a ja3Blocklist or ja4Blocklist is configured, symphony
reassembles the full ClientHello (bounded by size and a short timeout) before fingerprinting.
A connection whose ClientHello can't be fully reassembled — e.g. a client that fragments it to
expose SNI while withholding later extensions — is blocked (blocked reason
incomplete_handshake) rather than allowed on a partial/empty fingerprint. Without enforcement
configured, fingerprinting stays best-effort and an incomplete hello is not blocked.
License scope: Symphony implements core JA4 (TLS client fingerprinting) only. Core JA4 is BSD-licensed. The JA4+ suite of variants (JA4S, JA4H, JA4SSH, etc.) carries a separate FoxIO proprietary license and is not implemented here.
Hot-swapping protection config
Protection config is per-listener and fully hot-swappable via updateConfig. Push a new config atomically to each listener by port — no restart needed, in-flight connections are unaffected:
// Block a new CIDR range without restartingproxy.updateConfig({ protection: [ { port: 443, protection: { blocklist: ['198.51.100.0/24'], // added under attack rateLimit: { connectionsPerSecond: 50, burst: 100 }, requireSni: true, }, }, ],});
The entire ProtectionConfig is replaced atomically (one pointer swap). Any field not included in the new config reverts to its default. Existing per-IP rate-limit token buckets are preserved across a swap; if burst decreases, tokens are capped at the new ceiling on the next refill — no underflow.
Transitions (none→some, some→none): when using symphony-server, adding or removing a protection block in the config file triggers a seamless proxy recreate via SO_REUSEPORT — no bind gap, existing connections unaffected. Only contents-only changes (same presence, different CIDR/rate/etc.) stay on the pure hot-swap path. When calling updateConfig() directly, a listener that was started without protection cannot gain it and returns an error; restart the listener to change protection presence.
Security & compliance
For deployment hardening and security-questionnaire guidance — private key file
handling and rotation, protecting the (hot-reloaded) config file with restrictive
permissions and file-integrity monitoring, why an MD5 dependency appears (JA3
fingerprinting, not a cryptographic use), the current TLS parameters and FIPS/PCI
attestation roadmap, and the shared-responsibility split for DDoS — see
SECURITY.md.
Metrics & monitoring
In-process (proxy.metrics())
const m = proxy.metrics();// Proxy-wide// m.activeConnections — connections being proxied right now// m.blockedConnections — total rejected since start (protection + maxConnections)// m.pendingSuspended — connections held waiting for resolveConnection()// m.suspendedResolved — suspended connections that were resolved with a route// m.suspendedUnresolved — suspended connections that timed out or were rejected// m.routes — routes in the live table, including the default route// m.failingRoutes — rejected cert/protocol/carrier/allowedSources routes, duplicate wildcards, or invalid failover (or failover.spill) policies// Per listener, in configuration orderfor (const l of m.listeners) { // l.address, l.mode ('tls' | 'http') // l.activeConnections, l.accepted // l.bytesReceived — bytes read from clients (client → upstream) // l.bytesSent — bytes written to clients (upstream → client) // Counted where the proxy sees the bytes. On a terminated-TLS route that is the plaintext // stream, and the handshake — which precedes the counter — is excluded. On a passthrough // route the proxy has no plaintext view and forwards wire bytes, so the handshake records // are part of the stream and are counted. // l.blockedByReason — [{ reason: 'rate_limited', count: 12 }, ...] // l.errorsByReason — [{ reason: 'upstream_connect', count: 3 }, ...]}// Per configured route, aggregated across this proxy's listenersfor (const r of m.routeMetrics) { // r.route — configured exact SNI or wildcard pattern; never a client-provided hostname // r.metricsGroup — optional RouteConfig.metricsGroup, or '' // r.activeConnections, r.connections // r.bytesReceived, r.bytesSent // r.blockedByReason — sparse post-route refusals (`source_not_allowed`) // r.errorsByReason — sparse post-route failures; zero-valued reasons are omitted}const blocked = proxy.blockedIps();// blocked.rateLimited — IPs with a depleted per-second or sustained token bucket// blocked.concurrencyLimited — IPs at their maxConcurrentPerIp limit// blocked.cidrBlocklist — the configured static CIDR blocklist// blocked.penaltyBoxed — IPs currently in the penalty box
Every listener reason is reported on every call, including reasons still at zero, so a dashboard series
exists before the first incident rather than appearing mid-outage. l.blocked / l.errors are
summed from the very reason values reported alongside them, and m.blockedConnections from the
listener values in the same snapshot — so a reading taken mid-traffic is internally consistent
rather than only adding up while the proxy is idle. Route blocks and errors are sparse to avoid
multiplying every zero-valued reason by every configured tenant route.
Route counters begin only after resolution succeeds. Listener-level protection blocks,
maxConnections, malformed/no SNI, unmatched routes, and HTTP header failures therefore have no
route series. Exact routes are labeled with their configured SNI, and wildcard routes with the
configured wildcard pattern (*.example.com) rather than the client-provided hostname, keeping
label cardinality bounded by configuration. Counters survive route hot reloads while both sni
and metricsGroup are unchanged; changing the group starts a new series rather than moving
historical counters between tenants.
Removing or rejecting a route removes its series; adding that route again starts fresh counters.
tls_handshake and route_rate_limited are attributed after SNI selection but before client
authentication, so they describe traffic aimed at a route rather than confirmed tenant sessions.
Creating route metric objects and rendering them are linear in configured routes (four base
samples per route); deployments with thousands of routes should size their scrape interval
accordingly.
Block reasons:max_connections, cidr_blocked, ja3_blocked, ja4_blocked,
incomplete_handshake, no_sni, rate_limited, too_many_connections, penalty_boxed,
source_not_allowed. Only source_not_allowed is route-scoped (also in r.blockedByReason and
symphony_route_blocked_total).
symphony_listener_half_closed_connections is a gauge alongside the active one, counting the
subset of active connections whose upstream half has closed. A rising floor there is a retention
leak while it is still measured in connections — see Dead-peer detection.
idle_timeout counts terminations by idleTimeoutMs, which today is a total duration cap
rather than an idleness one — see #34. Busy
long-lived connections land in this bucket, not just quiet ones.
Out-of-process (symphony-server admin endpoint)
When symphony runs as its own process there is no JS API to call, so the server bin can expose
the same numbers over HTTP. Add an admin block to the config file:
Both bindings are optional; give either or both. Omit the admin block entirely and nothing is
exposed. socketPath may be relative to the config file's directory, and is chmodded to
socketMode (default 0o660) after bind. host defaults to 127.0.0.1. Route and group labels
can identify tenants, so keep access scoped to the host collector. Host Manager's standard Docker
bridge does not expose host loopback to tenant containers, but other host-local processes and
downstream metric systems remain inside this endpoint's trust boundary.
Route
Response
GET /metrics
Prometheus text exposition (v0.0.4)
GET /metrics.json
the same snapshot as JSON
GET /health
{ ok, pid, version, ports }
$ curl --unix-socket /run/symphony/admin.sock http://localhost/metrics# HELP symphony_build_info Always 1; the version is carried in the label.# TYPE symphony_build_info gaugesymphony_build_info{version="0.5.0"} 1...symphony_listener_accepted_total{proxy="80,443",listener="0.0.0.0:443",mode="tls"} 148213symphony_listener_blocked_total{proxy="80,443",listener="0.0.0.0:443",mode="tls",reason="rate_limited"} 27symphony_listener_errors_total{proxy="80,443",listener="0.0.0.0:443",mode="tls",reason="upstream_connect"} 4symphony_route_connections_total{proxy="80,443",route="api.example.com",group="tenant-1"} 148000symphony_route_errors_total{proxy="80,443",route="api.example.com",group="tenant-1",reason="upstream_connect"} 3
The proxy label is the port-set of the proxy entry the listener belongs to (each config entry
gets its own route table). Blocked and error counts are only ever emitted with their reason
label — the labelled series sum to the total, so use sum without(reason) rather than looking
for a separate unlabelled metric. Likewise the proxy-wide active-connection gauge is
sum without(listener) (symphony_listener_active_connections).
Route series use route (the configured SNI/pattern) and group (the configured
metricsGroup, or an empty string). Their active/connection/byte values aggregate across all
listeners in that proxy; Host Manager uses one proxy per external port, so its existing proxy
label keeps route traffic separated by port. Route error samples appear only after a reason is
non-zero.
Each appears only while its field is set. A generation can only be a label, so each new value starts
a new series: change it when the config changes, not on a timer.
The endpoint is strictly read-only and best-effort: it never blocks proxying, and a bind failure
is logged and retried every 5s rather than aborting startup. That matters during a version
upgrade, where the incumbent still holds the socket while the replacement is already serving
traffic through SO_REUSEPORT — the successor picks up the admin endpoint once the old process
exits. A socket file left behind by a SIGKILLed process is reclaimed automatically, but only
after a connect probe proves nobody is listening on it.
The Prometheus renderer is internal to the standalone server and is not exported from the
package root: a snapshot carries that process's pid, timestamps, and port-set grouping, which an
embedded consumer would have to synthesise. An embedded proxy has proxy.metrics() directly. If
a caller genuinely needs Prometheus text from an embedded proxy, open an issue — the right shape
is a ProxyMetrics-based renderer, not this one.
Hot config updates
// Replace the entire route table atomically — in-flight connections are unaffected.proxy.updateConfig({ routes: newRoutes,});
What can be hot-swapped: routes (destinations, TLS certs, suspension state, failover), per-listener protection (CIDR allowlist/blocklist, JA3 blocklist, rate limits, sustained rate limits, penalty box, concurrency caps, handshake timeout, requireSni), and a failover ingress listener's trustedSources.
What requires a restart: bind address, port, idle timeout, worker threads. When calling updateConfig() directly, protection presence (None↔Some) cannot change at runtime — the listener must be restarted. The symphony-server bin handles this automatically via seamless recreate.
start() binds every listener before it accepts a connection on any of them. If one can't bind, it
rejects and leaves nothing listening; calling it again retries.
Replacing a running symphony-server
A supervisor upgrades symphony-server by starting the new process on the same --config and
--status paths while the old one still runs; SO_REUSEPORT lets both accept on the same ports.
status.json names the newest live process (pid, version, startedAt, configPath, ports,
and listeners — the host:port of every listener it has bound). The supervisor waits for the
file to name the new process, then sends the old one SIGTERM. A new process that could not bind
a listener the running one has (a listener cert it can't read, say) does not take the file, so a
supervisor never retires the only process serving that listener; host-manager reads that as a
failed upgrade and keeps the old process.
If the old process is not stopped (a handoff that failed partway, or a supervisor that restarted
during it), it stops itself: once a newer process has held status.json for 10 seconds, serving
the same config on every listener the old process has, the old process runs its normal SIGTERM
stop and exits. Each second it checks again, and it starts the 10 seconds over if the newer
process exits, stops naming itself in the file, or stops covering one of its listeners. It never
exits for a process it can't verify through /proc, so on platforms without /proc it waits for
its supervisor. A process that stops never deletes status.json.
Only a process with this behavior stops itself. A symphony-server from 1.4.0 or earlier that is
already running keeps running until something stops it, so clearing out processes left over from
earlier upgrades needs the supervisor (or an operator) to stop them once.
Knowing which config symphony-server applied
reloadedAt in status.json says when a reload finished. It does not say which config that reload
read, or whether it applied. To find out, set a top-level generation in the config:
generation is an opaque string the writer chooses: a counter or a hash of the content. Use a new
value whenever the content changes, because symphony compares nothing and orders nothing; it only
reports back the string it read. A value that isn't a string is logged and treated as no generation,
and the rest of the config still applies.
Each reload then writes these to status.json:
Field
Meaning
appliedGeneration
The generation of the last config in which every proxy entry applied. It comes from the config that reload read, so a reload that read the previous config and fin