nacelle documentation

nacelle is an experimental Tokio-based Rust library for streaming application handlers across TCP and HTTP transports.

This book is the narrative documentation site. It follows the same broad delivery model as the Rust Book: chapter-oriented Markdown, search, keyboard navigation, and a local/offline build. Its content is organized like Django's documentation:

Tutorials take you through a working path.
Topic guides explain concepts and design choices.
How-to guides solve specific operational tasks.
Reference pages document exact behavior and APIs.

Rust API reference is still generated separately with cargo doc.

Start here

If you are new to nacelle, read:

If you are validating performance, read:

Internal readiness plans and assessments live under docs/internal and are not part of this public book.

Getting started

This tutorial gets a minimal TCP service running with the reference length-delimited protocol.

Add nacelle

In this workspace, the umbrella crate is nacelle. It re-exports the transport crates and owns the reference protocol:

#![allow(unused)]
fn main() {
use nacelle::prelude::*;
}

Build a handler

Handlers receive a NacelleRequest and return a NacelleResponse.

#![allow(unused)]
fn main() {
let handler = handler_fn(|mut request: NacelleRequest| async move {
    while let Some(chunk) = request.body.next_chunk().await {
        let _ = chunk?;
    }

    Ok(NacelleResponse::tcp_bytes("ok"))
});
}

Start the app

#![allow(unused)]
fn main() {
let addr = "127.0.0.1:8080".parse().map_err(NacelleError::protocol)?;
let protocols = NacelleProtocols::new()
    .tcp::<FrameRequest, _>("echo", addr, LengthDelimitedProtocol);

NacelleApp::new(handler)
    .with_telemetry(NacelleTelemetry::default())
    .with_ctrl_c_shutdown()
    .serve(protocols)
    .await?;
Ok::<(), NacelleError>(())
}

Next steps

Run cargo run --features reference_protocol --example app_core to see one app core served through multiple protocol adapters.
Read the architecture guide to understand the request path.
Read runtime limits and backpressure before raising connection counts.
Use Run the stress harness to validate a local build.

Run the stress harness

The stress harness has two binaries:

nacelle-stress-server, from nacelle-stress-server
nacelle-stress-test, from nacelle-stress-test

Run the convenience script:

./examples/run-stress-test.sh

The script reads root config.toml by default. Pass --config to select a repeatable benchmark profile. If the effective tls_self_signed value is true, it passes --tls-insecure to the client so the server and client speak the same transport.

For a plain TCP baseline, use examples/nacelle-stress-server/configs/tcp.toml.

For full details, see the how-to guide:

Run the server:

cargo run --release --package nacelle-stress-server -- --config examples/nacelle-stress-server/configs/tcp.toml

Run a bounded client smoke test:

cargo run --release --package nacelle-stress-test -- --connections 32 --pipeline 16 --duration-secs 15

The examples/run-stress-test.sh and examples/run-stress-test.ps1 helpers accept --config/-Config and pass --tls-insecure to the stress client only when the effective tls_self_signed value is true.

The stress client enables its Rustls support by default so --tls-insecure works with the local self-signed server. For a Rustls-free plain TCP build, run both stress binaries with --no-default-features and use examples/nacelle-stress-server/configs/tcp.toml.

Repeatable profiles:

examples/nacelle-stress-server/configs/tcp.toml: plain TCP baseline.
examples/nacelle-stress-server/configs/tcp-low-memory.toml: plain TCP with mimalloc low-memory behavior enabled.
examples/nacelle-stress-server/configs/tcp-tls.toml: TCP wrapped in self-signed TLS.

Linux example:

./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp.toml --server-threads 48 --connections 256 --pipeline 8 --duration-secs 30 --payload-bytes 256

PowerShell example:

.\examples\run-stress-test.ps1 -Config examples/nacelle-stress-server/configs/tcp.toml -ServerThreads 48 -Connections 256 -Pipeline 8 -DurationSecs 30 -PayloadBytes 256

OpenTelemetry metrics are enabled in the default stress server build. That build prints a compact OTel console snapshot every 5 seconds and enables request started/completed counters plus request/response byte counters by default. The generic telemetry API groups those switches under request_metrics; the stress server exposes byte accounting as byte_metrics = true. Use --no-byte-metrics for a lower-overhead OTel run, or use --no-default-features with the plain TCP config when you intentionally want a metrics-free peak throughput baseline.

The Tokio stress server default build includes tls-self-signed support. The checked-in root config.toml enables tls_self_signed = true, so the local stress client should use --tls-insecure with that default config. Use --no-default-features with examples/nacelle-stress-server/configs/tcp.toml when you need a Rustls-free plain TCP baseline.

CI-friendly scenarios should stay short and deterministic:

baseline echo throughput
max connection cap
max request cap
slow reader
slow writer
graceful shutdown under load

Heavy RPS and soak tests should run manually or nightly on dedicated Linux hosts.

Serve TCP with self-signed TLS

Use tls-self-signed for local load tests and auto-deploy flows that need a certificate immediately. It implies the rustls provider.

#![allow(unused)]
fn main() {
use nacelle::{
    FrameRequest, LengthDelimitedProtocol, NacelleTlsConfig, TcpServer,
    handler_fn,
};

let generated = NacelleTlsConfig::self_signed(["localhost", "127.0.0.1"])?;
let server = TcpServer::<FrameRequest, ()>::builder()
    .protocol(LengthDelimitedProtocol)
    .handler(handler_fn(|request| async move {
        Ok(nacelle::NacelleResponse::tcp(request.body))
    }))
    .build()?;

server
    .serve_tcp_tls("127.0.0.1:8443".parse()?, generated.tls_config)
    .await?;
Ok::<(), nacelle::NacelleError>(())
}

Self-signed certificates are for local and automated test flows. Public edge deployments should use managed certificate material and a documented rotation process.

For OpenSSL-backed TCP TLS, enable openssl and use NacelleOpenSslConfig::from_pem_files(...) with serve_tcp_openssl(...). Use openssl-vendored only when the build machine has the tooling needed to compile OpenSSL from source. The openssl feature enables provider-neutral tls without selecting Rustls.

How the documentation is organized

This book uses four kinds of documentation:

Tutorials are guided paths. They assume little context and aim for a working result.
Topic guides explain how nacelle works and why it is shaped the way it is.
How-to guides are recipes for specific tasks.
Reference pages are precise descriptions of behavior, APIs, and protocol contracts.

Use tutorials when you are new, topic guides when you need a model of the system, how-to guides when you have a concrete job, and reference pages when you need exact details.

Architecture

Nacelle is organized as a small core plus protocol-specific transport crates.

Crate Layout

nacelle-core: shared handler, request/response body, limits, lifecycle, telemetry, and TLS primitives.
nacelle-tcp: TCP/Unix socket server, protocol trait, connection loop, and listener runtime.
nacelle-http: Hyper HTTP/1 server, HTTP request policy, and HTTP TLS listener integration.
nacelle: convenience crate that re-exports the split crates and owns the reference length-delimited protocol.

The reference protocol intentionally stays out of nacelle-core and nacelle-tcp; it is a batteries-included implementation exported by the umbrella nacelle crate.

App Core And Protocol Adapters

Nacelle is organized so application behavior lives behind the Handler boundary. A handler receives a transport-neutral NacelleRequest and returns a NacelleResponse.

TCP Protocol implementations are adapters: they decode a wire format into request metadata and encode responses back into frames. Swapping protocols should not require rewriting the app core. The app-first serving path wires those pieces together with NacelleApp, NacelleProtocols, and NacelleApp::serve(...); lower-level TcpServer and NacelleHost APIs remain available for services that need direct listener control.

TLS lives in nacelle-core because the configuration and provider metadata are shared. tls is provider-neutral. rustls enables the Rustls provider used by HTTP and TCP. openssl enables the OpenSSL provider for TCP without selecting Rustls. Both providers feed NacelleTlsProvider and per-connection TLS metadata.

Request Flow

listener
  -> connection limit
  -> connection task
  -> protocol/HTTP decode
  -> request limit
  -> handler
  -> response body encode/stream

TCP and Unix socket listeners use the nacelle-tcp Protocol<Req> trait to decode request heads and encode response frames. HTTP uses nacelle-http with Hyper HTTP/1 and maps requests into the same NacelleRequest / NacelleResponse shape.

NacelleRequest::connection carries transport, a stable connection id, peer address, local address, local Unix socket path, effective peer IP, TLS metadata, and an optional typed extension. Raw protocol servers can populate that extension with connection_extension_factory(...) for auth/session state derived at accept or handshake time. Apps using serve(protocols, app) can set the same extension factory on NacelleApp.

HTTP-specific edge policy remains in nacelle-http: Host, method, URI/header shape checks, per-peer request rate limits, access logging, and security header injection. TCP keeps protocol semantics in the protocol implementation and shared lifecycle/limit enforcement in core.

Runtime State

NacelleRuntimeState owns shared budgets and counters. Connection, request, and streaming-task limits are non-blocking atomic bounded counters. Memory uses a checked allocation guard that releases on drop.

This keeps the common request path allocation-light while still enforcing bounded defaults.

Bodies

NacelleBody has three internal shapes:

empty/single chunk for fast small responses
buffered chunks for decoded TCP bodies already in memory
streaming channel for request/response bodies that move asynchronously

TCP large request bodies reserve their declared length while streaming. HTTP request bodies reserve Content-Length when Hyper exposes a bounded size hint. TCP protocols can override RequestMetadata::max_body_bytes(...) to choose a phase-aware body limit immediately after head decoding and before body buffering or streaming begins.

Shutdown

Listeners own a JoinSet of accepted connection tasks. Shutdown proceeds in stages:

signal shutdown
stop accepting
drain active connection tasks
abort remaining tasks after the drain deadline
emit shutdown telemetry

Task tracking is at the connection boundary, not the per-request hot path.

Observability

Telemetry is deliberately low-cardinality. Reasons are static strings such as connections, request_body_bytes, or http_body_read.

With otel, runtime gauges are observable instruments backed by runtime-state atomics, so collection reads current values without per-request metric writes. NacelleTelemetry owns lifecycle, request, phase, error, and byte metrics for all transports. Transports that can provide extra low-cardinality detail attach a NacelleMetricsContext with listener, protocol, transport, and TLS labels.

Request metric switches live under NacelleTelemetryConfig::request_metrics. Started/completed counters and byte counters are on by default; in-flight counters, duration histograms, and phase histograms are disabled by default. Enable them deliberately with NacelleTelemetry::default() builder methods on the server or app when you need diagnostic detail and can afford the extra per-request metric writes. Core/HTTP request paths do not start a request timer unless duration metrics or HTTP access logging are enabled.

Runtime limits and backpressure

Runtime limits are enforced through NacelleRuntimeState. They are intended to make overload predictable rather than perfectly invisible.

Key budgets include:

active connections
in-flight requests
streaming body tasks
optional per-peer connections
memory budget allocations
request and response body size
core handler timeout
TCP read, write, and idle timeouts through NacelleTcpLimits
HTTP header, body, write, keep-alive, and connection-age limits through NacelleHttpLimits
TLS handshake timeouts through the TLS config types

The important production habit is to size limits together. A high connection count with large read and response buffers is a memory budget decision, not just a concurrency decision.

For configuration details:

Start from NacelleLimits::default() and tune shared resource budgets for the deployment. Use NacelleTcpLimits for TCP socket timeouts and NacelleHttpLimits for HTTP edge timeouts and keep-alive behavior. Active connections, in-flight requests, streaming tasks, body sizes, handler timeouts, and transport timeouts are bounded by default. Memory allocation budgeting is opt-in: the default max_memory_bytes is usize::MAX, which disables Nacelle memory-budget enforcement until you set an explicit byte limit.

Recommended presets:

Internal service: keep defaults, set body limits to the largest expected payload, and run behind process supervision.
Internet-facing behind proxy: cap connections and requests to the container budget, keep 30 second transport timeouts, and let the proxy own coarse traffic filtering or certificate automation when desired.
Proxy-aware HTTP: configure NacelleHttpPolicy::with_trusted_proxy_ips(...) only with known proxy addresses before allowing Forwarded or X-Forwarded-For to affect per-peer request limits or request metadata.
Direct HTTPS listener: enable http,tls, load certificate/key material through NacelleTlsConfig, configure an SNI allowlist with from_pem_with_allowed_server_names or from_der_with_allowed_server_names, set a short TLS handshake timeout, configure max_connections_per_peer and max_connection_opens_per_peer_per_second, enable HTTP access logs, and attach NacelleHttpPolicy with Host, method, URI, header, security-header, and per-peer request-rate limits.
Direct TCP Rustls listener: enable tcp,tls, load certificate/key material through NacelleTlsConfig, use serve_tcp_tls or enable_tcp_tls, and keep protocol-level authentication/authorization in the application protocol.
Direct TCP OpenSSL listener: enable tcp,openssl, load certificate/key material through NacelleOpenSslConfig, use serve_tcp_openssl, enable_tcp_openssl, or NacelleProtocols::tcp_openssl, and configure the SslAcceptor yourself when you need OpenSSL-specific policy.
Local load-test/autodeploy HTTPS: enable tls-self-signed and call NacelleTlsConfig::self_signed(...); do not treat generated certificates as a public trust or rotation strategy.
High concurrency: reduce TCP buffer capacities before raising max_connections, and tune NacelleTcpLimits separately from shared resource budgets.

Memory budget:

connection_budget =
  max_connections * (read_buffer_capacity + response_buffer_capacity)
body_budget =
  concurrent_buffered_or_streaming_bodies * max_request_body_bytes
total_budget =
  connection_budget + body_budget + handler/backend/runtime headroom

When memory limiting is enabled with NacelleLimits::with_max_memory_bytes(...), Nacelle allocates from that budget for connection buffers and buffered or streaming request bodies. The limiter accounts for Nacelle-managed allocations, not total process RSS, so keep process or container memory limits in place. Request body allocations wait in FIFO order when the budget is full. The default wait limit is NacelleLimits::memory_allocation_timeout == Some(5s), and can be tuned with with_memory_allocation_timeout(...) or disabled with without_memory_allocation_timeout(). A timed-out waiter returns NacelleError::Timeout("memory_allocation").

The memory budget is an accounting guard, not a buffer allocator: it grants a NacelleMemoryAllocation that tracks bytes the transport or application intends to hold elsewhere, and releases those bytes when the guard is dropped. Applications can allocate from the same budget through NacelleRuntimeState::memory_budget(). Use try_allocate(...) for immediate admission, allocate(...) for FIFO waiting, or allocate_with_timeout_and_shutdown(...) when app work should stop waiting during shutdown.

TCP processes requests sequentially per connection. request_body_channel_capacity controls the queued streaming chunks between the socket reader and handler. HTTP uses Hyper's internal buffers plus Nacelle's body queue, so leave extra headroom when enabling large request bodies.

For TCP protocols, NacelleLimits::max_request_body_bytes is the default body limit. Custom request metadata can override RequestMetadata::max_body_bytes(connection, default_limit) to choose a per-request limit after head decoding and before body buffering or streaming. This is useful for phase-aware protocols that keep connection auth state in a typed connection extension and need a smaller unauthenticated body cap.

Dangerous configurations:

unbounded connections with large per-connection buffers
large body limits without a process/container memory limit
disabled timeouts on internet-facing listeners
direct internet-facing HTTP without Host/header/method/URI policy
direct internet-facing TLS without an SNI allowlist
direct internet-facing listeners without per-peer connection caps
direct internet-facing listeners without per-peer connection-open rate caps
direct internet-facing HTTP without per-peer request caps and access logs
trusting forwarded peer headers without an explicit trusted proxy list
generated self-signed certificates used as a long-lived public-edge certificate strategy
high keep-alive connection counts without proxy-level idle limits

TLS certificate rotation:

#![allow(unused)]
fn main() {
let tls = NacelleTlsConfig::from_pem_files("cert.pem", "key.pem")?;
tls.reload_from_pem_files("next-cert.pem", "next-key.pem")?;
}

Reloads affect new TLS handshakes. Existing connections continue with the configuration negotiated when they connected.

Operations model

Deployment Shape

Recommended internet-facing shape:

client -> proxy/load balancer/TLS -> Nacelle service

The proxy should own TLS, coarse connection filtering, and external idle timeouts. Nacelle owns application limits, protocol handling, body limits, and graceful shutdown.

Startup

Use explicit limits and print the effective config for stress or benchmark services. For production services, record:

process version and git SHA
configured limits
listener addresses
feature flags
allocator settings

Shutdown

Wire OS signals to NacelleHost::shutdown_and_wait_timeout. Pick a drain deadline that matches service semantics. Short deadlines protect deploy velocity but can abort in-flight work.

Expected shutdown telemetry:

shutdown requested
listener stopped accepting
drain started
drain completed or timed out
active connections aborted

Metrics To Watch

nacelle.connections.active
nacelle.requests.active
nacelle.streaming_tasks.active
nacelle.memory.used_bytes
nacelle.connections.accepted
nacelle.connections.closed
nacelle.connections.in_flight
nacelle.requests.started
nacelle.requests.completed
nacelle.rejections
nacelle.timeouts
nacelle.requests.failed
nacelle.request.bytes
nacelle.response.bytes

Alerts should focus on sustained saturation, rising rejections, timeout spikes, and memory approaching the configured budget.

Benchmarking

The default OpenTelemetry profile keeps lifecycle metrics on. Request metrics are grouped under NacelleTelemetryConfig::request_metrics: started, completed, and byte_counts are on by default, while in_flight, duration_ms, and phase histograms are opt-in. The stress server's default OTel build prints a compact console snapshot every 5 seconds.

Request duration metrics remain opt-in through NacelleTelemetryConfig. With the default config, core/HTTP request paths avoid request timer work unless HTTP access logging is enabled.

Canonical OpenTelemetry metric names are resource-first. Instrument type is documented here rather than embedded in the metric name:

Metric	Type	Notes
`nacelle.connections.active`	Gauge	Current runtime active connections.
`nacelle.requests.active`	Gauge	Current runtime active requests.
`nacelle.streaming_tasks.active`	Gauge	Current runtime streaming body tasks.
`nacelle.memory.used_bytes`	Gauge	Current bytes allocated by runtime memory accounting.
`nacelle.connections.accepted`	Counter	Accepted connections, labeled by listener/transport/TLS where available.
`nacelle.connections.closed`	Counter	Closed connections, labeled with close reason where available.
`nacelle.connections.in_flight`	UpDownCounter	Per-listener connection delta for transport-level detail.
`nacelle.requests.started`	Counter	Requests started.
`nacelle.requests.completed`	Counter	Requests completed, labeled by status where available.
`nacelle.requests.failed`	Counter	Requests failed before normal completion.
`nacelle.request.bytes`	Counter	Request bytes accounted by the transport/protocol path.
`nacelle.response.bytes`	Counter	Response bytes accounted by the transport/protocol path.
`nacelle.request.duration_ms`	Histogram	Request duration, opt-in.
`nacelle.phase.duration_ms`	Histogram	Internal phase duration, opt-in.

Run microbenchmarks before and after hot-path changes:

cargo bench -p nacelle --features bench,reference_protocol

Performance model

nacelle's high-throughput TCP path is sensitive to small per-request costs. When comparing runs, keep these variables fixed:

commit
Linux kernel and CPU governor
allocator configuration
server threads
connection count
pipeline depth
payload size
TLS versus plain TCP
stress client version

Use the performance how-to for repeatable command lines.

The current main branch has been observed around 1.9M RPS on Linux for the TCP benchmark path. This branch should be compared against that baseline on the same host, kernel, CPU governor, allocator settings, and command line.

Suggested local benchmark:

cargo bench -p nacelle --features bench,reference_protocol

The runtime_limits benchmark group covers connection/request permit acquire/drop and memory allocation overhead. Watch it closely after changes to NacelleRuntimeState.

Suggested RPS comparison:

./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp.toml --server-threads 48 --connections 256 --pipeline 8 --duration-secs 30 --payload-bytes 256

The default stress server build also prints a compact OTel console snapshot every 5 seconds. Request metrics are grouped under the generic telemetry request_metrics config; started/completed counters and byte counters are on by default, while in-flight and duration metrics remain opt-in. Request duration metrics are opt-in as well, which avoids request Instant work on core/HTTP paths unless duration metrics or HTTP access logs are enabled. Use --no-byte-metrics when comparing the cost of byte accounting, and use --no-default-features with the plain TCP config for a metrics-free baseline.

The checked-in root config.toml enables self-signed TCP TLS for local stress runs. For the plain TCP throughput baseline, use examples/nacelle-stress-server/configs/tcp.toml. Compare TLS and non-TLS runs separately:

./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp.toml
./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp-low-memory.toml
./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp-tls.toml

The examples/run-stress-test.sh and examples/run-stress-test.ps1 helpers apply root config.toml first, then the selected profile, and choose the matching client mode automatically.

Guardrails:

keep shutdown task tracking at the connection/listener boundary
avoid per-request locks in the TCP hot path
keep telemetry sinks optional; default operation should not push into in-memory sinks
preserve single-chunk body fast paths
tune TCP buffer sizes for the connection count instead of relying on large defaults

Configure production limits

Start from NacelleLimits::default() and tune shared resource budgets for the deployment. Use NacelleTcpLimits for TCP socket timeouts and NacelleHttpLimits for HTTP edge timeouts and keep-alive behavior. Active connections, in-flight requests, streaming tasks, body sizes, handler timeouts, and transport timeouts are bounded by default. Memory allocation budgeting is opt-in: set max_memory_bytes only after measuring that the limiter behaves correctly for your service.

Recommended presets:

Internal service: keep defaults, set body limits to the largest expected payload, and run behind process supervision.
Internet-facing behind proxy: cap connections and requests to the container budget, keep 30 second transport timeouts, and let the proxy own coarse traffic filtering or certificate automation when desired.
Proxy-aware HTTP: configure NacelleHttpPolicy::with_trusted_proxy_ips(...) only with known proxy addresses before allowing Forwarded or X-Forwarded-For to affect per-peer request limits or request metadata.
Direct HTTPS listener: enable http,tls, load certificate/key material through NacelleTlsConfig, configure an SNI allowlist with from_pem_with_allowed_server_names or from_der_with_allowed_server_names, set a short TLS handshake timeout, configure max_connections_per_peer and max_connection_opens_per_peer_per_second, enable HTTP access logs, and attach NacelleHttpPolicy with Host, method, URI, header, security-header, and per-peer request-rate limits.
Direct TCP Rustls listener: enable tcp,tls, load certificate/key material through NacelleTlsConfig, use serve_tcp_tls or enable_tcp_tls, and keep protocol-level authentication/authorization in the application protocol.
Direct TCP OpenSSL listener: enable tcp,openssl, load certificate/key material through NacelleOpenSslConfig, use serve_tcp_openssl, enable_tcp_openssl, or NacelleProtocols::tcp_openssl, and configure the SslAcceptor yourself when you need OpenSSL-specific policy.
Optional TCP OpenSSL listener: enable tcp,openssl and use serve_tcp_optional_openssl(...) or the matching host/app builder method when one listener must accept both plain and TLS clients; keep NacelleTlsDetectionOptions::timeout short enough to avoid tying up idle accepted connections.
IPv4 plus IPv6 TCP bind: use the NacelleProtocols::*_dual_stack(...) helpers when a serve-based app should bind both wildcard families for one protocol. The helpers register separate IPv4 and IPv6 listeners and force the IPv6 listener to v6-only mode.
Unix socket listener: enable tcp on Unix and call serve_unix(...) or NacelleHost::enable_unix_socket(...); use NacelleUnixSocketOptions only when this process owns stale-path cleanup or socket-file permissions.
Local load-test/autodeploy HTTPS: enable tls-self-signed and call NacelleTlsConfig::self_signed(...); do not treat generated certificates as a public trust or rotation strategy.
High concurrency: reduce TCP buffer capacities before raising max_connections, and tune NacelleTcpLimits separately from shared resource budgets.

Memory budget:

connection_budget =
  max_connections * (read_buffer_capacity + response_buffer_capacity)
body_budget =
  concurrent_buffered_or_streaming_bodies * max_request_body_bytes
total_budget =
  connection_budget + body_budget + handler/backend/runtime headroom

Set NacelleLimits::with_max_memory_bytes(...) when you want Nacelle to enforce the calculated budget. Without an explicit memory limit, Nacelle still enforces connection/request/body limits and transport-owned timeouts but leaves total memory governance to the application, runtime, process supervisor, or container. When the memory budget is full, request body allocations wait in FIFO order and time out after NacelleLimits::memory_allocation_timeout (5s by default). Tune this with with_memory_allocation_timeout(...), or call NacelleRuntimeState::memory_budget() when application code needs to allocate from the same budget as the transports.

For TCP protocols, NacelleLimits::max_request_body_bytes is the default body limit. Override RequestMetadata::max_body_bytes(connection, default_limit) when the decoded request head and connection extension state should choose a stricter phase-specific cap before Nacelle buffers or streams the body.

Use NacelleTcpOptions for accepted TCP stream behavior. Defaults preserve the existing behavior: TCP_NODELAY enabled and TCP keepalive disabled. Enable keepalive deliberately per deployment target because OS defaults and supported fields vary. NacelleTcpBindOptions adds listener bind controls such as IPv6-only mode for APIs that need explicit family behavior.

Use NacelleTcpLimits for TCP socket read, socket write, and idle timeouts. Use NacelleHttpLimits on HyperServer for HTTP header read, request body read, response write, keep-alive, and max connection age behavior.

Dangerous configurations:

unbounded connections with large per-connection buffers
large body limits without a process/container memory limit
disabled timeouts on internet-facing listeners
direct internet-facing HTTP without Host/header/method/URI policy
direct internet-facing TLS without an SNI allowlist
direct internet-facing listeners without per-peer connection caps
direct internet-facing listeners without per-peer connection-open rate caps
direct internet-facing HTTP without per-peer request caps and access logs
trusting forwarded peer headers without an explicit trusted proxy list
generated self-signed certificates used as a long-lived public-edge certificate strategy
high keep-alive connection counts without proxy-level idle limits
long TLS detection timeouts on optional TLS listeners
Unix stale-path cleanup for a socket path not exclusively owned by this process

TLS certificate rotation:

#![allow(unused)]
fn main() {
let tls = NacelleTlsConfig::from_pem_files("cert.pem", "key.pem")?;
tls.reload_from_pem_files("next-cert.pem", "next-key.pem")?;
}

Reloads affect new TLS handshakes. Existing connections continue with the configuration negotiated when they connected.

Run stress tests

Run the server:

cargo run --release --package nacelle-stress-server -- --config examples/nacelle-stress-server/configs/tcp.toml

Run a bounded client smoke test:

cargo run --release --package nacelle-stress-test -- --connections 32 --pipeline 16 --duration-secs 15

Repeatable profiles:

examples/nacelle-stress-server/configs/tcp.toml: plain TCP baseline.
examples/nacelle-stress-server/configs/tcp-low-memory.toml: plain TCP with mimalloc low-memory behavior enabled.
examples/nacelle-stress-server/configs/tcp-tls.toml: TCP wrapped in self-signed TLS.

Linux example:

./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp.toml --server-threads 48 --connections 256 --pipeline 8 --duration-secs 30 --payload-bytes 256

PowerShell example:

.\examples\run-stress-test.ps1 -Config examples/nacelle-stress-server/configs/tcp.toml -ServerThreads 48 -Connections 256 -Pipeline 8 -DurationSecs 30 -PayloadBytes 256

CI-friendly scenarios should stay short and deterministic:

baseline echo throughput
max connection cap
max request cap
slow reader
slow writer
graceful shutdown under load

Heavy RPS and soak tests should run manually or nightly on dedicated Linux hosts.

Compare performance profiles

Use separate profiles for each transport mode:

plain TCP
TCP with low-memory allocator behavior
TCP with TLS
HTTP

Do not compare TLS and non-TLS runs as if they measure the same path. Likewise, do not compare two runs if the stress client version changed.

Recommended plain TCP baseline config:

examples/nacelle-stress-server/configs/tcp.toml

Then run:

./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp.toml --connections 256 --pipeline 8 --duration-secs 30 --payload-bytes 256

More background:

Suggested local benchmark:

cargo bench -p nacelle --features bench,reference_protocol

The runtime_limits benchmark group covers connection/request permit acquire/drop and memory allocation overhead. Watch it closely after changes to NacelleRuntimeState.

Suggested RPS comparison:

./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp.toml --server-threads 48 --connections 256 --pipeline 8 --duration-secs 30 --payload-bytes 256

./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp.toml
./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp-low-memory.toml
./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp-tls.toml

The examples/run-stress-test.sh and examples/run-stress-test.ps1 helpers apply root config.toml first, then the selected profile, and choose the matching client mode automatically.

Guardrails:

keep shutdown task tracking at the connection/listener boundary
avoid per-request locks in the TCP hot path
keep telemetry sinks optional; default operation should not push into in-memory sinks
preserve single-chunk body fast paths
tune TCP buffer sizes for the connection count instead of relying on large defaults

Harden HTTP listeners

Nacelle's HTTP transport is Hyper HTTP/1. Configure HTTP timeout and keep-alive behavior through NacelleHttpLimits, shared body-size budgets through NacelleLimits, and request-shape policy through NacelleHttpPolicy.

Defaults:

NacelleHttpLimits::header_read_timeout: 30 seconds, enforced with Hyper's HTTP/1 header timeout and TokioTimer.
NacelleHttpLimits::request_body_read_timeout: 30 seconds, enforced while reading body frames.
NacelleHttpLimits::response_write_timeout: 30 seconds, enforced at Hyper's I/O write boundary.
NacelleHttpLimits::keep_alive: enabled.
NacelleHttpLimits::max_connection_age: disabled by default.
request and response body size limits: 16 MiB each.

NacelleHttpPolicy can reject requests before the handler runs:

allowed Host headers
allowed HTTP methods
maximum URI length
maximum header count
maximum aggregate header bytes
optional per-peer request rate limits through with_max_requests_per_peer_per_second
optional trusted proxy forwarded address handling through with_trusted_proxy_ips
optional security headers through with_security_header(...) or with_default_security_headers()
optional per-peer connection caps through NacelleLimits::with_max_connections_per_peer
optional per-peer connection-open rate caps through NacelleLimits::with_max_connection_opens_per_peer_per_second

Rejected requests receive deterministic HTTP responses where the request parser has already accepted the request: 405, 414, 421, 429, or 431. Rejections emit low-cardinality telemetry reasons such as host, method_not_allowed, uri_too_long, header_count, header_bytes, and peer_rate.

Enable the rustls feature to terminate HTTP over TLS. NacelleTlsConfig loads PEM certificate/key pairs, accepts explicit Rustls ServerConfig values, supports reloads for future handshakes, and enforces a TLS handshake timeout. Enable tls-self-signed only when local load tests or auto-deploying applications need to generate a self-signed certificate immediately; it implies rustls.

For direct edge HTTPS, build TLS config with NacelleTlsConfig::from_pem_with_allowed_server_names(...) or NacelleTlsConfig::from_der_with_allowed_server_names(...). When an SNI allowlist is configured, clients that omit SNI or send a name outside the list fail during the TLS handshake. HTTP Host policy is enforced after the handshake, so configure the SNI allowlist and NacelleHttpPolicy::with_allowed_hosts(...) with the same service names unless you intentionally need a narrower Host policy.

NacelleTlsConfig is the Rustls config shared with TCP TLS. NacelleTlsProvider reports Rustls for that config and OpenSsl for the TCP OpenSSL backend. HTTP TLS currently uses Rustls.

Enable HyperServer::with_access_log(true) when direct edge deployments need structured request logs. Access events are emitted with target nacelle::access and include transport, method, URI, status, request bytes, elapsed microseconds, and rejection reason.

Forwarded peer identity is disabled by default. Forwarded and X-Forwarded-For are considered only when the immediate socket peer is listed in NacelleHttpPolicy::with_trusted_proxy_ips(...); otherwise rate limits, request metadata, and access logs use the socket peer address.

For internet-facing deployments, a reverse proxy or load balancer can still own coarse traffic filtering and certificate automation. Nacelle now also enforces application-level body, request, connection, per-peer connection/request/connection-open-rate, timeout, TLS handshake, security header, and optional Host/header/method/URI limits in-process.

Slowloris-style clients are closed by NacelleHttpLimits::header_read_timeout. Trickle request bodies are closed by NacelleHttpLimits::request_body_read_timeout. Slow response readers are closed by NacelleHttpLimits::response_write_timeout when socket writes stop making progress.

Run security scans

Run vulnerability and dependency checks before release:

cargo audit
cargo tree -i serde_yaml
cargo tree -i unsafe-libyaml

serde_yaml and unsafe-libyaml should not appear in the dependency tree. If cargo-deny is adopted, add deny.toml with accepted licenses, advisory exceptions, and source policy.

Reference protocol

This document describes the optional LengthDelimitedProtocol reference implementation enabled by the reference_protocol feature. Custom protocols can implement Protocol<Req> directly and run over TCP or Unix domain sockets.

Frame Layout

All integer fields are little-endian.

Offset	Size	Field
0	4	`frame_len`
4	8	`request_id`
12	8	`opcode`
20	4	`flags`
24	`frame_len - 20`	body bytes

frame_len counts the fixed fields after itself plus the body. The minimum valid value is 20.

Flags

Flag	Value	Meaning
`FRAME_FLAG_START`	`0b0001`	First response frame for a request
`FRAME_FLAG_END`	`0b0010`	Last response frame for a request
`FRAME_FLAG_ERROR`	`0b0100`	Response body contains an error message

Request flags are decoded and preserved in FrameRequest, but the built-in server does not currently interpret request flags.

Requests

Each request frame contains one complete request body. The server decodes only the frame head before dispatch, then exposes the body to the handler as a NacelleBody. Small bodies are served from the connection read buffer. Larger bodies are streamed to the handler in configured chunks.

opcode is request metadata. The application handler decides whether to use it for routing, reject it, or ignore it. If the handler rejects an opcode after draining the body and returns an error, the server encodes that error as a response frame.

Handlers also receive connection metadata through NacelleRequest::connection. TCP listeners populate a stable connection id, peer/local socket addresses, and TLS metadata when a TLS backend is active. OpenSSL metadata includes negotiated protocol, cipher name, and cipher bit counts when available. Unix socket listeners populate the unix_socket transport label and local_path. Servers can attach typed per-connection state with connection_extension_factory(...); handlers retrieve it with request.connection.extension::<T>(). Apps built with serve(protocols, app) can attach the same state through NacelleApp::with_connection_extension_factory(...).

Responses

Handlers return a NacelleResponse with a streaming NacelleBody. The TCP transport encodes that response body into one or more response frames. By default, TCP responses inherit request_id and opcode from the request context. Applications can override either field with TcpResponseMeta.

The protocol guarantees:

the first response frame has FRAME_FLAG_START
the last response frame has FRAME_FLAG_END
a handler that returns an empty body still emits a start/end response frame
a handler error emits a start/end/error frame

Responses are written in request-processing order for a single connection. The prototype does not yet provide concurrent per-connection response interleaving.

Error Handling

Malformed frame heads, oversized frames, and EOF before a complete frame cause the connection to fail. Handler errors are encoded as error frames when enough request context is available. Unknown opcode handling is application policy.

Limits

The server enforces NacelleConfig::max_frame_len against frame_len. Buffer sizes and request-body chunking are configured through NacelleConfig. Runtime budgets, timeouts, and active counters are configured through NacelleLimits / NacelleRuntimeState.

TCP protocols can apply phase-aware request body limits by overriding RequestMetadata::max_body_bytes(connection, default_limit). The TCP runtime calls this after decoding the request head and before buffering or streaming the body. Implementations can inspect NacelleRequest::connection extensions, such as authentication/session state, and return a tighter pre-authentication body cap while keeping default_limit for authenticated requests.

TCP request handling is sequential per connection. Pipelined frames can sit in the socket/read buffer, but Nacelle does not run multiple handlers concurrently for one TCP connection. Streaming request bodies use request_body_channel_capacity for backpressure between socket reads and the handler, and declared streaming body bytes are allocated against the memory budget until the streaming request finishes.

HTTP hardening reference

Defaults:

NacelleHttpLimits::header_read_timeout: 30 seconds, enforced with Hyper's HTTP/1 header timeout and TokioTimer.
NacelleHttpLimits::request_body_read_timeout: 30 seconds, enforced while reading body frames.
NacelleHttpLimits::response_write_timeout: 30 seconds, enforced at Hyper's I/O write boundary.
NacelleHttpLimits::keep_alive: enabled.
NacelleHttpLimits::max_connection_age: disabled by default.
request and response body size limits: 16 MiB each.

NacelleHttpPolicy can reject requests before the handler runs:

allowed Host headers
allowed HTTP methods
maximum URI length
maximum header count
maximum aggregate header bytes
optional per-peer request rate limits through with_max_requests_per_peer_per_second
optional trusted proxy forwarded address handling through with_trusted_proxy_ips
optional security headers through with_security_header(...) or with_default_security_headers()
optional per-peer connection caps through NacelleLimits::with_max_connections_per_peer
optional per-peer connection-open rate caps through NacelleLimits::with_max_connection_opens_per_peer_per_second

NacelleTlsConfig is the Rustls config shared with TCP TLS. NacelleTlsProvider reports Rustls for that config and OpenSsl for the TCP OpenSSL backend. HTTP TLS currently uses Rustls.

Production configuration reference

Start from NacelleLimits::default() and tune shared resource budgets for the deployment. Use NacelleTcpLimits for TCP socket timeouts and NacelleHttpLimits for HTTP edge timeouts and keep-alive behavior. Active connections, in-flight requests, streaming tasks, body sizes, handler timeouts, and transport timeouts are bounded by default. Memory allocation budgeting is opt-in: set max_memory_bytes only after measuring that the limiter behaves correctly for your service.

Recommended presets:

Internal service: keep defaults, set body limits to the largest expected payload, and run behind process supervision.
Internet-facing behind proxy: cap connections and requests to the container budget, keep 30 second transport timeouts, and let the proxy own coarse traffic filtering or certificate automation when desired.
Proxy-aware HTTP: configure NacelleHttpPolicy::with_trusted_proxy_ips(...) only with known proxy addresses before allowing Forwarded or X-Forwarded-For to affect per-peer request limits or request metadata.
Direct HTTPS listener: enable http,tls, load certificate/key material through NacelleTlsConfig, configure an SNI allowlist with from_pem_with_allowed_server_names or from_der_with_allowed_server_names, set a short TLS handshake timeout, configure max_connections_per_peer and max_connection_opens_per_peer_per_second, enable HTTP access logs, and attach NacelleHttpPolicy with Host, method, URI, header, security-header, and per-peer request-rate limits.
Direct TCP Rustls listener: enable tcp,tls, load certificate/key material through NacelleTlsConfig, use serve_tcp_tls or enable_tcp_tls, and keep protocol-level authentication/authorization in the application protocol.
Direct TCP OpenSSL listener: enable tcp,openssl, load certificate/key material through NacelleOpenSslConfig, use serve_tcp_openssl, enable_tcp_openssl, or NacelleProtocols::tcp_openssl, and configure the SslAcceptor yourself when you need OpenSSL-specific policy.
Optional TCP OpenSSL listener: enable tcp,openssl, use serve_tcp_optional_openssl or the matching host/app builder method, and keep NacelleTlsDetectionOptions::timeout short enough for your accepted-connection budget.
IPv4 plus IPv6 TCP bind: use the NacelleProtocols::*_dual_stack(...) helpers when a serve-based app should bind both wildcard families for one protocol. The helpers register separate IPv4 and IPv6 listeners and force the IPv6 listener to v6-only mode.
Unix socket listener: enable tcp on Unix and use NacelleUnixSocketOptions only when this process owns stale-path cleanup or socket-file permissions.
Local load-test/autodeploy HTTPS: enable tls-self-signed and call NacelleTlsConfig::self_signed(...); do not treat generated certificates as a public trust or rotation strategy.
High concurrency: reduce TCP buffer capacities before raising max_connections, and tune NacelleTcpLimits separately from shared resource budgets.

Memory budget:

connection_budget =
  max_connections * (read_buffer_capacity + response_buffer_capacity)
body_budget =
  concurrent_buffered_or_streaming_bodies * max_request_body_bytes
total_budget =
  connection_budget + body_budget + handler/backend/runtime headroom

NacelleTcpOptions controls accepted TCP stream behavior. Defaults preserve the existing behavior: TCP_NODELAY enabled and TCP keepalive disabled. NacelleTcpBindOptions adds listener bind controls such as IPv6-only mode for APIs that need explicit family behavior.

NacelleTcpLimits controls TCP socket read, socket write, and idle timeouts. NacelleHttpLimits controls HTTP header read, request body read, response write, keep-alive, and max connection age behavior on HyperServer.

Dangerous configurations:

unbounded connections with large per-connection buffers
large body limits without a process/container memory limit
disabled timeouts on internet-facing listeners
direct internet-facing HTTP without Host/header/method/URI policy
direct internet-facing TLS without an SNI allowlist
direct internet-facing listeners without per-peer connection caps
direct internet-facing listeners without per-peer connection-open rate caps
direct internet-facing HTTP without per-peer request caps and access logs
trusting forwarded peer headers without an explicit trusted proxy list
generated self-signed certificates used as a long-lived public-edge certificate strategy
high keep-alive connection counts without proxy-level idle limits
long TLS detection timeouts on optional TLS listeners
Unix stale-path cleanup for a socket path not exclusively owned by this process

TLS certificate rotation:

#![allow(unused)]
fn main() {
let tls = NacelleTlsConfig::from_pem_files("cert.pem", "key.pem")?;
tls.reload_from_pem_files("next-cert.pem", "next-key.pem")?;
}

Reloads affect new TLS handshakes. Existing connections continue with the configuration negotiated when they connected.

OpenSSL builds need native OpenSSL development files. The openssl-vendored feature can build OpenSSL from source, but that build requires Perl on Windows.

API stability

Nacelle is 0.2.x, so public APIs are still experimental.

Stable enough for prototype integrations:

NacelleRequest, NacelleResponse, and NacelleBody
Handler and handler_fn
NacelleLimits and NacelleRuntimeState
NacelleHost
NacelleApp, NacelleProtocols, NacelleApp::serve(...), and serve(...)
nacelle::prelude::* for common application imports
NacelleTelemetry and NacelleTelemetryConfig
NacelleTelemetrySink for application telemetry bridges

Experimental:

transport-specific metadata
transport listener option structs
optional OpenSSL TLS detection on shared TCP listeners
telemetry sink details
stress tooling config
feature combinations involving tower, otel, and error-hints

New application code should prefer the app-first path: NacelleApp::new(handler).serve(protocols).await. Lower-level server and host APIs remain available when a service needs direct listener/runtime control. Telemetry docs should teach the generic NacelleTelemetry API.

Before 1.0, minor releases may change defaults or builder methods when production safety requires it. After 1.0, public API changes should follow semver, with migration notes for config/default changes.

Rust API reference

Generate the Rust API reference with:

cargo doc --workspace --all-features --no-deps

On Windows:

.\scripts\build-rustdoc.ps1

The generated index is:

target/doc/nacelle/index.html

Start with these public entry points:

nacelle::prelude::* for common application imports.
NacelleApp, NacelleProtocols, and NacelleApp::serve(...) for the app-first serving path.
Handler for the app-core boundary.
Protocol for TCP wire-format adapters.
NacelleTelemetry and NacelleTelemetryConfig for metrics and telemetry.
NacelleMemoryBudget, NacelleMemoryAllocation, and NacelleRuntimeState::memory_budget() for shared application/transport memory budget allocations.
TcpServer, NacelleHost, and transport runtime helpers when a service needs lower-level listener control.