LogBus

The Problem

Most log-processing toolchains force a choice: a heavyweight platform (Vector, Fluentd, Logstash) with its own DSL, or a pile of shell scripts that calcify the moment they leave the author's laptop. Teams want something between: declarative enough to read at a glance, programmable enough to handle the weird cases that always show up.

The Approach

LogBus runs event pipelines defined by stages (sources, parsers, transforms, formatters, sinks) in YAML. The engine wires the directed acyclic graph (DAG) from each stage's inputs: declarations and validates it statically - loops and bad wiring fail before any bytes move. Every stage is an async task connected by bounded multiple producer single consumer (MPSC) channels, and a plugin system keeps each stage swappable.

• Batteries included: Fourteen sources (journald, Kafka, OTel, HTTP, file tail & glob, S3 listings, sysinfo, a scheduler) and eleven sinks (OpenSearch, ClickHouse, CloudWatch Logs, Kafka, S3, email, OTel) cover the common wiring, plus codecs for JSON, YAML, Avro, Parquet, key-values, and lz4/deflate/snappy compression.
• Escape hatches, not lock-in: Built-in primitives cover the common case; embedded QuickJS handles the long tail.
• Single static binary: Drop it on a host, point it at a config, done. Binaries can easily be built to include only the requisite plugins.

Automatic Backpressure

The interesting engineering is in what happens when a sink slows down. Stages are connected by bounded MPSC channels with a tunable capacity, so backpressure is a property of the architecture rather than plugin code: when a sink's buffer fills, the upstream suspends, that stage stops reading its input, and the stall propagates stage by stage until the source itself pauses its I/O. No events are dropped and no CPU is burned waiting — suspended tasks are parked by the runtime. The maximum in-flight accumulation is exactly capacity × channels in the chain, so worst-case memory is a number you can compute, not a graph you discover in production.

Fan-in & fan-out fall out of the same primitives: multiple upstreams share clones of a downstream's sender (the receiver closes only when the last upstream finishes), and a lightweight fan-out task clones each event — cheap, since events are reference-counted — to every downstream.

The one deliberate exception proves the rule: the live-stats UI receives events via a non-blocking send, so a slow browser tab can never backpressure the data path. Observability should watch the pipeline, not wedge it.

Testing the Claims

The claim above is falsifiable on purpose: a slow or overloaded sink stalls the source, never loses events. A falsifiable claim you never measure is just a better-sounding vibe, so the rest of this page is the measurement.

It is deliberately narrow. One pipeline, one machine, ten repetitions. Everything below was produced by a load-test harness that lives in the repo and can be re-run with one command. What is not measured is said plainly at the end rather than estimated.

▶ Replay Load Test — visualize the throughput, queue depth, memory, and latency on one time axis. All ten repetitions are there, including the one the harness threw out. The tables below are that recording, summarised.

The Rig

Credibility here is not the peak number — it's that someone could reproduce it, even if they never do.

• Hardware. Apple Silicon, 8 cores, 16 GB RAM, macOS. This is a laptop, not a tuned server — no CPU governor pinning is available, and the numbers should be read as a floor rather than a ceiling.
• Pipeline. Seven stages: stdin → parse-lines → parse-json → stamp-ingress → transform (JS) → format-json → write-file(/dev/null). This is not a very representative pipeline since the source & sink a designed to play nice with the load harness and avoid IO.
• Event shape. 1 KB synthetic JSON records, generated once into a fixed corpus (100,000 lines, SHA-256 77b77c06…) so the load generator is never in the hot path. The feeder replays the corpus at a scheduled rate; timing measures LogBus, not the producer.
• Counting. Every run reconciles source-side emitted against sink-side committed. "0 drops" is arithmetic — emitted - committed = 0 — not optimism. A separate tripwire confirms every one of the six data-plane stages passed the same event count, so a silently-skipped stage cannot masquerade as a fast one.
• Method. Load is open-loop: each measurement is a fixed-rate dwell, never a ramp. A controller walks the rate up a fixed geometric grid until the engine fails to keep up, then narrows in, holds at capacity, deliberately overloads at 1.5×, drains, re-tests the same rate, and finally profiles latency at 50 / 70 / 90% of capacity. Ten independent repetitions, fresh engine each time, 73-78 s per repetition; every stage-to-stage channel is bounded at 4,096 events.
• Reproduce. just load::profile. Raw JSONL and a provenance sidecar (git rev, corpus hash, binary mtime) are written for every run.

Throughput

67,500 EPS sustained — ≈66 MiB/s of 1 KB JSON through seven stages, including a JavaScript transform. Nine of ten repetitions held that rate for a ten-second dwell and delivered 67.4k/s against it. The tenth did not.

67,500 is a rung, not a measurement. The controller climbs a fixed geometric grid, so every repetition can only answer held or tripped at the same discrete rates. Nine runs held 67,500 and all nine tripped on the next rung up, 78,750. What was actually measured is therefore an interval — capacity lies somewhere in [67,500, 78,750) — and 67,500 is the floor of it. A finer grid resolves a higher number off the same binary; that is grid resolution, not a faster engine, and the headline is quoted at the floor deliberately.

Within that grid, capacity is still a probability. Repetition 6 tripped on 67,500 — the rate the other nine held. Same binary, same machine, same offered load, different answer. That single disagreement is why this reports a quorum rather than a number.

Repetition 6 is excluded rather than quietly averaged in: its hold dwell never confirmed, so the harness marked it hold-unconfirmed and dropped it from the aggregate. Its backlog left the dwell at 64,591 events and never came back — it is the one run here that genuinely did not recover. Nine of ten clears the 8-run quorum the harness requires, so the result stands — but a run that discards its own bad data has to say which data it discarded. Its events still count toward the loss arithmetic below.

Latency

Latency at idle is meaningless and latency at 100% is just a queue depth. End-to-end here means source-read to sink-commit, measured against an ingress stamp the feeder writes at each event's scheduled arrival time — so a late-arriving event is charged for being late.

Percentiles are read from the engine's HDR histograms with the nine valid repetitions pooled, not averaged — a mean of nine p99s is not a p99 of anything. Each row is 2.2-6.0 million samples, so even p99.9 is a real measurement rather than a handful of stragglers.

The body of the distribution is flat all the way to capacity. From half load to 100%, p50 moves 3 ms → 4 ms and p90 moves 5 ms → 7 ms. Nothing in the typical path degrades as the engine fills; what degrades is the tail: p99 goes 16 ms → 90 ms on the way up, and 522 ms if you arrive at capacity from above. A mean would show none of this.

The two 100% rows are the same rate and differ 6× at p99, and that is the most useful number here. Both dwells offer 67,500/s for ten seconds. The first hold runs immediately after the controller has tripped the engine at 78,750, so it starts with a backlog still draining and pays for it: p90 38 ms, p99 522 ms. The re-test runs after a full recover-and-probe cycle and sees p90 7 ms, p99 90 ms. The engine's steady-state tail at capacity is the second number; the first is what you get if you arrive at capacity from above.

The p99 ordering between 70% and 90% is not physical, and it is noise rather than an artifact. Lower load cannot be slower than higher load. Per repetition, most of these dwells sit at a 5-9 ms p99 and one or two land a single multi-hundred-millisecond stall — rep 9 alone drags the 70% row to 211 ms, rep 3 the 90% row to 184 ms. Pooled, whichever row caught a stall wins. Nine repetitions is enough to see the stalls exist and not enough to put a rate on them.

The clean measurement of the ordering effect is the post-overload probe. It offers 60,750/s — the exact rate of the 90% row above — but runs immediately after the deliberate overload instead of after recovery:

Same rate, same binary, same run — a 19× difference at p99, and every one of the ten repetitions put that probe's p99 above 1.1 seconds. Latency measured against a scheduled arrival time charges each event for the queue it was born into, so a dwell that inherits a backlog reports that backlog no matter how fast the engine is running during it. This is the clearest argument here for why a load test has to publish the order its dwells ran in, not just its numbers.

One resolution caveat throughout: the engine's histogram buckets are milliseconds, so "4 ms" is one bucket and the sub-millisecond structure is not recoverable.

No Events Dropped

Across all ten repetitions — including the one excluded from the capacity figures, because a discarded measurement is still a real run of the engine: 35,414,488 events in, 35,414,488 events out, 0 lost, 0 errors.

That total includes the deliberate overload phase in every repetition — offered load above what the engine could process, sustained until it visibly failed to keep up. It still lost nothing. The events that could not be processed in time were not dropped; they waited, and the source was stalled until there was room. This is the design claim, and it held.

Memory: Bounded, Not Flat

Memory here means the physical memory the process holds (aka resident/working set size). It tracks in-flight work:

Within a single repetition that is better than a 10× range. What the design actually promises is not flatness but a bound, and that is what the data supports: memory rises as queues fill, peaks when the engine is overloaded and every channel is backed up, and falls back once the backlog drains.

This is the clearest thing in the replay: the memory trace is the backlog trace, one panel down and a moment later. Memory here is not a mystery to be discovered in production — it is queue depth, and queue depth is bounded by construction.

The peak is now repeatable, and it is repeatably the wrong number to quote. All ten fresh engines peaked in a 184-208 MB band, and every one of them peaked in the same phase — the drain immediately after the deliberate 1.5× overload. That looks like a tight, quotable bound. It is not one: an earlier pass of this same harness, tripping less hard, put the peak at 67-129 MB. The peak is a property of how far past capacity you push and for how long, not of the engine, and any number here is only the peak for this overload profile. Size for the worst case you can actually offer, not for the median of someone else's load test.

Two limits on this. The sampler takes roughly one snapshot per second, and the peak is a single sample in a fast-moving drain — the true maximum is certainly higher than any figure here. And the stronger claim above, that worst-case in-flight memory is arithmetic you can compute from channel capacity, is consistent with a bound in this range at the 4,096-event buffer this run used, but is not tested; confirming it needs a buffer-size sweep this run did not do.

Past Capacity

Every repetition ends with a deliberate overload: 101,250/s offered, 1.5× the sustainable rate, held for two seconds. Throughput does not plateau there — it goes down.

Pooled across nine valid repetitions, offering 50% more load bought 8% less throughput, and per repetition the overloaded rate ranged 57,409-64,927/s — never once above the un-overloaded rate. The scheduler churns on full MPSC channels and cache locality is lost, so an engine past capacity does strictly worse than one held just below it. This is the most useful thing here for anyone actually deploying it: the failure mode of offering too much is not "the same throughput plus queueing", it is less throughput and queueing.

Recovery is the other half of that claim, and it held. After each overload the harness drains, probes at 90%, then re-offers the original 67,500/s for a full ten seconds. All ten repetitions sustained the 90% probe, and 9 of 9 valid repetitions re-held full capacity — backlog flat, exiting the dwell at 68-479 events. The exception is repetition 6, the one already excluded: it entered the re-test with a backlog it had never cleared and left with 27,227 events outstanding. Overload is a performance event, not a state the engine gets stuck in — except in the one run out of ten where it was.

Summary

• The loss claim survived contact. 35.4 million events, seven stages, a deliberate 1.5× overload in every repetition, and nothing was lost. Backpressure is architecture here, not effort — no plugin code arranges the stall.
• The JS escape hatch is affordable. 67.5k EPS is with a JavaScript transform in the hot path. What it costs relative to a native stage is not measured yet, so no percentage is claimed.
• The headline is a floor, not a peak. Capacity was bracketed to [67,500, 78,750) by a coarse rate grid and is quoted at the bottom of that interval.
• The median is flat to capacity; only the tail moves. p50 goes 3 ms → 4 ms from half load to 100% while p99 goes 16 ms → 90 ms. The safe operating range is wide and its edge is sharp.
• How you arrive at capacity matters as much as the rate. The same 60,750/s dwell shows an 80 ms p99 after recovery and a 1,546 ms p99 straight after an overload. Publish the dwell order or the latency numbers mean nothing.
• Capacity is still a probability. One repetition in ten tripped on the rate the other nine held, and it was the only one that never recovered.
• Past capacity, throughput goes down, not sideways. 50% more offered load bought 8% less throughput. The most useful thing here for anyone actually deploying it.
• Memory is bounded; the bound belongs to the overload, not the engine. The peak was a tight 184-208 MB across ten runs — and 67-129 MB on an earlier pass that tripped less hard. Size for the worst case you can offer.
• Where the load test is thin. One pipeline on one laptop. No comparison against Vector or Fluent Bit. The rate grid is coarse enough that capacity is an interval rather than a number. The harness's second scenario is not reported because its load generator may be the bottleneck rather than the engine, and publishing it would mean measuring the harness instead. The computed-bound claim is not directly tested; that needs a buffer-size sweep this run did not do.