Why the Screen Shatters While Tokens Flow

In a chat UI that streams LLM responses token by token, the first place the screen falls apart is parsing. The moment ## arrives, it shows as a horizontal rule instead of a heading, and when only ** has landed, two asterisks sit raw on screen instead of bold text. Until a code fence closes, the code block flows out like an ordinary paragraph. Once the stream finishes everything looks right, but while it's flowing it always appears broken. This isn't a parser bug; it stems from the condition that a token stream always passes through an incomplete state. From a reader's perspective, the broken screen is what sticks in memory first.

The Moment Incomplete Syntax Is Exposed on Screen

The most common implementation re-parses the entire accumulated text on every chunk. It's simple, and given that most markdown parsers are designed for batch processing, it's also something of an unavoidable choice. But the cost splits in two directions. Re-parsing proportional to the accumulated length repeats on every token, so overall it approaches O(n²), and because it can't distinguish finished syntax from unfinished syntax, even parts that were already rendered correctly get re-rendered. The double cost is why the screen falls further and further behind on each frame as text grows. The key point is that time doesn't grow per token; it grows in proportion to the cumulative length stacked up so far. When you add the cost of rebuilding the DOM and recalculating layout on every token, the screen stutters no matter how fast data arrives.

There are two paths forward. One is to handle incomplete state separately. Until a structure like an unclosed fence, list, or table header is completed, that region is shown as plain text or hidden entirely. The renderer still bears the full re-draw cost, but the flicker of syntax jumping and overlapping shrinks. The other is to align chunk boundaries with what the server sends. If chunks are cut only at positions where markdown syntax isn't interrupted, the client-side parsing loses nearly all its burden. This option only opens up when the server is something you can touch, and it doesn't apply when you have to replay cached past responses verbatim. Switching to a renderer that supports incremental parsing is another route, but since most parsers are batch-designed, swapping libraries alone won't reduce the cost.

Both approaches use the same validation criteria: whether the final state is complete when the stream ends, and whether the screen stays at an acceptable level even if you cut at an arbitrary position with a code fence left open. The latter is hard to catch with parser tests, so a property test that slices the response token by token and walks every prefix is practical. Without deciding the meaning of "acceptable" up front, this validation never finishes.

The Path an Abort Signal Travels to the Server

Harder to detect than parsing collapse is abort handling. Passing a signal via AbortController and aborting the stream is only a few lines of code, but the path an abort actually travels is longer than you'd think.

If the read loop doesn't check for abort, the loop keeps consuming whatever data remains in the buffer even after fetch has been aborted. If the catch block doesn't filter out e.name === 'AbortError', an abort gets mistaken for an error and retry logic kicks in. Once retries start, an intended abort spirals into duplicate responses.

const controller = new AbortController();
try {
  const res = await fetch(url, { signal: controller.signal });
  const reader = res.body.getReader();
  while (true) {
    const { value, done } = await reader.read();
    if (done || controller.signal.aborted) break;
    push(value);
  }
} catch (e) {
  if (e.name === 'AbortError') return;
  throw e;
}

Even after the loop ends, an easy-to-forget step remains. If the server side doesn't check whether the connection was severed, it keeps paying the cost of producing a response after the client has given up. On a Node server, you should use res.writableEnded or req.closed as an abort signal to break out of the response loop. In SSE, a keepalive where the server periodically sends a single comment line is common; since comments aren't meant to be painted to the screen they don't leak into the response, and there's no simpler means for detecting a half-open connection.

Listener management also drifts often. If signal.addEventListener is registered per read iteration and never removed, listeners accumulate. Registering once per connection is safer. AbortController can't be reused, so you need a fresh one for every retry. It's also worth noting that aborting fetch and aborting a stream read don't always happen on the same timing. Tying both to the same signal, and closing whatever connection still remains loose with a client-side timeout, is the realistic approach. Validation is a matter of checking whether the server log records the connection closing after an abort, and whether retries fail to fire.

A Half-Baked Response: Salvage or Discard

Whether to resume and recover a response cut off mid-stream or discard it comes down to two conditions: whether there's room to keep receiving from the point of the break, and whether what arrived before the break has value as a partial result. Resuming requires the server to support resuming "from after the interrupted point," and if the resume position is off, you end up completing awkward sentences. When judgment gets fuzzy, discarding and starting over keeps state management simpler. But unless retry count, backoff interval, and jitter are managed in one place, the same cost repeats at the same point. Here the retry budget becomes the only rule you must follow. Whichever path you choose, the moment you drop the "generating" indicator and flip the state to done must come after the stream actually closes. Flip it earlier and a half-baked response stands as the final version.

Treating connection drops and aborts through the same path drags in another problem. An abort is an intentional end, so you must not retry, but a connection drop demands a retry. Without a flag distinguishing the two, pressing the "stop" button still lets automatic retries pull the same response again. Discarding a stream by its response id filters out this duplicate. The cleanest guard against the trap is to make streams idempotent: ignore any new stream bearing an id already handled, and tag retries with the same id.

When a connection drops, EventSource attempts automatic reconnection by default, and on error it fires the error event and enters a reconnect wait. When reconnection happens, the entire stream arrives again from the start, so content gets painted over what was already drawn. With frequent drops, the screen—unaware of the drop—keeps redrawing the same content over a half-baked response. Writing the loop yourself with a fetch and ReadableStream combination removes automatic reconnection, but then everything down to drop detection becomes your responsibility. The Last-Event-ID resume pattern is only natively supported in EventSource environments, so with a hand-rolled loop you have to build the resume feature yourself.

Backpressure: Separating Server Speed from Screen Speed

Behind the consistency problems waits the performance axis. A structure that updates state per token and redraws the whole component has a short time to first token (TTFT), but the per-frame burden then grows in proportion to accumulated length. Batching, which coalesces several updates into one frame, hides some of this, but it hits its limit with streams where chunks arrive tens of milliseconds apart. It's not purely a rendering problem either: the longer the response, the larger the accumulated DOM grows and the more the diff cost climbs along with it. Typical LLM streams arrive at tens to hundreds of tokens per second. The moment parsing and rendering can't keep up with that rate, the buffer fills, and past a few thousand characters the accumulated text itself delays frames.

The high-impact fix is backpressure: keep a buffer and only draw from it at the rate the screen consumes. Separate the speed the server sends at from the speed the renderer can paint, and drain the buffer on requestAnimationFrame (RAF) ticks. Since parsing and rendering happen at most once per frame, per-token re-rendering disappears. But cranking throttling to full from the start inflates TTFT and makes perceived responsiveness worse. An adaptive approach is steadier: only restrict the drain rate once the buffer has accumulated past a certain amount, and draw immediately when the buffer is empty. Throttle strength is set by whether the per-fix frame count stays above 60 and whether render time fits in the 16.6ms budget. Backpressure's failure mode is over-buffering. If the server pours data out quickly but you only tighten consumption, the buffer grows, and so does the amount discarded at the moment of abort. You need a mutual throttle: set a buffer ceiling and pause stream reads when it fills beyond that.

Keeping the DOM stable is the same story. Re-parsing and wholesale replacing the entire text every time makes the cursor and scroll position jitter together. Isolating the streaming region behind a component boundary so the outer tree isn't redrawn does more than micro-optimizations. On React, wrapping the boundary in memo so subtrees whose props haven't changed skip rendering makes a real difference. Auto-scroll also typically only follows near the bottom and stops updating when scrolled up. One threshold is enough, and roughly 40–80 pixels usually works.

Validation happens at three points. For parsing, cut with a code fence left open and slice token by token, capturing where the screen breaks. For abort, check that the server logs a connection close after the abort and that retries don't create duplicates. For performance, manipulate bandwidth and latency with CDP network throttling while measuring TTFT and frame time. None of the three surfaces on a healthy network, so you have to run with artificially inflated latency for it to mean anything. The fastest reproduction is running a small local stream server that emits tokens a few milliseconds apart. Without any bandwidth limiting, just replay the sequence of drop, retry, drop again in order.

Don't try to solve all these traps at once—start with the abort path. Parsing and performance are visible, so they're easy to fix. Abort and retry look like they work and only reveal their defects at the moment the stream actually breaks. Start by checking whether the server log records the connection closing after an abort, and most of the rest falls within that path.

Comments

Loading comments.