The event stream

Here's something that sounds like it should be bigger than it is: Datastar's entire server side is a text format with two event types and a handful of options. That's the whole protocol. We write all of it (six short functions of Clojure), test it against the expected output Datastar publishes for its own SDKs, and then hold a stream open so the server can keep talking. And talking. And talking.

Part 1 wrote one event by hand. That's plenty for saying hello, but Brew Board has bigger ambitions. It needs to replace a cart, append an order to the barista's list, remove a picked-up order, set a signal, and run a line of JavaScript. Five jobs! Surely that's five formats? Nope. All of those are options on two event types. The Datastar repository has an sdk/ADR.md that spells them out for anyone writing a server library, plus a directory of test cases (sdk/test/get-cases) with the exact bytes each call must produce. Datastar's own SDKs are written against those. So is ours. Which means when our tests pass, we're not "roughly compatible." We're byte-for-byte compatible.

One function writes every event

(defn send-event
  "Format one SSE event. Add an id line if id is set. Add a retry line unless retry is nil or 1000."
  [event-type data-lines {:keys [id retry]}]
  (str "event: " event-type "\n"
       (when id (str "id: " id "\n"))
       (when (and retry (not= 1000 retry)) (str "retry: " retry "\n"))
       (string/join (map #(str "data: " % "\n") data-lines))
       "\n"))

The ADR fixes the order, and it's not up for debate: event, then id if there is one, then retry unless it is the default of 1000 ms, then the data lines, then a blank line. By the way, id and retry aren't Datastar inventions. They're standard SSE fields, and they earn their keep when things go wrong. When a stream drops and Datastar reconnects, it sends the last id it saw as a last-event-id request header, and a retry field sets how long it waits before reconnecting. Part 6 uses both. For one-shot responses you'll rarely bother.

Everything else, and I do mean everything, is a function that decides which data lines to pass to that one. It's turtles, but only one turtle.

datastar-patch-elements

The data lines are key value pairs. For element patches the keys are:

Defaults are left out, so the common case is a single elements line, which is exactly part 1's event. (See? You already knew this.) The eight modes are where the actual decisions live:

That last column isn't folklore. It comes from one line in patchElements.ts: if there is no selector and the mode is neither outer nor replace, Datastar throws PatchElementsExpectedSelector. Think about why. Only those two modes can find their target from the payload's ids; "append" to what, exactly? A selector, on the other hand, can match lots of elements. Datastar runs querySelectorAll and applies the patch to every match, so mode remove with selector .picked-up deletes all of them in one go. Handy. Also, occasionally, alarming.

"Morphs" is the whole difference between outer and replace, and it's a bigger difference than it looks. Both put the new element where the old one was. outer walks old and new together and changes only what differs, so a text input the user is typing in keeps its focus and cursor. replace throws the old element in the trash, focus, cursor and all. Part 6 sends a whole page view on every change, which sounds insane, and relies on outer to make that harmless.

(defn patch-elements
  "A datastar-patch-elements event. Send only options that differ from the defaults."
  [html {:keys [selector mode use-view-transition view-transition-selector namespace] :as opts}]
  (send-event "datastar-patch-elements"
              (concat (when selector [(str "selector " selector)])
                      (when (and mode (not= "outer" (name mode))) [(str "mode " (name mode))])
                      (when use-view-transition ["useViewTransition true"])
                      (when (and use-view-transition view-transition-selector)
                        [(str "viewTransitionSelector " view-transition-selector)])
                      (when (and namespace (not= "html" (name namespace)))
                        [(str "namespace " (name namespace))])
                      (when html (prefix-lines "elements" html)))
              opts))

(defn remove-elements
  "Remove every element that matches the CSS selector."
  [selector opts]
  (patch-elements nil (assoc opts :selector selector :mode :remove)))

Multi-line HTML becomes several elements lines. That's not a style choice. A raw newline inside a data line would end the line right there, and the rest of your HTML would arrive as a malformed field, which is a fun one to debug. On the browser side, the parser groups data lines by key and joins each group with \n, so the HTML arrives back in one piece, none the worse for the trip.

Now go build some events. Change the mode, the selector, and the HTML, and watch both the wire text and what the toy page does with it. Try to break it. That's half the fun.

The order of data lines doesn't matter

Here's a little puzzle. Look at the SDK test cases closely, and you'll find that they disagree with each other. In patchElementsWithAllOptions the selector line comes before mode. In removeElementsWithDefaults and in every executeScript case, mode comes first. So which one's right? Both are. The script that runs those tests, sdk/test/compare-sse.sh, sorts data lines by key before comparing, keeping the order of lines within a key. And that matches the browser: in fetch.ts, the message handler splits each data line at its first space and collects lines into a map keyed by the word before it. Only the order of lines that share a key (the lines of one HTML block) is significant. Our lesson's golden-file test normalizes the same way, so it passes with either order. Mystery solved, and nobody had to file a bug.

Reading a stream, from the browser's side

Let's walk around to the other side of the counter for a minute. The network doesn't deliver events. It delivers chunks of bytes, and it has no respect whatsoever for your formatting: a chunk boundary can fall anywhere, including smack in the middle of a line. So Datastar's reader in fetch.ts is a three-stage pipeline: bytes to lines, lines to messages, messages to patches.

on chunk(bytes)
  buffer = buffer ++ bytes
  for each complete line in buffer          -- ends at \n, \r, or \r\n
    if line == ""            -> dispatch(message); message = new()
    else
      (field, value) = split line at first ":"   -- one optional space after it
      match field
        case "data"  -> message.data = message.data ++ "\n" ++ value   -- first one: no "\n"
        case "event" -> message.event = value
        case "id"    -> message.id = value; remember it for last-event-id
        case "retry" -> if value is an integer: retryInterval = value
  keep the unfinished tail in buffer

dispatch(message)
  if not message.event starts with "datastar" -> ignore
  args = group lines of message.data by the word before the first space
  apply(message.event, args)                 -- patch elements or patch signals

Step through a stream that arrives in deliberately awkward pieces. Notice that nothing happens to the page until a blank line completes an event, and the chunk boundaries never matter. Not once. The buffer just shrugs and waits.

datastar-patch-signals

The second event type changes signals instead of elements. Part 3 explains signals properly, so I'll give you just the elevator pitch: they're the page's reactive values, arranged as a JSON-shaped tree. The event carries a JSON object and applies it as an RFC 7386 JSON Merge Patch. That's a fancy name for three simple rules: keys in the patch are set, nested objects merge recursively, and a null value removes the signal.

(defn patch-signals
  "A datastar-patch-signals event. Signals is a map (encoded as JSON) or a JSON string.
  The client merges it as a JSON Merge Patch: a null value removes that signal."
  [signals {:keys [only-if-missing] :as opts}]
  (send-event "datastar-patch-signals"
              (concat (when only-if-missing ["onlyIfMissing true"])
                      (prefix-lines "signals" (cond-> signals (not (string? signals)) json/generate-string)))
              opts))

So (patch-signals {:quote 560 :error nil} {}) produces:

event: datastar-patch-signals
data: signals {"quote":560,"error":null}

which sets $quote to 560 (hello again, medium oat latte) and removes $error. And onlyIfMissing true sets only the keys that don't exist yet. That's exactly what you want for defaults that must not clobber what the user already typed, because nothing endears you to users like erasing their input.

Running a script

You might expect a third event type for JavaScript. There isn't one, and I think that's rather elegant. The ADR defines "execute script" as a patch-elements event that appends a <script> to body:

(defn execute-script
  "Run JavaScript in the browser: append a <script> to body. With auto-remove (the default)
  the script tag removes itself after it runs."
  [js {:keys [auto-remove attributes] :or {auto-remove true} :as opts}]
  (let [attrs (cond-> (mapv (fn [[k v]] (str " " (name k) "=\"" v "\"")) attributes)
                auto-remove (conj " data-effect=\"el.remove()\""))]
    (patch-elements (str "<script" (string/join attrs) ">" js "</script>")
                    (assoc opts :selector "body" :mode :append))))

Two details make that work, and both are sneaky. First, scripts inserted with innerHTML-style parsing don't run (the browser is protecting you from yourself), so patchElements.ts replaces each new <script> with a freshly created one, which does. Second, data-effect="el.remove()" is an ordinary Datastar attribute: an effect runs when the element is processed and again whenever a signal it read changes. el.remove() reads no signals, so it runs exactly once and deletes the tag. The script cleans up after itself, like a polite houseguest, so a page that runs a thousand scripts doesn't pile up a thousand tags. Use this sparingly, though. Brew Board uses it exactly once, in part 7, to talk to the native shell.

Holding the stream open (or: the response that wouldn't end)

Every response so far has ended after its events. But here's the thing: nothing in the format requires that. A stream can just… stay open, and deliver events whenever the server has something to say. That needs an asynchronous response, because a Ring handler that returns a map returns once and then it's done. http-kit's as-channel hands the handler a channel it can hang onto:

(defn clock-handler
  "Hold the response open. Send one patch per tick, at most `ticks` times, then close.
  Stop early when the client goes away."
  [req tick-ms ticks]
  (hk/as-channel
   req
   {:on-open  (fn [ch]
                (hk/send! ch {:status 200 :headers stream-headers} false)
                (future
                  (loop [n 1]
                    (when (and (hk/open? ch) (<= n ticks))
                      (hk/send! ch (patch-elements (clock-html n) {}) false)
                      (Thread/sleep (long tick-ms))
                      (recur (inc n))))
                  (hk/close ch)))
    :on-close (fn [_ status] (swap! closes conj status))}))

The first send! carries the status and headers. Each later one carries a complete event as one string, with close? false. That "whole event per call" bit matters more than it looks. The ADR asks implementations to "ensure ordered delivery", because two threads writing half-events to one connection would interleave into word salad. One string per event is the simplest possible way to get that guarantee. The SDK's http-kit adapter takes a lock instead, because it can also write through a compressing output stream.

From the page's point of view, a held-open stream is just a GET that takes a really long time to finish. That's all. The lesson starts it from a button, @get('/clock'). A page that should be live from the start uses <div data-init="@get('/clock')"> instead: data-init runs its expression once, when Datastar processes the element. Each event updates #clock. When the browser goes away, the connection closes and http-kit calls :on-close, which is where part 6 unsubscribes the connection from the broadcast hub. The lesson's demo checks this rather than taking it on faith: it reads three ticks, drops the connection, and waits to see on-close fire.

Now, let's be honest: the clock-handler above is a toy. It's one thread per connection, and it sends whether or not anything changed, like a coworker who gives you hourly updates on a project nobody's touched. Part 6 replaces the timer with "send when the state changes" and replaces the thread with a watch on an atom. The format doesn't change one bit. The stream is the same text, only spread out in time.

cd clojure && clojure -M:ex -d ex -n exercises.lesson02.event-stream-test