Fork me on GitHub
#missionary
<
2021-11-04
>
martinklepsch11:11:22

I’m not sure if this is a missionary specific question but I have a sequence of chunks from the function above like

["thing 1\n\n"
  "thing2: "
  "also thing2\n\n"
  "thing3\n\nthing4\n\n"]
These come in “one by one”, but the downstream consumer can only deal with “complete” items, e.g. it will fail when it receives
["thing 1\n\n"
  "thing2: "]
(complete meaning ending with \n\n) So kind of I want to hold back the "thing2: " string until I get the one that completes the segment. I got some help in #clojure using mapcat and partition-by but these approaches don’t really allow “holding back” an incomplete item.

Ben Sless11:11:27

are you implementing a SSE client?

leonoel11:11:14

I would write an ad-hoc stateful transducer

👍 1
martinklepsch12:11:15

(defn sse-chunk-xform
  "Returns a transducer that retains incomplete events (i.e. no trailing \n\n)"
  []
  (fn [rf]
    (let [incomplete (volatile! nil)]
      (fn
        ([] (rf))
        ([result] (rf result))
        ([result el]
         (cond
          (and (nil? @incomplete)
               (.endsWith el "\n\n"))
          (rf result el)

          (and @incomplete (.endsWith el "\n\n"))
          (do (rf result (str @incomplete el))
              (vreset! incomplete nil))

          :else
          (vswap! incomplete str el)))))))

Ben Sless12:11:13

@U050TNB9F in the front end or back end?

martinklepsch12:11:59

I started with https://github.com/oliyh/oxbow but I think the author may have only used it with small events where the chunks are usually complete

Ben Sless12:11:21

don't take it the wrong way, why torture yourself? the SSE specification is annoying. Let someone else solve the problem for you

Ben Sless12:11:30

Isn't there a SSE client in Javascript you can just use?

Ben Sless12:11:04

you need an entire state machine

Ben Sless12:11:25

your initial state will have to me something like {:id "" :type "" :data ""}

Ben Sless13:11:22

you may get several messages before you emit, so you need to handle all the transitions

martinklepsch13:11:33

Fair comments, I originally just used EventSource which was working fine but I need HTTP header auth and so I have to go the js/fetch route. It’s a good idea to use a library for the parsing, not sure I really looked for one to do just that but I’ll take a look

martinklepsch13:11:29

Kind of a shame that the EventSource API is so tied to the networking layer when most of the complexity seems to be in the stream parsing

Ben Sless14:11:12

you can implement it yourself, but it's a pain

martinklepsch16:11:24

I mean after reading the spec it doesn’t seem that painful

Ben Sless17:11:20

We call that "famous last words"

😛 1