Clojurians Log v2

(js/document.createElement "div")

Yes. But I meant that as an example of calling into deeply nested functions in goog

I have only ever seen it used like you described. You can probably also (:require [goog.global :as some-alias]) but I'm not sure whether there's any point in that.

That is, if we're talking about goog.global specifically. If it's about some goog module, I would definitely require it before using it. There's a doc somewhere on CLJS website describing how various parts of GCL should be required.

Also, I use the x.y pattern only for something that comes from js/something and has exactly one dot. Anything else becomes (-> something .-field1 .-field2 (.fn)).

Thanks!

I think you could also use .., I tend to favor that over the threading macros plus ./`.-`

I'd do (apply str (map my-fn ...)).

idk about speed but there's clojure.string/join

Or that, yes. Which does (apply str ...) for its 1-arity anyway.

yeah except it skips the allocation of the varargs seq?

> Are there any concerns with performance here And yes, there are - you're allocating strings on every item. > except it skips the allocation of the varargs seq Ah, probably. I tend not to think about the impl of varargs too much. :)

it doesn't usually matter unless you're in a loop anyways

Actually, I oversimplified - the sequence is partitioned 😄 so I cant straight away use apply. I only want to transform the second element in each partition. (reduce (fn [acc [s1 s2]] (str acc s1 (my-fn s2))) "" ...).

(str/join (mapcat #(update 1 my-fn) ...))

Wow!

https://github.com/cgrand/xforms/blob/master/src/net/cgrand/xforms/rfs.cljc#L134-L143 generally what you want is a loop around appending things to a growable buffer. java and js strings being immutable flat, not tree shaped things, means you end up appending to a mutable buffer and then transforming into a string. using something like reduce with a finalizing step (which transduce gives you) is a handy way to express that

alternatively I like iolists, the idea being you define your io primitives to operate on arbitrary trees of strings instead of just strings, so concating s1 and s2 is just [s1 s2]

Actually (str/join (mapcat #(update % 1 my-fn) ...)) did not work because the pairs are a seq and not a vector...

But (mapcat (fn [[s1 s2]] [s (my-fn s2)])) did it!

(transduce
  ; (map second) would also work on seq
  (map second)
  ;; or net.cgrand.xforms.rfs/str which does exactly (completing str! core/str)
  (completing
    net.cgrand.xforms.rfs/str!
    str)
  ""
  [[:whatever "a"]
   [:whatever "b"]
   [:whatever "c"]])
;; => "abc"

Scary.

Hahah… really?

It might be a few more characters but I personally do prefer how transducers lay out the “transformation process” as linear steps

(xf/str (mapcat (fn [[a b]]
                  [a (my-fn b)]))
        coll)

1. I have a collection 2. Start transduce… process elements one by one 3. Take every second element of step 2. 4. Use reducing fn to create a StringBuilder 5. As a completing step, return that StringBuilder as a regular string via str

The scariest part for me is using anything from net.cgrand.xforms, same with Specter. They're both so full of stuff, it makes it much harder to find what you need and much harder to later remember what the code that you wrote does.

The OP specifically asked for “performance”… For small collections I would likely not bother with the StringBuilder and just use clojure.string/join with the mapcat or similar… The value of transducers really shows if you have multiple transformation steps both in clarity (IMO) and performance (objectively)

While I can agree with Specter, I think xforms is quite great if you have the need for it, and doesn’t deviate much (or at all) from regular Clojure transducer semantics

Like str! is just a regular reducing fn one would write (if you can read past the :clj, :cljs , :cljd elaboration which are great if you need it) https://github.com/cgrand/xforms/blob/master/src/net/cgrand/xforms/rfs.cljc#L134-L143

In that sense, xforms is mostly “just” a library – simple collection of functions – and not a brand new syntax/way to do something

Yeah I don’t think it’s fair at all to compare xforms to Spector. There are a couple of goofy things, but a) it covers things missed by core (e.g. partition) and b) it plays within Clojure. It doesn’t invent novel abstractions.

My gripe with it comes from what I see in the wild and in some recommendations here. You learn about str! and a minute later you wrap transjuxt inside transjuxt inside by-key just to avoid a simple loop, and later can't disassemble it in your head without rewriting it completely, unless you use those kinds of constructs all the time.

Well you did identify the trickiest ones out of the lot, I’ll give you that! 🙂 If a simple loop is all you need, that’s fine. But sometimes the problem space is inherently both harder and not fully known, and an initially trivial loop might end up “growing” with a bunch of if’s and when’s and cond ’s inside… at which point you’re doing very imperative style programming… Using the right transducer for the job doesn’t make a hard problem disappear, but at least keeps it composable and various sub-problems compartmentalized within their own transducers.

… all while being able to maintain the execution “flow” as stream-like as possible, without having to constantly realize everything in every step. I have tried and done that pre-transducers (with loop , to extract max perfomance) and it ain’t fun…

Here’s some very old code that I wrote some years ago, before transducers existed, trying to “sip” things via loop – don’t ask me what’s going on there… 🙂 It’s very imperative but it was decently fast… with transducers would probably be almost just as fast and much more clear what’s going on… https://github.com/raspasov/neversleep/blob/master/src/neversleep_db/node_keeper.clj#L262-L323 I was relatively new to Clojure but I was very aware of all the functional ways of doing things and I still intentionally chose loop . Just showing it as an example of what loop that needs to be efficient/incremental can eventually morph into (not very pretty)

(I think transducers were released midway through that project… there’s some in the project but some of the code is a bit older than that)

And those problems I linked with the loop are not very hard… they are just doing “annoying” IO stuff, nothing actually novel/hard. There are harder problems.

> an initially trivial loop might end up “growing” with a bunch of if’s and when’s and cond ’s inside… at which point you’re doing very imperative style programming… Which is still totally fine in some cases. :) And conversely, starting to rely on xforms when you're still in that "growing" phrase might limit you to a point where instead of a simple imperative loop with a few conditions you end up writing a menagerie of stateful transducers that aren't used anywhere else, only to concoct something inscrutable. I agree that when you have a very fixed problem and when you already know all the relevant bits of xforms, using it might make sense. > don’t ask me what’s going on there… I don't understand why it's a loop in the first place - there's no recur?.. It just does a one-shot something and exits. Or am I missing something?

… that is a good question 😅 I am not going to try to uncover the old bodies to figure out why… might be just a mistake/extra loop Here’s another one with (recur...) https://github.com/raspasov/neversleep/blob/master/src/neversleep_db/command_log.clj#L72-L107

Side note: that was the first and last time I intentionally chose to use protobuf btw 😅

recur is another source of imperative added complexity… like are we going to reach that recur , or no… but overall I don’t have anything else to add to the discussion except that as a whole I mostly agree with you.

The second example to me is very straightforward, my only gripe with it is how many items there are in [...] after loop - it makes it harder to see what goes where at a glance. A few ways to fix it, and maybe just one of them would be enough: • Formatting, so that each item in [...] not only has its own line in [...] but also its own line in any recur • Combining multiple accumulators into an accumulator of tuples, since they're always modified together • Things that are changed separately can go into a single or a few state maps But now that we've seen the "imperative horror" - how would it look with transducers? I have no meaningful intuition for xforms, so I can't do the conversion myself, otherwise I would.

But yeah - the code is sufficiently complex but the comments actually they do help to re-understand what’s going on… It’s partitioning write by some id, and buffering up to 1ms before writing/sending data elsewhere.

Well, with trasducers it would be something like

(comp 
  (partition-by :blob-id)
  ;pseudo code, wouldn need to be created as a new transducer
  (buffer-for-max-time)
  ;etc
  )

Oh, and there's a duplicate branch in the logic - I would definitely try to rearrange the logic so that there are no duplicate branches. Something I usually do in such cases is to have something like an action value that's a simple keyword that gets cased into an actual action and a corresponding recur at the very end of a loop. The decision on which action to take gets done earlier in the loop, via that branching logic.

The trouble with "something like" demonstrations it that it's very easy to get the gist of something but can trivially become very hard to get the last detail correct. :)

I don’t think xforms specifically has some unique transducer to help with in this case, potentially the window one? But I haven’t looked into in depth

Well yes : ) It is overly simplified, it will get more complex; one of the main benefits over loop would be the clear separation of steps, while preserving the flow of data The loop at least as written, complects a sort of a “partition-by” (checking of we are still the same blob-id, etc) with the packing logic, and the sending logic; And I agree - are there ways to deduplicate the loop and make it more composable? Most likely yes. What I personally like about transducers in general (has nothing to do with xforms lib) is that they constantly and actively push you to separate your logic into steps, composed with comp

Vs when you get into the … loop you always add one more thing, and one more thing, until you are like … “oh I am getting data that I need to stop at, or skip, I can just add one more check”… Vs. if you use transducers, I am like “oh, ok, easy, just add (filter ...) as another step, virtually no cost of doing that”

Ultimately, both approaches can be structured well, but I think transducers give “just enough” structure to encourage separation of concerns… you just immediately know when you have (comp (map (fn [x] #_this_grows_very_big_logic_in_here_for_x))) … that you can separate something… And if you need to skip an item, you just don’t do it inside map, you add a (filter...) , if you need to partition items, (partition-by...) etc In a loop that is processing infinite/continuous data, you can’t just can’t do that “in the middle”… or at least not as straightforward as adding a filter /`partition-by`

> one of the main benefits over loop would be the clear separation of steps, while preserving the flow of data Just the fact that steps can be separated does not mean that the code will become more clear. It can easily be the opposite. That's exactly why I'm talking about details. You can have 5 stateful transducers separated by other transducers where the code of each transducer cannot be used in isolation and they all must be used together, in a very specific chain of transducers.

Oh that’s bad… yeah…

“each transducer cannot be used in isolation and they all must be used together, in a very specific chain of transducers” Ideally, that’s a one… transducer

And you can … do that… since they compose trivially.

Each transducer should be a usable piece, by itself, in a variety of contexts

If they need to be used in order, something has gone quite wrong…. they are basically all one transducer…

> Ideally, that’s a one… transducer But each step that's not custom has a perfectly fitting transducer from xforms. ;)

> If they need to be used in order, something has gone quite wrong…. Transducers always have to be used in a specific order, so I'm not sure what you mean. If you have a filter that needs a key :x, then the transducer that adds that key must always be before filter.

In any case, here's how I'd write that loop:

(defn protobuf-loop! []
  (letfn [(with-new-timeout [state]
            (merge state
                   {:last-fsync-ts (System/nanoTime)
                    :timeout-chan  (async/timeout timeout-length)}))]
    (loop [protobufs []
           confirm-chans []
           state (with-new-timeout {:blob-id                 -1
                                    :num-of-cmds-written     1
                                    :pipeline-command-log-ch nil})]
      (let [nano-secs-since-fsync (- (System/nanoTime) (:last-fsync-ts state))
            same-time-frame? (<= nano-secs-since-fsync 1000000)

            [[data api-confirm-ch pipeline-command-log-ch blob-id] _]
            (when same-time-frame?
              (alts!! [command-log-chan (:timeout-chan state)]))

            write! (fn []
                     (let [{:keys [blob-id num-of-cmds-written pipeline-command-log-ch]} state]
                       (write-protobufs-and-confirm blob-id confirm-chans protobufs num-of-cmds-written pipeline-command-log-ch)))]
        (cond
          (or (not same-time-frame?)
              ;; Happens via `alts!!`.
              (nil? data))
          (do (write!)
              (recur [] [] (with-new-timeout state)))

          (not= blob-id (:blob-id state))
          (do (write!)
              (recur (conj [] data) (conj [] api-confirm-ch)
                     (with-new-timeout {:blob-id                 (long blob-id)
                                        :num-of-cmds-written     1
                                        :pipeline-command-log-ch pipeline-command-log-ch})))

          :else
          (recur (conj protobufs data) (conj confirm-chans api-confirm-ch)
                 (update state :num-of-cmds-written inc)))))))

(defn start-write-protobuf-loop []
  ;; Using the value and setting it must either be an atomic operation or in a critical section.
  ;; The old version of the code is prone to multiple threads.
  ;start loop only once
  (when (compare-and-set! loop-started? false true)
    (thread
      (protobuf-loop!))))

Transducers always have to be used in a specific order, so I’m not sure what you mean.> If you have a filter that needs a key :x, then the transducer that adds that key must always be before filter. Well.. obviously… Transducers don’t just automagically make the sequential nature of most programs disappear 😆. But filter is not even stateful, you were referring to imaginary “stateful” transducers that would only work in a certain order… I was imagining something stateful that would literally be required to be after another stateful thing, in any and all cases. I’ve never seen such a thing but it would be very bad design, in most cases.

Are you questioning the value of transducers, as a whole (disregarding the xforms library)?

> I was imagining something stateful that would literally be required to be after another stateful thing, in any and all cases. > It can indeed be the case, I just reached for filter because it's on the surface. I didn't mean that a state of one transducer will be required for the next transducer, just that their logic and results are intertwined. > Are you questioning the value of transducers, as a whole (disregarding the xforms library)? > No, I'm questioning the knee-jerk reaction to reach to some third-party library with a lot of implications when str/join is likely to be enough. :) Or to reach for it without even a clear view of what the actual transducers should be, when a solution with loop is very simple, straightforward, and easily extensible.

I agree with keeping everything as few dependencies as possible, that’s one of the things that the Clojure community mostly does quite well; I don’t mind as much if we are just referencing a 3 line fn that is quite efficient and well written, even if the library does have some more … scary stuff. A lot of the smaller fns can literally be copy-pasted out of it and work as-is. str/join definitely works also 😉

We were talking about grabbing just a reducing fn, not even a transducer out of it! 😜 Anyhow, good chat, ttyl.