Clojurians Log v2

Nested expansions of some Clojure macros, perhaps also doseq , exhibit exponential growth in expanded code size. IIRC because some part of the expansion of each doseq includes the same code 2 times, so that part doubles at each nesting level.

https://clojure.atlassian.net/issues/?jql=textfields%20~%20%22doseq%22&selectedIssue=CLJ-2900

Known of at least for and doseq macros, according to JIRA linked above.

is this kind of problem caused by too much code, or too many local variables? I can start refactoring into smaller pieces, but if I rob peter to pay paul I wont be any better off.

The culprit (I think) is a doseq with 10 consecutive loops. That doesn’t seem too big to me.

Functions are compiled into jvm classes, with the body being a single method. jvm methods have a maximum size of 65535 bytes

the doseq has 10 nested loops?

OK, and if there is a local (fn [] …) inside a function, does that add to the size?

(doseq [dfa-1 dfas-1
            dfa-2 dfas-2
            dfa-and (with-timeout timeout-sec []
                      [(xym/synchronized-intersection dfa-1 dfa-2)])
            dfa-and-not (with-timeout timeout-sec []
                          [(xym/synchronized-and-not dfa-1 dfa-2)])
            dfa-empty-word (with-timeout timeout-sec []
                             [(rte-to-dfa :epsilon exit-value)])
            dfa-and-non-trivial (with-timeout timeout-sec []
                                  (xym/synchronized-and-not dfa-and dfa-empty-word))
            subset (with-timeout timeout-sec []
                     [(xym/dfa-subset? @dfa-1 @dfa-2)])
            overlap (with-timeout timeout-sec []
                      [(xym/dfa-inhabited? dfa-and)])
            non-trivial-overlap (with-timeout timeout-sec []
                                  [(xym/dfa-inhabited? dfa-and-non-trivial)])
            :let [[satisfiability path] (get (xym/find-trace-map @dfa-1) exit-value)]
            ]
 ...)

and with-timeout is a macro

i don't remember the specifics of how doseq expands, but i'm unsurprised this is too big

i removed a couple of the loops and the problem went away.

While 10 loops feels intense, I agree that it being a syntax error feels wack

Emmas-MacBook-Pro :: ~ % clj
Clojure 1.12.3
user=> (for [i (range 1)] (for [j (range 1)] (for [k (range 1)] (for [l (range 1)] (for [m (range 1)] (for [n (range 1)] (for [o (range 1)] (for [p (range 1)] (for [q (range 1)] (for [r (range 1)] [i j k l m n o p q r]))))))))))
(((((((((([0 0 0 0 0 0 0 0 0 0]))))))))))

yeah i think it's the complexity of the calls

I do wonder if there's an equivalent 10 levels deep of representing this that doesn't crash deep_thinking

https://gist.github.com/ghadishayban/fe84eb8db78f23be68e20addf467c4d4

Old thing I made for doseq. Doesn’t handle :let, :while etc, or optimize chunked seqs.

When you adding extra details, it's better to do so via a thread or an edit to the previous message, to avoid having multiple "heads" for threads. In any case, what's the original "java-esk code"?

Here is the scala code

def write(str: String): Unit = {
      for {c <- str} {
        dotStream.write(c.toByte)
      }
    }

Here’s the clojure code

(letfn [(write [str]
            (doseq [c str]
              (.write dot-stream (.toByte c))))
...

scala has a lot of stuff like implicits

There is an implicit conversion from scala.Char => scala.runtime.RichChar which provides useful non-primitive operations.

ahhhh, so it might be convering the java.blah.blah.Char into something which can be coverted to a byte?

in in clojure how should I write a string to a stream which has been created by

(java.io.FileOutputStream. (java.io.File. dot-path-name))

io/copy is what i go to to handle all that junk

(let [baos (java.io.ByteArrayOutputStream.)]
  ( "some input!" baos)
  (str baos))
"some input!"

which i think will go through

(defmethod do-copy [String OutputStream] [^String input ^OutputStream output opts]
  (do-copy (StringReader. input) output opts))

(defmethod do-copy [Reader OutputStream] [^Reader input ^OutputStream output opts]
  (let [^"[C" buffer (make-array Character/TYPE (buffer-size opts))
        out (OutputStreamWriter. output (encoding opts))]
    (loop []
      (let [size (.read input buffer)]
        (if (pos? size)
          (do
            (.write out buffer 0 size)
            (recur))
          (.flush out))))))

sorry, I don’t follow. do I need to convert through io/copy as youve shown above and then pass that value to .write?

or iterate over it and and all .write on each iteration?

(let [baos (java.io.ByteArrayOutputStream.)]
  ( "some input!" baos)
  (str baos))
"some input!"

i think this does in its entirety what you want to accomplish, by putting a string into some output stream

the rest was how it accomplishes that, implicitly saying it’s annoying to bother with buffers and loops and io/copy is pretty great

> in in clojure how should I write a string to a stream which has been created by >

(java.io.FileOutputStream. (java.io.File. dot-path-name))

i used a bytearrayoutputstream for simplicity but could change it to write to a file and then slurp it to demonstrate.

Ahh chatgpt suggested something elegant. seems to work

(letfn [(write [s]
            (doseq [c s]
              (.write dot-stream (byte (int c)))))]
  (write "Hello"))

token-check-test=> (let [input "this is a string"
      baos (java.io.ByteArrayOutputStream.)]
  (letfn [(write [s]
            (doseq [c s]
              (.write baos (byte (int c)))))]
    (write input)
    (str baos)))
"this is a string"
token-check-test=> (let [input "this is a string 😱"
      baos (java.io.ByteArrayOutputStream.)]
  (letfn [(write [s]
            (doseq [c s]
              (.write baos (byte (int c)))))]
    (write input)
    (str baos)))
Execution error (IllegalArgumentException) at metabase.premium-features.token-check-test/eval604279$write (REPL:417).
Value out of range for byte: 55357

might not be as elegant as you think

token-check-test=> (let [input "this is a string 😱"
      baos (java.io.ByteArrayOutputStream.)]
  (letfn [(write [s]
            (doseq [c s]
              (.write baos (byte (int c)))))]
    (write input)
    (str baos)))
Execution error (IllegalArgumentException) at metabase.premium-features.token-check-test/eval604279$write (REPL:417).
Value out of range for byte: 55357
token-check-test=> 
(let [baos (java.io.ByteArrayOutputStream.)]
  ( "this is a string 😱" baos)
  (str baos))
"this is a string 😱"

oh i see, my code is assuming every character can be converted to a byte?

(map int "a 😱")
(97 32 55357 56881)

also note that bytes are signed on the JVM so even something slightly outside the ASCII range will break it:

(byte (int \£))
;=> Execution error (IllegalArgumentException)
;   Value out of range for byte: 163

and also a reason why if you just want to copy a string to an output stream, just copying the bytes with io/copy lets you just faithfully copy the bytes themselves without having to understand the semantics in the intermediate bits. Just verbatim copy

> oops .indexOf returns -1 if not found. Yeah, it's a good idea to get familiar with the underlying java stdlib.

(fn [v x] (first (keep-indexed #(when (= %2 x) %1) v))))

.indexOf should work

This exists: (.indexOf [1 2 3] 2)

even if the object being searched for is clojure data?

oops .indexOf returns -1 if not found.

funny

Well, it sounds like this is a frequently-used lookup operation... Perhaps keeping an up-to-date reverse-index in memory will help?

[1 3 5 3 3 4] -> {1 0, 3 1, 5 2, 4 5} OR {1 [0], 3 [1 3 4], 5 [2], 4 [5]}