> :column 1 My immediate assumption, without checking, is that the last character is \newline.

That's my assumption as well, but I guess I don't know where that newline is. 🙂 There's no newline in the string I'm reading.

Ah, I failed to read the line you feed to the REPL correctly - I thought it's your line that had an issue and not something you put in a string there.

I would try wrapping it in a function and calling the function

Or maybe not

Doesn't seem to make a difference.

The compiler uses dynamic bindings to pass around position information if I recall, and it maybe that the state there from the repl's call to eval is cross talking with the code being evaled

Or it might still be the issue of there being an implicit newline, so to speak:

(def r (clojure.lang.LineNumberingPushbackReader. (java.io.StringReader. "(")))
=> #'dev/r
(.getLineNumber r)
=> 1
(.read r)
=> 40
(.getLineNumber r)
=> 1
(.read r)
=> -1
(.getLineNumber r)
=> 2

Hmm, maybe. I can call .setLineNumber on the LineNumberingPushbackReader, but the result is still confusing:

user=> (let [reader (clojure.lang.LineNumberingPushbackReader. (java.io.StringReader. "("))]
  (.setLineNumber reader (int 0))
  (clojure.lang.Compiler/load reader))
Syntax error reading source at (REPL:1:1).
EOF while reading, starting at line 0

Oh, that's probably it. :thumbsup:

It doesn't feel quite right to me, though. :thinking_face:

To kinda interpret every string as ending in a newline, that is. I haven't yet looked at the code for LineNumberingPushbackReader to see what's going on there, but this is confusing:

user=> (clojure.lang.Compiler/load (clojure.lang.LineNumberingPushbackReader. (java.io.StringReader. "(\n")))
Syntax error reading source at (REPL:2:1).
EOF while reading, starting at line 1

So this is interesting:

λ java -version
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)
λ clj -Srepro -M -e '(clojure.lang.Compiler/load (clojure.lang.LineNumberingPushbackReader. (java.io.StringReader. "(defn square [x] (* x x)")))'
Syntax error reading source at (REPL:1:1).
EOF while reading, starting at line 1

Full report at:
/var/folders/yb/qchyz_mx5w97ncww09ffrfz00000gp/T/clojure-17057255385196411412.edn

λ jabba use [email protected]
λ java -version
openjdk version "17" 2021-09-14
OpenJDK Runtime Environment (build 17+35-2724)
OpenJDK 64-Bit Server VM (build 17+35-2724, mixed mode, sharing)
λ clj -Srepro -M -e '(clojure.lang.Compiler/load (clojure.lang.LineNumberingPushbackReader. (java.io.StringReader. "(defn square [x] (* x x)")))'
Syntax error reading source at (REPL:2:1).
EOF while reading, starting at line 1

Full report at:
/var/folders/yb/qchyz_mx5w97ncww09ffrfz00000gp/T/clojure-6224778322342884943.edn

Seems relevant: https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8241020

Good find! :thumbsup:

Yep, case closed, I think. Thanks for the help. 🙂

Hi @U02CVMEFEUF I'm the author of μ/log. Trace events are just plain events so they can be filtered using the https://cljdoc.org/d/com.brunobonacci/mulog/0.8.2/doc/custom-transformations which all the built-in publishers support. the basic usage is:

(u/start-publisher!
  {:type :console
   :transform (fn [events] events)})

(u/start-publisher!
  {:type :console
   :pretty? true

   :transform
   (fn [events]
     (remove #(= (:mulog/event-name %) :app1/my-noisy-event) events))})

Similarly, if you wanted to remove all trace events (which I discourange) you could remove anything which has a :mulog/duration field.

(u/start-publisher!
  {:type :console
   :pretty? true

   :transform
   (fn [events]
     (remove :mulog/duration events))})

Feel free to post a discussion in Github if you need something else: https://github.com/BrunoBonacci/mulog/discussions

thanks a lot.

There's no robust way to get a version of a library, unless you control that library and put the version information there yourself.

> unless you control that library and put the version information there yourself. Supposing this assumption were true, how would I do it?

Just add a def into that library that has the version. :)

Exactly how we already have *clojure-version*.

Ah okay. Got it.

but beware that you have to conditionally check for it since there are versions released without it. You’ll need to check for the presence of the var before actually using it

to tag onto @U11BV7MTK - the easy way to do that is resolve - returns nil if the symbol is not bound

(ins)user=> (resolve 'foo/bar)
nil
(ins)user=> (resolve 'clojure.core/+)
#'clojure.core/+

Split the input into N parts, create a threadpool of size N, in each thread create a connection and process a single part sequentially, in the end combine all the results.

Unless you need for it to work in a lazy manner.

And with core.async, you can use pipeline with a set parallelism and a custom connection factory that would create and cache one connection per thread. After the work is done, you'd get rid of the factory and all the cached connections.

Also, perhaps https://github.com/clj-commons/claypoole could be of help but I haven't used it myself.

I looked into claypoole this morning. I may end up using that. I was hoping for something a bit more simple. Thanks for the pipeline suggestion, too.

Clojure is fully crossplatform - you can compare yourself, and it would make more sense because it would be working on the same machine.

I'm hoping to find someone curious to give it a try with me

Then nobody needs to know both

Yeah, but the results won't be comparable, because the machines will be completely different.

Both benchmarks would be run on the same machine

Ah, I see what you mean.

Although seems like a Clojure equivalent to a list in C# would be a vector and not a list.

I've done the .net side, and was hoping to find someone curious like myself to write the equivalent Clojure code

It's an ImmutableList, so basically a Persistent Data Structure that acts like a list

I get that. I meant from the sensible API point of view. In C#, a list is a collection that supports random access. In Clojure, that would be a vector, even though it implements Java's List.

Is it immutable?

Yes.

In .net, ImmutableList you can randomly access by index, and when you want to Add it gives you a new object instance for the ImmutableList with the value added. Internally it uses a binary tree I think

I have read that Clojure uses an n-ary tree

Which is why I am interested in comparing the speeds

I can write Clojure code that would do what you're doing in the "ImmutableList<T> benchmarks" section, but I can't run the .NET code and I can't promise that the result will be 100% equivalent. E.g. I.d implement "remove at zero" as creating a subvector in Clojure - I have no idea what .NET does here, whether it creates a fully new list or not.

I don't know a thing about .NET, so I have to ask - what's the difference between (l, x) => l.Add(x) and (l, x) => l = l.Add(x)?

Ah, because I am using the same code to do the loop I am passing an functions to do things like 1: Create the instance (List, ImmutableList, whatever) 2: To add values 3: To remove values

the tests are

1: How long does it take to add numbers sequentially 2: How long does it take to end up with an empty list when removing the last element (position length - 1) 3: When removing the first element (position 0) 4: When removing an element with a specific value that will be found at the start of the list 5: When removing an element with a specific value that will be found at the end of the list

So 1,2,3 are about mutations. 4 and 5 are about traversing the collection to find a single element

You should benchmark iteration as well, how quickly you can iterate over it

It's still more confusing to me that it could (or maybe even should) be. 1 and 2 - alright, got it. 3 - just as in 2, till you end up with an empty list? If so, Clojure's vectors are not optimized for that. Clojure's lists are proper linked lists - adding and removing items at the head makes the most sense. Clojure's vectors are n-ary trees indeed, where n is 32, and they're optimized for tail modification and random access. 4 and 5 - same as above, vectors aren't really for that. If that's something you need often, a different data structure should be considered. So, judging by just the .NET's List API, seems like comparing it to any single Clojure data structure would be comparing apples to oranges.

If so, it would be better to come up with scenarios that don't depend on any concrete type and then benchmark reasonable implementations of those scenarios, akin to Rosetta code.

Oh, I was under the impression that in Clojure it was optimised for both addition and removal. Is that not the case?

Addition - only at the tail. Removal - if a proper one, when the removed items can be garbage collected, then also only at the tail. But you can simulate removal with subvectors - those behave like vectors but retain a reference to the main vector. Lists are the reverse of that, with some additional differences, like not supporting fast random access and not having anything like subvectors.

I don't think the ImmutableList of C# optimizes to all these either though. I think they just offer all the convenience functions no matter the performance. That's my guess at least.

What does your test results for C# shows? Any one of them seems a lot slower than the other?

ImmutableList is only slow for finding values

1 sec

Note I had to peform List.Add 100 times more to get a benchmark

This is with what size list?

;; Size of n to test for.
(def n 1000000)

;; 1: How long does it take to add numbers sequentially

;; 18.9 msec
(time
 (do (into [] (range n))
     nil))

;; 2: How long does it take to end up with an empty list when removing the last element (position length - 1)

;; 79.3 msec
(let [v (into [] (range n))]
  (time
   (loop [v v]
     (when (> (count v) 0)
       (recur (pop v))))))

;; 3: When removing the first element (position 0)

;; 80.7 msec
(let [v (into [] (range n))]
  (time
   (loop [v v]
     (when (> (count v) 0)
       (recur (subvec v 1))))))

;; 4: When removing an element with a specific value that will be found at the start of the list

;; 49.3 msec
(let [v (into [] (range n))]
  (time
   (do (filterv (complement #{0}) v)
       nil)))

;; 5: When removing an element with a specific value that will be found at the end of the list

;; 48.7 msec
(let [v (into [] (range n))]
  (time
   (do (filterv (complement #{(dec n)}) v)
       nil)))

There'd be ways to optimize some of those specific scenarios, like with the find case, if you find at the start or end you can more easily pop or subvec, where as here I went with the more generic, no matter where I find it, which means I have to recreate a new vector which is slower.

Here it is with some of the more optimal ways that I could think of:

;; Size of n to test for.
(def n 1000000)

;; 1: How long does it take to add numbers sequentially

;; 18.9 msec
;; Already optimal
(time
 (do (into [] (range n))
     nil))

;; 2: How long does it take to end up with an empty list when removing the last element (position length - 1)

;; 79.3 msec
(let [v (into [] (range n))]
  (time
   (loop [v v]
     (when (> (count v) 0)
       (recur (pop v))))))

;; More optimal way
;; 28 msec
(let [v (into [] (range n))]
  (time
   (loop [v (transient v)]
     (if (> (count v) 0)
       (recur (pop! v))
       (persistent! v)))))

;; 3: When removing the first element (position 0)

;; 80.7 msec
;; Already optimal because Clojure subvec doesn't work on transient yet
;; see: 
(let [v (into [] (range n))]
  (time
   (loop [v v]
     (when (> (count v) 0)
       (recur (subvec v 1))))))

;; 4: When removing an element with a specific value that will be found at the start of the list

;; 49.3 msec
(let [v (into [] (range n))]
  (time
   (do (filterv (complement #{0}) v)
       nil)))

;; More optimal way
;; 1.3 msec
(let [v (into [] (range n))]
  (time
   (do (let [length (count v)
             i (.indexOf v 0)]
         (cond (= 0 i)
               (subvec v 1)
               (= (dec (count v)) i)
               (pop v)
               :else
               (let [v1 (subvec v 0 i)
                     v2 (subvec v (inc i))]
                 (into v1 v2))))
       nil)))

;; 5: When removing an element with a specific value that will be found at the end of the list

;; 48.7 msec
(let [v (into [] (range n))]
  (time
   (do (filterv (complement #{(dec n)}) v)
       nil)))

;; More optimal way
;; 24.4 msec
(let [v (into [] (range n))]
  (time
   (do (let [length (count v)
             i (.indexOf v (dec n))]
         (cond (= 0 i)
               (subvec v 1)
               (= (dec (count v)) i)
               (pop v)
               :else
               (let [v1 (subvec v 0 i)
                     v2 (subvec v (inc i))]
                 (into v1 v2))))
       nil)))

Do you have .net installed so could build + run some .cs benchmark code?

I am curious how it compares on the same machine

https://github.com/mrange/fsharpadvent2016

Not exactly what you are trying to do I think, but you may find it interesting. (Clojure map on CLR is to the right on the chart above)

I don't have c# sorry

Not entirely sure what the intended use case is, but sounds like just reading the source file and finding the form that defines the right function by name would be a solution.

It should be quite similar. The only reason that the classpath is relevant is that that’s how they find the resources/files. The source macro uses the var passed in to look up the var and get the namespace information. and namespaces must be findable on the classpath. If you come up with some other mechanism to give a file and symbol you can read the file for it. But now it is two bits of information instead of one (the var and from that the file)

but i’m also confused if you are working on a library why it isn’t on the classpath

right, so here's the thing I'd like to build: sometimes, when I'm exploring a codebase which isn't mine, maybe because I would like to do a PR, I want to understand how the execution of a function works. So suppose I have:

(defn foo [a b] (inc (* a b)))
(defn bar [a b c] (+ (foo a b) (foo a c)))

(bar 1 2 3)

(bar 1 2 3)
(+ (foo 1 2) (foo 1 3))
(+ (inc (* 1 2)) (foo 1 3))

@U11BV7MTK I mean, I can't do (source bar) in the example before, and I guess that is because the library I'm working on is not on the classpath, but only the dependencies (? please correct me if this is wrong)

@U2FRKM4TW yes that's possible, I kinda hoped someone knew of a solution that spared me some work; maybe rewrite-clj is what I should be reaching for (and using it just for parsing)?

oh neat. it’s like macroexpansion but function expansion. I can think of lots of problems that you’ll have to fix (polymorphism comes to mind immediately, figuring out what protocol body might come through, multimethod body, reify, all of which are runtime values and not statically determinable). But this could be a really cool tool

you’ll also get into figuring out variable scopes and shadowing when you substitute the actual arguments into the body

yes, it's exactly that, but with a bit of UI on top to let you choose what you want to expand (keeping macros in mind). For the parts that have "multiple choices" (polymorphism, protocols etc) I'd probably want the user to eval enough of the datastructure so that we know which path we want.

indeed, I'll need renaming. So I'm kinda interested of what kind of tools are already there to not duplicate effort

the obvious go-to is clj-kondo which should handle all of this for you (the analysis). You’ll have to piece it back together though

thank you, I never looked into how much analysis clj-kondo is capable of doing. It could very well be all I need!

it of course has an opaque macro barrier. so you’ve got your work cut out for you. But since you are making something like does expansion perhaps you can move past that barrier

what do you mean? which opaque macro barrier?

https://github.com/clj-kondo/clj-kondo/blob/master/doc/config.md#unrecognized-macros it doesn’t expand arbitrary macros

Oh I see, yes but I'm cool with expanding the macros I need myself - in fact I don't think I can avoid that work in the general case - for now I just want the source with as much AST annotation I can get, to help me with shadowing!

What you're talking about sounds like symbolic evaluation. If so, typed Clojure might be a place to look as well.

Something else you could do is mess with the reader or with defn. If you could attach the form as metadata to a var definition, would that work?

@UK0810AQ2 I might consider that, but in practice I also need to keep in mind the ns declaration in which the function is defined (so that I can disambiguate the calls inside the function). So I'm searching more for a tool that can parse all of the file and do some correlated operations

FWIW I've seen both, can't really say which is more frequent. If I had that question, I'd approach it from the perspective of the end user - what's needed more often, the implementations or the protocols? And you might even end up with both m.n.proto and m.n.impl with m.n having some factory functions or something like that.

generally, as protos are interfaces, I find it avoids a lot of entangling if you put them in a separate namespace than the impls

but I've probably done all the options at one point or another :)

right, i was more thinking if it's "better" to say (require '[my.namespace.proto :refer [cool-fn]]) or (require '[my.namespace :refer [cool-fn]]) . i know so much of this is just opinion and style, but i like to stick with the community standards when possible

my other piece of advice is that it's often useful to front protocol functions with a wrapper function

I know that seems like a lot more ceremony, but I have found that to be useful so many times now that I almost always just start that way

then the function wrapper is the api, and the protocol function becomes more like an spi

iiiiiinteresting, yeah

how does that interact with extending a protocol? or i guess it doesn't? you still extend the protocol like normal but then call the wrapping function in practice

yep

idk if you've written about this elsewhere (applied clojure maybe? just got my copy yesterday lol), but i would love to hear more about the benefits you've encountered

I'm not sure I had come to this when I wrote that yet, don't remember. don't think I've written about it anywhere, but it's something Rich and I have talked about and regularly do. spec (at least spec 2) is set up this way - there's a protocols namespace, then the main api which wraps and calls into the protocol functions, and then an impl (although that split is not necessarily interesting)

it gives you a place to put defaults, aspect stuff like logging, adaptation into other protocols, etc

cool, thank you

FWIW, it's a lesson I wish I'd learned before writing next.jdbc -- that mostly has the protocols separate, but I added some elsewhere in the library and it made both using the library and maintaining it more complex.

nnext.jdbc when? :)

so both implementation will live in the same system but only one will get started?

it depends how you use configuredchoice

the configuredchoice component replaces itself with whatever choice it makes when you start it

an instance of configuredchoice looked like

(map->ConfiguredChoice
 {:key :storage-backend
  :mysql (fn [this]
           (mysql/map->Store (assoc this :db (:database this))))
  :local (fn [this]
           (log/info "using local storage")
           (store/->LocalStore))})

so once it is started it would be replaced by the result of one of those functions

the key is to understand that start and stop on components are not side effecting functions that return nil, they return "updated" versions of the component they are called on

and those updated versions can be whatever you want, they don't have to be the same type

That's what I'm doing now, I'm trying to outsmart myself

Annoying part here is that I don't have C->B but B->C but only if I have one URI scheme and not another

Managed to figure it out with both protocols and multimethods

Or approach #3, have "create a C" be a logical method on B. Dummy B creates dummy C

still ultimately dependent on the URI, just a little more detached

Here's a https://clojurians.slack.com/archives/C03S1KBA2/p1651005353239669 (which became its own thread with some building blocks). Still seeking a working solution.

Calling (meta #'the-ns/the-fn) will give you the metadata and it will have a key :doc that you can apply to the wrapping function.

There's a bit of machinery at the end of expectations.clojure.test that pulls in functions from clojure.test and reuses their metadata but adds "Imported from clojure.test" to the docstrings: https://github.com/clojure-expectations/clojure-test/blob/develop/src/expectations/clojure/test.cljc#L540-L560

(it's complicated by catering for self-hosted cljs via planck as well as Clojure but...)

In use:

user=> (require '[expectations.clojure.test :refer :all])
nil
user=> (doc run-tests)
-------------------------
expectations.clojure.test/run-tests
([] [& namespaces])
  Runs all tests in the given namespaces; prints results.
  Defaults to current namespace if none given.  Returns a map
  summarizing test results.

Imported from clojure.test.
nil

user=> (doc clojure.test/run-tests)
-------------------------
clojure.test/run-tests
([] [& namespaces])
  Runs all tests in the given namespaces; prints results.
  Defaults to current namespace if none given.  Returns a map
  summarizing test results.
nil

Dang, being just data keeps getting cooler the more I learn about it

Another way to deal with that could be something like Emacs's advice mechanism or middleware. Problem is I'm not sure how it would play with compilation and direct linking

Finally came back to this. Thanks a lot, Sean. The intern-based solution worked. Observe the (defn wrap-fn solution works but Clj-Kondo underlines in red the produced var, obviously. Then, a lightweight (defmacro defwrapped wraps the call to make it understandable by clj-kondo. In this example, note it only allows to prefix and suffix docstrings.

This current implementation allows us to give a new name to the wrapped function. Not sure if it's really a good idea. The most important next step is to weed out how to override the original fn's body/bodies.

Screenshots are not a good way to share code but I suspect you are only calling your wrapper with a single argument?

(defn wrap-print [what-to-print]
  (fn wrapper [wrapped-fn]
    (fn with-print [& args]
      (print what-to-print)
      (apply wrapped-fn args))))

(defmacro wrap-fx
  [& body]
  `(let [s# (new java.io.StringWriter)]
     (binding [*out* s#]
       (let [r# ~@body]
         {:ret r#
          :out (str s#)}))))


(intern *ns* '+ ((wrap-print "Hello") clojure.core/+))
(wrap-fx (+ 1))   ; => {:ret 1, :out "Hello"}
(wrap-fx (+ 1 1)) ; => {:ret 2, :out ""}

Towards more simplicity

(clojure.core/defn wrap-add [what-to-add]
  (fn wrapper [wrapped-fn]
    (fn with-add [& args]
      (+ what-to-add
         (apply wrapped-fn args)))))

(intern *ns* '+ ((wrap-add 100) clojure.core/+))
(+ 1)   ; 101
(+ 1 1) ; 2

Try it with something other than a clojure.core function.

Several core functions are optimized and inlined for different arities. Core is also AOT compiled and direct linked so overriding parts of core can be tricky -- it doesn't behave like user-space functions in some areas.

Oh, looks like you got it... 🙂 will report back

A workaround would be to use a wrapper around clojure.core/+ for this case:

user=> (defn plus [& args] (apply + args))
#'user/plus
(defn wrap-print [what-to-print]
  (fn wrapper [wrapped-fn]
    (fn with-print [& args]
      (print what-to-print)
      (apply wrapped-fn args))))
#'user/wrap-print
user=> (intern *ns* '+ ((wrap-print "Hello") plus))
#'user/+
user=> (+ 1)
Hello1
user=> (+ 1 1)
Hello2
user=> 

haha I was trying this, but it gives a weird error:

(defn my-sub [& args] (apply - args))
; No value supplied for key: (apply - args)

user=> (defn my-sub [& args] (apply - args))
#'user/my-sub

Or maybe you overwrote defn?

oh yes, that's right, I had overwritten defn and forgot to use core's one. Developing and testing in the same file makes things easier, until it conflates them due to these clashing specificities.

Overriding core stuff is nearly always problematic 🙂

Thanks - I'll rewrite my tests in terms of other code than core's. :)

This did it, thanks again Sean!

You can read the string like so: (read-string (repl/source-fn 'frequencies))

even better:

(clojure.spec.alpha/conform
   :clojure.core.specs.alpha/defn-args
   (rest (read-string (clojure.repl/source-fn 'frequencies))))

which yielded:

{:fn-name frequencies,
 :docstring "Returns a map from distinct items in coll to the number of times\n  they appear.",
 :meta {:added "1.2", :static true},
 :fn-tail
 [:arity-1
  {:params {:params [[:local-symbol coll]]},
   :body
   [:body [(persistent! (reduce (fn [counts x] (assoc! counts x (inc (get counts x 0)))) (transient {}) coll))]]}]}

That seems to be baked pretty deeply into the core printing machinery: https://github.com/clojure/clojure/blob/master/src/clj/clojure/core_print.clj#L229-L236

I was about to go view the source. I am lazy and asked first. Currently I'm SOL. I can always transform to what I need before writing the files. I guess Rich had a preference for this.

Commas are whitespace so they're purely cosmetic 🙂

perhaps zprint would be tunable to your specifications

https://github.com/kkinnear/zprint