Fork me on GitHub
#clojure
<
2022-11-29
>
valerauko02:11:33

is there any way to analyze clojure/jvm startup time? it feels slow and i should be able to trim dependencies, but i don't know where to start

valerauko02:11:08

this is a dev environment and it takes almost a minute for the lein repl to start up (the nREPL server started on port... log)

Alex Miller (Clojure team)02:11:47

sometimes there is more work done in top level forms than needs to be. using the :verbose flag on require can sometimes help spot particular namespaces that are slow that way

Alex Miller (Clojure team)02:11:23

but more commonly, you're just loading a lot of classes and initializing a lot of vars and in that case, you might find the aot techniques in https://clojure.org/guides/dev_startup_time to be helpful

Alex Miller (Clojure team)02:11:55

tldr - loading and compiling your clojure code on every start is a lot of rework and there's no reason to do that for all of your deps especially (as they aren't changing)

valerauko03:11:19

tbh i don't think i have that much of my own code to cause issues, so i suspect it's gonna be my dependencies. do you happen to know how to precompile all my dependencies in lein? my deps change very infrequently so i would be fine with paying the price of a full recompile when that happens.

Alex Miller (Clojure team)04:11:10

with lein, I think you could just add :aot [your.app.namespace] to project.clj for whatever is the main class you start during dev, and then lein compile which should compile into target/classes by default. on subsequent startup, I think those will be in your classpath

valerauko05:11:56

thanks! i tried that but it didn't really work the way i hoped, so in the end what i did was 1. add :aot :all to my :dev profile 2. run lein compile 3. delete my own stuff from target/classes there were a few hiccups, i had to manually (compile) clojure.tools.reader.reader_types and clojure.tools.logging.impl (and some others) but otherwise it was smooth. startup time is now around 5 seconds and i'm okay with that i'll see if this broke anything but at this point it looks fine

Alex Miller (Clojure team)06:11:04

:aot :all is often more than you need, and subject to mis-ordering, which can be tricky with protocols and some other things. you also shouldn't need to delete your own stuff - newer timestamp .clj files take precedence over .class files so as you change your own code, any classes should fall out of use

kwladyka08:11:21

Do you use Integrant to load and prepare all stuff in the app or you have everything start up when load namespaces?

kwladyka08:11:34

Then your REPL should warm up pretty fast

kwladyka08:11:28

Maybe you are creating connections to DB etc. when starting REPL

valerauko08:11:13

i use mount for stuff like database connections, and i start those manually from the repl sadly this was just me pulling in way too many big libraries for convenience that had a pretty big loadtime cost

kwladyka08:11:44

I think it should be easy to identify which library make the higher cost by commenting half, re-run REPL, comment / uncomment another half.

kwladyka08:11:15

and maybe replace the slowest

kwladyka08:11:33

it can be some kind of bug in library which waiting for something

kwladyka08:11:43

1 minute sounds really long

kwladyka08:11:14

I think you mentioned about 1 minute, although I can’t see it right now

valerauko08:11:49

the thing is that at this point it's quite complicated and components depend on each other (such as caching stuff after a db query). that train of thought is exactly how i started this thread if you notice, asking about ways to analyze load times so i could figure out which libraries are causing it i have a few suspects (such as carmine which will expand macros for the whole redis command library), but since just precompiling it all eased the issue i'm not that desperate to dig deeper

kwladyka08:11:49

probably profiling is the most pro answer, but I didn’t try to analyse REPL time loading so far with profiling tools

kwladyka08:11:11

another way is which I mentioned to comment half of deps and re-run REPL

valerauko08:11:40

is it possible to profile the clojure (lein?) repl startup process? i've only used profilers from the repl or in web servers

kwladyka08:11:17

I think so, because it is Java process, but I have never tried

kwladyka08:11:14

can’t be sure how easy / hard is to do it

zakkor09:11:05

What would clojure be like if let bindings introduced the new binding into the current scope, rather than needing to create a new scope (thus, a new layer of indentation)?

(let x 3)
(println x)

borkdude09:11:22

@edward.partenie isn't that similar to using def? except that let bindings are immutable

mpenet09:11:51

defs are global, I think he meant (let [...] ..... (let x 3) (println x 3))

mpenet09:11:07

in any case, that's a bit weird, you loose the visual indicator from a normal let

zakkor09:11:03

^ yeah, similar, but the let just introduces the value into the enclosing/current scope. Like if you call it inside a defn, it'll be available until the defn 's scope ends. Basically exactly how let works right now, but with a bit of syntax sugar introduced

zakkor09:11:32

in my mind this is similar to how defn and def work in the top-level. Each defn doesn't introduce a new indentation level, instead the function is added to the global scope

p-himik09:11:55

Personally, I would prefer for such a syntax to never exist.

☝️ 7
1
zakkor09:11:46

@U2FRKM4TW so would you say it would be better for defn to work in the same way as let currently does? or how do we draw the line? 😄

p-himik09:11:07

By maintaining the status quo.

zakkor09:11:56

My questions are more of a thought experiment rather than a proposal to enact changes in clojure. Simply trying to understand current design decisions & maybe spark some interesting conversation related to them!

delaguardo10:11:26

personaly I would prefer to have only ns forms at the top level and everything that logically belongs to particular namespace placed within that form. Currently clojure needs to have a file in classpath with corresponding namespace with properly formatted name. It is possible to put (ns foo.bar) in the file bar/foo.clj but it will introduce too much problem at the end. There is implicit convention how to name source file and what you can write inside. But all of this seems redundant to me. Why not to have many namespaces in one file, why I have to remember that dashes in the name of a namespace should be translated into underscores in the file name. Those are minor problems but bugs me as long as I use clojure.

zakkor10:11:13

That's a good point. I think, in a way, the file itself should be the ns statement. The start of the file is the opening parens and the end of the file is the closing parens 😄

delaguardo10:11:04

I mean what is the point of introducing the limit here? 🙂 why just one namespace?

zakkor10:11:23

What bugs me is that we've accepted that it's more comfortable for defn and def to introduce their new units until the end of the current scope, because you don't need to wrap each call inside a new pair of parens, and this is a practical decision that makes everything easier to work with. But when it comes to let, which is arguably an even more common operation than defining functions, we don't have anything similar - each new variable is a new indentation level - and I don't really get why. How often do you want to control how long a let scope is? My guess is "never", you always want it to be as long as possible Additional note: In Ocaml, let bindings are expression based (like Clojure), but the syntax keeps the indent at the same level (this is something we can't really do in Clojure because we don't "have syntax").

let x = 3 in
let y = x + 1 in
x + y

zakkor10:11:09

@U04V4KLKC no clue, perhaps a decision made in the interest of keeping things simple?

delaguardo10:11:15

> My guess is "never", you always want it to be as long as possible

(defn impure-foo! [m]
  (log/debug m)
  (let [{:keys [x y]
         :or {x 1}} m]
    (submit-tuple! x y))
  (let [{:keys [x]} m]
    (log/debug x)))
it is not "never" — sometimes you need to rely on let creating a scope.

zakkor10:11:17

What would be the problem there? The second binding would shadow the first one

pavlosmelissinos10:11:01

let is awesome, you bundle all your context in a single expression. What would be the benefit of having what you describe besides fewer parentheses? In fact it would be much harder to reason about code if you had multiple sexps that affect each other. Wouldn't it also make linting much more complicated?

☝️ 2
zakkor10:11:48

> let is awesome, you bundle all your context in a single expression. I guess my point is that your context is already pre-bundled inside the outer expression in which you are writing your let . What let gives you is the opportunity to create a new scope and have more granular control over that scope (you can pick where the scope ends exactly). My other point is that you don't need this granular control, because in the vast majority of cases, you just want to create new variables and have them live until the current scope ends. > What would be the benefit of having what you describe besides fewer parentheses? Yes, this. Fewer parens, less indentation, less cumbersome to introduce new variables, more similar to how local variables work in other languages. > In fact it would be much harder to reason about code if you had multiple sexps that affect each other. Wouldn't it also make linting much more complicated? This is how ns, def, defn , deftype, defmethod, defmulti, defrecord, defprotocol, ....... currently work though. Would you say it makes code harder to reason about?

Hermann11:11:18

In my opinion, a let-in-this-scop like this would introduce some kind of statefulness into current scope. There is no symbol x in the scope until we processed the let-in-this-scope-form. Suddenly we're doing imperative style instead of fp. This also defeats "everything is an expression" (although you can and break that idiom sometimes to do effects). defXY forms also modify the global namespace scope, but at least they share a common prefix (`def...`), so that is clear.

☝️ 1
zakkor11:11:19

well sure, could call it defvar 😄

pavlosmelissinos11:11:03

I guess I still don't get what the problem is @edward.partenie, it has never felt cumbersome to me > This is how ns, def, defn currently work though. Would you say it makes code harder to reason about? Code needs to be able to access code (and be accessible) from other files, otherwise every Clojure program would be a single file that repeats clojure.core and all its dependencies. All those forms solve a different problem than the one you describe.

Hermann11:11:50

I'd expect defvar to create something mutable at namespace scope 😉

pavlosmelissinos11:11:57

> each new variable is a new indentation level Just noticed this part in your comment and I'm not sure that's true:

(let [x :foo
      y :bar
      z :baz])
You can have as many bindings as you want in a single let edit: If that's not what you mean, an example would help

zakkor11:11:18

Yeah, you can have multiple bindings in the same let statement, and this is only a single indentation level (I suspect the reason this multi-form let exists in the first place is because it would be insanely cumbersome otherwise 😉), but any other let statements inside the first let will be an additional indent level, examples:

zakkor11:11:30

@UEQPKG7HQ >it has never felt cumbersome to me It's more of a nitpick from me, or just spitballing how it could be nicer, but it definitely feels a bit cumbersome. Creating new local variables is nicer in every other language. (and this is coming from someone who is not parens-adverse, and uses structural editing, and etc...).

zakkor11:11:53

Imagine if you had to

int x = 12 {
   if (x) {
       int y = 13 {
          /// ... rest of your code goes here
       };
   }
}
in C or whatever. It would be madness!

zakkor11:11:07

Actually now that I think about it perhaps it'd be an interesting idea to tweak code formatters to keep what's inside the let expression at the same indentation level :thinking_face:

zakkor11:11:35

so it'd still be an expression, but doesn't create hadouken indents anymore

delaguardo11:11:00

(defn foo [x]
  (future-call
   #(do-something x))
  (let x 42)
  (do-something x))
what would be the value of x passed to the first do-something call?

👆 1
delaguardo11:11:38

> but doesn't create hadouken indents anymore "hadouken indent" is not a problem of language semantic. Clojure give a lot of instruments how to control indentation of your forms: let, threading macroses, destructuring, cheep function declaration

zakkor11:11:45

@U04V4KLKC yeah, not sure. It likely doesn't work with current clojure semantics. I'm only discussing this on a thought experiment level, not trying to make it a reality lol

pavlosmelissinos11:11:24

Ok, I get it, yes, nesting can sometimes be inconvenient but I don't think it's because let works like it does. The lack of a construct with function-wide scope is an example of good design. (having it would make stuff more convenient but it can also lead to bad code, it's kinda like :refer :all) I don't like nesting either, so at the hint of it, I always wonder "Would this make sense as a separate, named function? If not, can it be rewritten to be flat?". The answer is usually yes. For instance, those fns in your examples are long enough to deserve their own defns.

Hermann11:11:42

I also tend to "pile up stuff until it works, then separate and decompose ;)"

zakkor11:11:12

I agree with that 100%! Nesting is really the correct word I was looking for. I find myself rewriting stuff to make it look nicer in Clojure, way more frequently compared to other languages, and usually nesting is the reason for it. It's not necessarily a bad thing, but sometimes I just want to focus on solving the problem instead of making things look nicer 😭

marrs11:11:02

@edward.partenie Indentation is a programmer's practice. It has no meaning to Clojure. So just don't ident your code if you don't want to

(let [foo :bar]
(exp1 
(exp2 foo)))
(Edit: I see you thought of that already) I think the real question is "what do you want indentation to mean?" It seems you want to indicate visually how things are scoped right up until you don't. You could have a inline-let to do what you want. It would look a bit like the var keyword in JS and require similar hoisting rules behind the scenes. That's ok but it comes with its own problems, to the extent that some linters insist that you define all vars at the top of the scope (making it like let again). I think the with keyword did something similar with variables and it was decided that it was too dangerous in real life to justify keeping it in the language.

pavlosmelissinos11:11:45

@edward.partenie Writing good software is all about good design and (guess who said this 😛) good design is about separating problems into things that can be composed, so it's not just "making things look nicer" 🙂

👍 1
zakkor11:11:16

> So just don't ident your code if you don't want to Yeah, I think that's what I'll do (along with adding an option for this in cljfmt or zprint). I think for me, I want indentation to make it easiest to visually parse and understand code, not necessarily for it to always be an accurate representation of scopes > > I think the with keyword did something similar with variables and it was decided that it was too dangerous in real life to justify keeping it in the language. 👀 hadn't heard about that, any idea where can I read more about this?

👍 1
devn12:11:02

I recommend you do whatever you like, as long as no one else has to read your code ;)

👍 1
devn12:11:08

It would be extremely jarring to not have indents when reviewing someone else’s work.

zakkor12:11:05

I mean, there is an official clojure style guide full of special rules and exceptions for how to format code in order to make it easiest to read, right?

marrs12:11:15

@U06DQC6MA You never know, you might like it. I think the larger issue would be that other people's linters would just overwrite that stuff

👍 1
delaguardo12:11:36

it isn't official style guide 🙂

1
zakkor12:11:58

well, community style guide 😛

devn12:11:00

maybe I would like it, but it takes a village and there’s already enough variation in style that I’ll happily take the path of least resistance even if it might give me some minor improvement in readability

marrs12:11:40

Do that then.

devn12:11:24

haha yes, I’m just making the point that on a team: good luck selling in an indentation rule that no one else in the community uses

devn12:11:13

if you can get quorum, more power to you though!

zakkor12:11:28

of course, consistency is king. I would only use this in real projects if it were widely accepted as a good idea, and code formatters automatically formatted things in this way. Perhaps will never be a reality for Clojure, but it might be in other future projects (and it's fun to think about!)

👍 1
marrs12:11:33

@U06DQC6MA Maybe. Have you seen the Gnu indentation rules? If you want to contribute to their software then that's what you'll be coding in, so you conform to something But I think what @edward.partenie is doing is experimenting with aesthetics for his own curiosity. I think that's a worthwhile exercise

❤️ 2
👍 1
devn12:11:22

sure! that’s fun stuff, but felt obligated to point out what zakkor just said: consistency is king. anyway, pay me no mind, just being boring and practical over here. 😅

Joshua Suskalo15:11:07

on a practical level a lot of macros and forms expect one sexp for an argument. athe previous example of a let in an if is a good one: it would require a do. The same would be true if you want to use a let in a condition, to build an argument for a function call, to use in for-like macros that lack :let support, and so on.

👍 1
Joshua Suskalo15:11:44

Also we do have prior art on this. scheme lets you define local variables in this way.

seancorfield09:11:56

Threads please, folks.

🧵 6
emilaasa12:11:13

I'm writing a command line tool and would like to hear peoples recommendations / experience reports on using libraries that help with that. I want to create a pretty large cli tool that has: • several different sub commands • ability to create tools with 'unixy' feel to it • a bunch of different ways of passing arguments ( mytool score 'src/**.clj' --some-flag -v --depth=3) I've done some preliminary research and https://github.com/babashka/cli and https://github.com/clojure/tools.cli seems pretty interesting. Does anyone have some opinions on those or other tools?

borkdude12:11:35

babashka CLI is used in neil (and various other projects) which is a pretty big CLI https://github.com/babashka/neil/blob/3b61436e310bb97b1775eb110e8f1f54d9a6d5a5/src/babashka/neil.clj#L655

emilaasa12:11:04

It looks really neat, but I was a bit confused about the direction of the library because of the: > Turn Clojure functions into CLIs! tagline in the beginning

borkdude12:11:41

Maybe I should change that then, but the idea of bb CLI is that you can quickly whip up a CLI while also supporting more advanced use cases

emilaasa12:11:08

Yeah it seems like you have a ton of great functionality besides the metadata thing

emilaasa12:11:14

Overall pretty great approach I think!

emilaasa12:11:02

I think this is pretty clever:

(ns my-ns)

(defn foo
  {:org.babashka/cli {:coerce {:a :symbol
                               :b :long}}}
  ;; map argument:
  [m]
  ;; print map argument:
  (prn m))

marrs13:11:55

Some general advice: Less is often more. Simple commands that do one job are easier to reason about and use. If you're implementing a command with many sub-commands, try to treat those sub-commands as a collection of small, simple commands in their own right. The git command works this way and actually used to be a collection of little commands like git-log, git-commit, etc. Later on they were bundled into a single command. To maximise the utility of your command, its output needs to be easy for a computer to understand. That way it can be used in scripts to automate your workflow. Taking du as an example, the default output is quite hard to read as a human, but quite easy for a computer. The du -h version of the output is much easier to read, but also harder to parse and would be annoying to use in a script. This helps explain why du behaves the way it does. On the other hand, git log is primarily indended for use by a human, but it's still possible to use in a script if you play with the options. Nevertheless, it's quite fiddly and limits its utility in this regard. I'm not saying this is a bad trade-off, just that it is a trade-off, and one you should make consciously. Git makes it possible to alias complex commands, helping overcome this issue. Integration with other commands should also influence your use of stderr. stderr is badly named because it's not exactly for errors. It's really for output that's not intended to be piped to the stdin of another command. It's typically a place to put information that the person running the command might want to see, but that is not relevant to the final output (usually because they want to know what went wrong, hence the name). I'd recommend thinking about how your output might be used with tools like awk and grep . Experience with these tools is helpful. Finally, if you implement streams, other tools will be able to pipe their output to your command via stdin.

🙏 1
Mikko Harju14:11:02

It seems that clojure.walk/postwalk loses the additional metadata added by clojure.data.xml/parse – what would be the way to walk the structure with preserving the data? The side-effect is that when I xml/emit the postwalked version, I get unnecessary xml namespace applied to the root

borkdude14:11:28

@U32ST9GR5 The solution I'm usually using for this is write my own postwalk which does preserve metadata. Please upvote the ask issue about this. There is a patch in JIRA

borkdude14:11:15

I can't find the ask issue for this btw

borkdude14:11:55

@U064X3EF3 is there an ask issue for this? I'm not able to find it via the search

Mikko Harju14:11:02

👍 sounds like a nice change! I ended up just using xml-zippers for editing that preserved the metadata 😄

Alex Miller (Clojure team)14:11:16

if there's no ask link in the jira, then there probably isn't one

borkdude14:11:39

@U064X3EF3 Could one be created for upvoting purposes?

Mikko Harju15:11:27

Upvoted. Thanks for bringing this up!

borkdude15:11:43

well, you brought it up, so thank you too ;)

👍 1
Alex Miller (Clojure team)15:11:33

linked in the jira

🙏 1
borkdude18:11:27

@U064X3EF3 I will respond / update JIRA tomorrow

siva14:11:07

can clojure leverage loom?

Ben Sless14:11:14

Why not? There are already several projects which experimentally target it, including funcool/promesa, mpenet/mina, tenql/tapestry

borkdude16:11:49

Please upvote this clojure ask issue if you care about metadata preservation in clojure.walk: https://ask.clojure.org/index.php/12411/clojure-walk-walk-doesnt-preserve-metadata-on-lists-seqs

🎉 2
Alex Miller (Clojure team)20:11:06

If you upvoted this and you have an actual case in your code, it would be extremely helpful to add that to the Ask Clojure question https://ask.clojure.org/index.php/12411/clojure-walk-walk-doesnt-preserve-metadata-on-lists-seqs

Ben Sless20:11:42

While walk is getting looked at, how about the patch by Stuart Sierra (which I updated) to use protocol based dispatch? It improves the performance ~2x and would be really useful in places like reitit's middleware which keywordizes keys

borkdude20:11:18

Added an example to the ask issue. It's an issue for me anywhere I walk code that e.g. has line numbers.

👍 1
borkdude20:11:38

code usually has lists but also seqs (cons is not a list, but can be in code expressions as a result of syntax quote)

borkdude20:11:15

The protocol issue: https://clojure.atlassian.net/browse/CLJ-1239 - might suffer from the same metadata issue

borkdude10:12:53

@UK0810AQ2 I'm looking at the patch again and this time I tried incorporating the protocol. See attachment for the new clojure.walk namespace. I fail to see a difference for keywordize-keys:

user=> (let [m (vec (repeat 100 [{"a" true "b" false}]))] (time (dotimes [i 10000] (clojure.walk/keywordize-keys m))) nil)
"Elapsed time: 1020.560667 msecs"

borkdude10:12:04

both examples run about the same time

borkdude10:12:47

of course a benefit to the protocol would be extensibility

Ben Sless10:12:23

let me check again, but also try with more diverse data sets

👍 1
borkdude10:12:27

@U064X3EF3 My question to you: when working on the walk + metadata issue, would you like me to include a patch with also the protocol issue in scope? Since that addresses a different problem, I'd say probably not, but wanted to double check with you. I think we could solve the metadata issue first and then write a protocol which the API functions use, separately without making any breaking changes (of course). The uploaded .clj file has both, and I wonder if you could maybe look at it to see if I could make a patch out of it.

borkdude10:12:57

The solution I've chosen now is to write another -walk function (here implemented as a protocol, but this can be easily reverted) where outer receives both the old form and the new form. This gives users the opportunity to pick properties from the old form (like metadata) and apply it to the new form. The body of walk then becomes:

(defn walk
  "Traverses form, an arbitrary data structure.  inner and outer are
  functions.  Applies inner to each element of form, building up a
  data structure of the same type, then applies outer to the result.
  Recognizes all Clojure data structures. Consumes seqs as with doall."

  {:added "1.1"}
  [inner outer form]
  (-walk form inner (fn [_old new] (outer new))))

borkdude10:12:27

And a new version of postwalk , postwalk*, then becomes:

(defn postwalk*
  "Performs a depth-first, post-order traversal of form.  Calls f on
  each sub-form, uses f's return value in place of the original.
  Recognizes all Clojure data structures. Consumes seqs as with doall."
  {:added "1.12"}
  ([f form]
   (-walk form (partial postwalk* f) f)))

borkdude10:12:55

which lets us use it like this, to pick the old metadata:

(postwalk* (fn [old new]
             (if (instance? clojure.lang.IObj new)
               (with-meta new (meta old))
               new))
           x)

borkdude10:12:05

I've attached the above changes as a patch in the jira issue as well: https://clojure.atlassian.net/browse/CLJ-2568 I mentioned / linked issue CLJ-1239

Ben Sless11:12:19

@U04V15CAJ performance comparison example:

(def data {:a {:b {:c {:d {:e [1 2 3 4 5 6 7]} :x "123"} "y" #{'z [1 2]}}}})

(cc/quick-bench (postwalk identity data)) ;; Execution time mean : 3.277995 µs
(cc/quick-bench (walk/postwalk identity data)) ;; Execution time mean : 5.704277 µs

Ben Sless11:12:22

keywordize-keys (and stringify) could both benefit from this, and from an unrelated change to use reduce-kv instead of into

borkdude11:12:43

+ extensibility, seems like a good win

Ben Sless11:12:26

Absolutely, the original patch has been around since 10 years ago

borkdude11:12:53

note that "my" patch allows solving both issues, the original one would have been too restrictive

Ben Sless11:12:18

your latest patch also uses protocols?

borkdude11:12:45

I thought you benchmarked that

borkdude11:12:51

would be good to re-do it then

Ben Sless11:12:10

no, I benchmarked Stuart's

borkdude11:12:09

I'll try benchmarking myself with your example

Ben Sless11:12:46

saw it on my phone, then forgot 😅

Ben Sless11:12:22

But I also think you didn't see much of a difference with keywordize-keys because the datastructure was pretty shallow and didn't have plenty of types to dispatch on

Ben Sless11:12:10

(cc/quick-bench (postwalk* right data)) ;; Execution time mean : 3.282717 µs

Ben Sless11:12:12

With a slight variation

Ben Sless11:12:21

(defn walk* [inner outer form]
  (outer form (walkt form inner)))

(defn postwalk*
  [f form]
  (walk* (partial postwalk* f) f form))

(defn right [old new] new)

Ben Sless11:12:51

In Stu's implementation walkt doesn't take outer at all

(extend-protocol Walkable
  nil
  (walkt [coll f] nil)
  java.lang.Object
  (walkt [x f] x)
  clojure.lang.IMapEntry
  (walkt [coll f]
    (clojure.lang.MapEntry. (f (.key coll)) (f (.val coll))))
  clojure.lang.ISeq
  (walkt [coll f]
    (map f coll))
  clojure.lang.PersistentList
  (walkt [coll f]
    (apply list (map f coll)))
  clojure.lang.PersistentList$EmptyList
  (walkt [coll f] '())
  clojure.lang.IRecord
  (walkt [coll f]
    (reduce (fn [r x] (conj r (f x))) coll coll)))

(defn- walkt-default [coll f]
  (into (empty coll) (map f) coll))

;; Persistent collections that don't support transients
(doseq [type [clojure.lang.PersistentArrayMap
              clojure.lang.PersistentHashMap
              clojure.lang.PersistentVector
              clojure.lang.PersistentHashSet
              clojure.lang.PersistentQueue
              clojure.lang.PersistentStructMap
              clojure.lang.PersistentTreeMap
              clojure.lang.PersistentTreeSet]]
  (extend type Walkable {:walkt walkt-default}))

borkdude11:12:30

yes, I know, but this is too restrictive

Ben Sless11:12:20

How so? It still lets you carry around the meta, no?

borkdude11:12:14

how would you implement postwalk with metadata preservation (as opt-in by the user)?

borkdude11:12:24

based on that protocol

Ben Sless11:12:10

With these

(defn walk* [inner outer form]
  (outer form (walkt form inner)))

(defn postwalk*
  [f form]
  (walk* (partial postwalk* f) f form))
You just need a different arity outer

borkdude11:12:30

yes, that's basically my proposal, to have a new outer that takes the old and new collection. I'll leave the protocol out of the equation since that only complicates the issue further and can be added after this issue probably

Ben Sless11:12:09

Something which might be worth adding to the patch is a short circuit if the meta is nil, otherwise I think it'll just copy the object

Ben Sless11:12:27

Then you don't need the instance check, because meta for something which doesn't have a meta is always nil

Ben Sless11:12:37

(meta "1") ;; nil

borkdude11:12:05

the copying of the metadata isn't included in the patch itself, so I don't think that's necessary?

borkdude11:12:41

I'll leave the protocol for later after we get this out of the way

borkdude11:12:32

instance checks are very cheap btw

borkdude11:12:45

maybe even cheaper than calling meta

Ben Sless11:12:14

meta does an instance check 😄

Ben Sless11:12:53

(if (instance? clojure.lang.IMeta x)
          (. ^clojure.lang.IMeta x (meta)))

borkdude11:12:52

you mean, you could avoid doing with-meta, yes

borkdude11:12:09

up to the user :)

1
curtosis18:11:41

has anyone more-or-less publicly done a PrestoSQL/Trino connector written in clojure? I know the #C03RZMDSH analytics has a connector to datomic (obviously) but a) it’s not public, and b) it might not be in clojure. 😛 Essentially, I need to write a connector (not for datomic) and would prefer for all the obvious reasons to do it in clojure.

emccue18:11:26

there is a jdbc driver

emccue18:11:04

what is unsatisfactory about that + next.jdbc?

curtosis18:11:30

That’s the other direction. I’m talking about the connector underneath trino that maps the JDBC input SQL query onto some non-SQL structures.

bringe19:11:18

I noticed a difference in output of an expression if I run it in the repl vs using the Clojure CLI.

(Math/pow (inc 0.005689554233194338) 12)
=> 1.0704521809380758

clojure -M -e "(Math/pow (inc 0.005689554233194338) 12)"
=> 1.0704521809380756
What might be causing the (very slightly) different answer? More info:
clojure --version
=> Clojure CLI version 1.11.1.1113

;; Clojure version in REPL
(clojure-version)
=> "1.11.1"

R.A. Porter19:11:31

Different Java version, perhaps? In repl: (System/getProperty "java.version") Command line: java -version

ghadi19:11:39

I can't reproduce that. Which REPL?

Alex Miller (Clojure team)19:11:12

Both are likely valid given ulp constraints

Alex Miller (Clojure team)19:11:43

What do you get with StrictMath?

bringe19:11:59

It’s a repl started with clojure but yes, I see different Java versions are used. Maybe that’s the cause.

java --version
=> openjdk 11.0.15 2022-04-19

(System/getProperty "java.version")
=> "16.0.2"

bringe19:11:58

Ahh StrictMath gives me the command line result in the repl

bringe19:11:53

Wait, I lied about the java version. It’s the same in the terminal I started the repl in.

bringe19:11:17

I ran the above java --version in the VS Code terminal, but actually started the repl in a separate terminal.

bringe19:11:42

I didn’t understand how they’d be different anyway, so that makes sense I guess.

Alex Miller (Clojure team)19:11:52

Math only guarantees a difference down to a certain number of ulps (units in last place), see the javadoc for details per method

Alex Miller (Clojure team)19:11:58

StrictMath is ... stricter

bringe19:11:15

I see. Thanks!

Alex Miller (Clojure team)19:11:57

also, you might want to prefer clojure.math as it will get you better primitive use (and performance) in a lot of cases

bringe19:11:05

I see. I generally use clojure.math now, but I think I changed to Math in the above test code when investigating. I did notice, though, that clojure.math/pow gives the same answer as Math/pow in this case.

bringe19:11:43

I still don’t understand why Math/pow would give a different answer from the Clojure CLI than from a repl started from it, when the same java version is used.

bringe19:11:24

It’s not something I need to know though :man-shrugging:

Alex Miller (Clojure team)19:11:00

I suspect you could see both answers in both places as they are both valid

seancorfield19:11:04

@U9A1RLFNV Testing here with JDK 19 (set via JAVA_CMD so the Clojure CLI uses it consistently):

(~/oss)-(!2027)-> clj
Clojure 1.11.1
user=> (Math/pow (inc 0.005689554233194338) 12)
1.0704521809380758
user=>

Tue Nov 29 11:45:55
(~/oss)-(!2028)-> clojure -M -e "(Math/pow (inc 0.005689554233194338) 12)"
1.0704521809380758

seancorfield19:11:27

(that's 1.11.1200 for the CLI, BTW)

bringe19:11:01

Oh :face_palm: it is a difference in Java version between my terminal and the VS Code terminal.

seancorfield19:11:14

So it depends on how you are starting your REPL vs CLI "command-line" (`-e`)?

bringe19:11:33

I was running the clojure command in the VS Code terminal before and comparing to the output from my REPL, which was started from a Terminal instance. Glad I got to the bottom of that and learned about StrictMath today 😄.

seancorfield19:11:09

JAVA_CMD is your friend for consistent JDK version usage 🙂

👍 1
Alex Miller (Clojure team)19:11:11

there have been bugs in the past (even as recently as Java 8 timeframe) where different compiler levels/interpreter in Java had different math special cases and could produce different results

bringe19:11:13

@U04V70XH6 I think it depends on where I started the repl vs where I ran the clojure command.

seancorfield19:11:03

So different :jvm-opts could make a difference too...

Alex Miller (Clojure team)19:11:30

like the compiler used intrinsics or special cased x^2 as x*x, etc whereas interpreter level did not

Alex Miller (Clojure team)19:11:19

(but all of the answers are still correct according to the spec)

macrobartfast19:11:43

I’m using xtdb now although this is not an xtdb specific question. I’m finding myself, when I want to add a new value in the db having to update Malli, update the crud functions, then update the UI. It’s a lot of steps to add a single new value but maybe unavoidable in working with dbs. I wrote a function that dynamically generates a UI table from the document that is returned from the db irrespective of the number of keys. However, I am still manually doing the rest of the changes to the fns that interact with the db directly. And the order of keys of the returned document currently dictates how it appears in the UI… which I can write more code to handle. But I’m holding off for a second to evaluate my overall approach. What are some strategies for handling more ‘dynamically’ interacting with the db aspect of an app and the corresponding UI, such that I could add a new value to a db on the fly and change the codebase as little as possible? I prototype a lot so changes to the store are rapid while I’m figuring things out. Thanks!

seancorfield19:11:38

I guess I'd ask: • Why do you need to update Malli for each new column? • Why do you need to update your CRUD functions? For the former, are you perhaps over-spec'ing things such that you're using Malli like a "type system"? For the latter, why aren't the CRUD functions accepting (arbitrary) hash maps that they can use to interact with XTDB? At work, if we add new columns, we generally only need to modify the UI and the associated "edit" handler(s) (one to display the form, one to save the form).

macrobartfast19:11:23

I guess I have some idea that I need to update Malli because it’s ‘bad form’ to not have a check on what goes into the db… and then I am forced to update the CRUD functions.

seancorfield19:11:38

Type systems are brittle like that 🙂

macrobartfast19:11:27

So it seems like I could just skip the Malli schemas, especially while protoyping.

seancorfield19:11:41

If your specs are open for extension, having extra keys shouldn't be a problem (which is the concept behind clojure.spec.alpha -- not sure what Malli defaults to).

seancorfield19:11:24

But "open for extension" as a principle generally means much less churn in your system as you make changes. So pass hash maps instead of individual parameters, if the number/names of parameters might change often. Allow for arbitrary extra keys in hash maps (in general -- there are some situations where you will need to constrain the set of keys).

macrobartfast20:11:26

This is all good. And I hadn’t thought of this strategy… leaning toward maps.

dev-hartmann22:11:42

Hey folks, quick question about defrecords and protocols. From my understanding they are designed for java interop mainly, I had this talk with one of my colleagues and was arguing for the use of multimethods. I don’t want to be dogmatic or anything, just trying to understand the design philosophy

hiredman23:11:17

multi methods and protocols are different ways of doing polymorphism

hiredman23:11:53

defrecords and deftypes are ways to create custom types

hiredman23:11:15

(types in the jvm sense)

hiredman23:11:42

they are fairly mix and matchable

hiredman23:11:02

multimethods can have more complicated dispatching, protocols are fairly constrained, but not strictly type based since if a protocol is defined as being extendable via metadata you can attach a custom implementation without changing types

devn02:11:39

overuse of protocols is a classic smell imo

d-t-w05:11:25

Alex’s general guidelines on open/closed and type/value informed my use of the different clojure dispatch options.

d-t-w06:11:23

And yeah, when I see lots of protocols + component it’s often a team trying to recreate a spring object graph.

didibus07:11:50

Ya, I wouldn't say it's a smell in that protocol usage is bad. But record+protocol can be tempting from someone coming from OO, to just use those and try to design things similarly to what they'd do in an OO language. So the results would be a bit of a bad fit for Clojure, and probably overcomplicated. If you really need a set of functions to be polymorphic together. Like say plug some components and the protocol requires three functions implemented together. Then protocols are great.

👍 2
dev-hartmann09:11:56

thanks for all the answers folks! my knowledge seemed a bit limited as I was eager to smell something OOP’ish. thanks for the article which helps a lot!

dev-hartmann23:11:16

I know that defrecords are designed to dispatch on type and are faster at that than multimethods