Fork me on GitHub
#clojure
<
2021-05-04
>
Joshua Suskalo00:05:39

A library that I'm publishing on clojars requires some AOT compilation due to :gen-class. I'd ideally like to allow users to have this library as a git dependency too. My question is this: does the Clojure compiler produce bytecode that's dependent on the JVM version it's running on? e.g. if I compile on openjdk11, will I be able to rely on the classfiles generated being able to run on openjdk8?

seancorfield00:05:47

@suskeyhose More to the point, if you compile code against a specific version of Clojure, people using other versions might not be able to use your library.

Joshua Suskalo01:05:10

That's a good point, but an unfortunate requirement. In this case, the actual generated code is exceedingly simple, just a call to = and a couple of keyword lookups on the state of the generated class, which extends java.lang.Error

Joshua Suskalo01:05:40

Oh and the class implements an interface for a protocol

Joshua Suskalo01:05:02

So I'd imagine it's unlikely to see version conflicts.

hiredman01:05:14

Not a good idea

hiredman01:05:21

Aot compilation works best at the end of the dependency chain (you might aot your final program)

hiredman01:05:05

If you start inserting aot in the middle (aoting chunks of libraries) things can go very wrong

Joshua Suskalo01:05:12

Yeah, I've been over this before. The unfortunate reality is that with the current facilities Clojure provides I cannot extend Throwable in a way that I can catch the resulting class without :gen-class

hiredman01:05:14

The hazards you have to navigate are largely related to aot compilation being transitive, and order dependent

hiredman01:05:54

So aot compiling code will aot compile everything it depends on

Joshua Suskalo01:05:02

There is exactly one namespace that needs aot compilation, and it only needs to create a named subclass of Throwable.

Alex Miller (Clojure team)01:05:03

@suskeyhose stay tuned for some new features in the hopefully imminent future

Joshua Suskalo01:05:14

Thanks alex! I'm looking forward to it!

hiredman01:05:21

Which will, if you package it in a jar, result in something like an uberjar

hiredman01:05:40

Which depending on can lead to all kinds of dependency conflicts

Joshua Suskalo01:05:40

Sure, except in this case there are no transitive dependencies of this namespace.

seancorfield01:05:05

Well, clojure.core is a dependency I bet?

seancorfield01:05:35

So compiling your single namespace will cause a bunch of compiled .class files to appear I suspect.

Joshua Suskalo01:05:48

depstar allows me to elide all of the clojure.core class files

Joshua Suskalo01:05:56

and that has worked fine when depending on it from clojars.

hiredman01:05:13

Eliding class files can be done, but it will also break things sometimes

Alex Miller (Clojure team)01:05:26

clojure.core is already compiled so will not produce any class files there

seancorfield01:05:28

Fair enough. I don’t like to encourage AOT for libraries (and I think you get a warning, yes?) but some folks insist…

Joshua Suskalo01:05:53

Yeah, I would love to avoid aot if I could

hiredman01:05:54

Because the next time whatever elided clojure code is compiled it may not have the same class name

hiredman01:05:27

So your aot compiled code may refer to a class that doesn't exist any more

Joshua Suskalo01:05:40

Yes, it will not produce any clojure.core class files, but I had to elide some clojure.core.specs.alpha class files

Joshua Suskalo01:05:21

Yes, I would love to avoid aot completely, and if you have a way I can extend Throwable and catch the resulting class without aot, I will gladly switch.

Alex Miller (Clojure team)01:05:34

ideally, relying on ex-info is better

Joshua Suskalo01:05:16

Unfortunately that's not an option in this case because this is for control-flow constructs and I can't rely on third party code to re-throw the ex-info

seancorfield01:05:52

@alexmiller If you compile a very simple namespace, even though you don’t get classes from clojure.core, you do get classes from core.specs.alpha in your classes folder.

Joshua Suskalo01:05:05

Yup, I had to elide those

seancorfield01:05:28

Since @alexmiller is around: what about the OP’s Q which we haven’t answered yet… “if I compile on openjdk11, will I be able to rely on the classfiles generated being able to run on openjdk8?”

Alex Miller (Clojure team)01:05:51

should be, they are java 8 bytecode

Alex Miller (Clojure team)01:05:11

whether or not you make calls to jdk apis added after java 8 requires more discipline

seancorfield01:05:13

Cool. I had a feeling I’d seen that as an answer to another question here recently.

Joshua Suskalo01:05:14

Awesome, that helps out a lot.

Joshua Suskalo01:05:44

Yeah, I'm not calling any JDK api, all I'm doing is extending java.lang.Error with a named class that has a protcol implementation that checks if the argument is equal to an item in the state.

Alex Miller (Clojure team)01:05:28

there is a lot of fud around aot. it's not conceptually any different than all java libs that "aot compile". there are some issues for sure - we may take on making those issues less of a problem.

Joshua Suskalo01:05:37

Although now that I think about it, I could potentially reduce how much is AOTed by using extend-protocol in the other namespace.

Joshua Suskalo01:05:56

rather than directly extending the interface

seancorfield01:05:35

“fixing” the transitive compilation stuff would probably be the #1 thing folks seem to want (although there are tooling workarounds) — but I understand that there are complexities behind that.

seancorfield01:05:24

Making it easy to compile some thing(s) at the point of use, i.e., via a command that runs code, would also probably satisfy what most folks actually need 🙂

Joshua Suskalo01:05:08

Yeah, that or a way to extend classes while generating a named class is all I need

Joshua Suskalo01:05:17

in order to completely remove aot from the equation

Alex Miller (Clojure team)01:05:23

all of those things are up for consideration

dpsutton01:05:57

On mobile, but would it be possible to put your error class as a stand alone maven project and just depend on that from your clojure project?

dpsutton01:05:15

Then no aot and can use git deps and anything else you like

Joshua Suskalo01:05:38

Hmm. Except for the ns form, I think I could actually entirely remove the dependency on clojure.core

Joshua Suskalo01:05:25

That's actually a good question @dpsutton

Joshua Suskalo01:05:50

I'll consider that. It depends on what gets generated once I pare down what's actually in the aot-ed part.

Joshua Suskalo01:05:18

I think I can get everything that's there down to special forms.

dpsutton01:05:28

I think two jobs ago we did something like this and just compiled it once and stuck it on the class path.

dpsutton01:05:02

Checked in and everything if I remember correctly

Joshua Suskalo01:05:37

Alright, well after getting it down to just special forms and an ns clause it's still a few too many class files. I think I might just make a java artifact that I can put out on clojars.

Joshua Suskalo01:05:46

Thanks for the suggestion @dpsutton!

emccue03:05:13

@suskeyhose one thing that i think might be worth investigating is just using https://bytebuddy.net/#/

emccue03:05:48

making your class and loading it into the right classloader

Joshua Suskalo03:05:30

I think that adds more overhead than just making an additional jar with the single class file in it, and it certainly increases the runtime overhead for users of my library.

hiredman03:05:40

Clojure has a censored version of asm

hiredman03:05:45

(oh, I forgot that is such a heinous arrow)

tlonist09:05:47

I’ve just adopted using component(https://github.com/stuartsierra/component) in my project, and I’d like to know how others think about the experience. I haven’t checked, but there seem to be quite a few alternatives like integrant and mount. It might be because I’m inexperienced, but restructuring a non-component project (which was made of many global singleton defs and defns) into a component project wasn’t easy..

imre09:05:29

I like that using these libs seem to direct code organisation towards a more decoupled way.

imre09:05:49

I've worked with all 3 and currently prefer integrant then component, lastly mount

p-himik09:05:50

> restructuring [...] wasn't reasy Seems like it's one of the reasons behind https://github.com/juxt/clip since it can work with regular functions. Personally, I use Integrant. Wasn't that hard given that I had to just add a bunch of multimethods only for the lifecycle parts that I was interested in.

imre09:05:24

What I like in integrant is that system structure and component config are in the same source data structure

imre09:05:31

At least in my experience

ordnungswidrig09:05:04

I prefer stuartsierra/component for the simplicity of the model. In the ends it's a topological sort and a function to map over your graph. I used the generic mapping in the past to implement "system wide" functions, other than start and stop.

p-himik09:05:35

To add to what imre said - it can be separated with e.g. Aero. That's exactly what juxt/yada uses, and sometimes it makes sense - in my case, because I'm using shadow-cljs and I want to pull some of the config from the shadow-cljs.edn file.

dharrigan09:05:59

I use juxt clip, across a variety of applications. Works really well.

dharrigan09:05:55

Similar concepts as others (there is a small comparison here: https://github.com/juxt/clip#comparison-with-other-libraries)

flowthing09:05:29

I've used every library mentioned so far apart from Component, and I also prefer Clip. The only downside is that it is not (yet) possible to start/stop a partial system.

flowthing09:05:44

What I don't like about Mount and Integrant is that both conflate namespaces with components. I also found Clip pretty easy to grok, compared to e.g. Integrant. YMMV, of course.

Darin Douglass10:05:01

We’ve recently found redelay and have been really happy with how simply it deals with state. https://github.com/aroemers/redelay

Noah Bogart11:05:11

@U01TFN2113P I just did the same but with integrant. It’s a lot of up front work but when you get into the “reloaded workflow”, it makes things so much smoother.

Ben Sless11:05:28

Took me some time to "get" component. I think my problem was Component seemed too ad-hoc and using them was often left up to the implementation. Looking at https://github.com/juxt/aero#using-aero-with-components I landed at a usage mode I liked: Very generic components. Implement interfaces on them, including IFn. Take "constructor" functions as arguments. Constructor functions take the entire component as argument and destructure the keys they require out of it. Because of how component works and how the aero config got merged in, the component's this contains all of its dependencies and options. And finally, don't pass components as arguments to functions which aren't constructors. It creates code which looks a bit weird, but very easy and pleasant to work with.

tlonist13:05:37

Thanks for all the comments. When using component, I felt its way of coding makes the whole structure more nested than I originally thought. For instance to track down where the datasource is made -whether it be that of jdbc or from hikari congig - I needed to go to the definition of record database first, and then to its realiing function, and finally to where it actually gets injected. In the meantime it was quite verbose, with its structure code and business code mingled together.

tlonist13:05:28

But I believe the directive of component is a truly needed one. Will look more into it with other suggestions, thanks!

Noah Bogart13:05:10

yeah, it can make things feel much more interwoven/complex, but it also reveals how interconnected things already were through globals.

👍 3
wontheone114:05:12

Great thread! @U4ZDX466T I have a question specifically for you. > The only downside is that it is not (yet) possible to start/stop a partial system. Usually the way we can test apps using these libraries is not starting real HTTP web server and replace it with a handler functions and so on. If you can’t start/stop a partial system, how do you test them? Do you start whole dependencies and then mock all HTTP calls or such? Please share how you test a Clip app, very interested.

flowthing16:05:29

I rarely mock anything. The one application where I've had the opportunity to use Clip I just start the entire system and test against that. Might not be a viable option with larger applications, of course (although you might be able to split it into several systems, not sure).

👍 3
imre09:05:17

Is anyone aware of a utility lib that has a hybrid let/`letfn` macro? I often want to use letfn but end up just going with a let as I also need let-like bindings in the same scope and don't like the idea of adding yet another level of indentation. I'm looking for something where you can mix let and letfn style bindings in the same binding vector

borkdude10:05:58

I hardly use letfn, I just can't remember the syntax. Imo letfn is useful for mutually recursive fns, but in other cases, I would just use let

👆 15
imre10:05:30

It's a pretty simple and convenient syntax actually if you only need to let fns

imre10:05:46

(letfn [(foo [] 1)] (foo))

borkdude10:05:21

yeah, once I look it up it makes sense, but I just can't remember it

imre10:05:11

I prefer it a lot to using let when it's possible as it's a lot more concise

emccue15:05:17

@imre Let me take a stab at it - untested

emccue15:05:10

(defmacro let% [clauses]
  (loop [clauses clauses
         final-bindings []]
    (if (empty? clauses)
      `(let ~final-bindings)
      (if (list? (first clauses))
        (recur (rest clauses) 
               (conj final-bindings 
                     [(first (first clauses))
                      (cons 'fn (rest (first clauses)))]))
        (if (>= (count clauses) 2)
          (recur (rest (rest clauses))
                 (-> final-bindings
                     (conj (first clauses))
                     (conj (second clauses))))
          (throw (RuntimeException. "Not enough clauses in let")))))))
        

emccue15:05:13

give this a shot

emccue15:05:55

it won't let you do mutally recursive stuff like letfn but its somethign

imre15:05:50

nice one 🙂

imre15:05:59

I'll give this a go

imre16:05:03

(defmacro let% [clauses & body]
  (loop [clauses clauses
         final-bindings []]
    (if (empty? clauses)
      `(let ~final-bindings ~@body)
      (let [[fc & rcs] clauses]
        (if (list? fc)
          (recur rcs
                 (-> final-bindings
                     (conj (first fc))
                     (conj (cons `fn fc))))
          (if (>= (count clauses) 2)
            (recur (rest rcs)
                   (-> final-bindings
                       (conj fc)
                       (conj (second clauses))))
            (throw (RuntimeException. "Not enough clauses in let%"))))))))

imre16:05:17

did a bit of fixing & refactor on it

imre16:05:38

(let% [bar           1
       (foo [x y z] [x y z])
       {:keys [baz]} {:baz 2}
       [_ _ qux]     [0 0 3 4]]

      (foo bar baz qux))

;; => [1 2 3]

imre16:05:30

mutual recursion could be tough 🙂

Jim Newton10:05:29

my code is usually littered with letfn , and I’ve also often wished to mix let with letfn but not often enough to write the macro.

Jim Newton10:05:00

@imre what would be the clojure ideomatic way to distinguish the two types of bindings? For example, I very well might want to bind a variable to a vector. would it be the case that a list of 3 or more components whose 2nd element is a vector is necessarily an attempt to bind a function? and the macro could simply insert the (fn …) around it? or would you want something more exotic?

imre10:05:49

the elements of a letfn binding vector are list s

imre10:05:11

destructuring works off vectors and maps

imre10:05:14

so if the next thing you see in the binding vector is a list, that would be your letfn thing

imre10:05:23

(let+ [bar           4
       (foo [x y z] [x y z])
       {:keys [baz]} {:baz 2}
       [_ _ qux]     [0 0 3 4]]
  (foo bar baz qux))

Noah Bogart12:05:16

Is it possible to test macro expansion? I have a branch in my macro that throws an error when given incorrect code. I’d like to write a test to demonstrate that. Is that possible?

mpenet12:05:43

yes with macroexpand, macroexpand-1 or clojure.walk/macroexpand-all

Noah Bogart12:05:03

I’ve only ever used that at the repl. They don’t return strings, they return the actual code?

ghadi12:05:01

macroexpand takes a form and returns a form

Noah Bogart12:05:33

Cool, I’ll give that a try

Milan Munzar15:05:44

Coming from ClojureScript I was suprised that defrecord behaves differently in both:

(defrecord Service [send-query])

(defn make-send-query []
  (fn [foo] foo))

(let [s (->Service (make-send-query))]
  (.send-query s "foo"))
; ^ Throws in Clojure (no matching method), but works in Cljs
How to do this in Clojure (using Protocols?), why doesn't it work the same?

Alex Miller (Clojure team)15:05:45

you should not be using interop syntax to invoke

Alex Miller (Clojure team)15:05:27

record fields should be accessed like maps

Milan Munzar15:05:50

I see thx :thumbsup:

Milan Munzar16:05:43

so this would be a proper use for Protocols I guess

Alex Miller (Clojure team)16:05:47

((:send-query s) "foo") is I think what you're doing?

Alex Miller (Clojure team)16:05:59

you can certainly do the equivalent with protocols too

Alex Miller (Clojure team)16:05:38

(protocols are mostly just implemented as maps of types to functions)

Milan Munzar16:05:41

yeah that works in both

borkdude16:05:19

The lower level syntax for accessing these fields is (.-send-query s "foo") which works in both, but it's more preferable to access those via either keywords or protocol fns (in the case of deftype)

borkdude16:05:41

That (.send-query s "foo") works in CLJS might be just accidental since the . is quite overloaded in CLJS

Jim Newton16:05:36

does anyone know how I can contact Sam Ritchie ?

Adam Helins16:05:56

Through generative testing I found this behavior which I find surprising. In the context of maps and sets, [] and (list) are equivalent:

{[] :vec, (list) :list}  ;; Throw, duplicate keys

(hash-map [] :vec (list) :list)  ;; Super weird => {[] :list}

;; And similarly with sets

borkdude16:05:31

In CLJS (at least, planck) {[] :vec, (list) :list} doesn't throw though, it's something only the JVM Clojure seems to do

p-himik16:05:05

They have the same hash for some reason and are equal in CLJ.

Adam Helins16:05:39

But Lumo (CLJS) returns {'() :list} , which is inconsistent with Clojure JVM

Alex Miller (Clojure team)16:05:48

sequential collections compare equal. equal things have the same hash.

p-himik16:05:29

Ah, right, makes sense.

delaguardo16:05:01

user=> (= [] (list))
true

Alex Miller (Clojure team)16:05:55

hash-map docstring explains why different than literal (... "as if by repeated uses of assoc")

dpsutton16:05:01

in clj -A:cljs -M -m cljs.main -r {[] :vec, (list) :list} returns {[] :list}

borkdude16:05:33

$ bb -e '{[] :vec, (list) :list}'
{[] :list}
;)

Adam Helins16:05:42

Right, so this is not specific to empty colls then, it is indeed about hashing. It has never bothered me with = , but after years of Clojuring, I kind of freaked out seeing that behavior with keys :p

dpsutton16:05:38

it's just a subtle difference in the behavior when the key is already present and a new value is assoced.

👆 6
raspasov19:05:36

Yeah, I try to stick to [] (vectors as opposed to lists/seqs) nowadays for everything which is data but definitely not code in the “final” output; eliminates this kind of “worry”

raspasov19:05:06

Unless I specifically want to keep something as a, say, list, so I can keep adding to the front; That’s rare.

hipster coder22:05:25

@alexmiller I watched your video on Clojure Spec from about 1 year ago. Just wanted to say thank you for all the hard work you have done leading the community. It has kept me interested in using Clojure. Spec is an amazing tool. And you picked the perfect images for your talk. Of the bird.

nate sire23:05:54

I was just seconding what hipster said

nate sire23:05:02

I saw the video too