I enjoyed this: https://www.youtube.com/watch?v=-Qm09YiUHTs

...and here are the awesome slides: https://gfredericks.com/speaking/2018-11-30-compiling.pdf

i haven't gotten too far regarding the asm stuff, but i believe one very relevant file is Compiler.java in clojure's source code

Thanks folks, I'll take a look at your suggested resources. 🙂

Compiler.java is the only relevant file :)

i happened to be looking at core_proxy.clj today and noticed what looks like some use of asm there too:

(import
 '(clojure.asm ClassWriter ClassVisitor Opcodes Type) 

Not aware of such ways, but you can use a promise and await (deref) it, and add a watch to an atom that delivers to that promise when atom has 0

OK I see thanks :thumbsup: it might be a bit too much for what I need this time though

I would take a look at clj-java-decompiler results from the multiple compilation runs to see if anything changes from one run to the next: https://github.com/clojure-goes-fast/clj-java-decompiler

Or the output of the javap disassembler. It might help narrow down what is actually changing across different runs.

Here?

These will be a lot of text... I can put it here. Though I've made diff between javap output and got following:

1c1                                                                                                                     <   MD5 checksum 5451a8596af590278a48eaf4c54f0fc1                                                                       ---                                                                                                                     >   MD5 checksum ad944f6857a2468250c235ade9ea8808                                                                       305c305                                                                                                                 <        215: aload_0                                                                                                   ---                                                                                                                     >        215: aload_1                                                                                                   307,308c307,308                                                                                                         <        217: astore_0                                                                                                  <        218: aload_1                                                                                                   ---                                                                                                                     >        217: astore_1                                                                                                  >        218: aload_0                                                                                                   310c310                                                                                                                 <        220: astore_1                                                                                                  ---                                                                                                                     >        220: astore_0

checksums are obviously different, though other differences look like reordering of instructions

That looks like only the difference between using 'local variable slot 0' and 1 for different purposes across the two runs. Not a functional difference, but yes a byte code difference.

I do not have a guess why that might occur, but @U060FKQPN might.

Can we bug him?

Or rather "can I bug him?" 🙂

Out of curiosity, what is your use case for wanting the JVM byte codes across multiple AOT runs to be the same? Some kind of build/deploy tooling?

I already did by using the @ in front of his name. Whether he chooses to answer is really up to him.

He may be busy for the next N hours for all I know.

I want reproducible build https://reproducible-builds.org/

maven does kinda support it (not without major pain in arse, but does)

I highly doubt there is anything in the Clojure compiler that is causing this by design, but could easily imagine there is something that it just happens to pick different local temporary slot numbers (or whatever the correct technical name is for that 0/1 difference) in some cases.

The JVM byte code descriptions here: https://en.wikipedia.org/wiki/Java_bytecode_instruction_listings. call them "local variables 0, 1, ..."

Well as far as I know clojure compiler does use asm quite a lot

and can produce it's own bytecode without javac help

The Clojure compiler does use the asm library to produce JVM byte code, yes. I do not know if the asm library might be introducing this behavior on its own. Perhaps

There are also various dynamic names produced during compilation that are different between builds

e.g. client_4990$eval40104.class and client_9628$eval40104.class

FYI, it appears that achieving the goal of reproducible Clojure builds, i.e. identical JVM byte code, across runs will require changes to the Clojure compiler, at least, and perhaps the asm library, too. Changes to the Clojure compiler are far and few between in time these days, for stability reasons, so hopefully you are not in a hurry to see an officially released version of Clojure with such changes. You are of course welcome to experiment with your own local changes to these things all you wish.

I am pretty sure the client_4990 thing contains a value from a global counter, to generate unique names quickly.

And is thus likely to be very sensitive to the order that code is compiled.

e.g. swapping the order that two files are compiled is very likely to change those.

most likely, I would like to have some control over that... and I understand that changes to compiler are unlikely to happen soon if at all, though just hoped that maybe someone solved this using some hacks/tools

It seems at least possible that if you compile files in a consistent order, perhaps that would also eliminate the aload_0/1 differences you are seeing.

Worth a test on your part, perhaps.

but how? I use leiningen... and didn't see any options for that...

What command did you use to create the compiled files?

Whatever it was, perhaps try doing "Lein clean" between two separate build runs, if you weren't already doing that. One Lein build can leave behind many compiled class files in the target (or targets ? I forget) directory, and the compiler typically avoids recompiling Clojure source files when there is a newer time stamp .class file already in the file system.

do you have a minimal repro of this I could look at?

but generally the clojure compiler doesn't attempt to be reproducible at all

I always do lein clean, and this is issue at CI machine, so it even run in it's separate docker container every time. For build I just used :aot key in my project.clj file and give it bunch of namespaces to compile. This list is sensitive to order (that I've figured hard way long time ago). @U060FKQPN Hi 🙂 No I don't, but I probably can do that tomorrow (it is too late here to start).

You have probably already considered this factor and avoided it, but using anything like 'latest' or 'release' versions in a Lein project.clj file will go out over the network and look for newer versions of libraries. I do not know whether Leiningen can get different artifacts if you avoid those, guaranteed, or whether there might be reasons it could get different versions on different runs still.

I also do not know if anything in Leiningen tries to parallelize compilation at all, but I would guess no. Parallelism is usually a bad thing if you are trying to do things in a consistent order across runs.

Out of curiosity, I did a quick search for "clojure reproducible build" and didn't find anything promising there. I saw one email list conversation about one person trying to improve reproducibility in some Debian package for Clojure, and Elena Hashman replying that the change would not by itself make it reproducible, because Clojure builds were not reproducible for other reasons (but without details). Are you trying to make the builds reproducible for a project like Debian?

I don't use 'latest' version and similar stuff. I use on-premise nexus maven repo which I fully control and no dynamic artifacts can come from internets 🙂 I'm not trying to do anything for Debian. I want reproducible build because I'm making a custom CI/CD infrastructure for our company and having simple way to detect what should be deployed and what could be left untouched (based on file hashes) would be nice

git commit hash is good identifier - but it is not inherent property of produced artifact, while checksum is

Some people deploy Clojure source files rather than AOT compiled .class files, which if acceptable in your case could make the Clojure compiler behavior a non-issue.

Well, my clojure artifact is uberjar 🙂 Application inside need to compile some namespaces in order to provide handlers for pure java libraries it uses

so... alas AOT is unavoidable

Even uberjar's can contain Clojure source rather than .class files, I am pretty sure, but understood if there are reasons that is not an option for you.

I only compile what I absolutely have to 🙂

It's possible with #merge.

ah, so #merge [{:shared :config} #profile {:prod {} :dev {:only :dev}}]

that sort of thing at the top level?

I haven't used it at the top level, but it should be fine.

this works, ta.

You don't need while (it doesn't even exists) @atom is enough

well, it does exist

user=> (doc while)
-------------------------
clojure.core/while
([test & body])
Macro
  Repeatedly executes body while test expression is true. Presumes
  some side-effect will cause test to become false/nil. Returns nil

oops

Many moons ago (at a previous employer) we used to call these documents design rationale’s.

if the number of streams is fixed, I'd use alts! rather than a mix

well, fixed and small

Yeah I considered that, the example is simplified but it will be quite a few sources in the end

I'm actually more concerned with how to facilitate the comparison in a non-clunky way than how to get the stream data together in one place

something about my decision to just loop associng onto a map, and then compare when I've got enough values seems... off. I mean I think it would work, just feels wrong.

it's an inherently fuzzy kinda problem I guess

I think swapping into an atom which has a map, that then has a watch set on it might be slightly cleaner but I worry if I'm entering over-engineered territory there

IMO that code would be much more "surprising" to me

hmm ok... so maybe it's not completely off base. I'm just getting into using core async and the style is quite different from how i'd write normal clojure code, so I think I'm still developing that gut feel for what is good async code

obviously there's some consideration that you don't want that map to grow infinitely. also what does compare-them! do? definitely don't want it doing anything blocking

the keys will be fixed in amount, so i'm not worried about the map growing because i'll just be overwriting the vals each time for a given key

compare-them! is just a stand-in because i don't yet know how i'm going to compare the vals corresponding to matching keys from each stream

fair enough. yeah imo this is the implementation I'd probably go with

alright thanks for taking a look - i'll give it a go

IIRC :123 should not be valid, but it is due to historical reasons. I.e. undefined behavior.

Hm okay. That’s really odd then. I’m experimenting with crux db and I’m successfully able to transact : as a valid :crux.db/id value, even tho technically that keyword is invalid…

"not officially supported" according to http://clojure.org docs, does not mean the same thing as "is guaranteed to throw an exception if you try to use it"

some unsupported things kinda-sorta work in some contexts, but not necessarily all (e.g round tripping between printing and reading)

Makes sense. I was just surprised that I can’t eval that keyword in my REPL, yet somehow crux is able to digest it.

which one is accepted by crux, but doesn't work in REPL?

@U6GFE9HS7

user=> (keyword "a" "123")
:a/123

Maybe that's why. Or maybe not. :)

Oh, you mean something like :number/123 ? If so, then yes, the Clojure reader that the REPL uses throws an exception if you attempt to read it. If you end up creating such a keyword via some means that doesn't involve the reader, e.g. a call to the keyword function, you can pass it to other Clojure functions, and they will see it as a keyword. If they do not print it out and try to read it back with the Clojure reader, then no exception

Ahhh yes, I’m creating it using keyword.

It allows you to create all sorts of nightmares:

user=> (keyword "'" "\"/[/]")
:'/"/[/]

The fact that keyword doesn't prevent you from creating unreadable keywords has been asked N times, an issue was created for Clojure and declined, at least partly for performance reasons. There is an open issue to consider creating a different function that might do that, but no proposed implementation I am aware of. One could implement such a thing themselves, where the most general implementation I can think of would internally try converting it to a string and reading it back, throwing an exception or returning an error if reading threw an exception.

People deal with this in various ways, including using strings as keys instead of keywords in some cases.

There should probably be an FAQ for this -- or perhaps there is one or several already whose location isn't known to me.

So, I should likely avoid these sorts of keywords then. Crux supports a handful of valid IDs (https://opencrux.com/docs#transactions-valid-ids) I can likely use the map flavor: {:crux.db/id {:my/id 123}}

Some of it is mentioned here: https://clojuredocs.org/clojure.core/keyword

Nice, ty for the info and reference

Also this: https://clojure.org/guides/faq#unreadable_keywords

Cool. I should have checked that list of FAQs.

The best ORM is no ORM. ;)

I’ve heard from others that using next.jdbc with something like honeysql is quite pleasant. https://github.com/seancorfield/next-jdbc https://github.com/jkk/honeysql

https://www.hugsql.org/

And none of those are ORMs.

I use hugsql to query my Clickhouse Database. However, I have a self built adapter that is causing some issues with my APM agent. Not a big impact but it is a nuisance.

It’s probably a better idea to write custom code for the APM than it is to use an ORM

It doesn't have to do anything with ORMs though. Even if you'd find a perfect ORM for Clojure, chances are, it wouldn't support ClickHouse at all.

@U2FRKM4TW You are right Elastic APM has trouble making elasticsearch queries compatible

https://github.com/elastic/apm-agent-java/issues/1025

No idea what you mean by "ORM database" -- ORMs are wrappers for accessing "any" database and on the JVM those ORMs would all wrap whatever JDBC driver you are using.

I've never heard of a the Clickhouse database but not all JDBC drivers are created equal -- and it sounds like APM is relying on features that the Clickhouse JDBC driver does not support (which doesn't surprise me with some of those more esoteric JDBC drivers).

If you can switch database vendors to use a more "standard" DB (MySQL, PostgreSQL, SQL Server, etc) then that APM is more likely to work properly with it.

> ORMs are wrappers for accessing "any" database Not necessarily. There can be DB-specific stuff in a particular ORM. Or some DB might lack a particular piece of functionality required or recommended by an ORM.

(this also applies to HTTP servers that you use with APMs -- we wanted to use http-kit but ultimately switched back to Jetty because New Relic's APM didn't fully support http-kit)

@U2FRKM4TW That's why I put "any" in quotes 🙂

Bottom line: we don't use ORMs in Clojure because we don't have objects -- we use data instead 😉

Ah, I read the quotes as an emphasis, my bad.

@U04V70XH6 Exactly our use case. We had to change from immutant to jetty.

most APMs want you to stick to a narrow, boring path of supported libraries.

But I don't get value out of 99% of APMs

contextually we are talking about the cause as in .getCause - he can't test that the thing that caused the CompilerException was actually an IllegalArgumentException

https://github.com/clojure/clojure/blob/master/test/clojure/test_helper.clj#L134

feel free to lift that if needed

as-> is intended to be used inside -> and you do not need it here:

(-> time
    (str " 00:00")
    (->> (time/local-date-time "yyyy-MM-dd HH:mm"))
    (time/zoned-date-time "US/Central")
    (time/with-zone-same-instant "UTC")
    (->> (time/format (time/formatter :iso-offset-date-time)))

(but it's better not to mix ->/`->>` so I would avoid threading here altogether)

https://www.youtube.com/watch?v=4s9IjkMAmns

Oh, yeah, and it has a name: The Robustness Principle: https://en.wikipedia.org/wiki/Robustness_principle

Thoughtful criticisms of the principle are linked from the Wikipedia article, too, thankfully.

Ah, the vary-meta captures the entire form. Thanks!

Whoops. I misread the parens. I think I get what's going on now. Thanks