Fork me on GitHub
#clojure
<
2023-02-10
>
mx200010:02:51

Is it bad style to point refs to other refs ? Are references (atoms, refs ,etc ) themself immutable (not the content)

jpmonettas12:02:31

is there a way of starting a deps.edn or lein project repl with a forked version of the clojure compiler? Lets say I want to try an alternative forked version of the clojure compiler installed under org.my-own/clojure "1.11.0" , looks like I would need a way of excluding the default one

Alex Miller (Clojure team)13:02:16

Not easily - the CLI has a hard wired dep on Clojure in the root deps.edn

Alex Miller (Clojure team)13:02:06

But you can get around that a few ways, like by using an alias with :classpath-overrides or :extra-paths that points to a jar or dir

jpmonettas13:02:01

something like that would work also, let me see

jpmonettas13:02:09

hmm but it only works with a local folder or jar maybe, so no way of replacing your clojure compiler with a clojars one?

Alex Miller (Clojure team)13:02:27

No, this was not considered a core use case when we designed it :)

Alex Miller (Clojure team)13:02:47

Sure, not against it but would need to think about how it could happen

jpmonettas13:02:08

great! Thanks!

Alex Miller (Clojure team)13:02:25

I guess really it would require suppressing the root deps.edn, similar to how -Srepro ignores the user deps.edn

jpmonettas13:02:11

what else would that bring down? apart from the default compiler?

borkdude13:02:10

What about org.clojure/clojure {:local/root "yourown"}?

jpmonettas13:02:01

@U04V15CAJ yeah but if you want to provide that dev compiler as a tool for other people is kind of inconvenient

Alex Miller (Clojure team)13:02:40

That’s the only dep (well, transitively spec and core specs), but this is also how you get :deps alias and paths etc, so this would break other expectations

Alex Miller (Clojure team)13:02:10

Let me step back and ask if your goal is to test something or to actually distribute something that depends on this?

jpmonettas13:02:15

my goal is to create a fork of the clojure compiler that you can use only for dev. So it is the exact same clojure compiler but with a couple of patches I can keep rebasing with each version, patches for emitting some extra bytecode for debugging your code. So for it to be convenient you need a way of specifying the dev clojure compiler under your dev alias, and then have your standard clojure compiler for everything else

jpmonettas13:02:20

does it make sense?

jpmonettas13:02:44

to be more concrete I have been experimenting with https://github.com/jpmonettas/flow-storm-debugger but instead of instrumenting on demand by re-evaluating code, just modifying the compiler so everything is always automatically instrumented

borkdude13:02:07

if you want to have it under your dev alias, you can do {:aliases {:dev {your-compiler {:mvn/version ...}}}} and your compiler will be on the classpath before the official compiler and it would "just work" ™️ ?

jpmonettas13:02:46

oh, so it is guaranteed that every class and .clj file will be loaded instead of the ones of the original compiler by just doing that?

borkdude13:02:20

.class files are always loaded before .clj ones

borkdude13:02:34

but if your clojure compiler mirrors the .class + .clj structure of the original, then it'll work

jpmonettas13:02:13

yes they do, since it is the clojure compiler with a big patch which doesn't move files around

jpmonettas13:02:10

if that works then that is all I need

jpmonettas13:02:22

I'll give it a try

Alex Miller (Clojure team)13:02:37

I don’t get why you think that will work borkdude

borkdude13:02:17

deps and paths in aliases go before top level deps on the classpath, right? at least that's what I've been seeing. that is why it will work

Alex Miller (Clojure team)13:02:56

extra-paths, then paths, then the merged deps, which are in depth order then alpha sorted at each depth right now

borkdude13:02:07

oh I see, then it won't work

jpmonettas13:02:32

damn, was too good to be true

borkdude13:02:42

but classpath-overrides will still work, probably

Alex Miller (Clojure team)13:02:58

It does but you have to give it an explicit path

Alex Miller (Clojure team)13:02:18

(That’s one way I work with Clojure dev)

borkdude13:02:11

I'm also using classpath overrides in bb to remove clojure + spec from the classpath when loading bb deps

jpmonettas13:02:06

but you can't use mvn coordinates with classpath overrides, which makes it inconvenient for tooling

borkdude13:02:52

@U0739PUFQ What you do is:

{:aliases {:dev {:extra-deps {your-compiler {:mvn/version}}
  :classpath-overrides {org.clojure/clojure nil}}}

jpmonettas14:02:12

so that is basically the :exclude-clojure? true I was asking for the the Ask clojure question then

👍 2
jpmonettas14:02:44

nice, will give that a try! thanks!

borkdude15:02:32

I'd be surprised if it didn't:

/tmp $ clj -Spath -Sdeps '{:aliases {:no-clj {:classpath-overrides {org.clojure/clojure nil}}}}' -A:no-clj
src:/Users/borkdude/.m2/repository/org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.jar:/Users/borkdude/.m2/repository/org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.jar

jpmonettas15:02:52

yes, it worked!!

🎉 2
jpmonettas15:02:16

I'll add a comment to the ask clojure question

jpmonettas15:02:13

oh, you already added it @U04V15CAJ, great!

🎉 2
Alex Miller (Clojure team)15:02:00

the nil thing used to raise an error but pretty sure I fixed that for some use case like this in the past

borkdude15:02:51

I think I used to do an empty string or nil and then you suggested the reverse (can't remember which) but this ended up adding an empty path on the classpath, which you then fixed

borkdude15:02:16

This was exactly my use case in bb and I remember it being discussed around that time in 2021

Mario Giampietri13:02:24

Does anyone know if any other language (and which ones in case) has transducers, or something like Clojure transducers?

Alex Miller (Clojure team)13:02:44

None, as far as I know, as it’s something Rich invented

Alex Miller (Clojure team)13:02:16

We did publish impls in Java, JavaScript, and Python though, as examples

Alex Miller (Clojure team)13:02:58

It is a little similar to stream fusion in Haskell, but that’s compiler magic

mpenet13:02:47

racket has something somewhat similar

mpenet13:02:58

iirc it was written after the clj implementation, inspired by it

👀 2
Mario Giampietri13:02:44

@U064X3EF3 as far as you know, was there any source of inspiration (thinking of academic papers for example) for the concept itself? (the answer to this might well be in A history of Clojure I guess)

Alex Miller (Clojure team)13:02:12

It was Rich’s invention. He did actually search for papers where someone may have come up with it and did not find any

👌 2
Alex Miller (Clojure team)13:02:38

There are some papers with things called transducers that are different things

Ed15:02:09

I recall a scala person I know talking about something called transducers in the scala lib and found this : https://stackoverflow.com/questions/27816946/what-are-the-similarities-and-differences-between-scala-transducers-and-clojure ... but I think that they're slightly different things ... (my scala is pretty feak and weeble)

👀 2
Ben Sless16:02:43

Isn't it equivalent to operators fusion?

Tom H.12:02:37

Karsten Schmidt implemented transducers in Typescript: https://github.com/thi-ng/umbrella/tree/develop/packages/transducers

phill02:02:17

@UK0810AQ2 Operator fusion[1][2] minimizes the cost of the plumbing between the operations in a series by reducing the number of operations in the series. Transducers minimize the cost of the plumbing by eliminating the plumbing. [1] https://akarnokd.blogspot.com/2016/03/operator-fusion-part-1.html [2] https://learn.microsoft.com/en-us/windows/ai/directml/dml-fused-activations#how-to-fuse-activations

Ben Sless06:02:03

@U0HG4EHMH since java doesn't have multi stage evaluation, isn't operators fusion implemented like transducers in Rx streams?

phill11:02:15

I don't know if I'm up-to-date, but in any case what the linked pages [1] and [2] describe as "operator fusion" is significantly different from Hickey's transducers: In both [1] and [2], the framework relies on specially-coded special cases based on knowledge of the operations it provides, and the operators are built to operate in only one context, e.g., streams. Transducers, on the other hand, are an open set, each one blissfully unaware of the others and of the processing context.

cddr15:02:44

I was surprised that this did not work but instead gave me an error saying “call to fn did not conform to spec”. Let’s say func-expr here is actually a sexp representing the desired function defined in a config file that expects g and m parameters (and sometimes might refer to a handful of DSL operators of which some-binding is a representative example). How can I splice it in?

(let [func-expr '(fn [g m]
                   (some-binding))]
  (eval `(fn [game me]
           (binding [some-binding (fn [] nil)]
             (~func-expr game me)))))

cddr15:02:16

Ah, I forgot about autosyms. Think i need some #s.

andrea.crotti15:02:41

is there a way to automatically detect duplicated code over a large codebase? It should not care about style differences and ideally it should also find similar forms, they don't have to be identical.

andrea.crotti15:02:25

I could not really find anything, but in theory it's not hard, even though it might be very computationally expensive I guess

andrea.crotti15:02:51

comparing any form with any form at least, if we just limit to defns maybe then it's not too bad

respatialized16:02:56

https://github.com/Datomic/codeq you might want to look at codeq, which allows for Datalog queries against a Clojure codebase.

respatialized16:02:48

I think there was a more recent effort at putting the output of tools.analyzer in a Datalog DB but haven’t found it yet.

respatialized16:02:18

these might be interesting ways of trying to do form-level comparisons across a codebase

andrea.crotti21:02:44

ah yes I just remembered that there is this as well https://github.com/jpmonettas/clindex

andrea.crotti21:02:47

which might be handy as well

mloughlin18:02:41

I'm trying to import XML files into SQLite, are there any examples of prior art using Clojure? I'd love a clever generic library instead of my current plan of hand writing specific implementations for different XML

stand19:02:15

It's a bit old now but I like [this clojure-saxon library](https://github.com/pjt/saxon). It allows you to write clojure functions that are backed by XPath statements. XPath is a language you can use to extract nodes from a given xml document. It's the same sort of concept as the clojure specter library.

escherize19:02:06

hiccup makes a decent xml interop layer. Are you just putting the xlm straight in, or transforming it?

jmckitrick20:02:54

Has anyone here researched or successfully deployed a clojure app on a server running under heavy load? We’re looking at alternative to jetty and undertow. http-kit with virtual threads is promising, but it seems http-kit is not being actively maintained, and virtual threads are still experimental, IIUC.

thomas20:02:06

I guess you first want to define heavy load and on what kind of hardware(or VM) you want to run. (CPU count, memory etc.)

hiredman20:02:08

It might be helpful to discuss why you want an alternative to jetty or undertow

thomas20:02:50

and what is the limiting factor? CPU? network I/O or disk I/O?

hiredman20:02:24

Jetty has been pretty dependable at work, we dabbled in httpkit but of I recall switched back to jetty because newrelic was better able to report metrics for it out of the box

jmckitrick20:02:51

I’ll try to get those specific answers and report back.

hiredman20:02:40

Undertow has a particular threading model, which I don't think the ring adapters I have seen for it handle well

ghadi20:02:48

We also use jetty at work with very very high scale

hiredman20:02:49

We also use a netty based http server at work, netty being a kit for building http servers, and I hate it (mostly because netty is very commonly used and managing dependencies in a large project with it is gross)

jmckitrick20:02:51

That was my understanding as well, that it was very robust.

hiredman20:02:30

I would avoid any talk about virtual threads. I understand the enthusiasm for them, but it is way too early for any best practices or understanding how they impact api design

jmckitrick20:02:25

While I’m waiting on the answers (other than 8 core 64GB CPU) is it foolish to run benchmarks on a mac while deploying on Linux? Even if only to get general impressions of performance?

thomas20:02:12

general impression I would say yes, but the real thing might be different of course.

hiredman20:02:55

There are also guides out there for tuning jetty (although I don't think we change any of the defaults)

Ben Sless22:02:21

Yes I have to say in my experiments I got the best performance out of http kit and undertow But I want to revisit them

Ben Sless22:02:58

While HTTP Kit receives little maintenance, it works well and I couldn't get other servers to beat it in my benchmarks

Huahai23:02:04

we have been running http-kit for 8 years under all kinds of loads, never had performance issue with it.

emccue04:02:39

There is a new option with nima - Someone wrote a ring adapter recently too. Don't know its relative performance though

hifumi12305:02:16

For heavy load (as in maintain potentially a hundred active websocket connections that cause the server to run various handlers and make pretty expensive database queries) on top of ordinary HTTP API endpoints being hit, Undertow has been reliable for this purpose. I can’t comment on HTTP Kit besides it being relatively small and the client not supporting cookies yet… I’ve briefly tried using it for a server but didn’t feel comfortable with the lack of HTTP/2 support.

jumar06:02:57

It all depends on the nature of your application. In most situations, it's likely that the main bottleneck will be the database - running many non-trivial concurrent queries it's gonna impose a great load on it.

jmckitrick12:02:40

So we went through several options: aleph, jetty9, jetty11, undertow, httpkit w/virtual threads, and we ended up back at jetty9. Httpkit looked good, but doesn’t do streaming responses, and that’s a must for us.