Fork me on GitHub
#clojure
<
2023-10-10
>
seancorfield04:10:04

I'm looking for feedback on this new version of the Web Development Ecosystem page. Errors and omissions, primarily. It's not intended to explain how to write web apps -- it's meant to be an overview of the modern options in that space. Follow-up in #C02M6N5C137 or in a 🧵 here. Thank you!

💯 1
Bobbi Towers04:10:26

Fulcro could be included in the React section

1
👍 1
tatut05:10:39

https://github.com/tatut/ripley is somewhere between a templating and a framework, but sort of neither and both.

1
tatut07:10:28

I would also say Electric Clojure deserves a mention https://github.com/hyperfiddle/electric

vemv09:10:15

Small typo - sunng87 isn't a project name Perhaps Luminus could be mentioned in a less prominent way - I sense that it would be a favor to the Kit authors, given that they intend Kit to be the next-gen Luminus? From my experience with Kit, it "just works" and does use misc Luminus tech under the hood, so it's not unproven at all - it might be even considered very related tech, but renamed (which is better than a breaking v2). And lastly: > (currently an older 9.4.18, which is not the latest 9.4.x release) This seems something that could be fixed at any point, while this doc would remain pointing it out indefinitely. It would seem fair to consider the possible 'damage' that one can inadvertently do. (The general pattern IMO is that documentation shouldn't be too coupled to intricacies)

emccue13:10:07

I'm a bit mixed on the pairing of ring and compojure

seancorfield17:10:17

@U8LB00QMD Given that Fulcro is full-stack and is a collection of libraries, do you think it belongs in the frontend section or under frameworks, or perhaps somewhere else? @U11SJ6Q0K I hadn't heard of Ripley -- it looks interesting... Not quite sure where to put it so I've renamed Frameworks -> Frameworks and Integrated Libraries and added an Other Options section with Ripley in. I'll add Electric there too, at least for now. @U45T93RA6 Updated to say sunng87 ring adapter (the actual project name has jetty9 in it which is very confusing since it's for Jetty 12 now, with a Jetty 11 version still being maintained). Luminus is the only "framework" leftover from the previous iteration of the content and it's the subject of a book (multiple editions) and still widely used so... @U050CBXUZ Do you have input on this page and how you'd like Luminus and Kit described? I did not say "unproven" anywhere @U45T93RA6 and consider Kit preferable to Luminus (for many reasons) -- do you think the text on that page implies otherwise? Re: 9.4.18 -- I call it out because it's important from a security / CVE standpoint, and I'll be happy to update this page when (if) that gets fixed. The intent is that http://clojure-doc.org will continue to get maintained and updated frequently -- and I encourage the community, including you, to help with keeping it up-to-date 🙂 @U3JH98J4R It's what a lot of tutorials still feature and it's still the easier intro to web dev in Clojure -- Compojure's deps tree has just a handful of deps, including Ring; reitit's deps tree has nearly 100 lines of output. Reitit is powerful but it's a lot to learn, even on its own.

👍 1
yogthos18:10:06

I'd add a note regarding Kit on the Luminus page saying that Kit builds on lessons learned from Luminus and embraces the modern Clojure ecosystem. It's recommended as a modern alternative to Luminus

1
seancorfield18:10:45

@U050CBXUZ Thanks. I've added a similar note to the web dev page: > It builds on the lessons learned from Luminus and should be considered a modern alternative to it.

👍 1
emccue18:10:44

Remind me if I don't get back to this soon - I am going to try to write out what I think the "full story" is maybe with diagrams and articulate thinks

1
Bobbi Towers18:10:43

Although fulcro (which is the successor to David Nolen's Om Next) is broader in scope than re-frame and the other frontend solutions, since it involves coordination with the backend, it's still essentially relevant to the React ecosystem so I think it would fit best with the frontend options.

seancorfield18:10:00

Maybe I should split that into frontend and full-stack as a subset of the overall frontend development section... :thinking_face:

jwhitlark18:10:26

Htmx and hiccup (or selmer, I'd assume) seems a really interesting way for primarily back-end devs to quickly get a decent UX experience running. I'm not quite sure where it would fit, tho...

seancorfield19:10:15

Hiccup and Selmer are listed under backend / templating. HTMX is mentioned via Biff under frameworks / integrated libraries.

jwhitlark19:10:25

Fair enough. I'm excited about htmx, so I wanted to see it as a second level item, instead of third under Biff, but I don't have a good suggestion.

seancorfield19:10:59

Are there other HTMX offerings in Clojure-land, other than Biff? Does it need more than Hiccup? (Biff uses Rum, I believe)

jwhitlark19:10:57

As far as I can tell, htmx is just an added script tag, and then it's just some special html attributes, so I think it fits nice with the "just add a library" stuff. But I've not used it in anger yet.

yogthos19:10:36

yeah, htmx is backend agnostic, it's basically a small js lib that allows you to patch parts of the dom

yogthos19:10:07

you just return an html fragment and a hint of whether you want to append, replace, etc

yogthos19:10:22

there's also a ctmx library that's a light wrapper for Ring, might be worth mentioning https://github.com/whamtet/ctmx

1
👍 1
jwhitlark19:10:02

Oh, nice. I didn't realize it'd been added as a kit module. (docs don't seem to have it, but it's in the repo).

yogthos19:10:35

oh yeah, I should update the docs to mention it 🙂

jwhitlark19:10:25

:face_palm: I forgot who I was talking to. 😁

😆 1
seancorfield19:10:11

I added HTMX via ctmx to the Kit section 🙂

🎉 2
emccue03:10:56

I tried writing good words but gave up

emccue03:10:41

I guess what I am thinking about is that there is a whole "stack" of choices, and you can get away without including any "horizontal" of that stack if a web application doesn't need it

emccue03:10:37

which is why it felt odd to pair ring and compojure - since those are generally two distinct boxes in my brain and i'm always itching to give ^ this sort of explanation

seancorfield03:10:53

I guess because almost no one uses that first stack?

seancorfield03:10:09

And R+C is the second stack...?

emccue03:10:51

Well, in practice I think most of us use an iteration of the last stack

emccue03:10:14

(swap json for html, draw more arrows for observability, etc.)

seancorfield03:10:49

Haha... Ok, fair enough...

Tuomas-Matti Soikkeli06:10:18

error on aero link: <a href=":<//github.com/juxt/aero>">aero</a>

seancorfield05:10:19

@UM1PCCLNN Good catch! Thank you. Fixed.

hifumi12305:10:01

Has anyone here had luck using lein-cloverage’s :test-ns-regex or :ns-exclude-regex? The documentation states that both keys should be associated to a vector of Pattern instances, yet my regex never match. In particular

:cloverage {:ns-exclude-regex [#"dev"]
            :test-ns-regex [#"-test$"]}
neither excludes any namespaces with dev in them nor includes any namespaces ending in -test. Removing the :test-ns-regex key (and its value) results in cloverage finding all of the test namespaces, however.

1
seancorfield05:10:17

I suspect they need to match full namespaces not just part of it. Add .* to the patterns to see if I'm right.

hifumi12305:10:09

aha! you were right

hifumi12305:10:26

Thanks for the suggestion 🙂

1
pesterhazy07:10:08

Is there a way to see how long namespaces loading is taking? Today I'm working on a project where, after years of growth, running a single test takes 18s. That's an eternity. There are a number of factors but my strong suspicion is most of that time is spend compiling and loading code. When I send SIGQUIT (`Ctrl+\`) during startup I see

clojure.lang.RT.load
clojure.lang.Compiler.eval
featured prominently. So my plan is to instrument require or one of the internal Java functions to see which namespace loads are taking a long time. Has anyone done this before? Any other ideas for debugging slow startup? Solutions not involving a debugger preferred but I'll work with debuggers if necessary.

hifumi12307:10:34

I recommend using https://github.com/async-profiler/async-profiler and invoking clojure via java

hiredman07:10:36

The answer, broadly, is the namespaces that use core.async are slow to load

oyakushev07:10:52

Is it the same if you run the test once you already have a REPL? If so, you can use clj-async-profiler to profile the manual invocation of the test.

hifumi12307:10:53

This should give you at least a flamegraph of most java methods that are running while your tests start up

hiredman07:10:15

(I am just assuming if you have slow load times you must be using core.async)

1
oyakushev07:10:18

If you absolutely must profile from the launch of the process, then yes, injecting async-profiler agent to jvm-opts will give you the profile that you can then examine.

hiredman07:10:37

The person who perhaps has spent the most time looking at clojure startup time wrote this https://clojure.org/guides/dev_startup_time about improving startup time in development

pesterhazy08:10:58

@U0NCTKEV8 not using core.async in our project, although sadly it does appear in our -X:tree via com.cognitect.aws/api -> com.cognitect/http-client

pesterhazy08:10:44

> invoking clojure via java Already doing that but that's a great pointer because it highlights that the classpath is very big

pesterhazy08:10:06

> async-profiler That looks really interesting. From the docs: > -e java.util.Properties.getProperty will profile all places where getProperty method is called from.

pesterhazy08:10:38

I wonder if this would give me the namespace names

oyakushev08:10:24

Just do a regular cpu profiling, you will see the namespace names on the stacks anyway

pesterhazy08:10:27

Is it the same if you run the test once you already have a REPL?I can probably get the tests to run from a bare repl, but it would be great to have a generic way to profile requires that works from startup

lassemaatta08:10:19

Perhaps this is too simplistic, but I usually start by adding the :verbose flag to the require form and visually check what namespaces are being loaded. But of course this only helps when you want to analyze a particular (test) namespace and its dependencies.

Ben Sless08:10:07

Are you using test fixtures?

pesterhazy08:10:30

No test fixtures. Some of our tests are slow, but this question is specifically about the startup time (only a tiny fraction of those 18s is spent on the actual tests)

vemv09:10:23

What are your thoughts on running the tests from a repl-connected editor? Although certainly there are always optimizations to roll out in a given project, it is IME a more fruitful approach to simply accept that most large projects have some startup time. It's very tempting to bring the mentality from other languages that tests are best run from the CLI, but Clojure, being a repl-driven language, has a different story.

pesterhazy09:10:21

Running tests from a REPL is useful. However, 18s startup time for a trivial test on an M1 laptop is not acceptable – it's means we're doing too much in the project. My focus is on figuring out (measuring!) what we're doing wrong here

vemv09:10:18

Probably you'll be able to shave some seconds, and it will be great, but practically every large project will have similar load times. I'd call it "business as usual" around the 8 second mark. A fact of life (which I know very well as a tooling author) is that the Clojure compiler is slow. A project with 500 namespaces, many 3rd-party deps and dense ns graph (e.g. running your test requires 70% of all your namespaces) will be slow to load. ...a less obvious alternative is to make your project more modular, Polylith style, so that running a test implies requiring few transitive namespaces (yours or third-party). Either way, have a great one with the profiling 🙂 hopefully there will be some low-hanging fruit

👍 1
pesterhazy10:10:15

I'm not subscribing to the "fact of life" theory 😃 Modularity of course is a good approach (but of course quite expensive as far as changes go)

pesterhazy11:10:53

In any case, as @U0NCTKEV8 has pointed out I can use AOT to reduce the compile time, though I'd love to measure before I make changes, to know how much time is actually spent compiling (vs loading namespaces)

Ben Sless11:10:11

Sounds like it's time to instrument / profile some things. You can record the execution using JFR then analyze the results using jmc, you'll see exactly what's doing what

👍 1
vemv11:10:10

Caveat, AOT + tests (or dev, for that matter) is often a problematic combo. For local development, it's best done for 3rd party deps only. Expect opaque issues otherwise.

pesterhazy11:10:26

What kind of issues for tests?

vemv11:10:48

...personally I feel that a traditional profiler approach is too low-level here. One wants a require that tells you the time for each loaded namespace - that's 90% of the info you really need. A profiler doesn't give you such insights directly. With a bit of luck one would be able to either monkey-patch require, or roll your own (which might be easier than it sounds) (I was surprised to find no github projects in this direction?)

vemv11:10:04

> What kind of issues for tests? You could seach for "AOT" in #C03S1KBA2, probably there will be enough signals for the motivated searcher

pesterhazy11:10:57

> One wants a require that tells you the time for each loaded namespace - that's 90% of the info you really need. A profiler doesn't give you such insights directly. I agree. Trying to see if I can come up with something

jpmonettas11:10:03

I just tried, on Clojure master, replace in Compiler.java :

public static Object load(Reader rdr, String sourcePath, String sourceName) {
    long timeStart = System.nanoTime();
     ...
    System.out.println("Loaded " + sourcePath + " in " + (System.nanoTime() - timeStart)/1_000_000_000.0 + " secs ");
	return ret;
}
Then did a mvn -Dmaven.test.skip=true clean package install and run my project with org.clojure/clojure {:mvn/version "1.12.0-master-SNAPSHOT"} which now shows :
Loaded cider/nrepl/middleware/util.clj in 0.021725426 secs 
Loaded cider/nrepl/inlined/deps/orchard/v0v11v0/orchard/util/io.clj in 0.024584895 secs 
Loaded cider/nrepl/inlined/deps/orchard/v0v11v0/orchard/misc.clj in 0.062955573 secs 
Loaded cider/nrepl/middleware/util/meta.clj in 0.070580294 secs 
Loaded cider/nrepl/inlined/deps/orchard/v0v11v0/orchard/cljs/analysis.cljc in 0.049641564 secs 
Loaded cider/nrepl/inlined/deps/javaclasspath/v1v0v0/clojure/java/classpath.clj in 0.030271051 secs 
Loaded cider/nrepl/inlined/deps/toolsnamespace/v1v3v0/clojure/tools/namespace/parse.cljc in 0.096393728 secs 
Loaded cider/nrepl/inlined/deps/toolsnamespace/v1v3v0/clojure/tools/namespace/dependency.cljc in 0.083953341 secs 
Loaded cider/nrepl/inlined/deps/toolsnamespace/v1v3v0/clojure/tools/namespace/track.cljc in 0.108834121 secs 
Loaded cider/nrepl/inlined/deps/toolsnamespace/v1v3v0/clojure/tools/namespace/file.clj in 0.229194348 secs 
Loaded cider/nrepl/inlined/deps/toolsnamespace/v1v3v0/clojure/tools/namespace/find.clj in 0.296201815 secs 
Loaded cider/nrepl/inlined/deps/orchard/v0v11v0/orchard/java/classpath.clj in 0.022146213 secs 
Loaded cider/nrepl/middleware/track_state.clj in 1.0210087 secs 
Loaded cider/nrepl/inlined/deps/orchard/v0v11v0/orchard/inspect.clj in 0.158414203 secs 
Loaded cider/nrepl/middleware/inspect.clj in 0.188368304 secs 
Loaded cider/nrepl/inlined/deps/orchard/v0v11v0/orchard/util/os.clj in 0.014530388 secs 
Loaded cider/nrepl/inlined/deps/orchard/v0v11v0/orchard/clojuredocs.clj in 0.037726076 secs 
Loaded cider/nrepl/inlined/deps/orchard/v0v11v0/orchard/namespace.clj in 0.024732191 secs 
Loaded clojure/test/check/random.clj in 0.064458672 secs 
Loaded clojure/test/check/rose_tree.cljc in 0.041836996 secs 
Loaded clojure/test/check/generators.cljc in 0.368722733 secs 
Loaded cider/nrepl/inlined/deps/orchard/v0v11v0/orchard/spec.clj in 0.420228791 secs 
....
....

💯 2
jpmonettas11:10:27

those kinds of metrics could be a nice thing to have on the compiler, that you could activate with some jvm property

vemv12:10:19

> Loaded clojure/test/check/generators.cljc in 0.368722733 secs Btw, this is what I meant with "the Clojure compiler is slow". 0.3s is an eternity. Say that executing your project involves loading 2000 namespaces (your own + transitive), that's a fixed cost that cannot be tackled directly

pesterhazy12:10:28

@US3FQ5069 that's funny, that's close to what I came up with

diff --git a/src/jvm/clojure/lang/RT.java b/src/jvm/clojure/lang/RT.java
index 5d20ef49..eb38968d 100644
--- a/src/jvm/clojure/lang/RT.java
+++ b/src/jvm/clojure/lang/RT.java
@@ -418,20 +418,21 @@ static void compile(String cljfile) throws IOException{
 	}
 	else
 		throw new FileNotFoundException("Could not locate Clojure resource on classpath: " + cljfile);
 }
 
 static public void load(String scriptbase) throws IOException, ClassNotFoundException{
 	load(scriptbase, true);
 }
 
 static public void load(String scriptbase, boolean failIfNotFound) throws IOException, ClassNotFoundException{
+        System.err.printf("load_start,%s,%d\n",scriptbase,System.currentTimeMillis());
 	String classfile = scriptbase + LOADER_SUFFIX + ".class";
 	String cljfile = scriptbase + ".clj";
 	String cljcfile = scriptbase + ".cljc";
 	String scriptfile = cljfile;
 	URL classURL = getResource(baseLoader(),classfile);
 	URL cljURL = getResource(baseLoader(), scriptfile);
 	if(cljURL == null) {
 		scriptfile = cljcfile;
 		cljURL = getResource(baseLoader(), scriptfile);
 	}
@@ -454,20 +455,22 @@ static public void load(String scriptbase, boolean failIfNotFound) throws IOExce
 	}
 	if(!loaded && cljURL != null) {
 		if(booleanCast(Compiler.COMPILE_FILES.deref()))
 			compile(scriptfile);
 		else
 			loadResourceScript(RT.class, scriptfile);
 	}
 	else if(!loaded && failIfNotFound)
 		throw new FileNotFoundException(String.format("Could not locate %s, %s or %s on classpath.%s", classfile, cljfile, cljcfile,
 			scriptbase.contains("_") ? " Please check that namespaces with dashes use underscores in the Clojure file name." : ""));
+
+        System.err.printf("load_end,%s,%d\n",scriptbase,System.currentTimeMillis());
 }
 
 static public void init() {
 	doInit();
 }
 
 private static boolean INIT = false; // init guard
 private synchronized static void doInit() {
 	if(INIT) {return;} else {INIT=true;}
 

jpmonettas12:10:24

I guess great minds thinks alike XD

🪢 1
pesterhazy12:10:56

Out of curiosity, how do you configure your project to use the alternate clojure jar?

jpmonettas12:10:53

for this time, just added the new clojure dependency on the deps

👍 2
jpmonettas12:10:15

but if you want to have your "compiler with metrics" on Clojars, so other people can use it, you can do it like I do for ClojureStorm :

{:paths ["src"]
 :deps {...}
 :aliases {:metrics-compiler
           {:classpath-overrides {org.clojure/clojure nil} ;; for disabling the official compiler
            :extra-deps {com.github.pasterhazy/clojure {:mvn/version "RELEASE"}}}}}

jpmonettas12:10:08

that is an easy way of swapping compilers with aliases, even when you have the compiler under a different group name

pesterhazy12:10:38

wow that's great - thanks!

pesterhazy12:10:46

This seems to work great too:

clojure -Sdeps '{:deps {org.clojure/clojure {:local/root "/Users/user/prg/clojure/target/clojure-1.11.1.jar"}}}' -M:test:test-main ...

jpmonettas12:10:57

great! were you able to figure out the source of your slow tests?

pesterhazy12:10:10

Now I've got a csv file that should contain the data needed. Additionally it would be great to know how much of that time is spent compiling (vs loading) The next question is how to turn this into a useful graph or report ("10 slowest namespaces")

jpmonettas12:10:31

I'm not sure the compiling vs loading is possible, since they are kind of interleaved

oyakushev12:10:03

You'll get all of that if you use a profiler. JFR or VisualVM or async-profiler, any will do.

jpmonettas12:10:31

@U06PNK4HG do you have any pointers to that? like ways of configuring any of those so you can quickly get ns load times when you need it?

oyakushev12:10:32

It won't give you structured per-ns times because it won't know what a Clojure ns is. But you'll be able to see on the graph what ns is being loaded by the names of certain frames.

oyakushev12:10:35

Besides, per-ns times aren't that helpful anyway, because the time to load a namespace also sometimes includes the loading time of namespaces it requires (since it's all a graph), but sometimes it doesn't (if the namespace has been already loaded by someone else).

oyakushev12:10:10

I'd look at the whole graph as a whole and see if anything helpful stands out.

oyakushev12:10:58

It is as easy to try out as adding -agentpath:/path/to/libasyncProfiler.so=start,event=cpu,file=profile.html to :jvm-opts

oyakushev12:10:34

The .so file has to be downloaded for the correct platform from here https://github.com/async-profiler/async-profiler/releases/tag/v2.9

jpmonettas12:10:30

yeah, I just tried like this clj -J-agentpath:/home/jmonetta/non-rep-software/async-profiler-2.7-linux-x64/build/libasyncProfiler.so=start,event=cpu,file=profile.html -A:dev -e "(require ['dev]) (System/exit 0)" I looks like I can follow the loading graph by following the flamegraph

👍 1
jpmonettas12:10:22

I always use clj-asynprofiler, never tried it for this ns loading times thing

oyakushev12:10:30

The graph is pretty messy indeed, but on the other hand all the fine details are right there.

jpmonettas12:10:28

so thanks, one more idea for the toolbox!

oyakushev12:10:55

One way to make the output slightly nicer is to use async-profiler at startup to generate a collapsed.txt file, and then use clj-async-profiler to make a flamegraph out of it, like

(clj-async-profiler.core/generate-flamegraph "/path/to/collapsed.txt" {})

jpmonettas12:10:25

yeah I was thinking about it, since you can then filter or highlight stuff with the clj-async-profiler

oyakushev12:10:46

Yep, and also the frames will be demunged and Clojure frames will look more Clojurish

Alex Miller (Clojure team)13:10:09

the main sources of slow startup time are: 1. loading more stuff than you need initially (common when using user.clj) 2. compiling the world every time you start (AOT 3rd party deps and getting them on the classpath per the guide above. there is FUD about "problems" above but it is mostly FUD. there are reloading complexities when working at the REPL, but I don't think those apply here. 3. top-level def's that do expensive work like reading config or talking to 3rd party services (use delay or other such techniques)

👍 2
Alex Miller (Clojure team)13:10:01

in addition to all the good info above, I would also mention -verbose:class as a useful jvm tool to show loaded classes which can help show or understand #1. the timings I don't generally find very useful except to track down outliers per #3.

jpmonettas13:10:25

yeah, +1 on -verbose:class

pesterhazy14:10:54

I wrote a script to calculate load timings for namespaces from the csv output of my compiler patch. Here are the top results for our codebase (time in millis)

["clojure/core/matrix" 89]
["clojure/java/jdbc" 93]
["clojure/core/specs/alpha" 98]
["amazonica/aws/simpleemail" 102]
["clojure/core/async/impl/ioc_macros" 150]
["clojure/core" 181]
["clojure/core/matrix/impl/defaults" 184]
["malli/core" 192]
["taoensso/encore" 327]
["clojure/core/async" 343]
["clojure/core/matrix/protocols" 352]

pesterhazy14:10:14

Note that these include compile times. I'm going to try AOT'ing next Is there a clever way to AOT only 3rd party deps?

Alex Miller (Clojure team)14:10:09

the clever way is to not be clever

Alex Miller (Clojure team)14:10:30

if this is for tests, compile your test namespaces

Alex Miller (Clojure team)14:10:04

by definition, anything needed for those tests will be compiled transitively

👍 2
Alex Miller (Clojure team)14:10:44

trying to aot "libs" is the wrong way to think about it - you're trying to aot the code you are loading. if you want to delete a subset of those classes afterwards (from your project), you can, but there really is no need to - if a .clj file exists that is newer timestamp, it will be loaded instead

pesterhazy14:10:26

OK, I tried AOT. Some things I observed: • (compile 'my-test-namespace) wasn't sufficient to compile all necessary dependencies (not sure why). • Instead I used this as a wrapper for my test runner (binding [*compile-files* true] (apply (requiring-resolve 'kaocha.runner/-main) args)) • hugsql doesn't seem to work with AOT (related https://github.com/layerware/hugsql/issues/138). In my case I got "Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:641)". I had to rm -f classes/hugsql to make it run • With that done, AOT brings down startup time from 18s to 11s

pesterhazy14:10:04

(Leaving these here as breadcrumbs for the next person to embark on the startup-speed quest)

Alex Miller (Clojure team)14:10:59

in some cases, things that dynamically load namespaces may need to be added to the compile list

pesterhazy16:10:34

Here's the Clojure patch and analyzer script. Worked pretty well for me https://gist.github.com/pesterhazy/7f8af5701b51ccd86f99fc222436e6ea

hiredman16:10:57

you could just alter-var-root clojure.core/load ?

oyakushev16:10:29

Gotta alter it pretty early, I guess, not too reliable

chrisn13:10:40

Well - anther question is if you or anyone else would pay for a faster clojure compiler. How much is the clojure community effectively paying for those compile times and is there will and money there to make it faster?

Ben Sless14:10:26

If I had to bet money it would be on faster artifacts over faster compiler

chrisn15:10:12

I agree but there is a counter argument - devs and dev time are expensive. The artifact is running on a machine that costs far less an hour. Also as I mentioned in my last talk there is a class of java devs coming to clojure who use it like java - type some stuff in and the IDE auto-runs a unit test suite. Overall I think I agree though - the runtime matters a lot more than startup.

oyakushev17:10:37

People interested in startup profiling may want to check this: https://clojurians.slack.com/archives/C8NUSGWG6/p1697045429076009

pesterhazy17:10:43

Great. Will check it out

Ben Sless18:10:50

I never run an individual test from cold start, always from the REPL so I never cared about that. In stuff like CI, getting a runner usually takes longer than running the tests so my priorities are just in not doing stupid things in CI, making sure I cache deps, AOT compile artifacts, etc

Pavel Filipenco09:10:44

https://www.youtube.com/watch?v=isGKzmwDREg The same in clojure:

(defn two-sum
  "Two sum problem solution, using a map. O(n) time complexity, assuming nth is a O(1) operation (=vector index)."
  [target coll]
  (loop [acc {}
         i 0]
    (if (>= i (count coll))
      nil
      (let [current (nth coll i)
            needed (- target current)
            needed-index (get acc needed)]
        (if needed-index
          [i needed-index]
          (recur (assoc acc current i) (+ 1 i)))))))
Also, in the video he first builds the map and then iterates through the collection again, any reason why that's necessary?

Pavel Filipenco09:10:25

ah, he said it later in the video. Oops, got too impatient to do a solution)

Hendrik11:10:05

Is it possible to write a macro that returns a function definition with meta data, that will work in Clojure and Clojurescript? Basically what I want to achieve is this:

(defmacro example [label args & body]
  `(defn ^{:some :thing} ~label ~args ~@body))

p-himik11:10:01

defn accepts metadata as a separate argument as well: https://clojuredocs.org/clojure.core/defn#example-5a6764b1e4b09621d9f53a76 But you can use with-meta on the label symbol in a macro just fine. Note, however, that that metadata is attached to the var, and var support in CLJS is very limited.

Hendrik21:10:18

Thanks. I am using Clojure for a couple of years, but I never noticed the map form :D

dergutemoritz13:10:35

The docstring of eduction as well as https://clojure.org/reference/transducers#_eduction both mention that its result is "reducible/iterable". However, I couldn't find any place in the docs which clearly explains what that means. Is anyone aware of one?

dergutemoritz13:10:58

Of note: I know what it means via tribal knowledge. But I'd like to have a place to point to when somebody doesn't 🙂

Pavel Filipenco13:10:49

https://clojure.org/reference/reducers > reducible collection (a collection that knows how to reduce itself)

Alex Miller (Clojure team)13:10:42

it can be reduced (in this case, implements IReduceInit) and is java.lang.Iterable

Alex Miller (Clojure team)13:10:24

reducibility is really governed by the internal CollReduce protocol and IReduce/IReduceInit is an interface-based path into that for Java implemented classes

Alex Miller (Clojure team)13:10:21

I don't know that dark corners of reducibility are actually documented anywhere on the clojure site atm. Iterable of course has public javadoc

dergutemoritz14:10:18

@U060FHA3K28 right, that's the most explicit I could find but since it's only mentioned in passing in a different (but related) context didn't quite make it fit the bill for me 🙂

dergutemoritz14:10:48

@U064X3EF3 Ah, CollReduce is a good hint!

Alex Miller (Clojure team)14:10:50

it is something I would like to illuminate more :)

Alex Miller (Clojure team)14:10:04

having spent much time in these dark spaces :)

dergutemoritz14:10:50

That would be awesome!

dergutemoritz14:10:15

would be great!

Pavel Filipenco14:10:21

May I post simple ChatGPT generated initial contents in the issue?)

phill22:10:05

Well. This is all very well to giggle about. But it seems that we have dodged quite a bullet. Consider what might have become of the world if ChatGPT had been trained on Clojure docstrings? You'd ask it any question and the answer would be, like, "See Huet."

raspasov17:10:33

(Seems to have been trained on it)

Pavel Filipenco18:10:05

I don't get what "See Huet." means

phill22:10:33

Well, not having "seen Huet", I don't either. But here, for completeness, is the reference: https://clojure.github.io/clojure/clojure.zip-api.html

Filipe Silva17:10:02

Is there a way to make an existing defn dynamic?

(defn foo [] 1)
(defn indirect-foo [] (foo))

(binding [foo (fn [] 2)]
  [(foo) (indirect-foo)])
;; => Can't dynamically bind non-dynamic var: dynamic-redef.core/foo

;; This is what I need
(make-dynamic! #'foo)

(binding [foo (fn [] 2)]
  [(foo) (indirect-foo)])
;; => [2 2]

Filipe Silva17:10:56

.setDynamic will let binding run, but will still use the old, non-dynamic foo

(.setDynamic #'foo)
(binding [foo (fn [] 2)]
  [(foo) (indirect-foo)])
;; => [2 1]

isak17:10:38

Not sure. You can't use with-redefs?

Filipe Silva17:10:37

.bindRoot will update dynamic-foo to use the new fn, but won't pick up that it's dynamic now

(.bindRoot ^clojure.lang.Var #'foo
           (fn [] 3))
(binding [foo (fn [] 2)]
  [(foo) (indirect-foo)])
;; => [2 3]

Filipe Silva18:10:16

@U08JKUHA9 that's the thing, I'm trying to make an existing code base's test suite parallel, but that test suite uses with-redefs. I'd like to make a variation of with-redefs that uses the same syntax, but underneath uses bindinds:

(defn with-binding-redefs [...]
  ;; make all the vars dynamic
  (binding [...]))

Filipe Silva18:10:55

it sorta looks like it's possible since both .setDynamic and .bindRoot seem to do parts of it, but I don't know how to do both of those things together

isak18:10:58

Ah, gotcha

lassemaatta05:10:59

This https://modulolotus.net/posts/2022-06-22-tidd/ describes an existing https://github.com/mourjo/dynamic-redef library, which sounds a bit like what you’re looking for. Haven't used it myself though.

souenzzo09:10:53

> Is there a way to make an existing defn dynamic? No! The clojure compiler emits different JVM bytecode depending if a var is dynamic or not not only the var itself, but every code that references that var. so it should know about being dynamic or not at "compile" time, and it is hard to change it at runtime

Filipe Silva07:10:46

@U0178V2SLAY I tried that one a couple of days ago but since it relies on altering the bound fn with a wrapper it ends up having problems around metadata. I've worked around those problems with a proxy for now but wanted to know if the "just mark it as dynamic after the fact" approach could work since it'd be much simpler.

Filipe Silva07:10:53

@U2J4FRT2T so far it doesn't just seem hard, it seems downright impossible :D

jpmonettas18:10:39

hi everybody, is this expected ?

user=> (= (sorted-set-by compare ^:a []) #{[]})
true
user=> (= (sorted-set-by compare ^:a ()) #{()})
false
user=> (= (sorted-set-by compare ()) #{()})
true

1
hiredman18:10:32

it is a weird one

hiredman18:10:37

it comes down to

user=> (compare ^:a () ())
Execution error (ClassCastException) at user/eval44 (REPL:1).
class clojure.lang.PersistentList$EmptyList cannot be cast to class java.lang.Comparable (clojure.lang.PersistentList$EmptyList is in unnamed module of loader 'app'; java.lang.Comparable is in module java.base of loader 'bootstrap')
user=> 

hiredman18:10:24

and of course, if you flip it around it works fine

hiredman18:10:28

user=> (= #{()} (sorted-set-by compare ^:a ()))
true
user=>

jpmonettas18:10:05

when equality isn't commutative

hiredman18:10:36

() isn't comparable, so you cannot use compare as the comparator for a set that contains it

hiredman18:10:52

() should maybe be comparable

hiredman18:10:05

oh, I guess lists in general are not comparable? (vaguely tracks)

jpmonettas18:10:15

when adding meta to the empty list

hiredman18:10:07

because compare short circuits on identity before getting to the comparable stuff

hiredman18:10:24

() and () are identical? until you add metadata to one

marciol18:10:30

Hey, don't know if someone have got this, but as I'm using the .ftp.FTPSClient I'm getting a random set of errors Remote host terminated the handshake and Socket is closed for a different combination of user/password/directory in the same server, with multiple instances in multiple threads connecting to the server concurrently, but it's not heavy concurrent, I'm just spinning 20 threads, one for each combination. I'd like to know if anyone have seen this sort of error before. I'm using a retrying approach but I'm still getting failures.

Bailey Kocin19:10:22

Question about deps... Is there a guide that explains how keys are merged for the ~/.clojure/deps.edn vs my project one? I wanted to do something like add extra dependencies/extra paths to existing aliases in my project by adding them to my user deps and was wondering if I could do that/how it works....

hiredman19:10:28

> The merge is essentially merge-with merge, except for the :paths key, where only the last :paths found is used (they replace, not combine).

dpsutton19:10:57

i don’t think you can modify existing “aliases”. You combine aliases to modify the resulting classpath and invocation args

Alex Miller (Clojure team)19:10:15

aliases are merged across the different dep sources

dpsutton19:10:23

ah. i missed that it was a question about how :dev in the user dir merges with a :dev alias in a project directory. Wasn’t aware that there was a strategy for that. TIL

Bailey Kocin19:10:13

Thanks! This makes sense

seancorfield19:10:57

@U05NZDGDYG3 If you use non-overlapping names then clojure -A:project-alias:user-alias will be "additive" in terms of what gets run.

🙌 1
practicalli-johnny10:10:58

The official guide mentioned by Alex covers this really well. I also wrote an explanation with some other examples and a bit of graphics https://practical.li/clojure/clojure-cli/#configure-clojure-cli