We are deeply concerned by clojure startup time. In order to fix that I want to reimplement clojure’s lambdas with MethodHandle and invokeDynamic. And after that, make vars initialization lazy. But it’s quite hard task to do with current Compiler impl.

if you aren't doing anything dynamic at runtime, you could check out Graalvm native execution

already did, it just doesn’t work =(

try the later version of the GraalVM, they just released Release Candidtate 6 with better support for resolving reflection cases

troubles with core-async and our huge java codebase that is not compatible with it either

oh damn 😞

and by the way we don’t really want to lose all the dynamic stuff

have you spoken with @ghadi at all? he has done a lot of experiments with clojure and invoke dynamic

@hiredman not yet. Thank you

I’ve seen some of his commits, but didn’t find an attempt to handle lambdas

What's your startup time right now? @jetzajac

What do mean by attempts to handle lambdas?

~10 sec

AOT?

yep

Oh, replace IFns with method handles

Direct linking?

yep 😃

How many deps?

That is going to be an incompatible breaking change

not so much, really. datascript, core.async,

@hiredman can’t see where. even old compiled code will be compatible. it’s just another IFn impl

A MethodHandle is a reference to an already existing method in the JVM. Who loads that?

ns init

static method in ns__init class

all lambdas into one class into static methods

just like Java does with lambdas

I'd honestly look at the volume of code being loaded before hacking the compiler.

Not to say that there isn't room for improvement in the compiler

we’re building desktop app. it absolutely must be fast to start

profiler just shows class-loading and some other small stuff.

class-loading is a subject to safe-point bias though

I don't doubt that, but with compiler improvements realistically you're going to shave a couple seconds off, max

how would you measure that without real impl of that optimization?

Extrapolation based on years of experimenting

Let's say you double the loading perf, from 10s to 5s. Does that meet your SLA?

https://dev.clojure.org/display/design/Improving+Clojure+Start+Time

nope. we need ~2

You need a different stack

Want to eliminate vars loading too

Doubling the performance of startup is really really hard.

You need 5x

why not just emit bytecode equivalent to that java does and delay as much work on vars init as possible.

App CDS might help too (jdk 10+)

heard about that, yeah. but that is classloading optimization. which could be optimized with method handles a lot more efficiently. imagine emitting one static class with static method per ns

you are talking about shift the work from loading each fn class and its single method, to analyzing all the static methods on a ns

that is shifting work around, not actually changing the work that is done

and is in direct conflict with lazy loading

i meant not loading, but initialization

java is quite good at that. the overhead per class is much bigger. (based on IntelliJ bytecode volume)

I would be very surprised if loading and verifying the bytecode isn't the dominating factor there

have you compared with -Xverify:none?

no. definitely will, thanks

I think you are right to suspect that the volume of classes loaded is almost certainly the #1 issue

Class static initializers are generally on the lighter side when doing direct linking

I thinks the problem there might be the interpretation. class initializers most probably are interpreted, not jitted. even small amount of work there can lead to significant slow-downs.

Yes

Cold by definition

yep.

that can be solved not generally but on app level

just don’t use vars where you don’t need them. do something clever with macros or something. but it is possible only after addressing the #1 problem

Can you do dynamic classloading in this app?

sure I can

We can even delay some work on loading ns-es

Is this an intellij plugin? Because I know Jetbrains is religious about classloading

not actually plugin, but yeah. we do load intellij on classpath

interesting

we really do crazy stuff to optimize it, yeah

Yeah I remember talking to cfleming about this, I think he got spanked because cursive was slowing intellij startup

he switched parts to Kotlin, read it somewhere

it is the same problem, yeah

you might look at some of the clojure android stuff, they similar want basically and optimizing compiler

I wonder if you could do AOT then rewrite it with asm

how do they do clojure on android with this problem? some byte code tree-shaking and post-processing?

dunno

You can also do manual tree shaking if you ship a custom clojure

@ghadi how to do that? after emitting vars and all that?

you mean to tree-shake clojure.core?

clojure.core itself is quite fast to load. the problem seems to distribute between our codebase, datascript and core.async

You could remove things you know you won't call. Fork and delete...

core.async can be made much faster by delaying ioc_macros

ioc macros shouldn't be needed except during macro expansion

If it is, we could fix that

exactly! but is required eagerly!

that is about 10-15% or our load-time. we need much more 😞

Yeah. I'd file a bug for that. Seems fixable

Ok we're making progress :)

can even do that myself. just want to hit elephant first

What min JVM can you support?

I've been delaying class init work using constantdynamic on JDK 11

we can use 10 I think

Could simulate condy using indy...

never heard about constantdynamic

I hate to say it because I wrote part of it... but you could get rid of core.async

Not sure how much you lean on it

https://github.com/arrdem/oxcart

arrdem has some blog posts related to that which also might be interesting

core.async is awesome and we rely on it so much 😃

Changing the way clojure loads namespaces is a promising possibility... You might break stuff though

I'm surprised and impressed that datascript doesn't have any transitive deps!

that is ok, hava a compiler option 😃

datascript is awesome too

@jetzajac clojure-android doesn’t solve this problem, which is one of the main reasons basically no-one uses it.

tools.emitter.jvm is not in a usable state right now, but it’s something I’ve thought about updating for similar reasons to yours.

Basically having a more hackable compiler. It’s much much slower than the Clojure compiler, but since I AOT I don’t care about that. Currently I run a fork of Clojure, and I’d much prefer to be able to implement the extensions I need as macros or analyser passes.

However I think the Clojure language design places some fundamental restrictions on startup time right now. It’s possible that redesigning the var implementation using indy might help that, but @ghadi is the man to ask about that.

@priornix We'll take it to DM (there's #slack-help BTW for general Slack questions)

Thanks will do

I know one of the reasons you cant just trim out core statically is that any code can use "eval" and sidestep static analysis

but how much does the clojure ecosystem/real world projects actually use eval?

I use eval as a JIT for my DSLs but I’m probably unique there :)

@jetzajac what problems are you experiencing with slow startup times? Most apps I’ve worked on in my Clojure career have taken close to a 60 Sec to start, none of them have been a problem beyond a slight annoyance.

Ah, I see, a plug-in

I'm using it for emacs like user config on my new browser written in clojure, works like a charm

people can bind any code expression or existing calls to any key without having to touch the source

Hello everyone, So I'm trying to write a code for requesting quite large amount of URL using clj-http. My approach right now is by using pmap, it does the job, and I must note, faster than when I was using map. But then I notice that because for each request I didn't use any connection pool, the number of connections opened correlates linearly with the number of the URL I requested, this is unfortunate, since most of those URLs came from the same domain. When I tried to use with-connection-pool, wrapping my pmap call, it often results in an error stating that the connection pool has already closed. Has anybody ever succeeded in doing a bunch of HTTP requests in parallel, and by reusing connections?

@hawari.rahman17 you have to separate issues here. one is that http-kit does not use connection pooling by default and is very eager . for that, your connection pooling should work, but if you use different domains, it would not. the other problem is the use of macros like with* together wi realising lazy sequences. just wrap you pmap with a doall to prevent realising the lazy sequence after the pool is already closed.

for the general case of different domains I once use a java counting semaphore in combination with http-kit async api

You could wrap it in channels as well with core.async if you want to stay within Clojure.

Put the requests on a channel, have it take X requests, wait until those are done, put the results on another channel, take X more, and so on.

Hi @ido, your solution by using doall to wrap pmap successfully prevents the connection pool created by with-connection-pool to be closed, but I don't understand why does it runs way slower than the "use pmap, don't use connection pooling" solution, is it highly dependent on the number of maximum threads option assigned to it? Hi @henrik, yes, core.async isn't a solution I've already explored yet, but I think I might need to state my use case first to improve the quality of discussion here:

My program first reads a response from a service in the forms of JSON. From this response, I want to hydrate each of the object, the hydration process involves calling another service which ideally I want to run concurrently.

On second thought, for that use case, is using a new connection for each request necessarily a bad thing? Will connection reuse actually prove to be helpful instead of becoming a bottleneck?

What caused me to worry is if for some reason the number of requests increases greatly, won't it exhaust the ports quickly if I stick to my current approach? What is the conventional wisdom here?

@hawari.rahman17 Here’s an example. This solves the number of concurrent requests, but doesn’t deal with connection reuse.

Ah, I see, maybe I can use this approach to use connection pool exclusively per partition, let's see if that's better Thank you for the inspiration @henrik

Can anyone recommend a clojure project that they really rate in terms of code style? I'd like to see how the pros do it

Just wanted to give a heads up that I’ve created a new channel #healthcare for the discussion of clojure in healthcare. Thought it might be a good place for people working in or around that space to announce or collaborate on healthcare-specific libraries (FHIR specs, etc), for example.

Anyone with experience in message passing? I am trying to create a queue right now for a game where a user can cancel/accept.

newbie question: why this doesn't work : (swap! foo (#(str %) "foobar") )

look at the docstring for swap! and what arguments it takes

swap! requires a function to apply, (#(str %) "foobar") returns "foobar", which isn’t a function

an atom, followed by a function, followed by optional arguments

(#(str %) "foobar") is the function #(str %) applied to the argument "foobar" which is "foobar"

so your call to swap! is (swap! foo "foobar")

hmmmm, I see, there is a way to return a shorthanded lambda as itself?

just don't call it

lists are function application

but (#(str %) "foobar") is a closure, right?

no

lists are function application

#(str %) is a function (closure, procedure, etc)

(#(str %) "foobar") is invoking that function with the argument "foobar"

just like (+ 1 2) is invoking + with the arguments 1 and 2

well, I got that, but

#(str %) is also the same thing as str

ok, but how can I pass "foobar" and not doing the list application ?

ah, I get it

swap! uses the value of the atom you pass it

you are expecting automatic currying like from ml or haskell?

anyway, you can use partial, or better yet include foobar in you closure

yeah

#(str % "foobar")

clojure doesn't have that

(automatic currying)

ok, man, clojure is fun but so different, thanks a lot!