Fork me on GitHub
#clojure
<
2018-09-06
>
jetzajac00:09:15

We are deeply concerned by clojure startup time. In order to fix that I want to reimplement clojure’s lambdas with MethodHandle and invokeDynamic. And after that, make vars initialization lazy. But it’s quite hard task to do with current Compiler impl.

benzap00:09:30

if you aren't doing anything dynamic at runtime, you could check out Graalvm native execution

jetzajac00:09:54

already did, it just doesn’t work =(

benzap00:09:35

try the later version of the GraalVM, they just released Release Candidtate 6 with better support for resolving reflection cases

jetzajac00:09:36

troubles with core-async and our huge java codebase that is not compatible with it either

benzap00:09:45

oh damn 😞

jetzajac00:09:17

and by the way we don’t really want to lose all the dynamic stuff

hiredman00:09:46

have you spoken with @ghadi at all? he has done a lot of experiments with clojure and invoke dynamic

jetzajac00:09:53

@hiredman not yet. Thank you

jetzajac00:09:38

I’ve seen some of his commits, but didn’t find an attempt to handle lambdas

ghadi00:09:39

What's your startup time right now? @jetzajac

hiredman00:09:25

What do mean by attempts to handle lambdas?

hiredman00:09:21

Oh, replace IFns with method handles

ghadi00:09:27

Direct linking?

ghadi00:09:47

How many deps?

hiredman00:09:52

That is going to be an incompatible breaking change

jetzajac00:09:36

not so much, really. datascript, core.async,

jetzajac00:09:12

@hiredman can’t see where. even old compiled code will be compatible. it’s just another IFn impl

ghadi00:09:25

A MethodHandle is a reference to an already existing method in the JVM. Who loads that?

jetzajac00:09:47

static method in ns__init class

jetzajac00:09:14

all lambdas into one class into static methods

jetzajac00:09:25

just like Java does with lambdas

ghadi00:09:30

I'd honestly look at the volume of code being loaded before hacking the compiler.

ghadi00:09:00

Not to say that there isn't room for improvement in the compiler

jetzajac00:09:29

we’re building desktop app. it absolutely must be fast to start

jetzajac00:09:11

profiler just shows class-loading and some other small stuff.

jetzajac00:09:30

class-loading is a subject to safe-point bias though

ghadi00:09:33

I don't doubt that, but with compiler improvements realistically you're going to shave a couple seconds off, max

jetzajac00:09:48

how would you measure that without real impl of that optimization?

ghadi00:09:07

Extrapolation based on years of experimenting

ghadi00:09:39

Let's say you double the loading perf, from 10s to 5s. Does that meet your SLA?

jetzajac00:09:06

nope. we need ~2

ghadi00:09:19

You need a different stack

jetzajac00:09:30

Want to eliminate vars loading too

ghadi00:09:51

Doubling the performance of startup is really really hard.

ghadi00:09:55

You need 5x

jetzajac00:09:31

why not just emit bytecode equivalent to that java does and delay as much work on vars init as possible.

ghadi00:09:33

App CDS might help too (jdk 10+)

jetzajac00:09:30

heard about that, yeah. but that is classloading optimization. which could be optimized with method handles a lot more efficiently. imagine emitting one static class with static method per ns

hiredman00:09:37

you are talking about shift the work from loading each fn class and its single method, to analyzing all the static methods on a ns

hiredman00:09:51

that is shifting work around, not actually changing the work that is done

hiredman00:09:59

and is in direct conflict with lazy loading

jetzajac00:09:21

i meant not loading, but initialization

jetzajac00:09:23

java is quite good at that. the overhead per class is much bigger. (based on IntelliJ bytecode volume)

hiredman00:09:47

I would be very surprised if loading and verifying the bytecode isn't the dominating factor there

hiredman00:09:18

have you compared with -Xverify:none?

jetzajac00:09:44

no. definitely will, thanks

ghadi00:09:09

I think you are right to suspect that the volume of classes loaded is almost certainly the #1 issue

ghadi00:09:34

Class static initializers are generally on the lighter side when doing direct linking

jetzajac00:09:56

I thinks the problem there might be the interpretation. class initializers most probably are interpreted, not jitted. even small amount of work there can lead to significant slow-downs.

ghadi00:09:20

Cold by definition

jetzajac00:09:38

that can be solved not generally but on app level

jetzajac00:09:13

just don’t use vars where you don’t need them. do something clever with macros or something. but it is possible only after addressing the #1 problem

ghadi00:09:23

Can you do dynamic classloading in this app?

jetzajac00:09:14

We can even delay some work on loading ns-es

ghadi00:09:23

Is this an intellij plugin? Because I know Jetbrains is religious about classloading

jetzajac00:09:56

not actually plugin, but yeah. we do load intellij on classpath

ghadi00:09:07

interesting

jetzajac00:09:16

we really do crazy stuff to optimize it, yeah

ghadi00:09:06

Yeah I remember talking to cfleming about this, I think he got spanked because cursive was slowing intellij startup

johnj00:09:47

he switched parts to Kotlin, read it somewhere

jetzajac00:09:48

it is the same problem, yeah

hiredman00:09:05

you might look at some of the clojure android stuff, they similar want basically and optimizing compiler

ghadi00:09:52

I wonder if you could do AOT then rewrite it with asm

jetzajac00:09:55

how do they do clojure on android with this problem? some byte code tree-shaking and post-processing?

ghadi00:09:36

You can also do manual tree shaking if you ship a custom clojure

jetzajac00:09:06

@ghadi how to do that? after emitting vars and all that?

jetzajac00:09:37

you mean to tree-shake clojure.core?

jetzajac00:09:11

clojure.core itself is quite fast to load. the problem seems to distribute between our codebase, datascript and core.async

ghadi00:09:24

You could remove things you know you won't call. Fork and delete...

jetzajac00:09:28

core.async can be made much faster by delaying ioc_macros

ghadi00:09:57

ioc macros shouldn't be needed except during macro expansion

ghadi00:09:13

If it is, we could fix that

jetzajac00:09:15

exactly! but is required eagerly!

jetzajac00:09:50

that is about 10-15% or our load-time. we need much more 😞

ghadi00:09:55

Yeah. I'd file a bug for that. Seems fixable

ghadi00:09:11

Ok we're making progress :)

jetzajac00:09:19

can even do that myself. just want to hit elephant first

ghadi00:09:08

What min JVM can you support?

ghadi00:09:41

I've been delaying class init work using constantdynamic on JDK 11

jetzajac00:09:09

we can use 10 I think

ghadi00:09:29

Could simulate condy using indy...

jetzajac00:09:51

never heard about constantdynamic

ghadi01:09:27

I hate to say it because I wrote part of it... but you could get rid of core.async

ghadi01:09:43

Not sure how much you lean on it

hiredman01:09:44

arrdem has some blog posts related to that which also might be interesting

jetzajac01:09:31

core.async is awesome and we rely on it so much 😃

ghadi01:09:45

Changing the way clojure loads namespaces is a promising possibility... You might break stuff though

ghadi01:09:05

I'm surprised and impressed that datascript doesn't have any transitive deps!

jetzajac01:09:13

that is ok, hava a compiler option 😃

jetzajac01:09:28

datascript is awesome too

cfleming01:09:38

@jetzajac clojure-android doesn’t solve this problem, which is one of the main reasons basically no-one uses it.

cfleming01:09:18

tools.emitter.jvm is not in a usable state right now, but it’s something I’ve thought about updating for similar reasons to yours.

cfleming01:09:18

Basically having a more hackable compiler. It’s much much slower than the Clojure compiler, but since I AOT I don’t care about that. Currently I run a fork of Clojure, and I’d much prefer to be able to implement the extensions I need as macros or analyser passes.

cfleming01:09:06

However I think the Clojure language design places some fundamental restrictions on startup time right now. It’s possible that redesigning the var implementation using indy might help that, but @ghadi is the man to ask about that.

seancorfield02:09:05

@priornix We'll take it to DM (there's #slack-help BTW for general Slack questions)

priornix02:09:19

Thanks will do

emccue03:09:05

I know one of the reasons you cant just trim out core statically is that any code can use "eval" and sidestep static analysis

emccue03:09:39

but how much does the clojure ecosystem/real world projects actually use eval?

tbaldridge04:09:20

I use eval as a JIT for my DSLs but I’m probably unique there :)

tbaldridge05:09:21

@jetzajac what problems are you experiencing with slow startup times? Most apps I’ve worked on in my Clojure career have taken close to a 60 Sec to start, none of them have been a problem beyond a slight annoyance.

tbaldridge05:09:30

Ah, I see, a plug-in

ahungry05:09:53

I'm using it for emacs like user config on my new browser written in clojure, works like a charm

ahungry05:09:01

people can bind any code expression or existing calls to any key without having to touch the source

hawari08:09:04

Hello everyone, So I'm trying to write a code for requesting quite large amount of URL using clj-http. My approach right now is by using pmap, it does the job, and I must note, faster than when I was using map. But then I notice that because for each request I didn't use any connection pool, the number of connections opened correlates linearly with the number of the URL I requested, this is unfortunate, since most of those URLs came from the same domain. When I tried to use with-connection-pool, wrapping my pmap call, it often results in an error stating that the connection pool has already closed. Has anybody ever succeeded in doing a bunch of HTTP requests in parallel, and by reusing connections?

ido08:09:06

@hawari.rahman17 you have to separate issues here. one is that http-kit does not use connection pooling by default and is very eager . for that, your connection pooling should work, but if you use different domains, it would not. the other problem is the use of macros like with* together wi realising lazy sequences. just wrap you pmap with a doall to prevent realising the lazy sequence after the pool is already closed.

ido08:09:10

for the general case of different domains I once use a java counting semaphore in combination with http-kit async api

henrik08:09:03

You could wrap it in channels as well with core.async if you want to stay within Clojure.

henrik08:09:04

Put the requests on a channel, have it take X requests, wait until those are done, put the results on another channel, take X more, and so on.

hawari08:09:40

Hi @ido, your solution by using doall to wrap pmap successfully prevents the connection pool created by with-connection-pool to be closed, but I don't understand why does it runs way slower than the "use pmap, don't use connection pooling" solution, is it highly dependent on the number of maximum threads option assigned to it? Hi @henrik, yes, core.async isn't a solution I've already explored yet, but I think I might need to state my use case first to improve the quality of discussion here:

My program first reads a response from a service in the forms of JSON. From this response, I want to hydrate each of the object, the hydration process involves calling another service which ideally I want to run concurrently.
So I tried to do the hydration requests in parallel using pmap which works well until I realize that it is using a new connection for each request. Isn't using core.async will results in the same situation?

hawari09:09:53

On second thought, for that use case, is using a new connection for each request necessarily a bad thing? Will connection reuse actually prove to be helpful instead of becoming a bottleneck?

hawari09:09:31

What caused me to worry is if for some reason the number of requests increases greatly, won't it exhaust the ports quickly if I stick to my current approach? What is the conventional wisdom here?

henrik09:09:15

@hawari.rahman17 Here’s an example. This solves the number of concurrent requests, but doesn’t deal with connection reuse.

hawari09:09:08

Ah, I see, maybe I can use this approach to use connection pool exclusively per partition, let's see if that's better Thank you for the inspiration @henrik

👍 4
p14n10:09:00

Can anyone recommend a clojure project that they really rate in terms of code style? I'd like to see how the pros do it

manuel12:09:45

I tried following the conventions mentioned in that guide here: https://github.com/manuel-uberti/boodle But I won't go as far as considering myself a pro 😄

lukas.rychtecky14:09:06

Hmm I don’t see any conventions on given links

p14n14:09:24

Thanks all - I meant just code I could look through to see how other have done it but all of that is useful - appreciated

the2bears15:09:34

weavejester writes good code, https://github.com/weavejester

☝️ 4
dadair16:09:56

Just wanted to give a heads up that I’ve created a new channel #healthcare for the discussion of clojure in healthcare. Thought it might be a good place for people working in or around that space to announce or collaborate on healthcare-specific libraries (FHIR specs, etc), for example.

👍 4
💯 4
dangercoder19:09:28

Anyone with experience in message passing? I am trying to create a queue right now for a game where a user can cancel/accept.

dangercoder19:09:00

tried to get some advice in /beginner a few days ago i dont really find much info on this subject. Maybe just suck at googling on this subject..

noisesmith19:09:38

core.async is one way of doing coordination via queues (aka channels in this case) - it's a whole dsl for async

hiredman19:09:47

what do you mean by a queue for a game?

hiredman19:09:17

usually a queue used for message passing isn't going to be directly shown to a user

hiredman19:09:07

a "queue" displayed to a user is mode likely to be a model of a queue stored in something like a database

dangercoder19:09:12

Well, I kind of have theese steps: • User clicks "Start" from the UI - will hit my api • I batch together users using LinkedBlockingQueue and refs • When 10 users are in the queue, I send a notification to the users. If they accept, I will create the game for them. If 1 user declines I will remove the declining user and re-fill the queue again. I think it's best to go with core async, and when I am done with the whole batching/accept-decline stuff I will send it on the message queue.

hiredman19:09:20

ah, so the queue isn't directly exposed to the user

dangercoder19:09:40

If I restart the server/crash it's ok to lose track of the users who are in "queue" so I can hold them in-memory I guess.

hiredman19:09:46

I think something like an agent might work really well

dangercoder19:09:47

@hiredman I'll look into it. Thank you and @U051SS2EU for the advice, it's gold 🙂.

hiredman19:09:28

have some datastructure where you keep track of the state of each user :joined :joining :available, and when a user joins, send-off something that adds them to the datastructure in the :available state, then tries to select 10 :available users and switch them to :joining, before the send-off the user adds a watch to the agent and then when it sees its own record in the joining state, it can either accept (moves to :joined) or decline(moves back to :available)

parrot 4
hiredman19:09:59

you'll need some additional state so you know which group is joining into a game, etc

parrot 4
dangercoder12:09:49

@hiredman do you think it's easier to seperate the queues into different stages. Like available-queue which only shuffles all users to joining-queue when enough users are found. And in the joining-queue have a datastructure like this:

{
 ;; agent!  "78bc215a-5a52-442a-b5ed-acb4a446f0d3" {
                                           :state :waiting
                                           :start "yymmddssmm"
                                           :game-type :chess1v1
                                           "1337" :waiting
                                           "1338" :accepted
                                           "1339" :accepted
                                           "1340" :waiting
                                           }
   }

Right now every single player is sent to a available queue which is wrapped in a agent with a watch attached to it. As soon as an item is inserted into the available-queue the watch is triggered and I check count etc. When the count is over the min-limit (2 atm) i take 2 items from the vectors head and create a new datastructure (as you see above). Might not be the best way to group things, but it's my plan right now to "group" things together. If a player would send me cancel I will have the guid (key) of the map and send all the users who responded with "ok" into the available queue again. I need to automatically clear the map since it can be in :waiting for maximum of 10 seconds.

dangercoder12:09:19

with stages.. I mean something like this

(def available-queue (agent []))                
(def joining-queue (agent {}))                     

dangercoder12:09:38

A problem is that it's not syncronized though, so 1 player could possibly be joining the joining-queue twice I guess. Or well, with the watch this should not be a problem. Also since I have to "remove" users from the "available-queue" its costly.

aarkerio23:09:55

newbie question: why this doesn't work : (swap! foo (#(str %) "foobar") )

hiredman23:09:46

look at the docstring for swap! and what arguments it takes

dadair23:09:56

swap! requires a function to apply, (#(str %) "foobar") returns "foobar", which isn’t a function

hiredman23:09:03

an atom, followed by a function, followed by optional arguments

hiredman23:09:54

(#(str %) "foobar") is the function #(str %) applied to the argument "foobar" which is "foobar"

hiredman23:09:14

so your call to swap! is (swap! foo "foobar")

aarkerio23:09:57

hmmmm, I see, there is a way to return a shorthanded lambda as itself?

hiredman23:09:09

just don't call it

hiredman23:09:22

lists are function application

aarkerio23:09:55

but (#(str %) "foobar") is a closure, right?

hiredman23:09:26

lists are function application

hiredman23:09:07

#(str %) is a function (closure, procedure, etc)

hiredman23:09:35

(#(str %) "foobar") is invoking that function with the argument "foobar"

hiredman23:09:58

just like (+ 1 2) is invoking + with the arguments 1 and 2

aarkerio23:09:03

well, I got that, but

hiredman23:09:44

#(str %) is also the same thing as str

aarkerio23:09:23

ok, but how can I pass "foobar" and not doing the list application ?

hiredman23:09:37

ah, I get it

sam23:09:42

swap! uses the value of the atom you pass it

hiredman23:09:54

you are expecting automatic currying like from ml or haskell?

hiredman23:09:36

anyway, you can use partial, or better yet include foobar in you closure

hiredman23:09:47

#(str % "foobar")

hiredman23:09:55

clojure doesn't have that

hiredman23:09:02

(automatic currying)

aarkerio23:09:49

ok, man, clojure is fun but so different, thanks a lot!