Fork me on GitHub
#clojure
<
2021-02-04
>
Tanel03:02:11

What's the best way of debugging why an application takes so long to start? I'm using Clojure component along with a jetty HTTP server, a HikariCP instance and a few RabbiMQ listeners:

$ lein start
2021-02-04 05:50:48.848 INFO  [org.eclipse.jetty.util.log] - Logging initialized @17043ms to org.eclipse.jetty.util.log.Slf4jLog
start alias:
:aliases {"start" ["with-profile" "dev" "trampoline" "run"]}

p-himik03:02:36

FWIW I have found significant improvements in startup time when I got rid of lein altogether. It's been quite some time ago and I didn't try to dig deeper.

Tanel04:02:35

Are you using deps.edn? How does it compare to lein in your opinion?

seancorfield04:02:34

The Clojure CLI and deps.edn is focused on building a classpath and running some code. That's it. Leiningen is a full-on build tool that does "everything".

p-himik04:02:10

I find it much simpler and much more pleasant to work with in my use cases. The only hiccup for me was working with Java sources that are shipped within the same project. But there are ways around that.

seancorfield04:02:24

That said, there are lots of community-provided tools for the CLI that provide all the same "features" as Leiningen but a la carte instead of bundled.

seancorfield04:02:05

We run a 100K line codebase with the CLI. A monorepo with three dozen subprojects and we build over a dozen production services from it.

seancorfield04:02:56

(we switched away from Leiningen to Boot about five/size years ago, then switched to the CLI/`deps.edn` back in 2018 I think)

seancorfield04:02:51

As for your original Q @UBYSF8K2N building an uberjar with AOT compilation will really help startup time.

seancorfield04:02:47

We had several processes that took 10-20 seconds to start up from source but only take up to about five seconds to start as AOT'd uberjars.

Tanel04:02:57

I see. Thanks for the suggestions!

vemv04:02:49

> What's the best way of debugging why an application takes so long to start? How much time are we talking about? deps.edn can shave a few seconds (1 fewer JVM + cached classpath calculation) but that might not make a difference. My app at work takes ~4m to boot from source, it basically boils down to how many clojure namespaces are there. The clj compiler isn't particularly fast

seancorfield04:02:55

@U45T93RA6 See the OP -- it showed about 17 seconds to get to initializing the logging.

👍 3
Tanel04:02:07

@U45T93RA6 So 17 seconds is quite fast compared to yours. 😅

vemv04:02:32

@ Sean whoops, didn't squint hard enough 😇 Yeah in this case deps.edn can make a difference. To be sure, you can ps aux | grep java and copy the java command that Lein generates (`lein run` is a JVM program which ultimately simply builds a java invocation to be executed in a new process). Then you can run that java invocation from scatch, measuring your app with zero Lein overhead If you perceive a difference, sure go ahead with deps! (remove trampoline for extra accuracy)

👍 3
seancorfield04:02:05

@UBYSF8K2N If it's a fairly straightforward process, it's probably just going to be clojure -M:dev -m your.main.namespace where :dev is a deps.edn alias that is equivalent to whatever special stuff you have in your dev profile in project.clj and your.main.namespace would be, well, your main namespace.

👍 3
mpenet07:02:52

You might want to uberjar/aot the app if it is for prod, startup will then be faster too

noisesmith18:02:36

FWIW there's also a "LEIN_FAST_TRAMPOLINE" flag that makes lein use a cached classpath for the trampoline task trampoline makes lein use a single jvm (like clj does) and fast trampoline makes it reuse a cached classpath (like clj does)

West06:02:19

How do I use a java dependency that’s not on maven central repository? I want to use https://www.w3.org/Style/CSS/SAC/ in my Clojure project.

phronmophobic06:02:41

to answer your original question, are you using deps.edn? if so, you can use a local jar https://clojure.org/guides/deps_and_cli#local_jar

West06:02:39

I just found it on maven after searching.

West06:02:47

That was dumb lol

West06:02:54

But thank you.

West06:02:31

Ok, so now I can use it in my deps.edn yeah?

:deps {org.clojure/clojure {:mvn/version "1.10.2"}
        clj-css/clj-css     {:mvn/version "0.1.0-SNAPSHOT"}
        garden/garden       {:mvn/version "1.3.10"}
        org.w3c.css/sac     {:mvn/version "1.3"}}

phronmophobic06:02:19

yea, adding the maven dep should be the most straightforward

West06:02:28

Ok, I’m probably missing something dumb here. What’s going on?

phronmophobic07:02:09

if it's a java class, then you need an import, rather than require

West07:02:56

ah ok let me try that

West07:02:22

I think I got it. I used a dot when I called one of the methods, then I got a different error saying it got the wrong type. I think this is closer to what I’m looking to do.

West07:02:24

Thanks for your help!

👍 3
West07:02:54

Ok, I really don’t understand how to interop with Java libs at all. I’m trying to read http://cssparser.sourceforge.net/gettingStarted.html and get the methods working in clojure. net.sourceforge.cssparser/cssparser {:mvn/version "0.9.29"} Are there any resources on this topic you can recommend?

West04:02:07

I’m having a lot of trouble understanding this document. I have no idea how java works at all. I have no problem using java.lang stuff, but this I can’t seem to wrap my head around.

West04:02:34

At the very least my completion engine caught this information about the class/method.

beders05:02:25

are you familiar with OOP? Calling something in Java requires referencing a method, which only exists in two or three contexts: a static method of a class/interface or a non-static method. For that one you need an instance of a class - an object. CSSOMParser/parseStyleSheet refers to this: http://www.massapi.com/method/com/steadystate/css/parser/CSSOMParser.parseStyleSheet.html#Example0 and is a non-static method and can only be called on an object. The sample code linked shows you can create such an object. Does this level of detail help?

West05:02:38

I’m not familiar with OOP at all. So I have to create an instance of a class. Would I do that using (new CSSOMParser)?

seancorfield05:02:50

@U01KQ9EGU79 What's your background? Maybe that'll help us help you better...

seancorfield05:02:58

(if you already said and I missed it, sorry -- feel free to link me to an earlier post)

West05:02:40

I’m honestly a beginner with clojure and clojurescript. I have no idea really how OOP works other than the basic idea, i.e. Classes are like compound variables and methods are just functions.

seancorfield05:02:48

And your previous languages are...?

West05:02:49

The functional simplicity of clojure gave me a lot of confidence in building stuff early on, but I’m having trouble leveraging all these java libraries other than http://java.io and things like that.

West05:02:19

Before I did shell script and python.

West05:02:02

I couldn’t wrap my head around objects in python either, unless I was straight up calling a method to do something with strings.

seancorfield05:02:11

Okay, yeah, I think quite a few people have come to Clojure from Python but yeah, if you're working with Java libs, they'll be a lot more OO than even if you'd been comfortable with OO in Python...

West05:02:10

Aww man. I guess imma have to learn OOP after all.

seancorfield05:02:41

You might also want to focus on asking in #beginners rather than in here in #clojure -- the folks who've opted in to help in #beginners are willing to put in a lot of time helping folks with all aspects of coming up to speed on Clojure including spending time working with you on getting more comfortable with OO.

seancorfield05:02:47

In this channel, folks are likely to make assumptions about what you know and that will probably include assuming you are comfortable with OO.

West05:02:48

Got it, imma stay in beginners for awhile longer then. Thanks for your advice!

seancorfield05:02:03

You'll find quite a bit of overlap between the folks who'll likely help you -- but they'll approach it with different assumptions there so hopefully that will help!

Adam Helins11:02:46

There isn't any way of knowing what a namespace currently requires just by using clojure.core, is there ? (ie. without resorting to tools.namespace)

vemv12:02:30

note that t.n.parse does not have any side-effect (unlike (refresh) and friends)

👍 3
borkdude12:02:17

@adam678 You can use clj-kondo for this which determines this through static analysis. See https://github.com/borkdude/clj-kondo/blob/master/analysis/README.md

Adam Helins13:02:45

Good call but needs to be dynamic here

borkdude13:02:51

@adam678 ok, another way. use a combination of (ns-aliases *ns*) and (ns-refers *ns*). Iterate over the namespaces and vars returns from both and you will have the dynamic answer. This works in CLJ, not sure about CLJS.

Adam Helins13:02:19

Almost! Unless I am doing it wrong, it won't show namespaces which aren't aliased nor referred. But it might be good enough, who knows... thanks!

bendy13:02:56

Hello all - anytime I try to run lein repl now anywhere, even if there is no project.clj, it throws the following error:

$ lein repl
OpenJDK 64-Bit Server VM warning: Options -Xverify:none and -noverify were deprecated in JDK 13 and will likely be removed in a future release.
Retrieving fipp/fipp/0.6.23/fipp-0.6.23.pom from clojars
Retrieving fipp/fipp/0.6.23/fipp-0.6.23.jar from clojars
clojure.lang.Compiler$CompilerException: Syntax error compiling at (fipp/ednize.clj:1:1).
#:clojure.error{:phase :compile-syntax-check, :line 1, :column 1, :source "fipp/ednize.clj"}
 at clojure.lang.Compiler.load (Compiler.java:7648)
    clojure.lang.RT.loadResourceScript (RT.java:381)
Has anyone run into this, or does anyone know how to get around it?

borkdude13:02:46

@bendy This might come from something in your ~/.lein

Franklin09:04:18

this helped me solve a similar issue 🙂

bendy13:02:00

@borkdude legend, that solved it! No idea what was even in there 😂

1
🤯 1
borkdude13:02:51

@adam678 ok, another way. use a combination of (ns-aliases *ns*) and (ns-refers *ns*). Iterate over the namespaces and vars returns from both and you will have the dynamic answer. This works in CLJ, not sure about CLJS.

Shantanu Kumar18:02:12

Does mutating transient vector not change the root reference? E.g.

(let [t (transient [])] (dotimes [i 20000] (conj! t i)) (persistent! t))
appears to work fine

noisesmith18:02:24

@kumarshantanu it's allowed to, but doesn't promise to

noisesmith18:02:35

your code will break if you rely on it

dpsutton18:02:43

> Note in particular that transients are not designed to be bashed in-place. You must capture and use the return value in the next call. https://clojure.org/reference/transients

noisesmith18:02:20

"mutating a transient vector" - a conj! is allowed to mutate, doesn't promise to mutate

Shantanu Kumar18:02:22

Because for transient maps the root ref updates, so I was a bit surprised at transient vectors.

dpsutton18:02:48

another thing to remember is that reading code doesn't particularly help you. This is explicitly not in the contract so reading code has to be phrased as "the implementation right now does..."

jaihindhreddy18:02:56

The conj! impl of transient vectors https://github.com/clojure/clojure/blob/140ed11e905de46331de705e955c50c0ef79095b/src/jvm/clojure/lang/PersistentVector.java#L579-L608, but as others have pointed out, the docs instruct us to capture the return.

andy.fingerhut18:02:02

The supported contract is always to use the return value of any transient operation, just as you would for a persistent operation. It might be identical? to what you gave it, it might not. If you use the original reference for further operations, you are breaking the contract, and might never get any error or warning that you are doing so.

💯 3
andy.fingerhut18:02:23

and the confusion arises because in many cases, it works as you would hope, even if you break the contract

Alex Miller (Clojure team)18:02:41

There is a pending ticket to make the doc string clearer about this

jaihindhreddy18:02:46

I wonder why that is, however. Is the reasoning to keep code close to the functional equivalent (as the docs describe), or to have flexibility to return a different thing in the future (much like array-maps are switched out to hash maps)...

andy.fingerhut18:02:47

I would guess the second reason.

👍 3
hiredman18:02:06

transients are sort of a fusion between a mutable datastructure and an immutable one. internally they build the same tree structure as the immutable datastructure, they just mutate the structure when they can, otherwise they return a new tree. so for example if each node in the tree can hold 32 elements, when you add elements 0 to 31 you just mutate in place, and then once you add the 32 it has to split the tree

💯 3
andy.fingerhut18:02:15

If Rich didn't want that flexibility, then why would the current implementation sometimes return a non-identical transient collection?

andy.fingerhut18:02:06

I have a very difficult time believing the answer "that was an accident of the implementation"

3
jaihindhreddy18:02:48

> why would the current implementation sometimes return a non-identical transient collection? I didn't know this is the case.

hiredman18:02:33

but it is really two questions right? why does that behavior pattern exist, and why do transients expose it instead of hiding it

andy.fingerhut18:02:02

That is the reason that some people bring up the question of "hey, this code using transients isn't working" -- i.e. they wrote code that ignored the return value, and sometimes it works (when the implementation returns an identical object), but in other cases, it fails (when the transient operation returns a non-identical object)

andy.fingerhut18:02:49

@U0NCTKEV8 Agreed that an implementation of transients could guarantee to return an identical object always, if the implementer wanted to.

jaihindhreddy18:02:52

> why do transients expose it instead of hiding it The docs being what they are, isn't it fair to say that transients do hide this?

hiredman18:02:34

it depends on what you think is being hidden

jaihindhreddy18:02:50

"the fact that the output may be identical?"

hiredman18:02:50

they expose the fact that sometimes you get a new structure instead of mutating the existing one

👍 3
hiredman18:02:10

they could hide that if the whole thing was wrapped in a mutable reference

✔️ 3
jaihindhreddy18:02:19

ahh, I'm coming at it from the other side.

andy.fingerhut18:02:27

The true answer to "why is it this way?" is truly only known by the implementer, if you want all the details and tradeoffs involved in the decision

andy.fingerhut18:02:38

While it would be possible to design transients that guarantee returning an identical object always, I believe it would involve an extra level of indirection near the root, in cases where the implementer wants to switch data structures, e..g from array-map to hash-map. That would have some performance cost. Maybe one that some implementers would find acceptable, and others would not find acceptable.

hiredman18:02:11

La mort de l'auteur for programming

😂 3
dpsutton18:02:16

yeah i was disappointed that (find-doc "bash") didn't return any warning. glad there's a ticket

wombawomba20:02:00

What's a good lightweight library for making/unpacking tar files?

Alex Miller (Clojure team)20:02:42

apache io probably has that

borkdude20:02:48

@wombawomba clj-commons/fs also comes with this lib (the apache one): https://github.com/clj-commons/fs/blob/master/src/me/raynes/fs/compression.clj but note that tar is now available on all major OSes, including Windows, so shelling out isn't even that strange to avoid this dep

wombawomba20:02:19

cool, thanks — will take a look 🙂

ag21:02:56

I need a get-in that works not only for sequence of keys but also for functions, I can probably spend some time writing it, but perhaps someone already has done that before. So I basically I want something like this:

(get-in* {:foo [{:bar 1}]}
         [:foo first :bar]) ; => 1
(get-in* {:foo [{:bar 1}
                {:bar 42}]}
         [:foo second :bar]) ; => 42
Can you share a snippet?

Ed09:02:38

you could use reduce:

(let [get-in* (partial reduce #(%2 %1))] (get-in* {:foo [{:bar 1}]} [:foo first :bar]))

ag21:02:57

Please don't suggest to use Meander or Specter.

😃 3
dpsutton21:02:04

its not fully general but you can use 0 here

dpsutton21:02:47

it doesn't work for functions but in this tangible example vectors are associative with respect to index. so (get-in coll [:foo 0 :bar]) will achieve your result

ag21:02:30

ah, right... that does it. thanks Dan!

parens 9
caumond05:02:54

In case your conditions to navigate are getting more and more complex, have a look to specter : https://github.com/redplanetlabs/specter