This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2015-10-23
Channels
- # announcements (1)
- # aws (11)
- # beginners (28)
- # boot (235)
- # business (1)
- # cider (19)
- # clojure (34)
- # clojure-china (1)
- # clojure-czech (10)
- # clojure-japan (7)
- # clojure-poland (3)
- # clojure-russia (84)
- # clojure-sg (4)
- # clojure-uk (3)
- # clojurescript (114)
- # community-development (4)
- # core-async (15)
- # cursive (8)
- # datascript (5)
- # datomic (6)
- # editors-rus (27)
- # events (2)
- # hoplon (61)
- # jobs (2)
- # ldnclj (56)
- # ldnproclodo (5)
- # lein-figwheel (232)
- # luminus (1)
- # off-topic (5)
- # om (215)
- # onyx (436)
- # overtone (8)
- # re-frame (3)
- # reagent (3)
@martinklepsch: from your screenshot, it looks like it might be uncleared an Var.dvals
ThreadLocal, which would happen when something pushes thread bindings w/o a pop
@micha: does (some #{"boot.pod"} (map str (all-ns)))
seem like a good way to test if a project is running in boot?
@micha: is there a way to create an executable like boot
itself from an uberjar generated by boot pom uber --as-jars jar
?
i understand that by default i can't just give such an uberjar to java directly only to some servlet container because only such containers would understand how to add the embedded jars to the classpath
nvm, i misunderstood the use case
i'm trying to help the cider ppl
@onetom: there are a few things going on... to make an executable uberjar, you can just make a .sh with a leader similar to boot.sh's, can cat your uberjar to it
but yeah we have an idea to make a task called "uberboot" which generates one of these executables, and also packages the boot runtime... but the user supplies the main method
i made an experimental thing, https://github.com/adzerk-oss/boot-ubermain, which does some of the work
but we will need to make some changes to App for it to really work
@alandipert: so currently the boot uber --as-jars
output wouldn't work with the .sh leader?
well, they're somewhat unrelated
there are three big differences between the output of boot uber --as-jars
and what boot.sh is
first, the leader - this is trivial
second, the main method - this is tricky. one would need to AOT java or clojure and specify an entrypoint method other than boot's in order for java -jar uberjar.jar
to work
finally, with just a plain main method, the jars bundled with --as-jars wouldn't be added to the classpath
or "exploded" onto the classpath
that's something servlet containers do
it's also what my boot-ubermain thing does... it lets you specify an entrypoint function, and handles exploding the jars
but yeah, the stuff that's needed, is related to making the boot runtime available to apps that are distributed this way
currently the way App works is coupled to the way boot runs its own entrypoint, so other entrypoints don't have access to pods
ok, thx, i understand it roughly now. the whole question only came up because a, it takes 4 minutes to create an uberjar and it is 47MB b, on nixos it's a hassle to let boot download dependencies during runtime
i see
@onetom: perhaps what you want is a task that can build a nixos package with the dependencies?
Does anyone know why I'd be getting java.lang.Exception: missing jar file or repo not found
when trying to boot build-jar push-snapshot
? Here's the build.boot
I'm using: https://github.com/bmaddy/ring-proxy/blob/master/build.boot
(this is my first time trying to use boot, fwiw)
@micha: thanks for the "wagon" hint! we do have already a nixos package builder, it's just not very well tested yet: https://github.com/exicon/boot-nix
@bmaddy: build-jar
is a task provided by bootlaces. If you want to use it make sure all required settings are made by calling (bootlaces! version-of-your-lib)
Thanks @martinklepsch, that and changing the project group did the trick!
@tcrawley: would be happy to chat, let me know when you’re free
@martinklepsch: let me see if I can recreate the issue locally
tcrawley: There are repro instructions in the issue in case you haven’t seen
@tcrawley: thanks a ton in advance for your help! 😊
@micha: hey. In commit!
couldn’t we generate a patch op similar to how it’s done with sync!
(https://github.com/boot-clj/boot/blob/master/boot/pod/src/boot/tmpdir.clj#L67 vs. https://github.com/boot-clj/boot/blob/master/boot/pod/src/boot/file.clj#L185)
Currently every commit!
will wipe all tmp dirs if I understand correctly
@martinklepsch: what would be the advantage to generating a patch
less IO.
deleting things and creating links is io too, no?
creating those links has some cost and I assume esp. with large filesets that hurts
If I understand this whole profiling UI correctly linking makes 30% of commit!
time now the remainder seems to be mostly reflection
(could totally be that I’m misreading this thing)
@micha: makes 10% of overall cpu usage with some basic cljs thing, how do you define fast?
@micha: @alandipert will add the info to wiki
No problems with the gpg pr. Take your time. This is not a small PR, and everybody wants and deserves to have their voice heard. I think Deraen has a good point. I would support a gpg-agent based implementation.
Been using it for some days successfully (working on project with Datomic Pro). Only problem was configuring GPG chain on OS X.
i was thinking about maybe incorporating google measurement api in boot to get some visibility into which parts are used the most and where exceptions are coming from etc
I think it would be really cool
thought of this before. makes total sense to me that also open source apps need to have some analytics
fileset statistics
number of tasks in pipeline
@micha: maybe open a RFC issue and then we can list interesting metrics and get feedback if people are fine w/ it
@martinklepsch: I think I figured it out ^
@tcrawley: reading..
@martinklepsch: can you try your pod-leak script on clj 1.6.0 please?
@tcrawley: thanks for the detailed writeup! When I was looking into loaded classes I briefly searched for some of the Var
ones you mention but didn’t see any. Is that possible or did I just not look properly?
can confirm that class unloading happens with your thread shim
👏 👏 👏 👏
what tool were you using? I used Eclipse MAT, and could see paths to the class loader that were preventing GC: http://p.tcrawley.org/i/227d5.png
in that, you can see the ThreadLocal pointing to a Var$Frame, which would only happen if Var.dvals was set without being cleared
@tcrawley: used yourkit, now that you say it that image has it too.
@tcrawley: could you explain whats different from regular shimdandy usage in pods that made this Var$Frame thing occur? (do you know?)
does someone know if it’s possible to override a single function of a protocol implementation? from #C03S1KBA2 > I’d like to override a single method of a protocol for a specific type — is that possible at all? extend-type tells me the protocol is already defined for this type
@martinklepsch: re: fileset perf i fixed most of the reflection in core, didn't make a noticeable difference
I have a couple of changes that make a 13s > 3s drop for filesets with 3.5k files
excellent!
did you identify anything as the slowest thing?
not fully aware of all implications I guess but worth exploring more
@alandipert: commit! deletes everything every time and recreates all files/links
@alandipert: do you have an idea regarding the protocol q? Would like to make a test case repo with some changes to boot itself but without requiring people to rebuild boot
using alter-var-root for most stuff but that doesn’t work for protocol impls I think
@martinklepsch: you could try extend-type all the methods, but call-through to the existing methods in the bodies
that's the best i can do without looking at the protocol caching code... presumably you could expire entries manually
@martinklepsch: that's a good question. I used ShimDandy in Immutant 1.x and in the clojure impl for Vert.x 2.x. in both of those cases, shim creation happens on a deployment thread that is disposed of, so my first thought is that is what prevented it. but I don't think that would have been enough. In Immutant 1.x, there was a pool of threads that handled web requests that was shared across all apps, shim.invoke was called from those
@isaac: http://synergian.github.io/wagon-git/ for example
@martinklepsch: what are you trying to override? i can play with it
@isaac: https://github.com/boot-clj/boot/wiki/S3-Repositories is an example for an S3 wagon
@alandipert: @micha oh! thanks
@alandipert: delete this line https://github.com/boot-clj/boot/blob/master/boot/pod/src/boot/tmpdir.clj#L67
@micha: @alandipert https://github.com/martinklepsch/boot-fileset-perf/blob/master/build.boot
@martinklepsch: did you try (alter-var-root #'boot.tmpdir/commit! ... ) ?
The main change is really that copy-with-lastmod
does not always delete dest-file
but instead checks lastModified if the source file is older than dest-file … now that I’m reading what I’m writing it seems really confusing 😧
@martinklepsch: the fileset assumes that the files themselves (in the blob storage dir) are immutable (i.e. they will be written to only once and then read-only after that)
probably a better approach than changing copy-with-lastmod is to generate patch actions in commit!
the hardlinks that copy-with-lastmod
creates get their permissions and modtimes from the blob file they link to
yeah that makes sense (blob stuff)
although that function doesn't make a complete patch, it simplifies a little by ignoring deleted files i think
or boot.file/sync!
— at the point of commit I don’t have two filesets to compare
couldn’t we also just construct that tree from a list of tmpdirs?
also do we all feel pretty good about eliminating :target-path
in set-env!
in favor of a target
task that explicitly dumps to a directory of your choice?
I think that would be useful
see boot.file/tree-for
Would make it easier to dump uberjar directly to Ansible dir or such
or giant vendor resources with tens of thousands of files that the build doesn't care about
And I guess having separate target-dirs for dev and prod (uberjar) would be really good practive (or well, no target dir for dev at all)
Noticed one day that if I change files while doing deployment dev task will overwrite target dir and remove the jar file and thus the deployment will fail 😄
tree-for
doesn’t seem to recurse into nested dirs? if that’s somehow possible it would only be a couple of ls
and getting lastmod
Ah I see. In any case I think might be worth trying, reading is much faster than writing
i could work on a branch that compares before and after filesets in commit!
and see if that helps, too
as opposed to getting a tree from the fs?
It introduces more state 😛
I was also thinking that it would be cool to isolate the fileset stuff into a separate library with proper public api etc.
I’m heading off for a bit. Later)1
@martinklepsch: it doesn't introduce any more state, the fileset is all about state
@martinklepsch: @micha: more context on the memory leak - I may not have seen it before because a (binding [...
call in the right place could prevent it from occurring. That would trigger a Var.popThreadBindings()
call, which would remove the thread local
I do have a simpler solution than using a separate thread in shimdandy that I'm testing now
basically, it uses reflection to clear Var.dvals
after an invoke if it is the top of the call stack for the current thread
@juhoteperi: i was just talking with @alandipert and i think we have a way to do the target
task without a breaking change: the task would add metadata to the fileset before passing it down the pipeline, and boot could look for that and if it's present not write to the :target-path
@micha: Sounds reasonable
We should probably check if the GPG change can also made without breaking current options
yeah i think the gpg thing could accomodate new things when certain env vars are present
I'm not sure if we need any env vars for that
GPG binary only needs to be used when user has configured repository which should read credentials from encrypted file -> new feature anyway and when signing file and password hasn't been provided
because you'd be likely to use it in an unattended deploy situation and things like that
When signing a file and password is provided we could fall back to current stuff (or pass the password to gpg binary)
What about separate deploy-repositories?
If user doesn't set them it could fallback to normal repositories, so it would work like now?
Perhaps
Hmm, yeah task options is probably better place for them than env
I can find some references about using Gpg-agent together with Bouncy Castle lib. But looks like requires implementing some custom clases.
@micha I'm using :target-path
-- the build.boot has about a dozen tasks -- I would need to add (target ...)
to each?
@pandeiro also you'd add target at the end of the pipeline not for each task
@martinklepsch: ah yeah ok, so it would have to be in three or four comp
tasks only