Fork me on GitHub
#boot
<
2015-10-23
>
tcrawley01:10:51

@martinklepsch: from your screenshot, it looks like it might be uncleared an Var.dvals ThreadLocal, which would happen when something pushes thread bindings w/o a pop

tcrawley01:10:15

Var$Frame is only ever used on the dvals stack

tcrawley01:10:16

is this from issue #268?

tcrawley01:10:40

is that screenshot from an analysis of the test app from that issue?

tcrawley01:10:24

ping me in the morning (US-Eastern), and I'd be happy to take a deeper look

alandipert01:10:13

@micha: does (some #{"boot.pod"} (map str (all-ns))) seem like a good way to test if a project is running in boot?

micha01:10:18

or maybe look for boot.App Java class?

micha01:10:39

there is no way to use that as a library

micha02:10:08

what would the use cae be?

onetom02:10:03

@micha: is there a way to create an executable like boot itself from an uberjar generated by boot pom uber --as-jars jar?

onetom02:10:34

i remember a few month ago some work was done on that, no?

onetom02:10:39

i understand that by default i can't just give such an uberjar to java directly only to some servlet container because only such containers would understand how to add the embedded jars to the classpath

alandipert02:10:31

nvm, i misunderstood the use case

alandipert02:10:36

i'm trying to help the cider ppl

alandipert02:10:30

@onetom: there are a few things going on... to make an executable uberjar, you can just make a .sh with a leader similar to boot.sh's, can cat your uberjar to it

alandipert02:10:00

but yeah we have an idea to make a task called "uberboot" which generates one of these executables, and also packages the boot runtime... but the user supplies the main method

alandipert02:10:22

i made an experimental thing, https://github.com/adzerk-oss/boot-ubermain, which does some of the work

alandipert02:10:33

but we will need to make some changes to App for it to really work

onetom02:10:18

@alandipert: so currently the boot uber --as-jars output wouldn't work with the .sh leader?

alandipert02:10:17

well, they're somewhat unrelated

alandipert02:10:38

there are three big differences between the output of boot uber --as-jars and what boot.sh is

alandipert02:10:45

first, the leader - this is trivial

alandipert02:10:13

second, the main method - this is tricky. one would need to AOT java or clojure and specify an entrypoint method other than boot's in order for java -jar uberjar.jar to work

alandipert02:10:42

finally, with just a plain main method, the jars bundled with --as-jars wouldn't be added to the classpath

alandipert02:10:56

or "exploded" onto the classpath

alandipert02:10:03

that's something servlet containers do

alandipert02:10:24

it's also what my boot-ubermain thing does... it lets you specify an entrypoint function, and handles exploding the jars

onetom02:10:56

we just looked into the head.sh again and its functionality is clear 👌

alandipert02:10:50

but yeah, the stuff that's needed, is related to making the boot runtime available to apps that are distributed this way

alandipert03:10:11

currently the way App works is coupled to the way boot runs its own entrypoint, so other entrypoints don't have access to pods

onetom03:10:05

ok, thx, i understand it roughly now. the whole question only came up because a, it takes 4 minutes to create an uberjar and it is 47MB b, on nixos it's a hassle to let boot download dependencies during runtime

micha04:10:24

@onetom: perhaps what you want is a task that can build a nixos package with the dependencies?

micha04:10:00

or maybe a more general maven->nixos compatibility thing, like a nixos wagon

micha04:10:43

"wagons" are what they call the implementations of the maven dependency fetching layer

bmaddy04:10:52

Does anyone know why I'd be getting java.lang.Exception: missing jar file or repo not found when trying to boot build-jar push-snapshot? Here's the build.boot I'm using: https://github.com/bmaddy/ring-proxy/blob/master/build.boot (this is my first time trying to use boot, fwiw)

onetom05:10:35

@micha: thanks for the "wagon" hint! we do have already a nixos package builder, it's just not very well tested yet: https://github.com/exicon/boot-nix

martinklepsch08:10:48

@bmaddy: build-jar is a task provided by bootlaces. If you want to use it make sure all required settings are made by calling (bootlaces! version-of-your-lib)

bmaddy10:10:50

Thanks @martinklepsch, that and changing the project group did the trick!

martinklepsch11:10:01

@tcrawley: would be happy to chat, let me know when you’re free simple_smile

tcrawley12:10:41

@martinklepsch: let me see if I can recreate the issue locally

martinklepsch12:10:10

tcrawley: There are repro instructions in the issue in case you haven’t seen

tcrawley12:10:33

yep, looking at that now

martinklepsch12:10:45

@tcrawley: thanks a ton in advance for your help! 😊

tcrawley12:10:12

well, don't thank me yet :)

martinklepsch12:10:15

Currently every commit! will wipe all tmp dirs if I understand correctly

micha12:10:59

@martinklepsch: what would be the advantage to generating a patch

micha12:10:42

there's no IO involved in commit! though

micha12:10:50

it's just making links

martinklepsch12:10:04

deleting things and creating links is io too, no?

micha12:10:37

yes but very small, i imagine

martinklepsch12:10:53

creating those links has some cost and I assume esp. with large filesets that hurts

micha12:10:58

yeah it could be

micha12:10:19

commit is definitely a linear time operation now

micha12:10:51

maybe with patches you could get it down to O(log(n)) or so

martinklepsch12:10:31

If I understand this whole profiling UI correctly linking makes 30% of commit! time now the remainder seems to be mostly reflection

micha12:10:36

i'm surprised at the cost of reflection

micha12:10:25

but commit! is fast even with large filesets, no?

martinklepsch12:10:32

(could totally be that I’m misreading this thing)

martinklepsch12:10:20

@micha: makes 10% of overall cpu usage with some basic cljs thing, how do you define fast?

micha12:10:52

that's way more than i would have expected, yeah

danielsz12:10:24

There 's a new boot task in town!

danielsz12:10:05

@micha: @alandipert will add the info to wiki

micha12:10:23

this is great!

micha12:10:47

sorry for the delays with the gpg pr

danielsz12:10:53

thanks @micha!

danielsz13:10:19

No problems with the gpg pr. Take your time. This is not a small PR, and everybody wants and deserves to have their voice heard. I think Deraen has a good point. I would support a gpg-agent based implementation.

micha13:10:35

yeah me too

micha13:10:13

perhaps with fallback if its not too complicated

juhoteperi13:10:33

Been using it for some days successfully (working on project with Datomic Pro). Only problem was configuring GPG chain on OS X.

micha13:10:04

i was thinking about maybe incorporating google measurement api in boot to get some visibility into which parts are used the most and where exceptions are coming from etc

micha13:10:09

opt-in of course

micha13:10:23

and anonymous

micha13:10:32

is that crazy?

micha13:10:48

it's like google analytics for applications

martinklepsch13:10:57

I think it would be really cool

martinklepsch13:10:24

thought of this before. makes total sense to me that also open source apps need to have some analytics

micha13:10:27

yeah i think it could be a good QA resource

micha13:10:23

help with profiling too

martinklepsch13:10:49

fileset statistics

martinklepsch13:10:03

number of tasks in pipeline

martinklepsch13:10:54

@micha: maybe open a RFC issue and then we can list interesting metrics and get feedback if people are fine w/ it

micha13:10:15

yeah, a wiki page maybe?

danielsz13:10:40

do you have figures on adoption?

danielsz13:10:52

That's what I worry about.

micha13:10:06

we don't have any figures really

danielsz13:10:00

Boot is still the underdog, isn't it?

micha13:10:13

presumably

danielsz13:10:34

The world is full of injustice.

micha13:10:24

yeah where is kofi annan when you need him

tcrawley13:10:39

@martinklepsch: I think I figured it out ^

martinklepsch13:10:05

@tcrawley: reading..

micha13:10:30

@martinklepsch: can you try your pod-leak script on clj 1.6.0 please?

micha13:10:38

i'm thinking it might be a 1.7.0 change

micha13:10:50

i did all my profiling on 1.6

tcrawley14:10:14

@micha: I recreated it on 1.6 - see my comment above for the cause

martinklepsch14:10:11

@tcrawley: thanks for the detailed writeup! When I was looking into loaded classes I briefly searched for some of the Var ones you mention but didn’t see any. Is that possible or did I just not look properly?

martinklepsch14:10:38

can confirm that class unloading happens with your thread shim

martinklepsch14:10:56

👏 👏 👏 👏

tcrawley14:10:03

what tool were you using? I used Eclipse MAT, and could see paths to the class loader that were preventing GC: http://p.tcrawley.org/i/227d5.png

tcrawley14:10:53

in that, you can see the ThreadLocal pointing to a Var$Frame, which would only happen if Var.dvals was set without being cleared

martinklepsch14:10:00

@tcrawley: used yourkit, now that you say it that image has it too. simple_smile

tcrawley14:10:34

ah, right. your image yesterday is what made me suspect Var$Frame :)

tcrawley14:10:49

@micha: I'll see if I can get you a new ShimDandy today

martinklepsch14:10:52

@tcrawley: could you explain whats different from regular shimdandy usage in pods that made this Var$Frame thing occur? (do you know?)

martinklepsch14:10:00

does someone know if it’s possible to override a single function of a protocol implementation? from #C03S1KBA2 > I’d like to override a single method of a protocol for a specific type — is that possible at all? extend-type tells me the protocol is already defined for this type

alandipert14:10:47

@martinklepsch: re: fileset perf i fixed most of the reflection in core, didn't make a noticeable difference

martinklepsch14:10:17

I have a couple of changes that make a 13s > 3s drop for filesets with 3.5k files

alandipert14:10:47

did you identify anything as the slowest thing?

martinklepsch14:10:01

not fully aware of all implications I guess but worth exploring more

martinklepsch14:10:25

@alandipert: commit! deletes everything every time and recreates all files/links

martinklepsch14:10:35

@alandipert: do you have an idea regarding the protocol q? Would like to make a test case repo with some changes to boot itself but without requiring people to rebuild boot

martinklepsch14:10:10

using alter-var-root for most stuff but that doesn’t work for protocol impls I think

micha14:10:10

proxy might be a way

alandipert14:10:51

@martinklepsch: you could try extend-type all the methods, but call-through to the existing methods in the bodies

alandipert14:10:16

that's the best i can do without looking at the protocol caching code... presumably you could expire entries manually

isaac14:10:32

Is there any way use add git dependencies

tcrawley14:10:48

@martinklepsch: that's a good question. I used ShimDandy in Immutant 1.x and in the clojure impl for Vert.x 2.x. in both of those cases, shim creation happens on a deployment thread that is disposed of, so my first thought is that is what prevented it. but I don't think that would have been enough. In Immutant 1.x, there was a pool of threads that handled web requests that was shared across all apps, shim.invoke was called from those

tcrawley14:10:48

threads, so would have set dvals. so I'm at a loss

micha14:10:00

isaac: there is a git wagon for maven i think

alandipert14:10:09

@martinklepsch: what are you trying to override? i can play with it

micha14:10:10

is that what you're looking for?

isaac14:10:33

ruby's gem is a good package manage

isaac15:10:59

I think it is good if boot integrate wagon

micha15:10:29

you can use the :wagons key in set-env!

micha15:10:23

computer went crazy there

isaac15:10:27

@alandipert: @micha oh! thanks

alandipert15:10:46

@martinklepsch: did you try (alter-var-root #'boot.tmpdir/commit! ... ) ?

martinklepsch15:10:59

The main change is really that copy-with-lastmod does not always delete dest-file but instead checks lastModified if the source file is older than dest-file … now that I’m reading what I’m writing it seems really confusing 😧

micha15:10:23

@martinklepsch: the fileset assumes that the files themselves (in the blob storage dir) are immutable (i.e. they will be written to only once and then read-only after that)

micha15:10:11

but the files in blob storage are accessed by a hash of their contents

micha15:10:27

so it's possible that a file is updated without changing its contents

micha15:10:44

so we need to keep track of accurate modtimes in the fileset

micha15:10:59

to ensure that time moves forward as we mostly expect

martinklepsch15:10:59

probably a better approach than changing copy-with-lastmod is to generate patch actions in commit!

micha15:10:25

the hardlinks that copy-with-lastmod creates get their permissions and modtimes from the blob file they link to

micha15:10:59

yeah you could use fileset-diff function as a starting point i think

martinklepsch15:10:21

yeah that makes sense (blob stuff)

micha15:10:22

although that function doesn't make a complete patch, it simplifies a little by ignoring deleted files i think

micha15:10:33

ah, boot.tmpdir/diff* and friends looks like it does what you want

martinklepsch15:10:31

or boot.file/sync! — at the point of commit I don’t have two filesets to compare

micha15:10:13

i wonder if we should add that

micha15:10:45

somehow keep the fileset that corresponds to the filesystem

micha15:10:50

so we can compare

micha15:10:27

i think that could work

micha15:10:46

we'd add a boot.tmpdir/init! function that boot would call with the initial fileset

micha15:10:06

that would store the fileset in an atom

micha15:10:28

then commit! would compare against that and reset! it after it completes the commit

martinklepsch15:10:25

couldn’t we also just construct that tree from a list of tmpdirs?

micha15:10:30

also do we all feel pretty good about eliminating :target-path in set-env! in favor of a target task that explicitly dumps to a directory of your choice?

micha15:10:48

martinklepsch: we could construct the tree, but we'd have to do IO for that

juhoteperi15:10:51

I think that would be useful

micha15:10:56

reading dirs and files and stuff

martinklepsch15:10:56

see boot.file/tree-for

juhoteperi15:10:27

Would make it easier to dump uberjar directly to Ansible dir or such

micha15:10:53

yeah and also it would simplify the way we deal with things like node_modules

micha15:10:30

or giant vendor resources with tens of thousands of files that the build doesn't care about

juhoteperi15:10:33

And I guess having separate target-dirs for dev and prod (uberjar) would be really good practive (or well, no target dir for dev at all)

micha15:10:58

yeah this is a nice feature for me

micha15:10:07

i rarely need a target dir

micha15:10:53

i could imagine using it in the middle of a pipeline too

juhoteperi15:10:55

Noticed one day that if I change files while doing deployment dev task will overwrite target dir and remove the jar file and thus the deployment will fail 😄

micha15:10:02

like maybe for mking multiple builds in the same pipeline

martinklepsch15:10:48

tree-for doesn’t seem to recurse into nested dirs? if that’s somehow possible it would only be a couple of ls and getting lastmod

micha15:10:42

it's recursive, but the recursion is hidden in file-seq

micha15:10:12

file-seq unwinds the tree into a lazy sequence

micha15:10:21

flattens i mean

martinklepsch15:10:44

Ah I see. In any case I think might be worth trying, reading is much faster than writing

micha15:10:45

postorder

micha15:10:15

yeah it would be interesting to see if it makes a big difference

micha15:10:53

i could work on a branch that compares before and after filesets in commit! and see if that helps, too

martinklepsch15:10:30

as opposed to getting a tree from the fs?

micha15:10:40

i kind of like the fileset diff approach from an aesthetic point of view

martinklepsch16:10:04

It introduces more state 😛

martinklepsch16:10:47

I was also thinking that it would be cool to isolate the fileset stuff into a separate library with proper public api etc.

martinklepsch16:10:24

I’m heading off for a bit. Later)1

micha16:10:41

@martinklepsch: it doesn't introduce any more state, the fileset is all about state

micha16:10:51

it just memoizes really

micha16:10:55

or caches

tcrawley16:10:11

@martinklepsch: @micha: more context on the memory leak - I may not have seen it before because a (binding [... call in the right place could prevent it from occurring. That would trigger a Var.popThreadBindings() call, which would remove the thread local

tcrawley16:10:31

and that may have been happening in Immutant/vert.x

tcrawley16:10:47

I do have a simpler solution than using a separate thread in shimdandy that I'm testing now

tcrawley16:10:28

basically, it uses reflection to clear Var.dvals after an invoke if it is the top of the call stack for the current thread

tcrawley16:10:41

that seems to fix the issue with the test app

micha16:10:16

i've never seen this dvals stuff before

tcrawley16:10:00

it's an impl detail - it's how clojure keeps track of the stack of dynamic bindings

micha16:10:38

@juhoteperi: i was just talking with @alandipert and i think we have a way to do the target task without a breaking change: the task would add metadata to the fileset before passing it down the pipeline, and boot could look for that and if it's present not write to the :target-path

juhoteperi16:10:58

@micha: Sounds reasonable

micha16:10:25

i really don't want to make breaking changes if we can avoid them

micha16:10:49

we should try to be as stable as clojure itself

juhoteperi16:10:58

We should probably check if the GPG change can also made without breaking current options

micha16:10:00

or boot is a liability

micha16:10:46

yeah i think the gpg thing could accomodate new things when certain env vars are present

micha16:10:04

and fall back to current operation when they're not there

juhoteperi16:10:11

I'm not sure if we need any env vars for that

micha16:10:06

i think the gpg stuff is appropriate for env vars (in addition to task options)

juhoteperi16:10:11

GPG binary only needs to be used when user has configured repository which should read credentials from encrypted file -> new feature anyway and when signing file and password hasn't been provided

micha16:10:28

because you'd be likely to use it in an unattended deploy situation and things like that

juhoteperi16:10:40

When signing a file and password is provided we could fall back to current stuff (or pass the password to gpg binary)

micha16:10:56

yeah that's legit

juhoteperi16:10:48

What about separate deploy-repositories?

micha16:10:02

we could do that also

micha16:10:15

i think that might be appropriate as an option to the task actually

juhoteperi16:10:17

If user doesn't set them it could fallback to normal repositories, so it would work like now?

micha16:10:25

yeah totally

micha16:10:43

i want to avoid putting things in the env that aren't env related

juhoteperi16:10:49

Hmm, yeah task options is probably better place for them than env

micha16:10:10

yeah i really don't want boot to become a build tool 😆

juhoteperi16:10:18

I can find some references about using Gpg-agent together with Bouncy Castle lib. But looks like requires implementing some custom clases.

pandeiro17:10:01

@micha I'm using :target-path -- the build.boot has about a dozen tasks -- I would need to add (target ...) to each?

micha17:10:56

@pandeiro: no it would be optional

pandeiro17:10:59

oh cool :+1:

martinklepsch18:10:28

@pandeiro also you'd add target at the end of the pipeline not for each task

pandeiro18:10:03

@martinklepsch: ah yeah ok, so it would have to be in three or four comp tasks only

pandeiro18:10:29

I still prefer one place for it -- I'm just using it to rename target to something python guys are more familiar with simple_smile