Fork me on GitHub
#boot
<
2016-11-03
>
flyboarder00:11:34

If you use target task more than once

michael.heuberger00:11:13

hello guys - i’m looking for an example here how to include an external source map to build.boot. it’s for a new cljsjs package that comes with a link to an external one at bottom of the JS file. but somehow boot is not picking it up.

michael.heuberger00:11:01

that new cljsjs package has the minified JS file and the source map file both in the production folder (there is no development folder)

flyboarder00:11:28

What cljsjs package?

michael.heuberger00:11:30

a new one i am about to submit a PR for - it wraps the zxcvbn password strength checker

michael.heuberger00:11:44

because it already comes minified, i guess it is a special case

flyboarder00:11:35

Is the only version available minified?

michael.heuberger00:11:33

i think all zxcvbn releases are minified

ag00:11:21

guys trying to run (aot), seeing this error:

java.util.concurrent.ExecutionException: java.lang.ExceptionInInitializerError
  java.lang.ExceptionInInitializerError: 
clojure.lang.Compiler$CompilerException: java.lang.ExceptionInInitializerError, compiling:(boot/util.clj:1:1)
  java.lang.ExceptionInInitializerError: 
            clojure.lang.ArityException: Wrong number of args (0) passed to: digest/fn--37
what could this be?

michael.heuberger00:11:45

any suggestions @flyboarder re external source maps in boot?

micha00:11:28

did you check to make sure the map file is available at the URL specified in the minified source file?

michael.heuberger00:11:50

yes, and it is not available

michael.heuberger00:11:03

also checked in the temporary target folder of boot

michael.heuberger00:11:09

the external source map isn’t there either

micha00:11:23

is it in the jar file?

micha00:11:03

how are you extracting the source map from the jar? you will need to add it to the fileset, no?

micha00:11:27

unless you're making an uberjar and serving from the class path?

michael.heuberger00:11:43

well, i think book should do it for me? just listed it as a dependency [cljsjs/zxcvbn “4.4.1-1”]

micha00:11:08

that will put the files on the class path

michael.heuberger00:11:56

i am uploading that jar in a private, mirrored repository for our office like all the other cljsjs packages

michael.heuberger00:11:25

sorry, dont understand. what does the class path have to do with it?

micha00:11:10

when you add a dependency, the files in the jar are added to the classpath

micha00:11:18

that's pretty much it

michael.heuberger00:11:58

how can i check this in command line? that the source map is in the class path?

micha00:11:59

when you use something like :require [cljsjs.zxcvbn] and you have the deps.cljs file on the classpth, the cljs compiler will extract the js file from the class path for you

michael.heuberger00:11:03

in deps.cljs i see following (source map entry is missing there!):

{:foreign-libs [{:file "cljsjs/zxcvbn/production/zxcvbn.inc.js", :provides ["cljsjs.zxcvbn"]}], :externs ["cljsjs/zxcvbn/common/zxcvbn.ext.js"]}

chooie00:11:06

Can anyone help me figure out why map isn’t executing in this boot task:

(deftask my-test
  []
  (fn [next-handler]
    (fn [fileset]
      (map (fn [num]
             (println num))
        (range 1 5)))))

micha00:11:21

yes, i don't see anything about source maps in the deps.cljs documentation

michael.heuberger00:11:36

ok, so then it’s a cljsjs packaging problem

richiardiandrea00:11:01

@chooie it is executed, but because map is lazy you will never see the println, in order to see it you need (doall (map ....))

micha00:11:02

i mean i don't think source maps are supported in the deps.cljs file

micha00:11:13

so you will need to extract the source maps yourself

micha00:11:22

in a boot task

michael.heuberger00:11:35

hmmm … is there an example for inspiration?

micha00:11:30

you could find all the deps.cljs files on the classpath

micha00:11:10

and then look at them to get the :files

micha00:11:29

like in your package there is the file :file "cljsjs/zxcvbn/production/zxcvbn.inc.js"

micha00:11:59

then for each of those you could look for a file of the same name but with .map appended to the name

micha00:11:25

and if it exists you could copy it from the classpath to a temp dir and add the temp dir to the fileset

micha00:11:05

the best solution would be to have the deps.cljs file format include a :source-maps key or something

micha00:11:14

but that is the cljs compiler

micha00:11:28

there might be an open JIRA ticket for that already

michael.heuberger00:11:01

thanks heaps … thinking ...

micha00:11:04

a patch for that would be the best solution

micha00:11:28

then we could update the cljsjs task that generates the deps.cljs file to include the source maps

michael.heuberger00:11:55

yup but that might take a bit too long as i need to get this working pretty soon

michael.heuberger01:11:11

will try your above suggestion with a custom boot task

micha01:11:19

you can simply add the source map to your project :resource-paths

micha01:11:56

then you don't need to make a task that will just do that

michael.heuberger01:11:10

hmmm, but what path to pick there?

michael.heuberger01:11:29

when the source map lands in a temporary directory

micha01:11:32

it just needs to be in the same directory as the js file

michael.heuberger01:11:55

that’s the problem. which directory?

micha01:11:09

you can build the application and see where the js file is

micha01:11:28

suppose it's in the target dir at target/foo/bar/thingy.js

michael.heuberger01:11:35

with js file you mean the source map?

micha01:11:45

well the js file that references the source map

micha01:11:50

the minified js file

micha01:11:01

you can look in the target dir and find it

micha01:11:12

what is the path?

michael.heuberger01:11:18

but problem is that the source map is not in that target folder

micha01:11:27

yes, but what is the path to the js file

michael.heuberger01:11:01

./target/js/app.out/zxcvbn.inc.js

micha01:11:19

ok cool, and what does your :resource-paths look like?

michael.heuberger01:11:19

there is no ./target/js/app.out/zxcvbn.inc.js.map there

michael.heuberger01:11:29

:resource-paths #{”resources"}

micha01:11:29

yes, we'll get there in a moment

micha01:11:43

so now extract the source map file from the jar

micha01:11:12

and move it to ./resources/js/app.out/zxcvbn.inc.js.map

micha01:11:21

then rebuild your project

micha01:11:34

it should be in the target dir then and in the correct place

michael.heuberger01:11:36

i see what you mean

michael.heuberger01:11:44

but that wouldn’t be acceptable here

michael.heuberger01:11:59

needs to happen automatically with newer versions

micha01:11:29

seems like it would be easier to just upgrade it manually until you can really fix the issue

michael.heuberger01:11:15

what about a custom boot task you suggested above?

micha01:11:51

but it would need to put the file in the correct place

micha01:11:19

the correct place is really decided by the cljs compiler configuration and boot-cljs options

micha01:11:49

it's a shame that the source map can't be embedded in a comment in the js file itself

micha01:11:59

that would be the obvious way to do it

micha01:11:34

what about getting the source map from github?

micha01:11:49

like change the URL in the minified file to point to a github URL?

micha01:11:05

i think that might be the best solution

micha01:11:14

don't serve the source map locally

michael.heuberger01:11:16

ugh, you mean to turn //# sourceMappingURL=zxcvbn.inc.js.map into an absolute url pointing to the one on github?

micha01:11:40

seems ideal really

michael.heuberger01:11:07

back in half hour + many thanks!

richiardiandrea01:11:36

@micha can I compose watch tasks? Example one that watches all files and one that I can trigger manually ?

micha01:11:06

interesting, i don't know :)

richiardiandrea01:11:11

boot watch watch --manual ...

micha01:11:38

i think it might make an explosion

micha01:11:55

of waiting loops

micha01:11:16

interesting though

micha01:11:54

what if it always waits for both of you don't have --manual

micha01:11:15

i mean waits for either

richiardiandrea01:11:29

because I was thinking of solving the boot-figwheel issue with the manual trigger

micha01:11:11

yeah like what if without --manual it would trigger when files change or when you manually trigger it

richiardiandrea01:11:14

adding a hook in figwheel to call boot after it finishes compiling

micha01:11:22

and with --manual it just doesn't trigger on file changes

micha01:11:46

seems like something we could do without any breaking changes

richiardiandrea01:11:20

this can be extended to having the possibility to pass/bind your own atoms

micha01:11:25

man i need to get 2.7.0 shipped 🙂

micha01:11:48

i think it's easy enough to extend with your own atoms since we do expose a function you can call to trigger the update

micha01:11:58

so you can call that from a watch on your own atom too

ghadi01:11:59

Y'all tracking jdk 9?

micha01:11:25

hi @ghadi i haven't been

micha01:11:33

are there issues?

ghadi01:11:45

Usually yes

richiardiandrea01:11:46

@micha what is the name? I can only see the boot.core/new-build-at atom 😉

micha01:11:56

are they going to include the jigsaw module stuff?

micha01:11:12

hm i will check

richiardiandrea01:11:30

kk no rush, doing it too 😉

ghadi01:11:55

Oh yeah, JDK 9 is all modularized

richiardiandrea01:11:00

Cool thanks! I'll experiment

romain03:11:57

Anyone is experiencing slow issue ? For example a basic boot -hcan take 10 seconds to execute

micha03:11:17

@romain can you try boot -BPh

micha03:11:37

is that faster?

micha03:11:12

what's your setup? os, Java, etc

romain03:11:27

macOS Sierra, java version "1.8.0_102", boot 2.6.0

romain03:11:45

Hm... Boot clojure 1.7, I don't know if it changes something

micha03:11:35

seems OK to me

romain03:11:12

boot cljs takes time to start, then the compilation time seems normal

micha03:11:20

is Sierra the newest osx?

micha03:11:36

interesting

micha03:11:18

do you have any environment variables set?

micha03:11:02

like BOOT_JVM_OPTS

romain03:11:28

Nop, I only did brew install boot-cljthen boot -h`

micha03:11:55

very weird

romain03:11:15

For information, boot -hinstalled 2.5.x, and boot -h again detected the 2.6.x version and installed it

micha03:11:00

that's OK

micha03:11:33

it had the 2.5.x version hardcoded in

micha03:11:47

then updated itself via maven

micha03:11:10

are you using nfs or some kind of weird filesystem or anything like that?

micha03:11:43

no docker involved?

romain03:11:48

nop nop nop

micha03:11:11

very strange

micha03:11:47

with -BP it shouldn't be using any build.boot or anything

micha03:11:11

so you should see times of around 1.5sec

romain03:11:52

with macOS there are some new changes like new filesystem

romain03:11:14

but I don't think this is the problem

micha03:11:50

is boot using cpu while it's running?

romain03:11:19

Does boot -BPh a check on internet ?

micha03:11:46

it shouldn't

romain03:11:58

OK, because I have a really bad connection

romain03:11:52

My activity monitor showed me a "java" processus at 137%

micha03:11:26

interesting, so it's computing something

romain03:11:45

And it increases, second time 174% and now 224%

micha03:11:45

OK just as a test can you make an empty directory and run boot -BPh in there?

romain03:11:24

same behavior

micha03:11:39

the only thing i can think of is boot somehow needing to manage a huge number of files or something

micha03:11:49

or network

micha03:11:11

but i don't see how either could be the case here

romain03:11:01

I may try to update my java version, it's not normal to see a java process at 192% cpu usage

micha03:11:28

it's using 2 cores

micha03:11:41

seems reasonable

micha03:11:56

maven stuff maybe

micha04:11:55

what about trying to move your ~/.m2 dir to another name temporarily

micha04:11:16

maybe something is crazy in maven land

romain04:11:24

some seconds before retrieving dependencies. I let it finish and I retry boot -BPh

micha04:11:48

hmm very strange

micha04:11:14

what could it be doing before retrieving dependencies

micha04:11:02

could there be something going on with bash on your system?

micha04:11:31

boot.sh is a bash script

romain04:11:14

But I tried boot few months ago (with previous version of mac etc) and never had these delays

micha04:11:39

i mean boot.sh itself is a bash script

micha04:11:23

so if something is going on with bash on your system it could affect boot

romain04:11:41

(moving .m2 did nothing)

romain04:11:34

Let me try to update my java version

romain04:11:07

@micha well, it did nothing too 😞

romain05:11:37

I think I got a solution, I modified /etc/hosts to match this http://stackoverflow.com/questions/39636792/sbt-test-extremely-slow-on-macos-sierra and now it seems better. It's not blazing fast, but boot -BPh takes now 3 seconds instead of 10

romain05:11:00

Don't know if it's placebo but...

romain05:11:01

Tested 5 times with and without the fix, it really changes. From 8 seconds to 3 seconds

dominicm12:11:34

I've seen this a few times now with the latest mac update.

naomarik12:11:56

does latest mac update give devs anything useful? it looked like just siri and emojis to me

edannenberg12:11:55

last snapshot version seems to be from april

micha13:11:44

@edannenberg new release will be ready soon

naomarik13:11:20

i’m struggling a bit trying to figure out how to include a random github project as a dependency and hack on it. it seems checkout is the way to go but not sure how to use it. i’ve followed the lein guide and symlinked a project under ./checkouts, inlcluded that in the checkouts-path in boot, then the checkouts dep vector and it doesn’t find it

naomarik13:11:05

i’m guessing you have to ensure it installs in the local maven repo correctly?

naomarik13:11:14

that doesn’t seem to give me hackability though. what’s the workflow for this? just add as a source path?

edannenberg13:11:12

@naomarik did you see https://github.com/boot-clj/boot/issues/154 ? haven't used it myself though yet.

naomarik13:11:03

dat stuff is deprecated

naomarik13:11:42

(get-env) shows the checkouts stuff properly but i don’t know what the benefit is if i have to keep modifying the checkout dependency and running lein install

mynomoto13:11:21

@naomarik there is a --checkouts option now. -c --checkouts SYM:VER Add checkout dependency (eg. -c foo/bar:1.2.3)

naomarik13:11:47

{adzerk/boot-reload
 {:dir
  #object[java.io.File 0xfda1c1f "/Users/naomarik/.boot/cache/tmp/Users/naomarik/Sites/boot-test/5y2/-nu80k9"],
  :jar
  #object[java.io.File 0x615e77a7 "/Users/naomarik/.m2/repository/adzerk/boot-reload/0.4.14/boot-reload-0.4.14.jar"],
  :dep nil}}
(get-checkouts) gives me this and I can refer to it in the code properly, but I’m wondering why there’s reference to this dir since it doesn’t reload my changes. I’m running this in a simple repl with no tasks defined yet

mynomoto13:11:31

@naomarik the flow hasn't changed for using checkouts from boot, the task checkout was replaced by the option --checkouts. But the flow is you do auto builds that will generate jars on the checkout project and those will be picked on the project you are running with --checkouts.

naomarik13:11:02

ah so i should combine this with a watch task to get reloadability

mynomoto13:11:02

Generate jars installed on your local ~/.m2 I mean

micha13:11:42

yeah you want to run a separate process to build the checkout jar

micha13:11:16

the main process will rebuild when the jar is updated

micha13:11:16

i mean a separate process to build the checkout jar and install it in your local .m2 maven cache

christianromney13:11:35

what's the easiest way to have my build process replace one file with another? i'm striking out with both sift and cp in a custom task. Filesets are making me feel boneheaded 😄

micha13:11:38

the main process gets the jar from maven as usual

micha13:11:26

the easiest way is to create a file in a temp dir with the same path and the ad temp dir to the fileset

christianromney13:11:37

more importantly, i'd love to know how to think about them properly, and what i'm doing wrong.

micha13:11:40

it will replace the previous

micha13:11:28

the cp function is kind of specific and mostly only useful internally

pseud13:11:37

https://github.com/onetom/hoplon-layouts/blob/master/build.boot — anyone feel like explaining exactly what “compile” means in terms of maven scopes and clojure(?script) projects ? I know “provided” means the dependency isn’t transitive, for example.

micha13:11:15

@christianromney I'm on my phone but I'll look when i get to a computer

christianromney13:11:41

no hurry— 🙂 but if/when you get around to it, i'd be super grateful

christianromney13:11:56

really want to further my understanding

alandipert13:11:00

tl;dr it's the default scope, is transitive

micha13:11:04

s sure my pleasure

jfntn14:11:34

What’s the boot equivalent of lein install?

onetom14:11:52

@pseud as alan said, that's the default. i might have it there because i was lazy to delete after converting it from a maven pom.xml which had the scopes explicitly included (probably because of intellij's maven plugin)

pseud14:11:16

@onetom Ah. Btw thanks for maintaining the boot lein generate plugin 🙂

onetom14:11:46

of course! i need it too! 🙂

onetom14:11:41

it's a boot task, not a plugin, like the lein folks call it 😉

pseud14:11:39

Oh, right! Musn’t talk lein-speak 😛

romain14:11:42

@naomarik Well, it's the first mac update I find uninteresting... And it seems to have some thorns for advanced users

naomarik15:11:24

okay so with boot-reload installing itself every save with its default boot-dev i’m still not able to find new changes in the other project unless i restart the repl

micha16:11:29

@naomarik reloading things on the classpath doesn't update objects already constructed in memory

naomarik16:11:57

have to use clojure.tools.namespace.repl/refresh right?

naomarik16:11:42

this was throwing me errors about httpkit not being included

naomarik16:11:04

using latest stable clojure.tools instead of alpha removes the error, but still don’t get changes ;/

naomarik16:11:35

(require
 '[adzerk.boot-reload :as reload] :reload)
re-evaling this worked for me, couldn’t get clojure tools thing to work properly

naomarik16:11:39

boot-reload had a dependency on httpkit even when I added it to my own project calling refresh did nothing, even after setting the dirs shown here: https://github.com/boot-clj/boot/wiki/Repl-reloading

naomarik16:11:45

maybe clojure.tools.repl/refresh doesn’t work properly for checkouts?

micha16:11:14

(require 'some.name-space :reload) is i think what you want

micha16:11:21

or :reload-all

naomarik16:11:49

ya that’s what worked

naomarik16:11:55

not sure why the other thing didn't

christianromney17:11:37

@micha based on your last input, i figured it out thanks! turns out i was just forgetting to call (io/make-parents) on a new tempfile with a nested path

devn17:11:01

👏 on the episodes chaps

devn17:11:33

pretty rad to hear craig singing hoplon and boot's praises

christianromney17:11:32

now we need them to schedule their follow-up cognicast where we get to hear more about hoplon 🙂

devn17:11:36

@christianromney it just came out 🙂

christianromney17:11:15

@devn i heard the cast with them that was mostly boot and they agreed to come back to do hoplon

christianromney17:11:42

my podcast feed is behind!

devn17:11:06

it only came out like 1 hour ago

devn17:11:29

aside: what do you use to manage your podcast feed?

micha17:11:01

hi @devn! thanks!

ag18:11:19

hey guys is there a boot thing that makes “anonymous" deftask? something I can use within (comp). with-pre-wrap fileset does that but I’m not doing anything fileset realted

ag18:11:17

with-pass-thru?

ag18:11:24

is that what I need to use?

martinklepsch18:11:44

but you can also use with-pre-wrap and just return the unmodified fileset

martinklepsch18:11:54

with-pass-thru is just sugar

ag18:11:18

cool, thanks!

zane18:11:56

👍 for Overcast.

hiredman19:11:01

I am running boot uber jar at the command line, at the uberjar is being generated, but the fileset doesn't appear to be synced(I guess?) because the uberjar doesn't end up in target/ but I do see it in the cache in ~/.boot/

hiredman19:11:06

is that the expected behavior?

richiardiandrea19:11:31

@hiredman you probably need to append the target task to materialize the file if I understand correctly

richiardiandrea19:11:51

so boot uber jar target

zane19:11:53

Should I be thinking of target as "turn the files in the pod into real files"?

richiardiandrea19:11:30

@zane it should be thought as the task for materializing files from the fileset to the file system

zane19:11:58

Better way of saying the same thing. Okay, great.

richiardiandrea19:11:07

it is explicit in boot cause you might want to actually materialize the files somewhere else, on s3 for instance

zane19:11:13

Totally.

zane19:11:20

I like that it's explicit, I was just confused by the name.

richiardiandrea19:11:39

yeah agree, I guess it is a legacy thing 😉

hiredman19:11:22

ok, next, why does the uberjar not contain files from a :source-paths, which I am setting using set-env!

ag19:11:39

hey guys… I’m trying to use adzerk.boot-test. I have the dependency, I have it in require - still test task not showing up… wtf?

hiredman19:11:43

and it does contain all the classfiles and what not from dependencies that are set with the same call to set-env!

hiredman19:11:21

for some reason I thought :source-paths was a thing in the boot env, but once I put my src directory in :resource-paths now everything is showing up in the jar

mynomoto19:11:50

@hiredman :source-paths are input only, :resource-paths are input and output. I will get a link with a better explanation.

mynomoto19:11:41

@ag can I see your build.boot? Did you refer the task?

ag20:11:06

sorry, I was out for lunch… I tried removing ~/.m2 but still boot test -h doesn’t work: https://gist.github.com/agzam/e4819cd1255c7d99600e3cc461b26db6

ag20:11:52

oh… damn… I found it… for whatever crazy reason I had (ns-unmap 'boot.user ‘test)

mynomoto20:11:32

@ag the important part is that you found it 😉

lxsameer20:11:55

where is the default location for the boot-cljs compiled result ?

mynomoto20:11:16

@lxsameer it will put it on the fileset by default. If you want the fileset to be written to disk you need to use the target task. By default the target task will put the fileset on the target/ directory.

naomarik21:11:22

there any boot tasks to find updated deps for the project?

lxsameer21:11:35

@mynomoto dude, is there any document about these tasks and boot internals ?

micha21:11:19

@lxsameer you can see all the built-in tasks with boot -h

micha21:11:29

for any individual task you can do boot task -h

micha21:11:36

or in the repl you can do (doc task)

micha21:11:02

also there is documentation in the boot gihub repo, in the wiki and there are also api docs

lxsameer21:11:19

i read those, but to be honest, there are many things missing from these docs

micha21:11:52

it's unfortunate that there is no search functionality for github wikis

micha21:11:05

but there is a chrome extension that adds search

mynomoto21:11:13

Oh, I missed that @micha pointed the wiki...

naomarik21:11:56

any idea what this is? java[21479] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)

naomarik21:11:21

doesn’t seem to have any adverse effects other than spamming my log

micha21:11:33

do you have a lot of files that are being watched?

micha21:11:47

that's an issue iwth the native FSEvents APIs

naomarik21:11:03

not a lot of files but prob some intersection between figwheel and watch task

micha21:11:22

there are some workarounds supposedly, but the watcher code for OSX is not something i really want to mess with

micha21:11:32

native FSEvents are not supported for OSX in java

micha21:11:39

so it would need to fall back to polling

micha21:11:45

which uses a lot of CPU

naomarik21:11:50

ya thats gross

micha21:11:51

and adds latency

micha21:11:09

so we use this old java implementation, barbarywatchservice

micha21:11:24

and it's pretty convoluted and hard to understand

naomarik21:11:32

it seems to work perfectly

naomarik21:11:40

i think this is just 2 things watching the FS at same time

micha21:11:52

yeah it's just if you try to add too many watches it will choke

micha21:11:59

that's an OS limitation

naomarik21:11:13

which OS you using?

micha21:11:23

i use linux myself

naomarik21:11:49

i just installed a milllion things on my windows ubuntu bash thing and shocked it all worked

micha21:11:05

oh, you're using boot in windows?

naomarik21:11:09

no using osx

naomarik21:11:15

but will play around with the windows thing later

micha21:11:28

interesting yeah

micha21:11:47

we might be able to at least suppress the logging

naomarik21:11:57

that would be cool

naomarik21:11:08

i don’t see it normally though

micha21:11:14

i'm hesitant to disturb that code because it seems to be working reliably 🙂

naomarik21:11:19

only when i’m screwing around with figwheel + watch

micha21:11:26

we had to do many hacks to get it to be reliable

naomarik21:11:25

the watcher only watches resource paths right?

naomarik21:11:50

no nvm that doesn’t make sense

naomarik21:11:56

thinking of boot-reload

naomarik21:11:19

@mynomoto thanks, that’s super useful

naomarik21:11:40

you can imagine what i was doing beforehand

anmonteiro21:11:50

probably a long shot: does anyone have experience using Boot + Appveyor?

hlship22:11:04

So I got the boot repl working, and it was easy. Cursive can just connect to it as if it was a Leiningen REPL and off we go.

hlship22:11:18

But I’m having a different problem.

hlship22:11:30

I can build my Docker images the first time. No problem.

hlship22:11:52

Then I’m experimenting with updates and deletions of Maven artifacts, so in one window I: rm -rf ~/.m2/repository/org/apache/zookeeper/, but when I re-run my task:

(boot (wmsc-query-service-image))
java.nio.file.NoSuchFileException: /Users/hlship/.boot/cache/tmp/Users/hlship/workspaces/walmart/gleam/1ktp/6v03yy/zookeeper-3.4.6.jar -> /Users/hlship/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar

hlship22:11:03

I’ve already switched my code from using boot.file/hard-link boot.file/copy-with-lastmod, but that didn’t help.

hlship22:11:05

(with-pre-wrap fs
      (empty-dir! work-dir)
      (doseq [jar jars]
        (file/copy-with-lastmod jar (io/file work-dir (get-name jar))))
      (-> (cond-> fs
            (seq stable) (add-files stable target)
            (seq snapshot) (add-files snapshot target))
          (add-resource work-dir)
          commit!))

hlship22:11:41

So thar jars is a seq of File, for artifacts in my Maven repository.

micha22:11:02

what is add-files?

hlship22:11:37

That is part of my Docker stuff, it updates a datastructure attached to the fileset to add docker ADD instructions for those files.

hlship22:11:45

But it doesn’t otherwise manipulate the fileset.

hlship22:11:07

It works as expected if I execute boot from the command line twice, with the rm in between.

hlship22:11:19

But in a REPL, the rm in the middle causes the above exception.

micha22:11:44

i'm not sure how a link is being created from your .m2 into a temp dir

hlship22:11:50

So something is carrying over from the first execution of (boot (wmsc-query-service-image)) to the second.

micha22:11:26

and hard links will be fine if you delete the referent

hlship22:11:27

You know what, let me delete my ~/.boot/cache directory, maybe there’s something from a prior execution causing trouble.

micha22:11:43

only soft links need the original to exist

micha22:11:56

a hard link will just become "unlinked" if you delete the original

hlship22:11:17

I deleted ~/.boot/cache/tmp, but still persists:

(boot (wmsc-query-service-image))
java.nio.file.NoSuchFileException: /Users/hlship/.boot/cache/tmp/Users/hlship/workspaces/walmart/gleam/1l49/8uws5d/zookeeper-3.4.6.jar -> /Users/hlship/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar

java.nio.file.NoSuchFileException: /Users/hlship/.boot/cache/tmp/Users/hlship/workspaces/walmart/gleam/1l49/8uws5d/zookeeper-3.4.6.jar -> /Users/hlship/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar
         file: "/Users/hlship/.boot/cache/tmp/Users/hlship/workspaces/walmart/gleam/1l49/8uws5d/zookeeper-3.4.6.jar"
    otherFile: "/Users/hlship/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar"
sun.nio.fs.UnixException.translateToIOException           UnixException.java:   86
  sun.nio.fs.UnixException.rethrowAsIOException           UnixException.java:  102
   sun.nio.fs.UnixFileSystemProvider.createLink  UnixFileSystemProvider.java:  476
                                            ...                                   
                            boot.file/hard-link                     file.clj:  147
            gleam.tasks.docker/eval490/fn/fn/fn                   docker.clj:  105
            gleam.tasks.docker/eval328/fn/fn/fn                   docker.clj:   39
                            boot.core/run-tasks                     core.clj:  938
                              boot.core/boot/fn                     core.clj:  948
            clojure.core/binding-conveyor-fn/fn                     core.clj: 2019
                                            ...                                   

hlship22:11:21

(deftask artifact
  "Add one or more Maven artifacts to the image.

  Artifacts are retrieved from the Maven repository and added to the fileset, and
  a ADD instruction is added to the Dockerdata.

  All artifacts, along with transitive dependencies, are imported.
  "
  [d dependency ID:VER [[sym str]] "Vector of checkout dependencies"
   t target DIR str "Target directory for artifact(s)"]
  (assert target "--target is required")
  (assert (str/ends-with? target "/")
          "--target DIR must end with a slash")
  (let [env (get-env)
        coords (map pod/canonical-coord dependency)
        jars (map (comp io/file :jar) (pod/resolve-dependencies (assoc env :dependencies coords)))
        jar-names (mapv get-name jars)
        {:keys [stable snapshot]} (->> jar-names
                                       sort
                                       (group-by categorize))
        work-dir (tmp-dir!)]
    (with-pre-wrap fs
      (empty-dir! work-dir)
      (doseq [jar jars]
        ;; There's a fair assumption that the file names will never collide. Maven artifacts
        ;; include a version number that should do the trick.
        (file/hard-link jar (io/file work-dir (get-name jar))))
      ;; There's a method to my madness here. Doing all the stable dependencies first
      ;; will create an intermediate image, and that might be cached across executions.
      ;; The snapshots are much less likely to be stable, and should come second.
      (-> (cond-> fs
            (seq stable) (add-files stable target)
            (seq snapshot) (add-files snapshot target))
          (add-resource work-dir)
          commit!))))

hlship22:11:30

I think there’s an issue with pod/resolve-dependencies. It must be caching some prior results, from when the Maven artifact existed, but doesn’t redo the work of downloading the file now that it is missing.

micha22:11:46

it caches very aggressively

hlship22:11:48

It there a better way to resolve dependencies than my hack?

micha22:11:59

i don't think you want to do that

micha22:11:43

like i think it's best to leave maven cache alone, it's very complicated and it is shared among many systems that use maven

hlship22:11:56

Normally this might not be a problem, since we don’t expect the files in the local Maven repository to change during the boot execution … but it’s a minor problem while I’m testing some things. And we may be in a case where one task does builds that moves artifacts into the local Maven repo.

micha22:11:28

sure but you don't need to delete things from the cache

micha22:11:32

that's where the problems arise

micha22:11:08

like boot checkouts work fine, and you use them by updating jars in local m2

hlship22:11:25

this is true, for this particular test I can just run boot twice.

hlship22:11:37

I was just trying to do more in the REPL, now that I have it working.

micha22:11:48

there is a lot of memoization going on in the boot dependency resolution code

micha23:11:00

otherwise everything would be a lot slower

micha23:11:39

boot assumes that the maven cache is immutable, at least as far as the jar file names

micha23:11:15

you can use pomegranate directly if you need lower-level access to maven

micha23:11:33

or look in the boot.aether namespace

micha23:11:43

that's the boot wrapper for pomegranate

micha23:11:53

runs in a pod

hlship23:11:07

There’s a lot of features of Boot I haven’t even scratched yet.

hlship23:11:33

Any idea on a new release, one that (oh, just for example) has my io.aviso/pretty patch?

micha23:11:45

oh sweet, is there a PR?

hlship23:11:53

yes, last friday

micha23:11:53

i want to make a release this week

micha23:11:16

lots of stuff in master waiting to escape