Fork me on GitHub
#boot
<
2017-07-26
>
djebbz09:07:52

Hi guys, we keep running in problem when zipping assets with boot. Basically our task is ` (comp (build-assets) (sift :include #{"paths/to/assets"}) (target :dir #{"path/to/target"}) (zip :file (str "assets-" version ".zip")) (sift :include #{#"^(.+?).zip$)"}) (target :dir #{"path/to/target"}))`

djebbz09:07:13

The problem we have is that the zip randomly doesn't include some of our assets : sometimes it's images, sometimes it's css. We're keen on adding a (wait :time 1000) between target and zip to make sure everything is written to the fileset before zipping it. Anybody faced similar problems ? I read the source of zip, very small, I didn't see anything weird but I'm no Boot/Java guru.

alandipert14:07:05

djebbz i haven't heard of that problem and am not sure how to fix. but i did see this recently: https://github.com/manenko/boot-zip

alandipert14:07:36

might be worth trying to see if it fixes. but if you have more info about your setup and wouldn't mind filing a bug, that would be v. helpful

djebbz14:07:21

For the time being we added the (wait) call and it seems to do the job. Need more usage to see if it happens again. At this moment I'll investigate more.

djebbz14:07:45

It may be related to the setup, it always works locally but often doesn't on Jenkins...

djebbz14:07:07

Thanks for the pointer to boot-zip though

sekao15:07:58

to add a local jar to the classpath, do you just add the jar's path to :source-paths? i'm getting a java.nio.file.DirectoryNotEmptyException

sekao16:07:20

@martinklepsch thanks, am i correct that this will only work within a pod you create via make-pod?

richiardiandrea16:07:38

@sekao no, the function should work in general for the main (and children) pod

richiardiandrea16:07:43

> and will attempt to add that path to the right classloader (with the search rooted at the current thread's context classloader).

sekao16:07:32

got it, thanks!

sekao17:07:20

the weird thing is, after running add-classpath i can require a namespace from the jar at the top level of my build.boot. but if i try to build an uberjar of my project, which requires said namespace, it says it can't find it on the classpath

sekao17:07:38

if i remove the aot task, there is no error, so i suppose my project needs the namespace too early...

sekao17:07:30

i also tried the with-cp task to no avail

alandipert17:07:33

sekao currently the fileset entangles files and the classpath. as such the notion of environment differs between add-classpath and fileset

alandipert17:07:13

so add-classpath adds it to the build.boot context, but that's not where classpaths that filesets have starts from

alandipert17:07:51

luke vanderhart disentangled recently to nice effect, https://github.com/arachne-framework/arachne-fileset

richiardiandrea17:07:13

I liked the project above as well 😄

sekao17:07:53

looks interesting, so perhaps creating a boot task that wraps fs/add would do it?

sekao17:07:27

hmm, looks like that function doesn't work on boot's filesets

lwhorton17:07:17

is it common for the boot-cljs to run twice in a row on file change? I wonder if I have it configured incorrectly (bad ordering of task composition) or my source-paths are incorrect or something (change #1 invokes immediate change #2)

lwhorton17:07:03

100% of the time I get this on a .cljs file change:

Compiling ClojureScript...
• main.js
Compile sources, elapsed time: 1456.544556 msecs
Compile sources, elapsed time: 150.422101 msecs
Writing target dir(s)...
Elapsed time: 3.385 sec

juhoteperi17:07:46

@lwhorton That message is not from boot-cljs, but from ClojureScript compiler, there are two compile stages, that is why there are two "Compile sources" messages

lwhorton17:07:10

Ah. Bummer. It’s getting pretty slow now that I have a sizeable project and I was hoping I was just configuring things improperly. That being said, I also have another task that logs something like starting task foo... ended foo, elapsed time XXXms. that message shows up twice too, which is maybe not expected?

juhoteperi17:07:49

The second Cljs compile stage compiles :preloads namespaces and their dependencies

juhoteperi17:07:07

I can't comment about other tasks without more information

lwhorton17:07:51

Hmm, yea I’ll have to double check over the tasks

lwhorton17:07:10

but thanks for letting me know about the 2 stage compilation, i would have chased that for a while

juhoteperi17:07:08

One thing that can help with Cljs is to ensure that the project is constructed so that the often modified files are not dependecies of too many other namespaces, i.e. when a file is modified, as few as possible other files need to be recompiled

martinklepsch18:07:33

blogpost right there 🙂

juhoteperi17:07:11

But if the compile times are around 1-2 seconds, I doubt that is the problem. I have some projects where changing a common.* namespace triggers the recompile of everything and takes ~15 seconds.

juhoteperi17:07:01

Seeing that you are using target task, it might be helpful to check if some unnecessary files are being written, like node_modules which can contain thousands of files. You can use sift to remove the unnecessary files.

lwhorton18:07:09

Unrelated note, is there an api to get a particular dir in the tmp fileset? The fileset api seems to be all pointing to ways to get files, but suppose I have 2 tasks that want to share a particular dir between them?

lwhorton18:07:53

(this is a really weird task, but essentially I’m installing npm deps into a particular dir, such that I can install a whole bunch of things once and have steps 2, 3, 4, etc. utilize those installed programs inside of the node_modules dir)

minikomi10:07:36

I use:

(defn by-dir
  "Gets files matching directory names"
  [dirs files & negate?]
  ((boot/file-filter #(fn [f]
                        (= (.getParent f) %))) dirs files negate?))

minikomi10:07:44

(by-dir [dir1 dir2 …] fileset)

minikomi10:07:35

Useful for eg. collecting images which are grouped by directory etc.

lwhorton12:07:18

that’s a good idea

alandipert18:07:17

lwhorton filesets don't contain dirs per se, since the objects in them are addressed by content. works kind of like aws S3, if you are familiar. however you can scan the paths looking for some prefix

alandipert18:07:03

other than picking some prefix meaningful to your tasks, you could add metadata to the fileset object

alandipert18:07:31

or, add a single "manifest" file to the fileset that contains paths to other files that are meaningful for your task

lwhorton18:07:39

I see, so I could do a :

(binding [u/*sh-dir* my-special-prefix-path]
            (u/dosh "npm" "install" "--depth" "-1"))
and commit those files into the fileset, Then later lookup those files
(binding [u/*sh-dir* (.getPath my-special-prefix)]
  (u/dosh "my-special-path/node_modules/.bin/my-exec"))
or something along those lines*

lwhorton20:07:39

@juhoteperi +5 points for you and Gryffindor. filtering out files in post-task cleanup cut off a lot of recompilation time.

martinklepsch21:07:04

@lwhorton also consider that you might not need target at all (I don’t in most projects)

lwhorton23:07:25

really? that’s something i’ve never considered.. ill look into that too