Just to confirm my [lack of] understanding: When all of the tasks in a pipeline have been run, all files that are in the final fileset's temporary folders (even when not committed (?)) and that have output roles—those files are automatically copied into the environment's actual target folder. Is this correct?

And if that’s correct…is the actual target folder subject to being wiped clean any time before a pipeline’s final output-role temporary files are copied into it? Testing this gave me equivocal results.

I'm trying to set up a project where I use boot-http with a ring handler. I set resource-paths to #{"resources"}, ring's middleware resources to "public" (so that the ring server will serve static resources from resources/public), create a resources/public/js/main.cljs.edn so that my main.js will end up there. When I start the server if I go to localhost:3000/js/main.js it is indeed there which is cool, but the index page can't find goog's base.js. The created main.js points to main.out/goog/base.js while it should be pointing to js/main.out/goog/base.js. Can this somehow be set up to work correctly?

@xifi: :compiler-options {:asset-path “js/main.out”

Asset-path works automatically if you use “js” prefix instead of public for middleware.

hi @cigitia

so the pipeline, the contents of the pipe, is the immutable FileSet object

that's what passes between tasks

the target directory is the place where the pipeline ends, where the final fileset is emitted to the real filesystem

while the pipeline exists the target directory is in an undefined state

it must be because you could have more than one boot process running, sharing that directory

the contents of the target directory after the pipeline has completed should reflect the last fileset commit! operation

@juhoteperi: i retested with latest cljs and the problem still exists

i need to investigate further but it seems like namespaces are bleeding from one build into another

main namespaces?

no, other namespaces

well the main namespaces are shims

Cljs namespaces or could it be problem with macro reloading (clj/cljc)?

in my case cljs.test is involved

so it might be macros

or something cljs compiler is special casing for tests

Is there any exception or is it really slow (or doesn’t finish)?

no, it compiles normally

but the result is not the same

i mean the same as with :none

actually this is kind of a dumb case

i need to remove boot reload and so on

those tasks aren't meant to be used with :advanced

tests should work though

that's kind of important

@micha: Thanks for the reply! I think I understand better now. But, when you say that "the contents of the target directory after the pipeline has completed should reflect the last fileset commit! operation", this only includes temporary files with the output role, right? Otherwise, from what I understand, all the source files would get written into the target folder too, but I could be wrong.

no, you're correct

only files with output role are written to target dir

and those reflect the last FileSet committed during the pipeline

@cigitia: ^^

@micha: Thanks, I see! And, lastly, are any already-existing files in the target directory deleted before the final fileset gets written into the target?

@cigitia: the contents are merged, that is to say similar to the behavior of rsync --delete ...

it doesn't delete things unnecessarily

but it does remove things that are not in the output files of the fileset

also there is one more weird thing about the target dir

nothing is written there unless a task has created a new file in the fileset with the output role

tasks that just read things from the fileset and use their own private temp dirs therefore don't affect the target dir

this way you can run boot repl for instance without blowing away your target dir

I see, thanks, this is very helpful! I think I might add this information to the wiki for others too…unless it's already there and I just missed it, heh.

that would be :thumbsup: awesome

it's the best when new things are added to the wiki

@juhoteperi: this is a real puzzle

@juhoteperi: thanks for the tip. What I wanted to achieve was having 1 folder on the classpath that has static subfolders for css, js, images and such. The problem is that boot-cljs probably expects main.js and main.out to be emitted somewhere on the classpath so it always injects it as 'main.js'. I thought there might be a way to instrument it to consider a different root folder

@juhoteperi: https://github.com/adzerk-oss/boot-cljs/pull/92/files?w=1

see tests fail

xifi: you can make a .cljs.edn file in the :source-paths, like js/main.cljs.edn

i believe that will do what you want

@xifi: ^^

that will produce js/main.js and js/main.out/

@micha: I put main.cljs.edn under resources/public/js/ but main.js will try to link in main.out/goog/base.js instead of js/main.out/goog/base.js

wait, so you want like this:

you know, the small basic snippet you get in main.js,

if (typeof ... = undefined) {} {goog.require ...})

resources/
  public/
    js/
      main.js
      main.out/
        goog/
          base.js

right?

correct. I have :resource-paths set to #{"resources"} and the ring middleware for resources points to "public", so the web server can look into resources/public

oh right, yeah sorry

now my edn is under the js folder and the generated main.js tries to load main.out/goog/base.js

arbitrary routing isn't really supported like that

I see. what is the common setup?

i'd recomend just not doing that

serve from the root

if you're worried about secret codez getting out you can easily filter in middleware

normally you'll deploy static assets to a CDN anyway

by root you mean?

target?

so the issue of mixing cat GIFs and your database passwords is not so relevant

yeah

(wrap-resource {:root ""}) or whatver

ok, thanks. I had this setup from lein so I was trying to match it

people usually do that because they have supersecret/namespace.clj on the classpath mixed in with their static resources

so the easiest thing to do is to put the static resources in some arbitrary directory, like public/

and then authorize by path

but this has a lot of repercussions with the tooling stack

since now tools have no way to map resources to things in the classpath anymore

I'm plunging into this like a crazy person No js/html/css or web server experience whatsoever. Fun fun fun!

haha awesome

wait 6 months and I'll apply to adzerk

we'll be hiring, i'm sure!

I'm not sure I'd be able to work for an ad company though

haha that's what i thought too

we're the good guys though

stack overflow, reddit, imgur

those guys couldn't stay in business without ads

doesn't sound that bad. OK fine I'll take the job!

we basically protect those guys from being scavenged by the big ad networks

because we work just for them

we don't put them in competition with each other etc

@micha: https://github.com/boot-clj/boot/wiki/Target-Directory

@cigitia: that is an incredible writeup!

💥 🚀

Heh, thanks! And thanks for the help!

no thank you, this is super helpful

i mean "no, thank you..."

haha a comma makes all the difference

Haha

let's eat(,) grampa

@micha: Is there any way to make output files persist between task pipelines?

My situation is that I have some source files that I want to compile using task #1, and I also want to use those output files in task #2. But task #1 takes hours to do, and the source files that I’m compiling with change relatively rarely. I don’t want to run task #1 every time I want to run task #2, but task #2 cannot see task #1’s output unless it runs right after task #1 in the same pipeline; it can’t see any files in the target directory.

@cigitia: do you have a reliable way to invalidate your cache?

like can the program know for sure when the cache is valid?

The source files are going to change at most once or twice a year—they’re Unicode Character Database data.

Task #1 converts the UCD data into millions of JSON files.

one thing you can do in this case is make a jar file of task #1 and then depend on that

would that be a workable solution?

you can deploy your jar to private repo on s3 if you like

you can also version it then, which is nice

I’d have preferred keeping everything in one project, sigh, but you’re probably right: this might be the best way, yes.

you can cache the jar locally if you really want to

boot does have a persistent cache thing built-in

but then your fellow devs wouldn't be able to just get the jar from maven

they'd have to run task #1 on their machines

…Well, actually, I was wrong: the cache files are going to change way more often than that, because I’m going to be continuously adding new data to them.

This is a personal project, which does make things more flexible.

ok yeah so there is a thing you could use

one sec i forgot how it goes 😄

Task #1 actually is slow only on certain big files.

I’ve made it such that when I run Task #1 on consecutive source files (some of which are big and slow, some small and fast), it merges the cache data together.

https://github.com/boot-clj/boot/blob/master/boot/core/src/boot/core.clj#L281-L296

I’d like to be able to run Task #1 on small files often, but Task #1 on big files rarely, with the automatic merging of preexisting output files.

Oh, this looks nice.

we're actually working on a thing at the moment

a thing to allow you to cache partial filesets

so they can be merged into the fileset really fast, with everything precomputed and hashed and whatnot

it's for the uber task

which is slow now

This cache-dir! might just be what I’m looking for. What would be the difference between that new thing and cache-dir!?

https://github.com/boot-clj/boot/blob/b04ed7bcd2dbef49c24e8c283114cd601fa480ae/boot/pod/src/boot/tmpdir.clj#L159-L189

the new thing uses dirs created via cache-dir! for storage of cooked fileset info

the money is here https://github.com/boot-clj/boot/blob/b04ed7bcd2dbef49c24e8c283114cd601fa480ae/boot/core/src/boot/core.clj#L413

Oh, ah, so then is using cache-dir! slower than import-cache!, since it still has to compute a lot of stuff at the beginning of a pipeline?

well import-cache! doesn't exist yet

this is half done

but the idea is you create a persistent cache dir with cache-dir!

then you can use export-cache! to populate it with the contents of any directory you specify

this will populate your cache dir with files and a manifest in a special format that will allow boot to merge the contents with any fileset very efficiently

because the work of importing the things into the fileset is done when you do export-cache! and then amortized over all the time you merge

then once you have exported the dir to a cache dir you can use (add-resource fileset the-cache-dir :cached? true)

and it will be zipped right in there

it's like 50% complete, we still need to have it use the existing merge strategies (in case of name collisions boot allows the user to specify different merge strategies and bind to a dynamic var so add-resource et al will use those merge fns)

I see—that sounds amazing and exactly what I need. So cache-dir! is part of this; it’s not ready yet, either.

cache-dir is ready now

you can use that, totally

it will probably do exactly what you want

although millions of files will be taxing for any system

perhaps the best solution is a "bothism"

use cache-dir! as a place to cache JARs containing the millions of files

and then add the JAR to the classpath dynamically

The problem with using JARs is that the output files are actually going to change often: just only a few hundred output files at a time, but still frequently, at least while I’m developing.

i see

That is, I’m going to be running Task #1 often on small source files, and I wish to be able to use the data in already-existing output files, whenever they already exist, and merge that old data into the new data.

well cache-dir! will at least give you a place to cache things that will persist across multiple builds

that directory won't be deleted when boot exits

and you can retrieve it with the cache key in subsequent builds

Right, that’s what I’ll probably do. It’ll probably take a while to load the millions of files in the cache, though, right?

yeah probably 😕

i mean only the first time

they'll just be in there after that

it's basically just giving you a directory you can use as if you made a directory outside of boot

It won’t copy the cache into temporary files every time I run a task pipeline, right?

nope

not unless you add the cache dir to the fileset yourself

but you probably don't need to do that

I might eventually if another project needs to use those compiled files, but that’s further in the future anyway.

yeah and then maybe jars are the way to go

you'll need versioning and stuff probably

maybe sqlite is an option, too?

that might speed things up, depending on what you're doing of course

Maybe—in any case, thank you very much for the help, gosh.

If I’m understanding correctly, will import-cache! be for when you want to use cached files as a proper part of the fileset, e.g., when you want some of those cached files to be outputted into the target directory?

well the current use case is for the uber task

as it is right now the uber task is used in the pipline with the jar, war, zip tasks, etc

you'd do like boot uber jar

the uber task itself just looks at all the JARs on the classpath

and for each JAR it extracts all the entries and writes them to files which are added to the fileset

so this is nice from the composition perspective, because now we don't need separate uberjar, uberwar, uberzip etc tasks

but it's not so nice for performance, because it's writing a ton of files, and adding entries directly to jars, wars, zips, etc would be faster

so this is where import-cache! comes in

we extract all the files from a JAR into the fileset only once, when we first see this JAR

Ah, now I see.

Thank you so much for all your help!

sure, my pleasure!

so my task runs fine, boot-reload up through the websocket, then when I save a file I get this error

Load failed: Jsloader error (code #0): Error while loading script /public/main.out/stair_cljs/core.js?zx=s10xz74tt4l5
reload.js (line 222)

if I just refresh the page everything seems to be working fine

but that kind of misses the point of boot-reload 😛

hrm

if I click on reload.js I don't get to the script file. Not sure if firebug in FF is the best developmetn option

@juhoteperi: i figured it out

was a missing paren >_<

in a cljs file?

these errors can get pretty cryptic. Like now,

TypeError: src is null

Line 933

@micha: yes

the error is pointing deep inside GCL's events.js

haha

well I'm glad this tickles you 😛

you have to appreciate the little things

yeah, this stack being used is very little as well

html/css/js -> cljs -> gcl, with millions of libraries

not to mention the underlying complexity of the browsers and standards and whatnot

yeah i haven't been overly impressed with GCL libraries

what are your plans for some solid documentation on the hoplon stack?

i've tried but i ended up going with jquery, and i'm not missing anything

like castra github page has 0 info >_<

yeah it's not well documented at the moment

and the examples you and @alandipert posted have no server-side code

we have more people interested in working on docs and stuff now though

so hopefully soon things will improve

even an API doc would go a long way right now (if you have docstrings)

hm yeah we could do that i think

although generating docs for cljs has been problematic in the past

@micha: PR looks good. I'll merge that and do a release?

yep let's doit

Ouch, looks like automatically set asset-path is broken

by my patch?

Probably not

what's broken?

js/main.cljs.edn sets asset-path as "main.out"

when it should be "js/main.out"

the tests include asset paths, like demo/index.cljs.edn

Ah, could be caused by your patch

or no

I'm trying to decipher how the relative path is set

The test is now using simple optimization so asset-path doesn't matter

ah

We should have separate tests for none and simple

the tests happen to work as index.html is also inside demo path

you're saying html file not in same dir as js file?

It is in the same dir

But it's broken when it's not

yeah we don't really support that do we?

i mean how can cljs task know anything about html files?

It doesn't need to

It should work and it has worked

it worked with your shim

because it was able to detect things at runtime

with the current shim i don't see how it can

because everything is decided at compile time

It should currently set the asset-path relative to root of fileset

but it doesn't do that

no, that will break things

Hmmh

we should bring the original shim back

No

But I should talk with David about improving Cljs version

and not use :main when we're not in nodejs land

our shim was way better

it just didn't work in node

Saapas used to work...

having a requirement that the js file and output dir have a specific relationship with the html file seems totally reasonable to me

i think it's counterproductive to not have a Method there

and a standard thing

Before having .cljs.edn inside js/ folder would set asset-path as js/out

e.g. it used the relative path inside fileset

that's not what i need though

i might have like index.html, products/index.html etc

that seems like the more sane use case

rather than cosmetic path nonsense

nobody cares where the js file is, users don't see that

the only thing that matters is where the html file is

at some point barbie needs to pick out a dress and go to the prom

it's the same as if the JVM had to support arbitrary cosmetic paths in the classpath

like i like all my clojure files in clj/

and all my java classes in java/

it's a million times better that they don't support that

or it would break everything

Okay, I see where this changed https://github.com/adzerk-oss/boot-cljs/commit/ffa254a69a93fd52a8a029cfb2fb4020d23c5d8a

I'll have to think how to setup my projects

For now manually setting asset-path works

And probably that is the best solution, but we could make it possible to set asset-path without knowing about output dir name

Should be simple and allow all use cases

it's simple

they just need to provide a path to the html file in the .cljs.edn

then we can compute all things

or html files?

no, we only need to support one html file per js file

quite possible to load the same file from multiple paths

i don't think we should support that without optimizations

or without a better shim

If my blog used cljs I would need that

i don't see why

just compile with whitespace optimizations

Because each post is in it's own folder and there is only one js file

yeah so you can't use the shim then

Sure I can

without a better shim i think it's the sanest approach

I just need to set absolute path manually

how can you have one shim that works with different random html files?

"/js/main.out"

ah right

yeah that could be an option

There could be asset-path task option which wouldn't need to have output dir name appended

It would be really simple to document how to use that with all the use cases (and when it's not needed)

yeah the webroot option

Yeah

Anyhow, 1.7.48-3 is now released

why not?

ah lol

lol typo

🍌

i think the solution for the webroot business is modules

not :asset-path

it's a caching issue, you shouldn't be managing that manually

lol i'm so skeptical of anything :asset-path related

trigger warning :asset-path