Fork me on GitHub
#boot
<
2015-08-23
>
cigitia06:08:10

Just to confirm my [lack of] understanding: When all of the tasks in a pipeline have been run, all files that are in the final fileset's temporary folders (even when not committed (?)) and that have output roles—those files are automatically copied into the environment's actual target folder. Is this correct?

cigitia06:08:19

And if that’s correct…is the actual target folder subject to being wiped clean any time before a pipeline’s final output-role temporary files are copied into it? Testing this gave me equivocal results.

xifi09:08:35

I'm trying to set up a project where I use boot-http with a ring handler. I set resource-paths to #{"resources"}, ring's middleware resources to "public" (so that the ring server will serve static resources from resources/public), create a resources/public/js/main.cljs.edn so that my main.js will end up there. When I start the server if I go to localhost:3000/js/main.js it is indeed there which is cool, but the index page can't find goog's base.js. The created main.js points to main.out/goog/base.js while it should be pointing to js/main.out/goog/base.js. Can this somehow be set up to work correctly?

juhoteperi12:08:54

@xifi: :compiler-options {:asset-path “js/main.out”

juhoteperi12:08:55

Asset-path works automatically if you use “js” prefix instead of public for middleware.

micha14:08:46

hi @cigitia

micha14:08:09

so the pipeline, the contents of the pipe, is the immutable FileSet object

micha14:08:19

that's what passes between tasks

micha14:08:57

the target directory is the place where the pipeline ends, where the final fileset is emitted to the real filesystem

micha14:08:39

while the pipeline exists the target directory is in an undefined state

micha14:08:00

it must be because you could have more than one boot process running, sharing that directory

micha14:08:17

the contents of the target directory after the pipeline has completed should reflect the last fileset commit! operation

micha14:08:55

@juhoteperi: i retested with latest cljs and the problem still exists

micha14:08:29

i need to investigate further but it seems like namespaces are bleeding from one build into another

juhoteperi14:08:41

main namespaces?

micha14:08:56

no, other namespaces

micha14:08:09

well the main namespaces are shims

juhoteperi14:08:11

Cljs namespaces or could it be problem with macro reloading (clj/cljc)?

micha14:08:33

in my case cljs.test is involved

micha14:08:45

so it might be macros

micha14:08:58

or something cljs compiler is special casing for tests

juhoteperi14:08:07

Is there any exception or is it really slow (or doesn’t finish)?

micha14:08:16

no, it compiles normally

micha14:08:33

but the result is not the same

micha14:08:46

i mean the same as with :none

micha14:08:24

actually this is kind of a dumb case

micha14:08:35

i need to remove boot reload and so on

micha14:08:39

those tasks aren't meant to be used with :advanced

micha14:08:59

tests should work though

micha14:08:07

that's kind of important

cigitia17:08:25

@micha: Thanks for the reply! I think I understand better now. But, when you say that "the contents of the target directory after the pipeline has completed should reflect the last fileset commit! operation", this only includes temporary files with the output role, right? Otherwise, from what I understand, all the source files would get written into the target folder too, but I could be wrong.

micha17:08:52

no, you're correct

micha17:08:10

only files with output role are written to target dir

micha17:08:36

and those reflect the last FileSet committed during the pipeline

micha17:08:40

@cigitia: ^^

cigitia17:08:42

@micha: Thanks, I see! And, lastly, are any already-existing files in the target directory deleted before the final fileset gets written into the target?

micha17:08:22

@cigitia: the contents are merged, that is to say similar to the behavior of rsync --delete ...

micha17:08:34

it doesn't delete things unnecessarily

micha17:08:47

but it does remove things that are not in the output files of the fileset

micha17:08:15

also there is one more weird thing about the target dir

micha17:08:33

nothing is written there unless a task has created a new file in the fileset with the output role

micha17:08:59

tasks that just read things from the fileset and use their own private temp dirs therefore don't affect the target dir

micha17:08:18

this way you can run boot repl for instance without blowing away your target dir

cigitia18:08:09

I see, thanks, this is very helpful! I think I might add this information to the wiki for others too…unless it's already there and I just missed it, heh.

micha18:08:23

that would be 👍 awesome

micha18:08:00

it's the best when new things are added to the wiki simple_smile

micha18:08:28

@juhoteperi: this is a real puzzle

xifi18:08:12

@juhoteperi: thanks for the tip. What I wanted to achieve was having 1 folder on the classpath that has static subfolders for css, js, images and such. The problem is that boot-cljs probably expects main.js and main.out to be emitted somewhere on the classpath so it always injects it as 'main.js'. I thought there might be a way to instrument it to consider a different root folder

micha18:08:18

see tests fail

micha19:08:25

xifi: you can make a .cljs.edn file in the :source-paths, like js/main.cljs.edn

micha19:08:35

i believe that will do what you want

micha19:08:54

@xifi: ^^

micha19:08:59

that will produce js/main.js and js/main.out/

xifi19:08:33

@micha: I put main.cljs.edn under resources/public/js/ but main.js will try to link in main.out/goog/base.js instead of js/main.out/goog/base.js

micha19:08:13

wait, so you want like this:

xifi19:08:20

you know, the small basic snippet you get in main.js,

if (typeof ... = undefined) {} {goog.require ...})
or something like that

micha19:08:59

resources/
  public/
    js/
      main.js
      main.out/
        goog/
          base.js

xifi19:08:33

correct. I have :resource-paths set to #{"resources"} and the ring middleware for resources points to "public", so the web server can look into resources/public

micha19:08:02

oh right, yeah sorry

xifi19:08:28

now my edn is under the js folder and the generated main.js tries to load main.out/goog/base.js

micha19:08:31

arbitrary routing isn't really supported like that

xifi19:08:52

I see. what is the common setup?

micha19:08:58

i'd recomend just not doing that simple_smile

micha19:08:04

serve from the root

micha19:08:20

if you're worried about secret codez getting out you can easily filter in middleware

micha19:08:46

normally you'll deploy static assets to a CDN anyway

xifi19:08:54

by root you mean?

micha19:08:09

so the issue of mixing cat GIFs and your database passwords is not so relevant

micha19:08:35

(wrap-resource {:root ""}) or whatver

xifi19:08:10

ok, thanks. I had this setup from lein so I was trying to match it

micha19:08:52

people usually do that because they have supersecret/namespace.clj on the classpath mixed in with their static resources

micha19:08:18

so the easiest thing to do is to put the static resources in some arbitrary directory, like public/

micha19:08:30

and then authorize by path

micha19:08:46

but this has a lot of repercussions with the tooling stack

micha19:08:03

since now tools have no way to map resources to things in the classpath anymore

xifi19:08:24

I'm plunging into this like a crazy person simple_smile No js/html/css or web server experience whatsoever. Fun fun fun!

micha19:08:36

haha awesome

xifi19:08:55

wait 6 months and I'll apply to adzerk

micha19:08:11

we'll be hiring, i'm sure!

xifi19:08:32

I'm not sure I'd be able to work for an ad company though

micha19:08:43

haha that's what i thought too

micha19:08:47

we're the good guys though

micha19:08:56

stack overflow, reddit, imgur

micha19:08:07

those guys couldn't stay in business without ads

xifi19:08:26

doesn't sound that bad. OK fine I'll take the job!

micha19:08:28

we basically protect those guys from being scavenged by the big ad networks

micha19:08:37

because we work just for them

micha19:08:58

we don't put them in competition with each other etc

micha19:08:48

@cigitia: that is an incredible writeup!

micha19:08:59

💥 🚀

cigitia19:08:12

Heh, thanks! And thanks for the help!

micha19:08:38

no thank you, this is super helpful

micha19:08:47

i mean "no, thank you..."

micha19:08:57

haha a comma makes all the difference

micha19:08:56

let's eat(,) grampa

cigitia19:08:48

@micha: Is there any way to make output files persist between task pipelines?

cigitia19:08:55

My situation is that I have some source files that I want to compile using task #1, and I also want to use those output files in task #2. But task #1 takes hours to do, and the source files that I’m compiling with change relatively rarely. I don’t want to run task #1 every time I want to run task #2, but task #2 cannot see task #1’s output unless it runs right after task #1 in the same pipeline; it can’t see any files in the target directory.

micha19:08:24

@cigitia: do you have a reliable way to invalidate your cache?

micha19:08:46

like can the program know for sure when the cache is valid?

cigitia19:08:11

The source files are going to change at most once or twice a year—they’re Unicode Character Database data.

cigitia19:08:22

Task #1 converts the UCD data into millions of JSON files.

micha19:08:33

one thing you can do in this case is make a jar file of task #1 and then depend on that

micha19:08:08

would that be a workable solution?

micha19:08:20

you can deploy your jar to private repo on s3 if you like

micha19:08:29

you can also version it then, which is nice

cigitia19:08:34

I’d have preferred keeping everything in one project, sigh, but you’re probably right: this might be the best way, yes.

micha19:08:51

you can cache the jar locally if you really want to

micha19:08:01

boot does have a persistent cache thing built-in

micha19:08:17

but then your fellow devs wouldn't be able to just get the jar from maven

micha19:08:26

they'd have to run task #1 on their machines

cigitia19:08:26

…Well, actually, I was wrong: the cache files are going to change way more often than that, because I’m going to be continuously adding new data to them.

cigitia19:08:33

This is a personal project, which does make things more flexible.

micha19:08:42

ok yeah so there is a thing you could use

micha19:08:52

one sec i forgot how it goes 😄

cigitia19:08:55

Task #1 actually is slow only on certain big files.

cigitia19:08:21

I’ve made it such that when I run Task #1 on consecutive source files (some of which are big and slow, some small and fast), it merges the cache data together.

cigitia19:08:52

I’d like to be able to run Task #1 on small files often, but Task #1 on big files rarely, with the automatic merging of preexisting output files.

cigitia19:08:06

Oh, this looks nice.

micha19:08:13

we're actually working on a thing at the moment

micha19:08:22

a thing to allow you to cache partial filesets

micha19:08:46

so they can be merged into the fileset really fast, with everything precomputed and hashed and whatnot

micha19:08:08

it's for the uber task

micha19:08:12

which is slow now

cigitia19:08:21

This cache-dir! might just be what I’m looking for. What would be the difference between that new thing and cache-dir!?

micha19:08:59

the new thing uses dirs created via cache-dir! for storage of cooked fileset info

cigitia20:08:05

Oh, ah, so then is using cache-dir! slower than import-cache!, since it still has to compute a lot of stuff at the beginning of a pipeline?

micha20:08:18

well import-cache! doesn't exist yet

micha20:08:23

this is half done

micha20:08:42

but the idea is you create a persistent cache dir with cache-dir!

micha20:08:00

then you can use export-cache! to populate it with the contents of any directory you specify

micha20:08:45

this will populate your cache dir with files and a manifest in a special format that will allow boot to merge the contents with any fileset very efficiently

micha20:08:16

because the work of importing the things into the fileset is done when you do export-cache! and then amortized over all the time you merge

micha20:08:57

then once you have exported the dir to a cache dir you can use (add-resource fileset the-cache-dir :cached? true)

micha20:08:08

and it will be zipped right in there

micha20:08:11

it's like 50% complete, we still need to have it use the existing merge strategies (in case of name collisions boot allows the user to specify different merge strategies and bind to a dynamic var so add-resource et al will use those merge fns)

cigitia20:08:56

I see—that sounds amazing and exactly what I need. So cache-dir! is part of this; it’s not ready yet, either.

micha20:08:06

cache-dir is ready now

micha20:08:09

you can use that, totally

micha20:08:17

it will probably do exactly what you want

micha20:08:28

although millions of files will be taxing for any system

micha20:08:53

perhaps the best solution is a "bothism"

micha20:08:15

use cache-dir! as a place to cache JARs containing the millions of files

micha20:08:25

and then add the JAR to the classpath dynamically

cigitia20:08:59

The problem with using JARs is that the output files are actually going to change often: just only a few hundred output files at a time, but still frequently, at least while I’m developing.

cigitia20:08:27

That is, I’m going to be running Task #1 often on small source files, and I wish to be able to use the data in already-existing output files, whenever they already exist, and merge that old data into the new data.

micha20:08:39

well cache-dir! will at least give you a place to cache things that will persist across multiple builds

micha20:08:04

that directory won't be deleted when boot exits

micha20:08:25

and you can retrieve it with the cache key in subsequent builds

cigitia20:08:47

Right, that’s what I’ll probably do. It’ll probably take a while to load the millions of files in the cache, though, right?

micha20:08:59

yeah probably 😕

micha20:08:10

i mean only the first time

micha20:08:19

they'll just be in there after that

micha20:08:08

it's basically just giving you a directory you can use as if you made a directory outside of boot

cigitia20:08:41

It won’t copy the cache into temporary files every time I run a task pipeline, right?

micha20:08:57

not unless you add the cache dir to the fileset yourself

micha20:08:10

but you probably don't need to do that

cigitia20:08:59

I might eventually if another project needs to use those compiled files, but that’s further in the future anyway.

micha20:08:17

yeah and then maybe jars are the way to go

micha20:08:30

you'll need versioning and stuff probably

micha20:08:26

maybe sqlite is an option, too?

micha20:08:38

that might speed things up, depending on what you're doing of course

cigitia20:08:15

Maybe—in any case, thank you very much for the help, gosh.

cigitia20:08:05

If I’m understanding correctly, will import-cache! be for when you want to use cached files as a proper part of the fileset, e.g., when you want some of those cached files to be outputted into the target directory?

micha20:08:29

well the current use case is for the uber task

micha20:08:03

as it is right now the uber task is used in the pipline with the jar, war, zip tasks, etc

micha20:08:09

you'd do like boot uber jar

micha20:08:36

the uber task itself just looks at all the JARs on the classpath

micha20:08:56

and for each JAR it extracts all the entries and writes them to files which are added to the fileset

micha20:08:25

so this is nice from the composition perspective, because now we don't need separate uberjar, uberwar, uberzip etc tasks

micha20:08:00

but it's not so nice for performance, because it's writing a ton of files, and adding entries directly to jars, wars, zips, etc would be faster

micha20:08:16

so this is where import-cache! comes in

micha20:08:36

we extract all the files from a JAR into the fileset only once, when we first see this JAR

cigitia20:08:03

Ah, now I see.

cigitia20:08:44

Thank you so much for all your help!

micha20:08:51

sure, my pleasure!

xifi20:08:45

so my task runs fine, boot-reload up through the websocket, then when I save a file I get this error

Load failed: Jsloader error (code #0): Error while loading script /public/main.out/stair_cljs/core.js?zx=s10xz74tt4l5
reload.js (line 222)

xifi20:08:25

if I just refresh the page everything seems to be working fine

xifi20:08:43

but that kind of misses the point of boot-reload 😛

xifi20:08:46

if I click on reload.js I don't get to the script file. Not sure if firebug in FF is the best developmetn option

micha20:08:07

@juhoteperi: i figured it out

xifi20:08:10

was a missing paren >_<

micha20:08:31

in a cljs file?

xifi20:08:56

these errors can get pretty cryptic. Like now,

TypeError: src is null

Line 933
I have no idea what is that

xifi20:08:04

@micha: yes

xifi20:08:58

the error is pointing deep inside GCL's events.js

xifi20:08:01

well I'm glad this tickles you 😛

micha21:08:13

you have to appreciate the little things simple_smile

xifi21:08:08

yeah, this stack being used is very little as well simple_smile

xifi21:08:50

html/css/js -> cljs -> gcl, with millions of libraries

xifi21:08:09

not to mention the underlying complexity of the browsers and standards and whatnot

micha21:08:20

yeah i haven't been overly impressed with GCL libraries

xifi21:08:00

what are your plans for some solid documentation on the hoplon stack?

micha21:08:03

i've tried but i ended up going with jquery, and i'm not missing anything

xifi21:08:15

like castra github page has 0 info >_<

micha21:08:36

yeah it's not well documented at the moment

xifi21:08:56

and the examples you and @alandipert posted have no server-side code

micha21:08:17

we have more people interested in working on docs and stuff now though

micha21:08:25

so hopefully soon things will improve

xifi21:08:04

even an API doc would go a long way right now simple_smile (if you have docstrings)

micha21:08:30

hm yeah we could do that i think

micha21:08:43

although generating docs for cljs has been problematic in the past

juhoteperi21:08:53

@micha: PR looks good. I'll merge that and do a release?

micha21:08:45

yep let's doit

juhoteperi21:08:38

Ouch, looks like automatically set asset-path is broken

micha21:08:01

by my patch?

juhoteperi21:08:15

Probably not

micha21:08:31

what's broken?

juhoteperi21:08:51

js/main.cljs.edn sets asset-path as "main.out"

juhoteperi21:08:57

when it should be "js/main.out"

micha21:08:26

the tests include asset paths, like demo/index.cljs.edn

juhoteperi21:08:31

Ah, could be caused by your patch

juhoteperi21:08:06

I'm trying to decipher how the relative path is set

juhoteperi21:08:49

The test is now using simple optimization so asset-path doesn't matter

juhoteperi21:08:17

We should have separate tests for none and simple

juhoteperi21:08:34

the tests happen to work as index.html is also inside demo path

micha21:08:15

you're saying html file not in same dir as js file?

juhoteperi21:08:28

It is in the same dir

juhoteperi21:08:34

But it's broken when it's not

micha21:08:48

yeah we don't really support that do we?

micha21:08:59

i mean how can cljs task know anything about html files?

juhoteperi21:08:07

It doesn't need to

juhoteperi21:08:19

It should work and it has worked

micha21:08:27

it worked with your shim

micha21:08:42

because it was able to detect things at runtime

micha21:08:54

with the current shim i don't see how it can

micha21:08:01

because everything is decided at compile time

juhoteperi21:08:33

It should currently set the asset-path relative to root of fileset

juhoteperi21:08:36

but it doesn't do that

micha21:08:44

no, that will break things

micha21:08:38

we should bring the original shim back

juhoteperi21:08:53

But I should talk with David about improving Cljs version

micha21:08:53

and not use :main when we're not in nodejs land

micha21:08:06

our shim was way better

micha21:08:16

it just didn't work in node

juhoteperi21:08:50

Saapas used to work...

micha21:08:30

having a requirement that the js file and output dir have a specific relationship with the html file seems totally reasonable to me

micha21:08:49

i think it's counterproductive to not have a Method there

micha21:08:56

and a standard thing

juhoteperi21:08:03

Before having .cljs.edn inside js/ folder would set asset-path as js/out

juhoteperi21:08:17

e.g. it used the relative path inside fileset

micha21:08:39

that's not what i need though

micha21:08:55

i might have like index.html, products/index.html etc

micha21:08:14

that seems like the more sane use case

micha21:08:36

rather than cosmetic path nonsense

micha21:08:51

nobody cares where the js file is, users don't see that

micha21:08:01

the only thing that matters is where the html file is

micha21:08:22

at some point barbie needs to pick out a dress and go to the prom

micha21:08:39

it's the same as if the JVM had to support arbitrary cosmetic paths in the classpath

micha21:08:45

like i like all my clojure files in clj/

micha21:08:51

and all my java classes in java/

micha21:08:14

it's a million times better that they don't support that

micha21:08:25

or it would break everything

juhoteperi21:08:20

I'll have to think how to setup my projects

juhoteperi21:08:18

For now manually setting asset-path works

juhoteperi21:08:47

And probably that is the best solution, but we could make it possible to set asset-path without knowing about output dir name

juhoteperi21:08:12

Should be simple and allow all use cases

micha21:08:50

it's simple

micha21:08:03

they just need to provide a path to the html file in the .cljs.edn

micha21:08:10

then we can compute all things

juhoteperi21:08:15

or html files?

micha21:08:28

no, we only need to support one html file per js file

juhoteperi21:08:30

quite possible to load the same file from multiple paths

micha21:08:41

i don't think we should support that without optimizations

micha21:08:51

or without a better shim

juhoteperi21:08:08

If my blog used cljs I would need that

micha21:08:27

i don't see why

micha21:08:41

just compile with whitespace optimizations

juhoteperi21:08:45

Because each post is in it's own folder and there is only one js file

micha21:08:02

yeah so you can't use the shim then

micha21:08:14

without a better shim i think it's the sanest approach

juhoteperi21:08:16

I just need to set absolute path manually

micha21:08:32

how can you have one shim that works with different random html files?

juhoteperi21:08:48

"/js/main.out"

micha22:08:09

yeah that could be an option

juhoteperi22:08:27

There could be asset-path task option which wouldn't need to have output dir name appended

juhoteperi22:08:42

It would be really simple to document how to use that with all the use cases (and when it's not needed)

micha22:08:55

yeah the webroot option

juhoteperi22:08:39

Anyhow, 1.7.48-3 is now released

micha22:08:57

i think the solution for the webroot business is modules

micha22:08:03

not :asset-path

micha22:08:32

it's a caching issue, you shouldn't be managing that manually

micha22:08:50

lol i'm so skeptical of anything :asset-path related

micha22:08:04

trigger warning :asset-path