@onetom: are you seeing this performance with -SNAPSHOT or are you using -3?

it's possible the changes we made to support hot-reloading tests might have affected compilation times

im using -3 if it's just 1 big file, the difference is not so noticable

this weekend i'll look at perf again

see what can be done

the recompile time varies anyway, probably because of jvm warmup too

it seems like my application is taking seconds to recompile too

which seems like a lot since i haven't changed much

it should be < 1s

yup, same here. it was subsec b4

yeah

there must have been a regression in the cljs task or the compiler

jvm warmup will always be onerous

i think we want to concentrate on incremental build times

and minimize those as much as possible

maybe jvm options could help there, too

optimize more for long running jobs than for startup time

-server for example

we are on 1.8; there is no -server flag anymore, no?

oh, i wasn't aware of that change

if you have a lot of ram you might want to try running boot from a ramdisk

alan reports some improvement with that setup

like have boot write tempfiles to ramdisk

sorry, they only removed the -client

-server	  to select the "server" VM
                  The default VM is server,
                  because you are running on a server-class machine.

it's not a huge win because boot is already pretty efficient with temp files

but it might shave a second off the time

i was reading the tempfile handling code of boot a lot btw caused a lot of head scratching... like we couldn't really figure out how to create a TmpFile. the boot code only had 1 example of it. we tried (cp ...) too but it was complaining about the tmpdir not being recognized

it's definitely not a code base for beginners like ourselves

tests would have helped, alas 😕

sorry about that

with cp you really want to take an existing TmpFile and modify its path

but at the end my hack worked: https://github.com/exicon/homepage/blob/SPA/build.boot#L31-L49 have u seen it? do u have any immediate comment on it?

i've noticed it was creating symlinks for the index.html.out/ but not for the index.html.js

hmm that looks legit at first glance

line 41 looks suspect

you shouldn't have to deal with File objects

like given some TmpFile t with path "foo/bar.js" for example

if you want to copy that to "baz/foo/bar.js" say

you could do

i've seen later there are helper functions to lookup the various components of the TmpFile/TmpDir but not sure what's the purpose of those

(-> fileset (cp (tmp-file t) (assoc t :path "baz/foo/bar.js")) commit!)

hmmm.. let me try quickly

remember that the FileSet has a fixed set of underlying directories in which it stores the blob files and hard links

the blob files are the actual files, those are named by their md5 hash and are stored in the blob dir

the rest of the directories associated with the fileset are where the working tree is built

they're overlayed on each other to form the fileset tree on disk

there are a number of these directories because we need to have some files in the classpath and some not in the classpath

but once boot starts those directories are fixed for the lifetime of the JVM

you can't add new directories or remove directories

at least not without breaking filesets

ah, yeah, that's an other thing. i keep getting "can't delete target/something/something" errors since i use my copy task

im not sure i understand what do u mean by cant add/rm new dirs. doesn't the (tmp-dir! ...) fn make new ones?

no, that makes a cache directory

that's not in the fileset

when you do (add-resource fileset tmpdir) it doesn't add tmpdir to the fileset

it copies the files tmpdir contains into directories that are already in the fileset

aw... 😕 i thought for a while i understood things but now i don't understand again

the overall purpose of the fileset is to have a standard abstraction for the filessystem as it relates to the JVM

that abstraction only needs a certain number of directories

it needs a directory for files that will be on the classpath

it needs a directory for files that will not be on the classpath

and for boot's convenience it needs a directory for files what will be output files

and for files that won't be output files

and a few more things for organizing the files for boot's conveninece

but there is no need to ever add another directory

instead, when you do (add-resource ...) what happens is the files in the tmpdir are copied into blob storage

now we have the file contents decoupled from the path

that's super important

because now it's possible to refer to the file without referring to a specific path on the filesystem

this means we can make abstractions on top of it

which is what the FileSet is

the TmpFileSet is a mapping from object in blob storage to relative path in the tree

yeah, that part was roughly clear

when you call commit! on a fileset it takes those mappings and makes hard links on disk pointing to the blob files

with the appropriate relative paths

u mentioned FileSet and TmpFileSet. any difference?

first one was typo

lol

ok

i forget the name of the record type

any reason to call it Tmp?

it's all kind of related to managing temporary directories and whatnot, but not really

it's like git

git has an internal tree representation that you don't need to mess with

as i saw there was a new-fileset fn to create one but then no other one is created, but only that is evolving?

when you do git add mydir

what happens?

it doesn't add a directory to your .git

it goes into the object store

it adds the files to blob storage

and makes links internally to things in its tree graph

so it can generate the working set when it is commanded to

the blob storage is also something created by (tmp-dir! ...) no?

no

at least based on its path it looked like it

tmp-dir! just gives you a regular directory

the only reason boot has that function is because boot will clean those up for you

using the normal java temp directory machinery isn't as good because cleaning them up is a pain

like iwth boot if you make a tmp-dir! you can see things in the directory after boot exits

boot won't delete stale temp dirs until the next time it runs

so you can debug problems

if you use a regular java temp dir you can't do that

(it's all golden what u are saying. i will save it and try to rework it into some kind of documentation / article)

but (tmp-dir!) just returns a normal java.io.File object

and no other shenanigans other than that directory is in $BOOT_HOME, so boot can know when to delete it

(i just looked at the code, it was tmp-dir*; again a bit confusing as an "outsider"

ah yeah that's not part of the public API

that's part of what boot does when it initializes the fileset

that code is a little complicated maybe but there was a lot of reprtition and clerical errors so i did some refactoring that made it more confusing but less prone to clerical errors and typos

so during the lifetime of boot there is only 1 fileset is created newly?

the rest are all "mutations" of it?

well there is only one set of mutable global directories on the actual filesystem

the fileset objects themselves are immutable

the thing that's a singleton is the place where files are written when you call commit! on an immutable fileset

because the JVM's classpath is a singleton

and the disk is a singleton

a haskell person could tell you what kind of monad the fileset is

i'm sure it's some monad

so for example that was a bit unclear how to do a series of "changes" to the fileset

if u r doing that (cp ...) u showed above

how do u do a series of them?

use reduce?

you can chain them, yes or reduce

then you need to commit! of course

any operation that creates a new fileset will not affect the actual underlying filesystem

sure, that's clear too

so you can more efficiently separate the writing from the manipulation

batch the disk writes basically

and that's not really 100% true, because some things like add-resource etc. actually do add things to blob storage

but that's an implementation detail that shouldn't make a difference

because adding to blob storage is idempotent

and blob storage doesn't affect anything the user cares about

it's hidden in the fileset internals

though cp is doing an md5 calculation unconditionally

yeah

it needs to

but it was receiving a TmpFile which had an :id already 😕

oh true

yeah cp is messed up

i don't like that guy

that was another thing which made me scratch my head

yeah i have no idea what i was thinking there

is it used anywhere in boot?

btw, before i forget, pls push the boot-hoplon source!

hm

ok wow

i was looking into the source by unpacking the jar u published...

thx

dang sorry

and no, cp was not used anywhere; that's why i was a but clueless how to use it

we should fix it

any idea at 1st glance what can cause this:

java.io.IOException: Couldn't delete target/index.html.out/homepage/layout.cljs
                                           io.clj:  426
                                       ...
                     boot.file/delete-file                          file.clj:   52
                           boot.file/sync!                          file.clj:  190
                                       ...
                        clojure.core/apply                          core.clj:  634
                  boot.core/sync-target/fn                          core.clj:  687
                     boot.core/sync-target                          core.clj:  686
                    boot.core/run-tasks/fn                          core.clj:  695
tailrecursion.boot-hoplon/eval343/fn/fn/fn                   boot_hoplon.clj:   45

i have the feeling that my copy-index-html is doing something nasty which spits into the soup

i don't understand why it's failing to delete something in the target dir

that seems very strange to me

but i was not using cp... so im not sure if it was valid to just grab a TmpFile, change its path and shove it back to the (:tree fileset)

unless maybe a race condition

the target dir is unrelated to the flieset

yeah, it's a heisenbug, but happened quite often today when i was switching between prod and dev runs

so i don't think it is caused by something you did to that

hm

ok i ride to work

back in a bit

k, thx!

just a thought for the ride!

would it make sense to use javelin somehow for the fileset?

imean i saw a lot of code extracting the path from TmpFiles then putting them back to objects or the :tree

how would javelin help?

(defc= fileset {:tree (map (fn [f] {(:path f) f}) files)  :blob ...  :dirs ...})

interesting

it would probably clean up the fileset code

maybe

is it valid even to have {"path1/file" {:path "other/path/file"}} in the :tree?

that's interesting because the pipeline is what protects the consistency of the fileset

onetom: no, that would cause havoc and insanity

so that's why i felt that we are dealing with some derived view of some "stem" data

if the fileset was made of cells and more self-managed, the pipeline would be less important

and there should be code written to keep it consistent

or have to invent some restricted APIs to keep it consistent...

well messing around in there isn't part of the public API

that's why there are functions in boot.core, like tmp-file, tmp-path, tmp-dir, etc

we could add more assertions, but the public API won't create inconsistent things

i agree that boot.core/cp is probably beyond repair in its current form

and should be removed or fixed

i'm totally fine with breaking changes as long as we bump version to 3.0.0

and have a good changelog

i wouldnt know though how to do the cp if i wouldn't have had access to the underlying (Tmp)FileSet 😕

+1 for semver

no, cp would of course have access to the internal api

but it would be written in such a way that it does the sane thing

and doesn't corrupt the fileset

anyway, i don't have a lot of clarity yet on this fileset vs javelin, but i thought i would put it there so u can ponder about it

i haven't found javelin to be so necessary on the backend really

as i was reading the code i found it hard to decipher the reason why all that code is there

the purpose doesn't shine thru it

the difference is that the frontend is event driven by necessity

and the backend is top-down

javelin is what you need when you haev an event driven system

because you don't have a single entry point

with boot pipeline you do

and we have fs change events...

so javelin is less attractive

yeah but that is just a single event, basically

a "run pipeline" event

in the browser you have many many different events

and many different entrypoints

maybe we don't want a pipeline

unless u look at it as every input file is a possible event source

currently we don't

we batch fs updates

we debounce the filesystem events handler

so we don't rebuild until the filesystem changes are all in

i have a gut feeling it's actually cleaner to think about it as data-flow

yeah, i understand that but it's just like a graphics rendering pipe-line

we did consider this with boot1 originally

but it never really panned out

it flushes its content every few milliseconds

yeah i don't know if javelin helps with a rendering pipeline either

a pipeline just needs queues

that's what core.async is good at

ah...

with hoplon though, javelin is essential

because it's not a pipeline

it's a graph

there are many inputs and many outputs

and they are interconnected in complex ways

a rendering pipeline is simple, one input and one output

and a chain of things in the middle that have one input and one output

so at most you need a queue to decouple inputs and outputs

this is like the watch task

it's a queue in the pipeline

so it can batch inputs, etc

block on the queue

and so on

thx again for the explanation. i will ponder more about it.

i have another question, which is related to hoplon

so the <script> tags are synchronous by default

only when they're part of the HTML the browser loads

if i have a <script src="xxx.js"></script> which defines the variable

not when you create them dynamically

if you need them to be synchronous you must prerender them

and it's followed by a <script>xxx.something</script> then xxx would be accessible for sure if the xxx.js had a xxx = {something: ...}

you can use the new (static (script :src "...

but in development we are not using prerendering so (static (script ... )) would break, no?

it would be asynchronous, yes

(which is another thing which we looked into the source code of but couldnt understand...)

i think we should treat scripts specially in hoplon

your commit message was giving the example of (head :static (not prerendering?) ...)

and have hoplon itself prerender them

which would fail in dev mode

yeah

you don't need to do that though

i don't see any need for script tags like that, why do you need it?

i saw u were adding :static-id attributes too but their usage was unclear and also seemed unrelated to the (static ...) macro

:static-id is an implementation detail of the static macro

you shouldn't need to worry about them

well, it would be great to access that semantics of the script tags for integrating 3rd party code

but if you have external js you should use a deps.cljs file

that will properly integrate the js script into the dependency graph in clojurescript

and would it also download the script at compile time?

you could make a task that does that, yes

download script add to fileset

generate deps.cljs and add that to fileset

also it was not clear if i should reference the namespace it provides in the (ns (:require )) since the vars are accessible as js/... anyway

yeah you need to :require the external js

so that the dependency graph can be computed

but you use it as usual with js/...

(ns foo
  (:require [cljsjs.jquery]))

(js/jQuery ...

u made a task like that the other time but it scrolled out of the chat buffer... 😞 either way, im just surprised u can specify a remote URL in the foreign lib and havent found any docs on it how it works

this ensures that jquery.js will be loaded before foo.js

oh, wait, what, even for jQuery i have to explicitely require it?

yeah, but hoplon itself :requires it, so if you require hoplon it will be alraedy done

it thought it's part of ... ok

i mean it's slightly incorrect

u mean if im in a .hl file, then i don't need cljsjs.jquery?

but it happens to work currently

but i still would need other cljsjs. stuff, right?

yeah

you really should require cljsjs.jquery in the .hl files too

it's just that currently at least it works

because nothing is checking the correctness of js/... calls

i saw somewhere it's also possible to use cljsjs.jquery/jQuery, no?

no, that won't work

it's external js, meaning js that isn't proper GCL namespaces

if they were proper GCL namespaces we wouldn't need any of the deps.cljs stuff

deps.cljs just adds enough metadata to compute the dependency graph

but it can't make random javascript into a real gcl namespace

so external code can be initialized in the correct order?

yeah

and tree shaking

and if i dont require it i still can access it, how is it possible?

if your application doesn't :require cljsjs.jquery then that js won't be included

it just gets added to the list of modules as something which no one depends on explicitely?

?

but i think i can answer these for myself if i investigate the generated files a bit

if you don't require it you're just relying on someone else :require it in code that runs before your code

u helped tremendously again so i can continue hacking all night

aaaah!

because the cljs compiler doesn't have any way to know if js/jQuery is in the environment or not

it's on you to make sure the external js has been loaded, and you do that by :requier

wait, but what if u provide an externs file?

that's for advanced optimizations

wouldnt those symbols in the externs file become available under the namespace of the foreign-lib?

not really

i must have dreamt that then...

externs file is just like a blacklist sort of

"black"list

but u shall do your job too

it disables name munging

thanks for the lot of help!

sure anytime!

will you be around this weekend?

i will ask later tonight probably because we are trying to roll this new hoplon6-alpha6 homepage out tomorrow or the latest the day after

yeah, weekend too

i was thinking maybe we do the big hoplon reorganization this weekend

i got stable (unlimited traffic) internet too again

and start on all the things we need for a stable community

like the docs, etc

but now i have to help to put my daughter into bed... she is 10months old now and started to be naughty

hahaha awesome

hoplon reorg sounds like fun!

and i ride to work

good! cya

@martinklepsch: hi! we are trying to use your https://clojars.org/cljsjs/auth0-lock but with hoplon. it tries to insert some CSS into the <head> automatically already at the beginning of loading the library: https://github.com/auth0/lock/blob/master/lib/insert-css/index.js#L13-L17 but then the head gets blown away by hoplon. have you experienced this? do u have any solution for it? or you are not really using hoplon? if we include a <script> tag at the end of the (body) then it works, but that's ugly obviously.

@micha: what if

(reset! (.-hoplonKids elem) (vec kids))

(swap! (.-hoplonKids elem) conj (vec kids))

(swap! (.-hoplonKids elem) (partial conj (vec kids)))

i think it wouldn't work

did you try it?

on the surface of it, it seems to solve my problem with auth0 tinkering with the <head> even before (head) could kick in

yeah i think it would cause worse problems

yeah, the <style> auth0 inserts stays there

also tinkering with the head on load is really bad practice

it's just as bad as prevalent

at least a library should provide a constructor

maybe load it in an iframe

should, but practically NONE does which im using currently

you should try it

most of the time with these libraries you can work around them

all of them have the infamous document.getElementByTagName("head")[0].appendChild(document.createElement("script" or "style"))

yeah you can achieve the same as that by other means

but what made you say "it would cause worse problems"?

like what problems?

i think if you make that change you will see

i think it will break lots of things

i tried it but it looks okay to me so far

prerendering on?

no, in dev mode

try with prerendering

ok

works fine with simple optimizations too (all im doing in my example is to initialize the auth0 popup)

oh, try it on a real application

ah, wait a sec, i put the prerender in the wrong place. trying again

it has inserted shit twice, indeed, in that case

did you try doing (head :static true ...) in your app?

and then prerender

no because that would make the app useless without prerendering

why not use prerendering?

in dev mode?

yeah

never really thought about that

the question is more like why would i?

so you can do these things with the <head>

"it's slow o'readyyyy!"

that's why i didn't think about making it even slower

you can wrap most of the application in (if-not prerendering? ...)

and it wont run in phantomjs

that's what i do for my app, because we don't care about prerendering the content

we just need the loading spinner to be prerendered

with (head :static true ...) phantomjs doesnt even terminate...

maybe i will ping these auth0 guys... is there any library which you would point them to if you were to explain them how should a lib constructor look like?

google analytics

pretty much any serious company

adzerk even

lol

i imagine their bugtracker is filled with requests like thhis already

do u have a google analytics init snippet in cljs? we were just using (script "....") until now

yeah but inside the script tag is just js calling a function

you can call that function whenever you want, and it doesn't do anything until you tell it to

that's what you need with auth0 too

you just want to call some function to let it do its thing, but after hoplon has finished doing its own thing

no?