Fork me on GitHub
#hoplon
<
2015-08-25
>
micha11:08:55

@onetom: are you seeing this performance with -SNAPSHOT or are you using -3?

micha11:08:38

it's possible the changes we made to support hot-reloading tests might have affected compilation times

onetom11:08:20

im using -3 if it's just 1 big file, the difference is not so noticable

micha11:08:44

this weekend i'll look at perf again

micha11:08:50

see what can be done

onetom11:08:51

the recompile time varies anyway, probably because of jvm warmup too

micha11:08:07

it seems like my application is taking seconds to recompile too

micha11:08:18

which seems like a lot since i haven't changed much

micha11:08:23

it should be < 1s

onetom11:08:32

yup, same here. it was subsec b4

micha11:08:53

there must have been a regression in the cljs task or the compiler

micha11:08:34

jvm warmup will always be onerous

micha11:08:46

i think we want to concentrate on incremental build times

micha11:08:54

and minimize those as much as possible

micha11:08:11

maybe jvm options could help there, too

micha11:08:28

optimize more for long running jobs than for startup time

micha11:08:38

-server for example

onetom11:08:56

we are on 1.8; there is no -server flag anymore, no?

micha11:08:06

oh, i wasn't aware of that change

micha11:08:50

if you have a lot of ram you might want to try running boot from a ramdisk

micha11:08:05

alan reports some improvement with that setup

micha11:08:29

like have boot write tempfiles to ramdisk

onetom11:08:30

sorry, they only removed the -client

onetom11:08:39

-server	  to select the "server" VM
                  The default VM is server,
                  because you are running on a server-class machine.

micha11:08:21

it's not a huge win because boot is already pretty efficient with temp files

micha11:08:30

but it might shave a second off the time

onetom11:08:34

i was reading the tempfile handling code of boot a lot btw caused a lot of head scratching... like we couldn't really figure out how to create a TmpFile. the boot code only had 1 example of it. we tried (cp ...) too but it was complaining about the tmpdir not being recognized

onetom11:08:00

it's definitely not a code base for beginners like ourselves simple_smile

micha11:08:23

tests would have helped, alas 😕

micha11:08:32

sorry about that

micha11:08:11

with cp you really want to take an existing TmpFile and modify its path

onetom11:08:17

but at the end my hack worked: https://github.com/exicon/homepage/blob/SPA/build.boot#L31-L49 have u seen it? do u have any immediate comment on it?

onetom11:08:45

i've noticed it was creating symlinks for the index.html.out/ but not for the index.html.js

micha11:08:06

hmm that looks legit at first glance

micha11:08:39

line 41 looks suspect

micha11:08:54

you shouldn't have to deal with File objects

micha11:08:23

like given some TmpFile t with path "foo/bar.js" for example

micha11:08:48

if you want to copy that to "baz/foo/bar.js" say

micha11:08:52

you could do

onetom11:08:27

i've seen later there are helper functions to lookup the various components of the TmpFile/TmpDir but not sure what's the purpose of those

micha11:08:54

(-> fileset (cp (tmp-file t) (assoc t :path "baz/foo/bar.js")) commit!)

onetom11:08:24

hmmm.. let me try quickly

micha11:08:58

remember that the FileSet has a fixed set of underlying directories in which it stores the blob files and hard links

micha11:08:29

the blob files are the actual files, those are named by their md5 hash and are stored in the blob dir

micha11:08:03

the rest of the directories associated with the fileset are where the working tree is built

micha11:08:21

they're overlayed on each other to form the fileset tree on disk

micha11:08:04

there are a number of these directories because we need to have some files in the classpath and some not in the classpath

micha11:08:32

but once boot starts those directories are fixed for the lifetime of the JVM

micha11:08:42

you can't add new directories or remove directories

micha11:08:49

at least not without breaking filesets

onetom11:08:20

ah, yeah, that's an other thing. i keep getting "can't delete target/something/something" errors since i use my copy task

onetom11:08:43

im not sure i understand what do u mean by cant add/rm new dirs. doesn't the (tmp-dir! ...) fn make new ones?

micha11:08:04

no, that makes a cache directory

micha11:08:11

that's not in the fileset

micha11:08:34

when you do (add-resource fileset tmpdir) it doesn't add tmpdir to the fileset

micha11:08:49

it copies the files tmpdir contains into directories that are already in the fileset

onetom11:08:49

aw... 😕 i thought for a while i understood things but now i don't understand again

micha11:08:39

the overall purpose of the fileset is to have a standard abstraction for the filessystem as it relates to the JVM

micha11:08:58

that abstraction only needs a certain number of directories

micha11:08:10

it needs a directory for files that will be on the classpath

micha11:08:24

it needs a directory for files that will not be on the classpath

micha11:08:43

and for boot's convenience it needs a directory for files what will be output files

micha11:08:52

and for files that won't be output files

micha11:08:11

and a few more things for organizing the files for boot's conveninece

micha11:08:19

but there is no need to ever add another directory

micha11:08:44

instead, when you do (add-resource ...) what happens is the files in the tmpdir are copied into blob storage

micha11:08:57

now we have the file contents decoupled from the path

micha11:08:04

that's super important

micha11:08:22

because now it's possible to refer to the file without referring to a specific path on the filesystem

micha11:08:35

this means we can make abstractions on top of it

micha11:08:42

which is what the FileSet is

micha12:08:14

the TmpFileSet is a mapping from object in blob storage to relative path in the tree

onetom12:08:45

yeah, that part was roughly clear

micha12:08:59

when you call commit! on a fileset it takes those mappings and makes hard links on disk pointing to the blob files

micha12:08:08

with the appropriate relative paths

onetom12:08:13

u mentioned FileSet and TmpFileSet. any difference?

micha12:08:24

first one was typo

micha12:08:41

i forget the name of the record type

onetom12:08:45

any reason to call it Tmp?

micha12:08:16

it's all kind of related to managing temporary directories and whatnot, but not really

micha12:08:48

it's like git

micha12:08:01

git has an internal tree representation that you don't need to mess with

onetom12:08:07

as i saw there was a new-fileset fn to create one but then no other one is created, but only that is evolving?

micha12:08:14

when you do git add mydir

micha12:08:19

what happens?

micha12:08:29

it doesn't add a directory to your .git

onetom12:08:31

it goes into the object store

micha12:08:55

it adds the files to blob storage

micha12:08:12

and makes links internally to things in its tree graph

micha12:08:24

so it can generate the working set when it is commanded to

onetom12:08:26

the blob storage is also something created by (tmp-dir! ...) no?

onetom12:08:36

at least based on its path it looked like it

micha12:08:42

tmp-dir! just gives you a regular directory

micha12:08:56

the only reason boot has that function is because boot will clean those up for you

micha12:08:28

using the normal java temp directory machinery isn't as good because cleaning them up is a pain

micha12:08:00

like iwth boot if you make a tmp-dir! you can see things in the directory after boot exits

micha12:08:11

boot won't delete stale temp dirs until the next time it runs

micha12:08:16

so you can debug problems

micha12:08:31

if you use a regular java temp dir you can't do that

onetom12:08:09

(it's all golden what u are saying. i will save it and try to rework it into some kind of documentation / article)

micha12:08:20

but (tmp-dir!) just returns a normal java.io.File object

micha12:08:45

and no other shenanigans other than that directory is in $BOOT_HOME, so boot can know when to delete it

onetom12:08:01

(i just looked at the code, it was tmp-dir*; again a bit confusing as an "outsider" simple_smile

micha12:08:19

ah yeah that's not part of the public API

micha12:08:32

that's part of what boot does when it initializes the fileset

micha12:08:24

that code is a little complicated maybe but there was a lot of reprtition and clerical errors so i did some refactoring that made it more confusing but less prone to clerical errors and typos

onetom12:08:25

so during the lifetime of boot there is only 1 fileset is created newly?

onetom12:08:35

the rest are all "mutations" of it?

micha12:08:04

well there is only one set of mutable global directories on the actual filesystem

micha12:08:18

the fileset objects themselves are immutable

micha12:08:47

the thing that's a singleton is the place where files are written when you call commit! on an immutable fileset

micha12:08:01

because the JVM's classpath is a singleton

micha12:08:06

and the disk is a singleton

micha12:08:32

a haskell person could tell you what kind of monad the fileset is

micha12:08:39

i'm sure it's some monad

onetom12:08:06

so for example that was a bit unclear how to do a series of "changes" to the fileset

onetom12:08:25

if u r doing that (cp ...) u showed above

onetom12:08:31

how do u do a series of them?

onetom12:08:41

use reduce?

micha12:08:47

you can chain them, yes or reduce

micha12:08:16

then you need to commit! of course

micha12:08:42

any operation that creates a new fileset will not affect the actual underlying filesystem

onetom12:08:59

sure, that's clear too

micha12:08:05

so you can more efficiently separate the writing from the manipulation

micha12:08:13

batch the disk writes basically

micha12:08:00

and that's not really 100% true, because some things like add-resource etc. actually do add things to blob storage

micha12:08:17

but that's an implementation detail that shouldn't make a difference

micha12:08:35

because adding to blob storage is idempotent

micha12:08:05

and blob storage doesn't affect anything the user cares about

micha12:08:13

it's hidden in the fileset internals

onetom12:08:19

though cp is doing an md5 calculation unconditionally

micha12:08:33

it needs to

onetom12:08:24

but it was receiving a TmpFile which had an :id already 😕

micha12:08:37

yeah cp is messed up

micha12:08:45

i don't like that guy

onetom12:08:52

that was another thing which made me scratch my head simple_smile

micha12:08:03

yeah i have no idea what i was thinking there

micha12:08:08

is it used anywhere in boot?

onetom12:08:14

btw, before i forget, pls push the boot-hoplon source!

onetom12:08:43

i was looking into the source by unpacking the jar u published...

micha12:08:04

dang sorry

onetom12:08:29

and no, cp was not used anywhere; that's why i was a but clueless how to use it simple_smile

micha12:08:50

we should fix it

onetom12:08:53

any idea at 1st glance what can cause this:

java.io.IOException: Couldn't delete target/index.html.out/homepage/layout.cljs
                                           io.clj:  426
                                       ...
                     boot.file/delete-file                          file.clj:   52
                           boot.file/sync!                          file.clj:  190
                                       ...
                        clojure.core/apply                          core.clj:  634
                  boot.core/sync-target/fn                          core.clj:  687
                     boot.core/sync-target                          core.clj:  686
                    boot.core/run-tasks/fn                          core.clj:  695
tailrecursion.boot-hoplon/eval343/fn/fn/fn                   boot_hoplon.clj:   45

onetom12:08:33

i have the feeling that my copy-index-html is doing something nasty which spits into the soup

micha12:08:08

i don't understand why it's failing to delete something in the target dir

micha12:08:16

that seems very strange to me

onetom12:08:30

but i was not using cp... so im not sure if it was valid to just grab a TmpFile, change its path and shove it back to the (:tree fileset)

micha12:08:34

unless maybe a race condition

micha12:08:11

the target dir is unrelated to the flieset

onetom12:08:18

yeah, it's a heisenbug, but happened quite often today when i was switching between prod and dev runs

micha12:08:24

so i don't think it is caused by something you did to that

micha12:08:49

ok i ride to work

micha12:08:52

back in a bit

onetom12:08:14

just a thought for the ride!

onetom12:08:36

would it make sense to use javelin somehow for the fileset?

onetom12:08:36

imean i saw a lot of code extracting the path from TmpFiles then putting them back to objects or the :tree

micha12:08:22

how would javelin help?

onetom12:08:21

(defc= fileset {:tree (map (fn [f] {(:path f) f}) files)  :blob ...  :dirs ...})

micha12:08:48

interesting

micha12:08:31

it would probably clean up the fileset code

onetom12:08:51

is it valid even to have {"path1/file" {:path "other/path/file"}} in the :tree?

alandipert12:08:55

that's interesting because the pipeline is what protects the consistency of the fileset

micha12:08:09

onetom: no, that would cause havoc and insanity

onetom12:08:44

so that's why i felt that we are dealing with some derived view of some "stem" data

alandipert12:08:48

if the fileset was made of cells and more self-managed, the pipeline would be less important

onetom12:08:55

and there should be code written to keep it consistent

onetom12:08:13

or have to invent some restricted APIs to keep it consistent...

micha12:08:19

well messing around in there isn't part of the public API

micha12:08:37

that's why there are functions in boot.core, like tmp-file, tmp-path, tmp-dir, etc

micha12:08:08

we could add more assertions, but the public API won't create inconsistent things

micha12:08:31

i agree that boot.core/cp is probably beyond repair in its current form

micha12:08:39

and should be removed or fixed

micha12:08:58

i'm totally fine with breaking changes as long as we bump version to 3.0.0

micha12:08:11

and have a good changelog

onetom12:08:22

i wouldnt know though how to do the cp if i wouldn't have had access to the underlying (Tmp)FileSet 😕

onetom12:08:39

+1 for semver

micha12:08:40

no, cp would of course have access to the internal api

micha12:08:56

but it would be written in such a way that it does the sane thing

micha12:08:04

and doesn't corrupt the fileset

onetom12:08:24

anyway, i don't have a lot of clarity yet on this fileset vs javelin, but i thought i would put it there so u can ponder about it

micha12:08:51

i haven't found javelin to be so necessary on the backend really

onetom12:08:54

as i was reading the code i found it hard to decipher the reason why all that code is there

onetom12:08:03

the purpose doesn't shine thru it

micha12:08:08

the difference is that the frontend is event driven by necessity

micha12:08:16

and the backend is top-down

micha12:08:41

javelin is what you need when you haev an event driven system

micha12:08:51

because you don't have a single entry point

micha12:08:56

with boot pipeline you do

onetom12:08:01

and we have fs change events...

micha12:08:04

so javelin is less attractive

micha12:08:14

yeah but that is just a single event, basically

micha12:08:19

a "run pipeline" event

micha12:08:49

in the browser you have many many different events

micha12:08:02

and many different entrypoints

micha12:08:29

maybe we don't want a pipeline

onetom12:08:41

unless u look at it as every input file is a possible event source

micha12:08:00

currently we don't

micha12:08:07

we batch fs updates

micha12:08:20

we debounce the filesystem events handler

micha12:08:37

so we don't rebuild until the filesystem changes are all in

onetom12:08:41

i have a gut feeling it's actually cleaner to think about it as data-flow

onetom12:08:06

yeah, i understand that but it's just like a graphics rendering pipe-line

micha12:08:06

we did consider this with boot1 originally

micha12:08:19

but it never really panned out

onetom12:08:31

it flushes its content every few milliseconds

micha12:08:54

yeah i don't know if javelin helps with a rendering pipeline either

micha12:08:06

a pipeline just needs queues

micha12:08:15

that's what core.async is good at

micha12:08:40

with hoplon though, javelin is essential

micha12:08:45

because it's not a pipeline

micha12:08:50

it's a graph

micha12:08:58

there are many inputs and many outputs

micha12:08:11

and they are interconnected in complex ways

micha12:08:22

a rendering pipeline is simple, one input and one output

micha12:08:34

and a chain of things in the middle that have one input and one output

micha12:08:51

so at most you need a queue to decouple inputs and outputs

micha12:08:01

this is like the watch task

micha12:08:09

it's a queue in the pipeline

micha12:08:19

so it can batch inputs, etc

micha12:08:23

block on the queue

micha12:08:25

and so on

onetom12:08:12

thx again for the explanation. i will ponder more about it.

onetom12:08:51

i have another question, which is related to hoplon

onetom12:08:11

so the <script> tags are synchronous by default

micha12:08:42

only when they're part of the HTML the browser loads

onetom12:08:51

if i have a <script src="xxx.js"></script> which defines the variable

micha12:08:58

not when you create them dynamically

micha12:08:27

if you need them to be synchronous you must prerender them

onetom12:08:43

and it's followed by a <script>xxx.something</script> then xxx would be accessible for sure if the xxx.js had a xxx = {something: ...}

micha12:08:50

you can use the new (static (script :src "...

onetom12:08:16

but in development we are not using prerendering so (static (script ... )) would break, no?

micha12:08:27

it would be asynchronous, yes

onetom12:08:35

(which is another thing which we looked into the source code of but couldnt understand...)

micha12:08:03

i think we should treat scripts specially in hoplon

onetom12:08:08

your commit message was giving the example of (head :static (not prerendering?) ...)

micha12:08:09

and have hoplon itself prerender them

onetom12:08:31

which would fail in dev mode

micha12:08:59

you don't need to do that though

micha12:08:17

i don't see any need for script tags like that, why do you need it?

onetom12:08:39

i saw u were adding :static-id attributes too but their usage was unclear and also seemed unrelated to the (static ...) macro

micha12:08:04

:static-id is an implementation detail of the static macro

micha12:08:13

you shouldn't need to worry about them

onetom12:08:28

well, it would be great to access that semantics of the script tags for integrating 3rd party code

micha12:08:40

but if you have external js you should use a deps.cljs file

micha12:08:58

that will properly integrate the js script into the dependency graph in clojurescript

onetom12:08:22

and would it also download the script at compile time?

micha12:08:35

you could make a task that does that, yes

micha12:08:42

download script add to fileset

micha12:08:51

generate deps.cljs and add that to fileset

onetom12:08:12

also it was not clear if i should reference the namespace it provides in the (ns (:require )) since the vars are accessible as js/... anyway

micha12:08:29

yeah you need to :require the external js

micha12:08:35

so that the dependency graph can be computed

micha12:08:43

but you use it as usual with js/...

micha12:08:13

(ns foo
  (:require [cljsjs.jquery]))

(js/jQuery ...

onetom12:08:32

u made a task like that the other time but it scrolled out of the chat buffer... 😞 either way, im just surprised u can specify a remote URL in the foreign lib and havent found any docs on it how it works

micha12:08:38

this ensures that jquery.js will be loaded before foo.js

onetom12:08:49

oh, wait, what, even for jQuery i have to explicitely require it?

micha12:08:14

yeah, but hoplon itself :requires it, so if you require hoplon it will be alraedy done

onetom12:08:20

it thought it's part of ... ok simple_smile

micha12:08:46

i mean it's slightly incorrect

onetom12:08:48

u mean if im in a .hl file, then i don't need cljsjs.jquery?

micha12:08:56

but it happens to work currently

onetom12:08:01

but i still would need other cljsjs. stuff, right?

micha12:08:18

you really should require cljsjs.jquery in the .hl files too

micha12:08:27

it's just that currently at least it works

micha12:08:39

because nothing is checking the correctness of js/... calls

onetom12:08:44

i saw somewhere it's also possible to use cljsjs.jquery/jQuery, no?

micha12:08:52

no, that won't work

micha13:08:23

it's external js, meaning js that isn't proper GCL namespaces

micha13:08:45

if they were proper GCL namespaces we wouldn't need any of the deps.cljs stuff

micha13:08:16

deps.cljs just adds enough metadata to compute the dependency graph

micha13:08:38

but it can't make random javascript into a real gcl namespace

onetom13:08:57

so external code can be initialized in the correct order?

micha13:08:13

and tree shaking

onetom13:08:24

and if i dont require it i still can access it, how is it possible?

micha13:08:31

if your application doesn't :require cljsjs.jquery then that js won't be included

onetom13:08:42

it just gets added to the list of modules as something which no one depends on explicitely?

onetom13:08:01

but i think i can answer these for myself if i investigate the generated files a bit

micha13:08:24

if you don't require it you're just relying on someone else :require it in code that runs before your code

onetom13:08:34

u helped tremendously again so i can continue hacking all night simple_smile

micha13:08:58

because the cljs compiler doesn't have any way to know if js/jQuery is in the environment or not

micha13:08:25

it's on you to make sure the external js has been loaded, and you do that by :requier

onetom13:08:31

wait, but what if u provide an externs file?

micha13:08:43

that's for advanced optimizations

onetom13:08:54

wouldnt those symbols in the externs file become available under the namespace of the foreign-lib?

micha13:08:08

not really

onetom13:08:17

i must have dreamt that then...

micha13:08:21

externs file is just like a blacklist sort of

onetom13:08:33

simple_smile "black"list

onetom13:08:57

but u shall do your job too

micha13:08:59

it disables name munging

onetom13:08:10

thanks for the lot of help!

micha13:08:15

sure anytime!

micha13:08:35

will you be around this weekend?

onetom13:08:39

i will ask later tonight probably because we are trying to roll this new hoplon6-alpha6 homepage out tomorrow or the latest the day after

onetom13:08:49

yeah, weekend too

micha13:08:56

i was thinking maybe we do the big hoplon reorganization this weekend

onetom13:08:13

i got stable (unlimited traffic) internet too again

micha13:08:23

and start on all the things we need for a stable community

micha13:08:27

like the docs, etc

onetom13:08:36

but now i have to help to put my daughter into bed... she is 10months old now and started to be naughty simple_smile

micha13:08:44

hahaha awesome

onetom13:08:57

hoplon reorg sounds like fun!

micha13:08:00

and i ride to work

onetom15:08:43

@martinklepsch: hi! we are trying to use your https://clojars.org/cljsjs/auth0-lock but with hoplon. it tries to insert some CSS into the <head> automatically already at the beginning of loading the library: https://github.com/auth0/lock/blob/master/lib/insert-css/index.js#L13-L17 but then the head gets blown away by hoplon. have you experienced this? do u have any solution for it? or you are not really using hoplon? simple_smile if we include a <script> tag at the end of the (body) then it works, but that's ugly obviously.

onetom17:08:16

@micha: what if

(reset! (.-hoplonKids elem) (vec kids))
in https://github.com/tailrecursion/hoplon/blob/master/src/tailrecursion/hoplon.cljs#L291 would be
(swap! (.-hoplonKids elem) conj (vec kids))
or
(swap! (.-hoplonKids elem) (partial conj (vec kids)))
instead?

micha17:08:51

i think it wouldn't work

micha17:08:57

did you try it?

onetom17:08:17

on the surface of it, it seems to solve my problem with auth0 tinkering with the <head> even before (head) could kick in

micha17:08:37

yeah i think it would cause worse problems

onetom17:08:46

yeah, the <style> auth0 inserts stays there

micha17:08:55

also tinkering with the head on load is really bad practice

onetom17:08:07

it's just as bad as prevalent

micha17:08:12

at least a library should provide a constructor

micha17:08:20

maybe load it in an iframe

onetom17:08:57

should, but practically NONE does which im using currently

micha17:08:26

you should try it

micha17:08:54

most of the time with these libraries you can work around them

onetom17:08:54

all of them have the infamous document.getElementByTagName("head")[0].appendChild(document.createElement("script" or "style"))

micha17:08:11

yeah you can achieve the same as that by other means

onetom17:08:23

but what made you say "it would cause worse problems"?

onetom17:08:29

like what problems?

micha17:08:31

i think if you make that change you will see

micha17:08:40

i think it will break lots of things

onetom17:08:52

i tried it but it looks okay to me so far

micha17:08:02

prerendering on?

onetom17:08:22

no, in dev mode

micha17:08:29

try with prerendering

onetom17:08:12

works fine with simple optimizations too (all im doing in my example is to initialize the auth0 popup)

micha17:08:44

oh, try it on a real application

onetom17:08:43

ah, wait a sec, i put the prerender in the wrong place. trying again

onetom17:08:05

it has inserted shit twice, indeed, in that case

micha17:08:51

did you try doing (head :static true ...) in your app?

micha17:08:08

and then prerender

onetom17:08:47

no because that would make the app useless without prerendering

micha17:08:26

why not use prerendering?

onetom17:08:37

in dev mode?

onetom17:08:49

never really thought about that

onetom17:08:57

the question is more like why would i?

micha17:08:12

so you can do these things with the <head>

onetom17:08:33

"it's slow o'readyyyy!"

onetom17:08:55

that's why i didn't think about making it even slower simple_smile

micha17:08:56

you can wrap most of the application in (if-not prerendering? ...)

micha17:08:07

and it wont run in phantomjs

micha17:08:23

that's what i do for my app, because we don't care about prerendering the content

micha17:08:33

we just need the loading spinner to be prerendered

onetom17:08:37

with (head :static true ...) phantomjs doesnt even terminate...

onetom18:08:45

maybe i will ping these auth0 guys... is there any library which you would point them to if you were to explain them how should a lib constructor look like?

micha18:08:16

google analytics

micha18:08:24

pretty much any serious company

micha18:08:29

adzerk even

micha18:08:45

i imagine their bugtracker is filled with requests like thhis already

onetom18:08:25

do u have a google analytics init snippet in cljs? we were just using (script "....") until now

micha18:08:38

yeah but inside the script tag is just js calling a function

micha18:08:01

you can call that function whenever you want, and it doesn't do anything until you tell it to

micha18:08:08

that's what you need with auth0 too

micha18:08:27

you just want to call some function to let it do its thing, but after hoplon has finished doing its own thing