Fork me on GitHub
#boot
<
2016-02-25
>
richiardiandrea01:02:28

hey guys, in boot-test-cljs/`doo` is there are way to perform the equivalent of Phantom's require('webpage') ?

onetom07:02:24

@danielsz: thanks for the quick system release; we are putting it into production today or tomorrow.

onetom07:02:39

btw, i don't quite understand what's going on w the various github org names. the https://clojars.org/org.danielsz/system/versions/0.3.0-SNAPSHOT page shows a github link to tracksun/system which in turn redirects to https://github.com/tsonntag/system (which i can understand; the original org was renamed.) where is tracksun still coming from? is it a clojars org? i would expecet to see danielsz/system in that header too 😕

danielsz09:02:28

@onetom Yes, you are right, I would have expected that too. I'll check it out. Thanks for pointing it out. 👍

danielsz09:02:24

Oh, @juhoteperi already did. My hero, again. Well, thanks!

danielsz10:02:38

I'll fix it ASAP

dm311:02:39

driving benchmarks (highly unscientific) with boot: https://github.com/dm3/clojure-bench/blob/master/build.boot

danielsz14:02:33

@onetom: Unfortunately, I let the url from a contributor slip inside the scm url during the release of system 0.2.0. Because the artefact is already deployed and can't be overwritten, I can only fix it by releasing a new version. I can either bump the minor version and release now, or just wait it out until the proper 0.3.0 release.

onetom14:02:21

@danielsz: it doesn't cause any practical issues, so no point making a new release just fixing that. i just asked because it bothers me if i dont understand something which seems simple enough that i should simple_smile

danielsz14:02:20

I'm with you here: always be investigating. simple_smile Cool. We'll wait it out.

onetom14:02:24

however, it would be good if u could git push --tags just so the stable branch releases could be correlated with the commits easily from github

danielsz14:02:47

Yes, I was doing that and then I got sloppy. Sorry about that. I'll properly tag the releases in the future. It is important.

onetom14:02:29

it was a bit of annoyance when i tried to check if an issue was related to a specific version or not, that's why im asking

onetom14:02:34

as i see u created quite a crucial library which many people will rely on in the core of their app, so all the boring bits of maintenance should be done well otherwise users will annoy the hell out of u simple_smile

onetom14:02:30

i guess... i haven't maintained anything so important yet...

onetom14:02:31

btw, how are u managing the servers for running the tests for the various components?

onetom14:02:27

i dont know docker well enough yet, but i imagined it would be possible to ship a bunch of Dockerfiles w system, so a test instance of every service which it has a component for could be spun up temporarily for a test run.

danielsz14:02:38

In FOSS, there are often misconceptions about roles and responsibilities. I don't remember where I saw this tweet about FOSS users often behaving as customers.

danielsz14:02:54

I am doing what interests me, and make it available. Others are welcome to join, and put their expertise.

danielsz14:02:41

If you are good at setting up testing infrastructure, please give it a go.

danielsz14:02:09

For the moment, nothing fancy is in place.

danielsz14:02:44

I run the tests manually, and I have most of the external resources installed on my development machine.

danielsz14:02:06

That's an area where the devops expert would laugh at my amateurism. But I say, please step in, please contribute.

danielsz14:02:59

So all in all there is a fine line to tread. Both for the FOSS maintainer as for the FOSS consumer.

mobileink14:02:42

A question about conditional tasks. I have a situation where I want to do (comp (A) (if foo (B)) (C) (D)). That doesn't work so I tried to make (if ...) act more like a task: (comp (A) (if foo (B) (fn [next] (fn [fs] fs))) (C) (D)), i.e. tried to insert a nop/identity task. it sorta worked, but tasks C and D don't get run. in this case, B and C are both sift tasks so i solved it this way:

mobileink14:02:43

(let [sift-args (concat [:move {#"(.*\.clj$)" "WEB-INF/classes/$1"}] (if keep [:to-asset #{#"\.clj$"}]))] (comp (A) (apply builtin/sift sift-args) (D)))

mobileink14:02:56

what's the Right Way to handle conditional tasks?

dm314:02:54

(comp (a) (if cond? (b) identity) (c) (d))

mobileink14:02:54

egad. first thing i tried and it didn't work. i must have bungled the syntax, it works. thanks, sorry for the bother

micha15:02:07

i keep forgetting to make a ticket for that

micha15:02:23

i think the pipeline should be forgiving of nils

micha15:02:28

just ignore them

dm315:02:52

how would you modify how comp behaves?

micha15:02:26

haha nonsense, please to disregard

mobileink15:02:32

beat you to it

mobileink15:02:48

argh, just saw dm3's question.

dominicm15:02:57

I did that the other day, left me confused for a solid 20m

laforge4915:02:53

Isn't it more a matter of remembering how comp works? Not a boot issue at all but a matter of making the point in the comp documentation.

mobileink15:02:53

for me it was a matter of the concept of task. tasks may be functions but they don't feel like mere functions; at the least, they're higher-order fns. i think a doc section on pipelines would be helpful to newcomers.

mobileink15:02:36

... and i just noticed the section on "The Null Task" on the wiki, so i guess it's already there. oy.

mobileink15:02:04

its more a matter of remembering to actually read the docs, heh

laforge4915:02:34

Been there, not done that too.

jjttjj17:02:23

is there a way with boot-cljs-repl to have a file that corresponds to the default namespace of the repl, similar to how the clojure repl loads up in user.clj?

jjttjj17:02:51

i tried just a cljs.user namespace but when i do the browser doesn't seem to connect

jjttjj17:02:56

actually looking at the boot-cljs-repl source I'm probably better off just using the lower level piggyback/weasel stuff manually right?

pandeiro18:02:12

if I have a task that is doing setup prior to the with-pre-wrap, but I would like that setup to have access to the fileset somehow, I have to move it into the with-pre-wrap right?

micha18:02:13

@pandeiro: yes, you will need to do that setup each time it runs

micha18:02:26

you can cache the result of course

pandeiro18:02:46

right, we were using a delay before

micha18:02:47

if you have a sure way of knowing when to invalidate the cache

micha18:02:18

i think it's always best to do extra work if there is a chance that the cached values will be wrong

micha18:02:50

because correctness is usually faster in the long run than outright speed

micha18:02:23

since you don't waste time debugging against busted state

pandeiro18:02:35

that's what I am dealing with now anyway

micha18:02:49

like they say in the army, slow is smooth and smooth is fast

pandeiro18:02:29

I'm not 100% sure, but my hypothesis is that to serve non-stale resources, I need to get them directly from the fileset

pandeiro18:02:49

unfortunately I think that will mean I can't use the existing ring wrap-resource stuff

micha18:02:08

that's only the case if jetty is caching things

micha18:02:19

you can probably turn that off in the jetty config

micha18:02:40

yeah try it

micha18:02:49

resources are not cached in the JVM

micha18:02:04

well not for directories on the classpath

micha18:02:16

references to resources in jars are cached

micha18:02:25

because you can't change a jar that's on the classpath

micha18:02:37

the JVM will just crash if you do that

pandeiro18:02:05

right this is not jar stuff, just stuff that's in :resource-paths

micha18:02:37

you can do (slurp (io/resource "foo.txt")) in the repl

micha18:02:41

then change the file

micha18:02:43

and do it again

micha18:02:56

you'll see it produces the correct values

pandeiro18:02:58

yeah, that works

pandeiro18:02:37

wow yeah @micha, running with httpkit instead of jetty, the served resources aren't stale

pandeiro18:02:48

so it's something I have to turn off in jetty, i just have to figure out what

pandeiro18:02:58

thanks a bunch, that is a big break

pandeiro18:02:11

i wonder if the issue with jetty on windows is related

pandeiro18:02:34

@laforge49: weren't you setting something about a file-mapped buffer to get jetty to work on windows?

micha18:02:05

that was about keeping files open

micha18:02:12

that doesn't work on windows

micha18:02:18

not related

micha18:02:46

i mean i don't think it's related simple_smile

pandeiro18:02:50

ok, i'm taking wild swings via SO

pandeiro18:02:34

i guess i'll see how far i make it into http://www.eclipse.org/jetty/documentation/current/ tonight before i fall asleep

pandeiro19:02:12

i've entered bizarro programming land again... i made a simple println debugging statement in ring-jetty-adapter's proxy of the handler class and reloaded boot, and now i'm not seeing the stale resources from jetty anymore

pandeiro19:02:35

then i removed the println and the stale resources were still "fixed"

dm319:02:20

@micha - if you're interested, the result of my yesterday's struggles with with-env (which I didn't use in the end) : https://github.com/dm3/clojure-bench/blob/master/build.boot

dm319:02:18

so something like runboot in core makes sense, only synchronization is painful

pandeiro19:02:44

yeah so I misunderstood my problem, it doesn't seem to be jetty-specific at all, but rather something to do with ring's wrap-resource and how it resolves a filename

pandeiro20:02:58

like if a boot task modifies index.html and commits the modified version to the fileset, and later in the pipeline, another task slurps (io/resource "index.html") , it should obviously see the modified version right?

micha20:02:59

yes it should

pandeiro20:02:31

inching closer, i only see the behavior in pipelines without watch

pandeiro20:02:50

when watch is involved, the modified resource is being served

pandeiro20:02:58

without it, the original disk version

pandeiro20:02:01

ah-ha, and only if watch is not involved and the resource-modifying task comes before the serve task

pandeiro20:02:13

if it comes after, the modified resource is served

pandeiro21:02:15

how can I determine what classloader is being used / what classloaders exist within a boot process?

micha21:02:19

@pandeiro: how do you mean?

micha21:02:27

are you using pods?

micha21:02:47

what do you mean by "boot process" there?

pandeiro21:02:30

heh, not even sure

pandeiro21:02:41

It's just that the resource response in ring takes this option:

pandeiro21:02:28

and i'm testing and what is happening is that sometimes the resource URL is resolving to the modified version's resource, and sometimes to the previous

pandeiro21:02:34

it doesn't seem deterministic

pandeiro21:02:47

i wonder if I needed to pass something to that :loader option

micha21:02:59

oh yeah i added that

micha21:02:03

for this purpose

micha21:02:12

well sort of for this purpose

pandeiro21:02:22

please, tell me more

micha21:02:50

it's to avoid the need for :asset-paths basically

micha21:02:59

load static resources into a separate classloader

micha21:02:23

so you don't need to do the janky security by "public" subdirectory nonsense

micha21:02:40

i don't think it will help your problem though

pandeiro21:02:48

ah ok... so i can't tell... yeah doesn't seem related then

pandeiro21:02:41

i guess one factor is the time it is taking jetty to start up could determine which resource URL is used?

pandeiro21:02:05

b/c i have a simple reproduction case and literally half the time, it's one and half, the other

pandeiro21:02:27

i guess because of threads (?)

micha21:02:47

seems reasonable to me

pandeiro21:02:49

ok guess i will move the instantiation of the server into with-pre-wrap and see if that makes it more predictable

pandeiro21:02:56

does that sound like it may change things?

micha21:02:22

i don't think that will help

micha21:02:40

so with the watch task it works, right/

micha21:02:06

and without the watch task it works if you have the serve task after the task that modifies things, right?

micha21:02:18

is that the current status?

pandeiro21:02:36

@micha honestly i can't say i'm 100% certain, because of how flaky it is

pandeiro21:02:44

with watch it seemed to work

pandeiro21:02:02

but if that were true, would you have a certain hypothesis?

pandeiro21:02:57

what does seem true is that the resource is much less likely to be stale when watch is involved and when the resource-modifying task is after (not before) serve

pandeiro21:02:38

but again, even with no watch and resource-modifier before serve, sometimes the updated resource URL is used

micha21:02:49

what do you mean by updated resource URL

micha21:02:15

how would the URL change?

pandeiro21:02:01

like doing (.toString (io/resource "foo.html")) produces different things

pandeiro21:02:05

file:/home/mu/.boot/cache/tmp/home/mu/boothttpbug/csp/-rcsl8f/foo.html

pandeiro21:02:11

file:/home/mu/.boot/cache/tmp/home/mu/boothttpbug/csp/f0sqpx/foo.html

micha21:02:34

ah ok, this is starting to make sense

pandeiro21:02:37

and i instrumented ring resource response

pandeiro21:02:44

and found that sometimes it points to one, sometimes the other

micha21:02:55

how is that file being changed?

pandeiro21:02:09

in a boot task, added to the fileset

pandeiro21:02:13

with add-resource

micha21:02:17

like make a modified copy in a tmpdir and then do add-resource?

micha21:02:29

ok yeah that's ok

micha21:02:58

the original one should not exist anymore after you do that

pandeiro21:02:22

like in the filesystem?

pandeiro21:02:28

at .boot/cache/...

pandeiro21:02:50

i didn't check if it actually exists anymore, should I search for it

micha21:02:04

it would be good to make sure

micha21:02:24

it's not supposed to exist, if it does then there's a bug in the fileset code

pandeiro21:02:40

yeah both versions exist

pandeiro21:02:54

i did file /home/mu/.boot/cache/tmp/home/mu/boothttpbug/csp/f0sqpx/foo.html and it says it's ASCII text

micha21:02:18

one is in the tmpdir

micha21:02:24

and not on the classpath

micha21:02:30

the other is on the classpath

micha21:02:43

so that is ok i think

pandeiro21:02:11

yeah in one of those dirs, I see everything from my :resource-paths

micha21:02:14

find /home/mu/.boot/cache/tmp/home/mu/boothttpbug/csp -name foo.html

pandeiro21:02:18

in the other (modified), just the file in question

micha21:02:20

how many of those do you see

micha21:02:41

interesting

micha21:02:48

what's the 3rd one?

pandeiro21:02:08

heh, no idea, it never shows up in the printlns

pandeiro21:02:16

should i erase the cache or something and try?

micha21:02:18

i mean what else is in the directory

micha21:02:39

boot is very good about not having a bad cache

pandeiro21:02:53

just the file is in that dir

pandeiro21:02:07

and it is the modified version

micha21:02:29

is it possible that yo're serving from the fileset and the classpath?

micha21:02:41

seems like that's what's happening maybe

pandeiro21:02:54

i ... don't know

pandeiro21:02:14

the fileset never explicitly comes into things

pandeiro21:02:24

since the server setup and everything is done outside the with-pre-wrap

micha21:02:28

actually since http-kit doesn't have the problem, it must be jetty related

pandeiro21:02:33

no, http-kit does

pandeiro21:02:36

that was my mistake

pandeiro21:02:43

due to not removing watch as a variable

pandeiro21:02:53

without watch, http-kit and jetty both show the problem

micha21:02:05

well if you do something with :source-paths or something in your setup

pandeiro21:02:32

by "do something" you mean what?

pandeiro21:02:44

I just set it in set-env! in this case, nothing else

pandeiro21:02:58

and it's only one dir, as is :resource-paths

micha21:02:13

that will add directories to the classpath

micha21:02:47

which could be adding multiple versions of that file to the classpath at the same path

pandeiro21:02:42

i can't tell if that's what's going on or not, but i don't understand what the alternative would be, to serve the html files, then to include them in a dir specified by :resource-paths

pandeiro21:02:56

it's all moot too if I just choose to serve from the target dir, as the modified one is always what ends up there

micha21:02:42

can you link to the code that's doing the stuff with :resource-paths?

pandeiro21:02:39

let me just put the repro on github one sec

micha21:02:02

if i look at the set-env! i might see something

micha21:02:21

i don't really understand how it works

micha21:02:41

seems to me like the classpath doesn't need to be modified by your task

micha21:02:46

the user already set that up

micha21:02:52

so calling set-env! seems strange

micha21:02:38

no i mean in your task

pandeiro21:02:43

no sorry the task doesn't modify anything

pandeiro21:02:46

sorry, just the project

pandeiro21:02:00

i don't do any env mutating in the task

pandeiro21:02:12

except injecting deps when the pod is created

micha22:02:42

interesting, i can't reproduce here

micha22:02:05

i get the <h4> every time

micha22:02:44

barp $ boot dev-once-broken
file:/home/micha/.boot/cache/tmp/home/micha/git/pandeiro/boot-http-bug/kpy/-rcsl8f/foo.html
before update  ::  <h1>foo</h1>
file:/home/micha/.boot/cache/tmp/home/micha/git/pandeiro/boot-http-bug/kpy/f0sqpx/foo.html
after update  ::  <h4>foo</h4>
resources-handler called
2016-02-25 17:03:32.266:INFO::clojure-agent-send-off-pool-0: Logging initialized @3235ms
file:/home/micha/.boot/cache/tmp/home/micha/git/pandeiro/boot-http-bug/kpy/f0sqpx/foo.html
after serve  ::  <h4>foo</h4>

pandeiro22:02:19

in the task output yeah, that's always the same

pandeiro22:02:22

but with curl?

pandeiro22:02:40

curl request returns <h1> version like 70% of the time

pandeiro22:02:47

i didn't include the instrumented version of ring/util/resource-response in the repo, that just does another .toString and verifies which resource URL it matches

pandeiro22:02:06

but they always line up, the h1 points to the old, h4 to new

micha22:02:06

i can't make it return anything else that h4

micha22:02:35

oh ok finally

pandeiro22:02:03

i see more h1 than h4

pandeiro22:02:19

on a pretty fast machine

laforge4922:02:44

classic race condition!

pandeiro22:02:53

that makes it sound much more fun than it is

micha22:02:02

ok i have a modification to make

pandeiro22:02:09

i have another crazy observation in the meantime: if you wait 4-5s before curl, much more likely to be h1

naomarik22:02:44

what distro/windows manager is this? https://youtu.be/TcnzB2tB-8Q?t=7m31s

micha22:02:46

ok PR en route

pandeiro22:02:50

wow @micha, like a fix?

micha22:02:40

@pandeiro: yes

micha22:02:50

it copies the files into a docroot tmpdir

micha22:02:58

instead of using the build classpath

micha22:02:02

and only the output files

micha22:02:08

which i think is the correct behavior

micha22:02:55

you can get fancy with the fileset-diff and patch instead of all the copying

pandeiro22:02:40

wow, i don't even know what a docroot tmpdir is

micha22:02:53

it's a temp dir, named "docroot"

micha22:02:18

basically that's the only directory on the classpath of the pod

micha22:02:25

and you copy the files into that

micha22:02:45

in my PR there is also the "task-src" dir, but that's just because the task is not in a jar

pandeiro22:02:55

and if a file in :resource-paths gets changed during the watch pipeline, how does that work?

micha22:02:56

so i had to add that in for the demo to work

micha22:02:23

well whenever the serve task is hit in the pipeline it copies all the output files from the fileset into the docroot

micha22:02:26

thereby updating them

micha22:02:16

something weird is going on in the fileset, but i'm not 100% sure what

micha22:02:27

but this PR should fix the issue and be more robust

pandeiro22:02:45

awesome, looking now

pandeiro22:02:51

just one doubt: (assoc :directories #{"task-src" (.getPath docroot)})

pandeiro22:02:02

in the actual repo, that would be "src"?

micha22:02:17

no, that's just because of the demo

pandeiro22:02:34

ah so what do I do with that line when I migrate this back to boot-http?

micha22:02:34

the pod would not need that normally, but in this case it needs to load the impl namespace from there

micha22:02:43

just remove the "task-src"

pandeiro22:02:43

ah so kill that line?

micha22:02:55

no, you still need to add the docroot to the :directories

micha22:02:02

or you can't serve the things in there

pandeiro22:02:11

I don't get what :directories is

pandeiro22:02:17

it's input-files and output-files?

micha22:02:17

it's the classpath of the pod

micha22:02:36

that contains a set of directory paths

micha22:02:45

those directories will be on the classpath of the pod

micha22:02:55

no other directories are on the classpath of the pod

pandeiro22:02:58

so it would include everything from :source-paths, :resource-paths?

micha22:02:19

it just includes that one tmp dir

micha22:02:25

because that's how i set it

micha22:02:37

so :source-paths and so on are not on the classpath of the pod

micha22:02:11

when you make a pod you need to specify what the classpath will be

micha22:02:21

the classpath consists of jars and directorues

micha22:02:29

jars are specified via the :dependencies key

pandeiro22:02:33

the :resource-paths don't end up there automatically?

micha22:02:36

those are looked up in maven

micha22:02:47

directories are specified via :directories key

micha22:02:02

in the core pod set-env! manages :directories

pandeiro22:02:39

ok but worker pods need it manually set?

pandeiro22:02:54

i don't understand then why it was serving from :resource-paths at all before this

pandeiro22:02:02

or from wherever boot copies them to

micha22:02:37

because (get-env) has a :directories key

micha22:02:44

since your build.boot runs in a pod

micha22:02:51

that pod needs :directories too

micha22:02:59

but set-env! manages that

micha22:02:24

you'll notice that :directories are anonymous tmp dirs

micha22:02:32

if you do (get-env) in the repl

micha22:02:48

that's not the same thing as :source-paths etc

micha22:02:58

like the "src" directory is not on any classpath

micha22:02:25

instead, an anonymous temp dir is on the classpath, and files are copied from "src" to there whenever they change

micha22:02:43

so tl;dr; it was never serving from :resource-paths

micha22:02:54

it was serving from anonymous tmp dirs in :directories

micha22:02:02

if you want to be precise

naomarik23:02:43

micha i hope you and alan have more talks planned, you two are hilarious together

micha23:02:03

haha @naomarik thanks!

richiardiandrea23:02:35

Man I lost the talk, can you link me up? 😄

richiardiandrea23:02:16

I wanted to keep the generation isolated and I added what I needed through add-classpath

naomarik23:02:29

@richiardiandrea: this the one i'm referring to https://youtu.be/TcnzB2tB-8Q

naomarik23:02:30

@micha did chris granger (with eve) and you guys arrive independently on taking ideas from spreadsheets?

laforge4923:02:45

Got a real hoot from that boot video! Wonderbar! simple_smile