Fork me on GitHub
#boot
<
2015-06-14
>
micha01:06:15

ragge: thanks for looking into this!

micha01:06:01

ragge: interesting, it could be this: http://stackoverflow.com/a/3014860

micha01:06:27

who would expect it to be opening a stream and not closing it?

ragge08:06:35

@micha: yeah, I thought it might be that too at first... clojure has the same bug: http://dev.clojure.org/jira/browse/CLJ-1659

ragge08:06:59

current approach is def leaking, but it wasn't what is causing the issue I'm seeing

ragge08:06:02

ugh, disregaqrd those links

ragge08:06:40

however, although I think both are leaking file handles, they didn't cause my problem

ragge08:06:42

I ended up re-writing unpack-jarto use JarFile and control resources more explicitly

ragge08:06:36

that's fixes my problem

ragge08:06:44

could probably be tidied up a bit, but I had to go to bed

ragge08:06:58

let me know if youre interested in a PR for this

ragge08:06:34

I think, while using URLs is pretty, it comes with a bunch of issues in java stdlib

martinklepsch08:06:35

@micha: @juhoteperi related to pod dependencies I was wondering if it would make sense to add something like the output of boot show -d for each pod when called with -vv — thoughts?

alandipert12:06:08

@ragge: excellent sleuthing!

mitchelkuijpers13:06:33

Hi I have some problems with converting an leiningen build to boot

mitchelkuijpers13:06:03

I am currently using trapperkeeper, and trapperkeeper apparently expects your src content so be copied in the jar

mitchelkuijpers13:06:42

So I get the follwing exception after I created a uberjar

mitchelkuijpers13:06:47

java.io.FileNotFoundException: Could not locate nl/avisi/xodus/service__init.class or nl/avisi/xodus/service.clj on classpath.

mitchelkuijpers13:06:23

When I inspect the Jar I see this is true? Is there an option to force this?

estsauver14:06:57

@ragge: After I started running off of the master boot branch my uber problem disappeared.

estsauver14:06:17

I then started seeing the “too many files open” bug as well, your patch fixed it for me, thanks!

ragge14:06:09

@alandipert @estsauver I'm out now but will try to tidy up a PR later today

mitchelkuijpers14:06:52

Nvm fixed it -_-

mitchelkuijpers14:06:34

I have added the src/clj folder to the resources 😛

mitchelkuijpers14:06:52

that fixes it, stupid trapperkeeper does not support aot

estsauver14:06:17

(Just trying to minimize the https://xkcd.com/979/ effect simple_smile )

alandipert14:06:59

@mitchelkuijpers: curious, why are you AOT-ing?

mitchelkuijpers14:06:30

I am not anymore, because trapperkeeper does not support it

mitchelkuijpers14:06:44

but the main reason is that the application starts a bit faster

alandipert14:06:17

i ask because i try to spread the good news about the web + war tasks, and detest AOT

mitchelkuijpers14:06:43

Aha ok I don't exactly know what the web task is?

alandipert14:06:35

it allows you to generate servlets without AOT, by generating a web.xml file and putting it on the fileset that's compatible with https://github.com/tailrecursion/clojure-adapter-servlet

alandipert14:06:25

so if you are deploying to a servlet container you can both 1) not have to aot anything and 2) user uber --as-jars (much faster than default uber)

alandipert14:06:57

i have been working on and off on something similar, except for cli

alandipert14:06:27

that provides a proxy -main that runs a build.boot in the jar, so we can distribute boot cli programs with no deps except jvm

alandipert14:06:47

@ragge: re: PR that sounds great, thanks!

mitchelkuijpers14:06:45

Does that mean you could kind of use boot as an integral part of your distributed application?

alandipert14:06:15

well, you can already do that if you're brave enough simple_smile

alandipert14:06:27

more that it would make it possible to distribute standalone jars easily

mitchelkuijpers14:06:16

Hmm might need to look into that

mitchelkuijpers14:06:36

I am currently building the distribution of this application http://dash.avisi.com/ with leiningen

mitchelkuijpers14:06:53

Lol should fix that description of the page

mitchelkuijpers14:06:19

But that is the reason I am looking at boot creating a tar.gz with custom stuff is very narly with leiningen

alandipert14:06:05

that looks really cool!

alandipert14:06:36

i can imagine producing debs and rpms also, with fpm in a jruby pod

mitchelkuijpers14:06:42

Had never seen fpm, thank you sir!

alandipert14:06:32

sure! :thumbsup:

mitchelkuijpers15:06:55

I get some errors with instaparse

mitchelkuijpers15:06:15

I guess the fix for now is making sure boot uses the same clojure version?

mitchelkuijpers15:06:55

Yes that fixed it 😃

ragge15:06:45

@alandipert: was actually considering doing something for deb as that's what we're mostly deploying... we're currently using fpm, but using it from embedded jruby in a pod sounds both a bit wicked, and very exciting. is that something someone is working on?

ragge15:06:43

@alandipert: if not, it sounds like the perfect first proper task for me simple_smile

alandipert15:06:48

@ragge: not that i know of, go for it!

micha15:06:30

it would be awesome if we could ship updates to boot.sh via package manager

ragge15:06:44

@alandipert: just a question re the unpack-jar fn: what's the reason for catching this exception and just printing a warning? https://github.com/boot-clj/boot/blob/master/boot/pod/src/boot/pod.clj#L371

ragge15:06:46

if exception is swallowed, that means a file that was in the jar to be unpacked will not be in the uberjar, which is pretty serious...

ragge15:06:13

i would expect that to fail a task

micha15:06:48

a comment there is probably appropriate

ragge15:06:53

couldn't that be solved using exclude?

ragge15:06:00

if you have a jar file with that issue

micha15:06:01

yes, i think so

micha15:06:16

yeah you're right

ragge15:06:25

fix totally makes sense for that

ragge15:06:35

but it felt weird in my case

ragge15:06:41

just seeing a bunch of errors printed

ragge15:06:48

but task succeeding

micha15:06:56

i mean the only thing you can do is exclude them, right?

micha15:06:05

so here we just do that for you and move on

micha15:06:31

probably does what most users mean

micha15:06:46

but warns so you can do something special if you need to

ragge16:06:35

not sure I agree... as I said, it totally make sense in this case, but defaulting to automatically excluding files from your uberjar when an exception occurred somewhere in that fn...

ragge16:06:11

by this case I mean the case in the issue

ragge16:06:30

other cases might be mine (too many open files), but could be other stuff too

ragge16:06:02

just seemed like a dangerous default... was mostly asking to see if i should remove it in PR, but if you feel that should be the behaviour, I won't change it

micha16:06:48

osx filesystem is case insensitive too, right?

ragge16:06:53

they can be

ragge16:06:00

not by default I eblieve

ragge16:06:13

they are insensitive by default yes

ragge16:06:17

no, sensitve

micha16:06:35

it would be a shame if we make it so that the user always needs to jump through hoops of excluding things to get uber to work

micha16:06:52

like if it's more likely to get exceptions than not i would say just warn

micha16:06:14

but if most cases can work without failing then throwing would be okat

micha16:06:19

s/okat/okay/

micha16:06:58

like the only fix for something that's not working in the uber task is excluding

ragge16:06:28

well, it's only an issue when files do clash like in the issue... and currently boot makes a global decision that it's first-one-wins... how do you know that's what the user wants?

micha16:06:53

we don't, but the user only needs to intervene if it's not what they want

ragge16:06:00

like in the issue, you'd get an empty directory, not the license file, am I right?

micha16:06:03

and in many, maybe even most cases they won't probably care

micha16:06:20

no, they'd get the file i think

micha16:06:30

because the fileset doesn't know about directories

micha16:06:32

it's like git

micha16:06:53

that's just windows refusing to create a file with the same name as the non-empty directory

micha16:06:11

but even in the too-many-files case

micha16:06:36

there isn't any way to fix that other than excluding

micha16:06:59

of course you tell the user that their uber is not going to contain all the files

micha16:06:20

but the uesr can't fix it if excluding something doesn't fix it, right?

micha16:06:35

the too-many-files thing is a bug in boot

ragge16:06:42

well, in this case they could bump filehandles, if that was the case, or fix boot

micha16:06:13

perhaps we differentiate the file exists exception from ioerror or whatever

micha16:06:27

file already exists should i think just warn

ragge16:06:49

too me it just feels weird to not fail the task when an exception occurs like this (and if it's the license issue, let the user handle it, maybe print "if you want to exclude this failing file, add this /LICENSE to :exclude" etc...)

ragge16:06:06

it's like the javac task would succeed even if compilation failed, but just print warnings

micha16:06:22

well userjars are hairy

micha16:06:36

i can imagine getting 50 or 60 file exists warnings

micha16:06:50

am i going to add 50 options to the thing then?

micha16:06:57

that would be really annoying

ragge16:06:02

i don't have any data on that so I don't know

ragge16:06:10

50 is a lot, that would be painful yes

micha16:06:29

uberjar is a packaging thing

micha16:06:41

which makes it a little different than a build task i think

micha16:06:53

like you do it once, and you pay attention, right?

ragge16:06:15

not really

micha16:06:15

i guess that also means that you could take care to have proper exclusions too

ragge16:06:20

i build on ci on every commit

ragge16:06:30

if it started printing warnings but stil suceeding i wouldn't know

ragge16:06:36

until I have an uberjar that doesn't work

micha16:06:47

well your ci could run the uberjar lol

micha16:06:50

but i see your point

ragge16:06:12

to me, it's a task that fails to do what I wanted

ragge16:06:30

sorry to bang on aboutu this simple_smile

ragge16:06:38

I'll just leave it as is for now

ragge16:06:54

I have probably made my opinion known simple_smile

micha16:06:09

haha no worries

micha16:06:22

i think it's fine to throw really

micha16:06:28

just exploring the implications

micha16:06:43

would you consider this a breaking change, or a bugfix?

ragge16:06:18

changing to a throw?

ragge16:06:54

i think it would also be a good idea to get some data on collisions, if possible

ragge16:06:12

i agree the manual exclude solution is less ideal if it's common to have 50 conflicts

micha16:06:25

an additional flag to turn exceptions on is another option to consider

micha16:06:26

the uber task has been problematic from the start so i think it's maybe alright to change things around, because i think a lot of people are annoyed by how it currentlyworks

ragge16:06:39

not sure flag is the right way to go

ragge16:06:47

what other things are annoying people?

micha16:06:50

there are just more tickets about that task than anything else in boot i think

ragge16:06:47

oh ok, might have a look around then to get a better understanding

micha16:06:50

we're converging slowly simple_smile

podviaznikov21:06:08

thanks, will remove them now

podviaznikov21:06:41

@juhoteperi: it’s pretty cool. I’ll try it

martinklepsch21:06:08

@podviaznikov: in some tasks it would be neat if the exceptions give some more information about what file caused it

juhoteperi21:06:10

It's not yet pushed to clojars as I'm waiting for clj-livereload PR to be merged.

martinklepsch21:06:21

@podviaznikov: also is there documentation on specifying the initial metadata in markdown files? Is that just something that comes with markdown-clj?

podviaznikov21:06:41

@martinklepsch: what exceptions do you see?

martinklepsch21:06:05

@podviaznikov: yeah, I figured that out, but I was used as delimiters (jekyll) and surprised about the HTML comments

podviaznikov21:06:06

I think I can add support for jekyll style. I did that parses for my initial implementation 2 years ago and I don’t remember why I did it this way back then

martinklepsch21:06:46

I’m fine with either, was just confused initially, probably something that would be useful in the Readme/task docstring

martinklepsch21:06:50

@podviaznikov: re exceptions: getting them when having html inside markdown blockquotes, worked in kramdown but doesn’t with markdown-clj

podviaznikov21:06:45

I’ll try to reproduce error and see what can be improved for exception handling

podviaznikov21:06:18

also do you know other good markdown libraries in clojure?

martinklepsch21:06:48

No idea about markdown libs, I guess most just wrap the most popular Java lib (whichever that is)

akiva22:06:35

As far as I know, Dmitri’s is the standard: https://github.com/yogthos/markdown-clj

akiva22:06:14

However, in my latest project, I actually call out to the shell for multimarkdown.

micha22:06:04

it emits an AST basically, which you can extend

micha22:06:12

zero "rendering"

micha22:06:23

probably a more useful namespace simple_smile

micha22:06:07

emits markdown as s-expressions

micha22:06:16

you can implement the operations yourself

micha22:06:51

my objective there was to use markdown as a general purpose s-expression markup language

micha22:06:05

so your "markdown" is compiled to a program

micha22:06:50

like i want to be able to reimplement something like <<foo bar>> for instance to be something different than what markdown usually does (the french style of quotation)

micha22:06:09

and in hoplon markdown is a program, so the AST is really what i want anyway

micha22:06:49

basically the functions in that namespace are what you can call from your own macros

micha22:06:11

they generate the lisp expressions corresponding to the markdown