Fork me on GitHub
#boot
<
2015-12-27
>
laforge4902:12:26

@micha Something broke when I did the release of notify.

laforge4902:12:56

When I do a local build, it works. But when notify is downloaded from clojars, it does not.

micha02:12:25

what fails?

laforge4902:12:40

I did just boot build-jar push-release during release

micha02:12:56

yeah that sounds ok

laforge4902:12:20

can't find the notify rpc file

laforge4902:12:30

--gotta recreate, one sec

micha02:12:36

notify/
notify/notification_api.clj
notify/notification_rpc.cljs.hl
META-INF/
META-INF/maven/
META-INF/maven/hoplon/
META-INF/maven/hoplon/notify/
META-INF/maven/hoplon/notify/pom.properties
META-INF/maven/hoplon/notify/pom.xml
META-INF/MANIFEST.MF

laforge4902:12:42

deleted local repository

micha02:12:44

those are the contents of the jar in clojars

laforge4902:12:15

then ran boot dev and it downloads the notify jar and fails with this:

laforge4902:12:30

C:\Users\Bill\Documents\GitHub\demos\castra-simple2>boot dev Retrieving notify-0.0.1.jar from https://clojars.org/repo/ Starting reload server on <ws://localhost:59568> Writing boot_reload.cljs... Starting file watcher (CTRL-C to quit)... Compiling Hoplon pages... ò simple_notifications\index.cljs.hl Adding :require adzerk.boot-reload to index.html.cljs.edn... Compiling ClojureScript... ò index.html.js adzerk.boot_cljs.util.proxy$clojure.lang.ExceptionInfo$ff19274a: No such namespace: notify.notification-rpc, could not locate notify/notification_rpc.cljs, notify/notification_rpc.cljc, or Closure namespace "notify.notification-rpc" data: {:tag :cljs/analysis-error, :from :boot-cljs} clojure.core/ex-info core.clj: 4593 adzerk.boot-cljs.impl/handle-ex impl.clj: 60 adzerk.boot-cljs.impl/compile-cljs impl.clj: 97 ... clojure.core/apply core.clj: 630 boot.pod/eval-fn-call pod.clj: 223 boot.pod/call-in* pod.clj: 230 ... boot.pod/call-in* pod.clj: 233 adzerk.boot-cljs/compile boot_cljs.clj: 71 adzerk.boot-cljs/compile-1/fn boot_cljs.clj: 125 clojure.core/binding-conveyor-fn/fn core.clj: 1916 ... Elapsed time: 9.791 sec

laforge4902:12:02

I'll rebuild locally and look at the working jar...

micha02:12:03

you didn't use the --manifest option to hoplon task when you built the jar, maybe?

micha02:12:15

there is no hoplon/manifest.edn file in that jar

micha02:12:32

look at the contents of the jar

micha02:12:48

there is the notify/notification_rpc.cljs.hl entry

micha02:12:55

in the notify jar from clojars

laforge4903:12:03

ok, still have that

micha03:12:07

but that needs to be in the hoplon manifest

micha03:12:20

because hoplon needs to compile it

micha03:12:43

so when you build the jar you should use the --manifest option to the hoplon task

micha03:12:25

i don't understand why it would work locally though

micha03:12:31

that's puzzling

laforge4903:12:15

locally I have a bunch of things defined in bot.build

laforge4903:12:28

both the pom task options and dev

laforge4903:12:31

again, I did just boot build-jar push-release during release

laforge4903:12:54

so the processes must have differed somehow?

laforge4903:12:31

some magic in dev that is not captured in build-jar push-release?

laforge4903:12:00

@micha are you saying I should have done a boot build-jar --manifest push-release instead?

micha03:12:04

what do you do locally?

laforge4903:12:45

I should have done a boot build-jar --manifest push-release?

micha03:12:50

what you need to do is boot hoplon -m build-jar push-release

laforge4903:12:29

release 0.0.2 coming up!

micha03:12:57

when you put the .hl files in the jar, you need to have the hoplon compiler write the manifest, which is what the hoplon compiler will use to figure out which files from dependency jars will need to be compiled when you use the jar from another project

laforge4903:12:09

can I define this release in my boot.build?

micha03:12:51

you can make a deploy task if you like

laforge4903:12:16

deftask deploy-release, yes

micha03:12:24

(deftask deploy
  []
  (comp
    (hoplon :manifest true)
    (build-jar)
    (push-release)))

micha03:12:56

ah i see now

micha03:12:07

the dev task has the manifest business in there already

micha03:12:16

that's why it was working locally

laforge4903:12:27

many thanks. again!

laforge4903:12:39

--going to release and then test it!

laforge4903:12:33

@micha I did not get too far:

laforge4903:12:36

clojure.lang.ExceptionInfo: java.lang.AssertionError: Assert failed: project repo is not clean (or (not ensure-clean) clean?) data: {:file "C:\\Users\\Bill\\AppData\\Local\\Temp\\boot.user4061139093341627544.clj", :line 29} java.util.concurrent.ExecutionException: java.lang.AssertionError: Assert failed: project repo is not clean (or (not ensure-clean) clean?) java.lang.AssertionError: Assert failed: project repo is not clean (or (not ensure-clean) clean?) boot.task.built-in/fn/fn/fn/fn built_in.clj: 784 adzerk.bootlaces/eval769/fn/fn/fn bootlaces.clj: 54 adzerk.bootlaces/eval808/fn/fn/fn bootlaces.clj: 62 boot.task.built-in/fn/fn/fn/fn built_in.clj: 716 boot.task.built-in/fn/fn/fn/fn built_in.clj: 618 boot.task.built-in/fn/fn/fn/fn built_in.clj: 342 hoplon.boot-hoplon/eval565/fn/fn/fn boot_hoplon.clj: 119 boot.core/run-tasks core.clj: 791 boot.core/boot/fn core.clj: 801 clojure.core/binding-conveyor-fn/fn core.clj: 1916 ... C:\Users\Bill\Documents\GitHub\notify>

laforge4903:12:19

@micha it looks like (hoplon :manifest true) might be putting something in the repository, like

laforge4903:12:26

_maven.repositories

laforge4903:12:37

notify-0.0.2.jar and

laforge4903:12:49

notify-0.0.2.pom.

laforge4903:12:10

and then the next step checks to see that there is nothing in the repository for that release.

laforge4903:12:31

I tried deleting the 0.0.2 folder, but the same thing keeps happening.

micha03:12:00

that's just warning you that you're trying to deploy a release without committing your changes in git

micha03:12:32

if the changes aren't committed then the git commit info in the pom.xml won't be correct

micha03:12:50

if you commit your changes you should be able to push the release

micha03:12:10

it'll also tag the commit for you, so you can push the tags to github too

laforge4903:12:28

manifest.edn now contains ["notify/notification_rpc.cljs.hl"]

laforge4903:12:52

@micha It works! 0.0.2 is a good release.

laforge4903:12:38

So by dropping the v from the version, github's own release mechanism is disabled. Fine by me!

seancorfield05:12:58

How many folks are storing dependencies in EDN files and reading them into build.boot files?

flyboarder05:12:27

@seancorfield: if you are looking to create a task to slurp in that vector I was just looking for that 😛

seancorfield05:12:46

Oh, that was easy... I was just curious as to whether it was common practice...

flyboarder05:12:07

@micha: i thought I had the checkout task working, however upon further inspection im not sure the hoplon compiler is picking up the changed hl file still

seancorfield05:12:08

Now that we're not tied to declarative project.clj files, we can use dynamic Clojure! simple_smile

micha05:12:38

@flyboarder: i will merge here and test

flyboarder05:12:48

might want to keep it out of master for now

seancorfield05:12:10

We have pinned versions of several libraries now (in xyz.version files) and we map over the dependencies and fix the versions of those libraries to the pinned versions. It's really nice.

seancorfield05:12:34

Moving to Boot has been a huge win all round for us, for flexibility.

micha05:12:57

eliminate the build tool, and just program (tm)

micha05:12:04

@flyboarder: for a truly seamless experience we may need to move the checkout task into boot itself

micha05:12:19

so the dependency loading machinery is aware of checkout dependencies

flyboarder06:12:28

@micha: so even with my fix if the symbols are in deps it fails to build with the new files

flyboarder06:12:47

i removed those lines so they are only in checkout and it works

flyboarder06:12:58

im missing something

micha06:12:15

ah i see something

micha06:12:05

should be something more like (->> "hoplon/manifest.edn" pod/resources (filter #(= "jar" (.getProtocol %))))

micha06:12:52

@flyboarder: 0.1.12 is in clojars now

micha06:12:05

but you still need to remove the checkout dep from your :dependencies

micha06:12:13

but at least it works then

flyboarder06:12:43

@micha: still had to remove the symbol manually from the deps list, is it possible they are already loaded by the time checkout runs?

micha06:12:27

there's no way around that unless the underlying dependency stuff in boot is changed

flyboarder06:12:40

i modified checkout to print before and after that line and the deps are being removed, but if they were there prior to being removed new file are not compiled

micha06:12:58

that's the problem

micha06:12:11

it's able to see the old things on the classpath, too

flyboarder06:12:02

but why does that prevent the hoplon compiler from picking up the changed files? I’m confused a bit

micha06:12:20

it's picking up the one with the same path in the jar

flyboarder06:12:20

cljs picks up changes fine?

micha06:12:31

the one that isn't changing

micha06:12:10

actually i should make sure

flyboarder06:12:12

but isnt it?

micha06:12:25

well the one in the jar isn't changing

flyboarder06:12:40

but the jar itself is changing?

micha06:12:41

and hoplon compiler sees the one in the jar and emits a cljs file

flyboarder06:12:55

isnt that what the checkout task is watching for

micha06:12:09

yes the checkout task is dumping the .hl file into the fileset

micha06:12:18

but the classpath can have only one thing with that path

micha06:12:29

now there are two things with the same path in the classpath

micha06:12:41

the .hl file in the jar and also the .hl file that was extracted to the fileset

micha06:12:46

which one will it see?

micha06:12:54

it can only see one of them

micha06:12:27

it must be seeing the one in the jar instead of the one in the fileset

micha06:12:38

so that one will not change

flyboarder06:12:17

so it’s only seeing the one from the first time it saw the file, even tho the jar gets updated and the new file pulled out

micha06:12:09

hm i think i was slightly incorrect there

micha06:12:30

1. dependency jar is loaded, containing manifest file

micha06:12:50

2. checkout task unzips dependency jar into temp dir, including the .hl file

micha06:12:08

3. hoplon task sees manifest file in jar and extracts the .hl file to the fileset

micha06:12:24

(now there are two .hl files with the same path, in different tempdirs)

micha06:12:40

4. the hoplon compiler compiles the .hl file

micha06:12:49

question: which .hl file will it compile?

flyboarder06:12:29

well it’s obviously the first file is what we are seeing

micha06:12:39

the hoplon task overwrote the one from the checkout task, perhaps

micha06:12:08

it's puzzling

micha06:12:26

seems like the checkout task should be clobering the other things

micha06:12:30

which is what we want

micha06:12:19

that's clobbering the file from the checkout

micha06:12:31

with the one hoplon extracted from the jar itself

flyboarder06:12:11

but shouldnt that file be new everytime the jar is recreated?

micha06:12:28

yes it is

micha06:12:38

then hoplon replaces it with the old file again

micha06:12:52

it stashes the .hl file from the jar in the tmp-hl dir

micha06:12:13

each time it comes around in the pipeline, the first thing hoplon does is add those tmp-hl files to the fileset

flyboarder06:12:14

right and then adds that folder to source

flyboarder06:12:47

ok im following with what your saying, how is the extracted folder clobbering the new file? doesnt the extracted folder contain the new file? That line makes sense to me???

flyboarder07:12:19

oh i see what you mean!!!

flyboarder07:12:29

wow that is hard to spot

micha07:12:55

yeah pretty sneaky ofit

seancorfield07:12:26

(you know you can edit a previous message, right?)

micha07:12:57

irc for too long

micha07:12:31

well i have a fix

micha07:12:48

there is a :mergers option to add-source that we can use here to resolve conflicts

micha07:12:03

i can use the boot.pod/first-wins-merger

micha07:12:11

should be a simple fix

micha07:12:38

really though i think the optimal solution involves moving the checkout machinery deeper into boot

micha07:12:48

for it to really work transparently

flyboarder07:12:28

im not sure i see how that helps, just so it knows more about dependencies?

micha07:12:55

it would know to not load that dependency from the jar in the first place

micha07:12:06

and unzip the jar instead

micha07:12:49

it would maybe be cleaner to do what you originally suggested, like have boot treat all dependencies as checkout dependencies when the flag is set

micha07:12:50

it could be quite slow though if it needs to unzip a lot of jars

flyboarder07:12:11

yeah i see that being a slow process for a project which already takes minutes to build

flyboarder07:12:45

and that would require the new (possibly development) version being in your build file

micha07:12:45

i wonder what happens if i modify a jar in place, when it's loaded into a classloader already

micha07:12:03

probably jvm crashes

flyboarder07:12:23

yeah i would imagine it would error on the next call

flyboarder07:12:13

i think it might just be easier to write a task which loads the deps from a file, this allows me to process the prod and dev deps vectors for conflicts before they ever actually get added to the environment

micha07:12:57

ah, it's fixed

jaen08:12:49

@seancorfield: I don't know about "many", but I do it - read-dependencies! and update-deps task here - https://gitlab.com/jaen/clj-cljs-presentation/blob/master/build.boot. Works pretty nice; as a bonus you get updating of dependencies without killing boot for free (most of the time that is, if there's a conflict you still have to restart it, since you can't unload jars).

jaen08:12:06

I've been asking here whether it would be possible to run the whole thing inside a pod, so even restarting on conflicts could be avoided, but so far this doesn't seem to work exactly as I want it

jaen08:12:50

(for example the boot-cljs compile task errors out complaining that files don't exist, which it doesn't do without running things inside a pod)

alandipert14:12:58

@magomimmo: iirc the problem on win7 is that open files can't be deleted, which we must do to manage the filesets. In win10 MS relaxed this restriction

magomimmo14:12:54

@alandipert: thanks. I'll forward it to the mlist

sveri14:12:15

@alandipert: Good to know, maybe you could document that please in the windows section of boots README?

micha15:12:40

@sveri: have you tried with windows 10 btw?

sveri16:12:52

@micha: No, I don't have one at hand. But @seancorfield reported that he tried it on W10 without problems

flyboarder16:12:14

@jaen: exactly what I was looking for!!

jaen16:12:01

@flyboarder: re: dependencies? Nice.

jaen16:12:13

Though keep in mind it doesn't solve all cases, like I described

jaen16:12:26

If you get the yellow warning about deps conflict, you have to restart anyway.

flyboarder16:12:51

yeah, but looking at your example will update-deps run every time a resource file is changed?

jaen16:12:25

Yes, I am aware of that, I think I've fixed that in another project, but I don't have it in a public repo

jaen16:12:11

(defn- modified-files? [before-fileset after-fileset files]
  (->> (fileset-diff @before-fileset after-fileset)
       input-files
       (by-name files)
       not-empty))

(deftask lein-generate
  []

  (let [fs-prev-state (atom nil)]
    (with-pre-wrap fileset
      (when true ; (modified-files? fs-prev-state fileset #{"resources/dependencies.edn"})
        (util/info "Regenerating project clj...\n")
        (generate-lein-project-file! :keep-project true)
        (reset! fs-prev-state fileset))
      fileset)))

(deftask update-deps
  []

  (let [fs-prev-state (atom nil)]
    (with-pre-wrap fileset
      (when true ; (modified-files? fs-prev-state fileset #{"resources/dependencies.edn"})
        (util/info "Dependencies changed, updating...\n")
        (set-env! :dependencies (fn [deps] (read-dependencies!)))
        (reset! fs-prev-state fileset))
      fileset)))

jaen16:12:16

This should be it

jaen16:12:17

I think.

jaen16:12:45

Though I'm unsure why I have modified-files? calls commented out '

flyboarder16:12:04

hahah cool, thanks for that I’ll play around with it!

jaen16:12:17

But in general it should work, I've lifted modified-files? from system.boot where it works.

jaen16:12:21

Let me know if it does.

flyboarder17:12:14

@jaen: it works! I made a slight modification so i can keep it out of my boot file.