This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-09-07
Channels
- # admin-announcements (2)
- # arachne (1)
- # bangalore-clj (2)
- # beginners (39)
- # boot (349)
- # cider (31)
- # clara (2)
- # cljs-dev (9)
- # cljsjs (67)
- # cljsrn (7)
- # clojure (300)
- # clojure-art (4)
- # clojure-greece (11)
- # clojure-hk (3)
- # clojure-israel (1)
- # clojure-italy (17)
- # clojure-japan (1)
- # clojure-russia (33)
- # clojure-sg (2)
- # clojure-spec (41)
- # clojure-uk (86)
- # clojurescript (123)
- # clojurex (3)
- # code-reviews (1)
- # component (6)
- # crypto (1)
- # cursive (36)
- # datomic (32)
- # devcards (3)
- # emacs (11)
- # events (3)
- # funcool (4)
- # luminus (10)
- # om (28)
- # onyx (88)
- # pedestal (2)
- # re-frame (84)
- # reagent (7)
- # ring-swagger (3)
- # specter (33)
- # sql (2)
- # vim (21)
hopefully-not-too-boneheaded question about boot-logservice
: I want to use it with clojure.tools.logging/with-logs
to capture off the stdout
output of a process I don’t control (lives in a closed-source jar) but with-logs
wants me to provide a namespace from which to pretend to log, and my core namespace isn’t visible from within the log-service pod
how can I either make it visible there, or divine the log-service pod’s *ns*
from my core ns so as to pass it uh
“back in” to with-logs
?
@chris_johnson hm, that doesnt' just work? looks like it takes ns as an arg, which is a symbol iirc
No, it yields the kind of ClassNotFoundException you'd see if you tried to run something that you had eval'd in a REPL but not built anywhere
ok, i see the problem now too
There's a problem? I was seriously just hoping for someone smarter and more experienced than me to tell me what I did wrong! 😄
i did (boot (repl :pod "adzerk.boot-logservice"))
and now i'm in the pod poking around
the bug is something in https://github.com/adzerk-oss/boot-logservice/blob/master/src/adzerk/boot_logservice.clj#L39-L50
Let me see if anything helpful shows up in my stack trace
[adzerk.boot_logservice$slf4j_service_factory_factory$reify$reify__1512 write_BANG_ boot_logservice.clj 41]
?
scanned right past it before because I didn’t think I was bug hunting
that's the one
and we see pod-eval/eval so we know it's in the with-eval-in block
(get-logger* ~(str logger-ns))
is the suspicious line
logger-ns is a string there, and we're in the with-eval-in block so the ~ should splice it in
the exception is acting like the generated code is (get-logger* foo.bar)
instead of (get-logger* "foo.bar")
which is what i would expect from a call like (log/with-log foo.bar ...)
I’m actually sort of following this - I spent a lot of time last week poring over this code before I realized that my Hiccup-fu was weak and was making a slightly-incorrect logback.xml
yeah basically it implements a logger for tools.logging purposes, but the implementation defers to stuff running in a pod
yeah, it’s really elegant
thanks 😊
now, if it would work correctly lol
I like it a lot, and not just because it saves me hours of reading (boot (show “-p”))
so, I’m not as experienced as I’d like to be with debugging the internals of a function call going awry - would it be helpful to you if I pastebin’d up the core.clj
where this is all happening?
na, i think i can reproduce the problem OK
thanks though
i have to go shortly but i'll spend a few more minutes on it, if no fruit, perhaps yuo can make a ticket and we can dive in later
I’d be happy to, it will give me practice boiling a problem down to a minimal repro case, for which I would be grateful
sure, that would be handy
thanks
thank you kindly!
i think i fixed it - i pushed 1.2.0, can you give it a try?
https://github.com/adzerk-oss/boot-logservice/commit/c9b5e576cd9d98e613a26815aa53601266ce7fe8 is what i did
i think it's a boot bug but i can't reliably reproduce, will try more tomorrow
@alandipert - indeed, that seems to work! I updated the dependency to 1.2.0
in the minimal-repro project, restarted the REPL, and now I get logging to nrepl-server instead of a stack trace.
I will plug 1.2.0 into my larger project and report back if there are any problems there, but I think you did fix it. 😄
good to hear
Git commands for boot, and workflows! https://github.com/degree9/boot-semgit
Looking for an example on how I should add development-only dependencies - I know that deftask + set-env! can be used to customize an environment, but is that the way to go ? What to people do ?
@pseud usually just adding them with :scope "test"
is a good option as well
https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Scope worth reading about scopes
@martinklepsch is that sufficient ? how does that later work when I publish a jar and such ? (thinking of lein and their profiles, here)
@pseud anything scoped as test would be used to compile code on test directory tree and on the classpath for test goals executed by lein but not packaged in a jar or war.
@pseud it depends on what your goals are 🙂 :scope "test"
dependencies are not transitive so when your project is used as a dependency your development dependencies won't leak
exactly what I needed to know - that seems super. Wow.. To think, lein was a radical simplification of the hell that is ant and maven.. And boot a radical simplification of that.. Everything is relative indeed 😛
when using set-env!
in a task you can avoid them showing up in pom.xml
as test dependencies but this may cause differences in dependency resolution during dev/test compared to prod
@martinklepsch Would a dev task which adds the deps be better?
@dominicm that’s what I looked into, but I can’t quite figure out how to do that. At least I can’t setup the dev environment in one task and then use that task in another
@dominicm it may affect dependency resolution so you need to be aware that there might be different versions running in dev vs. prod
(which is something you usually want to avoid 🙂 )
@martinklepsch Isn't that a problem anyway? I use the show pedantic task as part of my pipeline
@pseud You can put the set-env!
at the top of another task as normal. You need to return identity
if it's the last statement in the task.
> Isn't that a problem anyway? I use the show pedantic task as part of my pipeline It's a problem when you change dependencies (versions or set-env!) as long as they're stable, resolution should be stable as well no?
@martinklepsch No idea if it would be stable if you used the test scope either though.
You would hope, but if it's not pulled in outside the test scope, then the transitive deps would be different there too, no?
@dominicm riiiight... I now think maybe my previous statements were based on the assumption that all transitive deps+versions are listed in pom.xml which isn't the case I think
though in the context of an application that will never be a dependency itself that might be different? (i.e. uberjar contents are determined at build time including effects of :scope "test"
on resolution?)
@martinklepsch What effect does :scope "test"
have on transitive deps during an uberjar?
Are transitive deps of a test scope still included in a non-test scope (e.g. building an uberjar)?
I think the dependencies might not be included but overall dependency resolution is affected
(i.e. they're loaded by maven but might not end up in the artifact)
I'm guessing a lot here so probably we should test this or something
Yeah, I wondered if maven was smart enough to do that. We should absolutely test it.
We could collaborate on a "Dealing with Dependencies in Clojure" guide 😄
@dominicm I'm 99% certain Maven would not include transitive deps in an uberjar (or any packaged artifact) if they were only in the test tree and in test scope.
When I've built Java projects packaged as jar's or war's using maven and checked dependencies in built artifact this has been the case.
@agile_geek But what if your test tree, included a newer/different version than something in your non-test tree. What would happen then?
@dominicm can you list the same dependency twice?
@dominicm I think it's non-deterministic which version would be loaded in test phase but only the version in the 'production' version would be packaged (assuming it's compiled scope and not provided)
I've been caught out other way...compiled production code using a dependency only in test scope and the build runs but the packaged artifact fails with class not found at runtime.
@martinklepsch I'm thinking of a transitive dependency coming from a different dependency, that's only in your test tree. Not two top-level dependencies, one scoped to test, sorry.
@martinklepsch definitely can happen transitively but yes you can list same dependency twice in a pom in maven, most often when parent poms are involved.
I can't remember but think that maven loads the first version it finds...but don't quote me
Different versions of dependencies happen all time in mvn in Java projects.
I spent the last two weeks of my last contract trying to debug this exact problem!
@agile_geek Youch 😞
@dominicm I wouldn't have minded but ran out of time without solving it!
@martinklepsch so I guess a good way to prevent dependencies from your development environment causing your tests to mysteriously pass, but your uberjar to fail, would be to put them in a dev
task.
no way to test locally as problem only happened when deployed in WAS and we didn't have local licenses for Websphere and could only deploy twice a day to a test environment. Ended up removing about 60% of the dependencies as they were not required. Copy and paste dependency management when project started!
@agile_geek Doesn't sound very agile 😉 where was your influence? 😛
> so I guess a good way to prevent dependencies from your development environment causing your tests to mysteriously pass, but your uberjar to fail, would be to put them in a dev
task.
but in this case won't you have a different set of deps for dev and prod potentially causing trouble?
@dominicm main reason for not renewing contract.
@agile_geek have you seen the way @stuarthalloway et al handle dependencies?
@martinklepsch They're different with the test scope anyway. Or that's what I thought @agile_geek said 😛
@martinklepsch I think I watched a talk @stuarthalloway did on this.
@dominicm are you coming to ClojuTRE btw?
@martinklepsch @dominicm test scope will load dependencies in app class loader for test and non test scopes but only package compile scoped to jars so diff versions of same jars would be loaded non-deterministically on class path.
@martinklepsch Not this year unfortunately.
@agile_geek I'm afraid I still don't fully understand that. Can I try with an example, and ask you questions about it? My app (APP) depends on A and B (B is in test scope) A depends on B v1 B from APP is v2 (in test scope)
Trying again: 1) When I run my tests, would B be v2, because it is declared top level explicitly? 2) When I build my uberjar, would B be v1, because the top level B is in test scope?
@dominicm sorry, crappy coffee shop wifi dropped!
If B dependency is in both test scope and compile scope (transitively) then I suspect the version of B you would get in test phase would be undetermined. However, the version of B packaged would be that transitively referenced by library in compile scope.
If B is explicitly in both test and compile scope at different versions then same I think the version you'd run against in test scope would be whichever was last loaded (not sure which that would be) but only the version referenced in 'compiled' scope would be packaged (not the version test scoped)
@agile_geek I thought so, I just wanted to confirm my understanding. This has somewhat scary implications for development tools right?
Right
Like I said just 2 weeks ago was trying to untangle dependency hell - WAR packaged and run in Jetty worked, same artifact deployed in Websphere App Server on same JVM version failed with missing class!
The class missing is present in rt.jar which is in the bootstrap class loader (highest priority and MUST be on classpath)!
Never got to the bottom of it but some weirdness in WAS classloader order and transitive deps in pom (must have been in compile or provided scope as the 'packaged' artifact ran in one env and failed in other)
@alexyakushev https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Scope any help?
@dominicm: Thank you. This enumerates the scopes, which is already nice; but it is still not clear which scopes Boot applies during which tasks.
@alexyakushev I'd be interested to know too!
@alexyakushev @dominicm I guess to understand that you'd need a mapping of boot tasks to maven goals and phases used by each task.
@martinklepsch Boot pulls in :scope "test"
dependencies into your uberjar!
Have a small problem, if I want a fixed nrepl port - cljs-repl has an example like so (cljs-repl {:nrepl-opts {:port 9000}}) — however, sticking that in a composition won’t work, something about attempting to convert a persistentarraymap to a charsequence.
That example looks wrong now I'm looking at it. Try:
(cljs-repl :nrepl-opts {:port 9000})
@dominicm @alexyakushev boot doesn't have any knowledge of scopes itself
scopes are respected during dependency resolution by the Aether library, which is what boot uses to fetch maven deps
basically it boils down to three scopes that you care about: compile, test, and possibly provided
when you make a pom from your project, which is needed when you upload a jar to a maven repo, the scopes are annotated in its dependencies section
dominicm: you’re right about that, get on it, will ya ? 😛 Just kidding, I’ll look into it
the sole purpose of the scopes as far as boot is concerned are to establish a separation of transitive and nontransitive dependencies
the dependencies in your pom that are marked test or provided scope will not be visible to aether when another project uses your project as a dependency
By default, entries from dependencies with the following scopes will be
copied to the fileset: compile, runtime, and provided. The --include-scope
and --exclude-scope options may be used to add or remove scope(s) from this
set.
(set-env!
:source-paths #{"src"}
:dependencies '[[org.clojure/clojure "1.8.0"]
[enlive "1.1.2" :scope "test"]])
(deftask build
"Builds an uberjar of this project that can be run with java -jar"
[]
(comp
(aot :namespace #{'app.main})
(pom :project 'app
:version "1.0.0")
(uber)
(jar :main 'app.main)
(sift :include #{#"project.jar"})
(target)))
This compiles, against a namespace that requires in enlive. I don't think it should:
;; src/app/main.clj
(ns app.main
(:gen-class)
(:require [net.cgrand.reload :as reload]))
(defn foo
[]
(println reload/auto-reload))
(defn -main
[& args]
(foo))
yeah, the AOT is happening at compile-time in the presence of the lib, which creates .class files, which end up in your uberjar
but then you get runtime exceptions if your code in the uberjar really does depend on the dependencies with test scope
when i do aot for a jar, the namespace i AOT compile has no compile-time references to any other namespace
oh yeah, lemme fix that up with a java shim
PRs neglecte^D err accepted
@micha Is this why many projects do some kind of funkage with eval in their (-main)
?
usually like
(ns app.main-class
(:gen-class))
(defn -main [& args]
(require 'app.main)
(apply (resolve 'app.main/-main) args))
it's AOT compiling the minimum possible thing that will still work with the java classloader
imho it's worse than pointless to AOT compile vast swaths of clojure namespaces just to get a class that can be used as the Main of a jar 🙂
well if you're on that train then you need to be aware that you're tinkering in perfland
Fair enough. I don't really care that much about the performance personally. I just thought it was a free feature toggle.
"free feature toggle" 😂
So AOT exists solely for making a main class, and Clojure projects don't want AOT at their dependencies. I need to make a patch to edge.
Aren't their different jars for clojure, one AOTd the other not?
so top level forms must be wrapped in delay
etc when they would perform side effects at compile time
So if you avoided top level forms, you'd be okay though? Or are there innumerable problems. Yeah, I think that's a general JVM problem, I've seen jars which have had complaints about wanting compilation to go back (compiled for 1.8, they used 1.7, or whatever)
So, back in my days fiddling with java, I compiled a jar or two, and you could specify the JDK to target.
yeah there are ways to tell jave to cross compile, but i couldn't figure out how to do it reliably
sudo: required
services:
- docker
before_install:
- "make -C docker build-travis"
script:
- "./run-heimdallr-docker-noninteractive"
Compatible with dockerfiles as well. CoreOS created an independent standard (AppC) that others can use to build images, and anything that understands AppC can run them.
@dominicm i updated the example to use a java shim, no aot https://github.com/adzerk-oss/boot-uberjar-example/blob/master/src/main/Main.java
an alternative to what micha proposed, which is to basically write the same thing in clj and AOT just it
@dominicm in Java you would usually control the source and target jvm versions in maven (or ant) so you don't get differences across development machines but the final build would usually be on a consistent env like a CI server (server farm) - https://maven.apache.org/plugins/maven-compiler-plugin/examples/set-compiler-source-and-target.html
@alandipert we should make a main task in boot built_in perhaps
@micha it would be worht adding if it sets up the boot runtime
then we can uberboot
yeah, i was gonna say jar
yeah, that's what we're contemplating
just, as a flag to an existing task
i forgot about this: https://github.com/adzerk-oss/bootlet
@alandipert Your boot-uberjar example, puts the source code in :resource-paths
, I can't seem to get the old version of yours to bring in other .clj files, if I use :source-paths
, that seems like a mistake?
source-paths don't make it to jars
is that what you're seeing?
it works if you AOT them because the resulting .class files do
so yeah, if you're not aot-ing, should put them in resource-paths
well a 'source' in boot lingo is a file without an 'output' role, ie it is destined to be consumed/transformed by a task
https://github.com/boot-clj/boot/wiki/Filesets#fileset-components this table makes way more sense now
RE: AOT. Clojure libraries are not meant to be AOT-compiled, but final applications can be AOT-compiled. I AOT all of my apps in production.
It causes problems here and there, but mostly with things that are sloppy and dangerously written anyway.
@alexyakushev what is the benefit of AOT in your applications? are you making desktop applications?
I’m having an issue with boot-cljs-repl
let’s say I set up a cljs-repl, call start-repl
then (require ‘my-project.ns) (in-ns ‘my-project.ns)
however there don’t seem to be any definitions there?!
say I have a (def foo 42)
in that namespace, when evaluating foo
at the CLJS REPL I get
#object[TypeError TypeError: Cannot read property ‘foo' of undefined]
hrm, solved it by adding an explicit require
in my main.cljs.edn
@martinklepsch reproduced the sift
thingy again
without watch
it seems to work
don't delete the cache now 😛
have you tried the two placements (before/after) watch task?
yeah, this happened with sift
before the watch
task
and does it happen with after as well? Any chance you can create a repro?
@martinklepsch hrm, I’ll try to create a repro, but can’t do it today though
and yeah, it happens with the watch
task after too, that’s how I reproed the first time
Anyone sitting on a good cljs project in using boot which builds an uberjar (ideally which is pushed to clojars)
usually you don't push uberjars to clojars, what are you trying to achieve generally?
I admittedly have very little knowledge of what I’m doing (well, aside from writing code, hopefully)
@martinklepsch but what I’m trying to do, in a nutshell, is to convert one project of mine from lein to boot for which I have a (über?)jar up on clojars (via “lein deploy”…)
So I’m basically trying to make everything build, create a (über?)jar, deploy the thing and hopefully stop using lein for that project.
@pseud is it a library others consume via maven?
@alexyakushev That's good to know. I might try AOT'ing everything, especially if I can figure out a way to keep out the test deps from the AOT
and a follow-up question, why is it I can’t seem to find any info on compiling cljc files as part of a boot-based cljs project ?
@pseud They don't need compiling really 🙂. They're just treated as normal clojure files.
@pseud so an uberjar usually refers to a jar that includes all dependencies — is that what you want?
well, my boot project right now complains they’re not on the classpath during compilation. Of course, I freely admit to knowing very little. But it seems to me that it’s helpful to have some sort of quick-start project examples because if there’s anything I’ve learned from the many languages I’ve played with, it is that oftentimes those who write code are oblivious to packaging etc and thus there’s less code shared than there could be 😞
It's hard to know what leiningen is actually doing on your behalf sometimes, did you go out of your way to make it an uberjar that gets pushed?
@dominicm nope - I really admit I have no clue. “lein deploy” pushed to clojars, I could use my lib in another project, I was a happy camper
any chance you can paste the project.clj
?
yes, seeing that we can probably infer what you're trying to reproduce
OK - keep in mind I’m forced to go through the motions here faster than I’d like (need to share such that some colleagues can start playing with it) - that’s why code might be rushed and especially why the project files and such might be messy. That said: https://github.com/readux/readux And this is what I’m using with boot right now (~1hr into it): https://gist.github.com/jwdevantier/093af02cc1d65994c34d4c45ba33b7a3
@alandipert https://github.com/adzerk-oss/boot-cljs-example/blob/master/build.boot should this be using :source-paths
for the cljs? As they're an input with an output?
@pseud so you don't need uber
and to push the jar to clojars try (push :repo "clojars")
also aot
and target
are not necessary
@martinklepsch but if I leave off target I don’t really know where the output file goes ?
(And I’m assuming then the aot(ahead-of-time) only applies to actually compiling clojure code proper to what.. java bytecode to avoid being interpreted at first?)
@dominicm well with lein I had the ability to do “lein install” a few times over to use a local test project just to see that I hadn’t royally screwed everything up 😉 I’d like that option still, if possible
try this
(deftask build
"Build a jar file"
[]
(comp (pom) (jar) (install)))
and once you want to push to clojars just do
boot build push --repo clojars
@dominicm re: source/resource in cljs example, i think yes. the distinction usually boils down to that of application vs. library
@dominicm applications don't care if cljs ends up in artifact, but that's the purpose of libraries and they must include them in jar
@alandipert Oh I see. I hadn't considered that.
@alandipert I guess that means that @pseud should do: :resource-paths #{"src/cljs" "src/cljc"}
then?
@dominicm seems like it, if I stuff everything in :source-paths the files are simply not included at all. However, if I use :resource-paths then the files are included in the jar.
(Oh, and I should be said, I started digging into that when I got errors from using my newly minted jar in a local project - w/o using :resource-paths I would get errors pertaining to missing files)
Wondering if anyone uses checkout dependencies. I’ve uploaded a small-ish example and annotated in the readme how I’ve put the tasks together, what stumps me is that whether using these, or the verbatim (well, sans artifact-id & version numbers) instructions of “boot checkout -h” I get an error like "ERROR: No such namespace: readux-0.1.2-SNAPSHOT.readux.core”. Anyone used checkout dependencies successfully ? Anyone got any public projects to share ?
@pseud You just add :checkouts
to your boot environment with the value being a vector of the dependencies you want checked out.
Then in the checkout project you can do something like this:
$ boot watch pom jar install
@pseud don't forget the '
in font of that vector
If you want to reload a file in the REPL from foo/bar you just (require 'foo.bar.core :reload)
assuming that’s correct - how’d you figure out that adding a “:checkouts” key to the env of the dependent project would do the trick ? I mean, I looked at “boot checkout -h"
> Checkout dependencies task. DEPRECATED
That task is deprecated but probably that could be stated more prominently
it has moved to the global env which is what @kenny described
Ah.. Snap. yea.. Well, in my defence (and indeed the output from “boot -h checkout” 😛 ), I’m tired as hell. It’s probably clear enough.
no worries, after a good night of sleep it'll be clear as day 🙂