Fork me on GitHub
#boot
<
2015-10-15
>
cigitia02:10:23

Let’s say that I’m unit testing, and in those tests I need to write temporary test files, like SQLite database files. However, if I want to run the tests using Boot and adzerk/boot-test, there’s a problem—there seems to be no way to pass a Boot temporary directory to the unit tests. This would mean that the unit tests have to write the test files into the actual project folder, which I know is strongly discouraged practice in usage of Boot. Is there a better way?

martinklepsch09:10:01

@asolovyov: re detecting what changes the cljs version: boot show -d

martinklepsch09:10:51

@chrisn: probably best to paste the task and options you’re using to S3

asolovyov09:10:07

@martinklepsch: when it changes in runtime, show -d doesn't help at all simple_smile

asolovyov09:10:19

and boot-cljs-repl did that in runtime simple_smile

martinklepsch09:10:51

@asolovyov: that’s true. most of the times though the culprit is in the list of static deps

martinklepsch09:10:03

@asolovyov: what version of boot-cljs-repl was that?

martinklepsch09:10:10

@asolovyov: huh, that makes sense. Never encountered this before somehow

chrisn17:10:08

@martinklepsch: I figured out what is going on. I assumed get-env pulled from the actual environment of the process and it does not. Can I make a library available to my boot.build script without having it included in the aot command as a dependency? Also, aot does not appear to respect file times in its default config.

chrisn17:10:29

I guess more specifically how does any boot script specify its dependencies?

chrisn17:10:00

These would be independent of the dependencies of its targets

chrisn17:10:11

Does boot aot itself? For a command line tool it would be great if its startup time was minimized as much as possible.

alandipert17:10:24

@chrisn: you can pull a dependency without implying it should be transitive with (set-env! :dependencies '[[foo/bar "1.0.0" :scope "test"]])

alandipert17:10:22

also, the boot core is aot'd

chrisn17:10:59

OK, so in this use case transitive means that that dependency will not be carried into child functions or pods, so if I call (jar) that dependency will be filtered.

chrisn18:10:12

Hmm. So does that mean I can only use that dependency at the top level of the file or can I use it in a task defined in the file?

alandipert18:10:51

well, pods do not necessarily inherit the dependencies of the environment in which they were created

alandipert18:10:22

the ":scope test" business applies only to the creation of pom.xml... it would be a way to introduce a dependency that should not show up in the packaged artifact, if the project is a library

alandipert18:10:58

which is not to say you're wrong... (jar) would be a part of that packaging process, but the manifest that goes into the jar, the pom.xml, would be created via (pom)

alandipert18:10:40

@chrisn: which is all not very clear 😞 if that doesn't make sense, happy to answer more q's

alandipert18:10:01

boot embraces the maven stuff, but unfortunately there's some overlap between java/maven packaging things and boot's pod model

chrisn19:10:42

OK, so the next problem I have is that aot takes a long time.

chrisn19:10:01

Which doesn't make sense when I have 100 lines of clojure sitting on a bunch of jars that don't change.

chrisn19:10:33

This would actually be a strong argument to switch from leiningen for us as that uberjar build time hurts our turn around time for things.

alandipert19:10:09

yeah, the problem there isn't aot, it's fileset overhead... 'uber' causes a lot of things to go on the fileset

alandipert19:10:21

where are you deploying?

alandipert19:10:34

to a servlet container?

chrisn19:10:59

docker for now

chrisn19:10:23

In general I am trying to figure out a sane way to manage clojure code.

chrisn19:10:00

aot all is pretty slow and I think it is because the clojure compile function doesn't check if it has already compiled something in a sane way because it works at the namespace level not at the jar level.

chrisn19:10:13

but there isn't a strong argument to aot all

alandipert19:10:23

yeah, we deploy to docker... not needing to aot is great

chrisn19:10:08

I would like a simple compilation of every clj file meaning it would be cool if I could guarantee all clj files in the project actually do compile

alandipert19:10:32

yeah, that's doable in a task with the require function

alandipert19:10:22

e.g. (require 'some.appns) in your build.boot somewhere

chrisn19:10:24

So do you need to aot all your docker stuff? I think we don't because startup time is pretty irrelevant to us.

chrisn19:10:28

Good point about require.

alandipert19:10:37

i try to avoid aot as much as possible

alandipert19:10:51

and no, i don't do it with docker

chrisn19:10:02

interesting.

alandipert19:10:33

we have the 'web' task in boot... which generates java stubs for you, so you don't need to AOT clojure

alandipert19:10:12

for CLI things i made https://github.com/adzerk-oss/boot-ubermain which stubs your main function

chrisn20:10:30

Actually, where docker is concerned I am pretty sure uberjar is a mistake.

chrisn20:10:57

If something small changes then the docker image uploads the entire uberjar because I believe docker works on a per-file basis with no ability to diff binary files.

chrisn20:10:27

If you didn't uberjar but just shoved all the dependencies into the docker image along with the jar then most of the time when you publish your docker it would be almost instance.

chrisn20:10:41

I haven't tested this assumption.

chrisn20:10:57

Right, so my first nontrivial thing with boot is to write a task that takes the set of dependencies and writes them all into a directory.

chrisn20:10:15

This is complicated by the fact that one of our dependencies is opencv and thus deals with native library issues.

chrisn20:10:59

uberjar as-jars is damn close.

chrisn20:10:05

Or perhaps there.

chrisn21:10:09

There except for native dependencies.

chrisn22:10:55

so...here is the code in leiningen that deals with native dependencies:

(defn extract-native-deps [files native-path native-prefixes]
  (doseq [file files
          :let [native-prefix (get native-prefixes file "native/")
                jar (try (JarFile. file)
                      (catch Exception e
                        (throw (Exception. (format "Problem opening jar %s" file) e))))]
          entry (enumeration-seq (.entries jar))
          :when (.startsWith (.getName entry) native-prefix)]
    (let [f (io/file native-path (subs (.getName entry) (count native-prefix)))]
      (if (.isDirectory entry)
        (.mkdirs f)
        (do (.mkdirs (.getParentFile f))
            (io/copy (.getInputStream jar entry) f))))))

(defn resolve-dependencies
  "Delegate dependencies to pomegranate. This will ensure they are
  downloaded into ~/.m2/repository and that native components of
  dependencies have been extracted to :native-path. If :add-classpath?
  is logically true, will add the resolved dependencies to Leiningen's
  classpath.

  Returns a seq of the dependencies' files."
  [dependencies-key {:keys [repositories native-path] :as project} & rest]
  (let [dependencies-tree (apply get-dependencies dependencies-key project rest)
        jars (->> dependencies-tree
                  (aether/dependency-files)
                  (filter #(re-find #"\.(jar|zip)$" (.getName %))))
        native-prefixes (get-native-prefixes (get project dependencies-key)
                                             dependencies-tree)]
    (when-not (= :plugins dependencies-key)
      (or (when-stale :extract-native [dependencies-key] project
                      extract-native-deps jars native-path native-prefixes)
          ;; Always extract native deps from SNAPSHOT jars.
          (extract-native-deps (filter #(re-find #"SNAPSHOT" (.getName %)) jars)
                               native-path
                               native-prefixes)))
    jars))

joelkuiper22:10:00

eehm, I have a weird problem. I’m on a OS X El Capitan hackintosh, just did a fresh install with brew install boot-clj

joelkuiper22:10:36

Exception in thread "main" org.sonatype.aether.resolution.DependencyResolutionException: Failed to collect dependencies for [#<Dependency org.clojure:clojure:jar:1.7.0 (compile)> #<Dependency boot:boot:jar:RELEASE (compile)>]
	at org.sonatype.aether.impl.internal.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:371)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
….
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:992)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
	at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:747)
	at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
	at org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:131)
	at org.apache.http.impl.io.AbstractSessionOutputBuffer.flush(AbstractSessionOutputBuffer.java:138)
	at org.apache.http.impl.AbstractHttpClientConnection.doFlush(AbstractHttpClientConnection.java:270)
	at org.apache.http.impl.AbstractHttpClientConnection.flush(AbstractHttpClientConnection.java:275)
	at org.apache.http.impl.conn.AbstractClientConnAdapter.flush(AbstractClientConnAdapter.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:258)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:645)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:464)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
	at org.apache.maven.wagon.shared.http4.AbstractHttpClientWagon.execute(AbstractHttpClientWagon.java:674)
	at org.apache.maven.wagon.shared.http4.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:793)
	... 8 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
	at sun.security.ssl.InputRecord.read(InputRecord.java:505)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
	... 24 more

chrisn22:10:06

what version of java are you running?

chrisn22:10:14

@joelkuiper: ?

joelkuiper22:10:45

~/Repositories » java -version
java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
~/Repositories » 
Installed the JRE from the Oracle site

joelkuiper22:10:08

same with lein, just checked

chrisn22:10:18

happens every time?

chrisn22:10:25

Oh, same error?

joelkuiper22:10:27

internet connection is fine

joelkuiper22:10:24

OS X unfortunately doesn’t have that command

micha22:10:01

seems like a similar problem though

chrisn23:10:11

So, where in the fileset system does out-of-date come into play?

joelkuiper23:10:13

hmm different issue it seems

joelkuiper23:10:22

I tried copying over some certs from my macbook, but no avail

joelkuiper23:10:42

looks like it was a poorly timed maven central issue

micha23:10:32

ah, well good that it works now 🍻