Fork me on GitHub
#tools-build
<
2021-10-22
>
dvingo16:10:46

fairly minor dev-time thing, I'm curious what other people's experiences are: as I understand things build.clj must exist in the top level of the repo. the top-level directory of a repo is usually not on the classpath so when developing/iterating the commands in build.clj I cannot use a REPL as I normally would. Does anyone else run into this and how do you deal? (What I'm doing now is just developing "blindly" - using edit, run from command-line, loop).

Alex Miller (Clojure team)16:10:35

You can put it anywhere you like

Alex Miller (Clojure team)16:10:12

It's just a namespace resolved from the paths, which defaults to :paths [“.”]

Alex Miller (Clojure team)16:10:38

You can rely on that root and put it in a namespace, or change the paths, or both

Alex Miller (Clojure team)16:10:29

There is a gap here with making it easier to get a -T esque classpath and I am going to work on that at some point

Alex Miller (Clojure team)16:10:13

As a workaround you can basically use :replace-deps and :replace-paths to simulate the same thing to start a build repl

seancorfield16:10:59

@danvingo When I'm developing a build.clj script, I start a REPL with -A:build to get the dependencies, open up build.clj and do whatever is your editor's "load file" command. In most editors, a file doesn't need to be on the classpath as long as all the namespaces and libraries it depends on are on the classpath (or already loaded).

seancorfield16:10:56

You can always load an arbitrary Clojure file into the REPL, as long as its dependencies are accessible.

dvingo17:10:09

At least in cursive, if you namespace isn't on the classpath it cannot be loaded into the REPL

seancorfield17:10:21

Really? That's weird. Clojure has a load-file function built-in that lets you read in an arbitrary file -- that's what all the editors I've used do.

seancorfield17:10:59

But then I hear of all sorts of other weird restrictions with Cursive... It doesn't work well with Polylith because it doesn't support :local/root deps properly (I think that's the issue?) and it doesn't currently support the whole tools/build thing very well (again, because of classpath issues).

seancorfield17:10:51

I think most of those are due to IntelliJ's model of the world being very Java-centric, based on what I've seen Colin say about his frustrations with trying to support this stuff.

dvingo17:10:03

aha yea I was referring to cursive's "load current namespace in REPL" feature. For sure, you could use load-file if you wanted. - This is all about ergonomics so wanted the convenient thing to be available. That sounds about right re IntelliJ issues

dvingo17:10:40

Thanks for the info - I haven't done the requisite reading up on the -T/tools feature in depth yet so I think that will clarify things for me.

AustinP20:10:22

What is the correct way to pass :java-opts along to an uberjar build? Ive got the java-command fn and :java-opts being passed into the uber task, tried to add :cp with the file path as well. Im trying to pass a production config edn file into the uberjar process to load prod specific vars. We use :jvm-opts ["-Dconf=dev-config.edn"] in the :run alias for dev mode, just trying to reflect that for prod in our uberjars. What am I missing here? Do I need a compile step before running uber to utilize the :java-opts correctly? I put a prod-config.edn in both env/prod/resources and at the top project level next to build.clj . I am using @seancorfield’s build-clj lib. Currently my uberjar function looks like this:

(defn uber "Build the uberjar." [opts]
      (-> opts
          (assoc :lib lib :main main
                 :command-args (b/java-command {:basis basis
                                                :main main
                                                :cp ["env/prod/resources/"]
                                                :java-opts ["-Dconf=./prod-config.edn"]}))
          (bb/clean)
          (bb/uber)))

dpsutton20:10:02

bb/uber doesn't start a new process so there is are no jvm options to it.

Alex Miller (Clojure team)21:10:04

what code is supposed to be affected by these jvm-opts?

Alex Miller (Clojure team)21:10:59

you can't "set" jvm opts in an uberjar - that has to be done on the java command line that uses it

AustinP21:10:06

Were using the cprop lib thats loading some system configs in at startup. It looks for a file at compile time to load from, and fails to compile without it. Hence trying to pass it into the process at compile time, and not just after the uberjar is built when I run the jar. Since its failing to build the jar when it cant find a config file to load.

dorab21:10:47

If you do end up looking at another config library, I'd recommend looking at juxt/aero. They recommend just one config file that contains both dev and prod (and any other) profiles. I've transitioned from cprop to aero myself.

AustinP21:10:15

Will definitely look into that! Thanks! In the market for any lib to handle various configs throughout our system. Were using a monorepo with Polylith as well which makes things an extra layer of interesting.

dorab21:10:09

I have surveyed config libs three times so far. First time, I ended up with environ, the next time with cprop and now have been happy with aero for a few years now. perhaps the progression reflects the sophistication of the projects I've been working on at the times I did the surveys.

👍 1
AustinP21:10:30

Might just switch to a different lib for dev/prod config loading, or modify the way cprop is handling this so it can at least compile without a config. But was curious if there was a way to handle this current case.

dpsutton21:10:08

could you put :jvm-opts ["-Dcconf=./prod-config.edn"] in your build alias?

AustinP21:10:09

Just gave that a shot but no luck. Wasnt sure how/if :jvm-opts was being utilized by tools.build

AustinP21:10:26

Its okay if there isnt a solution for this the way we've got it set up, I really just wanted to make sure I wasnt missing something obvious in the usage of tools.build. I understand why/how it doesnt function this way with uberjars. Seems like we need to look into other config libs, or at least re-evaluate our usage of cprop

seancorfield21:10:10

Blocked by TBUILD-18 in tools.build itself.

hiredman21:10:28

sounds like grossness on the use of cprop, it recommends a top level side effecting form (load-config) which of course means it tries to do stuff when loading the code for compilation

seancorfield21:10:32

Malli is another library that relies on a JVM property at compile-time 😐

AustinP21:10:22

yep @hiredman, thats exactly where were hitting this wall.

hiredman21:10:23

if you replace (load-config) with (delay (load-config)) and wrap a (force ...) where needed then you'll at least side step that

AustinP21:10:41

Thanks for the info @seancorfield! Sounds like it might be taken care of in the future. Might look into the juxt/aero lib, or try @hiredman’s suggestion in the meantime

hiredman21:10:47

(assuming of course you don't have other top level side effects that would force the config)

sun-one22:10:35

For Mono repos with local/roots and git dependencies how do you set it up too compile to an uberjar with .class files for those deps? Currently it appears to just be moving the source code of these dependencies into the uberjar but not compiling dependencies to java byte code? The issue I'm hitting (and maybe theirs a work around) is running java -jar <my_uberjar> fails with java.lang.ClassNotFoundException for my local dependency presumably because the libs source not java bytecode is in the jar?

hiredman23:10:46

that is generally not a safe assumption

hiredman23:10:12

a class not found exception can happen for a number of reasons, and clojure source can be loaded just fine from jars

hiredman23:10:58

clojure's aot compilation is transitive, so if you are are aot compiling your source, anything it loads will also be aot compiled

hiredman23:10:16

the best thing to do might be to start from the class not found exception, does that class exist in the uberjar, and what is on the stack that is triggering the class not found exception

hiredman23:10:35

and what kind of construct is supposed to generate the class

hiredman23:10:47

are you expecting java source to be compiled?

sun-one23:10:13

The error I was getting was Caused by: java.lang.ClassNotFoundException: gen_fhi.fhir_client.protocols.Client When I ran jar tf target/workspace-0.0.313-standalone.jar | grep gen_fhi/fhir_client I can see the libs source is included but not a .class file (it's referenced as a local dependency in my deps.edn where I'm running tools.build

$ jar tf target/workspace-0.0.313-standalone.jar | grep gen_fhi/fhir_client
gen_fhi/fhir_client/
gen_fhi/fhir_client/utils.cljc
gen_fhi/fhir_client/protocols.cljc
So the clj source is in the uberjar but no .class file.

hiredman23:10:05

you are using the interface generated by the protocol somewhere, but not actually loading the code that creates the protocol

sun-one23:10:41

It's referenced as local dep in my edn file where I'm using it.

sun-one23:10:54

` gen-fhi/fhir-client {:local/root "../cljc/fhir_client"}

hiredman23:10:06

nah, I mean in your source code

hiredman23:10:28

somewhere you using gen_fhi.fhir_client.protocols.Client (the java interface that is generated from the protocol definition) without requiring gen-fhi.fhir-client.protocols (the clojure namespace that defines the protocol)

sun-one23:10:39

In the stack trace it's coming from

at clojure.lang.RT.classForName(RT.java:2212)
	at clojure.lang.RT.classForName(RT.java:2221)
	at gen_fhi.workspace.db.postgres.core$fn__52985.<clinit>(core.clj:41)
	at gen_fhi.workspace.db.postgres.core__init.load(Unknown Source)
	at gen_fhi.workspace.db.postgres.core__init.<clinit>(Unknown Source)
Which I am requiring it
(ns gen-fhi.workspace.db.postgres.core
  (:require [integrant.core :as ig]
            [next.jdbc :as jdbc]
            [hugsql.core :as hugsql]
            [clj-json-patch.core :as patch]
            [clojure.core.match :refer [match]]

            [gen-fhi.operation-outcome.core :as oo]
            [gen-fhi.workspace.db.postgres.extensions]
            [gen-fhi.workspace.ws.protocols :refer [WSDataHandler get-patches-after]]
            [gen-fhi.fhir-client.protocols :refer [Client] :as fdb]))

hiredman23:10:50

what is line 41?

hiredman23:10:43

how is gen-fhi.workspace.db.postgres.core being required?

hiredman23:10:09

is there a class file for gen-fhi.workspace.db.postgres.core in your uberjar?

sun-one23:10:36

Line 41 is a long defrecord where I use the protocol

(defrecord PGFHIRDatabase [source]
  WSDataHandler
  (get-patches-after [{:keys [source] :as this} {:keys [workspace] :as ctx}]
    (jdbc/with-transaction [tx source]
      (let [res (get-latest-version-id tx {:workspace workspace})]
        (get-patches-after this ctx (- (:version_id res) 1)))))
  (get-patches-after [{:keys [source]} {:keys [workspace]} version-id]
    (let [version-id (if (string? version-id) (Integer/parseInt version-id) version-id)]
      (get-patches-and-prev-version source {:workspace  workspace
                                            :version-id version-id
                                            :limit      10})))

  Client
  (metadata    [this ctx]
    (throw (ex-message "Not Implemented")))
  (invoke      [this ctx op parameters]
    (throw (ex-message "Not Implemented")))
  (invoke      [this ctx type op parameters]
    (throw (ex-message "Not Implemented")))
  (invoke      [this ctx type id op parameters]
    (throw (ex-message "Not Implemented")))

  (search [{:keys [source]} {:keys [workspace]} parameters]
    (let [res (search-system source {:workspace workspace
                                     :cols ["resource"]
                                     :where (when-let [where-clause (parameters->where-clause parameters)]
                                              (where-snip
                                               where-clause))})]

hiredman23:10:51

is gen-fhi.workspace.db.postgres.core being loaded possibly as part of some kind of plugin system?

sun-one23:10:08

It shouldn't be. When I look at the jar it has .class files for it

gen_fhi/workspace/db/postgres/core$fn__53038.class
gen_fhi/workspace/db/postgres/extensions$__GT_pgobject.class
gen_fhi/workspace/db/postgres/core$fn__53034.class
gen_fhi/workspace/db/postgres/core$loading__6737__auto____52859.class
gen_fhi/workspace/db/postgres/core$eval52915.class
gen_fhi/workspace/db/postgres/core$property_search.class
gen_fhi/workspace/db/postgres/core__init.class

sun-one23:10:30

$ jar tf target/workspace-0.0.313-standalone.jar | grep workspace/db/postgres.core
gen_fhi/workspace/db/postgres/core$fn__52985.class
gen_fhi/workspace/db/postgres/core$parameters__GT_where_clause.class
gen_fhi/workspace/db/postgres/core$fn__52867.class
gen_fhi/workspace/db/postgres/core$fn__52947.class
gen_fhi/workspace/db/postgres/core/
gen_fhi/workspace/db/postgres/core$parameters__GT_where_clause$fn__52955$fn__52960.class
gen_fhi/workspace/db/postgres/core$fn__52985$map__GT_PGFHIRDatabase__53030.class
gen_fhi/workspace/db/postgres/core$fn__52861.class
gen_fhi/workspace/db/postgres/core$eval52885.class
gen_fhi/workspace/db/postgres/core$fn__52903.class
gen_fhi/workspace/db/postgres/core$parameters__GT_where_clause$fn__52955.class
gen_fhi/workspace/db/postgres/core.clj
gen_fhi/workspace/db/postgres/core$fn__52985$__GT_PGFHIRDatabase__53028.class
gen_fhi/workspace/db/postgres/core$fn__53038.class
gen_fhi/workspace/db/postgres/core$fn__53034.class
gen_fhi/workspace/db/postgres/core$loading__6737__auto____52859.class
gen_fhi/workspace/db/postgres/core$eval52915.class
gen_fhi/workspace/db/postgres/core$property_search.class
gen_fhi/workspace/db/postgres/core__init.class
gen_fhi/workspace/db/postgres/core$property_search$fn__52951.class
gen_fhi/workspace/db/postgres/core$fn__52911.class
gen_fhi/workspace/db/postgres/core$eval52871.class
gen_fhi/workspace/db/postgres/core$eval52929.class
gen_fhi/workspace/db/postgres/core/PGFHIRDatabase$fn__53010.class
gen_fhi/workspace/db/postgres/core/PGFHIRDatabase$fn__53014.class
gen_fhi/workspace/db/postgres/core/PGFHIRDatabase$fn__52994.class
gen_fhi/workspace/db/postgres/core/PGFHIRDatabase$reify__52989.class
gen_fhi/workspace/db/postgres/core/PGFHIRDatabase.class
$ 

hiredman23:10:09

the aot compilation of gen-fhi.workspace.db.postgres.core should cause aot compilation of gen-fhi.fhir-client.protocols then

hiredman23:10:34

are you using the latest tools-build/tools-deps?

sun-one23:10:56

Potentially not let me check and bump if needed.

hiredman23:10:05

there were some bugs in the aot compilation stuff (the order in which it aot compiled could lead to problems), I don't see how that could cause this, but you never know

sun-one23:10:35

This was my build script:

(ns build
  (:require [clojure.tools.build.api :as b]))

(def workspace-lib 'gen-fhi/workspace)
(def version (format "0.0.%s" (b/git-count-revs nil)))
(def class-dir "target/classes")
(def basis (b/create-basis))
(def uber-file (format "target/%s-%s-standalone.jar" (name workspace-lib) version))

(def src-dirs ["clj" "cljc" "test" "migrate" "resources" "cli"])

(defn clean [_]
  (b/delete {:path "target"}))

(defn figwheel []
  (b/process {:command-args ["clojure" "-m" "figwheel.main" "-O" "advanced" "-bo" "dev"]}))



(defn uber [_]
  (clean nil)

  (figwheel)

  (b/copy-dir {:src-dirs src-dirs
               :target-dir class-dir})

  (b/compile-clj {:basis basis
                  :src-dirs src-dirs
                  :class-dir class-dir})

  (b/write-pom {:class-dir class-dir
                :lib workspace-lib
                :version version
                :basis basis
                :src-dirs src-dirs})

  (b/uber {:class-dir class-dir
           :uber-file uber-file
           :basis basis
           :main 'gen-fhi.workspace.core}))
It's mostly copied from the reference material with modification to source dirs. Tools.build I'm on the latest sha from the repo.

hiredman23:10:31

you may want clean to delete class-dir

sun-one23:10:45

It should delete the directory containing it "target" I can also delete that dir specifically as well for sanity.

hiredman23:10:55

ah of course

sun-one23:10:44

Should have immediatlly included this but this is the entire stack trace when I run java -jar

hiredman23:10:57

have you looked in target/classes after running uber, are is there a class file from gen-fhi.fhir-client.protocols in there?

hiredman23:10:53

is there a user.clj in your jar file?

sun-one23:10:12

No their isn't .class files in there for gen-fhi.fhir-client.protocols. The only files that exist there appear to be compiled assets and resources from the project I'm running tools.build. I do have user.clj file

$ jar tf target/workspace-0.0.313-standalone.jar | grep user                      
user$start_server.class
user$fn__52521.class
user__init.class
user$reset.class
user.clj
user$loading__6737__auto____52519.class

hiredman23:10:36

that user.clj will be what is breaking it

sun-one23:10:53

Really why would that be?

hiredman23:10:01

user.clj is special

hiredman23:10:23

if clojure finds it, it loads it before doing anything else

hiredman23:10:53

so it is being loaded before aot compilation happens, which is causing anything it loads not to be properly aot compiled

hiredman23:10:16

and then it is being loaded again as your uberjar loads clojure before anything else is loaded

hiredman23:10:19

if you take your user.clj file and comment everything out then rebuild that will likely fix everything

sun-one23:10:29

Ah so this is happening when I'm executing the jar itself it's running the user.class without having AOT compiled for the deps in there?

sun-one23:10:40

Okay let me try that.

hiredman23:10:23

because clojure will load the first user.clj it finds some tutorials use it for setting up dev tooling

sun-one23:10:43

I'm assuming this would only affect my local project or would dependencies user.clj also present issues?

hiredman23:10:58

but it is kind of brittle, and you need to make sure it isn't included when aot compiling or jaring

hiredman23:10:08

dependencies as well

hiredman23:10:43

the first user.clj clojure finds, clojure itself just pulls stuff from the classpath

hiredman23:10:53

doesn't know if things from a dependency or not

sun-one23:10:39

Yea I think that solved it. I'm getting a different issue now but its unrelated and probably from some generated code I have

Execution error (IndexOutOfBoundsException) at clojure.asm.MethodWriter/computeMethodInfoSize (MethodWriter.java:2061).
Method code too large!

Full report at:
/var/folders/7m/3hkb8jgx3bnfcx4dpm1hqfqr0000gn/T/clojure-18076644096480273651.edn
Execution error (ExceptionInfo) at clojure.tools.build.tasks.compile-clj/compile-clj (compile_clj.clj:89).
Clojure compilation failed
Thanks @hiredman I would not have figured this out on my own.

hiredman00:10:07

The fact that you got class files for a namespace, but not for namespaces it depended on is kind of the tell

hiredman00:10:14

It generally means something was causing those depended on namespaces to be loaded before compilation

👌 1
seancorfield23:10:06

@chesslunatic You probably want to ensure user.clj is in a separate folder (e.g., dev) that is only added to your classpath via an alias (e.g,. :dev {:extra-paths ["dev"]}) so that it won't interfere with JAR building.

sun-one23:10:37

:thumbsup: Yep I'll be doing that in the future thanks @seancorfield