Fork me on GitHub
#beginners
<
2023-03-14
>
Kari Marttila11:03:28

I'm a bit puzzled with running a jar file created using tools.build. When running the app with REPL the logging works just fine. But. I created a jar file using tools.build. And when running the standalone jar file I get: java.lang.NoClassDefFoundError: clojure/tools/logging/impl/LoggerFactory I provide some more information in the:thread:not to pollute this main level with long listings.

Kari Marttila11:03:44

deps.edn:

{:paths ["resources"]
 :aliases {:dev {:extra-paths ["dev-resources" "classes"]
                 :extra-deps {org.clojure/clojure {:mvn/version "1.11.1"}
                              thheller/shadow-cljs {:mvn/version "2.20.20"}
...
           :backend {:extra-paths ["src/clj"]
                     :extra-deps {metosin/ring-http-response {:mvn/version "0.9.3"}
...
                                  org.clojure/tools.logging {:mvn/version "1.2.4"}
                                  commons-logging/commons-logging {:mvn/version "1.2"}
                                  org.slf4j/slf4j-api {:mvn/version "2.0.6"}
                                  ch.qos.logback/logback-classic {:mvn/version "1.4.5"}
...
           :build {:deps {io.github.clojure/tools.build {:mvn/version "0.9.4"}}
                   :ns-default build}

Kari Marttila11:03:13

build.clj:

(ns build
  (:refer-clojure :exclude [test])
  (:require [clojure.tools.build.api :as b]))

(def lib 'karimarttila/webstore)
(def main 'backend.core)
(def class-dir "target/classes")

(defn- uber-opts [opts]
  (assoc opts
         :lib lib 
         :main main
         :uber-file (format "target/%s-standalone.jar" lib)
         :basis (b/create-basis {:project "deps.edn"
                                 :aliases [:common :backend :frontend]
                                 }
                                )
         :class-dir class-dir
         :src-dirs ["src/clj" "src/cljc"]
         :ns-compile [main]))

(defn uber [opts]
  (println "***** Building uberjar *****")
  (println "Cleaning...")
  (b/delete {:path "target"})
  (let [opts (uber-opts opts)]
    (println "Copying files...")
    (b/copy-dir {:src-dirs   ["resources" "src/clj" "src/cljc"]
                 :target-dir class-dir})
    (println "Compiling files...")
    (b/compile-clj opts)
    (println "Creating uberjar...")
    (b/uber opts)))

Kari Marttila11:03:49

Build uberjar:

clj -T:build uber
Run uberjar:
java -jar target/karimarttila/webstore-standalone.jar
=> java.lang.NoClassDefFoundError: clojure/tools/logging/impl/LoggerFactory

Kari Marttila11:03:11

When I unzip the uberjar and search for log / factory related classes:

λ> fd | grep -i log | grep -i factory
./org/slf4j/LoggerFactoryFriend.class
./org/slf4j/ILoggerFactory.class
./org/slf4j/helpers/SubstituteLoggerFactory.class
./org/slf4j/helpers/NOPLoggerFactory.class
./org/slf4j/LoggerFactory.class
./org/slf4j/spi/LoggerFactoryBinder.class
./org/apache/commons/logging/impl/LogFactoryImpl.class
./org/apache/commons/logging/impl/LogFactoryImpl$1.class
./org/apache/commons/logging/impl/LogFactoryImpl$3.class
./org/apache/commons/logging/impl/LogFactoryImpl$2.class
./org/apache/commons/logging/LogFactory$3.class
./org/apache/commons/logging/LogFactory$1.class
./org/apache/commons/logging/LogFactory$6.class
./org/apache/commons/logging/LogFactory$5.class
./org/apache/commons/logging/LogFactory.class
./org/apache/commons/logging/LogFactory$2.class
./org/apache/commons/logging/LogFactory$4.class
./ch/qos/logback/core/sift/AppenderFactoryUsingSiftModel$1.class
./ch/qos/logback/core/sift/AppenderFactoryUsingSiftModel.class
./ch/qos/logback/core/sift/AppenderFactory.class
./ch/qos/logback/core/net/ObjectWriterFactory.class
./ch/qos/logback/core/net/QueueFactory.class
./ch/qos/logback/core/net/ssl/SSLContextFactoryBean.class
./ch/qos/logback/core/net/ssl/KeyStoreFactoryBean.class
./ch/qos/logback/core/net/ssl/TrustManagerFactoryFactoryBean.class
./ch/qos/logback/core/net/ssl/SecureRandomFactoryBean.class
./ch/qos/logback/core/net/ssl/ConfigurableSSLServerSocketFactory.class
./ch/qos/logback/core/net/ssl/ConfigurableSSLSocketFactory.class
./ch/qos/logback/core/net/ssl/KeyManagerFactoryFactoryBean.class
./ch/qos/logback/core/testUtil/MockInitialContextFactory.class
./ch/qos/logback/classic/util/StatusViaSLF4JLoggerFactory.class
./ch/qos/logback/core/model/ModelHandlerFactoryMethod.class
./ch/qos/logback/core/joran/util/beans/BeanDescriptionFactory.class

Kari Marttila12:03:02

And the main entry point of the app starts with:

(ns backend.core
  (:require
   [clojure.tools.logging :as log]
...
  (:gen-class))
...

Kari Marttila12:03:58

Actually. Now investigating this a bit further, I realized that the clojure/tools/logging package is not compiled:

λ> fd -HI | grep -i clojure | grep -i logging
./clojure/tools/logging.clj
./clojure/tools/logging
./clojure/tools/logging/test.clj
./clojure/tools/logging/impl.clj
./clojure/tools/logging/readable.clj
...
(but no .class files found)

Alex Miller (Clojure team)12:03:17

You probably need to set :src-dirs and :class-dir for compile-clj

Kari Marttila12:03:05

I thought I have already done that:

(def class-dir "target/classes")
...
(defn- uber-opts [opts]
  (assoc opts
...
         :class-dir class-dir               ;;; ======> HERE
         :src-dirs ["src/clj" "src/cljc"]   ;;; ======> HERE
         :ns-compile [main]))

(defn uber [opts]
...
  (let [opts (uber-opts opts)]
    (b/copy-dir {:src-dirs   ["resources" "src/clj" "src/cljc"]
                 :target-dir class-dir})
    (b/compile-clj opts)

Alex Miller (Clojure team)13:03:30

Sorry, lot of stuff to read

Kari Marttila13:03:19

No problem. 🙂

Kari Marttila13:03:42

There is no hurry regarding this issue. I think I will investigate this a bit more in the weekend.

Alex Miller (Clojure team)13:03:44

You’re not getting the class-dir in your uberjar

Alex Miller (Clojure team)13:03:56

Or at least that’s the thing to check - are the classes in target/classes

Kari Marttila13:03:39

λ> ll target/classes/
total 36
drwxrwxr-x 5 kari kari 4096 Mar 14 15:11 ./
drwxrwxr-x 3 kari kari 4096 Mar 14 15:11 ../
drwxrwxr-x 3 kari kari 4096 Mar 14 15:11 backend/
-rw-rw-r-- 1 kari kari 1099 Feb 20 20:18 config.edn
drwxrwxr-x 2 kari kari 4096 Mar 14 15:11 data/
-rw-rw-r-- 1 kari kari 3292 Feb 16 15:02 logback.xml
drwxrwxr-x 2 kari kari 4096 Mar 14 15:11 public/
-rw-rw-r-- 1 kari kari  218 Feb 13 19:33 README.txt
-rw-rw-r-- 1 kari kari  769 Mar 14 12:18 user.clj
=> should I have at this point all dependencies as class files here as well?

dpsutton13:03:14

from your deps edn, you have a very unusual classpath: :paths ["resources"]

dpsutton13:03:51

ah nevermind. i see how your basis is created.

Alex Miller (Clojure team)14:03:24

> should I have at this point all dependencies as class files here as well? no

Alex Miller (Clojure team)14:03:34

but are there classes under backend/ ?

Kari Marttila14:03:19

λ> ll target/classes/backend/
...
-rw-rw-r-- 1 kari kari  1100 Mar 14 15:11 'core$create_users.class'
-rw-rw-r-- 1 kari kari  1315 Mar 14 15:11 'core$env_value.class'
...
-rw-rw-r-- 1 kari kari  2492 Mar 14 15:11 'core$fn__20803.class'
-rw-rw-r-- 1 kari kari  3267 Mar 14 15:11 'core$loading__6789__auto____20754.class'
-rw-rw-r-- 1 kari kari  3167 Mar 14 15:11 'core$_main.class'
-rw-rw-r-- 1 kari kari  2118 Mar 14 15:11 'core$read_config.class'
-rw-rw-r-- 1 kari kari  3152 Mar 14 15:11 'core$system_config.class'
-rw-rw-r-- 1 kari kari   851 Mar 14 15:11 'core$system_config_start.class'
-rw-rw-r-- 1 kari kari  1956 Mar 14 15:11  core.class
-rw-rw-r-- 1 kari kari  3664 Mar 14 13:10  core.clj
-rw-rw-r-- 1 kari kari  5650 Mar 14 15:11  core__init.class
drwxrwxr-x 2 kari kari  4096 Mar 14 15:11  db/
-rw-rw-r-- 1 kari kari 13234 Mar 14 11:14  webserver.clj

Alex Miller (Clojure team)14:03:19

so revising my answer above, you shouldn't have dependent jars here but you should have any needed compiled classes from those dependencies. in particular, tools.logging will detect (at runtime) a logging factory to load and because of that, you're probably not getting that logging factory to compile just from loading backend.core (which pulls in tools.logging)

Alex Miller (Clojure team)14:03:49

I'm not sure which logging factory you intend to use - seems like there are a bunch of loggers available in your deps

Alex Miller (Clojure team)15:03:57

I think maybe just explicitly including the logging impl to the namespaces to compile would fix the issue though :ns-compile [main clojure.tools.logging.impl]

Kari Marttila15:03:32

Ok. Thanks! I will try! 🙂

Alex Miller (Clojure team)15:03:25

or actually, maybe better to swap the order of those

seancorfield16:03:39

You have user.clj in clj/src -- that is breaking things.

seancorfield16:03:02

If you move that to a folder that is only added in :dev, your uberjar will work.

Kari Marttila16:03:14

Ah! Let's try!

seancorfield16:03:09

I was curious about it because we use tools.logging and we don't need to explicitly AOT compile the impl ns -- just the app's main ns -- so I cloned your repo and rebuilt it without user.clj and, sure enough -- compiled tools.logging stuff:

> jar tvf target/karimarttila/webstore-standalone.jar |fgrep tools/logging
  1421 Tue Mar 14 09:39:51 PDT 2023 clojure/tools/logging$find_factory.class
  2469 Tue Mar 14 09:39:51 PDT 2023 clojure/tools/logging$loading__6789__auto____142.class
  5145 Tue Mar 14 09:39:51 PDT 2023 clojure/tools/logging$logp.class
  1856 Tue Mar 14 09:39:51 PDT 2023 clojure/tools/logging$fatalf.class
     0 Tue Mar 14 09:40:02 PDT 2023 clojure/tools/logging/
   657 Tue Mar 14 09:39:51 PDT 2023 clojure/tools/logging$log_stream$fn__291$fn__292.class
  1949 Tue Mar 14 09:39:51 PDT 2023 clojure/tools/logging$fn__299$log_uncapture_BANG___304.class
 18496 Tue Mar 14 09:39:51 PDT 2023 clojure/tools/logging__init.class
...
   767 Tue Mar 14 09:39:51 PDT 2023 clojure/tools/logging/impl$fn__192$G__188__195.class
 13861 Tue Mar 14 09:39:51 PDT 2023 clojure/tools/logging/impl__init.class
  6668 Tue Jan 04 14:42:06 PST 2022 clojure/tools/logging/readable.clj
  1232 Tue Mar 14 09:39:51 PDT 2023 clojure/tools/logging/impl$reify__216.class
  1356 Tue Mar 14 09:39:51 PDT 2023 clojure/tools/logging/impl$reify__218.class
  1909 Tue Mar 14 09:39:51 PDT 2023 clojure/tools/logging/impl$find_factory.class
...

Kari Marttila16:03:47

Thank you so much! 🙂

Kari Marttila16:03:56

Yep! It works now! Thank you all so much!

Kari Marttila16:03:42

Now I just have to add frontend to the standalone jar. I believe I can do that. 🙂

2
Alex Miller (Clojure team)17:03:18

User strikes again, maybe I should detect and warn about that

Kari Marttila17:03:44

Damn. The frontend step was easy.

Kari Marttila17:03:32

Maybe I write a blog post "Creating a production build for a full-stack Clojure app"?

Kari Marttila17:03:26

I mostly write these blog posts for myself - what I have learned. But maybe someone might benefit also regarding a blog post like that.

💯 8
seancorfield17:03:17

I see so many people get tripped up with user.clj -- it seems like a real foot-gun feature of Clojure these days... what was the original idea behind why it was added?

Kari Marttila17:03:16

At least for now on, I remember to keep it in:

:aliases {:dev {:extra-paths ["dev-resources" "dev-src"]
=> dev-src 🙂

Kari Marttila17:03:37

This is a great community. I'm pretty sure I couldn´t have figured that out myself.

seancorfield17:03:54

I avoid the auto-load altogether by having an explicit file -- not called user.clj -- in a dev-only path and a dev-only alias that starts my REPL and explicitly loads that file -- so there's no chance that Clojure will start loading code on its own. Color me paranoid 🙂

Alex Miller (Clojure team)17:03:59

user.clj is very old, added well before I was even using Clojure, so not sure what the original thinking was

Kari Marttila17:03:26

"user.clj considered harmful" ? 🙂

seancorfield17:03:56

I'm certainly in that camp. I've never used user.clj in... nearly 13 years of Clojure programming...

Kari Marttila17:03:23

I write a blog post about this building story - and add a chapter called "user.clj considered harmful" 🙂

seancorfield18:03:26

@U064X3EF3 Interesting. That adds a load of user.clj but doesn't remove the current machinery that auto-loads it? and that machinery has changed since to use maybeLoadResourceScript() 🙂 (nice to see the DCL for the root classpath -- that will address socket REPL etc, yes?)

Alex Miller (Clojure team)18:03:02

that's a very old commit from when user.clj was added

seancorfield18:03:34

Oh, I misread the stuff at the top of that page since I saw 1.12-alpha1

Alex Miller (Clojure team)18:03:29

I looked at the google group and irc log from around that time, which was when all the ns/require/use stuff was percolating (grep for jewel.clj or lib.clj for some ancient discussions there), but didn't actually find any discussion specifically of user.clj

seancorfield18:03:49

"committed on Apr 20, 2008" 🙂 My bad!

seancorfield18:03:35

I'd be interested to hear if Rich remembers why it was decided to auto-load user.clj 😉

Alex Miller (Clojure team)18:03:04

I'll see if he remembers

2
seancorfield20:03:10

It's good to see articles about tools.build!

✔️ 4
Petrus Theron23:05:56

Thanks for this, @U04V70XH6! I just lost 3 days on this because I couldn’t get uberjar to work in prod with beta libraries, so thought it was the libs. Hard to debug w/o logs 😅 . Why is user.clj breaking logging? Why does it prevent some .class files from being built?

seancorfield23:05:41

@U051SPP9Z If user.clj is on the classpath, it is automatically loaded by Clojure itself, which means it will get loaded when you try to compile files (AOT) for building an uberjar. You should make sure user.clj is only ever added to the classpath under an alias that you only use for development.

seancorfield23:05:55

(or just not use user.clj at all, which is my approach)

Petrus Theron23:05:33

Thanks, I understand that user.clj is being loaded automatically at startup, but why is it preventing the .class file from being built? Why do all other .class files get built?

seancorfield23:05:15

It depends what is in user.clj, what it requires, and so on.

seancorfield23:05:55

Looking at the OP's project, their user.clj was requiring Integrant and their backend.core ns and starting up part of their system(?), so all those required nses would be loaded too... that's potentially a lot of side-effects that could interact with logging and all sorts of other things.

Alex Miller (Clojure team)23:05:37

The stuff that user.clj loads will not be loaded during compilation. Because compilation is a side effect of loading, that stuff won’t be compiled

👍 2
Petrus Theron22:05:49

Thanks for the explanation, @U064X3EF3. So is my understanding correct that: during an uberjar build, user.clj blocks the compilation of all the namespaces it :require’s because it loads them automatically before the uberjar process can kick off and load deps for compilation? (or only some under certain conditions?) Well, that certainly is a footgun.

phill23:11:12

While user.clj is not a good place for side effects - and everything is a side effect - user.clj is not a bad place for the things people usually put there as long as they sheathe it all in (comment...)

oddsor10:02:35

I just got hit by this same footgun while attempting to debug why a websocket-protocol wasn’t found in a demo-app I was making! If I have a nrepl in my demo-app and want to connect to some sort of “dev-namespace” in production (🤷), is there a way to set another default namespace than user.clj? Either via the nrepl-server’s config, or when connecting?

oddsor14:02:50

Great! This one was quite mysterious; I spent a good amount of time wondering what went wrong 😅

seancorfield18:02:36

I'll repost my comment from that Ask, just to get more feedback: I'd be very happy to see an env var that you could set "globally" for your system that disabled loading user.clj completely. I've pretty much always felt the auto-loading of user.clj was a mistake. I get that it is "convenient" but it really smacks of "easy" vs "simple" to me.

Alex Miller (Clojure team)19:02:02

you've already posted that on ask, I dropped the link on your comment

seancorfield19:02:00

Oh, thanks! A year ago... Heh, it's been a bugbear of mine for a decade 🙂

oddsor11:02:30

I have a followup-question in the meantime! For the rare use-cases where I want a user-namespace (a namespace containing helper-fns in interactive demos on deployed apps), is it possible to do this:

(require 'user :reload-all)
somewhere in the build.clj-file? thinking-face

Alex Miller (Clojure team)13:02:20

There is no affordance for this right now

👍 1
slk50012:03:01

I am a bit puzzled by the following definition in book: "Everything Is a Sequence Every aggregate data structure in Clojure can be viewed as a sequence." from Programming Clojure, 3rd Edition by Alex Miller, Stuart Halloway, Aaron Bedra. link https://www.oreilly.com/library/view/programming-clojure-3rd/9781680505719/f_0028.xhtml 1. "Everything Is a Sequance" I don't think so i.e. number 1 is not a sequance. So we can't say "Everything Is a Sequance". 2. "Every aggregate data structure in Clojure can be viewed as a sequence." This term "aggregate data" is only once used in book and there is no definition of it. I've found a definiton in my notes (but I don't remember the source) 'data type that holds more than one thing'. I was thinkig about it and I think it should be 'data type that can hold more than one thing' because list with one or zero elements is also a 'aggregate data' and can be viewed a sequence. What do you think?

pyry12:03:31

1. Of course there are primitive elements like 1 and true that can't easily be interpreted as sequences. As the next phrase indicates, the title doesn't try to to hint at such elements. But the aggregate data structures can be thought of as sequences. 2. Think of aggregate data structure as just your typical data structure. The qualifying word aggregate just seems to be included to clarify that indeed it's those typical data structures that they're referring to here. https://www.merriam-webster.com/dictionary/aggregate

👍 4
Bob B12:03:55

"Everything is a sequence" is a section title, not an assertion. I think it's just there to theme the idea that collections can be viewed as sequences.

8
dgb2313:03:59

This section is very important for Clojure, partly because it discerns it from other Lisps that you might know. Much of its core library and the runtime deal in sequences, which is an abstraction over concrete data structures like vectors, maps, lists, sets and lazy collections generated by functions. The key idea here is that you're programming against this abstraction if you are dealing with anything that has a concrete collection behind it. This section wants you to think of all of these things as sequences in order to see why all of the sequence functions work on these concrete data structures. Note that later you'll be introduced to transducers, which take this a step further. If you think of sequence transformations as reducing them with conj , producing a new sequence, then that conj can be parametrized or pulled out so to speak in order for you to compose generic step transformations that can be applied to sequences, channels, streams maybe websocket messages or whatever can be seen as doing work on individual steps in a series of steps. So ultimately the "Everything is a Sequence" statement is very contextual. A more precise statement would be "All of those things are sequences and here's what you can do with them".

dgb2313:03:59

Oh and in a sense the statement is very true, since Clojure is a Lisp, it is defined in terms of its own data structures. In that sense everything is really a sequence. You might get error messages from say your REPL or from a linter when some form isn't proper. That's because your code is just a data structure in a certain shape. You can produce these forms and transform them with sequence functions. So everything kind of is a sequence.

xbrln14:03:38

In the web app am working on, am defining spec in one namespace and using it in another. I use the spec in my namespace like this :path-to-where-spec-resides/name-of-spec-definition When I run the application, I dont get any errors and app runs as expected. But when I compile using leinuberjar and run the jar I get the following error.

Started app
Execution error (AssertionError) at spec-tools.core/create-spec (core.cljc:536).
Assert failed:  Unable to resolve spec: :path-to-where-spec-resides/name-of-spec-definition
(get-spec spec)
Does anyone have an idea what this might be and how it can be solved 🙂

Ben Lieberman14:03:24

Specs are in a global registry and not the ns where you def them with s/def which might be the issue

xbrln14:03:24

I think that is the issue. So if not with the namespace then how can one call it ? When I give just the spec name like :name-of-spec-definition it give Unable to resolve spec: error.

dpsutton15:03:40

is the namespace which defines your spec in your uberjar? And do you require that namespace before using the spec?

xbrln15:03:52

No I did not require it. Also I don't know how to find out if it is in the uberjar (am not familiar with java 😐) This is the spec definition in the spec NS (s/def ::spec-test (s/keys :req-un [::post-id])) And am using it in my route file like {:body :path-to-spec-file/spec-test} Previously I was defining the spec in the route file and using it like this and it works. {:body ::spec-test}

dpsutton15:03:16

I want to focus on the namespace that defines the specs, not any particular namespace of the keyword of the spec if that makes sense to you. Which namespace includes the s/def form defining the spec?

xbrln16:03:33

app_name/spec/request.clj is where spec is defined. app_name/domain-name/routes.clj is where that spec is used.

dpsutton17:03:18

do you require app-name.spec.request anywhere in another namespace? I suspect the code isn’t getting loaded

xbrln17:03:49

I don’t require it anywhere.

dpsutton17:03:57

if it’s not required, it will never get loaded and those specs won’t be defined

2
xbrln05:03:24

I was not aware that spec has to be required, I thought referring them with the fully qualified keyword alone is enough. For anyone checking this in the future, it works after I require them like (:require [app_name.spec.request :as spec-request]) and use it like ::spec-request/name-of-spec Thank you @U11BV7MTK for pointing this out !! 🙌