Fork me on GitHub
#clojure
<
2020-06-23
>
Franklin09:06:54

when you create a jar file for a Clojure application and then run the jar file with java, can you still get a repl to that clojure application?

Franklin09:06:03

How can you do this?

dharrigan09:06:21

You can get a basic repl like this:

dharrigan09:06:28

java -Dclojure.server.repl="{:port,44444,:accept,clojure.core.server/repl}" -jar helloworld.jar

3
dharrigan09:06:39

❯ telnet localhost 44444                
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
user=> (+ 1 2)
3
user=> 

dominicm10:06:36

I always create my jars with clojure.main as their main, so I can do java -jar myjar.jar to get a REPL, or java -jar myjar.jar -m myapp.main to run the app.

3
dharrigan12:06:19

That's a great idea!

Ed19:06:13

you could always do java -cp myjar.jar clojure.main didn't set clojure.main as your main class

rickmoynihan08:06:58

Doesn’t the reverse make more sense? Having java -jar myjar.jar to boot the app, and using java -cp myjar.jar -m clojure.main to start a REPL? The intention of packaging an app is to normally run the app not a repl right?!

dominicm10:06:59

clojure.main can't start clojure.main

dominicm10:06:13

but also, either way it's generally hard-coded in some kind of launch script.

dominicm10:06:37

I already know how clojure.main works, having to learn each project's special main when I'm trying to debug a production issue is just pain waiting to happen.

dominicm10:06:11

"oh, we use -m for migrations. -repl is for a repl, yeah, like in go. Barry used to do Go, he said it's the posix standard now"

rickmoynihan10:06:09

Sorry I meant to say java -cp myjar.jar clojure.main

dominicm10:06:40

That doesn't work with some types of jar, e.g. ones with a Class-Path manifest.

rickmoynihan11:06:40

Sure but doesn’t your way also depend on creating the jar in a particular format too?

wombawomba12:06:44

How can I get deftype to use interface default implementations? I’m trying to implement https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/serialization/Serializer.java via

(deftype Foo []
  Serializer (serialize [...] ...))
which should suffice because all the other methods have default implementations, yet I’m getting this error:
java.lang.AbstractMethodError: Receiver class my-ns.Foo does not define or inherit an implementation of the resolved method 'void configure(java.util.Map, boolean)' of interface org.apache.kafka.common.serialization.Serializer.

wombawomba12:06:38

…or should I be using something other than deftype for this? I need to be able to resolve the class by name afterwards.

noisesmith14:06:54

in order to use concrete inheritance with a predictably named class, you need gen-class afaik mind you proxy does create a class with a name, just not a predictable one you can hard code from outside the app

wombawomba15:06:50

I do seem to get a predictable class name with deftype

wombawomba15:06:00

but maybe that’s a coincidence?

wombawomba15:06:31

Either way, my question was mainly how I can get Clojure to indicate that the default interface methods should be used

noisesmith15:06:26

deftype provides a predictable class name, and canot do concrete inheritance. proxy provides concrete inheritance but does not offer a predictable class name

noisesmith15:06:47

you could combine the two (using deftype as a wrapper) or use gen-class, or write a java shim

noisesmith15:06:59

I wasn't as clear originally as I intended to be, though I thought I was answering the question I skipped some details

noisesmith16:06:18

oh, the other (implicit) detail - abstract class default implementations are just inheritance by a subclass plus delegation to a super as far as the bytecode is concerned

noisesmith16:06:36

which is why deftype, defrecord etc. can't do it

wombawomba17:06:37

got it, thanks

murtaza5214:06:29

can someone help on why the macro below throws an error -

(defmacro make-defs
  [coll]
  ~@(map (fn [[var-name# v#]]
           `(def ~var-name# ~v#))
         ~coll))

(make-defs [[n1 1] [n2 1]])
I want the macro to generate -
(def n1 1)

(def n2 2)

borkdude14:06:29

@murtaza52 Try wrapping the body in a do:

`(do ~@(map ...))

borkdude14:06:13

or: (cons 'do (map ...))

murtaza5214:06:47

@borkdude it still throws the same error - Attempting to call unbound fn: #'clojure.core/unquote

delaguardo14:06:42

@murtaza52

(defmacro make-defs
  [coll]
  `(do ~@(map (fn [[var-name# v#]]
                `(def ~var-name# ~v#))
              coll)))

delaguardo14:06:36

~ is working only inside of (...)`

murtaza5214:06:26

@delaguardo thanks that works !

agata_anastazja (she/her)19:06:48

I am having problems with import, when I am trying to import org.apache.commons.lang3.StringUtils in lein repl, it works, but when I do it in my program it says

Could not locate org/apache/commons/lang3/StringUtils__init.class, org/apache/commons/lang3/StringUtils.clj or org/apache/commons/lang3/StringUtils.cljc on classpath. 

seancorfield20:06:42

@U2J6U9D1R Can you share the relevant piece of code from your program?

Ed20:06:51

Have you got apache commons string utils in your dependencies in your project.clj file?

seancorfield20:06:12

@U0P0TMEFJ If it works in the REPL, they must have the dependency.

seancorfield20:06:44

It sounds like you're trying to reference org.apache.commons.lang3.StringUtils as if it were a Clojure namespace...

Ed20:06:45

oops ... didn't spot that 😉

noisesmith20:06:15

yeah, that error looks like an error message from require or use, which won't work on a class

agata_anastazja (she/her)20:06:38

okie dokie

(ns bob
(:require [org.apache.commons.lang3.StringUtils :as StringUtils]))

agata_anastazja (she/her)20:06:49

this is what I have in my program

agata_anastazja (she/her)20:06:24

and I import it in repl differently

agata_anastazja (she/her)20:06:28

(import org.apache.commons.lang3.StringUtils)

Ed20:06:39

Yeah, I think they're saying it should be (ns bob (:import [org.apache.commons.lang3 StringUtils]))

agata_anastazja (she/her)20:06:21

Syntax error macroexpanding clojure.core/ns at (bob.clj:1:1).
((:import [org.apache.commons.lang3.StringUtils])) - failed: Extra input spec: :clojure.core.specs.alpha/ns-form

agata_anastazja (she/her)20:06:36

and I haven't added anything to the project.clj

agata_anastazja (she/her)20:06:03

it looks like this

(defproject bob "0.1.0-SNAPSHOT"
  :description "bob exercise."
  :url ""
  :dependencies [[org.clojure/clojure "1.10.0"]])

noisesmith20:06:50

you have extra parens on :import

Ed20:06:20

can you paste the actual code? I think you have extra brackets in there.

noisesmith20:06:30

((:import ...)) is wrong, it should be (:import ...) - parens are not free in clojure

agata_anastazja (she/her)20:06:35

so in the actual code it's

(ns bob
(:import [org.apache.commons.lang3.StringUtils]))

noisesmith20:06:56

OK, there's two versions of import, one with extra coll delimiters one without

seancorfield20:06:56

Also, you might want to focus on the #beginners channel if you're running into these sorts of problems and can't figure them out yourself -- that channel has folks who have opted in to spending time helping folks who are very new to Clojure and are making straightforward mistakes like this.

seancorfield20:06:20

(both myself and @U051SS2EU are in that channel, so we'll still help you there!)

noisesmith20:06:33

you either want (:import foo.bar.Baz) or (:import (foo.bar Baz))

agata_anastazja (she/her)20:06:19

yes @U04V70XH6, should I replicate it there?

seancorfield20:06:28

For the future.

seancorfield20:06:43

I invited you (back) into #beginners so it should be easy to find in your Slack sidebar.

noisesmith20:06:17

finally - the class you want from apache commons should not be available if you haven't added it to project deps

seancorfield20:06:56

Yup. Not sure how you got it working in lein repl with it being in your :dependencies.

noisesmith20:06:31

maybe it's a dep pulled in by a plugin(?)

seancorfield20:06:34

Unless Leiningen adds that as a dependency behind the scenes?

noisesmith20:06:12

it won't do that normally, but plugins can do anything to your deps before launching the target vm

seancorfield20:06:37

Looks like lein brings that dep in without any plugins

(! 611)-> lein repl
nREPL server started on port 55265 on host 127.0.0.1 - 
REPL-y 0.4.4, nREPL 0.6.0
Clojure 1.10.0
OpenJDK 64-Bit Server VM 14+36-1461
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

user=> (ns bob
  #_=> (:import (org.apache.commons.lang3 StringUtils)))
nil

noisesmith20:06:04

oh wow, thats new to me

noisesmith20:06:11

might be an nrepl thing?

seancorfield20:06:47

You would need to add [org.apache.commons/commons-lang3 "3.7"] to your :dependencies vector in project.clj to be able to access that class...

3
parrot 3
noisesmith20:06:49

yeah, I suspect it comes in via reply, anyway the right thing to do is to explicitly declare deps for classes you use directly in your code

seancorfield20:06:38

(! 634)-> cat project.clj 
(defproject bob "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url ""
  :license {:name "EPL-2.0 OR GPL-2.0-or-later WITH Classpath-exception-2.0"
            :url ""}
  :dependencies [[org.clojure/clojure "1.10.1"]
                 [org.apache.commons/commons-lang3 "3.7"]]
  :main ^:skip-aot bob.core
  :target-path "target/%s"
  :profiles {:uberjar {:aot :all}})
(! 635)-> lein run
Hello, World!
(! 636)-> cat src/bob/core.clj 
(ns bob.core
  (:import (org.apache.commons.lang3 StringUtils))
  (:gen-class))

(defn -main
  "I don't do a whole lot ... yet."
  [& args]
  (println "Hello, World!"))

agata_anastazja (she/her)19:06:05

Does it mean I need to install stringutils on my machine?

agata_anastazja (she/her)19:06:21

feels like I'm missing something very basic

hiredman19:06:57

When you import in the repl what are you doing?

agata_anastazja (she/her)20:06:07

(import org.apache.commons.lang3.StringUtils)

agata_anastazja (she/her)20:06:18

so it's different way of importing it

hiredman21:06:59

that isn't importing it

hiredman21:06:20

huh, I guess it is

agata_anastazja (she/her)22:06:55

anyhow, it fixed now! thank you for help

hiredman22:06:02

it's been a decade since import became a macro and accepted unquoted symbols, you'd think I'd be on top of that

murtaza5219:06:40

What is the purpose of the symbol @ in the code below ?

(def a [1 2])
@(ns-resolve *ns* (symbol "a"))

noisesmith20:06:29

it's a reader macro for deref

user=> '@a
(clojure.core/deref a)

Ed20:06:32

It deref's the var you just resolved

noisesmith20:06:56

resolve returns a var, a var is a container, you can use deref to get the value inside a var

dpsutton20:06:49

check out (doc deref) for what deref does on vars, agents, atoms, delays, futures, etc

murtaza5221:06:54

I have a macro which is creating defs. It works when called in the same ns, however when I require it and call it from another ns it fails -

(defmacro make-def
  [coll]
  (let [coll# (if (symbol? coll) @(ns-resolve *ns* coll) coll)]
    `(def a ~coll#)))

(def b [1 2])
(make-def b)
I get the following error when called from a different ns - Cant create defs outside of current ns. How do I write the above macro sot hat it can be called from a different ns ?

kaaninho21:06:11

What should this macro do?

kaaninho21:06:18

What do you want do achieve?

noisesmith21:06:51

the a in the macro body gets turned into a namespaced symbol by `, and creates an invalid def

noisesmith21:06:44

btw resolve implicitly uses the bindings of *ns* and ns-resolve is only needed if looking up symbols as if you were in another ns, you can just use

@(resolve coll)

noisesmith21:06:23

also, that error with a getting namespaced should occur no matter where you call the macro from

murtaza5221:06:24

@UMNSSKZL3 I have some data for which I need to generate defs. So instead of writing 50+ defs by hand, I decided to write a macro that will generate it.

murtaza5221:06:16

@U051SS2EU the error does not occur when I call the macro in the same ns it has been defined, it only happens when I call it from another ns, which is what I need.

noisesmith21:06:04

you said that before, but I find it hard to beleive - maybe because a already exists in that ns, so it accidentally works...

noisesmith21:06:19

hmm - def must have a special case for the current ns

noisesmith21:06:38

anyway, the solution is to not use a namespaced symbol - ` is doing nothing in your code

noisesmith21:06:27

(well, not really - the only thing it's doing is creating a namespaced symbol, which causes this error outside the ns where it was defined)

murtaza5205:06:54

thanks that worked

murtaza5205:06:36

one question what is pro / con of not using ` and using just ' . Is there a disadvantage of not having namespaced symbol ?

kaaninho07:06:20

I think you should look into macros a bit more (no offense!). In you can "unquote" (~`) some symbols, with ' you cant. Regarding your question: It differs. When defing, you dont want namespaced symbols, in other cases you want them. Can you provide an example, (with example input and output) what code the macro should create?

noisesmith17:06:32

is for templating, so it's usually used in macros (but you can write macros without 
and you can use
outside macros)
namespaces symbols to prevent a common sort of error in macros - accidentally capturing symbols in the use context

murtaza5217:06:45

@UMNSSKZL3 yes my macro understanding is pretty rudimentary, I need to read up more. I am using the fulcro UI framework, and it uses a certain def to define UI elements. There is a large form I need to create, and instead of typing all the elements, I am using the macro to generate them.

murtaza5217:06:24

@U051SS2EU thanks for the explanation - that is what I was looking for - accidentally capturing symbols.

noisesmith17:06:43

a paper on the problem of symbol capture https://www.csee.umbc.edu/courses/undergraduate/331/resources/lisp/onLisp/09variableCapture.pdf - I guess maybe "variable capture" is the preferred name?

murtaza5217:06:14

@UMNSSKZL3 any reading recommendations ?

kaaninho08:06:35

@murtaza52 I like the book "Clojure for the Brave and True", its lightweight and fun to read. Also the author lets you read it online for free. For macros in particular https://www.braveclojure.com/read-and-eval/ and the following chapter. Have fun!

murtaza5206:06:59

@UMNSSKZL3 thanks let me go through it

kwladyka21:06:46

I run app like this on localhost and it works perfect docker run --rm -it -p 80:80 -m 128M --memory-swap 0 --memory-swappiness 0 foo I run the same image on cloud run (kubernetes) and I get Memory limit of 128M exceeded with 333M used. Consider increasing the memory limit Even if I use -Xms and -Xmx, check memory limits and usage by Java functions and by bash it shows me I use 60-80 MB. Only set to 512MB fix the memory issue, but I literally see in all possible way app use around 60-80 MB. Any hints? I tried literally everything. How is it even possible the same docker image works on my localhost but not in google cloud run.

dpsutton21:06:41

what java version?

kwladyka21:06:18

I tried with 14 and 11 using official clojure docker images with buster and scratch

kwladyka21:06:39

so the app even not start, the memory is exceeded when starting and even first print from main functions is not processed

kwladyka21:06:27

I was trying to run this with java -jar foo.jar (uberjar) and with clj -m foo.core

Cory21:06:26

two things: cloud run has hard memory limits while docker doesn't and heap is isn't the only memory allocated

hiredman21:06:51

"official" is kind of a loaded term, they are the official images from docker as far as I know, not official from the people who make clojure

kwladyka21:06:48

> while docker doesn’t Do you mean docker run --rm -it -p 80:80 -m 128M --memory-swap 0 --memory swappiness 0 foo doesn’t limit memory in the same way?

kwladyka21:06:42

also -Xms and -Xmx should be me pretty confident about memory right?

kwladyka22:06:04

(let [free (/ (.freeMemory (Runtime/getRuntime)) 1048576.)
        total (/ (.totalMemory (Runtime/getRuntime)) 1048576.)
        used (- total free)]
    (println "max memory" (/ (.maxMemory (Runtime/getRuntime)) 1048576.))
    (println "free memory" free)
    (println "total memory" total)
    (println "used memory" used))
this too

hiredman22:06:06

-Xms and -Xmx limit the size of the gc'ed heap

hiredman22:06:19

anything outside of that is not counted

kwladyka22:06:45

what else I can check then?

lukasz22:06:25

you can try using -XX:InitialRAMPercentage and -XX:MaxRAMPercentage to set the heap size as percentage of available limits within the container

👍 3
kwladyka20:06:52

@U0JEFEZH6 what values do you recommend for clojure apps?

lukasz20:06:54

I usually start with 70% and then see how low I can go

lukasz20:06:04

(as per the gist I shared yesterday)

kwladyka20:06:15

What is the default heap size ? I can’t find this in google

kwladyka20:06:37

ah true, I have it in my todo to learn this file

lukasz20:06:01

IIRC starts at 128mb and goes up to whatever it can, MaxRAMPercentage I believe has no default or it's 20%

kwladyka20:06:24

oh then this values really make sense

kwladyka20:06:33

for docker containers

lukasz20:06:06

Depends - we use fargate which has its own limits, so it does require some tuning (instance resources -> docker resources -> jvm resources). If you co-locate many applications on one host that's a different challenge altogether - I'd say it's 20% learning, 80% experimentation

hiredman22:06:32

your operating system will have some way to get the used memory size for a process (interpreting that can be tricky as well)

kwladyka22:06:43

but the point is when process start I see it use very low memory (I also printed bash commands to logs). The issue about memory is happening only when start the container.

kwladyka22:06:57

So while I already run the app it is probably too late

kwladyka22:06:26

unless it consume memory in some secret way which I don’t see

kwladyka22:06:22

I cut this code of the app to so simple, almost hello world. This is so strange. It looks like I can cut more random code and it start to work. But add a few lines and it consume +hundreds of MB. Really doesn’t make sense.

kwladyka22:06:34

Unless this is some cloud run bug

kwladyka22:06:04

As I understand this docker run --rm -it -p 80:80 -m 128M --memory-swap 0 --memory swappiness 0 foo should make it fail also on my computer

kwladyka22:06:30

in cloud run even 256M is not enough

lukasz22:06:13

Cloud run gives you runtime monitoring - so it should report memory/cpu usage for your containerized process - that would be a starting point. In my experience you always need more resources in environments like these (e.g. my lambda functions do work under 128mb but run faster in 515mb envs). Lastly, do you see the same problem running in docker: as in app starts, but eventually gets killed by the OOM while receiving the same traffic as in cloud run? Also you will always get better results with a uberjar + fine tuned jvm flags

hiredman22:06:47

I am not familiar with the memory limiting options for docker, but in general I have found things that deal with linux cgroups to be very complex, and sometimes (due to running on old versions where certain limits don't exist, etc) fail to actually limit things

Cory22:06:53

yeah, the memory resource controller is very hairy in my experience, especially with the JVM, i tend to over provision by default

kwladyka22:06:47

> so it should report memory/cpu usage for your containerized process it does but it doesn’t help debug information for this condition practically doesn’t exist > Lastly, do you see the same problem running in docker: as in app starts, but eventually gets killed by the OOM while receiving the same traffic as in cloud run? no, when it start it is fine and showing very low memory usage. never exceed. so it looks like this is only starting app issue. I guess. Even more saying. Loading Java issue, because even first line of the code is not processed. > Also you will always get better results with a uberjar + fine tuned jvm flags I was trying many things. I use uberjar. What other jvm flags can I use?

lukasz22:06:11

I've shared our entrypoint/launcher script in #docker

seancorfield22:06:03

There's a #docker channel and a #kubernetes channel which this should be in. This is not about Clojure.

seancorfield22:06:46

Or at least use a thread please instead of cluttering up the main channel with a lot of non-Clojure questions and speculation. Thank you.

kwladyka22:06:21

Sorry. It looks like it is about clojure / java running app, but maybe docker issue. I don’t know.

kwladyka22:06:35

Is it possible clojure run app have a peak of memory usage when loading the app? Did you have such condition before?

seancorfield22:06:00

We don't run code in small containers so we don't "care". The problem you are having is specific to the Docker/Kubernetes environment you are running in, which is why the conversation doesn't belong here.

kwladyka23:06:33

> The problem you are having is specific to the Docker/Kubernetes environment you are running in I don’t know that. Maybe high memory peek during start the app is also outside docker, then it is not docker related issue.

kwladyka23:06:57

but it is fine, just saying I have no idea what is the source of the issue

jumar04:06:20

I think it was good to discuss this in #clojure because it's related to clojure app memory consumption. It would be better if it was in a single thread but so many conversations fail to do so. I'll follow up in #docker