Fork me on GitHub

i am reminded of george jahad's debug repl


Hello all, what is the best way to process same data with async process? Each process will do things like logging, calculate something, transform to something with data. I tried to use but if I include some Thread/sleep in one process I see that it slow down other process. How it processing can not depends each other?


What do you mean same data?


Just same input and process different independent result from it?


You could just use future, or if you want a more reactive model core.async


this process will listen events that came from main stream line


Hi all! I'm trying to use tools.deps to pull a dependency from a private git repo. If I've understood everything correctly, there is no configuration required for private repos (other than having a valid ssh key). So basically, if I can git pull the "git@..." url, I should be able to pull it with tools.deps, right? However, I am hitting this error 😕 (most of the stacktrace ommitted for brevity, let me know if it might be relevant):

Cloning: :********REDACTED*******.git
Error building classpath. connector is not available: 
com.jcraft.jsch.agentproxy.AgentProxyException: connector is not available: 
        at com.jcraft.jsch.agentproxy.ConnectorFactory.createConnector(ConnectorFactory.jav
        at clojure.lang.Delay.deref(
        at clojure.core$deref.invokeStatic(core.clj:2320)
        at clojure.core$deref.invoke(core.clj:2306)


I've been digging a bit on the issue, and found some people that encountered the same error when running clojure inside a docker container (for example: Note that this is not my case.


You need to run an ssh agent


I assumed that was already the case? Can you use ssh without it?


Yes, though many systems run an ssh agent by default. In that case, you'd need to add a key to it with ssh-add.


Hmm.. strange. It seems ssh-add returns "Could not open a connection to your authentication agent." So this might be it


Oh, if this is inside a container, chances are any ssh agent run in the host system is inaccessible.


no, that's currently not my case


Ok, that did it! I started ssh-agent (as described in and I was able to pull the git dep!

👍 4

This is all ssh-related, nothing specific to clojure or even git, but if you don't provide any auth info (eg, password), ssh will look for an identity/keypair with one of the default names; often ~/.ssh/id_dsa or ~/.ssh/id_rsa; if those keys are encrypted, it will ask for a password. This is where ssh-agent comes in: you can store 1 or more private ssh keys, decrypted if necessary (`ssh-add` will ask for the passphrase on adding), and any ssh session that can find the ssh-agent will try those keys to establish a session.


I currently don't have a passphrase, so I assumed tools.deps would just get my ~/.ssh/id_rsa just like git cmdline does, but it seems that is not the case.


Anyway, thanks for the help! 😀


Huh. Looks like it's using org.eclipse.jgit underneath. Maybe it doesn't do any identity lookup.


I have another question. Are transitive dependencies not available when using tools.deps? I have two projects, project B depends on project A via git, and project A depends on core.async (via regular mvn dependency). When I open a repl in project B, it fails to load because B was also using core.async (without a direct dependency). Is this the intended behavior?


nevermind, that was just me writing :dependencies instead of :deps 😅


@penryu Actually, by checking the issues, I've seen they're trying to migrate to just shell out to git cli to solve most current issues


That would do it.


What is the best way to deal with coercions/validation in a REST API in clojure? I was trying to use compjure-api + spec-tools + clojure.spec.alpha, but things just don 't work as expected. My undestanding is that spec was not created for coercion at application boundaries. But I don't know what to use instead.


@zara I've used schema in the past. They explicitly support coercion:


So do you use Schema for API and spec to document and validate data inside your programs?


we use schema for both @zara, although we've also been using schema since before spec was a thing, and i'd rather not have both schema+spec versions of datastructure shapes, so i'm not in a hurry to change


juxt/yada supports schema for API coercion and description (i.e. swagger)


I think i will go with Schema then.


To be honest, I don't have a good answer for that. I used schema before spec even existed. Currently I use spec, and since I'm working mostly on an all-clojure codebase, I have no need for coercion.


We use nippy to communicate between services, instead of encoding/decoding JSON: (


Is there a way to define something like this in Schema?

  :measures (s/keys :req-un [::name ::length ::width ::height])
  :predefined (s/keys :req-un [::name ::predefined_package]))


The issue about coercion when using spec vs schema is interesting to me, as I'm planning to use spec but haven't started yet. Say that JSON is coming in from the outside. With schema there is a way to coerce it during validation, but not with spec. So with spec it must be coerced prior to validation. That seems straightforward to me, but perhaps I'm missing something because I haven't tried it yet. In your experience does coercion often need to be done differently, depending on the particular spec? I was hoping to simply convert JSON to maps and vectors in a generic way, and use spec to validate the result.


I looked at the schema blog post about coercion: And I think I can do the coercion generically (before validating with spec) as long as I convert JSON types to lowest common denominator types (e.g., Number rather than Long), and don't support convenience conversions (e.g., String to Keyword).


I usually do the coercion on the parser (like cheshire or jsonista), then validate the json parsed object


Thanks @U6CN6JQ22, I'll take a look at that.

Ramon Rios15:01:24

Folks, is that a way to get a octet-stream and read it?

Ramon Rios15:01:14

i got a .docx by request, the content type is octet-stream, i'm using the byte-streams library and was able to transform it on a ByteArrayInputStream, but i'm kind of lost in how to transform it on a docx file again.


you can use to write that ByteArrayInputStream to a .File

Ramon Rios18:01:19

(with-open [xin (bs/to-input-stream (get-template))
              xout ( "resources/test.docx")]
    (io/copy xin xout))

Ramon Rios18:01:26

I tried to do something with it


well, resources is a weird destination


and I don't think output-stream is that convenient for files


I'd explicitly do (io/output-stream (io/file "/some/path/test.doxc"))

Ramon Rios18:01:56

I'm just goofing around for a test.

Ramon Rios18:01:03

Not something that would go for a project


the reason I say resources is weird, is in most setups it's a "magic" folder that lets you read files locally but read from inside a jar in prod


oh, I guess output-stream is that convenient

(ins)user=> (with-open [o (io/output-stream "foo.txt")] (io/copy (io/input-stream "user.clj") o))
(ins)user=> (= (slurp "foo.txt") (slurp "user.clj"))

Ramon Rios18:01:30

I'm just creating a POC, so no rules haha

Ramon Rios18:01:46

Saying this about the resources stuff


cool - just mentioning because it's a really common problem for new clojure users (if you don't make the resources mistake, someone else reading this thread will, and it's good to be aware of why it happens and how to avoid it)

Ramon Rios18:01:50

(defn test []
  (with-open [xin (bs/to-input-stream (get-template))
              xout (io/output-stream (io/file "resources/test.docx"))]
    (io/copy xin xout)
    (slurp "resources/test.docx")))

Ramon Rios18:01:58

My file is corrupt

Ramon Rios18:01:51

i can pass the outputstream with no issues and my slurp brings me a empty string

Ramon Rios18:01:13

When i open on word, it's corrupted


the file doesn't get flushed until the with-open exits, for one thing

Ramon Rios18:01:40

So with-open do the flush job?


one thing to look out for is encoding - did it get base64 encoded over the wire? is some step using a text function (you should be using binary ops, not text ops, on a docx)


the with-open flushes, but the slurp is guaranteed to happen before the flush in your example


and slurp is convenient, but it's not for binary data like docx, it's for text

Ramon Rios18:01:52

Let me show you my response body


what is (get-template) returning?

Ramon Rios18:01:06

"PK \b\b\b ?n/P
x]h?{?8\f??S4G?A?y?Y8X???(?[Fw?i4o|??l?^?????PK\b??#?   =  PK \b
\b\b ?n/P               docProps/core.xml?R]O?0}?W,}??`L?F??'IL?h|??e
9?zo(?oA1????? m?\r?_?<I?[??3?<ÝalFGt?|?4;[u?c?@??????U???P*??Jr
v?v??}??\bB??O??X|PK\b??pM`  ?  PK \b\b\b ?n/P               d
?uR?\n-?E??Ruz?%??gJ?A+????b??%?($(W??{????F?? ?????j;??Vrx?
                         d??PK\b-]?:    PK \b\b\b ?n/P
           word/_rels/document.xml.rels??M\n?0???\"??U??nDp+? 1???6\
?^R,}¦T'? ????O&?U????7???m]k??=???n?H??A??>.???|m\r????????
?????@????????I??wPK\b?/0??     PK \b\b\b ?n/P               wo
rd/document.xml?U?n?0??+?c?[B? ????~ M?$?|aIYv??K=??(???\n\b?????
????Q?? ?5k?\\d,#l!M?f_w?nX?7W?????????6/?h4????]?M?E\r??+-Zo?p
B?????L0???gZ?.?\\ o??|Nd?'gD9?X??????
p?????z?????G??se,?@@Id?6???8??u??\r?? i?WkvU+?v??c<?C??PP?>?'??
?rm?m??;.??C??`?]-}BOH?T\t?v???P4 My?E?B????{j??]\r\b?{&?!pQw?]?.
?0  ?  PK \b\b\b ?n/P               word/styles.xml?UMO??W?|
%H??`K$vv??d1 ??HY??4Xl??9?9*?=cQ?Xe??n?/?+?3?By%?~?(W 4k??nR?;C
a??l1 ?B?JL??Fgt?b!4??#?@??l<jR?\br\n/?\\??M@??A??t??,??\n??G?E??
?I?C??<??a?#4e 9??8&?M??f?vs
                              ???w???5?!?[???Jn ,??.?4????/?\
??~Ów?X????{???Z?M?????PK\b??t|  \t  PK \b\b\b ?n/P
       word/fontTable.xml?PAN?0??\n?w??BQ??z??[g?X???4???N#!? ????
.^~ PK\b??;b  PK \b\b\b ?n/P               word/settings.xml
E?K?0\fD??\"?X?H???Bk?RbG????\t+??73z??yb????r?? ?<?t?p>?[0?????
               PK\bv???   ?   PK \b\b\b ?n/P               [Con
?4? ??$??R7y???+2+'y????? ??5??/@1?
                                   +?q?|t\r??????tzI??tHC? e~

5[?!?[??r??l?\"? ??Vr?Lc?????p???ti?Y???/??g_?n? R???????░?+???vR
                               ?#??????%?X*?d? ?d?pw??Aw?
                            76??z<\b5?B??0?i??,? PK\b
^  PK  \b\b\b ?n/P??#?   =
 \b\b\b ?n/P??pM`  ?                 docProps/core.xmlPK  \b\b
\b ?n/P-]?:                 ?  docProps/app.xmlPK  \b\b\b ?n/
P?/0??                  )  word/_rels/document.xml.relsPK  \b\b
\b ?n/Px?@?0  ?               8  word/document.xmlPK  \b\b\b ?n/
P??t|  \t               ?  word/styles.xmlPK  \b\b\b ?n/P??;b
 U               `\n  word/fontTable.xmlPK  \b\b\b ?n/Pv???   ?
                 word/settings.xmlPK  \b\b\b ?n/P

Ramon Rios18:01:06

??T  ^     
?\f [Content_Types].xmlPK \t \t < 4 "

Ramon Rios18:01:43

This code above

Ramon Rios18:01:49

a octet-stream


that's not a java type though, what does (type (get-template)) actually return?

Ramon Rios18:01:40

(defn get-template []
  (:body (client/get (:oci-url env))))

Ramon Rios18:01:45

This is my function

Ramon Rios18:01:10

It shoud return a .docx file that contains some text to be used as a template for a email


OK clj-http.client/get will return a hash-map, and the :body key has an input-stream


unless you are using middleware that messes with it

Ramon Rios18:01:02

What you mean by middleware?


I don't know if you understand that there's literally a function clojure.core/type that tells you the class of an object


clj-http uses middleware to modify its behavior and add features, this can include changing the body of a response


or even automatically putting it in a file

Ramon Rios18:01:21

I had no idea of it

Ramon Rios18:01:40

My coleague is using a .txt and worked.

Ramon Rios18:01:52

Probably the issue is with my document


there are things that are safe for text that corrupt binary files - try a text file and a jpg too

Ramon Rios18:01:13

Thanks for the tips


I think I found your problem


user=> (type (:body (client/get "")))


haha! slack is terrible


anyway - it's a string of the contents, which means that client/get is corrupting the contents

Ramon Rios19:01:19

What would you think it should be the best approach for it?»



(client/get "" {:as :byte-array})


this should do the right thing


then the io/copy will work etc.

Ramon Rios19:01:56

I'll test that


this worked for me:

user=> (io/copy (:body (client/get "" {:as :byte-array})) (io/file "google.png"))


(io/copy already promises to close a file if it opens it, so no with-open needed)


awesome - glad I could help

Ramon Rios19:01:15

I really appreciate your help 😄


you can also use {:as :stream} - that might even be a perf / resource usage improvement if that matters (with byte-array it's consuming a stream for you and putting it in a new byte array for you, with as stream you get to decide what you do with the stream, which can include using the input in a way that is less wasteful of memory -- probably doesn't matter for your use case though)


another way to put it: with stream, you directly put the stream into a file, with byte-array it's putting the data into an array, then you turn the array into a stream and put that into a file - so there's steps being done that aren't needed


Hmm, I was going to recommend docjure, but it looks like that only handles Excel files. Maybe you can go directly to the Apache POI site and find something you can use with Java interop?

Ramon Rios17:01:43

That's what i'm trying to do, i got a library for get the bytes and put into a inputStream


I'm trying to build my project inside docker using tools.deps and got a very weird issue. I'm able to build and run my project locally. However, inside docker, after everything is built successfully, attempting to run my project yields this obscure java error:

Exception in thread "main" java.lang.ExceptionInInitializerError
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(
        at clojure.lang.RT.classForName(
        at clojure.lang.RT.classForName(
        at clojure.lang.RT.loadClassForName(
        at clojure.lang.RT.load(
        at clojure.lang.RT.load(
        at clojure.core$load$fn__6824.invoke(core.clj:6126)
        at clojure.core$load.invokeStatic(core.clj:6125)
        at clojure.core$load.doInvoke(core.clj:6109)
Caused by: java.lang.ClassNotFoundException: org.objectweb.asm.Type
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.ja
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders
        at java.base/java.lang.ClassLoader.loadClass(
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(
        at clojure.lang.RT.classForName(
        at clojure.lang.RT.classForNameNonLoading(
        at Source)
        at<clinit>(Unknown Source)
        ... 226 more


The relevant line is this: java.lang.ClassNotFoundException: org.objectweb.asm.Type Any ideas?

Alex Miller (Clojure team)16:01:40

seems like you're missing the asm dep (transitive dep of tools.analyzer.jvm)

Alex Miller (Clojure team)16:01:31

does clj -Spath show the asm lib? if so, does the path it refers to actually exist?

Alex Miller (Clojure team)16:01:49

possibly relevant, how are you running your project


That's strange 😕, I ran clojure -Spath and I can't find any reference to either asm nor tools.analyzer.jvm. I think both libraries should be there because of core.async, right? Somehow core.async is not there either


it seems like you're AOT compiling as well


To run the project I tried both clojure -m my-app.core and java -jar with the uberjar (generated with uberdeps)


Yes, I'm AOT compiling indeed

Alex Miller (Clojure team)16:01:59

you can just jar tf the.jar to list the jar contents (might want to | grep for something). sounds like maybe the uberjar is incomplete?


Let me copy the relevant sections of my dockerfile:

FROM clojure:openjdk-8-tools-deps- AS BUILDER
WORKDIR /usr/src/app
COPY deps.edn .

# Download dependencies
RUN clojure -A:uberjar-deps -e '(println "Downloaded dependencies")'

# Copy the source code
COPY --from=local-libraries:latest /usr/src/deps /usr/src/deps
COPY src src
COPY test test

# AOT-compile
RUN mkdir classes
RUN clojure -A:local-libs -e "(compile 'my-app.core)"

# Generate uberjar
RUN clojure -A:local-libs -A:uberjar-deps -A:uberjar \
  --main-class my-app.core
And also my deps.edn
{:paths   ["src/" "classes/"]
 :aliases {:uberjar-deps {:extra-deps {uberdeps {:mvn/version "0.1.8"}}}
           :uberjar {:main-opts ["-m" "uberdeps.uberjar"]}
            {my-local-lib-1 {:local/root "/usr/src/deps/..."}}}}
 {org.clojure/clojure    {:mvn/version "1.10.0"}
  dali                   {:mvn/version "0.7.5"}
  org.clojure/data.json  {:mvn/version "0.2.6"}
  org.clojure/data.xml   {:mvn/version "0.0.8"}
  com.rpl/specter        {:mvn/version "1.1.2"}
  com.taoensso/carmine   {:mvn/version "2.19.1"}
  clj-pipeline           {:mvn/version "0.3.0-SNAPSHOT"}}}


@alexmiller Weird... If I just grep the jar contents as you suggested I can see the asm libraries, the clojure/asm/Type.class entry in particular seems to be in the jar


That's a different class @jsanchezf


than the one that was not found


oh, ok! thanks por pointing that out 😅


So, by grepping the jar I can definitely see tools.analyzer.jvm:

But if I grep for asm the only matches are in clojure/asm/*

Alex Miller (Clojure team)17:01:24

clojure includes a vendored (renamed) version of asm

Alex Miller (Clojure team)17:01:14

I think I would look to the stuff creating the jar as the likely culprit, not sure what there though


I don't know if the issue is in the jar. I tried running with clojure -m my-app.core and got the same error 😕

Alex Miller (Clojure team)17:01:10

Is the structure of that deps.edn right?

Alex Miller (Clojure team)17:01:40

Do you have a brace out of place so the deps aren’t being seen or something?


That may have been me editing the file after copying it to slack 😅 The same deps.edn project builds just fine locally. But I'll just double-check just in case


@jsanchezf I would docker run into every intermediate container produced by the build process and verify that what you expect is what's in the container


it's also a multi stage docker build but only one FROM directive in it


I want to make it multi-stage for dev/prod. But first I need to make it work 😅 That's why I only have a single FROM, but it's the whole file


it's broken then


or is local-libraries:latest a container?


oh, no, that's an image built by another Dockerfile, sorry about the confusion


maybe it's an excerpt


I think I've been able to isolate the issue a bit more. The problem seems to be that during the uberjar generation, the -A:local-libs is ignored.


I thought specifying multiple aliases, each one with its own :extra-deps would concatenate all the extra-deps vectors, is that not the case? That would mean my -A:uberjar-deps is overriding the previous alias and thus dropping some of my dependencies during the uberjar generation.


Your uberjar tool may be reading and parsing the deps.edn file itself, which would mean it ignores any options you pass to clj


it looks like it has its own --aliases option that is separate from clj's


Yes, that definitely seems to be it!


Indeed, it worked!! Thank you very much 😄🎉


For anyone that might've been following this, I'd also like to mention that the project built correctly on my machine, but that was before I decided to split my local dependencies into an alias, which was the source of the issue. I should've been more careful and tested with the new setup to rule that out 😅

Graham Seyffert21:01:47

Does anyone here have much experience with Eclipse .launch files and the various options it provides?…

Graham Seyffert21:01:11

I’m wondering if there’s an option to say “here’s the path for my source files”

Graham Seyffert21:01:05

I’m trying to get away from using AOT’d Clojure code in our app, but can’t do that if our launch config in dev can’t find the source files 😬


@gseyffert it should suffice to specify that src is a classpath element and clojure.main is the entrypoint class, with -m my.ns as args


even better, if you can have it ask clj or leiningen what the classpath should be


with depstar, I attempt to use a pom (made with clj -Spom ) so that java -jar ... will work with the resulting uberjar, but it crashes - is this a known issue, could it be related to using a local/root dep?


the tail of the output, it crashes after some normal clashing item warnings

{:warning "clashing jar item", :path "META-INF/ASL2.0", :strategy :noop}
{:warning "clashing jar item", :path "META-INF/services/com.fasterxml.jackson.core.JsonFactory", :strategy :concat-lines}
Execution error (NullPointerException) at hf.depstar.uberjar/copy-pom (uberjar.clj:264).


looks like that line of uberjar.clj is trying to string/replace on the main-class which is maybe null here...


@noisesmith Bug. Please open an issue on depstar on GitHub with as much repro info as possible please.


@seancorfield there's a lot of company specific stuff, but I'll try to find the time to do a generic repro


Even the above info is probably enough, I think, but any extra helps.

👍 4

I'll try to get a fix out quickly (but I'm looking after my wife, post-surgery, so it may not happen until the weekend).