This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-01-16
Channels
- # aleph (2)
- # announcements (1)
- # aws (2)
- # babashka (5)
- # beginners (122)
- # boot-dev (1)
- # cider (3)
- # clara (7)
- # cljdoc (11)
- # clojure (161)
- # clojure-dev (45)
- # clojure-europe (8)
- # clojure-france (1)
- # clojure-india (1)
- # clojure-italy (3)
- # clojure-nl (11)
- # clojure-uk (34)
- # clojurebridge (2)
- # clojurescript (13)
- # cryogen (10)
- # cursive (13)
- # datomic (25)
- # emacs (8)
- # fulcro (76)
- # graalvm (2)
- # jackdaw (5)
- # jobs-discuss (2)
- # juxt (13)
- # off-topic (13)
- # pathom (5)
- # pedestal (7)
- # quil (2)
- # reitit (9)
- # remote-jobs (4)
- # schema (1)
- # shadow-cljs (33)
- # spacemacs (8)
- # sql (9)
- # vim (2)
- # vrac (2)
Hello all, what is the best way to process same data with async process? Each process
will do things like logging, calculate something, transform to something with data. I tried to use https://github.com/clojurewerkz/meltdown but if I include some Thread/sleep
in one process
I see that it slow down other process
. How it processing can not depends each other?
Hi all! I'm trying to use tools.deps to pull a dependency from a private git repo. If I've understood everything correctly, there is no configuration required for private repos (other than having a valid ssh key). So basically, if I can git pull
the "git@..." url, I should be able to pull it with tools.deps, right?
However, I am hitting this error 😕 (most of the stacktrace ommitted for brevity, let me know if it might be relevant):
Cloning: :********REDACTED*******.git
Error building classpath. connector is not available:
com.jcraft.jsch.agentproxy.AgentProxyException: connector is not available:
at com.jcraft.jsch.agentproxy.ConnectorFactory.createConnector(ConnectorFactory.jav
a:120)
at clojure.tools.gitlibs.impl$fn__932.invokeStatic(impl.clj:30)
at clojure.tools.gitlibs.impl$fn__932.invoke(impl.clj:28)
at clojure.lang.Delay.deref(Delay.java:42)
at clojure.core$deref.invokeStatic(core.clj:2320)
at clojure.core$deref.invoke(core.clj:2306)
at clojure.tools.gitlibs.impl$call_with_auth.invokeStatic(impl.clj:49)
at clojure.tools.gitlibs.impl$call_with_auth.invoke(impl.clj:41)
at clojure.tools.gitlibs.impl$git_clone_bare.invokeStatic(impl.clj:71)
I've been digging a bit on the issue, and found some people that encountered the same error when running clojure inside a docker container (for example: https://github.com/tokenmill/docker-images/tree/master/meta-clojure-private-git-deps). Note that this is not my case.
Yes, though many systems run an ssh agent by default. In that case, you'd need to add a key to it with ssh-add
.
Hmm.. strange. It seems ssh-add
returns "Could not open a connection to your authentication agent." So this might be it
Oh, if this is inside a container, chances are any ssh agent run in the host system is inaccessible.
Ok, that did it! I started ssh-agent (as described in https://wiki.archlinux.org/index.php/SSH_keys#SSH_agents) and I was able to pull the git dep!
This is all ssh-related, nothing specific to clojure or even git, but if you don't provide any auth info (eg, password), ssh will look for an identity/keypair with one of the default names; often ~/.ssh/id_dsa
or ~/.ssh/id_rsa
; if those keys are encrypted, it will ask for a password.
This is where ssh-agent comes in: you can store 1 or more private ssh keys, decrypted if necessary (`ssh-add` will ask for the passphrase on adding), and any ssh session that can find the ssh-agent will try those keys to establish a session.
I currently don't have a passphrase, so I assumed tools.deps would just get my ~/.ssh/id_rsa
just like git cmdline does, but it seems that is not the case.
Huh. Looks like it's using org.eclipse.jgit
underneath. Maybe it doesn't do any identity lookup.
I have another question. Are transitive dependencies not available when using tools.deps? I have two projects, project B depends on project A via git, and project A depends on core.async (via regular mvn dependency). When I open a repl in project B, it fails to load because B was also using core.async (without a direct dependency). Is this the intended behavior?
@penryu Actually, by checking the issues, I've seen they're trying to migrate to just shell out to git cli to solve most current issues
What is the best way to deal with coercions/validation in a REST API in clojure? I was trying to use compjure-api + spec-tools + clojure.spec.alpha, but things just don 't work as expected. My undestanding is that spec was not created for coercion at application boundaries. But I don't know what to use instead.
@zara I've used schema in the past. They explicitly support coercion: https://github.com/plumatic/schema
So do you use Schema for API and spec to document and validate data inside your programs?
we use schema for both @zara, although we've also been using schema since before spec was a thing, and i'd rather not have both schema+spec versions of datastructure shapes, so i'm not in a hurry to change
juxt/yada
supports schema for API coercion and description (i.e. swagger)
To be honest, I don't have a good answer for that. I used schema before spec even existed. Currently I use spec, and since I'm working mostly on an all-clojure codebase, I have no need for coercion.
We use nippy to communicate between services, instead of encoding/decoding JSON: (https://github.com/ptaoussanis/nippy)
Is there a way to define something like this in Schema?
(s/or
:measures (s/keys :req-un [::name ::length ::width ::height])
:predefined (s/keys :req-un [::name ::predefined_package]))
The issue about coercion when using spec vs schema is interesting to me, as I'm planning to use spec but haven't started yet. Say that JSON is coming in from the outside. With schema there is a way to coerce it during validation, but not with spec. So with spec it must be coerced prior to validation. That seems straightforward to me, but perhaps I'm missing something because I haven't tried it yet. In your experience does coercion often need to be done differently, depending on the particular spec? I was hoping to simply convert JSON to maps and vectors in a generic way, and use spec to validate the result.
I looked at the schema blog post about coercion: http://plumatic.github.io//schema-0-2-0-back-with-clojurescript-data-coercion And I think I can do the coercion generically (before validating with spec) as long as I convert JSON types to lowest common denominator types (e.g., Number rather than Long), and don't support convenience conversions (e.g., String to Keyword).
I usually do the coercion on the parser (like cheshire or jsonista), then validate the json parsed object
Thanks @U6CN6JQ22, I'll take a look at that.
Folks, is that a way to get a octet-stream and read it?
i got a .docx
by request, the content type is octet-stream, i'm using the byte-streams library and was able to transform it on a ByteArrayInputStream, but i'm kind of lost in how to transform it on a docx file again.
you can use
to write that ByteArrayInputStream to a
(with-open [xin (bs/to-input-stream (get-template))
xout ( "resources/test.docx")]
(io/copy xin xout))
I tried to do something with it
well, resources is a weird destination
and I don't think output-stream is that convenient for files
I'd explicitly do (io/output-stream (io/file "/some/path/test.doxc"))
I'm just goofing around for a test.
Not something that would go for a project
the reason I say resources is weird, is in most setups it's a "magic" folder that lets you read files locally but read from inside a jar in prod
oh, I guess output-stream is that convenient
(ins)user=> (with-open [o (io/output-stream "foo.txt")] (io/copy (io/input-stream "user.clj") o))
nil
(ins)user=> (= (slurp "foo.txt") (slurp "user.clj"))
true
Got it
I'm just creating a POC, so no rules haha
Saying this about the resources stuff
cool - just mentioning because it's a really common problem for new clojure users (if you don't make the resources mistake, someone else reading this thread will, and it's good to be aware of why it happens and how to avoid it)
Thanks
(defn test []
(with-open [xin (bs/to-input-stream (get-template))
xout (io/output-stream (io/file "resources/test.docx"))]
(io/copy xin xout)
(slurp "resources/test.docx")))
My file is corrupt
i can pass the outputstream with no issues and my slurp brings me a empty string
When i open on word, it's corrupted
the file doesn't get flushed until the with-open exits, for one thing
So with-open
do the flush job?
one thing to look out for is encoding - did it get base64 encoded over the wire? is some step using a text function (you should be using binary ops, not text ops, on a docx)
the with-open flushes, but the slurp is guaranteed to happen before the flush in your example
and slurp is convenient, but it's not for binary data like docx, it's for text
Got it
Let me show you my response body
what is (get-template)
returning?
"PK \b\b\b ?n/P
_rels/.rels??MKA\f???C??l+????\"Bo\
"???????3i???A\n?P???y???m???N???AÛiAq0?0jx?=/`?/?W>??J?\\*???aI???L
?41q??!fOR?<b\"???q???2??1??j?[???H?76z?$?&f^?\\??8.Nyd?`?y?q?j4?\f
x]h?{?8\f??S4G?A?y?Y8X???(?[Fw?i4o|??l?^?????PK\b??#? = PK \b
\b\b ?n/P docProps/core.xml?R]O?0}?W,}??`L?F??'IL?h|??e
T?i
?o????o??sz?Wg?=X'k?#??(?k!u????\"?C??L
V?rt????k
?6`?#?(7
9?zo(?oA1????? m?\r?_?<I?[??3?<ÝalFGt?|?4;[u?c?@??????U???P*??Jr
T??M?$M?IC??-?V????8?bvj?r
?????\r?k??^?b?N?8%1??IF??f??\f?z??qm??
=?
p?J??\r{?G\"???r^??_V?dL??????p?q\fWrCG???H%?#\r]e
{????tEG?
v?v??}??\bB??O??X|PK\b??pM` ? PK \b\b\b ?n/P d
ocProps/app.xml?QAN?0?????8M?4?W\b?\t\t$?V9?&1Jl?6??=.?B??f<??]/??!:
?uR?\n-?E??Ruz?%??gJ?A+????b??%?($(W??{????F?? ?????j;??Vrx?
?k?q?$??%@,??????o???2?{??&?QR?h??k??P??2<O??3H?|??$
???8???
TB??GY?<?Ya?O?o?&K`?????I?<]mr^??u+??u?????
??]????V?|E?g?q?
E?`?????5???u67?PeYg???.}
d??PK\b-]?: PK \b\b\b ?n/P
word/_rels/document.xml.rels??M\n?0???\"??U??nDp+? 1???6\
t?(z{?Z(????}?1/__??]?m??,I??Q??p(??%??I??NR\\\t?v??Dn?yP-?2$????
?^R,}¦T'? ????O&?U????7???m]k??=???n?H??A??>.???|m\r????????
?????@????????I??wPK\b?/0?? PK \b\b\b ?n/P wo
rd/document.xml?U?n?0??+?c?[B? ????~ M?$?|aIYv??K=??(???\n\b?????
????Q?? ?5k?\\d,#l!M?f_w?nX?7W?????????6/?h4????]?M?E\r??+-Zo?p
%??mYJ????fu\b.O?a??:0?+-jh?U?/????,{?\"(?????#??w?Z?u?%][??C+?{2
B?????L0???gZ?.?\\ o??|Nd?'gD9?X??????
??;???}B?M?K?j???F?R?
p?????z?????G??se,?@@Id?6???8??u??\r?? i?WkvU+?v??c<?C??PP?>?'??
?rm?m??;.??C??`?]-}BOH?T\t?v???P4 My?E?B????{j??]\r\b?{&?!pQw?]?.
??uf???\"\f5'7?1p\f?????P?????????6?????U6|?H????X????Lk?`TD?`
?)?\rSfoC?zNVM?C??F?z??&????????0?(??@??I.]M?????t:??E:???PK\bx?@
?0 ? PK \b\b\b ?n/P word/styles.xml?UMO??W?|
(bA4\"M?=N???????»???BH6?j/Y?????{??????:Fl??2??B?\nvs}?;fy?
%H??`K$vv??d1 ??HY??4Xl??9?9*?=cQ?Xe??n?/?+?3?By%?~?(W 4k??nR?;C
??{??T???J???~Z)?P?5D????B=
^L?~??L??h????\f?>?4?Z?+????u?\\?mv?q
a??l1 ?B?JL??Fgt?b!4??#?@??l<jR?\br\n/?\\??M@??A??t??,??\n??G?E??
?I?C??<??a?#4e 9??8&?M??f?vs
Q??0H?\\%?&y??|K?4N?p??Ax
f?<?I?Q
?????m/\r?z?}?|?w3Z???????%7??
???w???5?!?[???Jn ,??.?4????/?\
\????????Q???)????H*?.\\ko?9Z??ryR?[{Ce???N??,<???#N<??iE?}?
F?'1??????`??\tL??5??)}??{???J`W/?h???r?Ó??????E??_?c????????Va?h?
??~Ów?X????{???Z?M?????PK\b??t| \t PK \b\b\b ?n/P
word/fontTable.xml?PAN?0??\n?w??BQ??z??[g?X???4???N#!? ????
?????Yq???S%?f?H???|?=?>H???\t+yB????/?EZ'.?J?1v?R?[t?3?!%???AL?p
P?u?F??Är`H^d?52?i??G??RDZ??nM?ryI'?????;??{??P&??s[??*?3?4
NC?g?3Q???\b????R??7?????????:Q??&???0?????1???i?+?I??o?/?N?S??????
.^~ PK\b??;b PK \b\b\b ?n/P word/settings.xml
E?K?0\fD??\"?X?H???Bk?RbG????\t+??73z??yb????r?? ?<?t?p>?[0?????
?v??A??SH???]57?J?d???+???r??!?Q?NS?+??6??????d?&c??8???G
?S??s?<C??q????
PK\bv??? ? PK \b\b\b ?n/P [Con
tent_Types].xml???N?0E????[???@\b%???\bkd?Ij?????=?4?P??e<s?N???MV
?4? ??$??R7y???+2+'y????? ??5??/@1?
+?q?|t\r??????tzI??tHC? e~
5[?!?[??r??l?\"? ??Vr?Lc?????p???ti?Y???/??g_?n? R???????░?+???vR
@2g.<0?\r?9nB??3D?????Av8???N-?
?#??????%?X*?d? ?d?pw??Aw?
??_??pe??*?I=:????????kDV???
76??z<\b5?B??0?i??,? PK\b
??T
^ PK \b\b\b ?n/P??#? =
_rels/.relsPK
\b\b\b ?n/P??pM` ? docProps/core.xmlPK \b\b
\b ?n/P-]?: ? docProps/app.xmlPK \b\b\b ?n/
P?/0?? ) word/_rels/document.xml.relsPK \b\b
\b ?n/Px?@?0 ? 8 word/document.xmlPK \b\b\b ?n/
P??t| \t ? word/styles.xmlPK \b\b\b ?n/P??;b
U `\n word/fontTable.xmlPK \b\b\b ?n/Pv??? ?
?
word/settings.xmlPK \b\b\b ?n/P
??T ^
?\f [Content_Types].xmlPK \t \t < 4 "This code above
a octet-stream
that's not a java type though, what does (type (get-template))
actually return?
(defn get-template []
(:body (client/get (:oci-url env))))
This is my function
It shoud return a .docx
file that contains some text to be used as a template for a email
OK clj-http.client/get will return a hash-map, and the :body
key has an input-stream
unless you are using middleware that messes with it
What you mean by middleware?
I don't know if you understand that there's literally a function clojure.core/type
that tells you the class of an object
clj-http uses middleware to modify its behavior and add features, this can include changing the body of a response
or even automatically putting it in a file
I had no idea of it
My coleague is using a .txt and worked.
Probably the issue is with my document
there are things that are safe for text that corrupt binary files - try a text file and a jpg too
Thanks for the tips
I think I found your problem
user=> (type (:body (client/get "")))
java.lang.String
I seee
haha! slack is terrible
anyway - it's a string of the contents, which means that client/get is corrupting the contents
Got it
What would you think it should be the best approach for it?»
https://github.com/dakrone/clj-http#output-coercion here's the relevant documentation
`
(client/get "" {:as :byte-array})
`this should do the right thing
then the io/copy will work etc.
I'll test that
this worked for me:
user=> (io/copy (:body (client/get "" {:as :byte-array})) (io/file "google.png"))
nil
It worhed
worked!!
(io/copy already promises to close a file if it opens it, so no with-open needed)
awesome - glad I could help
I really appreciate your help 😄
you can also use {:as :stream}
- that might even be a perf / resource usage improvement if that matters (with byte-array it's consuming a stream for you and putting it in a new byte array for you, with as stream you get to decide what you do with the stream, which can include using the input in a way that is less wasteful of memory -- probably doesn't matter for your use case though)
another way to put it: with stream, you directly put the stream into a file, with byte-array it's putting the data into an array, then you turn the array into a stream and put that into a file - so there's steps being done that aren't needed
Hmm, I was going to recommend docjure, but it looks like that only handles Excel files. Maybe you can go directly to the Apache POI site and find something you can use with Java interop? https://poi.apache.org/apidocs/index.html
That's what i'm trying to do, i got a library for get the bytes and put into a inputStream
I'm trying to build my project inside docker using tools.deps and got a very weird issue. I'm able to build and run my project locally. However, inside docker, after everything is built successfully, attempting to run my project yields this obscure java error:
Exception in thread "main" java.lang.ExceptionInInitializerError
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:416)
at clojure.lang.RT.classForName(RT.java:2207)
at clojure.lang.RT.classForName(RT.java:2216)
at clojure.lang.RT.loadClassForName(RT.java:2235)
at clojure.lang.RT.load(RT.java:453)
at clojure.lang.RT.load(RT.java:428)
at clojure.core$load$fn__6824.invoke(core.clj:6126)
at clojure.core$load.invokeStatic(core.clj:6125)
at clojure.core$load.doInvoke(core.clj:6109)
(...)
Caused by: java.lang.ClassNotFoundException: org.objectweb.asm.Type
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.ja
va:602)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders
.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:416)
at clojure.lang.RT.classForName(RT.java:2207)
at clojure.lang.RT.classForNameNonLoading(RT.java:2220)
at clojure.tools.analyzer.jvm.utils$loading__6706__auto____8809.invoke(utils.clj:9)
at clojure.tools.analyzer.jvm.utils__init.load(Unknown Source)
at clojure.tools.analyzer.jvm.utils__init.<clinit>(Unknown Source)
... 226 more
The relevant line is this: java.lang.ClassNotFoundException: org.objectweb.asm.Type
Any ideas?
seems like you're missing the asm dep (transitive dep of tools.analyzer.jvm)
does clj -Spath
show the asm lib? if so, does the path it refers to actually exist?
possibly relevant, how are you running your project
That's strange 😕, I ran clojure -Spath
and I can't find any reference to either asm nor tools.analyzer.jvm. I think both libraries should be there because of core.async, right? Somehow core.async is not there either
To run the project I tried both clojure -m my-app.core
and java -jar
with the uberjar (generated with uberdeps)
you can just jar tf the.jar
to list the jar contents (might want to | grep for something). sounds like maybe the uberjar is incomplete?
Let me copy the relevant sections of my dockerfile:
FROM clojure:openjdk-8-tools-deps-1.10.1.483-stretch AS BUILDER
WORKDIR /usr/src/app
COPY deps.edn .
# Download dependencies
RUN clojure -A:uberjar-deps -e '(println "Downloaded dependencies")'
# Copy the source code
COPY --from=local-libraries:latest /usr/src/deps /usr/src/deps
COPY src src
COPY test test
# AOT-compile
RUN mkdir classes
RUN clojure -A:local-libs -e "(compile 'my-app.core)"
# Generate uberjar
RUN clojure -A:local-libs -A:uberjar-deps -A:uberjar \
--main-class my-app.core
And also my deps.edn
{:paths ["src/" "classes/"]
:aliases {:uberjar-deps {:extra-deps {uberdeps {:mvn/version "0.1.8"}}}
:uberjar {:main-opts ["-m" "uberdeps.uberjar"]}
:local-libs
{:extra-deps
{my-local-lib-1 {:local/root "/usr/src/deps/..."}}}}
:deps
{org.clojure/clojure {:mvn/version "1.10.0"}
dali {:mvn/version "0.7.5"}
org.clojure/data.json {:mvn/version "0.2.6"}
org.clojure/data.xml {:mvn/version "0.0.8"}
com.rpl/specter {:mvn/version "1.1.2"}
com.taoensso/carmine {:mvn/version "2.19.1"}
clj-pipeline {:mvn/version "0.3.0-SNAPSHOT"}}}
@alexmiller Weird... If I just grep the jar contents as you suggested I can see the asm libraries, the clojure/asm/Type.class entry in particular seems to be in the jar
That's a different class @jsanchezf
So, by grepping the jar I can definitely see tools.analyzer.jvm
:
...
clojure/tools/analyzer/jvm$_deftype$fn__10823.class
clojure/tools/analyzer/jvm$desugar_host_expr.class
clojure/tools/analyzer/jvm$build_ns_map$fn__10736.class
clojure/tools/analyzer/jvm$parse_deftype_STAR_$fn__10845.class
clojure/tools/analyzer/jvm$parse_import_STAR_.class
clojure/tools/analyzer/jvm$analyze_PLUS_eval$fn__10925.class
clojure/tools/analyzer/jvm$loading__6706__auto____10313.class
clojure/tools/analyzer/jvm$desugar_symbol.class
clojure/tools/analyzer/jvm$analyze_method_impls.class
...
But if I grep for asm
the only matches are in clojure/asm/*
clojure includes a vendored (renamed) version of asm
I think I would look to the stuff creating the jar as the likely culprit, not sure what there though
I don't know if the issue is in the jar. I tried running with clojure -m my-app.core
and got the same error 😕
Is the structure of that deps.edn right?
Do you have a brace out of place so the deps aren’t being seen or something?
That may have been me editing the file after copying it to slack 😅 The same deps.edn project builds just fine locally. But I'll just double-check just in case
@jsanchezf I would docker run
into every intermediate container produced by the build process and verify that what you expect is what's in the container
I want to make it multi-stage for dev/prod. But first I need to make it work 😅 That's why I only have a single FROM, but it's the whole file
I think I've been able to isolate the issue a bit more. The problem seems to be that during the uberjar generation, the -A:local-libs
is ignored.
I thought specifying multiple aliases, each one with its own :extra-deps would concatenate all the extra-deps vectors, is that not the case? That would mean my -A:uberjar-deps is overriding the previous alias and thus dropping some of my dependencies during the uberjar generation.
Your uberjar tool may be reading and parsing the deps.edn file itself, which would mean it ignores any options you pass to clj
https://github.com/tonsky/uberdeps/blob/master/README.md#project-setup--extra-aliases
For anyone that might've been following this, I'd also like to mention that the project built correctly on my machine, but that was before I decided to split my local dependencies into an alias, which was the source of the issue. I should've been more careful and tested with the new setup to rule that out 😅
Does anyone here have much experience with Eclipse .launch
files and the various options it provides?…
I’m wondering if there’s an option to say “here’s the path for my source files”
I’m trying to get away from using AOT’d Clojure code in our app, but can’t do that if our launch config in dev can’t find the source files 😬
@gseyffert it should suffice to specify that src
is a classpath element and clojure.main
is the entrypoint class, with -m
my.ns
as args
even better, if you can have it ask clj or leiningen what the classpath should be
with depstar, I attempt to use a pom (made with clj -Spom
) so that java -jar ...
will work with the resulting uberjar, but it crashes - is this a known issue, could it be related to using a local/root dep?
the tail of the output, it crashes after some normal clashing item warnings
{:warning "clashing jar item", :path "META-INF/ASL2.0", :strategy :noop}
{:warning "clashing jar item", :path "META-INF/services/com.fasterxml.jackson.core.JsonFactory", :strategy :concat-lines}
Execution error (NullPointerException) at hf.depstar.uberjar/copy-pom (uberjar.clj:264).
null
looks like that line of uberjar.clj is trying to string/replace on the main-class which is maybe null here...
@noisesmith Bug. Please open an issue on depstar
on GitHub with as much repro info as possible please.
@seancorfield there's a lot of company specific stuff, but I'll try to find the time to do a generic repro
I'll try to get a fix out quickly (but I'm looking after my wife, post-surgery, so it may not happen until the weekend).