This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-05-14
Channels
- # beginners (19)
- # boot (28)
- # cider (1)
- # cljs-dev (12)
- # cljsrn (1)
- # clojure (246)
- # clojure-belgium (4)
- # clojure-berlin (1)
- # clojure-france (4)
- # clojure-italy (7)
- # clojure-russia (11)
- # clojure-serbia (3)
- # clojure-spec (5)
- # clojurescript (106)
- # cursive (4)
- # datomic (7)
- # emacs (1)
- # jobs (6)
- # leiningen (3)
- # luminus (1)
- # lumo (11)
- # off-topic (1)
- # om (7)
- # onyx (1)
- # pedestal (6)
- # re-frame (16)
- # reagent (4)
- # rum (3)
- # spacemacs (1)
- # unrepl (1)
- # untangled (5)
- # vim (1)
@taylor You could use symbols. Alternatively, use keywords and a multimethod for dispatch. That would ensure that your edn file isn’t tied to a specific namespace structure.
Unrelated to my previous message, I have a data structure that uses metadata. I’m using spec to help me parse it, but of course spec ignores the metadata. The current solution I have is to use a custom conformer, but this messes up explain
. Does anyone have a better suggestion?
weavejester: I saw someone ask a related question in #clojure-spec the other day: https://clojurians.slack.com/archives/C1B1BB2Q3/p1494345116258721
Ah, thanks
Hm, no solution, though.
(defn- meta-conformer [spec]
(s/conformer #(let [val (s/conform spec %)]
(if (= val ::s/invalid)
::s/invalid
{:value val, :meta (meta %)}))))
@weavejester I was actually wondering the same thing as taylor recently. How do you unquote the symbol? Assuming we're in the correct namespace?
@john this looks relevant http://stackoverflow.com/a/7355158 I’ve never seen resolve
fn before 🙂
find-var
or resolve
, usually. And you can require
the namespace
of the symbol automatically.
My suggestion would be to use multimethods. I don’t think it’s a good idea to attach edn to Clojure code directly, anyway.
We have data readers that are separate, so I think we should take the same approach with functions.
You could also have something that walks the tree and substitutes values for symbols. I’ve had a lot of success with multimethods, however, such as in Integrant.
Which might be worth looking at if you want a configuration that can be loosely attached to code.
@weavejester thanks, looking at it now. Do you mind explaining a bit what’s going on with #ig/ref
in :handler #ig/ref :handler/greet
? Specifically, what does the #
indicate there?
@taylor #
is a tag in edn. It’s a way of augmenting basic structures into more specific implementations. For example, #inst
modifies strings to turn them into times, #inst "2017-05-14T00:17:51.749-00:00"
.
In Integrant, #ig/ref
is a tag that indicates that a keyword refers to another top-level key.
I am trying to figure why I can’t seem to call this method from commons-codec
: https://github.com/apache/commons-codec/blob/trunk/src/main/java/org/apache/commons/codec/binary/Hex.java#L225
I am trying to use buddy-core .. which uses this library and am trying to narrow down why I can’t seem to call this method. Here’s a REPL session which seems to not show this method.
@verma are you sure you aren't just doing this?
+user=> Character/isUpperCase
CompilerException java.lang.RuntimeException: Unable to find static field: isUpperCase in class java.lang.Character, compiling:(NO_SOURCE_PATH:0:0)
+user=> (Character/isUpperCase \a)
false
methods are not first class
not even static methods
you’re correct, it does seem like when I call it it seems to think its there .. I forgot to include that in REPL stuff, didn’t know it was the same for Character
class as well, good point
user=> (Hex/encodeHexString (.. "hello" getBytes))
CompilerException java.lang.IllegalArgumentException: No matching method: encodeHexString, compiling:(/private/var/folders/rw/7cgwzwhj1bvgqjdn25ygjsvc0000gn/T/form-init8722891713944001492.clj:1:1)
well, how would bytes be a string?
I don't know that method, but that seems odd to me
oh, never mind https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/binary/Hex.html#encodeHexString(byte[])
this is where the problem started, this is failing to compile: https://github.com/funcool/buddy-core/blob/master/src/buddy/core/codecs.clj#L40
my point is that static methods are not first class
it isn't valid to return a method
that's the error in your initial paste you shared (your other error is another issue of course, I'm playing with it now)
codecing-deleteme.core=> (import org.apache.commons.codec.binary.Hex)
org.apache.commons.codec.binary.Hex
codecing-deleteme.core=> (Hex/encodeHexString "hello")
ClassCastException java.lang.String cannot be cast to [B codecing-deleteme.core/eval1346 (form-init4539385857918328383.clj:1)
codecing-deleteme.core=> (Hex/encodeHexString (.getBytes "hello"))
"68656c6c6f"
oh wait, .. works for me too
what version of commons-codec do you have?
mine is the latest I think [commons-codec/commons-codec "1.9"]
oh, not the latest then, haha
I just tried 1.10, works there too
so something in your system is weird - have you run lein clean
recently?
maybe you should if you haven't (/me is out of ideas)
oh, I was totally wrong on .. above, I was thinking of doto (d'oh)
oh, so your project is hosed
@john clojure.repl/doc is your friend!
=> (doc ..)
-------------------------
clojure.core/..
([x form] [x form & more])
Macro
form => fieldName-symbol or (instanceMethodName-symbol args*)
Expands into a member access (.) of the first member on the first
argument, followed by the next member on the result, etc. For
instance:
(.. System (getProperties) (get "os.name"))
expands to:
(. (. System (getProperties)) (get "os.name"))
but is easier to write, read, and understand.
nil
john: haha
$ rlwrap lumo -d
Lumo 1.2.0
ClojureScript 1.9.482
Docs: (doc function-name-here)
Exit: Control+D or :cljs/quit or exit
+cljs.user=> (doc ..)
-------------------------
cljs.core/..
([x form] [x form & more])
Macro
form => fieldName-symbol or (instanceMethodName-symbol args*)
Expands into a member access (.) of the first member on the first
argument, followed by the next member on the result, etc. For
instance:
(.. System (getProperties) (get "os.name"))
expands to:
(. (. System (getProperties)) (get "os.name"))
but is easier to write, read, and understand.
nil
cljs.repl / clojure.repl should really be taught thoroughly in people's first intro to the language...
see also apropos
and find-doc
I used to use it. But many cljs repls - once you're outside user
- you have to type out the whole cljs.repl/doc
so I've gotten in the habit of using web docs
(use 'cljs.repl)
there's no way to make it universal, but it's a quick thing to type out
/Users/verma/.m2/repository/commons-codec/commons-codec/1.10/commons-codec-1.10.jar
seems to be the only reference in class path dealing with commons-codec
well, ring-mock is silly anyway
(I mean yes, it's convenient, but you can just use hash-maps)
are you running deps tree with the same profile that runs the repl you are using?
that could be a gotcha
maybe lein needs an :anti-deps key
"never load this version of this lib"
here’s the deps tree: https://gist.github.com/verma/0be2f8201bb33a59c583c88bf554974d
could it be that ring-mock somehow bundles commons-codec so you can't override?
looks pretty basic: https://github.com/ring-clojure/ring-codec/blob/master/project.clj#L7
what about a PR to get them to use a newer commons-codec?
are you sure the double vector for exclusions works?
just checking - because an exclusion would never have other data needing a vector, and the sample shows no vector
oops, no, there are valid uses of a vector for exclusions, my bad
compojure and ring-core are trying to use ring-codec which uses older version of commons-codec, but lein deps :tree is not reporting those
however, the suggestion here is working wonderfully: http://stackoverflow.com/questions/33093833/display-complete-dependency-tree-with-leiningen
fascinating
although mvn says that 1.6 has been omitted, but I feel that’s more relevant for uberjar purposes may be 😕
hey, you could try lein uberjar; java -cp target/*standalone.jar clojure.main
@verma wouldn't a top level :exclusions key in project.clj do that?
yeah, just like under a dep
none of them need to include it, if you explicitly ask for the version you need
and I'd rather have one exclusion and an explicit dep, rather than needing multiple exclusions
finally I can move on, @noisesmith thanks for your help and support
dominicm: he started with that, it was showing an erroneous output
manytimes, I've founds lein vizdeps
(https://github.com/walmartlabs/vizdeps) to be more useful to see all the dependencies and quickly identify possible conflict
lxsameer: you can set keys in the environment with the help of env
, and in an uberjar the project.clj will be visible via io/resource
exactly
+~$ lein new app foo
Generating a project called foo based on the 'app' template.
+~$ cd foo
+~/foo$ lein uberjar
Compiling foo.core
Created /home/justin/foo/target/uberjar/foo-0.1.0-SNAPSHOT.jar
Created /home/justin/foo/target/uberjar/foo-0.1.0-SNAPSHOT-standalone.jar
+~/foo$ rlwrap java -cp target/uberjar/foo-0.1.0-SNAPSHOT-standalone.jar clojure.main
Clojure 1.8.0
+user=> (require '[ :as io])
nil
+user=> (slurp (io/resource "project.clj"))
"(defproject foo \"0.1.0-SNAPSHOT\"\n :description \"FIXME: write description\"\n :url \""\n :license {:name \"Eclipse Public License\"\n :url \""}\n :dependencies [[org.clojure/clojure \"1.8.0\"]]\n :main ^:skip-aot foo.core\n :target-path \"target/%s\"\n :profiles {:uberjar {:aot :all}})\n"
user=>-
Hey guys I got a luminous project setup with compojure-api. Currently all my responses are in JSON except for internal server errors when the schema is invalid. How do I get these internal server errors in JSON as well?
i was trying do such things yesterday but i failed so i am asking again here today 🙂
I have queue and i want get from this queue max 20 items on each iteration (when 10 items i want get 10 items and do job, not wait for 20). How to do it in most simple way? After each iteration i want do X seconds break.
The hard part is i want do job when fill to 20 items or after X seconds from last run, even where is 10 items in queuethis is very ugly, but it works
(require '[clojure.core.async :as >])
(def c (>/chan 40))
(defn deliver-result
[items]
(>/go (println (java.util.Date.) "found" items)
(>/<! (>/timeout 5000))))
(def l
(>/go-loop [found []]
(let [next-item (delay (>/poll! c))]
(cond
;; c is closed
(and (seq found)
(nil? (peek found)))
(deliver-result (conj (pop found) ::done))
;; we have our quota
(= (count found) 20)
(do (>/<! (deliver-result found))
(recur []))
;; we don't have quota, c isn't closed, and there was more on the channel
(some? @next-item)
(recur (conj found @next-item))
;; there was nothing on the channel, but we have a partial result
(seq found)
(do (>/<! (deliver-result found))
(recur []))
;; we had no partial result, wait for more on the channel...
:else
(recur (conj found (>/<! c)))))))
(>/onto-chan c (range 55) false)
(Thread/sleep 30000)
(>/onto-chan c (range 22) true) ; closes c, stops l
(>/<!! l)
hmm are you sure? I mean onto-chan
close channel on the end, but in real situation c
channel wouldn’t be close
the example demonstrates that it still returns partial results
I only close the channel to show that l exits on that condition
$ bench /tmp/part.clj
#inst "2017-05-14T12:07:26.352-00:00" found [0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19]
#inst "2017-05-14T12:07:31.377-00:00" found [20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39]
#inst "2017-05-14T12:07:36.381-00:00" found [40 41 42 43 44 45 46 47 48 49 50 51 52 53 54]
#inst "2017-05-14T12:07:56.356-00:00" found [0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19]
#inst "2017-05-14T12:08:01.357-00:00" found [20 21]
#inst "2017-05-14T12:08:06.359-00:00" found [:user/done]
the timestamps show no extra delay is added, the partial results show that it can return results before the full 20 if less than 20 are available
(def queue (async/chan))
(def queue-portion (async/chan 100 (partition-all 20)))
(async/pipe queue queue-portion)
This partition values by 20, but problem is when less than 20. queue
channel has to be close to run job with less than 20right, which is why this code doesn't use partition all
look at the output, it works
async/take has the same problem
right, there's nothing built in that has the "as much as available" logic, so I had to construct it using poll!
no matter what you use, you'll need to balance your timeout logic with the batching, which is inherently compex
using poll!, you can figure out how much is available, and via the channel read condition we can still figure out the channel is closed (which is still useful - you don't want to ever loop on a closed channel)
there might be a way to improve it by separating the batching from the as-much-as-available logic...
cleaner version, separates getting as much as possible from the channel from the batching logic
(require '[clojure.core.async :as >])
(def c (>/chan 40))
(defn deliver-result
[items]
(>/go (println (java.util.Date.) "found" items)
(>/<! (>/timeout 5000))))
(defn as-much-as-available
[c n]
(>/go-loop
[found []]
(if (>= (count found) n)
found
(if-some [el (>/poll! c)]
(recur (conj found el))
found))))
(def l
(>/go-loop
[found []]
(if (= found [nil]) ; this means c is closed
(deliver-result [::done])
(let [more (>/<! (as-much-as-available c (- 20 (count found))))]
(>/<! (deliver-result (into found more)))
(recur [(>/<! c)])))))
(>/onto-chan c (range 55) false)
(Thread/sleep 30000)
(>/onto-chan c (range 22) true) ; closes c, stops l
(>/<!! l)
See? You just gotta catch @noisesmith when he's around 🙂 That's a pretty good solution
Here's an atom/add-watch solution along a similar vein:
(defn add-to-q [q items]
(apply conj q items))
(defn work [f xs n]
(mapv
#(do (Thread/sleep n)
(f %)
(vec %))
(partition-all 20 xs)))
(defn sub-work-unit [x]
(println
(apply str (java.util.Date.) " found " (interpose " " x))))
(def in-q (atom []))
(def out-q (atom []))
(defn process-q [w]
(let [res (work sub-work-unit w 1000)]
(swap! out-q into res)
res))
(add-watch in-q :work-q #(process-q %4))
(defn run-q [items]
(swap! in-q #(add-to-q % items))
@out-q)
(run-q (range 44))
which results in:
Sun May 14 13:12:47 EDT 2017 found 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Sun May 14 13:12:48 EDT 2017 found 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
Sun May 14 13:12:49 EDT 2017 found 40 41 42 43
[[0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19] [20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39] [40 41 42 43]]
@wimomisterx hi, the exception handling is described here: https://github.com/metosin/compojure-api/wiki/Exception-handling - the default exception handler should return JSON too. There is #ring-swagger for c-api discussion.
how do I check whether a given object is a float array? https://clojuredocs.org/clojure.core/float-array
=> (= "class [F" (str (class (float-array []))))
true
@qqq there's no other way to do it, because the class literal for float array is not readable
you could compare equality of the class and another float-array's class, that's even more hacky to me - an array that only exists so you can compare its class to something
I want use async
to queue to run functions. (this question is other topic than my question before). I think the easiest way would be do something like (add-to-queue #(orders/sync shop-id erp-id))
and in <!
just run functions (f)
. I want >!
there different functions which can’t work in the same time. Is it good idea? How would you do this?
On the other hand i could run specific functions based on some key like :sync
or :another-function
, but then i will have to care about parameters etc. Also editor wouldn’t give me hints about parameters.
So mainly i want just confirm what i am doing is right pattern design or i am doing something antipattern or just bad design 🙂
@kwladyka My inclination would be to make the items in the queue data instead, then use a multimethod to dispatch off a type.
e.g. (>!! queue #:event{:type :order, ...})
thx for suggestion. What advantage vs my solution? As disadvantage i see lost hints as parameters to functions. Less clear what data should be stored in queue.
You can define a spec that validates the event data. The main advantage is that the data in your queue is transparent. Easier to debug problems, and it can be more easily distributed or made persistent.
Functions can be a little opaque if they’re being passed through a queue.
Oh, it’s just Clojure 1.9 shorthand for {:event/type :order, …}
.
#:foo.bar{:x 1, :y 2}
= {:foo.bar/x 1, :foo.bar/y 2}
You’re missing the #
and it’s only for Clojure 1.9.
It’s just shorthand - you can always write it out the long way.
@kwladyka I’ve done something similar using a “processing pipeline” discussed here http://www.braveclojure.com/core-async/. I put wrapped fns on it and called like you suggested and it seems to work well
thx, actually there is a pipe
fucntion to do this. Also pipeline
but this one i didn’t use.
I had made a queue at one point but removed it because it didn’t seem necessary for my use case
i've been getting hung up on how map
seems to pull individual characters out of the map
@tstirrat Yes. I believe if you want to use map
on a string seq
should be called on it first, which turns it into a list of characters. There’s also the clojure.string library for working with strings or Java interop
but that means that you then have to call join
on what comes back out to get a string back, correct?
that's the case no matter how you call map - map cannot return a string
but in general, it's better to use host interop or clojure.string and avoid doing any sequence operations on strings, if you care about performance at all
https://pastebin.com/JWFqkVEg - any better way to deal with exception in go
blocks then try
and catch
?
no, and running a job inside a go block is a bad idea too
well, to be more precise, waiting on a job in a go block is a bad idea
@noisesmith so how to do that in the right way?
(go-loop [] ... (let [answer (<! (async/thread (try (job) (catch ...))))] ...))
otherwise you risk starving go's thread-pool
@noisesmith mainly exception in catch
part
not just that, I mean that go has a small number of threads, and if you block on something that takes a significant amount of time (as most jobs would) then everything in core.async stops
but as i understand code which you gave as an example has the same issue. It is still waiting to finish the job and wouldn’t let next job to start until this one finish. The same situation or i miss something?
it parks on a channel, which frees the thread to do other work
just calling (job) can't free the thread
no more channel reads, no more channel writes, etc.
the try/catch is just common sense, being able to handle errors instead of silently ignoring them
no, that's not what I'm saying
go has a small number of threads, when you block them doing work, nothing core.async does can happen
but how your example solve it? It is still blocked because wait for <! (async/thread ...)
?
no, you park
<! never blocks
when you call <!, you let somebody else use your thread, and your go block will wake up again (maybe in a different thread) when the channel you called <! on has data for you
when you call (job) this can't happen, and you are stealing a thread from core.async
<! (and also >!) give up your block's thread, and lets something else use it
go doesn't ever own a thread, it is picked up by core.async threads
(def worker
(go-loop []
(let [job (<! queue)]
(<! (thread (job)))
(if (nil? job)
(println "queue finished")
(recur)))))
so this is right way to not block go
threads?kwladyka: wouldn't you usually bind the return value of the job?
(def worker
(go-loop []
(let [job (<! queue)
result (when job
(<! (thread (try (job) (catch ...))))]
(if (nil? job)
(println "queue finished")
(recur)))))
OK - you should still check for nil before calling it, and use a try/catch
also i am just not sure… if i load to the REPL worker twice it will make two go
threads parked forever? or kill old one during load new (def woker …)?
kwladyka: unless you do something about it, it leaves both running. The usual solution is to have the loop exit if a chan closes.
public static int cuGetErrorString(int error,
java.lang.String[] pStr)
Gets the string description of an error code
Sets *pStr to the address of a NULL-terminated string description
of the error code error.
If the error code is not recognized, ::CUDA_ERROR_INVALID_VALUE
will be returned and *pStr will be set to the NULL address.
Parameters:
error - - Error code to convert to string
pStr - - Address of the string pointer.
Returns:
::CUDA_SUCCESS, ::CUDA_ERROR_INVALID_VALUE
See Also:
CUresult
how do I call this function from clojure? I have an int to pass it for the error code, it's the [] String I'm not sure how to pass itI don’t know what value of pStr
that funtion expects, but see http://stackoverflow.com/questions/8614251/how-to-a-cast-to-a-string-in-clojure for how to make a java.lang.String[]
out of a collection of strings. So something like (JavaClass/cuGetErrorString 42 (into-array String ["Awesome","Sauce"]))
.
@noisesmith thank you for your help with async