This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-03-16
Channels
- # aatree (1)
- # admin-announcements (6)
- # alda (3)
- # beginners (66)
- # boot (41)
- # cider (4)
- # cljsjs (3)
- # cljsrn (18)
- # clojure (146)
- # clojure-android (2)
- # clojure-nl (1)
- # clojure-russia (35)
- # clojure-sdn (2)
- # clojure-sg (5)
- # clojure-uk (41)
- # clojurescript (116)
- # datomic (12)
- # dirac (40)
- # docker (2)
- # editors (2)
- # hoplon (85)
- # immutant (19)
- # jobs (1)
- # keechma (2)
- # lein-figwheel (8)
- # mount (33)
- # off-topic (1)
- # om (114)
- # onyx (159)
- # parinfer (24)
- # proton (3)
- # reagent (4)
- # ring-swagger (15)
- # uncomplicate (7)
- # untangled (93)
- # yada (30)
@tf0054: re https://clojurians.slack.com/archives/announce-to-everyone/p1458083127000076 – You can depend on Java libraries, and use (import '(org.corp.project Classname))
Random example from my work today: Depending on [com.amazonaws/aws-java-sdk-core "1.10.59"]
and using (import '(com.amazonaws.util EC2MetadataUtils))
will make you able to call the static methods via (EC2MetadataUtils/staticMethod)
.
hey @bpicolo: have a specific question/point of confusion?
hey @arohner did you check out https://commons.apache.org/proper/commons-lang/javadocs/api-2.6/org/apache/commons/lang/StringEscapeUtils.html#unescapeJava(java.lang.String)
@josh.freckleton: I remember something about infinite loops being mentioned in #C0617A8PQ. Maybe try the bleeding-edge version?
@jonahbenton: Heya. The issue is that is doesn't actually say what reify does at any point in that doc box
it depends on the goal right, you are assuming the goal is for the docstring to cover everything about the thing
@hiredman: compare it with, say, https://clojuredocs.org/clojure.core/assoc-in
not only that, it’s horrible documentation for a function that’s a difficult concept to understand
Don't really know why I came to clojure slack to complain other than that it's been the best slack ever for me in the last few days ;D
@echristopherson: Thanks, I didn't know that channel existed, so I posted it there!
I have a blocking function that I want to wrap in a timeout — is there something in Java I could easily reach for?
Currently reaching for core.async but I'm thinking there must be something simple than this:
(defmacro with-timeout [ms & body]
(let [c (async/thread-call #(do ~@body))]
(let [[v ch] (alts!! [c (timeout 100)])]
(if-not (= ch c)
(throw (ex-info "Operation timed out" {}))
v))))
well, that macro is definitely not correct, because it is doing the call instead of making the call
well, that macro is definitely not correct, because it is doing the call instead of emitting code that does it
you can just use future, and deref, as of I dunno, some clojure release awhile back, deref of futures can take extra arguments for timing out
@hiredman: yeah that macro is "wip" 😉
@martinklepsch: what happens to the thread that you're starting when a timeout occurs?
@hiredman: @dialelo thanks that sounds much better indeed. Just for my own macro-fu:
(defmacro with-timeout [ms & body]
`(let [c# (async/thread-call #(do ~@body))]
(let [[v# ch#] (async/alts!! [c# (async/timeout ~ms)])]
(if-not (= ch# c#)
(throw (ex-info "Operation timed out" {}))
v#))))
this is better?@hans: not sure I understand?
@martinklepsch: I was just wondering whether you're prepared to let the thread continue to run, or whether you could run into the problem that you're starting more and more threads facing timeout conditions.
@martinklepsch: It could be completely okay with your use case, but it could also be preferable to handle the timeout situation at a lower level rather than using threads.
@hans: in my case on timeout I'll throw an exception which will be caught & invalidate the resource that has caused the timeout
@martinklepsch: so you're okay with the fact that the thread may be running forever?
@hans well that sounds bad... I guess I should terminate the thread? fwiw this should probably done in the normal case as well no? (sorry my knowledge about thread handling is ... lacking)
@martinklepsch: it gets worse, because if you think that you can just terminate a thread, you're wrong
@martinklepsch: so effectively, you need to either accept that your threads are leaking, or you need to find a way to detect the timeout in a synchronous fashion, without creating a thread just for the purpose of being able to wait. modern i/o subsystems usually support safe timeout methods, but it won't be as simple as wrapping some arbitrary call in a with-timeout form.
@martinklepsch: for me, it was a hard to accept fact that i cannot terminate a thread from another thread in a safe fashion, but i eventually gave in.
@martinklepsch: (i'll stop after this) the reason why you cannot safely terminate a thread from another thread is that the system does not perform its resource control on a per-thread fashion. consider that the thread might hold a lock when you terminate it - how would you ever get the lock released?
@hans: thanks, makes sense (internet broke)
so what did you end up with?
also wondering, given these issues, how do @dialelo and @hiredman deal with leaking threads? is the deref with a timeout/.get magically cleaning things up correctly?
if i need timeouts, i ask myself how to implement them in a synchronous fashion first. i don't usually think that i could have a with-timeout macro that'd help me avoid going into the details.
@dm3: right, a thread pool with a fixed number of threads may be helpful, as one can monitor its utilization and, say, issue a warning or just die if the pool is exhausted.
but essentially, using a thread pool is just pushing the problem to another level. i'd begin by thinking about how the timeout can be detected synchronously.
@martinklepsch: what kind of operation are you dealing with that can take too long?
@hans: @dm3 right so with the core.async approach there would already be a threadpool if I understand correctly
@hans: database operations. Sometimes connections just become stale and won't do anything anymore (I think this is caused by an issue in the DB driver which will be fixed soon) so after thats solved mostly to make sure things break instead of taking forever
as i said, the thread pool only pushes the problem out, it does not actually solve it. if you could run into the situation that a thread could hang forever, the thread pool will just push the detection of that issue to a later point in time, or prevent you from detecting it at all. that may be a viable strategy, but it'd better be a conscious decision
just trying to point out that understanding the execution model is essential for any serious work
@martinklepsch: for that kind of thing, it may make sense as an intermediate measure. just make sure that with-timeout is not promoted to your utils.clj and becomes a regular thing that people use to avoid thinking about timeouts properly.
@martinklepsch: i'd really be interested in knowing why a database connection becomes stale and can't be used anymore. for me, that'd be a perfect procrastination issue.
@hans: There's some bogus core.async code. It's not really the connection but rather the driver code. Not much interesting..
hey @martinklepsch: it's pretty common, across databases, drivers, etc, for individual connections to become unusable. Most pools have connection validation behavior for this reason
@hans: do you have any more pointers that document "the right way"?
@jonahbenton: I can imagine, by connection validation do you mean "validation" or "invalidation" (i.e. pre/post action)
@martinklepsch: i'm not sure what to suggest - learning how multithreading works is one aspect of it, advanced i/o systems are another. http://www8.cs.umu.se/kurser/TDBC64/VT03/pthreads/pthread-primer.pdf was the canonical reference back in my day, but you may find it to be going too much into the gory details of systems programming.
much of the pthread primer applies to multi threaded programming in general, so don't be put off by the title if you're into understanding the principles.
both- the usual behavior for a pool is to maintain a maximum ttl for any given connection- automatic invalidation- and to also, sometimes optionally, to utilize a dummy query to re-validate individual connections, either on receiving custody of them or periodically in the background
and most pools- jdbc, anyway- expose an interface that makes this transparent to you
if your blocking function is a database call, and you want an application-level timeout to stop execution of the query at the database server- you will want to check the driver and database to confirm that cancellation actually works. sometimes you can only cancel a query from a dba control point
If I send a message to an agent that takes some time to execute, how do I check whether the processing has been finished or not?
I'm trying to see if there's a perfect idiom for doing something I find myself doing quite often. Let's say I have a stream of incoming events which need to augment a view of the world, with clojure I usually do:
(reduce some-reducer (init-world) my-source)
Now there are often time cases where there are two things to do in the reducer: update the state of the world and output a list of side-effects. I usually produce the side-effects onto a channel and handle them in a separate function, but I think it hinders legibility somewhat.
What I'm trying to do is to have a transducer able to hold on to a value, a bit like iterate
in a sense, where I'd be able to say something along the lines of:
(defn update-world-and-provide-effects
[world event]
[new-world side-effect-to-perform-vector])
(def my-xform
(comp (map sanitize-input)
(fooduce update-world-and-provide-effects (init-world))
(mapcat identity)))
So two questions:
1. Does this make any sense
2. Is there a way of doing this I missed?hmm looking again at clojure.core source it seems I'm expected to write my own transducer for this and hold on toworld in a volatile
so here's a transducer that does this, in case anyone is interested:
(defn induce
([f]
(induce f (f)))
([f init]
(fn [rf]
(let [state (volatile! init)]
(fn
([] (rf))
([result] (rf result))
([result input]
(let [[new-state output] (f @state input)]
(vreset! state new-state)
(rf result output))))))))
I’m trying to test some Clojure code that does Java interop with some side effects:
(defn ^:private run-media-driver!
[options]
(let [ctx (doto (MediaDriver$Context.)
(.dirsDeleteOnStart (options :delete-dirs)))]
(try (MediaDriver/launch ctx)
(catch IllegalStateException ise
(throw (Exception. aeron-launch-error-message ise))))))
What’s the preferred way of testing that? IIRC :import doesn’t create vars, so with-redefs doesn’t work, right?roberto: Thanks! That appears to know how to redef vars, but not classes. Is that incorrect?
Creating a fake MediaDriver isn’t super hard with gen-class; the problem is convincing (MediaDriver/launch ctx)
to point to my fake.
I mean, sure, that means I can mock that out, but it seems like that just kicks the can a bit further down the street, right?
I keep state out of my unit tests, so they don’t touch IO nor anything like that. But the moment you are asking about redef
, it means you want to mock or stub.
Yes, keeping state and side effects out of my unit tests is what I’m trying to accomplish; I’m trying to mock out MediaDriver/launch. I could write a fn like:
(defn launch-media-driver [x] (MediaDriver/launch x))
… but then I can’t test that 😄because MediaDriver has behaviors that I need to handle: raise an exception, or don’t (+ have some magical side effects)
here is an example https://github.com/dakrone/clj-http/blob/master/test/clj_http/test/client_test.clj#L80
I’m also saying that you don’t cop out once you hit a class in other environments, including Java.
If timbre prints to standard out by default, if i'm executing (timbre/info "test") in a repl, shouldn't it print "test" in the repl ?
I have a KafkaConsumer object that will throw an exception when modified concurrently https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html and I’m trying to just close the darn thing when I CTRL-C my program. Any ideas?
I’ve tried closing it in clojure.repl/set-break-handler!
and .addShutdownHook
but both throw that same exception… (which I guess isn’t surprising). So I wrapped it in an atom
and STILL it throws that exception. I guess I don’t understand why.
yeah, it is. The main thread, and I’m trying to shut it down from the shutdown hook which I’m assuming is its own thread. i actually just found a bit in the docs about how to shut down the consumer from a separate thread using .wakeup
so hopefully I can get that going
@ccan I can confirm that using wakeup from another thread is the correct way, but all it really does is make sure you're not deadlocked polling since poll will block. Sounds to me like your issue is more how you're closing it as you suggest.
@ccann: Are you also sure you are not holding a reference to the consumer somewhere, preventing shutdown from closing the consumer?
@ymilky thanks for the tip, I think you’re right. I was under the impression that I was releasing the reference but I’ve just discovered that bit might be failing silently so the reference is maintained in a tight polling loop.
@ccan hope that helps. I also have written a suite of libraries for Kafka including a wrapper on the consumer you are using. See https://github.com/ymilky/franzy . Some people are testing it right now, but after that I hope to get a real release out on clojars. Adding some features and such this week, but it's pretty comprehensive for all of kafka.
pull requests welcome. The goal was to be comprehensive, reasonably fast all things considered, and stay pretty close to the official APIs, building via add-ons. That and minimizing depedencies, i.e. no referencing the server jar for just the consumer. It should be completely modular, among many other bonuses. Expect a bit of churn, but so far the users are happy and I'm mostly throwing up some more add-ons this week when I get a chance.
@ccan awesome, glad to hear. You had me slightly nervous for a minute there was some serious issue in the Java client I hadn't hit yet
I'd also recommend the admin stuff I have there as it really helps when you have issues. You can see in Zookeeper for example exactly what is going on, for instance if you have a stuck consumer. You can also explicitly delete a consumer/client if needed.
if you want to swap!
an atom multiple times is this valid clojure code:
(-> some-atom
(swap! assoc-in [:key] new-value)
(swap! update-in [:editable] #(not %)))
Is this idiomatic?The two operations won’t be atomic
explains a lot why might code doesn't behave as expected 😄
More to the point, swap!
returns the new value, not the atom.
There is that minor point
So what is passed into your second swap!
isn’t an atom, so it will fail...
ah ok!
How do I achieve this?
(swap! some-atom #(-> % (assoc-in [:key] new-value) (update-in [:editable] #(not %)))))
is one way
Hah @donaldball types faster than me!
@donaldball: Nested #()
?
I’d probably lift the anonymous fn into a var tho, and yeah, @rauh is correct, you can’t nest anonymous fn literals
jr: beat me
Thanks a lot 😄
you can also use doto
like ->
for atoms
(doto (atom 0) (swap! inc) (swap! inc))
didn't know that, thanks
Oh, that’s clever
Although multiple swap!s are still not atomic