Fork me on GitHub
#clojure
<
2016-03-16
>
hypirion00:03:42

@tf0054: re https://clojurians.slack.com/archives/announce-to-everyone/p1458083127000076 – You can depend on Java libraries, and use (import '(org.corp.project Classname)) Random example from my work today: Depending on [com.amazonaws/aws-java-sdk-core "1.10.59"] and using (import '(com.amazonaws.util EC2MetadataUtils)) will make you able to call the static methods via (EC2MetadataUtils/staticMethod).

jonahbenton02:03:56

hey @bpicolo: have a specific question/point of confusion?

echristopherson02:03:25

@josh.freckleton: I remember something about infinite loops being mentioned in #C0617A8PQ. Maybe try the bleeding-edge version?

bpicolo03:03:16

@jonahbenton: Heya. The issue is that is doesn't actually say what reify does at any point in that doc box

bpicolo03:03:33

@hiredman: sure, that doesn't make that set of docs better though hah

hiredman03:03:07

it depends on the goal right, you are assuming the goal is for the docstring to cover everything about the thing

bpicolo03:03:17

Not really

smw03:03:23

reify defines both an anonymous type and creates an instance of that type

smw03:03:00

Oh, you are very correct

bpicolo03:03:03

typically I expect those docs to tell me what it does first and foremost

smw03:03:05

That’s horrible documentation simple_smile

smw03:03:38

not only that, it’s horrible documentation for a function that’s a difficult concept to understand simple_smile

bpicolo03:03:11

Don't really know why I came to clojure slack to complain other than that it's been the best slack ever for me in the last few days ;D

bpicolo03:03:24

probably smart

bpicolo03:03:27

Will file a ticket

josh.freckleton04:03:31

@echristopherson: Thanks, I didn't know that channel existed, so I posted it there!

martinklepsch11:03:30

I have a blocking function that I want to wrap in a timeout — is there something in Java I could easily reach for?

martinklepsch11:03:23

Currently reaching for core.async but I'm thinking there must be something simple than this:

(defmacro with-timeout [ms & body]
  (let [c (async/thread-call #(do [email protected]))]
    (let [[v ch] (alts!! [c (timeout 100)])]
      (if-not (= ch c)
        (throw (ex-info "Operation timed out" {}))
        v))))

hiredman11:03:27

well, that macro is definitely not correct, because it is doing the call instead of making the call

hiredman11:03:45

well, that macro is definitely not correct, because it is doing the call instead of emitting code that does it

hiredman11:03:41

you can just use future, and deref, as of I dunno, some clojure release awhile back, deref of futures can take extra arguments for timing out

dialelo11:03:13

i'd go with future + deref with timeout as suggested by @hiredman

martinklepsch11:03:27

@hiredman: yeah that macro is "wip" 😉

hiredman11:03:34

previous to that, you could just use future and call the java .get with a time out

hans11:03:16

@martinklepsch: what happens to the thread that you're starting when a timeout occurs?

martinklepsch11:03:52

@hiredman: @dialelo thanks that sounds much better indeed. Just for my own macro-fu:

(defmacro with-timeout [ms & body]
  `(let [c# (async/thread-call #(do [email protected]))]
     (let [[v# ch#] (async/alts!! [c# (async/timeout ~ms)])]
       (if-not (= ch# c#)
         (throw (ex-info "Operation timed out" {}))
         v#))))
this is better?

martinklepsch11:03:29

@hans: not sure I understand?

hans11:03:29

@martinklepsch: I was just wondering whether you're prepared to let the thread continue to run, or whether you could run into the problem that you're starting more and more threads facing timeout conditions.

hans11:03:31

@martinklepsch: It could be completely okay with your use case, but it could also be preferable to handle the timeout situation at a lower level rather than using threads.

martinklepsch11:03:42

@hans: in my case on timeout I'll throw an exception which will be caught & invalidate the resource that has caused the timeout

hans11:03:36

@martinklepsch: so you're okay with the fact that the thread may be running forever?

martinklepsch11:03:22

@hans well that sounds bad... simple_smile I guess I should terminate the thread? fwiw this should probably done in the normal case as well no? (sorry my knowledge about thread handling is ... lacking)

hans11:03:52

@martinklepsch: it gets worse, because if you think that you can just terminate a thread, you're wrong simple_smile

hans11:03:39

@martinklepsch: so effectively, you need to either accept that your threads are leaking, or you need to find a way to detect the timeout in a synchronous fashion, without creating a thread just for the purpose of being able to wait. modern i/o subsystems usually support safe timeout methods, but it won't be as simple as wrapping some arbitrary call in a with-timeout form.

hans11:03:37

@martinklepsch: for me, it was a hard to accept fact that i cannot terminate a thread from another thread in a safe fashion, but i eventually gave in.

hans11:03:32

@martinklepsch: (i'll stop after this) the reason why you cannot safely terminate a thread from another thread is that the system does not perform its resource control on a per-thread fashion. consider that the thread might hold a lock when you terminate it - how would you ever get the lock released?

martinklepsch11:03:30

@hans: thanks, makes sense (internet broke)

martinklepsch11:03:35

so what did you end up with?

martinklepsch11:03:38

also wondering, given these issues, how do @dialelo and @hiredman deal with leaking threads? is the deref with a timeout/.get magically cleaning things up correctly?

dm311:03:09

usually you use a thread pool for this kind of work

hans11:03:24

if i need timeouts, i ask myself how to implement them in a synchronous fashion first. i don't usually think that i could have a with-timeout macro that'd help me avoid going into the details.

dm311:03:25

this way you control the number of threads that get created

hans11:03:25

@dm3: right, a thread pool with a fixed number of threads may be helpful, as one can monitor its utilization and, say, issue a warning or just die if the pool is exhausted.

dm311:03:28

e.g. ztellman/manifold allows passing in a custom executor service

dm311:03:38

to a deferred/future

hans11:03:01

but essentially, using a thread pool is just pushing the problem to another level. i'd begin by thinking about how the timeout can be detected synchronously.

hans12:03:02

@martinklepsch: what kind of operation are you dealing with that can take too long?

martinklepsch12:03:06

@hans: @dm3 right so with the core.async approach there would already be a threadpool if I understand correctly

dm312:03:25

it would be global to the whole application though IIRC

martinklepsch12:03:26

@hans: database operations. Sometimes connections just become stale and won't do anything anymore (I think this is caused by an issue in the DB driver which will be fixed soon) so after thats solved mostly to make sure things break instead of taking forever simple_smile

hans12:03:30

as i said, the thread pool only pushes the problem out, it does not actually solve it. if you could run into the situation that a thread could hang forever, the thread pool will just push the detection of that issue to a later point in time, or prevent you from detecting it at all. that may be a viable strategy, but it'd better be a conscious decision simple_smile

dm312:03:14

totally agree - you need to have a timeout strategy defined

dm312:03:49

just trying to point out that understanding the execution model is essential for any serious work

hans12:03:49

@martinklepsch: for that kind of thing, it may make sense as an intermediate measure. just make sure that with-timeout is not promoted to your utils.clj and becomes a regular thing that people use to avoid thinking about timeouts properly.

hans12:03:54

@martinklepsch: i'd really be interested in knowing why a database connection becomes stale and can't be used anymore. for me, that'd be a perfect procrastination issue.

martinklepsch12:03:36

@hans: There's some bogus core.async code. It's not really the connection but rather the driver code. Not much interesting..

jonahbenton12:03:54

hey @martinklepsch: it's pretty common, across databases, drivers, etc, for individual connections to become unusable. Most pools have connection validation behavior for this reason

martinklepsch12:03:16

@hans: do you have any more pointers that document "the right way"?

martinklepsch12:03:08

@jonahbenton: I can imagine, by connection validation do you mean "validation" or "invalidation" (i.e. pre/post action)

hans12:03:31

@martinklepsch: i'm not sure what to suggest - learning how multithreading works is one aspect of it, advanced i/o systems are another. http://www8.cs.umu.se/kurser/TDBC64/VT03/pthreads/pthread-primer.pdf was the canonical reference back in my day, but you may find it to be going too much into the gory details of systems programming.

hans12:03:18

much of the pthread primer applies to multi threaded programming in general, so don't be put off by the title if you're into understanding the principles.

jonahbenton12:03:28

both- the usual behavior for a pool is to maintain a maximum ttl for any given connection- automatic invalidation- and to also, sometimes optionally, to utilize a dummy query to re-validate individual connections, either on receiving custody of them or periodically in the background

jonahbenton12:03:11

and most pools- jdbc, anyway- expose an interface that makes this transparent to you

jonahbenton12:03:51

if your blocking function is a database call, and you want an application-level timeout to stop execution of the query at the database server- you will want to check the driver and database to confirm that cancellation actually works. sometimes you can only cancel a query from a dba control point

thiagofm13:03:37

If I send a message to an agent that takes some time to execute, how do I check whether the processing has been finished or not?

thiagofm13:03:49

(realized? agent) doesn't seem to work

thiagofm13:03:24

Okay, solved.

pyr14:03:33

Hi everyone

pyr14:03:02

I'm trying to see if there's a perfect idiom for doing something I find myself doing quite often. Let's say I have a stream of incoming events which need to augment a view of the world, with clojure I usually do:

(reduce some-reducer (init-world) my-source) 
Now there are often time cases where there are two things to do in the reducer: update the state of the world and output a list of side-effects. I usually produce the side-effects onto a channel and handle them in a separate function, but I think it hinders legibility somewhat. What I'm trying to do is to have a transducer able to hold on to a value, a bit like iterate in a sense, where I'd be able to say something along the lines of:
(defn update-world-and-provide-effects
      [world event]
      [new-world side-effect-to-perform-vector])

(def my-xform
   (comp (map sanitize-input)
                 (fooduce update-world-and-provide-effects (init-world))
                 (mapcat identity)))
So two questions: 1. Does this make any sense 2. Is there a way of doing this I missed?

pyr15:03:55

hmm looking again at clojure.core source it seems I'm expected to write my own transducer for this and hold on toworld in a volatile

pyr15:03:34

so here's a transducer that does this, in case anyone is interested:

(defn induce
  ([f]
   (induce f (f)))
  ([f init]
   (fn [rf]
     (let [state (volatile! init)]
       (fn
         ([] (rf))
         ([result] (rf result))
         ([result input]
          (let [[new-state output] (f @state input)]
            (vreset! state new-state)
            (rf result output))))))))

lvh16:03:51

I’m trying to test some Clojure code that does Java interop with some side effects:

(defn ^:private run-media-driver!
  [options]
  (let [ctx (doto (MediaDriver$Context.)
              (.dirsDeleteOnStart (options :delete-dirs)))]
    (try (MediaDriver/launch ctx)
         (catch IllegalStateException ise
           (throw (Exception. aeron-launch-error-message ise))))))
What’s the preferred way of testing that? IIRC :import doesn’t create vars, so with-redefs doesn’t work, right?

roberto16:03:44

I’ve been happy with conjure

roberto16:03:02

I would use that to mock the java api and just test that they are called

lvh16:03:39

roberto: Thanks! That appears to know how to redef vars, but not classes. Is that incorrect?

lvh16:03:24

Creating a fake MediaDriver isn’t super hard with gen-class; the problem is convincing (MediaDriver/launch ctx) to point to my fake.

roberto16:03:36

I would wrap all that nasty java thingy in a function

lvh16:03:57

How do I test that?

lvh16:03:23

I mean, sure, that means I can mock that out, but it seems like that just kicks the can a bit further down the street, right?

roberto16:03:20

I keep state out of my unit tests, so they don’t touch IO nor anything like that. But the moment you are asking about redef, it means you want to mock or stub.

lvh16:03:21

Yes, keeping state and side effects out of my unit tests is what I’m trying to accomplish; I’m trying to mock out MediaDriver/launch. I could write a fn like:

(defn launch-media-driver [x] (MediaDriver/launch x))
… but then I can’t test that 😄

roberto16:03:44

why do you want to mock MediaDriver ?

roberto16:03:47

don’t mock things you don’t own

roberto16:03:54

so wrap it, and mock the wrapper

lvh16:03:36

because MediaDriver has behaviors that I need to handle: raise an exception, or don’t (+ have some magical side effects)

lvh16:03:46

mocking the wrapper lets me do that, but then there’s nothing to test the wrapper

roberto16:03:48

then stub the wrapper to raise an exception

roberto16:03:01

there is no need to test the wrapper

lvh16:03:15

(def launch-media-driver [x] (System/exit 1))

roberto16:03:18

other functions that use will test it, plus intergration tests

lvh16:03:25

OK, thanks

roberto16:03:06

not using conjure but gives you some ideas

roberto16:03:21

it also includes some integration tests

lvh16:03:02

Yes, I understand what you’re saying, and how to write that test.

lvh16:03:10

I’m saying that that doesn’t test that the wrapper does anything sane.

lvh16:03:45

I’m also saying that you don’t cop out once you hit a class in other environments, including Java.

danielstockton18:03:21

If timbre prints to standard out by default, if i'm executing (timbre/info "test") in a repl, shouldn't it print "test" in the repl ?

ccann18:03:26

I have a KafkaConsumer object that will throw an exception when modified concurrently https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html and I’m trying to just close the darn thing when I CTRL-C my program. Any ideas?

ccann18:03:12

I’ve tried closing it in clojure.repl/set-break-handler! and .addShutdownHook but both throw that same exception… (which I guess isn’t surprising). So I wrapped it in an atom and STILL it throws that exception. I guess I don’t understand why.

pbostrom19:03:10

@ccann: is the consumer polling on another thread?

ccann19:03:29

yeah, it is. The main thread, and I’m trying to shut it down from the shutdown hook which I’m assuming is its own thread. i actually just found a bit in the docs about how to shut down the consumer from a separate thread using .wakeup so hopefully I can get that going

ymilky19:03:59

@ccan I can confirm that using wakeup from another thread is the correct way, but all it really does is make sure you're not deadlocked polling since poll will block. Sounds to me like your issue is more how you're closing it as you suggest.

ymilky19:03:36

@ccann: Are you also sure you are not holding a reference to the consumer somewhere, preventing shutdown from closing the consumer?

ccann19:03:04

@ymilky thanks for the tip, I think you’re right. I was under the impression that I was releasing the reference but I’ve just discovered that bit might be failing silently so the reference is maintained in a tight polling loop.

ymilky19:03:56

@ccan hope that helps. I also have written a suite of libraries for Kafka including a wrapper on the consumer you are using. See https://github.com/ymilky/franzy . Some people are testing it right now, but after that I hope to get a real release out on clojars. Adding some features and such this week, but it's pretty comprehensive for all of kafka.

ccann19:03:41

I was working on something similar actually

ymilky19:03:00

pull requests welcome. The goal was to be comprehensive, reasonably fast all things considered, and stay pretty close to the official APIs, building via add-ons. That and minimizing depedencies, i.e. no referencing the server jar for just the consumer. It should be completely modular, among many other bonuses. Expect a bit of churn, but so far the users are happy and I'm mostly throwing up some more add-ons this week when I get a chance.

ccann19:03:51

@ymilky: sounds awesome, much more than I was planning on doing. btw, problem solved!

ymilky19:03:03

@ccan awesome, glad to hear. You had me slightly nervous for a minute there was some serious issue in the Java client I hadn't hit yet

ymilky19:03:08

I'd also recommend the admin stuff I have there as it really helps when you have issues. You can see in Zookeeper for example exactly what is going on, for instance if you have a stuck consumer. You can also explicitly delete a consumer/client if needed.

iwankaramazow20:03:14

if you want to swap! an atom multiple times is this valid clojure code:

(-> some-atom
                   (swap! assoc-in [:key] new-value)
                   (swap! update-in [:editable] #(not %)))
Is this idiomatic?

donaldball20:03:09

The two operations won’t be atomic

iwankaramazow20:03:23

explains a lot why might code doesn't behave as expected 😄

seancorfield20:03:28

More to the point, swap! returns the new value, not the atom.

donaldball20:03:37

There is that minor point

seancorfield20:03:12

So what is passed into your second swap! isn’t an atom, so it will fail...

iwankaramazow20:03:52

How do I achieve this?

donaldball20:03:32

(swap! some-atom #(-> % (assoc-in [:key] new-value) (update-in [:editable] #(not %))))) is one way

seancorfield20:03:52

Hah @donaldball types faster than me!

donaldball20:03:16

I’d probably lift the anonymous fn into a var tho, and yeah, @rauh is correct, you can’t nest anonymous fn literals

jr20:03:30

#(not %) == not

iwankaramazow20:03:41

Thanks a lot 😄

bostonaholic20:03:16

you can also use doto like -> for atoms

bostonaholic20:03:38

(doto (atom 0) (swap! inc) (swap! inc))

iwankaramazow20:03:54

didn't know that, thanks simple_smile

donaldball20:03:10

Oh, that’s clever

donaldball20:03:24

Although multiple swap!s are still not atomic