Anyone know a good open source clojure library for User Management?

please message me directly

A core.async question: what is the preferred way to determine if a channel is already closed before you try to put to it? Or is my desire to do that a sign that I'm Using It Wrong?

Ah, not really supported it seems: http://stackoverflow.com/questions/24912971/how-to-determine-when-a-core-async-channel-is-closed-from-the-writers-side

@alexisgallagher: every time I’ve gone down that path I’ve realised I should think about the problem in a different way

Ok. My question situation is a lot like the OP's. I've got a process polling an API, and I'd like to expose the results as a channel, and stop polling if the consumer closes the channel.

I suppose I need to use another mechanism to signal to turn polling on/off.

This post by @bbloom is insightful: https://groups.google.com/d/msg/clojure/_KzEoq0XcHQ/j46hKin2-tsJ

Does anybody know if "educe" made it into the Clojure language?.. http://www.youtube.com/watch?v=4KqUvG8HPYo&t=40m43s

Is that the same as http://clojure.github.io/clojure/clojure.core-api.html#clojure.core/eduction ?

(seems to be, right?)

aha! thanks..

thats the one

Why is that? (Clojure 1.6.0)

(unchecked-multiply 62 839299365868340224)
=> -3303671537291560960
(unchecked-multiply 62 (Long/valueOf 839299365868340224))
java.lang.ArithmeticException: integer overflow

Same in Clojure 1.8.0

The problem here is that I try this for over half an hour in the REPL with (unchecked-multiply 62 839299365868340224) and my code still fails because it uses Long instead of long.

How would I add up e.g. (1 1 1 0 1 0 0 1 1) and get all intermediate results (1 2 3 3 4 4 4 5 6)?

reductions

http://clojuredocs.org/clojure.core/reductions

@robert-stuttaford: That's it. Thanks!

To answer my question: http://dev.clojure.org/jira/browse/CLJ-1832

@akiel: Actually, it's because of https://github.com/clojure/clojure/blob/f76b343d917532d0b2e1397587e3fdefdd5bd166/src/jvm/clojure/lang/Numbers.java#L1862 if I'm reading things correctly. The primitive long will end up being coerced into a Long, the value of both Longs will then be turned into a primitive long and a normal checked multiply will be called afterwards.

@sgerguri: the issue http://dev.clojure.org/jira/browse/CLJ-1832 explains the situation very good

@akiel: I was just adding it here for completeness. There is a single link from that issue into a line in the file that actually does not highlight the root cause of why this is happening.

@sgerguri: Yes I saw this also. Thanks.

I would be happy to see a patch on that ticket if someone wanted to work on it

System has an open ticket that seems like a good fit for #C03S1KBA2 learners who want to contribute to a FOSS project. It's a relatively simple task, and I'll provide guidance. https://github.com/danielsz/system/issues/51

@alexmiller: In which direction? Doc or correct implementation?

How do I silence a redefinition warning?

I have a Clojure namespace with a function that needs to be called from java. The function takes no parameters and returns a record. I'm having trouble knowing how to specify this in :gen-class. I keep getting class not found exception for the record. Assuming because it hasn't been compiled yet, but not sure how to influence that (I'm building with Leiningen). Or I could be completely off track . Oversimplified example below for illustration:

(ns foo.bar
  (:gen-class
    :name com.foo.Bar
    :methods [^:static [mymethod [] foo.bar.MyRecord]]))

(defrecord MyRecord [param1 param2]
  proto
  (method1 [this] ... )
  ;; etc)

(defn mymethod [] (->MyRecord "baz" "qux"))

(defn -mymethod [] (mymethod))

@akiel: impl

@manderson: you might need to define the record in a different ns or use gen-class directly, after the record is defined

Thanks @alexmiller. I'll give that a try. I just got it working returning Object, but that's not as nice for the user of my API. (eg: :methods [^:static [mymethod [] Object]])

FYI: in case anyone is curious, calling the gen-class function after the defrecord in my namespace worked per Alex's suggestion.

@mbertheau: (:refer-clojure :exclude [symbol]) in namespace definition.

@danielsz: Thanks!

(In build.boot you can use ns-unmap instead since the ns form is autogenerated.

is there a reason why clojure.string/join takes the coll as a second argument when given a separator ?

for instance if I want tot split / update I can't put join in the thread first macro since the coll is the last argument of join

(->> things (string/join \;))

https://github.com/rplevy/swiss-arrows >_> <_<

there's always as->

it's not a big deal anyway I was just wondering, here is an exemple function

(hello-clojure)

yenda: if you're working with map, reduce, filter the coll is also always the last argument.

what is a good light weight and stable http server for clojure?

Jetty + Compojure

or http://www.http-kit.org/

cool

but http-kit is lower level than Jetty

not sure about the "stable" part

but it's used in production AFAIK

Does anyone run web servers in cljs + node?

I'm using http-kit without problems.

@yenda: it's make it easier to partial

I’m using http-kit now, wasn’t sure how active the development is though

most string functions take the string first (similar to collection functions), but this is one of the exceptions

I'm trying to get clojure.java.jdbc to work with sqlite on windows, but keep getting this error: CompilerException java.sql.SQLException: No suitable driver found for jdbc:sqlite:resources/database.db

alexmiller: thanks that's what I was wondering.

partial explains it

I downloaded a sqlite-jdbc jar from https://github.com/xerial/sqlite-jdbc and put it in my src dir but that didn't help

that doesn't get the jar on the classpath

arijun: try [org.xerial/sqlite-jdbc "3.8.7"] in your dependencies

(or whatever the current version is)

oh I incorrectly translated the dependency from maven when I tried to do that before

but I printed my classpath and it included that src directory?

arijun: but the classpath would have to include the jar file, not the directory containing the jar file

Ok, failed assumption from python

what's the point of including the directory in the classpath at all then?

@lucasbradstreet: that is by design: https://clojure.github.io/core.async/#clojure.core.async/&gt;!!

puts a val into port. nil values are not allowed. Will block if no
buffer space is available. Returns true unless port is already closed.

Thanks, I realise that it should block

I just think it should release when the channel is closed

Otherwise your thread that is >!! onto the ch will become blocked forever unless you take from the channel just to get it unstuck

even when you have called close! on the channel

My point being that when you close! the channel it should return false (as if you closed the channel before the >!!)

@lucasbradstreet: use thread macro if you are worried about your main thread blocking: https://clojure.github.io/core.async/#clojure.core.async/thread

This is within a thread, which is the issue

Because the thread never exits

It ends up blocked on >!! when the channel it’s >!! onto is closed

but closed after the >!! call

I’m trying to avoid creating lots of zombie threads as a result

My current workaround is to do this, but it’s not great and uses the impl fn closed?

I think the current behaviour is racey

You can >!! 1000 messages onto your ch, and depending on your order of the final >!! and your closed! you get different results

Anyway, I’m mostly looking for advice about whether everyone thinks it’s a bug. I realise that technically it does what it says on the tin

My feeling is that absent a performance related reason, you would never want the current behavior

@lucasbradstreet: does the bug reproduce w/o the future? just >!!, close!, >!! ?

Nope

Because it has to be >!!ing before the close!

ah, it has to already be blocked

Yup

bug, IMO, but my opinion doesn’t count for much

We use >!! to fill up a buffer from within a thread, but we use the return code to determine whether the thread should bail. I’m OK with it blocking until we’ve close!d the channel that it’s >!!ing on, at which point it should bail

doc for close!: https://clojure.github.io/core.async/#clojure.core.async/close!

Logically closing happens after all puts have been delivered. Therefore, any
blocked or parked puts will remain blocked/parked until a taker releases them.

Yeah, that doc is a bit more clear cut

Thanks @pbostrom. I don’t really agree with the choice, but at least it’s obviously been considered

lamina makes the distinction between close (stop accepting inputs) and drained (no data left) which I think is useful in thinking about the lifecycle

no problem, there is some amount of inherited design from golang channels I believe, here is a discussion on the mailing list someone posted earlier: https://groups.google.com/forum/#!msg/clojure/_KzEoq0XcHQ/j46hKin2-tsJ there is this implicit idea that the producer (i.e. the thread making the puts/>!!) should do the close!

I’ll read through that thread. I had seen that in the past, but didn’t connect it to this. I do have some trouble seeing how the thread doing the >!! should also close! if it is blocked at the time. I should read more and figure it out in my head before I speak further

you could also use offer! to only put if not closed, not sure if that helps

http://clojure.github.io/core.async/#clojure.core.async/offer!

offer! will not block - it only puts if the put will succeed

Yeah, I did that in my alternative approach. But you need to also check whether the channel is closed? if you’re going to retry, and that is in the impl protocol

offer! is a great addition

Thanks to you both

offer! will not put if channel is closed

and returns false

That is true, but it also returns false if the channel is full

So I have no way of distinguishing between the two

oh

one is false, the other is nil

Apologies

Ok, that’s excellent. Problem solved.

I remember a previous discussion on the clojure mailing list that there was no way to distinguish between them

well, I don't know that I would rely on that difference :)

I don't know that that is intentional

(I wrote the code)

so let me rephrase - that was not intentional :)

Ha, ok, that is very good to know!

@lucasbradstreet what I found most clarifying about that @bbloom google groups link I posted earlier, was the idea that closing the channel is only intended to be a signal from the producer to the consumer.

Yeah, I think that’s especially enlightening

What that seems to imply is that the only mechanism a channel offers for a consumer to communicate with the producer is back-pressure, and that you shouldn't use close! as a kind of "fast acting back-pressure" which is one way I was thinking of it.

Whether or not it's consistent with the original design plan, I think that "fast acting back-pressure" perspective on close! is an easy one to fall into.

I just did a tutorial where it was basically used that way, so maybe that's what got it in my head.

Also, there are many infinite producer processes, where the consumer is the one who decides when they don't care any more, rather than the producer knowing that there is nothing left to offer.

And in those processes, back-pressure is a poor way for the consumer to say they don't care any more.

First it's delayed by the size of the buffer.

Second, the producer process may have side-effects (like draining some other source outside of the core.async abstraction). So you'd like the producer to know it should shut off as soon as the consumer doesn't care, rather than later.

That's the situation I'm working on right now.

Same situation here

Interesting. So my producer is polling an SQS queue, and putting the messages onto a channel.

I just want a way to shut off the polling.

I’m writing onyx plugins that read from datasources like the datomic log, bookkeeper, kafka, etc

My first was to have the consumer close the channel.

Okay. Very similar.

and need some way to stop the producer from reading from those datasources and writing to the channel. Same situation basically

You see my problem. I want the producer to know if it should stop before it polls SQS and takes an item from the SQS collection. So offer! is no good because I need to get the item from SQS before I try offer! to see if the channel is closed.

I think that situation is always going to be racey, assuming you can’t read from SQS without it being consumed

I notice there's a library Squeedo that does an SQS->core.async polling listener, and they ran into the same problem, but they solved it by breaking the abstraction and going into the implementation to get closed?: https://github.com/TheClimateCorporation/squeedo/blob/master/src/com/climate/squeedo/sqs_consumer.clj#L36

Yep, that’s basically what I’ve done

I don't like reaching into the core.async implementation like that b/c I assume it's not stable API.

I don't think anything here needs to be racey. But it may be I'm misunderstanding it.

Sorry, I just meant that if you close from outside, and you want to know that the channel is closed before you offer

ah. maybe.

Since the close could happen inbetween. I think with SQS you’re safe because you just don’t ack the message?

Yeah, but that's ugly. I'd rather not build a dependency into SQS's retry logic since I don't think that's necessary.

The fundamental problem is we want to create a way to "send a message" in some sense to the polling process but we can't use the channel for it.

So I see two alternatives at the moment:

1. just set a flag in an atom.

This is racey the way I've done it. Maybe it needs to be a ref for synchronization of the state-change and the side-effect of closing the channel.

Here's how it looks:

(defn create-queue-listener
  "Kicks off background process that will eagerly grab messages and
  put them on the channel. 

If no messages are available, it waits 1 second, then repeats.

If a message is available, it tries to put it on the chan, then
  repeats.

Returns an opaque queue-listener object, which can be used later to kill the queue-listener.

Inputs:

- get-messages-fn. Zero-arg function which returns collection of messages, by doing sychronous I/O.
- message-chan. channel on which to put messages.

Stops grabbing messages when the channel is closed."

  [message-chan get-message-fn]
  (let [ql (atom {:open true :message-chan message-chan})]
    (go
      (while (:open @ql)
        (if-let [message (get-message-fn)]
          ; taking from the channel onto-chan returns blocks here.
          ; but doesn't onto-chan block while putting to message-chan?
          (<! (onto-chan message-chan (vector message) false))
          (<! (async/timeout 1000)))))
    ql))

(defn kill-queue-listener
  [ql]
  (swap! ql assoc :open false)
  (close! (:message-chan @ql)))

(Sorry about the wall of code everyone.)

So I think the more correct, non-racey option is

2. Create a second channel, a "control channel", that is used only for sending the "stop polling" message to the go loop that is polling.

Instead of checking a flag in while, that go worker uses alts! with :priority to first try to check the control channel and then try to pull data.

That's my rough idea.

I’ve done 2 in the past, but I’m not sure how it helps here

I'm not sure if this is clever and correct, or clever and gratuitous and channels galore.

It’d help with the consumer, but not a producer?

Well, my thinking (maybe I'm confused!) is that what I have now is incorrect because swap! only atomically modifies the flag, but doesn't atomically modify the flag and close the channel. and maybe alts! fixes that?

But that hunch might be wrong. And the more I think about it, I don't actually see why I should close the channel at all.

All I want is a way to stop the polling process from polling.

I hand the channel in when i create the polling process.

Maybe I'll want to use it again.

Or hand it to a different kind of producer later.

Using a second shutdown channel to stop the polling will work fine

So if I'm not closing the channel as a way to shut off polling, then why close it at all? Maybe that idea was just a vestige of my old thinking.

Yeah, you don’t really need to. The point of the second channel is just to short circuit draining the first channel, right?

The producer is certainly a tougher case

Well, I just wanted a way to shut off polling, full stop. Then I thought "oh close the channel". But now that I think about it the two things are unrelated.

I was just confused.

Maybe this is all an argument for the illuminating clarity of the original design idea that close! should not be a form of backward message.

I think it probably is, actually

Yeah, and I guess it’s an argument for draining the channel to unblock the producer

Rather than using a hack around offer! and checking if the channel is closed

If the idea is that close! represents "nothing more to send", then it is incorrect to close! a channel that represents an infinite stream of items coming from an external source that you are polling.

What's really going on is that, sometimes, you just don't care anymore, so you stop taking from the channel. It's not that there's nothing more that could be put into it.

:thinking_face:

Yeah, that’s essentially my problem too

Thanks this chat has helped me straighten out my head on this.

Thanks. I’m still not sure what the correct solution is. Maybe channels are inappropriate for this use case.

Anyone using Light Table regularly?

@alexisgallagher: I think using a control channel (or something like it) is a coordination problem, not a message conveyance problem and I think relying on the channel to do it is conflating the two.

Ok. So you agree with the conclusion I've been stumbling toward?

I think so, not sure I read 100% of the backchat

it would be an interesting experiment to create a priority buffer though where different messages have different priority (control / data) so that you could send a high priority "stop" control message through the channel and have it emerge in front of the low priority data messages

I think what slipped me up is that "I don't want any more" seems like a special case of "whoa not so fast!", so it's easy to think of close! as a special form of back-pressure, or vice versa, but this is basically incorrect.

yeah, there is a reason Rich didn't put close? in the api

(although we may add it)

Yeah, just trying to understand the design patterns the API is promoting. As I pointed out others are digging into implementation to find close? ( https://github.com/TheClimateCorporation/squeedo/blob/master/src/com/climate/squeedo/sqs_consumer.clj#L36 ). I think that's because the intended design patterns are not obvious every one.

If you all had good reasons not to include it originally, I'd rather personally prefer to see those reasons explained than just see it added. 😉

well, as you said above, it's not intended to be a "back message" from consumer to producer

there are however some cases where it's pretty useful

one issue is what if closed? returns false?

and then you do something, but asynchronously another thread calls close!

there's a race there that does not have a solution (in the general sense, although your code may not be structured to allow this)

http://dev.clojure.org/jira/browse/ASYNC-126 is the ticket for this btw

I know @tbaldridge has elaborated on a better use case for this somewhere but not sure where that was now

Are #C05423W6H folks missing out on this discussion?

totally. suckers.

seriously though, yeah would be nice to move this to #C05423W6H

Woah, didn't realize there were 220 Slack channels here. (This is the problem with inexpensive channel abstractions. Ka-ching!)

damn

any idea what could cause this error? #<CompilerException java.lang.NoClassDefFoundError: clojure/lang/Tuple, compiling:(reverie/module/entity.clj:1:1)>

i get that when running lein repl

tried lein clean as well, but to no avail

maybe AOT compilation against 1.8 and running with Clojure < 1.8 ?

or generally just AOT compilation of external deps when there shouldn't be

seems to be. i recently moved reverie over to 1.8, but have a couple of sites running on 1.6. thought clearing .m2 should have fixed it, but it only went away when i upgraded the site project.clj to 1.8

but shouldn't lein clean and removing .m2 clear that?

or are there things still lingering?

as in: almost 100% certain it was all on 1.6, but might've missed it

it looks to me like reverie is compiled with Clojure 1.8 (where Tuple was added)

yeah, newer version. alpha1 is 1.6

probably a mixup somewhere in there with all the modules 😕

Is there a ClojureScript library for creating HTML slide decks? As in, I want to trade in Keynote for ClojureScript (+ Figwheel)? Curious if there's a good solution before I try to roll my own.

@alexmiller: Ken Tilton used a priority queue like that in his GUI frameworks based on cells: https://github.com/kennytilton/celtk/blob/master/celtk.lisp#L75-L108

@hlship: I would love to know what you find. I need this as well for Clojure/west.

@hlship see this talk too… https://www.youtube.com/watch?v=G7Z_g2fnEDg

Yeah, I was going to brush off my devcards for this.