@jkrueger: So how are you building your Clojure? Are you just shipping as source?

for a service that has a lot of video uploads, how do i chose between clojure and clojurescript + nodejs?

@spradnyesh I think the argument for node comes when you have short lived connections, because node spins up faster than the JVM, but if your service is long running then I personally feel that the JVM is a better option because of its rock solid foundation.

Also I always had the feeling that profiling in clojure was a lot more mature, and this would be very useful if your service becomes slow and you want to figure out your bottleneck

The plus side of CLJS is I guess, js everywhere?

If you use cljc, it is relatively easy to have the same code running on both the server side and the browser. I'd definitely go with Clojure for the server side on average, better performance especially in highly concurrent situations. Startup time is pretty irrelevant for server applications.

hey @spradnyesh: how much is a lot of video uploads? you are streaming to your own storage or to someone elses?

Question: has anyone here used kafka & clojure together?

bonus points if you’ve used this library and would let me pick your brain a little bit https://github.com/gerritjvv/kafka-fast

hi guys I have a question regarding core.async, I want to untap and close a channel when I unmount a component. My strategy is to send a message so it can closes itself, but I'm not so familiar with how to do it right with alt! and go. This is the code :

(async/tap components-mult c-tap)
     (go
       (alt! c-tap ([v]
                    (if (= (:action v) :close-tap)
                      (do
                        (async/untap mult c-tap)
                        (close! c-tap))
                      (do
                        (prn "v" v)
                        (do-stuffs v)
                        (recur))))))))

@shayanjm: used kafka, not that library

bfabry: gotcha. having trouble spinning up consumers using that library

which is unfortunate, because making producers work was (deceivingly) simple

Is there any way to set things up so that Class.forname finds a var defined by a reify?

@cddr depends when the thing is reified. I just used Class.forName to find one that was created during compilation. but if it's at runtime I doubt it

Oh, that might be good enough. I was just testing at the repl and it couldn't find it

user=> (compile 'bar.foo) bar.foo user=> (use 'bar.foo) nil user=> (str (class reified-thing)) "class bar.foo$reify__2350" user=> (Class/forName "bar.foo$reify__2350") bar.foo$reify__2350 user=>

so you'll need to aot the reify statement probably

11326-storage:bar bfabry$ cat src/bar/foo.clj (ns bar.foo) (def reified-thing (reify Object))

Sweet! I think that's going to work. Thanks @bfabry

np

@shayanjm: funnily enough I'm integrating kafka as we speak, using that libary

@shayanjm: I also had trouble with consumers because the library mandates a redis-based setup and I am too unfamiliar with kafka to stray from the ZK setup, so I've actually ended up using clj-kafka for the consumer and kafka-clj (kafka-fast) for the producer

@shayanjm: worth having a look at Bifrost which uses Kafka to write to S3. https://github.com/uswitch/bifrost

there's also https://github.com/pyr/kinsky worth mentioning, it's relatively recent but knowing the author it should be a quality lib.

thanks for the kind words

kinsky targets kafka 0.9.0, and so far if you do, safe for the java client there is no other clojure lib

@pyr, quick question - did you try ztellman/manifold and if so, why did you choose core.async for kinsky?

I'm not a 100% sold on manifold just yet.

is it mostly reliability concerns or API-wise?

you can use them together if that's your thing (manifold can interface with core.async), or write a thin interface to it like kinsky.async does for core.async

@dm3: but it really boils down to our use of core.async internally

and kinsky is a factor-out of our kafka lib

sure, just wanted to hear an opinion

when stuff ends up copied across several projects, we often end up publishing a lib

i didn't split the core.async part like i did in mesomatic

but might do that eventually

the idea behind kinsky is to factor our more

especially our processing logic which relies on dual topics + log compaction

(roughly outlined here: http://spootnik.org/entries/2015/04/23_hands-on-kafka-dynamic-dns.html)

ok thx

event sourcing with snapshots in kafka

@dm3: and in general, I would say I am very conservative with external libraries, I tend to default to clojure.core and then bolt-on as few as possible

I can see your point. Although I default to manifold instead of core.async if I don't need cljs

however not running anything super critical

@jasonbell: @shayanjm: unrelated to clojure, but kafka connect will now service that use case well also

anyone some favorite clojure util libs that have stuff like map-vals? ideally cljc

medley

@dm3: perfect, thanks!

@lvh: I think you can try with conso (https://clojuredocs.org/clojure.core.logic/conso) and some recursion to define the goal (`l/membero result events`) for every member of results. You could also try with permuteo and appendo but not sure if it will work (they have to be proper collections, so they have to be grounded). You can also try asking in #C0566T2QY

nberger: thanks for the hints! my attempts at writing a recursive version always fell short in that it expected to know a priori how big the result set should be

I may be able to statically define that by inspecting the query I’m getting in (that’s a data structure)

I've got a mental block, there's got to be a way to simplify this: (if-let [v (:v m)] (hash-set v) (hash-set)) I'm trying to end up with either an empty set out a set of one item, and not a set containing nil. A separate issue I have is that i can't find a curly brace key on my phone :p

@acron: ahhh, yeah I think the guy brings up some good points about ZK, so I wanted to try out his redis set up

it looked cool, too bad I can’t make it run 😕

@sashton: (conj (hash-set) (:v m))?

(that's assuming you can't find your curly brace key

if you do probably (conj #{} (:v m))

@jethroksy that leaves me with a set containing nil, rather than an empty set

@sashton: my bad i meant into

eh

wait i had this working just now

wait into should work

(into #{} (:v m))

@jethroksy I should have mentioned that the value in :v is not a collection: (into (hash-set) (:v (hash-map :v 1)))

That would throw an error

you could use quote/unquote

but i'm not sure if that's simpler anymore

ah wait the nil problem would persist

Quote/unquote is probably a bad habit because i don't think this won't work in Clojurescript.

@sashton: how about (if (some? x) #{x}) #{}) if I undertand you properly?

hm, no that;s not exactly what you want

feels like that's about as simple as it gets

conj and into both won't work

Yeah, I think (if-let [v (:v m)] #{v} #{}) is the most sensible option.

is there a reason you need the empty set?

today I learnt...

Or alternatively something like this (or (some->> (:v m) (conj #{})) #{}) though it's a bit too clever maybe.

oh wait

One reason he might want a set I can think of is that he might be testing against the set.

seq doesn't work on sets

It doesn't?

It should.

Vectors aren't seqs either

seq returns a sequence from the collection.

i meant seq?

Yeah I'm testing the set against another sequence

so (seq #{1 2 3}) will net you a sequence of 1, 2 and 3 in some order.

yup

I guess at this point I should just wrap my if-let into a function and call it good.

if having a nil inside doesn't hurt you could just #{:v m}

i meant #{(:v m)}

Truth be told, I am doing this in clojurescript, but it seems like it's general enough clojure functions so I asked here

I could see a case where I need to check if the set is empty, so the nil gets in the way

@sashton: incidentally

do you have more of those maps?

or just one?

Because if you have more of those something like this might make sense (into #{} (keep :v ms))

No, it's actually a nested map so I am calling get-in.

Yeah, so what you're doing already is probably the most sensible then; (into #{} (keep :v [m])) feels to me like an attempt to be too clever.

Is there a way to handle runtime config in an idiomatic way?

is there a way to destructure a let binding with a prefix ?

like r1/path r2/path...

You mean :namespaced/keywords by that or something else?

@polygloton, @seriousbug @jaen - of ring-swagger - not all Prismatic Schema records can be mapped to JSON Schema and vice versa. We are planning to build a pure Schema-based web/api documentation lib - should be a better fit for e2e Clojure(Script).

Also, did a spike on visualizing schemas with Graphviz, UMLGraph-like. Might be useful as a repl-tool.

jaen: I mean with the current form path method and params are redefined on the second line

I would add something like r1 in front of them to avoid that

Oh, I see. Nothing I'm aware of, you probably have to {path2 :path method2 :method params2 :params} unfortunately

@yenda: actually you can do (let [{:keys [:r1/path :r2/method]} req] (do-thing path method)) @jaen ; the namespace is stripped and the second half is used as the vbl name

@jaredly: Yeah, I know about the namespace being stripped, but this is exactly the problem - in his case that would mean they would clash, wouldn't it?

there's no trick to avoid that problem afaik

other than renaming per the second set of keys like @jaen did above

ohhh right

you didn't share what's in the let, but it's entirely possible there is some way to restructure the code at a higher level (via merge / merge-with)

can anyone point me to some reference reading for making a (stuartsierra) component depend upon a core async channel? My case is very simple right now, I don’t need to broadcast to many components with a mult or anything like that.

hey @uwo- do you have a specific problem in mind? I haven't seen it documented explicitly but a common pattern is to keep 2 chans in component state- a work-chan and a stop-chan- and have component/start spin up a go loop that consumes work on work-chan and listens for a signal on stop-chan to stop work. component/stop would use stop-chan to signal the go loop to stop. Does that help? Of course there are many permutations depending on the needed communication pattern.

@jonahbenton: hey, thanks. I should probably make use of that pattern. I’ve got a watcher component and a worker component, and I figured it would make sense to have them both depend on a channel; the watcher would produce and the worker consume. That said, I’m mostly curious where or when I add the channel to the system-map. Would the system construction look like this?:

(defn system
  [cfg]
  (let [system-map (c/system-map :work-chan (a/chan 10)
                                 :stop-chan (a/chan)
                                 :watcher (ew.watcher/component cfg))]
    (c/system-using 
      system-map
      {:watcher [:work-chan]})))

@uwo gotcha- so the members of the system-map have to be components and participate in the lifecycle protocol, and chans can't do that directly. so maybe instead have 2 components, :watcher and :worker, have watcher create the channels and have them be part of component state, then have :worker depend on :watcher, and pull out the channels from :watcher state?

oh? Did I misunderstand this quote from the library? Any type of object, not just maps and records, can be a component if it has no lifecycle and no dependencies. For example, you could put a bare Atom or core.async Channel in the system map where other components can depend on it.

(in https://github.com/stuartsierra/component#usage-notes)

channels can be in the system map.

ah, indeed, sorry, i misspoke, was speaking from personal preference has been to keep resources within components so that lifecycle of those resources are managed together

is referring directly to reloaded.repl/system in an application bad? like this example: https://github.com/danielsz/sente-system/blob/master/src/clj/example/my_app.clj#L63

trying to figure out how to accomplish that same thing, accessing a sente component's members, but it seems like the way it's done there isn't ideal?

Use something that doesn't require you to define things globally, I suppose?

@bfabry: Cascalog looks great but I can't wrap my mind around it. I don't come from a datalog background. Just trying to do something simple like a substring search in a field seems pretty obtuse. Maybe it's just me. Guess I'll have another look at PigPen.

jaen: thanks, so you're suggesting the issue may be tied to sente's design? (not trying to come off as snarky here, or incite trash talkign a lib, just feel like i've been struggling to grasp the proper way to do it every time i try to add a new type of component)

@jjttjj: nah sente's fine, I was talking about compojure. defroutes is a global state, you can't wrap that in a function so you can pass sente as a parameter to close over.

I don't know if compojure supports non-macro route definitions though.

Something like bidi might be a better fit with component.

Though now I look at weavejester's duct and it seems you can use compojure with component somehow

@jjttjj: If you're not comfortable referring to the system directly, you can do it with dependency injection. There are examples in the system repo.

Hey guys! I was thinking of introducing a lazy value-based dependency graph for a pure Clojure (not ClojureScript) project. I tried looking around for libraries but only found https://github.com/hoplon/javelin, which appears to be CLJS only, https://github.com/gcv/dgraph, which doesn't seem to be battle-tested or actively maintained, and https://github.com/richhickey/clojure-contrib/blob/master/src/main/clojure/clojure/contrib/dataflow.clj, which doesn't appear to have been migrated over. Are there other libs that do this that I haven't found? Either way, what's a good choice?

(Must not only support describing transitive dependencies, but must specifically include the Excel-like lazy/caching formula capabilities.)

@artiavis: Graph may fit that definition https://github.com/plumatic/plumbing

@jjttjj: But the general idea that seems sensible with component to me is have a protocol like RouteProvider and HandlerProvider that return handlers and routes that then a webserver component can compose together and make a handler out of. You could take a look at juxt's modular for that approach. I think duct is somewhat similar, but I didn't yet have a chance to try it.

@jjttjj: I tried to rip some simplified example from my thesis, maybe it'll help you somewhat - https://gist.github.com/jaen/6d9fdc0819ce662ba24b

@bfabry: wow this looks pretty interesting

@bfabry: I'll follow up on the mailing list if I have more questions

@artiavis: I’m the author of dgraph

You’re right that I don’t actively maintain it at the moment, but am happy to talk about it.

FWIW, dgraph does have the lazy and caching semantics you’re describing.

Lack of bugs not guaranteed. 😄

@gcv: hey! I really liked what I saw, but it has very few downloads on clojars, and you even mentioned on the readme that it has potential bugs with things like threading and mutating the graph on the fly

Yes, I remember discussing it with Rich a few years ago, and he pointed out a potential memory leak. I’d have to spend some time thinking about it to remember the exact scenario that would cause it to happen.

for the use case I have in mind, I'd need to be able to wire/unwire bits from the graph on the fly

How large a graph?

For https://github.com/cryogen-project/cryogen-core, so basically for caching the compilation process for very large sites. The graph won't necessarily be very large, but I'd need to be able to leave the graph running and keep adding or removing from it.

got it

@gcv: out of curiosity, why did you write dgraph instead of using clojure-contrib.dataflow?

IIRC dataflow is destructive.

dgraph is purely persistent in the same way as Clojure maps are (it uses maps under the hood).

I currently think dgraph can be (1) brought up-to-date with modern Clojure, and (2) the leak bug fixed in a few days of work. The trick is finding those few days in my schedule to do that.

@gcv: I see. I got that dgraph is nondestructive from the examples, but didn't find any samples of dataflow handy from which to compare.

@gcv: have you seen the plumbing library? at first blush it looks pretty similar in premise but more fleshed-out