Bore da

@gjnoonan You'll be at ClojureX this year! W00t! I'm usually running around in a panic but come and find me.

morning

morning.

@otfrom not a direct Clojure question but more about architectural decisions. I've used Kafka and I'm a bit of a fan but I've got a problem that needs a messaging infrastructure that has fairly low volume, low rates per second i.e. < 1000 messaages per second (small messages). I think Kafka is probably not required and RabbitMQ is already in place so thinking to use that as it saves us setting up and maintaining new/different infrastructure and don't have to write our own mechanism for delivery guarantees. Does that sound sensible? I guess the real question is what advantages do I get with Kafka and what trade off's and when do you think the additional work of implementing delivery guarantees using Zookeeper are worth the advantages (i.e. high volume/performance)?

morning

@otfrom follow up question: Is there a good Clojure library out there that helps with the consumer delivery guarantees (I know about https://github.com/yieldbot/marceline for Storm but what's good for just a simple Clojure consumer? I can see options like: https://github.com/pingles/clj-kafka and https://github.com/ymilky/franzy but not sure what trade off's are?

@agile_geek it sounds like you've pretty much made your decision and it's always easier to make use of what you've got, especially if there's operational experience with Rabbit.

not that familiar with rabbit, but kafka get's you at least once delivery, easy scaling, and persistency

we're using franzy, but it's a bit dead in the water at the minute, making in enquires about resurrecting it

@tcoupland yeah. Not fully my decision (I'm just advising based on fact I'm only person whose used Kafka). I've not used Kafka from Clojure or used it for over 12 months so I wasn't sure what was available now around libraries to implement message consumer semantics.

it has changed quite a bit recently. consumer offsets are not longer in ZK by default and the client has completely changed (see our kixi.comms project for some insight)

OK. I'll take a look. What are the other (sensible options) for offsets in a Kafka cluster apart from ZK?

it now stores them in itself. The consumer offsets are transmitted around on a special topic within the cluster

which i find a bit weird 🙂

the client is in some ways simpler, it just comes down to one function 'poll', but it's borked a bit because 'poll' is also the mechanism for clients sending keep-alive to the cluster. so if you have a long process time on messages, it gets a bit painful

Hmm. I know having ZK was another 'moving part' to manage but not sure I like the idea of Kafka relying on itself for this. I liked the fact that Kafka did one thing and did it well. However, it's just my initial gut feeling and I guess you can still use ZK if you want but it adds work.

o you still have ZK there, it handles the partition assignment. no reduction in parts

and some broker bits and bobs, now i'm looking at it

OK so I guess the reason offsets are in Kafka is to simplify code in consumers?

well, not really. As you still end up managing your offset as much as you every did.

the main justification seems to be performance

o, if your a fan of core.async (like me), it's worth knowing about this multi-year old bug that's still rolling on: http://dev.clojure.org/jira/browse/ASYNC-100. hopefully save someone else having a seriously bad day 🙂

re. clojure kafka libraries. clj-kafka does not support newer kafka versions afaik: https://github.com/pingles/clj-kafka/issues/87

@tcoupland not sure I'm a 'fan' (as it sometimes makes my head hurt, I'm a simple soul and extracting values from non blocking go blocks always trips me up) but I certainly use it a fair bit. So the workaround is one 'catch' and check the exception manually? Not hard but a bit annoying.

one catch, must be Throwable it looks like now. Yeah, not hard, but a mega pain in the bum if you do anything else!

Yeah I saw that. The worry is more that ppl don't know this and naively implement an exception specific catch, don't test it thoroughly and get caught out. It should be documented in big red letters in the core.async docs!

it's how old the bug is that is becoming my main concern. Opened it on 27/Oct/14 and I think it's fair to say it is a major bug that can cause people serious harm

Definitely.

@tcoupland I assume it's a pretty tough one to fix? Otherwise someone would have submitted a PR?

well they seemed to think it wasn't that hard to fix and patches have been submitted to this dupe: http://dev.clojure.org/jira/browse/ASYNC-169. Patched since may, although it looks like the tests just got tests added in september

i think this comment sums it up nicely: "are there any plans/estimations regarding this bug? we were recently hit by it, and was a very special pleasure to analyze the issue."

I’ve used Rabbit pretty heavily, and it’s performed well for me

kafka’s persistence/replay stuff always sounds good, but it just seems more fiddly to use when i’ve tried it (in small scenarios)

this was the async bug that got me: http://dev.clojure.org/jira/browse/ASYNC-178

and I’m just off to check my use of catch blocks...

@tcoupland So this raises a bigger question. What happens to the stewardship of Clojure if Cognitect go under? I'm never too sure who has rights to do what, other than Rich holds copyright for language.

@agile_geek - There's a scary thought...

I might finally be in a position to use Clojure for more than "a little tiny bit of the stack" and you're talking about stuff like this..? Argh! *hair turns white*

that is a scary thought. I dread to think what would happen.

looks like my catch blocks are broken 😞

@agile_geek guess we’d better give them lots of money for consultancy so that they don’t go under!

what if some big fish buys them? <evil grin>

maybe some more bugs will get fixed 😛

worth mentioning what David Polak(?) said a few years ago about this.. Cognitect doesn't depend on the success of Clojure.

agile_geek as it is licensed under eclipse we could carry on from the last changes under that license under a fork. We might have to change the name for trademark reasons though

@otfrom who's this 'we' you speak of?

we the community

obviously we'd really miss Rich's stewardship, but emacs survived RMS handing it over

sorry, rms

well, luckily the community is larger than just you or me

we could always move over to lisp flavoured erlang. That might be better way of handling core.async things anyway. 😉

I've watched a bit of a talk by Richard on LFE but it didn't go into any code just the language architecture so I have little opinion. I am a bit of a JVM fan boy tho

@mccraigmccraig what's ticking all those boxes for you at the mo?

@tcoupland i have been following the development of https://github.com/LuxLang/lux - it's always looked conceptually awesome, and it's getting close to being usably awesome

other than that there is shen, but that's not properly open-source which very much puts me off

there is a new racket based one as well

a bit too new though https://github.com/lexi-lambda/hackett

Lux does sound interesting, i to am pretty bound to the jvm

scan the book @agile_geek - it's a very easy read, at least until you hit the monadic macros stuff https://www.gitbook.com/book/luxlang/the-lux-programming-language/details

@mccraigmccraig now, now... you should know 'ease' is a relative term based on your formative experiences therefore what's easy to you may not be to me. 😈

true @agile_geek , though i remain unconvinced by your self-deprecation !

Ask @otfrom...he knows how slow I can be 😉

ha, you broke the slack @mention regex

Morning

I'm good at breaking stuff

I just like breaking shit

I usually break my own sh*t tho.

I don't need to break things... it never works in the first place... :thinking_face:

@mccraigmccraig your self-deprecation comment has made me think...maybe I need to mark myself with '@deprecated @see daughters for currently supported implementation'

@agile_geek i shall read your nametag carefully next time i see you at a clojurey thing !

Hi all.

I'm trying to write a macro generating macro.

Struggling with the quoting

(defmacro build-macro
  [n]
  `(defmacro ~(symbol n)
     [env]
      (print ~'~env)
      ))

It's the 'env bit that's causing issues.

ah sorted it 🙂

(defmacro build-macro
  [n]
  `(defmacro ~(symbol n)
     [~'env]
      (print ~'env)
      ))

I can't say I find the quoting/unquoting particularly friendly.

@mattford did you try using an auto-gensym (e.g. env# ) instead of quoting ? it makes bindings easier to read and avoids the capture pitfall (find "auto-gensym" here http://www.braveclojure.com/writing-macros/ for details)

ah yes that works much more nicely.

I have these macros

(defmacro with-env
  [env & body]
  `(dopartial [~env] ~@body))

(defmacro with-service-tags
    [env & body]
  `(dopartial [~env] ~@body))

(defmacro with-user
   [env & body]
  `(dopartial [~env] ~@body))

I was trying to write a macro to generate them but failing miserably.

Something like this

(defmacro defwith
  [n]
  `(defmacro ~(symbol (str "with-" n))
     [env# ~'& body#]
       `(dopartial [~'~env#] ~'~@body#)
     ))

what's the expected output from those macros @mattford ? they all look identical from here...

They are identical excepting the names.

and what does dopartial do ?

I can write

(with-env "sit" 
  (with-user "stg" 
    (println "matt"))

"sit stg matt"

dopartial is more general form of doto.

(defmacro dopartial
  [ x & forms ]
  `[~@(map (fn [f]
             (if (seq? f)
               (if (= 'dopartial (first f))
                 `(dopartial ~(into x (second f)) ~@(next (next f)))
                 `((partial ~(first f) ~@x) ~@(next f)))
               `(partial ~f ~@x)))
           forms)])

I also wrote that. This really isn't the right approach for what I want. But I've a need to get the broken design working to satisfy my ocd 🙂

There's a problem with arrays vs strings in the macro aliases (for want of a better term).

But I just want my macro generating macro to work as I'd expect it to.

have you discovered macroexpand-1 @mattford ?

yeah

it's sensible output for a whilst

then just explodes with seqs and concats

yeah, writing macros is often a pita

i can see a possible problem with your dopartial macro too

in that it's matching against a naked 'dopartial symbol, but probably wants to work with a namespace qualified symbol too

if the macros all do the same thing, can you just write one macro and then alias it under different names?

^ winner

Could do 🙂

What's that called then?

just def

eg. (defmacro my-macro …)

(def some-alias my-macro)

(def another-alias my-macro)

ah right easy peasy

easier than learning how macro syntax works!

not that you shouldn’t do that, but....

Can I conquer the OCD and walk away from my macro generating macro.....

Sadly I suspect not 🙂

So I should use name for the if check matching?

actually, apparently you can’t def like that on a macro 😞

my bad

but apparently you can do this: (def #^{:macro true} my-or #‘or)

anyone booked accomodation for EuroClojure yet?

@benedek are the uswitch gang going out?

to EuroClojure you mean @jonpither ?

Yeah

Is there a way in clojure to specify a function body based on the form/type of the arguments?

eg. different bodies for string and arrays?

Multi methods.

Would detail further but 🍕

nice!

But I really want multimethod macros

hmm can a function know it's own name?

Great to see we have ~50 people coming to the London Clojurians talk on Tuesday night at SkillsMatter (so we should get a decent sized room) https://skillsmatter.com/meetups/8885-yada-2-0-and-clojure-efficiency-pitfalls

there is nothing stopping you calling a function from a macro @mattford ... you can also use the syntax-quote from regular functions

@jonpither: not yet, any hotel recommendations?

we are thinking staying south of the river. Local sources tells me it's a bit quiet close to the venue

great stuff @jr0cket I need to come down to one.. one of these days! Unless anyone has a spare time machine?

I've come to the end of my great macro experiment 🙂 Many thanks to all for the help and for putting up with my spam. The final working version looks like this:

(defmacro defwith
  [n]
  `(defmacro ~(symbol (str "with-" n))
     [ x# ~'& forms# ]
     `[~@(map (fn [f#]
                (if (seq? f#)
                  (if (re-find #"^with-" (name (first f#)))
                    `(~(first f#) ~(into x# (second f#)) ~@(next (next f#)))
                    `((partial ~(first f#) ~@x#) ~@(next f#)))
                  `(partial ~f# ~@x#)))
              forms#)]))

(defwith profile)
(defwith env)
(defwith user)
(defwith tags)

(with-profile ["non-prod"]
  (with-env ["sit"]
    (println ": parameters are prepended")
    (println ": println will be swapped out to an `ssh` command"))
  (with-env ["stg"]
    (println ": different env")
    (with-tags ["bastions"]
      (with-user ["matt"]
        (println ": use params to discover/login and then do some stuff on a box")))))

non-prod sit : parameters are prepended
non-prod sit : println will be swapped out to an `ssh` command
non-prod stg : different env
non-prod stg bastions matt : use params to discover/login and thendo some stuff on a box

@jonpither i guess so but don’t know anything specific yet. apart from some really nice looking tshirt designs 😉

will ask around tmrw

Haha 😂 I love how ML has been used here http://ahogrammer.com/2016/11/15/deep-learning-enables-you-to-hide-screen-when-your-boss-is-approaching/

https://github.com/Hironsan/BossSensor