Fork me on GitHub
#clojure
<
2017-07-28
>
joshjones02:07:50

@mordus and though you’re building your own, clojure has a PersistentQueue, if that helps you in some way, and peek/pop/conj work on it

josh_tackett03:07:58

Anyone know a good way to do synchronous REDIS pub/sub? Basically publish a message, and wait for the message response before moving forward evaluating?

joshjones03:07:15

@josh_tackett in a pub/sub model, you publish a message, but you don’t receive any response. Maybe you mean that you’d like to wait until the publish has completed before continuing. I’m not a redis expert by any means ,but I don’t think it’s possible to do this in redis (Kafka, by contrast, allows you to both pass in a callback function and also returns a Future that you can use to determine completion of the publish). Consider whether you actually need to know when the publish is complete, as one of the main benefits of a pubsub architecture is that the publisher really knows very little about who will consume its messages

josh_tackett04:07:32

@joshjones Yep I've used the classic pubsub structure, but what I'm describing is a special use case. No worries though I think I figured it out.

donyorm05:07:00

So I'm making a clojure desktop app that relies heavily on plugins. I'm thinking about using pomegrante to load them on the classpath (from a specific directory). Is that a reasonable idea, or is it likely to break?

zilti07:07:45

In datascript, is there a way to only return results based on how many entries in a ref they have? E.g. only return results X that have 2 Y stored in X.refY

Niki07:07:36

you mean, from queries?

Niki07:07:00

I can’t think of a simple way to do that

Niki07:07:51

you can

(d/q '[:find ?x (count ?y)
       :where [?x :ref ?y]]
     db)
and then filter the results yourself

zilti07:07:50

Ah. So, no countin :where.

qqq10:07:25

normally, macro expansion is outside in, however, is it possible to write a macro foo which says: (foo ...) ==> do all macro expansion of ..., then pass it to some other function for post processing ?

qqq10:07:28

seems to suggest that macroexpand-all is broken since 'pure data' is being expanded

bronsa10:07:13

macroexpand-all is not lexical scope aware

weavejester11:07:51

@hmaurer I find it useful 🙂

hmaurer11:07:53

(aka does it have any known significant issues)

hmaurer11:07:21

does it auto-wrap lines above a certain length?

weavejester11:07:49

@hmaurer It could have more features; I haven’t had time to work on it recently, but aside from that, not that I’m aware of… It might not support reader conditionals; I can’t recall.

weavejester11:07:06

@hmaurer No, cljfmt is pretty conservative.

weavejester11:07:47

There’s another one, zprint, that just removes all formatting and reapplies it.

weavejester11:07:57

Whereas cljfmt just tries to fix formatting that’s wrong.

hmaurer12:07:36

@weavejester Thanks, I’ll take a look at zprint too then!

donyorm13:07:40

I'm looking to create a plugin system for my application that will allow me to distribute it as an uberjar and then load the plugins from the classpath, ideally as jar-files without the source but I'm willing to fudge on that. Does using pomegrante (https://github.com/cemerick/pomegranate) for dependency management seem reasonable? I get the impression that having the same libraries with different versions will usually wreak havoc.

tap13:07:27

My friend asked me to give him one resource link for him to start learning Clojure. I proudly gave him this https://clojure.org/guides/getting_started. Shout out to everyone who involves in making it this great! Thank you!

qqq13:07:39

(require '[clojure.walk :as walk]) 
(println (walk/macroexpand-all '(quote (let [a 1] a)))) 
; (quote (let* [a 1] a))
 
is above viewed as a bug or a feature ?

qqq13:07:46

(and if a feature, what is the reasoning)

bronsa13:07:31

I'd say it's neither a bug nor a feature, it's a limitation of macroexpand-all, it has no awareness of lexical scoping or evaluation rules (and doesn't claim to)

bronsa14:07:12

if you want to macroexpand all with lexical scoping & evaluation awareness, use either tools.analyzer.jvm or riddley

qqq14:07:06

one of the things that makes it easy to write new DSLs in racket is the ability to write

#lang foobar
at the top of a file, then define a foobar reader/expander, and viola you can write foobar code in *.rkt files. Does Clojure have any feature like this (perhaps as a library) ?

bronsa14:07:32

clojure's reader and macroexpander are not extensible in the way that racket's are

qqq14:07:34

@bronsa : to the best of your knowledge is there anyway to get racket's "dsl building" features in clojure ?

bronsa14:07:08

use racket :)

qqq14:07:22

if it had decent jvm / js support, I would 🙂

bronsa14:07:46

clojure's philosophy is not really compatible with the language extensibility features of racket

qqq14:07:47

the tco/continuations I don't really need, but the dsl building capabilities are something else

johnj14:07:15

TCO would be really nice to have

bronsa14:07:36

would it? how many times do you actually use recursion (and more specifically, non-tail recursion) in clojure?

johnj14:07:22

0 times cause there is no TCO 😛

johnj14:07:54

we have partial TCO at least with recur

bronsa14:07:38

yes, my point is, if you can express your programs using reduce/map and loop recur if necessary, how would TCO be really nice to have

johnj14:07:43

lazy sequences somewhat alleviates the need for TCO but some algorithms are nicely expressed using mutual recursion

cjhowe14:07:01

i like that loop/recur enforces tail position for the recur call actually

tbaldridge14:07:03

And as Guido pointed out more than once about Python. TCO has a nasty habit of removing bits of your call stask you may have wanted during debugging.

joshjones15:07:05

@lockdown- trampoline for that 😉

johnj15:07:13

@tbaldridge that's feature of TCO 🙂 - if you really need that add an accumulator

tbaldridge15:07:08

You think you have confusing stack traces now? Just wait until you see "foo called bar", and you look at the source and foo never calls bar.

tbaldridge15:07:15

That's a good way to waste a few hours in debugging.

joshjones15:07:10

given that you really don’t want to use stack-consuming recursion in general, so you’ll be calling from tail position most of the time anyway, you replace foo with recur in your call, and it’s quite literally the same, except you’re being explicit that this is TCO’d … implicit would be fine, but in most cases you quite literally gain nothing, IME anyway @lockdown-

joshjones15:07:50

(defn foo [x] (if (g x) "done" (foo (f x)))) versus (defn foo [x] (if (g x) "done" (recur (f x))))

tbaldridge15:07:29

and with recur you get compiler checked tco, so you know when you messed up.

johnj15:07:02

@joshjones yes, recur was mentioned, was referring more about tail calls to other functions rather than self calls

joshjones15:07:46

yeah, using trampoline has its drawbacks but happy there’s the option

johnj15:07:00

@tbaldridge I'm not saying its a must have just that it would be nice to have the option of full TCO, Clojure being a lisp and all.

johnj15:07:32

Does the JVM even has plans to ever implement TCO?

bfabry15:07:30

I really doubt it'll ever happen

donyorm16:07:45

What's the best way to add a directory to the classpath after a clojure program has started up? I found pomegrante (https://github.com/cemerick/pomegranate), but from the readme: "add-classpath and add-dependencies should be considered escape hatches to be used when necessary, rather than a regular part of your development workflow," which makes it seem like it would be unsuitable for a production program.

hiredman16:07:44

the classpath is just something the system classloader uses to load things, you don't really want to alter the classpath, you want to create yourself a classloader and use it to load code

hiredman16:07:35

but if you are not experienced enough with the jvm to know how the classpath and classloaders work, I would suggest you find an alternative means to your ends. getting classloaders to both do what you want and behave correctly is non-trivial, and you will immediately be plunged in to the deep end

donyorm16:07:40

I'm definetely not experienced with the classpath and classloaders. Is there an alternate way to make a sane plugin system? Clojure doesn't really have any libraries dedicated to that it seems

qqq16:07:20

Are there any good guides for writing compilers inside of clojure (inside of writing compilers from clojure to some other target language.)

seancorfield16:07:38

There won't be anything Clojure-specific -- but the principles of compiler writer are the same regardless of language.

seancorfield16:07:14

The main issue you'll have is that most compiler-writing material is going to be illustrated with non-functional languages.

hiredman16:07:34

the clojurescript compiler and the go macro implementation in core.async are pretty decent examples (I don't know of any prose write ups)

seancorfield16:07:56

@qqq Here's a book about writing a compiler in a functional language http://www.cs.princeton.edu/~appel/modern/ml/ (Standard ML, but the same functional principles will apply).

qqq16:07:09

is there a nice way to translate union types from SML to clojure ?

noisesmith16:07:35

Object is a supertype of all union types on the jvm, no?

seancorfield16:07:14

@qqq Use tagged maps and multimethods?

ajmagnifico17:07:53

Transducers question: When combining functions with (comp), the functions end up being applied in right-to-left order. When combining transducers with (comp), the transducers end up being applied in left-to-right order. Is that intentional?

ajmagnifico17:07:56

I’m just trying to gain an intuition here, if there is one to be had

josh_tackett17:07:03

Anyone know the lighttable plugin for moving parens via keyboard shortcuts?

bfabry17:07:09

@ajmagnifico it's not so much intentional as just a consequence of how transducers work

ajmagnifico17:07:23

okay, this is what I’m looking for

ajmagnifico17:07:26

please explain

ajmagnifico17:07:35

if you would be so kind

ajmagnifico17:07:50

very helpful, thank you @bfabry !

bfabry17:07:47

no worries, I remember finding that consequence pretty interesting myself

juliobarros17:07:48

Anybody have tips or know of a write up or library for implementing fine grained role based permissions in a ring/compojure web app? I mean something like check that this particular user has a role with permissions to delete/modify this particular record.

juliobarros18:07:06

Thanks @bja I want something like the authorize macro but more focused and easier to use (with docs) to recommend to another developer.

juliobarros18:07:20

Also, that doesn’t depend on friend specific attributes (like identity being bound, hierarchical keywords, etc.)

plins18:07:33

is it possible to update 2 keys at once inside a map using clojure’s update function? something like

(update {:a 20 :b 10} :a inc :b dec)

bja18:07:06

all you really need to do is have some ring middleware that populates a session key containing a user-id in the ring request. then your controllers can take that user-id and build whatever authz (role-based, fine-grained permissions based on models, etc) that you want

bja18:07:57

there are a bunch of helpers for doing things like working with encrypted+signed session cookies

bfabry18:07:44

@plins not with update afaict, reduce works though (reduce #(update %1 %2 inc) [:a :b])

hmaurer18:07:57

@plins what about this?

(merge-with #(%2 %1) {:a 20 :b 10} {:a inc :b dec})

bfabry18:07:59

^that's pretty bloody cool

noisesmith18:07:51

or (-> m (update :a inc) (update :b dec))

noisesmith18:07:17

which I think tends to be the version that is easy to read and refactor

currentoor19:07:36

So how do you provide high availability? Right now in prod we just have on JVM instance that handles our web traffic, datomic peer, and background jobs. It’s handling the load fine but we don’t want it to be a single point of failure. Also it’s been nice to reason about things by just having one process. I figured we could take two approaches. The Datomic transactor approach, have a backup instance that just sits idle and takes over if the main instance dies. The other is to have a pool of instances that can operate independently, possible separate web workers from background job workers. Sort of like the way Rails does it.

tbaldridge19:07:41

@currentoor most systems I've worked on use a load balancer and more than one web server. If one server goes down the loadbalancer stops sending requests to it

currentoor19:07:11

@tbaldridge do you see anything inherently flawed about the Datomic transactor approach for web servers? Just because we built our whole architecture assuming one instance.

tbaldridge19:07:39

continuing here to allow others to use main chat

tbaldridge19:07:01

So the other problem is that a single server is rather limiting. A app that is unhappy with multiple instances needs to be redesigned, imo.

tbaldridge19:07:19

Single writer is fine, but limiting reads to a single box is a scaling issue long-term.

tbaldridge19:07:23

It's really hard to implement

tbaldridge19:07:55

you have to make sure you somehow don't have both instances running at the same time.

currentoor19:07:57

I see, Stuart Halloway made it sound simple in on of his videos. But I’m probably overlooking a lot of stuff.

currentoor19:07:26

Anyway thanks for the response. And by the way those core asnyc videos you did a while back were fantastic!

tbaldridge19:07:00

Well, remember that the Datomic transactor doesn't handle reads, only writes.

currentoor19:07:20

Ah, yeah that’s true!

tbaldridge19:07:31

And those writes are serialized. So you can do stuff like: finish your current transaction, then reject the remaining writes, then shutdown

tbaldridge19:07:50

Also Datomic only fails over once. It won't fail back to the original transactor automatically, as that could cause flip-flop issues.

currentoor19:07:15

How does serialized writes change anything? Not sure I understand that point.

tbaldridge19:07:08

You only ever have a single thread writing to storage, and that simplifies things a bit. Transactions during a shutdown can be in 3 phases: pending, writing, written.

tbaldridge19:07:35

And there will only ever be one that's being written. Pending transactions can be discarded, and then you have to figure out something to do with written.

tbaldridge19:07:59

But in general it sounds like your app is a bit more complicated than "a process that reads from a queue, indexes data, and writes it to disk".

tbaldridge19:07:23

So what are you doing that makes the server hard to duplicate?

currentoor19:07:47

Yeah it’s a SaaS for marketers. Well we have every client connect to the webserver via websockets. Then we use the TxReportQueue to push novelty to the each client over the websocket.

tbaldridge19:07:13

so what about that can't be done with multiple boxes?

currentoor19:07:48

We also have situations where we make an HTTP request for several blobs of data, the HTTP response is empty, but we load those blobs asynchronously over the websocket.

tbaldridge19:07:54

Sounds perfect, incoming data is pushed to Datomic, the TxReportQueue pushes to all the connected peers.

tbaldridge19:07:23

as opposed to sending the request over the websocket?

currentoor19:07:45

yeah, I suppose we could send the request over websockets instead.

currentoor19:07:00

the only reason we send it over http is for legacy reasons

tbaldridge19:07:42

so your current pattern would have problems with a loadbalancer out-of-the-box, but most load balancers also support things like routing all requests from a single client to the same server

currentoor19:07:12

hmm yeah that might work, then the websocket and the http requests would be guaranteed to be connected to the same instance

tbaldridge19:07:34

another approach I've used, which might work with smaller numbers of clients, is that you maintain a message queue for each connected websocket. When a user connects to the system they start receiving messages from their queue. That way if the websocket connection breaks, they can reconnect via a different server, and pick up where they left off by reading from the same queue, but on a different server.

tbaldridge19:07:13

then your HTTP requests can also go into a message queue with a "response queue" name as part of the request.

tbaldridge19:07:27

That's a rather complex model that works quite well, but has more moving bits.

currentoor19:07:44

and what would you use for the message queue bit?

tbaldridge19:07:04

I've used RabbitMQ for this, but Id probably use Kafka these days since it's immutable

tbaldridge19:07:28

so when a client reconnects they can say "I last read from message #4434" and then they can start getting data from that point on.

currentoor19:07:13

yeah the load balancer approach might be most appropriate for us

currentoor19:07:32

This has been really helpful, thank you.

hmaurer19:07:04

Hi! I would like to metaprogrammingly define a bunch of tests (based on descriptions in an EDN file). Roughly speaking I want to loop over the content of an EDN file and define deftest blocks on the fly

currentoor19:07:37

sounds like a job for a macro

tbaldridge19:07:34

Look at the implementation of deftest. You may not even need a macro. Most of the clojure.test interface is data driven, so it's possible to run a single function that does a ton of work, and reports results with custom report logic.

hmaurer19:07:02

@tbaldridge ah, so I could just have a single deftest with a bunch of assertions?

hmaurer19:07:37

e.g. a single “rest api” test block that runs tests on all endpoints?

tbaldridge19:07:03

Sure, I do that sort of thing all the time

tbaldridge19:07:16

also, although testing says it wants a string, it will take an expression as well: https://github.com/clojure/clojure/blob/master/src/clj/clojure/test.clj#L596

tbaldridge19:07:43

So you can do: `(testing (str "running-endpoint-" url) ....)`

martinklepsch19:07:11

;; breaks with EOF while reading
(some->> (identity [:#something])
         (filter identity))
;; works
(some->> (identity [:#something]) (filter identity))
whhyyyy? 😄

noisesmith19:07:36

I’d blame that on :#anything being invalid

matthavener19:07:40

is there a predicate equivalent to (or (fn? %) (and (var? %) (fn? (var-get %))) ?

hmaurer19:07:09

Is there a project (more specifically a web application) which lets you search the std lib by giving one (or many) input-output samples?

noisesmith19:07:26

@hmaurer there’s findfn which is a lib

hmaurer19:07:32

e.g. give it #(* 2 %), [1 2 3] [2 4 6] and it would tell you it’s map

hmaurer19:07:41

by trying it out on very clojure.core function

hmaurer19:07:43

exactly what I was look for; thanks! I thought about building a web app for this but if it already exists…

ghadi19:07:58

not exactly what you're asking

hmaurer19:07:16

@ghadi ha, still cool!

ghadi19:07:24

good for finding examples out in the wild.

noisesmith19:07:48

findfn gives really surprising results sometimes (eg. the long list of clojure.core functions that are effectively identity if called on one arg)

noisesmith19:07:17

but history shows I’m easily amused, so there’s that

ghadi20:07:53

is there a vim repl plugin that works without nREPL?

ghadi20:07:29

similar to emacs/inf-clojure

noisesmith20:07:20

the newer vim can do socket comms - so it should exist even if it doesn’t

noisesmith20:07:16

@ghadi using this to connect to clojure socket server would be a fun weekend hack I bet http://vimhelp.appspot.com/channel.txt.html

ghadi20:07:22

interesting, thanks!