This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-07-28
Channels
- # aleph (1)
- # arachne (4)
- # beginners (13)
- # boot (18)
- # clara (7)
- # cljs-dev (156)
- # cljsrn (278)
- # clojure (163)
- # clojure-conj (5)
- # clojure-dev (1)
- # clojure-losangeles (1)
- # clojure-poland (2)
- # clojure-sg (1)
- # clojure-spec (15)
- # clojure-uk (17)
- # clojurescript (275)
- # data-science (5)
- # datomic (23)
- # emacs (10)
- # leiningen (1)
- # lumo (16)
- # off-topic (98)
- # onyx (10)
- # parinfer (83)
- # re-frame (18)
- # reagent (47)
- # remote-jobs (1)
- # ring (1)
- # ring-swagger (5)
- # rum (6)
- # specter (8)
- # vim (5)
@mordus and though you’re building your own, clojure has a PersistentQueue
, if that helps you in some way, and peek/pop/conj work on it
Anyone know a good way to do synchronous REDIS pub/sub? Basically publish a message, and wait for the message response before moving forward evaluating?
@josh_tackett in a pub/sub model, you publish a message, but you don’t receive any response. Maybe you mean that you’d like to wait until the publish has completed before continuing. I’m not a redis expert by any means ,but I don’t think it’s possible to do this in redis (Kafka, by contrast, allows you to both pass in a callback function and also returns a Future that you can use to determine completion of the publish). Consider whether you actually need to know when the publish is complete, as one of the main benefits of a pubsub architecture is that the publisher really knows very little about who will consume its messages
@joshjones Yep I've used the classic pubsub structure, but what I'm describing is a special use case. No worries though I think I figured it out.
So I'm making a clojure desktop app that relies heavily on plugins. I'm thinking about using pomegrante to load them on the classpath (from a specific directory). Is that a reasonable idea, or is it likely to break?
In datascript, is there a way to only return results based on how many entries in a ref they have? E.g. only return results X that have 2 Y stored in X.refY
you can
(d/q '[:find ?x (count ?y)
:where [?x :ref ?y]]
db)
and then filter the results yourselfnormally, macro expansion is outside in, however, is it possible to write a macro foo which says: (foo ...) ==> do all macro expansion of ..., then pass it to some other function for post processing ?
https://clojuredocs.org/clojure.walk/macroexpand-all <-- is this bug for real ?
@weavejester would you recommend using https://github.com/weavejester/cljfmt ?
@hmaurer I find it useful 🙂
@hmaurer It could have more features; I haven’t had time to work on it recently, but aside from that, not that I’m aware of… It might not support reader conditionals; I can’t recall.
@hmaurer No, cljfmt is pretty conservative.
There’s another one, zprint, that just removes all formatting and reapplies it.
Whereas cljfmt just tries to fix formatting that’s wrong.
@weavejester Thanks, I’ll take a look at zprint too then!
I'm looking to create a plugin system for my application that will allow me to distribute it as an uberjar and then load the plugins from the classpath, ideally as jar-files without the source but I'm willing to fudge on that. Does using pomegrante (https://github.com/cemerick/pomegranate) for dependency management seem reasonable? I get the impression that having the same libraries with different versions will usually wreak havoc.
My friend asked me to give him one resource link for him to start learning Clojure. I proudly gave him this https://clojure.org/guides/getting_started. Shout out to everyone who involves in making it this great! Thank you!
(require '[clojure.walk :as walk])
(println (walk/macroexpand-all '(quote (let [a 1] a))))
; (quote (let* [a 1] a))
is above viewed as a bug or a feature ?I'd say it's neither a bug nor a feature, it's a limitation of macroexpand-all
, it has no awareness of lexical scoping or evaluation rules (and doesn't claim to)
if you want to macroexpand all with lexical scoping & evaluation awareness, use either tools.analyzer.jvm or riddley
one of the things that makes it easy to write new DSLs in racket is the ability to write
#lang foobar
at the top of a file, then define a foobar reader/expander, and viola you can write foobar code in *.rkt files.
Does Clojure have any feature like this (perhaps as a library) ?@bronsa : to the best of your knowledge is there anyway to get racket's "dsl building" features in clojure ?
clojure's philosophy is not really compatible with the language extensibility features of racket
the tco/continuations I don't really need, but the dsl building capabilities are something else
would it? how many times do you actually use recursion (and more specifically, non-tail recursion) in clojure?
yes, my point is, if you can express your programs using reduce/map and loop recur if necessary, how would TCO be really nice to have
lazy sequences somewhat alleviates the need for TCO but some algorithms are nicely expressed using mutual recursion
And as Guido pointed out more than once about Python. TCO has a nasty habit of removing bits of your call stask you may have wanted during debugging.
@lockdown- trampoline for that 😉
@tbaldridge that's feature of TCO 🙂 - if you really need that add an accumulator
You think you have confusing stack traces now? Just wait until you see "foo called bar", and you look at the source and foo never calls bar.
That's a good way to waste a few hours in debugging.
given that you really don’t want to use stack-consuming recursion in general, so you’ll be calling from tail position most of the time anyway, you replace foo
with recur
in your call, and it’s quite literally the same, except you’re being explicit that this is TCO’d … implicit would be fine, but in most cases you quite literally gain nothing, IME anyway @lockdown-
(defn foo [x] (if (g x) "done" (foo (f x))))
versus
(defn foo [x] (if (g x) "done" (recur (f x))))
and with recur you get compiler checked tco, so you know when you messed up.
@joshjones yes, recur was mentioned, was referring more about tail calls to other functions rather than self calls
@tbaldridge I'm not saying its a must have just that it would be nice to have the option of full TCO, Clojure being a lisp and all.
What's the best way to add a directory to the classpath after a clojure program has started up? I found pomegrante (https://github.com/cemerick/pomegranate), but from the readme: "add-classpath and add-dependencies should be considered escape hatches to be used when necessary, rather than a regular part of your development workflow," which makes it seem like it would be unsuitable for a production program.
the classpath is just something the system classloader uses to load things, you don't really want to alter the classpath, you want to create yourself a classloader and use it to load code
but if you are not experienced enough with the jvm to know how the classpath and classloaders work, I would suggest you find an alternative means to your ends. getting classloaders to both do what you want and behave correctly is non-trivial, and you will immediately be plunged in to the deep end
I'm definetely not experienced with the classpath and classloaders. Is there an alternate way to make a sane plugin system? Clojure doesn't really have any libraries dedicated to that it seems
Are there any good guides for writing compilers inside of clojure (inside of writing compilers from clojure to some other target language.)
There won't be anything Clojure-specific -- but the principles of compiler writer are the same regardless of language.
The main issue you'll have is that most compiler-writing material is going to be illustrated with non-functional languages.
the clojurescript compiler and the go macro implementation in core.async are pretty decent examples (I don't know of any prose write ups)
instaparse may be of interest https://github.com/Engelberg/instaparse
@qqq Here's a book about writing a compiler in a functional language http://www.cs.princeton.edu/~appel/modern/ml/ (Standard ML, but the same functional principles will apply).
Object is a supertype of all union types on the jvm, no?
@qqq Use tagged maps and multimethods?
Transducers question: When combining functions with (comp)
, the functions end up being applied in right-to-left order. When combining transducers with (comp)
, the transducers end up being applied in left-to-right order. Is that intentional?
I’m just trying to gain an intuition here, if there is one to be had
Anyone know the lighttable plugin for moving parens via keyboard shortcuts?
@ajmagnifico it's not so much intentional as just a consequence of how transducers work
okay, this is what I’m looking for
please explain
if you would be so kind
I think this thread should explain it for you https://groups.google.com/forum/#!topic/clojure/OeXZOJYydAs
very helpful, thank you @bfabry !
Anybody have tips or know of a write up or library for implementing fine grained role based permissions in a ring/compojure web app? I mean something like check that this particular user has a role with permissions to delete/modify this particular record.
@juliobarros friend can help with that: https://github.com/cemerick/friend
Thanks @bja I want something like the authorize macro but more focused and easier to use (with docs) to recommend to another developer.
Also, that doesn’t depend on friend specific attributes (like identity being bound, hierarchical keywords, etc.)
is it possible to update 2 keys at once inside a map using clojure’s update
function? something like
(update {:a 20 :b 10} :a inc :b dec)
all you really need to do is have some ring middleware that populates a session key containing a user-id in the ring request. then your controllers can take that user-id and build whatever authz (role-based, fine-grained permissions based on models, etc) that you want
there are a bunch of helpers for doing things like working with encrypted+signed session cookies
@plins not with update afaict, reduce works though (reduce #(update %1 %2 inc) [:a :b])
or (-> m (update :a inc) (update :b dec))
which I think tends to be the version that is easy to read and refactor
So how do you provide high availability? Right now in prod we just have on JVM instance that handles our web traffic, datomic peer, and background jobs. It’s handling the load fine but we don’t want it to be a single point of failure. Also it’s been nice to reason about things by just having one process. I figured we could take two approaches. The Datomic transactor approach, have a backup instance that just sits idle and takes over if the main instance dies. The other is to have a pool of instances that can operate independently, possible separate web workers from background job workers. Sort of like the way Rails does it.
@currentoor most systems I've worked on use a load balancer and more than one web server. If one server goes down the loadbalancer stops sending requests to it
@tbaldridge do you see anything inherently flawed about the Datomic transactor approach for web servers? Just because we built our whole architecture assuming one instance.
continuing here to allow others to use main chat
So the other problem is that a single server is rather limiting. A app that is unhappy with multiple instances needs to be redesigned, imo.
Single writer is fine, but limiting reads to a single box is a scaling issue long-term.
It's really hard to implement
you have to make sure you somehow don't have both instances running at the same time.
I see, Stuart Halloway made it sound simple in on of his videos. But I’m probably overlooking a lot of stuff.
Anyway thanks for the response. And by the way those core asnyc videos you did a while back were fantastic!
Well, remember that the Datomic transactor doesn't handle reads, only writes.
Ah, yeah that’s true!
And those writes are serialized. So you can do stuff like: finish your current transaction, then reject the remaining writes, then shutdown
Also Datomic only fails over once. It won't fail back to the original transactor automatically, as that could cause flip-flop issues.
How does serialized writes change anything? Not sure I understand that point.
You only ever have a single thread writing to storage, and that simplifies things a bit. Transactions during a shutdown can be in 3 phases: pending, writing, written.
And there will only ever be one that's being written. Pending transactions can be discarded, and then you have to figure out something to do with written.
But in general it sounds like your app is a bit more complicated than "a process that reads from a queue, indexes data, and writes it to disk".
So what are you doing that makes the server hard to duplicate?
Yeah it’s a SaaS for marketers. Well we have every client connect to the webserver via websockets. Then we use the TxReportQueue to push novelty to the each client over the websocket.
so what about that can't be done with multiple boxes?
We also have situations where we make an HTTP request for several blobs of data, the HTTP response is empty, but we load those blobs asynchronously over the websocket.
Sounds perfect, incoming data is pushed to Datomic, the TxReportQueue pushes to all the connected peers.
as opposed to sending the request over the websocket?
yeah, I suppose we could send the request over websockets instead.
the only reason we send it over http is for legacy reasons
so your current pattern would have problems with a loadbalancer out-of-the-box, but most load balancers also support things like routing all requests from a single client to the same server
hmm yeah that might work, then the websocket and the http requests would be guaranteed to be connected to the same instance
another approach I've used, which might work with smaller numbers of clients, is that you maintain a message queue for each connected websocket. When a user connects to the system they start receiving messages from their queue. That way if the websocket connection breaks, they can reconnect via a different server, and pick up where they left off by reading from the same queue, but on a different server.
then your HTTP requests can also go into a message queue with a "response queue" name as part of the request.
That's a rather complex model that works quite well, but has more moving bits.
and what would you use for the message queue bit?
I've used RabbitMQ for this, but Id probably use Kafka these days since it's immutable
so when a client reconnects they can say "I last read from message #4434" and then they can start getting data from that point on.
i see
yeah the load balancer approach might be most appropriate for us
This has been really helpful, thank you.
Hi! I would like to metaprogrammingly define a bunch of tests (based on descriptions in an EDN file). Roughly speaking I want to loop over the content of an EDN file and define deftest
blocks on the fly
sounds like a job for a macro
Look at the implementation of deftest. You may not even need a macro. Most of the clojure.test interface is data driven, so it's possible to run a single function that does a ton of work, and reports results with custom report logic.
@tbaldridge ah, so I could just have a single deftest with a bunch of assertions?
Sure, I do that sort of thing all the time
also, although testing
says it wants a string, it will take an expression as well: https://github.com/clojure/clojure/blob/master/src/clj/clojure/test.clj#L596
So you can do: `(testing (str "running-endpoint-" url) ....)`
@tbaldridge brilliant, ty
;; breaks with EOF while reading
(some->> (identity [:#something])
(filter identity))
;; works
(some->> (identity [:#something]) (filter identity))
whhyyyy? 😄I’d blame that on :#anything being invalid
is there a predicate equivalent to (or (fn? %) (and (var? %) (fn? (var-get %)))
?
Is there a project (more specifically a web application) which lets you search the std lib by giving one (or many) input-output samples?
@hmaurer there’s findfn which is a lib
exactly what I was look for; thanks! I thought about building a web app for this but if it already exists…
findfn gives really surprising results sometimes (eg. the long list of clojure.core functions that are effectively identity if called on one arg)
but history shows I’m easily amused, so there’s that
the newer vim can do socket comms - so it should exist even if it doesn’t
@ghadi using this to connect to clojure socket server would be a fun weekend hack I bet http://vimhelp.appspot.com/channel.txt.html
oh, nice