This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-09-28
Channels
- # ai (1)
- # beginners (239)
- # bitcoin (1)
- # boot (4)
- # cider (5)
- # clara (3)
- # cljs-dev (16)
- # cljsjs (5)
- # cljsrn (1)
- # clojure (152)
- # clojure-android (3)
- # clojure-dev (3)
- # clojure-greece (4)
- # clojure-italy (5)
- # clojure-spec (14)
- # clojure-uk (24)
- # clojurescript (81)
- # data-science (1)
- # datomic (47)
- # devcards (34)
- # docs (3)
- # emacs (6)
- # ethereum (1)
- # events (9)
- # flambo (5)
- # fulcro (11)
- # graphql (1)
- # immutant (3)
- # lein-figwheel (2)
- # leiningen (2)
- # luminus (6)
- # lumo (90)
- # off-topic (25)
- # proton (2)
- # re-frame (44)
- # reagent (5)
- # ring (7)
- # ring-swagger (11)
- # shadow-cljs (11)
- # spacemacs (6)
- # vim (13)
- # yada (25)
Hey everyone I'm new to clojure and trying to pick a stack for my first production project. I was looking at korma for the ORM... does anyone have experience with it? would you recommend it?
Most clojurists would probably recommend looking at honeysql or hugsql instead of korma if you need a sql interaction library.
korma is heavy on macros, while honeysql uses data, and hugsql uses a different approach altogether
I might be misrepresenting it; I am a beginner and haven’t used Korma, but that’s what I remember from their doc at least
> (defdb) creates a connection pool for your db spec (using the wonderful c3p0 library) to make sure resources are used efficiently. It also sets the last created pool as the default for all queries. As such, connection management is essentially taken care of for you.
when working with compojure, if you have routes defined in let’s say an project.items.route namespace and a project.lists.route namespace; is there a facility to combine them all?
like can you call defroutes macro “again” in your core.clj something like
(defroutes routes (list-routes item-routes))
I apologize for this vague question. But is it possible to use a db to persist a function? Let's say you have a function A that gets passed function B as a parameter. Is there any way A can persist B, so that it can be retrieved and used later?
There are some ways, but the better approach is to persist data. So instead of saving the function, save the name of the function, or perhaps an id that can be used to lookup the function in a hashmap
@astrashe what @tbaldridge suggests is how Onyx does it afaik
I know that I'm going down the wrong road here, that I shouldn't be persisting functions
@astrashe Np. Check out https://www.youtube.com/watch?v=kP8wImz-x4w if you have some time. It’s a talk by the creator of Onyx (@michaeldrogalis); he talks about several ways of designing with data, and I think he mentions your issue along the way
Off the top of my head, it seems the issue with persisting functions would be: what about dependencies?
If the function’s code you are persist is only calling clojure.core functions then it’s fine, but if it’s calling other functions in your codebase you would need to persist those too
No dependencies here. We have a bunch of contracts with various companies, that set up sales targets. I want to use data in our db to figure out if we're likely to hit our targets. A contract could say anything -- our business people make odd deals. 95% of them fall into two kinds of categories. But I'm trying to figure out how to store logic that will decide whether we hit a target in the edge cases
I'd like it to be possible to set it up without adding a new function to the source, but that's probably not a good goal
Mmh, in that scenario maybe storing functions directly isn’t too crazy. Don’t take my word for it, I am a beginner too, but if your contract can do anything and yet can be expressed as functions with no dependencies besides clojure.core, then maybe it’s the way to go
I think that letting people type a function into a web interface is probably a bad idea, even if I assume our users are trusted
Someone who can handle writing the function can probably handle modifying the source
Mostly, I was thinking about a problem, and that made me curious about procedure serialization and persistence
@schmee eh, depends on the language. Don't allow them to use arbitrary code from a full language, but that's kindof where DSLs come into play
@astrashe but perhaps allow them to compose predefined operators/functions?
I've used and worked with systems like that before, works really well in conjunction with dataflow programming: https://en.wikipedia.org/wiki/Dataflow_programming
@tbaldridge The predefined operators/functions approach makes sense. In practice, I think I'll be the person entering the formulas, though.
Yeah, there are some similarities
spreadsheet dataflow is a subset of the overall concept of dataflow programming
But for a really basic approach, writing a lisp interpreter for simple expressions is quite trivial:
(+ 3 (foo "bar"))
is easy to evaluate. If it's a number return it, if it's a symbol, look it up and return it, if it's a list, eval the contents and call apply
that gives you the ability to a) store arbitrary code in a database, and b) protects you against "bad code" by only supporting a small number of valid inputs.
I was curious because old lisps let you save an image of the system, and you can use that to build up data structures that blend code and data in complex ways. I'm not sure how to do that in clojure
Or just store it as a string and call eval
, lol
that's not a feature Clojure supports, mostly because the JVM makes it mostly impossible.
@tbaldridge I think there's an example of a restricted eval function in the land of lisp
@tbaldridge but the larger question is, let's say you have a db with lots of records. For most records, the value of field1 is just static data. But for a few, it's the result of a function. I think the answer is to just accept that you'll have to put your functions in the source code, and use a schema that will let you figure out what to do
If I’m looking for help understanding why a Hackerrank solution is timing out, would this be the best channel?
It’s the Maximum Element problem 1 x -Push the element x into the stack. 2 -Delete the element present at the top of the stack. 3 -Print the maximum element in the stack.
(defn get-query [] (map #(Integer/parseInt %) (clojure.string/split (read-line) #" ")))
(defn stack-stepper [query stack]
(condp = (first query)
1 (conj stack (second query))
2 (rest stack)
3 (do (println (apply max stack)) stack)))
((fn []
(loop [stack '()
queries-left (Integer/parseInt (read-line))]
(if (> queries-left 0)
(recur (stack-stepper (get-query) stack) (dec queries-left))))))
general rule of thumb that i've seen is that if you want to implement protocols, use records, else use maps
So...
(ns mercadolibre-srv.oauth
(:require [coremgr-connector.access :as access]
[coremgr-connector.oauth :as oauth]))
(defrecord oauth [auth credential])
(defn- store->credential [store]
(first (filter #(= "MercadoLivre" (:type %)) (:credential store))))
(defn status-ok? [status]
(= status 200))
(defn- valid? [result]
(and (status-ok? (:status result))
(= (count (:body result)) 1)))
(defn get [config]
(let [result @(oauth/generate (settings :username)
(settings :password)
(settings :client-id)
(settings :client-secret))]
(when (valid? result)
(->oauth (:body result)
(store->credential (first (:body result)))))))
(defn get-credential [oauth]
(when-not (empty? oauth)
(let [result @(access/credential (-> oauth :auth :token)
(:credential oauth))]
(when (status-ok? (:status result))
(:body result)))))
It would've been better with just a map@mjcleary i notice that your loop has order stack, queries-left
but then you recur with the new inputs swapped
it works for me in my repl but i took the unnecessary function call out:
(loop [stack '()
queries-left (Integer/parseInt (read-line))]
(when (> queries-left 0)
(recur (stack-stepper (get-query) stack) (dec queries-left))))
i modified it to run for a bit longer:
(time (loop [stack '()
queries-left 10000000
iteration 1]
(cond
(= queries-left 1)
(recur (stack-stepper [3] stack) (dec queries-left) (inc iteration))
(> queries-left 0)
(recur (stack-stepper (get-query iteration) stack)
(dec queries-left) (inc iteration)))))
9999999
"Elapsed time: 13762.254026 msecs"
nil
changing into a vector shaved off ten seconds:
user> (defn stack-stepper [query stack]
(condp = (first query)
1 (conj stack (second query))
2 (pop stack)
3 (do (println (apply max stack)) stack)))
#'user/stack-stepper
user> (time (loop [stack []
queries-left 10000000
iteration 1]
(cond
(= queries-left 1)
(recur (stack-stepper [3] stack) (dec queries-left) (inc iteration))
(> queries-left 0)
(recur (stack-stepper (get-query iteration) stack)
(dec queries-left) (inc iteration)))))
9999999
"Elapsed time: 3752.576336 msecs"
nil
user>
meaning the time remained the same as using lists or the time was the same as my results?
I really appreciate you looking. More than anything, I wanted to make sure I wasn’t complicating the solution due to my inexperience with FP and Clojure
I'm trying to get my head around a problem here and would be nice to get some insights from someone more experienced. I have an API endpoint that gives me back a status on a file upload. Possible values for a status
field are in progress
and complete
. I want to be able to write a function, that returns a promise that resolves only when status
is complete
, otherwise forces another API hit after an X delay or something. What's the best way to approach this?
why a promise?
Doesn't have to be a promise, it's just the first thing that came to my mind. Because I never know when the status is gonna be complete.
if you use future to start the thread of the upload, you can use realized?
to check if it is done
in order to check back with another api call you would need to associate that future into eg. a top level atom, perhaps a hash-map with request-id as a key
That's true, I can use futures too. My biggest problem though is how to keep hitting the API until I get the value I want.
that’s if there’s only ever one server - with multiple servers things get more complex
but, if you can use just one server (at least for now) check out atom
and swap!
and you could have eg. (defonce uploads-in-progress (atom {}))
at the top of your namespace
then inside your handler (swap! uploads-in-progress assoc unique-id (future (do-upload))
- or something a lot like that
@rinaldi you can’t have multiple clients?
@rinaldi in order to have the data accessible between two requests you would need an atom, or something like it
but how do you know which upload to check on?
OK - so really all you need to do is hit the remote api every time that endpoint is hit
@noisesmith since you are here, could you help me with this? I'm using record, see below, would hashmap be better? if so why?
(ns mercadolibre-srv.oauth
(:require [coremgr-connector.access :as access]
[coremgr-connector.oauth :as oauth]))
(defrecord oauth [auth credential])
(defn- store->credential [store]
(first (filter #(= "MercadoLivre" (:type %)) (:credential store))))
(defn status-ok? [status]
(= status 200))
(defn- valid? [result]
(and (status-ok? (:status result))
(= (count (:body result)) 1)))
(defn get [config]
(let [result @(oauth/generate (settings :username)
(settings :password)
(settings :client-id)
(settings :client-secret))]
(when (valid? result)
(->oauth (:body result)
(store->credential (first (:body result)))))))
(defn get-credential [oauth]
(when-not (empty? oauth)
(let [result @(access/credential (-> oauth :auth :token)
(:credential oauth))]
(when (status-ok? (:status result))
(:body result)))))
@rinaldi you mentioned having something that keeps track of when something has completed, you would need an atom for that, but you don’t seem to need that
@noisesmith Pretty much, yeah. I'm just struggling on how to keep checking the status until it's complete.
oh, so you don’t want to return until it’s complete?
(let [p (promise)] (future (do-upload) (deliver p true)) @p)
you don’t need to loop
oh wait, you don’t have a call that returns only when upload is done, in that case
(loop []
(Thread/sleep 1000)
(if (done)
true
(recur)))
and you might want to put that loop in a future with the promise etc. in some cases
if I have an async operation (e.g. writing a file) that returns a channel; is there a clojure idiom to signal "we succeeded!" or "we failed, here's an error..."?
@rcustodio why is it a record?
yeah, I’d just use a hash-map instead, there’s no reason to use a record for that
the basic concept is that we don’t create new datatypes unless there’s some feature we need from it
clojure has different standard design practices than erlang, records are a little faster than hash-maps, but only switch to one if you actually need it
or the rare cases where you need lookup to be faster (the difference is pretty small)
I’ve also used them to make profiling easier
(ns mercadolibre-svc.rmq
(:require [langohr.basic :as basic]
[langohr.channel :as channel]
[langohr.core :as rmq]
[langohr.consumers :as consumers]
[langohr.queue :as queue]
[mercadolibre-svc.settings :refer [settings]]))
(set! *warn-on-reflection* true)
(defprotocol IRMQ
(ack
[this ch delivery-tag])
(create-channel
[this])
(publish
[this exchange rk payload]
[this exchange rk payload props])
(subscribe
[this prefetch queue cb]
[this prefetch queue cb options]))
(defrecord RMQ [conn chan]
IRMQ
(ack [this ch delivery-tag]
(basic/ack ch delivery-tag))
(create-channel [this]
(channel/open conn))
(publish [this exchange rk payload]
(publish this exchange rk payload {}))
(publish [this exchange rk payload props]
(basic/publish chan exchange rk payload props))
(subscribe [this prefetch queue cb]
(subscribe this prefetch queue cb {}))
(subscribe [this prefetch queue cb options]
(let [ch (create-channel this)
pr (or prefetch (settings :rabbitmq :prefetch))]
(basic/qos ch pr)
(consumers/subscribe ch queue cb options))))
(defn rmq [uri]
(let [conn (rmq/connect {:uri uri})
chan (channel/open conn)]
(log/info "connected")
(->RMQ conn chan)))
in case this is a good choice? (rmq is a state passed though args)that seems reasonable if performance is critical, or you might make other things that implement IRMQ
you can just use functions and hash-maps otherwise
@rcustodio the bigger concern around all of this is that if you use functions and vanilla data, things compose and extend quite easily, and you can reuse all the standard functions and idioms of the clojure ecosystem. The more you use custom types with their own methods, the more surrounding code needs to be specially designed for your code. records and protocols are meant to act a lot like hash-maps and functions but there are differences
using a hashmap and functions only in this rmq connector would've been more idiomatic, more "clojure"
@noisesmith It works!
(defn- response-ready? [campaign-id response-id api-token]
(let [will-complete (promise)]
(loop []
(let [status (fetch-response-progress-status campaign-id response-id api-token)]
(printf "[%s] Status for response \"%s\" is \"%s\"\n" campaign-id response-id status)
(Thread/sleep 250)
(if (= "complete" status)
(deliver will-complete true)
(recur))))
will-complete))
(defn- handle-response [campaign-id survey-id api-token]
(let [response-id (fetch-response-id campaign-id survey-id api-token)
response-ready (future (response-ready? campaign-id response-id api-token))]
(when @response-ready
(printf "[%s] Zip file for response \"%s\" is ready.\n" campaign-id response-id))))
so I assume some error handler might set will-complete to false at some point?
you are checking the value of dereferencing the promise, but the only value it can ever have is true
and deref blocks, so it waits until it returns true
also, you aren’t actually looking at the value inside will-complete, you are only checking whether will-complete itself is fals or nil - you would need @@response-ready to check the actual value
as written right now, you could eliminate the usage of promise completely and just use the return value of loop itself
also, as written right now you don’t need a future either (and in fact since you don’t need it, would be better off not using it)
the code you wrote loops until complete is true and then prints and returns
@rcustodio records are not state management tools, as they are immutable - something else has to do the actual management
@rinaldi deliver isn’t doing anything useful in that code
Also I'm using a future there because Thread/sleep
would block thread execution until deliver is called
nor is the promise - you unconditionally wait on the promise
@rinaldi the @ deref blocks
nothing you are doing in that code is async
you start a future and loop synchronously, and do a blocking wait on the future
you’ve effectively wasted a thread because the calling thread can’t do anything until the new thread returns
might as well have done the work in the original thread in the first place
you can use realized?
to check if the future is done, if you don’t want to wait and loop
but waiting and looping is by its nature blocking
it seems like what you need to do is the big picture flow chart of the interactions between the client and your code and the API you use, which things should block, which things should return immediately, etc.
once you have a clear picture of the expectations there’s likely a straightforward way to implement it
I literally just wanted to keep checking for progress until done, then have a way to signal completeness
so you want to block and not respond until the async thing completes
in parallel with what?
But since this function is gonna be called with different values at the same time, would be nice having it running in a different thread
other requests? other actions inside the handling of the same request?
called by who?
@rinaldi my question is what has to be parallel, whose call needs to be parallel - some consumer of an http api you implement? some caller of your library from other code in the same vm?
So in order to check the progress status I hit a vendor API end point, but meanwhile I'm doing other things too (different API calls, in different vendors)
all in one function?
That's why I wanted to execute this specific job separated... No service relies on each other, so they can run in parallel
(let [upload (future (loop …))] … (realized? upload) …)
so the response from vendor1 is passed to vendor2?
that can’t be parallelized
then you don’t want a threading macro
Yeah, I said threading macro thinking about something else. It's something like:
(let [credentials (fetch-credentials)]
(vendor1 credentials)
(vendor2 credentials)
(vendor3 credentials))
(let [credentials (fetch-credentials)
response-1 (future (vendor1 credentials)
response-2 (future (vendor2 credentials)]
{:vendor-1 @response-1
:vendor-2 @response-2})
runs both requests in parallel, returns when both are available, errors if either errorsGotcha, this part is fine. We were working inside one of these vendors, which require me to constantly check for progress, as you saw.
Since they are being called in parallel I guess I could do it in a sequential manner?
@rinaldi sure, put the loop inside the future with the vendor call
but future itself returns immediately, and all the futures can thus work in parallel
but you don’t need a promise in order to do the loop - just return from the loop when you get a positive result
you return from a loop by not calling recur, so yes, you replace (deliver foo true)
with true
and it just works
But if I don't call recur
how can I request another status check? That's the part I don't get
why would you call another status check if you are done?
one branch of an if calls the check, the other returns
Exactly, if I'm done I don't need it but until it's done, that could take a few requests
you don’t need a delay for that
call recur until you are done, that’s it
why would it?
there’s no debounce in that code…
do you mean the sleep call?
I have no idea what you think that promise is doing, but I assure you that it does not make your code less agressive
you can sleep in a loop without using a promise, and get the exact same behavior
I come from JavaScript so I'm still making sense of all these features Clojure gives me to handle async stuff
a promise is a mutable container that receives a value exactly once and is afterward immutable - that’s all it is
the thing that makes this code async is future
- that returns immediately, you can use realized?
to ask if it is done, and you can use @
to do a blocking wait for its thread to finish
What if instead of recur I wanted to wait 1 second before checking the status again, because of business constraints, API quota usage, etc?
you already have a sleep inside the loop though
I guess you could have two sleeps but I don’t see how it would accomplish anything
@rinaldi one thing that might help your transition is asking yourself which parts of async programming patterns in js are essential to what you are trying to do (eg. the ability to launch a thing and not immediately block on the result) and which parts are accommodating limits that the jvm doesn’t have (attaching callbacks to realization of an async task etc. - things we might also do if our own logic benefits from it but might be simpler to avoid)
oh, this is all if you are using jvm clojure of course (I guess I assumed this was the case)
@noisesmith atom would be for manage state, but async.core can do it as well?? depending how write the app
core.async is good at coordination - if you have things that are not synchronized that shouldn’t wait on one another, and you need to coordinate between them in some way
so to me it solves some of the things you might solve with shared state in a simpler way
but it isn’t a way to manage shared state