This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-11-29
Channels
- # aws (1)
- # aws-lambda (2)
- # bangalore-clj (3)
- # beginners (26)
- # boot (25)
- # braveandtrue (1)
- # cider (5)
- # cljsrn (7)
- # clojure (144)
- # clojure-android (2)
- # clojure-czech (1)
- # clojure-greece (3)
- # clojure-italy (17)
- # clojure-poland (5)
- # clojure-russia (25)
- # clojure-spec (9)
- # clojure-uk (100)
- # clojurescript (85)
- # core-async (42)
- # cursive (11)
- # datascript (2)
- # datomic (25)
- # duct (3)
- # emacs (5)
- # figwheel (2)
- # fulcro (49)
- # graphql (16)
- # hoplon (8)
- # klipse (10)
- # leiningen (10)
- # lumo (9)
- # off-topic (12)
- # om (14)
- # onyx (25)
- # planck (34)
- # portkey (6)
- # re-frame (43)
- # reagent (4)
- # remote-jobs (2)
- # ring (36)
- # ring-swagger (1)
- # rum (1)
- # shadow-cljs (187)
- # specter (25)
- # sql (5)
- # unrepl (75)
(ins)user=> ,(def foo:bar 1)
#'user/foo:bar
(ins)user=> foo:bar
1
(ins)user=> {::a foo:bar}
#:user{:a 1}
@bfabry the docs specifically say it’s allowed… but it definitely feels weird
That's the thing about having a library that processes other people's code, it has to handle any possibility
Hey all. If anyone else is interested in having official New Relic support for Clojure, please upvote my support ticket. 🙂 https://discuss.newrelic.com/t/feature-idea-current-state-of-clojure-instrumentation/53105
I’d prefer to see this done via their http://opentracing.io compatibility. The opentracing api is pretty easy to deal with via java interop. I have been messing with a clojure wrapper as well, but pretty much it all converges to a with-open
that opens the trace context, and some (still being debated) machinery to propagate tracing contexts in-process across threads and such.
@gonewest818 Interesting. Thanks, will check it out
The downside to opentracing is how early it is. Elements of the api are not settled, like the in-process propagation thing I mentioned, but basic tracing is in place and multiple backends are available.
warning: clojure n00b here. i'm trying to process data from a gzipped file that has JSON documents concatenated together in it (no delimited, not even a newline). so far I have two functions that can gunzip the file and produce a reader. i've been trying to find ways to parse this into JSON. one idea is to process the character stream from the reader, keep track of curly braces and emit/print when the i'm at the appropriate place ("}{"). but I'm having a hard time keeping track of state in absence of mutable variables. another idea is to use something like re-find but it works only on strings. I don't want to read in the while file since the data I have is too large and I'd rather process it lazily since i plan to run this in aws lambda (limited memory). the functions I have so far (apart various broken experiments):
(defn in [f]
(->
( f)
(java.util.zip.GZIPInputStream.)
()))
(defn char-seq
[^java.io.Reader rdr]
(let [chr (.read rdr)]
(if (>= chr 0)
(cons (char chr) (lazy-seq (char-seq rdr))))))
i could make the process and algorithm work in python (very slow) but would prefer to use clojure. i've spent too much time (2 days) trying to figure out a way but no luck so far.. any pointers/advice at all would be greatly appreciated.You're on the right track @clojurian! You have the reader all ready to go. Let's focus on the other half of the problem: parsing a series of JSON values (i.e. concatenated)
To parse the example string, we want to turn this back into a stream, because the most JSON parsers take a stream (i.e. java Reader) as their input. And you also said you want to parse this lazily.
user=> (require '[cheshire.core :as json])
nil
user=> (doc json/parse-stream)
-------------------------
cheshire.core/parse-stream
([rdr] [rdr key-fn] [rdr key-fn array-coerce-fn])
If multiple objects (enclosed in a top-level `{}' need to be parsed lazily,
see parsed-seq.
cheshire is ready 🙂
I've elided some useless docs, and it's pointing us to parsed-seq
for your/this use case.
user=> (doc json/parsed-seq)
-------------------------
cheshire.core/parsed-seq
([reader] [reader key-fn] [reader key-fn array-coerce-fn])
Returns a lazy seq of Clojure objects corresponding to the JSON read from
the given reader. The seq continues until the end of the reader is reached.
Note that because it returns us a lazy seq, you'll have to consume it fully before closing the underlying Reader.
(This doesn't mean you have to load the whole stream eagerly into memory, we'll still be mindful of efficiency)
Typical ways to consume the seq fully are:
1. do some collection operations using the seq, and wrap that code in doall
2. reduce
/ transduce
(collecting accumulation)
3. doseq
Ok... so to make this concrete let's parse that example above. (I'll leave the task of glueing it to the GZipReader to you)
user=> (with-open [r (java.io.StringReader. example)] (vec (json/parsed-seq r)))
[{} {} {}]
@ghadi thank you so much! this is very educational. that parsed-seq
function is great. that combined with one of the 4 things you listed should do the trick.
i'm trying to digest what you said and seeing some light. i'll come back after a little while after I've tried a few things.. thank you again.
ya @clojurian. clojure is stupidly fun
it is but it's incredibly humbling too.
i really like how you built that small example to test and show the idea.
my brain needs to be disinfected of procedural thinking. unlearning is hard.
i don't know whether to be happy or be sad... after hours upon hours of frustration, all of it works in just a couple of lines.
you're awesome! @ghadi
@clojurian I think one of the things with Clojure is that simplicity is king. If you're struggling with a problem and your code seems overly complex, then there's probably a much simpler, more idiomatic solution 🙂
i'll keep that in mind.
on that a related question. i don't have any local user group or colleagues who do clojure. i was super frustrated, had a headache on, dozens of tabs open and what not.. and had no one I could ask. normally I hesitate to bother but i find that my progress is also super slow. when is it appropriate (or inappropriate) to come ask for help here?
@clojurian #beginners
sounds like the place for me. thanks @eriktjacobsen
Edit: Sorry I notice that there's a dedicated aws-lambda channel, I'll move my question there
i see that there's brandon bloom's backtick lib, but is there any way to make clojure not impose namespaces on syntax-quoted forms?
`[:find [(~'pull ~'?task ~tasks/task-shape) ...]
:in ~'$ ~'?docref
:where
[~'?task :property1 ~'?docref]
[~'?task :property2 ~'?tc]
[~'?tc :tags "tag"]]
the tasks/task-shape. I'm trying to put the query for what a task looks like in one place and then let others query against this common shape however they like
probably easiest to build it up in pieces I guess, but yeah bit annoying
[:find [`(~'pull ~'?task ~tasks/task-shape) ...]
:in '$ '?docref
:where
'[?task :property1 ?docref]
'[?task :property2 ?tc]
'[?tc :tags "tag"]]
yeah good point. use the vectors and keywords to my advantage. still not sold on it. gonna roll it around a bit
@dpsutton it seems like it would be easier to just not use ` at all there
[:find [(list 'pull '?task tasks/task-shape) ...]
:in '$ ?docref
:where
'[?task :property1 :docref]
'[?task :property2 ?tc]
'[?tc :tags "tag"]]
@dpsutton I just use https://github.com/gfredericks/misquote/blob/master/src/misquote/core.clj for that.
thanks. that's looking the cleanest so far. just makes me nervous that this stuff just silently and happily fails on datomic if its namespaced. i'm deciding which is worse the super important syntax or the duplication of the structure
thanks @rauh i saw bblooms backtick as well. any reason to choose one over the other that you know of? I think i saw brandon handles gensyms but not sure how important that is for my purposes
@dpsutton Not sure, but I use the above for exactly that purpose: Creating datomic queries. It works well. Though I changed it slightly to make Cursive happy
@dpsutton Here the full example: https://gist.github.com/rauhs/d2575e77e6e063ae94abbd9f1bca226d
yes, type hints are only to avoid reflection
if you're interested in enforcing types, you should probably consider spec or plumatic schema
or pre/post/raw assertions, if you're only using them on occasion
I use spec, but I also have a bunch of defrecords hanging around, wanted to squeeze some extra value out of those
right now I am getting familiar with a module in an app, and sprinkling type hints all over fn signatures helps to understand teh mess, also IDE's "show usages"
but it seems like specs is the ultimate way to go, if records are not really used for extending protocols, etc.
Spec is the way to go with this stuff. You can turn it on and off. I think you are going to regret extensive manual assertions and hints.
A :: map
B :: set
I want to remove all (k,v) pairs from A where k is in B
is the best way to do this (apply dissoc A B)
or is there a more efficient way ?
@qqq (reduce dissoc A B)
user=> (def A {:a 1 :b 2 :c 3})
#'user/A
user=> (def B #{:a :c})
#'user/B
user=> (time
#_=> (dotimes [_ 1000000]
#_=> (reduce dissoc A B)
#_=> ))
"Elapsed time: 120.139233 msecs"
nil
user=> (time
#_=> (dotimes [_ 1000000]
#_=> (apply dissoc A B)
#_=> ))
"Elapsed time: 335.459923 msecs"
I would imagine that with a larger sized B apply would evenually win, as the vararg version can iterate over the coll w/o having to pay an invocation price
from the peanut gallery over here: neither choice will/should dominate in your codebase
(def s1 (range 1000000))
(def s2 (map #(* 2 %) (range 500000)))
(def m (into {} (for [x s1] [x x])))
(time (do (apply dissoc m s2)
nil))
(time (do (reduce dissoc m s2)
nil))
I'm not getting a noticable difference(def s1 (range 1000000))
(def s2 (map #(* 2 %) (range 500000)))
(def m (into {} (for [x s1] [x x])))
(cc/quick-bench
(do (apply dissoc m s2)
nil))
(comment
Execution time mean : 456.384541 ms
Execution time std-deviation : 69.565195 ms
Execution time lower quantile : 411.034707 ms ( 2.5%)
Execution time upper quantile : 545.198448 ms (97.5%)
Overhead used : 2.943666 ns)
(cc/quick-bench
(do (reduce dissoc m s2)
nil))
(comment
Execution time mean : 469.385020 ms
Execution time std-deviation : 83.522059 ms
Execution time lower quantile : 393.009549 ms ( 2.5%)
Execution time upper quantile : 546.717419 ms (97.5%)
Overhead used : 2.943666 ns)
what surprises me most: (apply dissoc ...) doesn't cause a stack overflow for having so many args on the 'stack frame'
with criterium not seeing any cases where apply
version outperforms reduce
version
still seeing reduce
version out-perform for small inputs by ~10%
https://github.com/clojure/clojure/blob/clojure-1.9.0-alpha14/src/clj/clojure/core.clj#L1501-L1513
transients do actually give the reduce version a significant edge
(criterium.core/quick-bench (apply dissoc m s2))
Evaluation count : 6 in 6 samples of 1 calls.
Execution time mean : 336.386302 ms
Execution time std-deviation : 19.057339 ms
Execution time lower quantile : 319.084193 ms ( 2.5%)
Execution time upper quantile : 360.243222 ms (97.5%)
Overhead used : 1.529080 ns
=> nil
(criterium.core/quick-bench (reduce dissoc m s2))
Evaluation count : 6 in 6 samples of 1 calls.
Execution time mean : 330.780186 ms
Execution time std-deviation : 30.145095 ms
Execution time lower quantile : 291.699187 ms ( 2.5%)
Execution time upper quantile : 360.677617 ms (97.5%)
Overhead used : 1.529080 ns
=> nil
(criterium.core/quick-bench (persistent! (reduce dissoc! (transient m) s2)))
Evaluation count : 6 in 6 samples of 1 calls.
Execution time mean : 226.415054 ms
Execution time std-deviation : 7.032915 ms
Execution time lower quantile : 217.464830 ms ( 2.5%)
Execution time upper quantile : 233.430482 ms (97.5%)
Overhead used : 1.529080 ns
(def lst (for [x (range 1000000)] [x x]))
(defn massoc! [o [k v]]
(assoc! o k v))
(cc/quick-bench (into {} lst))
(comment
Execution time mean : 591.524037 ms
Execution time std-deviation : 68.112686 ms
Execution time lower quantile : 555.200521 ms ( 2.5%)
Execution time upper quantile : 709.664574 ms (97.5%)
Overhead used : 2.943666 ns
Found 1 outliers in 6 samples (16.6667 %)
low-severe 1 (16.6667 %)
Variance from outliers : 31.1510 % Variance is moderately inflated by outliers)
(cc/quick-bench (persistent! (reduce massoc! (transient {}) lst)))
(comment
Execution time mean : 515.178533 ms
Execution time std-deviation : 69.452690 ms
Execution time lower quantile : 482.675140 ms ( 2.5%)
Execution time upper quantile : 635.692003 ms (97.5%)
Overhead used : 2.943666 ns
Found 2 outliers in 6 samples (33.3333 %)
low-severe 1 (16.6667 %)
low-mild 1 (16.6667 %)
Variance from outliers : 31.7413 % Variance is moderately inflated by outliers)
Why doesn't clojure core use transients by default ?see here (relevant part of into
source from core.clj
):
(defn into
([to from]
(if (instance? clojure.lang.IEditableCollection to)
(with-meta (persistent! (reduce conj! (transient to) from)) (meta to))
(reduce conj to from))))
I have a long list of validation checks, and steps that happen interleaved among them. The validation checks can throw errors: right now I have:
(let [_ (check)
a (f)
_ (check2)
b (g)
...]
z)
basically, those validations/throwing errors are a way to short circuit the logic
the main alternative I see is a series of if
each with a nested let
is there a cleaner way of achieving this sort of Either
monad pattern?@josh.freckleton some->
?
yeah, I usually use some-> or some->> for that kind of thing
but reduce, then returning a reduced as appropriate can work too
(reduce (fn [acc v] (if (borked? acc) (reduced acc) (frob acc v)) init coll)
The nice thing about reduce
/`reduced` is you can return something other than nil
if any of the steps fail.
indeed
re some->
, kind of, but I need to report details of where it failed
re reduced
, interesting, I hadn't considered that, I kind of like that
your coll is likely to be a vector of checks to run in that case
I'm not really moving across a collection though...
It’s that weird fp stuff, instead of reducing a function over a collection of args, you reduce an arg over a collection of fns. 😉
so (reduce (fn [context check] (let [checked (check context)] (if (OK? checked) checked (reduced checked)) {} [check1 check2])
sort of like this
and then the hash map can hold a, b, etc. under keys
and then the collection I'm accumulating would be my otherwise let
bound vars. ya, the context
though at this point loop/recur might be easier to read?
your call
@noisesmith that's a genius pattern, I'll see if it works for this, thanks
:thumbsup:
it’s something I have used to success in my own code
you might also want to look at the new halt-when transducer in 1.9 (although it’s a little tricky to use well)
Has anyone used a clojure wrapper around git's cli? There are some that wrap JGit, but as far as I can tell JGit does not yet support 'clone --mirror ...' and I need that functionality.
@the2bears I’d generally look at doing interop before opting for a wrapper anyway - if the API is a total pain I then check for a wrapper, but often just using interop is fine
agreed, in that I don't need any more functionality other than 'clone --mirror' and then 'push --mirror'
that’s straightforward to use, if it’s good enough
yeah, I'll try that out. Thanks! And by the way, you're absolutely one of the busiest sources of help here... really appreciated.
oh, thanks
By 'busiest' I mean as soon as a see someone with a question, I see 'noisesmith typing...' 🙂
I would be living on the street without @noisesmith