This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-04-19
Channels
- # beginners (69)
- # boot (10)
- # cider (18)
- # cljsrn (6)
- # clojure (225)
- # clojure-austin (1)
- # clojure-dusseldorf (7)
- # clojure-france (2)
- # clojure-greece (3)
- # clojure-italy (9)
- # clojure-madison (6)
- # clojure-miami (1)
- # clojure-mke (2)
- # clojure-russia (58)
- # clojure-sg (36)
- # clojure-spec (25)
- # clojure-uk (54)
- # clojurescript (20)
- # core-async (15)
- # cursive (16)
- # datomic (71)
- # emacs (7)
- # hoplon (3)
- # jobs (5)
- # jobs-discuss (4)
- # klipse (3)
- # leiningen (2)
- # lumo (60)
- # mount (1)
- # off-topic (16)
- # om (6)
- # om-next (7)
- # onyx (21)
- # pedestal (15)
- # planck (1)
- # re-frame (3)
- # ring-swagger (4)
- # specter (33)
- # uncomplicate (31)
- # untangled (8)
- # vim (35)
@brycecovert, @noisesmith, @madstap thanks for the pointers
@hught I think it is, but different people on each, so sometimes worth it to ask on both
Another option since I could not let go of group-by
:
(let [g (group-by first [[:a 1] [:b 2] [:a 3]])]
(reduce #(update %1 %2 (partial map second)) g (keys g)))
=> {:a (1 3), :b (2)}
it does return lists, but I did get to use group-by : )Does anyone have an example of a project that uses integrant? Got excited for it after watching https://skillsmatter.com/skillscasts/9820-enter-integrant-a-micro-framework-for-data-driven-architecture-with-james-reeves but it would be cool to see it used in an actual project.
ah, on the above, changed to mapv
and it now returns lists:
(let [g (group-by first [[:a 1] [:b 2] [:a 3]])]
(reduce #(update %1 %2 (partial mapv second)) g (keys g)))
=> {:a [1 3], :b [2]}
still learning the ropesHow can I add multiple params to a map function?
Ex: Where to add x
in the following case, where x
is not a list?
I have a feeling there’s another high-order function I could use, but I’m missing it 😅
(map (fn [i x] (do-something i x)) items)
@vargaradu you have to reverse the argument order, but partial works:
(map (partial do-something x) items)
but that would require the signature of do-something to be (do-something x i)
which is why clojure often takes the viewpoint of "the argument most likely to be partially applied comes first"
@tbaldridge Awesome! I tried partial but couldn’t make it work, makes sense with the param order. Thanks!
If x
is coming from the lexical environment, the fn
will automatically make a closure
(let [x ...] (map (fn [i] (do-something i x)) items)
Using the reloaded pattern (tools.namespace) with component my UI seems to stick to old version of an altered function although the whole system has been restarted -
Yes, there are known scenarios where that can happen. I wrote about my findings here: http://danielsz.github.io/2016/05/06/Ghosts-in-the-machine
are there known error scenarios where this happens. I checked that the function is actually recompiled via a macro form (notify-compilation) placed into the function: (defmacro notify-compilation [] (println "compiled!"))
@gv what UI toolkit are you using?
@tbaldridge JavaFX - but every UI component is reconstructed after reloading
If I check the function identity via (println (format "fn hash = %X" (hash the-function)))
within the caller function and on the repl I get different results after reloading
oh well, seems that it is not a good idee to (:require [myapp.system :as sys]) and use (sys/create-system ...) - via an indirection ((resolve 'myapp.system/create-system) ...) it works now as expected ...
ok going really basic here, given a function f
which returns a number, what would be an idiomatic way to return the return value from f if larger than zero, else return zero, I can do this with let:
(defn g []
(let [val (f)]
(if (pos? val) val 0)))
but it feels clunky and non-functional, is there perhaps some function I can chain with f
?mbjarland: the problem with the code is not your conditional, but more that f
is a global side effect
Yeah, that was a somewhat contrived example, I was looking for the 'max' functionality in the reply on the clojure channel
I'd think pos?
would be more performant than max
. I'd recommend (defn g [v] (if (pos? v) v 0))
and then just (g (f))
@john the nice thing about programming is that we don't need to speculate, we can measure things
$ rlwrap ~/bin/bench
Clojure 1.9.0-alpha13
+user=> (require '[criterium.core :as crit])
nil
+user=> (defn g [v] (if (pos? v) v 0))
#'user/g
+user=> (crit/bench (do (g -1) (g 0) (g 1))
)
Evaluation count : 6856866060 in 60 samples of 114281101 calls.
Execution time mean : 7.528191 ns
Execution time std-deviation : 1.000423 ns
Execution time lower quantile : 7.177274 ns ( 2.5%)
Execution time upper quantile : 10.556685 ns (97.5%)
Overhead used : 1.543663 ns
Found 3 outliers in 60 samples (5.0000 %)
low-severe 1 (1.6667 %)
low-mild 2 (3.3333 %)
Variance from outliers : 80.6832 % Variance is severely inflated by outliers
nil
+user=> (crit/bench (do (max 0 -1) (max 0 0) (max 0 1)))
Evaluation count : 11319087960 in 60 samples of 188651466 calls.
Execution time mean : 3.651514 ns
Execution time std-deviation : 0.100580 ns
Execution time lower quantile : 3.564811 ns ( 2.5%)
Execution time upper quantile : 3.848574 ns (97.5%)
Overhead used : 1.543663 ns
Found 4 outliers in 60 samples (6.6667 %)
low-severe 3 (5.0000 %)
low-mild 1 (1.6667 %)
Variance from outliers : 14.2277 % Variance is moderately inflated by outliers
nil
@noisesmith interesting! Is there a blog post somewhere on how to set up that ~/bin/bench
business?
oh, bin/bench is just clojure.jar plus a few deps (criterium, core.async) in an uberjar - it allows me to test things directly with as little cruft and uncertainty as possible
so I literally just created a leiningen project with no code in it, created deps for criterium and a few other libs, and ran lein uberjar and renamed the resulting jar bench
. On linux there's a kernel extension that automatically calls your default java if your jar is executable.
(defn max
"Returns the greatest of the nums."
{:added "1.0"
:inline-arities >1?
:inline (nary-inline 'max)}
([x] x)
([x y] (. clojure.lang.Numbers (max x y)))
([x y & more]
(reduce1 max (max x y) more)))
it’s a way to call the static method max
of clojure.lang.Numbers
with the args x and y
@dpsutton it's calling a different arity
ah. i was thinking this is strict calling so the (max x y)
in the call to (max x y)
would blow up
oh right, that's a funny looking interop
it's not calling itself in that form
i'm not super familiar with the .
stuff so i'm assuming that's what's changing it. but on its face it looks bad
@john my theory of the performance difference is that max can be calculated without branching, and if can't
the .
special form is another way of calling (clojure.lang.Numbers/max x y)
, so it’s a different max
. That interop form is common in core, and always causes me to do a double take as well, since it’s generally the same name as the function and looks like an infinite recursion situation 🙂
exactly my confusion. thanks @tanzoniteblack
@noisesmith yeah, I tried your same bench with pos?
and an if
, without the function indirection... still slower than max
static Ops ops(Object x){
Class xc = x.getClass();
if(xc == Long.class)
return LONG_OPS;
else if(xc == Double.class)
return DOUBLE_OPS;
else if(xc == Integer.class)
return LONG_OPS;
else if(xc == Float.class)
return DOUBLE_OPS;
else if(xc == BigInt.class)
return BIGINT_OPS;
else if(xc == BigInteger.class)
return BIGINT_OPS;
else if(xc == Ratio.class)
return RATIO_OPS;
else if(xc == BigDecimal.class)
return BIGDECIMAL_OPS;
else
return LONG_OPS;
}
i suspect max will be slower when there's not a long literal as one of the arguments
@dpsutton trying that now, keeping the 0 and using vars for the other args
also trying the arbitrary high number suggested in the same test
well, these are longs, but yes
so, max took 10 times as long when one arg was a var and the large/small values were made more extreme, retesting g with that input
+user=> (def lo -29128192921)
#'user/lo
+user=> (def hi 29128192921)
#'user/hi
+user=> (def z 0)
#'user/z
+user=> (crit/bench (do (max 0 lo) (max 0 z) (max 0 hi)))
Evaluation count : 1485761220 in 60 samples of 24762687 calls.
Execution time mean : 39.103974 ns
Execution time std-deviation : 0.708623 ns
Execution time lower quantile : 38.284591 ns ( 2.5%)
Execution time upper quantile : 40.400809 ns (97.5%)
Overhead used : 1.543663 ns
Found 1 outliers in 60 samples (1.6667 %)
low-severe 1 (1.6667 %)
Variance from outliers : 7.7826 % Variance is slightly inflated by outliers
nil
+user=> (crit/bench (do (g lo) (g z) (g hi)))
Evaluation count : 3605141940 in 60 samples of 60085699 calls.
Execution time mean : 15.243799 ns
Execution time std-deviation : 0.199153 ns
Execution time lower quantile : 14.989091 ns ( 2.5%)
Execution time upper quantile : 15.755367 ns (97.5%)
Overhead used : 1.543663 ns
Found 3 outliers in 60 samples (5.0000 %)
low-severe 3 (5.0000 %)
Variance from outliers : 1.6389 % Variance is slightly inflated by outliers
nil
so g does nicely when we have vars and larger values (I suspect vars were the bigger equalizer here)
basic pattern: get a request, validate request, start a task running in the background, return a 200 response. assuming this is a kind of one off case (don't need to whip out core.async and build a whole nonblocking webserver) would using a future be an acceptable way to handle this? I'm not concerned about the return value, so I'm never going to dereference the future
Yes, that's fine, just be mindful that any errors in futures get implicitly swallowed up unless you deref them.
so you mean a future for every request?
this guarantees you achieve terrible performance for an extended period of time as GC consumes 99.9% of your CPU usage and after a long excruciating period eventually crash once you reach N clients
(if you know N is larger than the number you will ever see, this could be OK)
right, claypoole makes using futures via threadpools easy
@noisesmith @jstew you guys are amazing. saved me some serious trouble down the line.
I learned it the hard way, lol
(if you couldn't tell by my poetic description of the failure behavior)
@tjtolton for a small system, this approach would work fine, but if you really are kicking of N tasks at a time where N > NUM_CPUs, then I highly recommend a durable queue. Just enqueue data, once the queue acks the message return a 200. Then any job processor can grab that message and kick off a task.
Most message queues also have "max messages in flight" which works well for rate limiting the number of threads you have allocated at one time.
The worst-case situation for a "real" production server is when a user can call a HTTP endpoint and quickly and directly spawn a background thread. That's a great way to get servers that crash.
looks like you need to loop and put the bytes in until there's no more bytes to read from the file
using the .read method on java.io.InputStream
also, io/copy is happy to write into a byte-array, and you can make a ByteBuffer out of one of those
or even use .put
on a ByteBuffer and a byte-array to put its contents into one if you need to do it that way
@qqq oh! you might like this https://docs.oracle.com/javase/7/docs/api/java/nio/MappedByteBuffer.html
if you don’t necessarily need a ByteBuffer, you can use clj-mmap:
(defn file-to-byte-array ^bytes [file-name]
(with-open [mapped-file (mmap/get-mmap file-name)]
(mmap/get-bytes mapped-file 0 (.size mapped-file))))
for a ByteBuffer you can do something like:
(defn file-to-byte-array-bb ^ByteBuffer [^String file-name]
(with-open [channel (.getChannel (FileInputStream. file-name))]
(let [len (.size channel)
mbb (.map channel FileChannel$MapMode/READ_ONLY 0 len)]
mbb)))
where, now that I look at it, len could be inlined etc, cutting this from some experimentation I did
@mbjarland that creates a mappedbytebuffer as I linked above, it looks slightly a lot more convenient to use though https://github.com/thebusby/clj-mmap/blob/master/src/clj_mmap.clj#L64
does anyone know of a good way to write the result of a transducer to a db without having to realize the whole sequence at once?
(doseq [data (sequence xf input)] (write-result! data))
?
eh, I wouldn't mix seqs and transducers, that can do goofy stuff
goofy how? if it’s relevant, i think i’m planning on using redis to help with denormalization
I'd recommend just using transduce:
(transduce
xf
(fn [_ val]
...do something...
nil)
coll))
but transduce is eager
If you use lots of calls to (map)cat, calling sequence on the transducer can realize more data than you think.
oh wait, is the issue "keep fewer results at a time in memory" rather than "I want to stop inserting at an arbitrary point" ? if so yeah use transduce
I could have misunderstood
i have a big transducer stack, and i want to denormalize the results and store them in a sql db
i’m worried about (i) memory/performance (ii) locking issues (iii) pks/fks getting mixed up
OK then I misunderstood the question and I agree that transduce is correct here
and if you're doing de-normalization, what you're asking for is an upsert. And I highly recommend doing that in SQL, not in Clojure. Can be done with a query, or even better via a stored-proc.
You can hand the stored-proc your data, and that proc can decide what needs to be updated/inserted.
But yeah, whatever you do, you'll want to do it in some form of SQL as you'll need to have the rows locked while you look up FKs and do the inserts
I have a predicate fn, (yes? ...)
that returns nil
if the predicate doesn't even apply. Is this an anti-pattern since, being a predicate, suggests it returns a boolean?
sensible alternatives?
I seem to remember conversation here a few days ago about clojure.core having predicates which return something falsey, because it was faster. That being said, it’s probably an anti-pattern to have a predicate that is actually trinary, true, false, and nil
@josh.freckleton afaict, returning anything but true or false from a predicate is the literal end of the world
@alexmiller shoot, I already eval'd it, I hope it's not too late? 😉
Have you ever seen an elegant encoding of Maybe Bool
, which is essentially what I have?
Dang it, now we are goners
Encoding from what perspective?
I'd say you can just return whatever you want, but that's just me :)
> treat nil and false differently yes
just curious if there was something more clojuric than ad hoc doing it
i've always thought truthy/falsey is as good as true/false in clojure. unless we're talking about alternative t/f.
("clojurian"?)
@josh.freckleton seriously, lemme know if you figger it out. of course you can always throw an exception. "Maybe x" seems a little too strong-typish for clojure, to me at least. you'd have to make all your consumers deal with it.
::not-found?
would unfortunately be truthy, so not quite right for a predicate, but then again nil
isn't the best fit either
(speaking of "end of the world"...)
if you have a non-truthy semantics, you’ll have to check for the special case first, don’t see a way around that
Without a lot of context, I would recommend having two predicates - applicable? and yes? in this case
> if you have non-truthy... special case ...
(filter false? [nil nil false])
not too special-casey
if you really want it to be a binary predicate still id probably agree with @brycecovert to split it into two, otherwise it really seems like you have an actual trinary use case
nuttin wrong with n-valued logic. if you want to avoid confusion - assuming the reader will take yes? to be a boolean predicate - you could elaborate the symbol, e.g. yes?+, yes???, or whatever.
fwiw i've used "maybe-x" for this sort of thing and found it works quite well - when i return to it months later i know what it means.
@mobileink > n-valued logic
is that a thing? I've wondered this frequently in the past, like what if there were more than T/F, but my math knowledge is patchy.
Are there binary fns, like for ternary T/F/X
, T AND X
, X XOR F
, or something?
(shoulda googled first https://en.wikipedia.org/wiki/Three-valued_logic)
the fun part is the soviet computers that considered three value logic necessary for ideological reasons going back to hegel
@josh.freckleton yeah, classical logic is binary but modern logic goes way beyond that. not just 3-valued, n-valued. its just systems more or less.
consider that "true" and "false" are just meaningless symbols, they do not mean true and false in the ordinary sense. from there it's a short step to "the logical values,in this system are 0, 1, 2, ... " - all of which are just symbols.
> hree value logic necessary for ideological reasons going back to hegel where does this sentence come from?
from the article about ternary computer i see > it had notable advantages over the binary computers which eventually replaced it, such as lower electricity consumption and lower production cost
i'm guessing thesis, antithesis, synthesis. it's the logic of scientific history, man!
Does anyone have any suggestions for a faster clojure.data/diff? I don’t know if anyone’s e.g. tried specter for this (or if it’d even help). I imagine zippers might, at least. I currently have two mostly-similar nested trees (flattened about 2E5 entries) and clojure.data/diff just takes forever
: ) figured I’d bring down the level a notch or two, any terse and/or idiomatic way of checking if a character is part of a string in clojure, not via regex or by converting the character to string, but just as a character:
(contains? "bob" \o)
IllegalArgumentException contains? not supported on type: java.lang.String clojure.lang.RT.contains (RT.java:814)
(string/includes? "bob" \o)
ClassCastException java.lang.Character cannot be cast to java.lang.CharSequence clojure.string/includes? (string.clj:376)
(contains? (seq "bob") \o)
IllegalArgumentException contains? not supported on type: clojure.lang.StringSeq clojure.lang.RT.contains (RT.java:814)
@mbjarland => (.indexOf "hello" "e")
@mbjarland I’m sure there’s a better way to say it but String is a CharSequence so I’d use includes?
and use str to turn the char into a string
@mbjarland interop is idiomatic
clojure.string/index-of also exists
=> (source clojure.string/index-of)
(defn index-of
"Return index of value (string or char) in s, optionally searching
forward from from-index or nil if not found."
{:added "1.8"}
([^CharSequence s value]
(let [result ^long
(if (instance? Character value)
(.indexOf (.toString s) ^int (.charValue ^Character value))
(.indexOf (.toString s) ^String value))]
(if (= result -1)
nil
result)))
([^CharSequence s value ^long from-index]
(let [result ^long
(if (instance? Character value)
(.indexOf (.toString s) ^int (.charValue ^Character value) (unchecked-int from-index))
(.indexOf (.toString s) ^String value (unchecked-int from-index)))]
(if (= result -1)
nil
result))))
nil
=> (.indexOf "hela" (int \a)) => 3 (this is what @hiredman alluded to above)
@bronsa correct me, but I think that use of ^long
there isn't even a valid type hint
(not that there is any reflective code there that would be avoided if it was a valid type hint)
user=> (def x 1)
#'user/x
user=> (set! *warn-on-reflection* true)
true
user=> (let [w ^long x] (Long/valueOf w))
Reflection warning, NO_SOURCE_PATH:13:18 - call to static method valueOf on java.lang.Long can't be resolved (argument types: unknown).
1
user=> (let [w (long x)] (Long/valueOf w))
1
user=>
I think what I am remembering is you had to do some special handing for those type hints on that core.async locals clearing patch
yeah, "casting" of objects is fine (when there's no real casting done in bytecode, just on the compiler bit for method matching) -- when casting OTOH is on primitives then that's not ok
I think type hints work as casts for primitive type hints only if they're on method args (e.g. (.foo Bar ^long (something))
) but I gave up trying to make sense/remember what works or doesn't
Transit doesn't serialise a datomic function "Not supported: class clojure.lang.Delay"
This is way late, but I was just catching up on slack. I don't know datomic, but check out https://clojars.org/yieldbot/serializable-fn for serializing functions. It has some limitations, but it works.
transit isn't meant to serialize functions
Any idea how I can transact a db fn into the schema using the datomic client api? This is kind of a downer.
my guess is the api takes forms that evaluate as functions, and you shouldn't have a delay in t here
the #db/fn tag can introduce pretty much anything after reading, so likely what it produces introduces a delay object