This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2018-10-09
Channels
- # 100-days-of-code (6)
- # announcements (4)
- # atlanta-clojurians (1)
- # aws (1)
- # beginners (65)
- # boot (21)
- # cider (9)
- # cljsrn (3)
- # clojure (186)
- # clojure-android (4)
- # clojure-conj (1)
- # clojure-dev (12)
- # clojure-germany (3)
- # clojure-italy (8)
- # clojure-nl (1)
- # clojure-russia (32)
- # clojure-spec (19)
- # clojure-uk (41)
- # clojurescript (83)
- # core-async (5)
- # cursive (18)
- # datomic (15)
- # emacs (5)
- # events (4)
- # fulcro (7)
- # hyperfiddle (4)
- # leiningen (4)
- # liberator (1)
- # off-topic (65)
- # overtone (4)
- # pedestal (5)
- # perun (1)
- # planck (3)
- # re-frame (1)
- # reagent (3)
- # rum (5)
- # shadow-cljs (8)
- # spacemacs (19)
- # testing (3)
- # tools-deps (4)
- # yada (6)
Hello. I'm trying to use JVM11 + Clojure 1.10.0-beta1 but i'm getting
Boxed math warning, clojure/core/rrb_vector/rrbt.clj:1859:30 - call: public static boolean clojure.lang.Numbers.isPos(java.lang.Object).
Syntax error compiling deftype* at (midje/util/ordered_set.clj:16:1).
Cause: Must hint overloaded method: toArray
I'm already using [org.clojure/core.rrb-vector "0.0.12"]
.also warns Boxed math warning, clojure/core/rrb_vector/rrbt.clj:1859:30 - call: public static boolean clojure.lang.Numbers.isPos(java.lang.Object).
on JVM8, but works.
can you share a repro?
sounds like https://dev.clojure.org/jira/browse/CLJ-2374 in another lib
yeah it’s this toArray in midje https://github.com/marick/Midje/blob/master/src/midje/util/ordered_set.clj#L75
as of JDK 11, this is now ambiguous
I was thinking about having an atom with a flag. After every read id check the flag and stop reading
Is there anything against scheduling a function that only triggers something (and doesn’t run very long itself) using a go loop and a timeout channel compared to a scheduled thread pool executor
Spec question: do I really have to write generators for specs that simply must conform to common predicates like integer? or string? - it's not hard, but just seems like unnecessary boilerplate to do this again and again
let's suppose you have this generator
(defn string-generator []
(gen/fmap str/join
(gen/vector gen/char-alphanumeric 2 10)))
and you define this spec for a string
(spec/def ::string
(spec/with-gen string? string-generator))
if you define a spec using that string spec, it will also use that generator
(spec/def ::name ::string)
You can use built-in generators for most common predicates.
(require '[clojure.spec.alpha :as s])
(require '[clojure.spec.gen.alpha :as gen])
(s/def ::my-thing string?)
(gen/generate (s/gen ::my-thing))
=> "aegaeG33fa"
;; this seems to work also
(gen/generate (s/gen string?))
(gen/generate (s/gen my-thing))
ClassCastException clojure.test.check.rose_tree.RoseTree cannot be cast to clojure.lang.IFn clojure.test.check.rose-tree/fmap (rose_tree.cljc:77)
@joelsanchez right, but then I still have to write boilerplate generators for every single base type, right?
you don't need to write generators for the base types if you don't want to, but the built-in ones are pretty terrible, and you only have to do it once anyway
I’ve been quite happy with the built-in generators. However sometimes I get the “tried 100 times but couldn’t generate satisfying blablab” error.
@valtteri still get the same exception using the other require (swapped [clojure.test.check.generators :as gen] for [clojure.spec.gen.alpha :as gen])
looks like these can https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/FutureTask.html
How accurate is a timeout channel? If I read from a timeout channel with a 15 minute timeout, should that work accurate on the millisecond?
@borkdude tldr: 10ms precision, uses System/currentTimeMillis
. look here for the details: https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async/impl/timers.clj
10ms is good enough for me. I was seeing some weird behavior. 20 seconds worked fine, 30 seconds too… but 15 minutes, I saw something happening after 4 minutes… trying from a clean REPL 😉
the question boils down to: is core.async timeout suitable to build a lightweight scheduler
and is there anything against spawning ~400 go loops that trigger a very short running function every 15 minutes
@borkdude not sure, but here is another approach to consider: https://github.com/duct-framework/scheduler.simple/blob/master/src/duct/scheduler/simple.clj#L7-L16
yeah, the thing is, I need to be able to reset the delay/interval dynamically, so building my own thing on top of core.async is a bit easier
Anyone have any experience wrapping a Java library in Clojure? Really interested in making an RTree library in Clojure.
here are some that I've used that come to mind: clj-time, clojure.java.jdbc, faraday. clj-time probably being the smallest example of those. I know there are many more. Java interop is pretty easy in Clojure, so I think most of the effort is in designing an "idiomatic" Clojure wrapper
I was actually just looking at clj-time, thinking about following that model.
The RTree library I’m using was designed to be functional. Adding a point to the RTree makes a new RTree. So I figured it’s a great candidate for this.
I will take a look at those other libraries thanks for the tip.
in some cases Java interop is trivial enough that you don't need much of a wrapper :man-shrugging:
I'd probably approach it as 1) use the Java lib from Clojure 2) alleviate the pain points as you go
That is true, the interop is not terrible. Though debugging is a pain and the RTree library returns Observables which makes putting the data into a collection a real pain.
I haven’t had any luck finding any good implementations of RTree in Clojure. Mainly have been writing a lot of interop to handle geospatial data. I had to deal with iterators yesterday and it felt so un-functional. Will spend some time researching on my own but any guidance would be awesome.
What is the elegant way to conj a value to a vector inside of a map, when the vectory may not be there yet? Or, in other words, how to implement (defn conj-to-vector-in [m path val])
?
Is there a way to know if a function has changed? I'm happy for the definition to be heavier rather than lighter. That would involve checking whether any functions it uses have changed too.
It is an undecidable problem in general, given Clojure's dynamic nature, e.g. you could calculate Vars to call as functions at run time if you really wanted to. The common case where people don't do that is still made a bit trickier by the fact that functions are often passed as args and returned as return values.
the tools.analyzer
library can be used to analyze Clojure code and produce lots of data about its contents, that might make it fairly straightforward to determine a 'call tree' for the cases where it is obvious in the source code that you mean Var X in namespace Y.
I'm thinking of caching in a build system. Self tracking is really valuable I think.
https://github.com/Datomic/codeq generates a sha of each top level function (it tries to support java too so it calls it a code segmenet)
That might be interesting to look at. I think it was mentioned that a rewrite is planned that uses tools analyzer. Or rather, it was going to be looked at.
Here is a simple problem that I'm not sure how to solve in Clojure idiomatically. I have a large text file describing conference talks. The structure looks similar to this:
Speaker: Andrew
Title: On penguins
Some lines describing the talk,
a. k. a. "abstract".
Speaker: Julia
Title: Five flavours of cleanliness
This talks covers ...
Joint work with Andrew.
Given this file, I would like to build a map with the keys [:speaker :title :abstract]
.What I ended up doing is short, but a bit ugly:
(defn add-speaker
[sps sp abs]
(cond-> sps
(seq sp) (conj (assoc sp :abstract (trim (join "\n" abs))))))
(defn speaker-builder
[[sps sp abs] l]
(cond
(starts-with? l "= Speaker:") [(add-speaker sps sp abs)
(parse-name l)
[]]
(starts-with? l "= Title:") [sps
(into sp (parse-title l))
abs]
:else [sps sp (conj abs l)]))
(defn get-speakers
[ls]
(let [[sps sp abs] (reduce speaker-builder [[] {} []] ls)]
(add-speaker sps sp abs)))
Is there something easier than that? (Please, don't offer writing a regular expression.)@U3DAE8HMG I will, thank you
Would the desired result look something like the following?
[{:speaker "Andrew"
:title "On Penguins"
:abstract "rest of lines joined by \n"}
...]
@UCQL6E7PY Yes, exactly
Here's my rough version using partition-by to group the lines into their own sequences for one map
(def lines
(list "Speaker: Andrew"
"Title: On Penguins"
"abstract line 1"
"abstract line 2"
"Speaker: Julia"
"Title: Five Flavours of Cleanliness"
"abstract line 3"
"abstract line 4"
""
"abstract line 5"))
(defn lines->map
[]
(->> lines
(partition-by #(clojure.string/starts-with? % "Speaker:"))
(partition 2)
(reduce (fn [m [[speaker] [title & abstract]]]
(conj m {:speaker speaker
:title title
:abstract (clojure.string/join "\n" abstract)}))
[])))
=> (lines-map)
[{:speaker "Speaker: Andrew"
:title "Title: On Penguins"
:abstract "abstract line 1\nabstract line 2"}
{:speaker "Speaker: Julia"
:title "Five Flavours of Cleanliness"
:abstract "..."}]
;; use map instead of reduce
(defn lines->map
[]
(->> lines
(partition-by #(clojure.string/starts-with? % "Speaker:"))
(partition 2)
(map (fn [[[speaker] [title & abstract]]]
{:speaker speaker
:title title
:abstract (clojure.string/join "\n" abstract)}))))
@UCQL6E7PY Completely overlooked this approach. Thank you!
why is clojure.core.rrb-vector
enabling :warn-on-boxed
here[1]? It seems to be creating a wall of warnings in my REPL when I try to load something which depends on this lib. So far I was unable to somehow disable it. What am I missing here?
a warning example: Boxed math warning, clojure/core/rrb_vector/nodes.clj:88:11 - call: public static boolean
[1] https://github.com/clojure/core.rrb-vector/blob/master/src/main/clojure/clojure/core/rrb_vector.clj#L46
worst case you could do (set! *unchecked-math* nil)
immediately after loading rrb-vector right?
I guess that could mean moving a require out of the ns form, which is annoying
it is a transitive dependency AFAIK, don't have a good control there and all the warnings seem to be originated in the lib itself
I have java11, tried with clojure 1.9 and beta 1.10, maybe I should try with java8. That might be relevant.
if you explicitly require the ns, then call set!, outside the ns form, the later require becomes a no-op
this is ugly but it does undo the behavior for everything outside that one lib
or you're saying it's already only warning for things inside that lib and not other code?
the real fix might be telling the rrb-vector devs to only turn that on during dev? - ignore what I was saying about setting unchecked-math yourself, that clearly wouldn't help here
I see, I wonder if I'm the only one hitting this issue, btw. it does not appear when running tests or entering via -main
core.rrb-vector causes many such warnings. It was written before the Clojure compiler implemented those warnings, and core.rrb-vector has not been updated much since then. It is a performance warning only, and not nearly as bad of a performance issue as reflection. It would be nice if core.rrb-vector didn't spew all of those warnings for non-developers of the library, I agree.
@andy.fingerhut thanks for the explanation, I'm going to use a patched version locally
I can't even run lein test
on the latest core.rrb-vector
:
Exception in thread "main" java.io.FileNotFoundException: Could not locate clojure/test/check/generators__init.class or clojure/test/check/generators.clj on classpath:
@seancorfield same error here, under java11
That's with Java 8 for me.
Looks like mvn test
works tho'...
(and doesn't spew those warnings)
I never see such errors in my project that uses core.rrb-vector “0.0.11”
those *unchecked-math*
things are in multiple files, like setting them should work only locally for given file:
https://github.com/clojure/core.rrb-vector/blob/master/src/main/clojure/clojure/core/rrb_vector/rrbt.clj#L23
https://github.com/clojure/core.rrb-vector/blob/master/src/main/clojure/clojure/core/rrb_vector/nodes.clj#L7
commenting out all those (set! ...)
commands and using the patched library locally fixed my issue, the warnings are gone
I’d want to ping Michał but seems like we could release a new version w/o those
I've started to dig into core.rrb-vector a bit to learn the data structure and code, to see if I can fix some bugs and do some comparative performance testing of it vs. Clojure's built in vectors, but it could be weeks before I get there part time.
There are about 10 public forks of the library on Github, where others look like they have started something similar, but not gotten to something they were willing to release -- hopefully I don't end up there.
well I won’t end up there :)
note that commenting out those set!’s actually changes the result - the key is to set to true vs :warn-on-boxed
the only canonical build file in contrib libs is the pom.xml so you should always run mvn to build / test
which is clean for me
(but I have experienced the behavior above)
hello everyone! rest-api question, do you recommend to use more liberator
and go with a “full loaded” solution or go with compojure-api
with a more DIY solution then?
I don't have a recommendation for you. I'll just say that using liberator
doesn't feel "full loaded" to me. The library helps me structure my app, but it doesn't like I'm getting sucked into a framework.
yeah I meant “full loaded” for structuring rest api
@UC78Y0DR9 there is also #reitit, like compojure-api, but just data and no magic.
I am hoping to reach the more time-consuming goal of changing the code so that even with the warnings enabled, there are none (or only in places where they won't matter for performance) -- again, no promises, but that is why it won't be quick.
well, those are real warnings. all that code is using java.util.concurrent.atomic.AtomicInteger and so falls into Number cases for all the math
changing that would mean a significant rewrite
my maven is failing with that generators file not found error when trying to test/build/install
you run X and see what Y?
just saying “failing” doesn’t help me at all
ah. sorry, cloned the core.rrb-vector
and tried to run mvn there like seancorfield above
and Y is?
FYI, in core.rrb-vector, the non-authoritative Leiningen project.clj file uses a more recent version of collection-check, which causes one of the namespaces to give an error since a namespace in collection-check seems to have changed. That should only be an issue if you use Leiningen on it, though.
Yeah, lein install
IIRC will overwrite the pom.xml file, and given the inconsistent state of that project's pom.xml
and project.clj
files, would likely cause such problems.
I backed down the warnings and am releasing 0.0.13 now
Alex I think I am missing something about what you said earlier "note that commenting out those set!’s actually changes the result - the key is to set to true vs :warn-on-boxed". Do you mean that the compiled code is different with those set!'s commented out?
yes - it turns on unchecked math by default
user=> (doc *unchecked-math*)
-------------------------
clojure.core/*unchecked-math*
While bound to true, compilations of +, -, *, inc, dec and the
coercions will be done without overflow checks. While bound
to :warn-on-boxed, same behavior as true, and a warning is emitted
when compilation uses boxed math. Default: false.
Cool that it is in the doc string for when I forget again 🙂
release is done, takes about 10 minutes to show up in Maven central usually
I don't know how I was unaware of this before today -- cool for algorithm/data structure enthusiasts like me: https://github.com/lacuna/bifurcan/blob/master/doc/comparison.md
how can I avoid this reflection?
user=> (ArrayDeque. [{}])
Reflection warning, /private/var/folders/q5/s1n6ttwx1x12mzmbk2mmx3t80000gp/T/form-init5839443831084402899.clj:1:1 - call to java.util.ArrayDeque ctor can't be resolved.
#object[java.util.ArrayDeque "0x49d61ca8" "[{}]"]
@andy.fingerhut i think zach published that only two days ago
oh, well then I am marvelously quickly informed about it 🙂
haha you're up to date. if you follow zach on twitter he and alexander yakushev (spelling?) had a discussion about some of the benchmarks you might be interested in
@schmee type hinting literal collections is tricky, (let [^java.util.Collection c [{}]] (ArrayDeque. c))
may work better
hmm so native java is quite fast it seems
doh, I thought I tried that but I put the hint on the value instead of the variable name thanks @hiredman!
Would anyone have a functional approach (in clojure) of bin packing algo? like the first-fit decreasing here https://www.geeksforgeeks.org/bin-packing-problem-minimize-number-of-used-bins/ ?
Well, the decreasing part is easy via sort
on the pieces. If you write a function that takes an existing set of bins and their current occupancy, and one new item to put in them, and returns the new occupancies after the item is placed into one of the bins, you can iterate that function once for each item via reduce
I did it in an imperative language, I'm just starting the part where I loop each pieces and see if the occupancy of bins are enough, but I was overwhelming about the way to
clojure's for is not a loop
A reasonable way to represent an assignment of pieces to bins is a vector of one element per bin, and one bin could be represented as a Clojure map with a key :pieces
whose value is a vector of piece sizes, and a key :occupancy
that is the total of all piece sizes (the latter part is easily calculated from the set of pieces, but could speed up the algorithm by not having to recalculate it many times).
@noisesmith for / loop sorry
Me personally, a function for determining which bin to put the next new piece in, I would write using loop
, but I may be in the Clojure minority there.
Using loop
doesn't mean you have to mutate data, though -- it could just calculate and return the bin number into which the new piece is placed.
@romain np - not trying to be pedantic, I actually don't know what you mean by for / loop
. I can picture attempting to do this with for
(a list comprehension), or loop
(low level iteration), or some combination of the two, but I don't know what approach you are describing
I tried both approach, I may have though they are similar, that's why I typed them. But don't worry you didn't offense me in any way 😉
@andy.fingerhut thanks, I'll give a try
my typical approach is to replace loop
with reduce
in cases where an input sequence of some sort is consumed in strict order
but if it's helpful to consume out-of-order, then yeah use loop instead of reduce
As I mentioned, a separate function that just handles "given a current arrangement of previous pieces placed in bins, plus one new piece size, return the new arrangement with the new piece included" is a nice dividing point for thinking about it. Such function could be implemented with loop
. It is good for efficiency to use a technique that can stop as soon as it finds the place to put a new piece, without considering all bins if it doesn't need to.
No matter how that function is implemented internally, you can then write another that calls it once for each piece, using reduce.
reduce can also return early with reduced
(not sure if this applies here)
The second function I mentioned doesn't have to use reduce, of course -- it could also use loop or any of several other techniques. reduce is especially good for a situation where you have a starting state, and for each element of a sequence, you apply a function to the current state and the new element to get a next state, and then repeat.
(starting state in the example = "empty set of bins")
And state might be misleading term there -- I just mean "an immutable value representing a state"
@andy.fingerhut for the data representation, you'd have a vector containing a set of 2 elements (a vector of pieces and a number of occupancy) ?
For the occupancy I was thinking just the total size of all pieces occupying the bin so far. That is redundant information if you have the collection of all piece sizes in the bin, since you could easily recalculate it as the sum of all piece sizes, but it seems like a reasonable idea for efficiency not to have to recalculate that value for each new piece.
Ok, now I need to figure how I "break" my loop using loop
statement and it should be ok
you can break from loop
by having a path that doesn't call recur
Body of loop
can look like (if <condition to keep going> (recur ...) <return value of loop>)
user=> (loop [x 0] (if (> x 100) x (recur (+ x (rand-int 20)))))
106
or the pathological case
user=> (loop [] :OK)
:OK
Or the sociopathological case (loop [] :hit-neighbor)
there's a lot of things in clojure that act like identity, under the right conditions
user=> (+ (* (and (or (do (let [] (-> (loop [] (identity (doto 42))))))))))
42
(couldn't help editing to add a few things)@romain nice exercise, something like this? https://repl.it/repls/GrumpyAnxiousDistributedcomputing
@darwin it's a question of style, but the outer recur on that function is something I would always replace with reduce
Yeah I got those in the wrong place somehow, will fix