Fork me on GitHub
#clojure
<
2018-10-09
>
souenzzo01:10:04

Hello. I'm trying to use JVM11 + Clojure 1.10.0-beta1 but i'm getting

Boxed math warning, clojure/core/rrb_vector/rrbt.clj:1859:30 - call: public static boolean clojure.lang.Numbers.isPos(java.lang.Object).
Syntax error compiling deftype* at (midje/util/ordered_set.clj:16:1).
Cause: Must hint overloaded method: toArray
I'm already using [org.clojure/core.rrb-vector "0.0.12"].

souenzzo01:10:38

also warns Boxed math warning, clojure/core/rrb_vector/rrbt.clj:1859:30 - call: public static boolean clojure.lang.Numbers.isPos(java.lang.Object). on JVM8, but works.

alexmiller02:10:47

can you share a repro?

alexmiller02:10:56

as of JDK 11, this is now ambiguous

vheuken04:10:12

What would be the best way to close a stream from a different thread?

vheuken04:10:00

I was thinking about having an atom with a flag. After every read id check the flag and stop reading

vheuken04:10:05

is there a better way?

emccue05:10:23

atom should work fine, but perhaps (mumble mumble something volatile?)

emccue05:10:49

you could also use a core async channel as a signalling mechanism

vheuken05:10:01

i was thinking about using channels

vheuken05:10:10

or even just storing the futures and using future-cancel

emccue05:10:26

yeah lots of options. I aint an expert

vheuken05:10:59

Yeah. I got the atom solution working, so im going with that.

vheuken05:10:28

I feel like there's no way to do it that's not kinda complex

emccue05:10:01

yep, come back when its a performance bottleneck

emccue05:10:27

as they say

vheuken05:10:28

I can't imagine it ever will be.

vheuken05:10:41

there's much more shit going on than an if statement lol

borkdude09:10:00

Is there anything against scheduling a function that only triggers something (and doesn’t run very long itself) using a go loop and a timeout channel compared to a scheduled thread pool executor

simongray09:10:59

Spec question: do I really have to write generators for specs that simply must conform to common predicates like integer? or string? - it's not hard, but just seems like unnecessary boilerplate to do this again and again

simongray09:10:39

if a spec is simply string? then shouldn't the :gen be inferred?

joelsanchez09:10:38

let's suppose you have this generator

(defn string-generator []
  (gen/fmap str/join
            (gen/vector gen/char-alphanumeric 2 10)))
and you define this spec for a string
(spec/def ::string
  (spec/with-gen string? string-generator))
if you define a spec using that string spec, it will also use that generator
(spec/def ::name ::string)

valtteri09:10:01

You can use built-in generators for most common predicates.

(require '[clojure.spec.alpha :as s])
(require '[clojure.spec.gen.alpha :as gen])

(s/def ::my-thing string?)
(gen/generate (s/gen ::my-thing))
=> "aegaeG33fa"

;; this seems to work also
(gen/generate (s/gen string?))

simongray09:10:54

(gen/generate (s/gen my-thing))
ClassCastException clojure.test.check.rose_tree.RoseTree cannot be cast to clojure.lang.IFn  clojure.test.check.rose-tree/fmap (rose_tree.cljc:77)

simongray09:10:21

@valtteri so it doesn't seem to work when I try

simongray09:10:57

@joelsanchez right, but then I still have to write boilerplate generators for every single base type, right?

simongray09:10:50

ah wait, I see that you're using a different gen namespace

valtteri09:10:54

I edited my post to include also requires. Please check those

joelsanchez09:10:38

you don't need to write generators for the base types if you don't want to, but the built-in ones are pretty terrible, and you only have to do it once anyway

valtteri09:10:46

I’ve been quite happy with the built-in generators. However sometimes I get the “tried 100 times but couldn’t generate satisfying blablab” error.

simongray09:10:58

@valtteri still get the same exception using the other require (swapped [clojure.test.check.generators :as gen] for [clojure.spec.gen.alpha :as gen])

valtteri09:10:18

Did you restart REPL?

simongray09:10:11

I imported as gen2 and just used that instead of gen/..., but I can try

simongray09:10:18

@valtteri ok now it works ^-^ thanks

borkdude10:10:19

can I cancel a future from within a future?

borkdude14:10:45

How accurate is a timeout channel? If I read from a timeout channel with a 15 minute timeout, should that work accurate on the millisecond?

schmee14:10:54

@borkdude tldr: 10ms precision, uses System/currentTimeMillis. look here for the details: https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async/impl/timers.clj

borkdude14:10:40

10ms is good enough for me. I was seeing some weird behavior. 20 seconds worked fine, 30 seconds too… but 15 minutes, I saw something happening after 4 minutes… trying from a clean REPL 😉

borkdude14:10:56

the question boils down to: is core.async timeout suitable to build a lightweight scheduler

borkdude14:10:32

and is there anything against spawning ~400 go loops that trigger a very short running function every 15 minutes

borkdude14:10:11

15 minutes seem to work ok now, so it was probably some weird state I had

borkdude15:10:30

yeah, the thing is, I need to be able to reset the delay/interval dynamically, so building my own thing on top of core.async is a bit easier

christian.gonzalez15:10:17

Anyone have any experience wrapping a Java library in Clojure? Really interested in making an RTree library in Clojure.

taylor15:10:33

here are some that I've used that come to mind: clj-time, clojure.java.jdbc, faraday. clj-time probably being the smallest example of those. I know there are many more. Java interop is pretty easy in Clojure, so I think most of the effort is in designing an "idiomatic" Clojure wrapper

christian.gonzalez15:10:21

I was actually just looking at clj-time, thinking about following that model.

christian.gonzalez15:10:57

The RTree library I’m using was designed to be functional. Adding a point to the RTree makes a new RTree. So I figured it’s a great candidate for this.

christian.gonzalez15:10:37

I will take a look at those other libraries thanks for the tip.

taylor15:10:56

in some cases Java interop is trivial enough that you don't need much of a wrapper :man-shrugging:

taylor15:10:36

I'd probably approach it as 1) use the Java lib from Clojure 2) alleviate the pain points as you go

christian.gonzalez15:10:53

That is true, the interop is not terrible. Though debugging is a pain and the RTree library returns Observables which makes putting the data into a collection a real pain.

christian.gonzalez15:10:59

I haven’t had any luck finding any good implementations of RTree in Clojure. Mainly have been writing a lot of interop to handle geospatial data. I had to deal with iterators yesterday and it felt so un-functional. Will spend some time researching on my own but any guidance would be awesome.

witek16:10:43

What is the elegant way to conj a value to a vector inside of a map, when the vectory may not be there yet? Or, in other words, how to implement (defn conj-to-vector-in [m path val])?

dominicm17:10:57

Is there a way to know if a function has changed? I'm happy for the definition to be heavier rather than lighter. That would involve checking whether any functions it uses have changed too.

andy.fingerhut17:10:32

It is an undecidable problem in general, given Clojure's dynamic nature, e.g. you could calculate Vars to call as functions at run time if you really wanted to. The common case where people don't do that is still made a bit trickier by the fact that functions are often passed as args and returned as return values.

andy.fingerhut17:10:42

the tools.analyzer library can be used to analyze Clojure code and produce lots of data about its contents, that might make it fairly straightforward to determine a 'call tree' for the cases where it is obvious in the source code that you mean Var X in namespace Y.

dominicm18:10:38

I'm thinking of caching in a build system. Self tracking is really valuable I think.

dominicm18:10:22

Doing something might be better than nothing though. Not sure.

hiredman18:10:12

https://github.com/Datomic/codeq generates a sha of each top level function (it tries to support java too so it calls it a code segmenet)

dominicm18:10:08

That might be interesting to look at. I think it was mentioned that a rewrite is planned that uses tools analyzer. Or rather, it was going to be looked at.

avfonarev19:10:02

Here is a simple problem that I'm not sure how to solve in Clojure idiomatically. I have a large text file describing conference talks. The structure looks similar to this:

Speaker: Andrew
Title: On penguins
Some lines describing the talk,
a. k. a. "abstract".

Speaker: Julia
Title: Five flavours of cleanliness
This talks covers ...

Joint work with Andrew.
Given this file, I would like to build a map with the keys [:speaker :title :abstract].

avfonarev19:10:30

What I ended up doing is short, but a bit ugly:

(defn add-speaker
  [sps sp abs]
  (cond-> sps
    (seq sp) (conj (assoc sp :abstract (trim (join "\n" abs))))))

(defn speaker-builder
  [[sps sp abs] l]
  (cond
    (starts-with? l "= Speaker:") [(add-speaker sps sp abs)
                                   (parse-name l)
                                   []]
    (starts-with? l "= Title:") [sps
                                 (into sp (parse-title l))
                                 abs]
    :else [sps sp (conj abs l)]))

(defn get-speakers
  [ls]
  (let [[sps sp abs] (reduce speaker-builder [[] {} []] ls)]
    (add-speaker sps sp abs)))
Is there something easier than that? (Please, don't offer writing a regular expression.)

taylor20:10:48

take a look at Instaparse, might be pretty easy for this

avfonarev20:10:13

@ I will, thank you

trailcapital20:10:58

Would the desired result look something like the following?

[{:speaker "Andrew"
  :title "On Penguins"
  :abstract "rest of lines joined by \n"}
 ...]

avfonarev20:10:16

@ Yes, exactly

trailcapital20:10:46

Here's my rough version using partition-by to group the lines into their own sequences for one map

trailcapital20:10:50

(def lines
  (list "Speaker: Andrew"
        "Title: On Penguins"
        "abstract line 1"
        "abstract line 2"
        "Speaker: Julia"
        "Title: Five Flavours of Cleanliness"
        "abstract line 3"
        "abstract line 4"
        ""
        "abstract line 5"))

(defn lines->map
  []
  (->> lines
       (partition-by #(clojure.string/starts-with? % "Speaker:"))
       (partition 2)
       (reduce (fn [m [[speaker] [title & abstract]]]
                 (conj m {:speaker speaker
                          :title title
                          :abstract (clojure.string/join "\n" abstract)}))
               [])))

=> (lines-map)
[{:speaker "Speaker: Andrew"
  :title "Title: On Penguins"
  :abstract "abstract line 1\nabstract line 2"}
 {:speaker "Speaker: Julia"
  :title "Five Flavours of Cleanliness"
  :abstract "..."}]

trailcapital20:10:19

;; use map instead of reduce
(defn lines->map
  []
  (->> lines
       (partition-by #(clojure.string/starts-with? % "Speaker:"))
       (partition 2)
       (map (fn [[[speaker] [title & abstract]]]
              {:speaker speaker
               :title title
               :abstract (clojure.string/join "\n" abstract)}))))

avfonarev21:10:50

@ Completely overlooked this approach. Thank you!

darwin20:10:56

why is clojure.core.rrb-vector enabling :warn-on-boxed here[1]? It seems to be creating a wall of warnings in my REPL when I try to load something which depends on this lib. So far I was unable to somehow disable it. What am I missing here? a warning example: Boxed math warning, clojure/core/rrb_vector/nodes.clj:88:11 - call: public static boolean (long,java.lang.Object). [1] https://github.com/clojure/core.rrb-vector/blob/master/src/main/clojure/clojure/core/rrb_vector.clj#L46

noisesmith20:10:08

worst case you could do (set! *unchecked-math* nil) immediately after loading rrb-vector right?

noisesmith20:10:22

I guess that could mean moving a require out of the ns form, which is annoying

darwin20:10:55

it is a transitive dependency AFAIK, don't have a good control there and all the warnings seem to be originated in the lib itself

darwin20:10:58

I have java11, tried with clojure 1.9 and beta 1.10, maybe I should try with java8. That might be relevant.

noisesmith20:10:43

if you explicitly require the ns, then call set!, outside the ns form, the later require becomes a no-op

noisesmith20:10:59

this is ugly but it does undo the behavior for everything outside that one lib

darwin20:10:26

ok, just tested it under java8 and behaves the same

noisesmith20:10:42

or you're saying it's already only warning for things inside that lib and not other code?

noisesmith20:10:46

the real fix might be telling the rrb-vector devs to only turn that on during dev? - ignore what I was saying about setting unchecked-math yourself, that clearly wouldn't help here

hiredman20:10:37

what version of rrb is that?

darwin20:10:49

I see, I wonder if I'm the only one hitting this issue, btw. it does not appear when running tests or entering via -main

darwin20:10:05

latest version 0.0.12

darwin20:10:44

ah, actually when running lein test it also appears

andy.fingerhut20:10:41

core.rrb-vector causes many such warnings. It was written before the Clojure compiler implemented those warnings, and core.rrb-vector has not been updated much since then. It is a performance warning only, and not nearly as bad of a performance issue as reflection. It would be nice if core.rrb-vector didn't spew all of those warnings for non-developers of the library, I agree.

darwin20:10:08

@andy.fingerhut thanks for the explanation, I'm going to use a patched version locally

seancorfield21:10:31

I can't even run lein test on the latest core.rrb-vector:

Exception in thread "main" java.io.FileNotFoundException: Could not locate clojure/test/check/generators__init.class or clojure/test/check/generators.clj on classpath: 

darwin21:10:40

@seancorfield same error here, under java11

seancorfield21:10:17

That's with Java 8 for me.

seancorfield21:10:34

Looks like mvn test works tho'...

seancorfield21:10:55

(and doesn't spew those warnings)

waffletower21:10:29

I never see such errors in my project that uses core.rrb-vector “0.0.11”

darwin21:10:12

I'll have to google more about this, it is first time I encountered this

darwin21:10:41

commenting out all those (set! ...) commands and using the patched library locally fixed my issue, the warnings are gone

alexmiller21:10:35

I’d want to ping Michał but seems like we could release a new version w/o those

andy.fingerhut21:10:08

I've started to dig into core.rrb-vector a bit to learn the data structure and code, to see if I can fix some bugs and do some comparative performance testing of it vs. Clojure's built in vectors, but it could be weeks before I get there part time.

andy.fingerhut21:10:04

There are about 10 public forks of the library on Github, where others look like they have started something similar, but not gotten to something they were willing to release -- hopefully I don't end up there.

alexmiller21:10:38

well I won’t end up there :)

alexmiller21:10:48

note that commenting out those set!’s actually changes the result - the key is to set to true vs :warn-on-boxed

alexmiller21:10:32

the only canonical build file in contrib libs is the pom.xml so you should always run mvn to build / test

alexmiller21:10:41

which is clean for me

alexmiller21:10:54

(but I have experienced the behavior above)

andrea.imparato21:10:50

hello everyone! rest-api question, do you recommend to use more liberator and go with a “full loaded” solution or go with compojure-api with a more DIY solution then?

ericcervin22:10:14

I don't have a recommendation for you. I'll just say that using liberator doesn't feel "full loaded" to me. The library helps me structure my app, but it doesn't like I'm getting sucked into a framework.

andrea.imparato22:10:30

yeah I meant “full loaded” for structuring rest api

ikitommi05:10:36

@ there is also #reitit, like compojure-api, but just data and no magic.

andy.fingerhut21:10:20

I am hoping to reach the more time-consuming goal of changing the code so that even with the warnings enabled, there are none (or only in places where they won't matter for performance) -- again, no promises, but that is why it won't be quick.

alexmiller21:10:07

well, those are real warnings. all that code is using java.util.concurrent.atomic.AtomicInteger and so falls into Number cases for all the math

alexmiller21:10:31

changing that would mean a significant rewrite

darwin21:10:37

my maven is failing with that generators file not found error when trying to test/build/install

alexmiller21:10:01

you run X and see what Y?

alexmiller21:10:11

just saying “failing” doesn’t help me at all

darwin21:10:32

ah. sorry, cloned the core.rrb-vector and tried to run mvn there like seancorfield above

andy.fingerhut21:10:57

FYI, in core.rrb-vector, the non-authoritative Leiningen project.clj file uses a more recent version of collection-check, which causes one of the namespaces to give an error since a namespace in collection-check seems to have changed. That should only be an issue if you use Leiningen on it, though.

darwin21:10:09

ok, now it works for me, after reverting that dirty pom touched by lein

darwin21:10:36

sorry for the noise, but it was a footgun 🙂

andy.fingerhut21:10:44

Yeah, lein install IIRC will overwrite the pom.xml file, and given the inconsistent state of that project's pom.xml and project.clj files, would likely cause such problems.

darwin21:10:00

yes, that's what I did as when I saw project.clj...

darwin21:10:33

anyways, thank you all for prompt help, my issue is resolved now

alexmiller21:10:13

I backed down the warnings and am releasing 0.0.13 now

andy.fingerhut21:10:02

Alex I think I am missing something about what you said earlier "note that commenting out those set!’s actually changes the result - the key is to set to true vs :warn-on-boxed". Do you mean that the compiled code is different with those set!'s commented out?

alexmiller21:10:44

yes - it turns on unchecked math by default

alexmiller21:10:21

user=> (doc *unchecked-math*)
-------------------------
clojure.core/*unchecked-math*
  While bound to true, compilations of +, -, *, inc, dec and the
  coercions will be done without overflow checks. While bound
  to :warn-on-boxed, same behavior as true, and a warning is emitted
  when compilation uses boxed math. Default: false.

andy.fingerhut21:10:47

Cool that it is in the doc string for when I forget again 🙂

alexmiller21:10:28

release is done, takes about 10 minutes to show up in Maven central usually

alexmiller22:10:51

now available

andy.fingerhut22:10:19

I don't know how I was unaware of this before today -- cool for algorithm/data structure enthusiasts like me: https://github.com/lacuna/bifurcan/blob/master/doc/comparison.md

schmee22:10:35

how can I avoid this reflection?

user=> (ArrayDeque. [{}])
Reflection warning, /private/var/folders/q5/s1n6ttwx1x12mzmbk2mmx3t80000gp/T/form-init5839443831084402899.clj:1:1 - call to java.util.ArrayDeque ctor can't be resolved.
#object[java.util.ArrayDeque "0x49d61ca8" "[{}]"]

schmee22:10:57

I tried type hinting the array with ^java.util.Collection but no dice

dpsutton22:10:39

@andy.fingerhut i think zach published that only two days ago

andy.fingerhut22:10:03

oh, well then I am marvelously quickly informed about it 🙂

dpsutton22:10:28

haha you're up to date. if you follow zach on twitter he and alexander yakushev (spelling?) had a discussion about some of the benchmarks you might be interested in

hiredman22:10:58

@schmee type hinting literal collections is tricky, (let [^java.util.Collection c [{}]] (ArrayDeque. c)) may work better

coinedtalk22:10:59

hmm so native java is quite fast it seems

schmee22:10:17

doh, I thought I tried that but I put the hint on the value instead of the variable name facepalm thanks @hiredman!

schmee22:10:04

if you use it the right way java can be way fast

romain22:10:07

Would anyone have a functional approach (in clojure) of bin packing algo? like the first-fit decreasing here https://www.geeksforgeeks.org/bin-packing-problem-minimize-number-of-used-bins/ ?

andy.fingerhut22:10:03

Well, the decreasing part is easy via sort on the pieces. If you write a function that takes an existing set of bins and their current occupancy, and one new item to put in them, and returns the new occupancies after the item is placed into one of the bins, you can iterate that function once for each item via reduce

romain22:10:17

I did it in an imperative language, I'm just starting the part where I loop each pieces and see if the occupancy of bins are enough, but I was overwhelming about the way to

romain22:10:17

I started using for loop, but the old imperative way shows up lol.

romain22:10:44

However I want the list of bins, not just the total bins needed

noisesmith22:10:41

clojure's for is not a loop

andy.fingerhut22:10:06

A reasonable way to represent an assignment of pieces to bins is a vector of one element per bin, and one bin could be represented as a Clojure map with a key :pieces whose value is a vector of piece sizes, and a key :occupancy that is the total of all piece sizes (the latter part is easily calculated from the set of pieces, but could speed up the algorithm by not having to recalculate it many times).

romain22:10:38

@noisesmith for / loop sorry

andy.fingerhut22:10:04

Me personally, a function for determining which bin to put the next new piece in, I would write using loop, but I may be in the Clojure minority there.

andy.fingerhut22:10:09

Using loop doesn't mean you have to mutate data, though -- it could just calculate and return the bin number into which the new piece is placed.

noisesmith22:10:48

@romain np - not trying to be pedantic, I actually don't know what you mean by for / loop. I can picture attempting to do this with for (a list comprehension), or loop (low level iteration), or some combination of the two, but I don't know what approach you are describing

romain22:10:15

I tried both approach, I may have though they are similar, that's why I typed them. But don't worry you didn't offense me in any way 😉

romain22:10:46

@andy.fingerhut thanks, I'll give a try

noisesmith22:10:55

my typical approach is to replace loop with reduce in cases where an input sequence of some sort is consumed in strict order

noisesmith22:10:24

but if it's helpful to consume out-of-order, then yeah use loop instead of reduce

romain22:10:35

I thought about reduce, but it's a long time I haven't thinking in a functional way 😕

romain22:10:02

I'm a bit rusted ^^'

andy.fingerhut22:10:53

As I mentioned, a separate function that just handles "given a current arrangement of previous pieces placed in bins, plus one new piece size, return the new arrangement with the new piece included" is a nice dividing point for thinking about it. Such function could be implemented with loop. It is good for efficiency to use a technique that can stop as soon as it finds the place to put a new piece, without considering all bins if it doesn't need to.

andy.fingerhut22:10:16

No matter how that function is implemented internally, you can then write another that calls it once for each piece, using reduce.

noisesmith22:10:30

reduce can also return early with reduced (not sure if this applies here)

andy.fingerhut22:10:09

The second function I mentioned doesn't have to use reduce, of course -- it could also use loop or any of several other techniques. reduce is especially good for a situation where you have a starting state, and for each element of a sequence, you apply a function to the current state and the new element to get a next state, and then repeat.

andy.fingerhut22:10:34

(starting state in the example = "empty set of bins")

andy.fingerhut22:10:22

And state might be misleading term there -- I just mean "an immutable value representing a state"

romain22:10:18

@andy.fingerhut for the data representation, you'd have a vector containing a set of 2 elements (a vector of pieces and a number of occupancy) ?

andy.fingerhut23:10:13

For the occupancy I was thinking just the total size of all pieces occupying the bin so far. That is redundant information if you have the collection of all piece sizes in the bin, since you could easily recalculate it as the sum of all piece sizes, but it seems like a reasonable idea for efficiency not to have to recalculate that value for each new piece.

romain23:10:14

Ok, now I need to figure how I "break" my loop using loopstatement and it should be ok

noisesmith23:10:41

you can break from loop by having a path that doesn't call recur

andy.fingerhut23:10:31

Body of loop can look like (if <condition to keep going> (recur ...) <return value of loop>)

noisesmith23:10:45

user=> (loop [x 0] (if (> x 100) x (recur (+ x (rand-int 20)))))
106

romain23:10:59

Oh you're right... It might me later in night then I though ^^'

noisesmith23:10:45

or the pathological case

user=> (loop [] :OK)
:OK

andy.fingerhut23:10:27

Or the sociopathological case (loop [] :hit-neighbor)

noisesmith23:10:25

there's a lot of things in clojure that act like identity, under the right conditions

user=> (+ (* (and (or (do (let [] (-> (loop [] (identity (doto 42))))))))))
42
(couldn't help editing to add a few things)

aisamu03:10:44

this url 😂

romain23:10:17

It does the job :+1::skin-tone-2:

noisesmith23:10:50

@darwin it's a question of style, but the outer recur on that function is something I would always replace with reduce

alexmiller00:10:51

Yeah I got those in the wrong place somehow, will fix