This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-08-21
Channels
- # announcements (1)
- # babashka (39)
- # beginners (91)
- # cider (9)
- # clj-kondo (10)
- # cljsrn (1)
- # clojure (54)
- # clojure-europe (45)
- # clojure-italy (2)
- # clojure-nl (1)
- # clojure-spec (39)
- # clojure-uk (21)
- # clojurescript (7)
- # core-typed (1)
- # cursive (9)
- # data-science (1)
- # datomic (2)
- # docker (3)
- # emacs (11)
- # figwheel-main (11)
- # fulcro (19)
- # java (1)
- # juxt (1)
- # kaocha (68)
- # malli (7)
- # meander (5)
- # off-topic (76)
- # pedestal (1)
- # re-frame (6)
- # reveal (1)
- # rum (2)
- # shadow-cljs (48)
- # sql (6)
- # tools-deps (47)
- # vim (8)
- # xtdb (23)
btw i think i found the problem
(apply concatv rest)
can blow up the stack / is not in tail position.Oh, interesting... Does that introduce laziness somehow?
OH! Yeah, that sounds so obvious once you say it out loud! 🙂
let me explain. sorry, i noticed that even if I remove the :fn I still get the crash with the original impl of concatv. but with this loop/recur one (inspired by the Stuart Sierra blog on concat
) that crash won't occur:
(defn concatv
"Strict version of concat."
[head & tail]
(loop [seqs (cons head tail) acc []]
(if (empty? seqs)
acc
(recur (rest seqs) (into acc (first seqs))))))
Seems weird to have head & tail
when you only cons
them together and never use them again.
You might want to consider whether (concatv)
should work and what it should produce (I'd say yes and []
respectively).
i'm just trying to match behavior of the original concat
(which seems to agree w/ you) 🙂
cljs.user=> (concat)
()
I just tried it in Clojure and got an error
Oh, it works after changing to & args
and (loop [seqs args ...
(defn concatv
"Strict version of concat."
[& args]
(loop [seqs args acc []]
(if (empty? seqs)
acc
(recur (rest seqs) (into acc (first seqs))))))
Also, how much memory will it use? the loop/recur one is tail optimized so should use a constant amount of memory
The version with [head & tail]
requires 1+ arguments 🙂
here is a "final" version that works (tested up to 75 iterations at a time):
(s/def ::any-coll
(s/with-gen
(s/coll-of any?)
#(s/gen (s/coll-of any? :max-count 125))))
(s/fdef concatv
:args (s/cat :args (s/* ::any-coll))
:fn #(let [r (:ret %1)
args (-> %1 :args :args)
sumhash (fn [c] (apply + (mapv hash c)))]
(and
(= (count r) (apply + (mapv count args)))
(= (sumhash r) (apply + (mapv sumhash args)))))
:ret (s/coll-of any? :kind vector?))
(defn concatv
"Strict version of concat."
[& args]
(loop [seqs args acc []]
(if (empty? seqs)
acc
(recur (rest seqs) (into acc (first seqs))))))
so yea i think i was kinda confounded earlier by the fact that there was a potential stack overflow in the code under test. so even when i got the test code "right" the code under test could screw me
Complected, even 🙂
small update: so ... there turned out to be 3 problems. my hunch about s/*
was also correct. some of the crashes were emanating from it as well.
here's a "more final" version (at least until I get another random, intermittent failure some time in the future):
(defn arbitrarily-partition [coll freq]
(let [signals (take (count coll) (cycle (cons true (repeat freq false))))
shuffled (shuffle signals)
zipped (map vector shuffled coll)
partitioned (partition-by first zipped)]
(for [c partitioned]
(map second c))))
(s/def ::coll-of-colls
(s/coll-of (s/coll-of any?)))
(s/def ::distinct-coll-of-colls
(s/with-gen
::coll-of-colls
#(gen/let [elems (gen/set (s/gen any?) {:max-elements 150})
freq (s/gen (s/int-in 1 10))]
(arbitrarily-partition elems freq))))
(s/fdef concatenate
:args (s/cat :args ::distinct-coll-of-colls)
:fn #(let [r (:ret %1)
args (-> %1 :args :args)
sumhash (fn [c] (apply + (mapv hash c)))]
(and
(= (count r) (apply + (mapv count args)))
(= (sumhash r) (apply + (mapv sumhash args)))))
:ret (s/coll-of any? :kind vector?))
(defn- concatenate
"Strict version of concat."
[args]
(loop [seqs args acc []]
(if (empty? seqs)
acc
(recur (rest seqs) (into acc (first seqs))))))
(defn concatv
"Strict version of concat."
[& args]
(concatenate args))
I did not read entire discussion, so it might be irrelevant, but:
there is a :distinct
option in s/coll-of
(s/coll-of int? :distinct true)
it's supposed to be distinct across the insides of the inner collections if I recall correctly (i had to read the code to refresh my own memory but that's what it appears to me at this point)
based on the fact that the generator for ::distinct-coll-of-colls first generates a set of elements and then merely chooses random places to "divide" those into buckets