moin moin

bon dia

morning

any datomic experts around here? I get an exception on a transact and I have no clue how to debug it

or should I ask #datomic?

you'll tend to have to wait for the afternoon for help from over the pond

Bore da

found the problem of my datomic issue. see #datomic for more details

(and it wasn't a datomic issue anyway)

same feeling here @maleghast I wish I also knew more about Datomic.. I kinda know what it does conceptually... but no idea how to actually use it and what it all means.

@thomas - I am in the same boat. I know about some cool things it can do, but rather expect that I need to have a baptism of fire (i.e. do something with it, make lots of mistakes, learn the hard way), or work with / along side someone generous of spirit who has already got the knowledge (however they got it) and learn from them.

There do not appear to be any good books, and online tutorials I have found suffer dreadfully from the “toy app” problem.

that sounds like the plan indeed!!

Just need my CEO to get the damn funding in… 😉

I never got round to sit down and just do it... (like so many things...)

’tis the way the cookie crumbles, sometimes…

I have two side-projects that I am not doing anything with properly and one of them would use Datomic, so maybe, one day…

(or if our funding comes in and we have operating money and stuff, there are clear applications for it in what I am doing now)

[Idiomatic / Clojuric Question] - I have a vector of maps, and I want to know how many have a certain value for a certain key. I know I can do this with a reduce, but is there a better / snappier way of doing this? (I am trying to get better at being idiomatic / thinking like a Clojurist)

what's the preferred way nowadays to integrate test.check w/with clojure.test?

chicken blood and a red moon

@maleghast how big is the vector?

My brain goes to filter and count tbh @maleghast

(count (filter #(= :certainly (:key %)) xs))

@dominicm - I like the look of that - seems less heavy-handed than doing a reduce with an incrementing accumulator.

I use reduce very rarely

@reborg - Right now, in development it’s small, but it could reasonable go into the hundreds, perhaps thousand+ maps.

@dominicm - I am SURE I use it too much, i.e. when there are better / simpler solutions that I don’t realise I have at my disposal. 🙂

the count filter above is perfectly fine then (imho)

@reborg is there something more efficient for larger collections? I'm curious

🙂 you can go all the way up to enterprise edition

??

@reborg Only idea I've come up with is core.reducers for parallelizing the counting. No idea if that would produce a noticeable improvement (or if it would be a detriment)

right, I'm curious to measure it, coming shortly

😂

😄

@dominicm - your help above, exactly what I needed - thanks again. This is all in the pursuit of thinking more Clojuric thoughts, so thanks for that too 🙂

a few options for larger stuff http://sprunge.us/dSTN

going from normal to totally overkill

That’s pretty new to me, but the outputs do tell an interesting tale of optimisation…

I guess the highlight is that the fold:

(r/fold +
    ((comp
       (map :samplevalue)
       (filter #(= 75584 %))
       (map (constantly 1))) +)
    data)

is reasonably easy to follow and ~x3 faster

I wonder if there's much impact by the number of matches you get. I'd be curious to know how a filter that matched >50% impacts the performances.

yeah, my example is pretty sparse, 5 overlapping elements only

out of curiosity @reborg what are the timings for building up the vector from the sequences, the into step… reducers are fine when you have the data in a tree already; but I’ve found seqs can be so pervasive that in practice it takes a lot of care with mapv/`filterv` etc to know things are in the right form for a fold

yes, the "problem statement" above was "I have a vector" so I was happy to acknowledge the happy path already 🙂

yup appreciated… just curious how bad it is when it’s not already in that shape

it doesn't make any sense to time how much building the vector takes

that time is eventually gonna have to be spent, whether eagerly or lazily on demand

sorry I don’t know the usecase… but generally isn’t it an extra cost you may have to pay to put the seq into a tree for the fold to be parallelisable?

not necessarily, you could e.g. implement CollFoldable for chunked seqs

e.g. if the API to a database gives you a lazyseq, I’d have to pay some presumably linear cost for into in order to parallelise processing each result.

but usually if you're using reducers/transducers you shouldn't operate on lazy seqs, but on reducibles

> not necessarily, you could e.g. implement CollFoldable for chunked seqs Interesting idea…

with databases, files, etc, there are approaches where the fold can load the chunk directly from the source, without the need for you to load the entire thing in memory

with DBs the primary key is a natural choice to chunk a big table up

@reborg: Yes agreed. I’ve been looking to do such things for e.g. CSV parsing, etc… i.e. splitting a file into roughly equal chunks seqing to row ends, and batching it up with a reducible API. But to my knowledge such things aren’t widespread yet… Do you know of any examples?

Links foldable-seq https://github.com/paulbutcher/foldable-seq/blob/master/src/foldable_seq/core.clj and iota for files https://github.com/thebusby/iota

ahh yes I’d seen iota… but foldable-seq is new to me… I have a side project that I keep meaning to pick up again around doing (trans|re)ducible I/O… but maybe I don’t need to…

stuff like foldseq is usually a bad idea

it can easily cause the the heap to explode

see https://groups.google.com/forum/#!topic/clojure-dev/qJo7z_9CVdw

@bronsa: doesn’t it depend what you’re reducing into? e.g. if I were counting lines in a file you should be ok, no?

no

or is it because fold is presuming a tree-like reduction?

lazy seqs are impossible to fold in parallel w/o holding onto the original seq, period

@bronsa: ahh yes lazy-seq’s totally

hence why i suggested chunked seqs not lazy seqs

ok… understood… sorry had missed that detail

yes, in that case I agree… the (trans|re)ducible I/O stuff I was trying to do wasn’t backed by seqs. Basically underneath it would just keep feeding you the mutable IOStream/Reader and let higher level things parse them into collections etc.

but interesting idea about ChunkedSeqs… will need to do some digging

here's a question:

(update-in #{1 2 3} [1] inc)
ClassCastException clojure.lang.PersistentHashSet cannot be cast to clojure.lang.Associative  clojure.lang.RT.assoc (RT.java:820)

(#{1 2 3} 2)
=> 2

what would (assoc #{} 1 2) return

associative has to do with assoc not get

I suppose, in a world where update-in would work, it would return #{2}

and then (update-in #{2} [2] inc) would return #{3}

that doesn't make any sense

I agree, I don't really think it makes much sense either.

I think the idea is that essentially, updating a value, renames the key?

you need to think about the assoc behaviour not update-in

update-in follows from assoc, not the other way round

and there's just no sensical behaviour for assoc over sets

isn't there? surely associng a key into a set adds the value to that set?

so (assoc #{1 2} 3) => #{1 2 3}

that's not how assoc works

that's conj

it seems to me that is how i would expect it to work

it just happens that conj and assoc are the same thing for sets

(i mean i know the language doesn't currently work like that, but i don't understand why not)

assoc takes k,v not just k

It would be for hashmaps or vectors yes?

?

maps map arbitrary keys to arbitrary values, vectors map natural number keys to arbitrary values, and sets map arbitrary keys to themselves

look, to make assoc work on sets there's 2 alternatives: - either accept (assoc my-set k v), but it only makes sense when k == v and when k <> v the behaviour would make no sense - or accept (assoc my-set k), changing the signature of assoc, but then (assoc my-map k) would make no sense

it just doesn't make sense either way you look at it

yes i'm not so worried about the syntactical implications

we don't need assoc to work for sets because we have conj

but conceptually i don't see why it shouldn't

and hence why update-in wouldn't work

then what? a set is trivially a specialization of a map, so if that's your argument then ok. but it's not an argument for making assoc work on sets

or update-in

assoc can't work on sets as it has to take a kv pair

because assoc (update-in is just an abstracion over assoc) promises to work with any k,v pair and sets restrict the domain to k,k

i'm not really suggesting anything, maybe it's just that clojure.lang.Associative seems like a slightly bad name

maybe

just read KVAssociative instead ;)

one legitimate question would be why doesn't IPersistentSet implement ILookup

but it seems the answer to my question is "because assoc works that way and clojure.lang.Associative is for assoc)"

no idea

yes

an interesting clojure curio for a thursday afternoon

back to work now i suppose, thanks!

the contract for assoc is defined in Associative