This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2018-10-11
Channels
- # announcements (1)
- # aws (2)
- # beginners (43)
- # calva (7)
- # cider (5)
- # cljs-dev (1)
- # cljsrn (2)
- # clojure (68)
- # clojure-android (10)
- # clojure-conj (2)
- # clojure-italy (10)
- # clojure-nl (3)
- # clojure-spec (3)
- # clojure-uk (128)
- # clojurescript (10)
- # core-async (47)
- # cursive (32)
- # datomic (35)
- # events (1)
- # fulcro (24)
- # funcool (2)
- # graphql (2)
- # jobs (3)
- # juxt (2)
- # leiningen (3)
- # luminus (1)
- # off-topic (7)
- # om (1)
- # onyx (2)
- # pedestal (32)
- # perun (2)
- # portkey (2)
- # re-frame (4)
- # ring-swagger (2)
- # rum (3)
- # shadow-cljs (137)
- # spacemacs (4)
- # testing (3)
- # tools-deps (101)
- # uncomplicate (2)
- # unrepl (5)
- # vim (2)
How about a derived topic that contains your groupings, that another group of consumer consumes? It's not clear how KSQL can cut this, however KStreams should. Disclaimer: I'm very new to kafka and ksql.
@jarvinenemil kafka-streams may be worth a look - i don't think it can do windowing based on message counts out of the box, but it supports various time and session based windows, which may be ok ? https://docs.confluent.io/current/streams/developer-guide/dsl-api.html#windowing. alternatively, you may be able to achieve exactly what you want with a state-store https://docs.confluent.io/current/streams/developer-guide/processor-api.html#state-stores
👋 Hello friends. I have been tearing my hair out (of that which I have left) over this: I'm trying to maintain the order of the key/value pairs within a map going in/out of a core.async
chan
, but no matter what I do (use array-map
, the linked/map
library from @frankie, applying those in transducers / applying before they get put
/`>!` into the chan
) nothing seems to preserve the insertion order of a map flowing through... Is this even possible?
A note: even when I check the (type
of what's coming out of the chan
I get the correct type, e.g.: cljs.core/PersistentArrayMap
PersistentArrayMap
(from the docs):
When doing code form manipulation it is often desirable to have a map which maintains key order. An array map is such a map - it is simply implemented as an array of key val key val
Ah sorry. I’m on a phone. Can you post some example showing some code and the behavior your get?
sure, one sec...
Ok, still on phone - what happens when you print the constructed array before it goes in the channel?
Your hypothesis is correct. Let me prn the construct before
ah! Let me try that...
@loganpowell this has nothing to do with core.async -- the ordering is lost as you construct the literal {..} map, you can't magically re-order it into an array-map
you have to construct the map using (array-map k v k v )
instead of doing (into (array-map) {k v k v})
@bronsa @orestis Thank you guys! Now I have to figure out a way to apply that recursively 😄
keep in mind that operations on array-maps can automatically promote the map to an unordered hash-map
I can tell 😉
@loganpowell i've successfully used https://github.com/frankiesardo/linked for insert-ordered maps to avoid the promotion problem
@mccraigmccraig Sweet! I'm trying that currently, but I still get rearrangement. Can you share a snippet?
i don't have a simple snippet - it was in this stream-join lib - https://github.com/employeerepublic/promisespromises/blob/master/src/prpr/stream/cross.clj
but i fear that won't help you much
If this works for a single-depth linked/map
I can make it recursive... Is the (sort m)
the trick?
this works fine:
(into (linked.core/map) [[:a 1][:b 2][:c 3][:d 4][:e 5][:f 6][:g 7][:h 8][:i 9][:j 10]])
For some reason, I'm still getting re-arrangement
i think core.async is obscuring your problem here - the re-ordering is happening either in the into
or the creation of the literal map... you are putting the resultant map whole onto the promise-chan
Ah, I realize what I need to do: Turn the original map into a vec
first. Edit: This also seems to work for `(into (array-map) (vec map1))
what about replacing {:vintage "2016" ...}
with [[:vintage "2016] ...]
- as far as into is concerned it's identical, and it won't reorder at read time
@noisesmith that's exactly right
conversations like these are the reason that maps in Go (and later Java) have random iteration order
guaranteed to be random, or just no guarantee of a specific order?
@markmarkmark Sorry for air-hogging. Thank you @mccraigmccraig and @noisesmith for branstorming with me!
@noisesmith they're guaranteed to be random
I'm not sure about Go, but in Java the iteration order is chosen by a random seed that is initialized when the JVM starts (or the first map is iterated probably). So it's random per JVM startup rather than per iteration.
oh - yeah that's a security feature
(see the php hash collision attacks for reference)
I thought you meant that two iterations in the same runtime would have different orders
my understanding is that it's because developers depend on the iteration order of maps being the same and then get mad when an internal change causes the arbitrary order of iteration to not match the past.
that's explicitly the reason they do it in Go, but I'm having trouble finding something like that about Java.
ah, it might only be in the new Immutable collections that you get from a call like Map.of(...). From java.util.ImmutableCollections:
/**
* A "salt" value used for randomizing iteration order. This is initialized once
* and stays constant for the lifetime of the JVM. It need not be truly random, but
* it needs to vary sufficiently from one run to the next so that iteration order
* will vary between JVM runs.
*/
static final int SALT;
static {
long nt = System.nanoTime();
SALT = (int)((nt >>> 32) ^ nt);
}
apologies for the tangent
there was a vulnerability several years ago that effected many web frameworks, that was the result of being able to guess how query param names would hash in the hashmap the web frameworks used
ah, this comment on the issue where the immutable collections were added specifically calls out people depending on the iteration order coincidentally being the same: https://bugs.openjdk.java.net/browse/JDK-8048330?focusedCommentId=13843815&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13843815