Fork me on GitHub
#core-async
<
2018-10-11
>
fmjrey04:10:51

How about a derived topic that contains your groupings, that another group of consumer consumes? It's not clear how KSQL can cut this, however KStreams should. Disclaimer: I'm very new to kafka and ksql.

mccraigmccraig08:10:24

@jarvinenemil kafka-streams may be worth a look - i don't think it can do windowing based on message counts out of the box, but it supports various time and session based windows, which may be ok ? https://docs.confluent.io/current/streams/developer-guide/dsl-api.html#windowing. alternatively, you may be able to achieve exactly what you want with a state-store https://docs.confluent.io/current/streams/developer-guide/processor-api.html#state-stores

Logan Powell15:10:20

👋 Hello friends. I have been tearing my hair out (of that which I have left) over this: I'm trying to maintain the order of the key/value pairs within a map going in/out of a core.async chan, but no matter what I do (use array-map, the linked/map library from @frankie, applying those in transducers / applying before they get put/`>!` into the chan) nothing seems to preserve the insertion order of a map flowing through... Is this even possible?

Logan Powell15:10:05

A note: even when I check the (type of what's coming out of the chan I get the correct type, e.g.: cljs.core/PersistentArrayMap

orestis15:10:15

But that type doesn’t preserve order, does it?

Logan Powell15:10:10

PersistentArrayMap (from the docs):

When doing code form manipulation it is often desirable to have a map which maintains key order. An array map is such a map - it is simply implemented as an array of key val key val

orestis15:10:00

Ah sorry. I’m on a phone. Can you post some example showing some code and the behavior your get?

Logan Powell15:10:12

sure, one sec...

orestis16:10:13

Ok, still on phone - what happens when you print the constructed array before it goes in the channel?

orestis16:10:45

That is, capture the created array in the let, then print it.

orestis16:10:17

My hypothesis is that you could even do identical? and it would be true.

Logan Powell16:10:49

Your hypothesis is correct. Let me prn the construct before

orestis16:10:07

Actually, the issue is obvious now. Nothing to do with core async.

orestis16:10:48

You are using a map literal...

Logan Powell16:10:25

ah! Let me try that...

bronsa16:10:08

@loganpowell this has nothing to do with core.async -- the ordering is lost as you construct the literal {..} map, you can't magically re-order it into an array-map

bronsa16:10:28

you have to construct the map using (array-map k v k v ) instead of doing (into (array-map) {k v k v})

Logan Powell16:10:27

@bronsa @orestis Thank you guys! Now I have to figure out a way to apply that recursively 😄

bronsa16:10:04

keep in mind that operations on array-maps can automatically promote the map to an unordered hash-map

Logan Powell16:10:39

I can tell 😉

mccraigmccraig17:10:55

@loganpowell i've successfully used https://github.com/frankiesardo/linked for insert-ordered maps to avoid the promotion problem

Logan Powell17:10:55

@mccraigmccraig Sweet! I'm trying that currently, but I still get rearrangement. Can you share a snippet?

mccraigmccraig17:10:23

but i fear that won't help you much

Logan Powell17:10:56

If this works for a single-depth linked/map I can make it recursive... Is the (sort m) the trick?

mccraigmccraig17:10:05

this works fine:

(into (linked.core/map) [[:a 1][:b 2][:c 3][:d 4][:e 5][:f 6][:g 7][:h 8][:i 9][:j 10]])

Logan Powell17:10:10

For some reason, I'm still getting re-arrangement

mccraigmccraig17:10:13

i think core.async is obscuring your problem here - the re-ordering is happening either in the into or the creation of the literal map... you are putting the resultant map whole onto the promise-chan

Logan Powell18:10:10

Ah, I realize what I need to do: Turn the original map into a vec first. Edit: This also seems to work for `(into (array-map) (vec map1))

noisesmith18:10:15

what about replacing {:vintage "2016" ...} with [[:vintage "2016] ...] - as far as into is concerned it's identical, and it won't reorder at read time

Logan Powell18:10:27

@noisesmith that's exactly right

markmarkmark18:10:35

conversations like these are the reason that maps in Go (and later Java) have random iteration order

noisesmith18:10:20

guaranteed to be random, or just no guarantee of a specific order?

Logan Powell18:10:27

@markmarkmark Sorry for air-hogging. Thank you @mccraigmccraig and @noisesmith for branstorming with me!

markmarkmark18:10:51

@noisesmith they're guaranteed to be random

noisesmith18:10:01

that seems like a waste of resources

8
markmarkmark18:10:17

I'm not sure about Go, but in Java the iteration order is chosen by a random seed that is initialized when the JVM starts (or the first map is iterated probably). So it's random per JVM startup rather than per iteration.

noisesmith18:10:34

oh - yeah that's a security feature

noisesmith18:10:51

(see the php hash collision attacks for reference)

noisesmith18:10:18

I thought you meant that two iterations in the same runtime would have different orders

markmarkmark18:10:23

my understanding is that it's because developers depend on the iteration order of maps being the same and then get mad when an internal change causes the arbitrary order of iteration to not match the past.

markmarkmark18:10:50

that's explicitly the reason they do it in Go, but I'm having trouble finding something like that about Java.

markmarkmark18:10:54

ah, it might only be in the new Immutable collections that you get from a call like Map.of(...). From java.util.ImmutableCollections:

/**
 * A "salt" value used for randomizing iteration order. This is initialized once
 * and stays constant for the lifetime of the JVM. It need not be truly random, but
 * it needs to vary sufficiently from one run to the next so that iteration order
 * will vary between JVM runs.
 */
static final int SALT;
static {
    long nt = System.nanoTime();
    SALT = (int)((nt >>> 32) ^ nt);
}

markmarkmark18:10:28

apologies for the tangent

hiredman18:10:46

there was a vulnerability several years ago that effected many web frameworks, that was the result of being able to guess how query param names would hash in the hashmap the web frameworks used

markmarkmark18:10:43

ah, this comment on the issue where the immutable collections were added specifically calls out people depending on the iteration order coincidentally being the same: https://bugs.openjdk.java.net/browse/JDK-8048330?focusedCommentId=13843815&amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13843815