Fork me on GitHub
#meander
<
2020-03-04
>
grounded_sage12:03:52

Still have problems around joining the large datasets. Put the first transformation step side by side and meander simply blows the other option away.

timothypratley05:03:25

Anything you can share as an example? Sounds interesting 🙂

grounded_sage10:03:10

There is a slight error in the dataset one as I typed it out purely to see what the difference was side by side.

grounded_sage10:03:25

With respect to code size, clarity and ability to change Meander is definitely my choice.

jimmy15:03:54

If you ever want to pair on joining up some datasets I'd be happy to do that. Even if you can't share your stuff. I'm sure we could make some mock data and try things out.

grounded_sage19:03:02

Really appreciate it @U5K8NTHEZ. At the moment I am in datahike land as the datasets are too large to be done in memory with Meander. I also broke Tech.ml.dataset haha

grounded_sage19:03:40

Though they are arguably still moderate in size. <500mb for the largest.

timothypratley00:03:09

Hmmm FWIW it looks like the code on the right just needs to be changed slightly… IMO you should be doing something like:

(for [x (aws/get-data)]
  (m/match x ~~~~ ~~~~~))

timothypratley00:03:33

search is intended for finding multiple logical matches (not sequential processing) I think! 🙂

grounded_sage08:03:05

I’ve had search fail on me for some reason when I am doing a simple match. I think it might be due to m/pred or m/app. I’ll revisit at some point for now I just flatten the results