Clojurians Log v2

@gigasquid with data in parquet, what could I do to accomplish the task? which is essentially unshredding a bunch of paper

Depends on what you want your final form. If it’s a data lake, you could crawl with glue to expose in Redshift. Then you could transform with any database tool like DBT

But if just a one off, then transform with tablecloth something else

yeah but what transform op?

sort? group by?

is there like a shuffle operation a la carte?

I haven’t used tablecloth but it’s built on this https://techascent.github.io/tech.ml.dataset/200-quick-reference.html there are your options You can also ask in the Zulip channel, it’s a bit more active there

500GB/300Kb ~= 13M, so there's tens of millions of documents? How is the input stored? (Guessing: Five hundred 1GB files with one edn [id, ordinal, chunk] per line? Or something else?) And what's the desired output? One file per id with the correctly ordered concatenated chunk s of bytes? Or something else? Could be an interesting case for DuckDB, where once loaded, the GROUP BY 'id' and ORDER BY 'ordinal' might be elegantly expressed (and I'd predict efficiently executed, if that matters).

Also, thinking about this for 1min more... iiuc, there may ultimately be no need to group or sort (!) Can the 'ordinal' be used as an offset into the output file (`ordinal * 1200`)? If so, then one could just go linear over the input, writing into each output file (named by id) at the offset implied by that ordinal (growing the file as necessary). At the end, every chunk would end up in the correct place in every file. With careful use of the file system, it might even be possible to parllelize over the input (!) Fun problem.

it is a fun problem, and I have considered both of those solutions (write @ offset ordinal*1200) and duck db

let's say 50M docs, it's all in one file, database output

https://www.youtube.com/watch?v=HAY4TKIvSZE&t=40s

Wow, one 500GB edn file? Is it a vector of vectors?

EDNL

like JSONL

no surrounding vec

it's horrible, i know

huh, and how are the chunks expressed? As quoted strings?

[uuid, int ordinal, string]

conceptually

but, like, reading to \n is safe? Or have to respect quotes, like if a chunk has a \n in it...?

yea

(in reality, there are 5 columns that serve as the compound primary key. I synthesize a uuid from them)

so my solution was to shave 12 bits off the uuid, use it as a partition key to spray all the rows into one of 4096 different files

each output file is small enough to sort in memory, then partition-by the primary key, then coalesce

is there like an a la carte "shuffle" or "partition" op in tablecloth or tmd?

or is group-by efficient enough not to matter?

There are certainly efficient paths through group-by in TMD... By "shuffle" do you mean randomize?

I mean like the Spark shuffle op, related keys go to the same dest

like I described with the 12-bit shaving

user> (shuffle (range 10))
[8 2 7 4 6 0 9 5 3 1]

ok I mean partition then 🙂

(I thought spark's shuffle was locality aware)

I think with TMD those types of operations would result in just a proliferation of datasets, you'd still need to deal with the out of core aspect. If you got the thing into one big arrow chunk (or perhaps ~5,000 arrow chunks), there might be a memory mapped pathway. But then, these ideas bump into re-implementing parts of DuckDB.

I guess it sort of matters too if this just needs to be done exactly once, or if we get a new 500GB dump every hour, or... ?

just once, toy problem

This function may be of interest: https://techascent.github.io/tech.ml.dataset/tech.v3.dataset.reductions.html#var-group-by-column-agg

I've already solved this with 100 lines of intense file manipulation, but I wanted to use something off the shelf

Each row is edn, a vector of [id, ordinal, chunk]

also good ref https://techascent.com/blog/clojure-csv-parquet.html

See you in ~30 minutes: https://bit.ly/data-recur-zoom

Tomorrow ☝️