Clojurians Log v2
specter
Hi! I'm a Clojure newbie and found specter after googling for nested/complex data processing on Clojure. I am considering it for my learning journey and (hopefully) applying to some challenges I'm currently facing, but I have a question regarding its functionality:
Does specter allows flattening JSON data dynamically? And by that I mean:
• Unnest each and every nested key without knowing their names or paths;
• Explode each and every array key, also without knowing their names or paths;
• Move all exploded/unnested keys to the top/root level;
• Able to handle a moderate amount of JSON data at once (~2GB).
The idea is to dynamically transform a deeply nested JSON into a flat data file (it can be CSV, JSON, Parquet, etc., anything works for me), that is, without knowing its keys beforehand.
Currently, I have a Scala application that does this by recursively using ClassTag/reflective functions to check if each key is a Struct or an Array and then performing the necessary steps to make it flat.
If this kind of task is not possible or recommended with specter, let me know if there are any other tools/libraries that you find to be better suited for this particular challenge.
i've done this with clojure/walk and merge-with
Probably possible with specter, but not sure. You may be interested in this, depending on your use case: https://github.com/cloojure/tupelo/blob/master/docs/forest.adoc
always wanted to get around to learning forest
Yea it seems like a great idea. I'm not sure he mentions it on that page, but the way it works is by flattening the tree, which is what the OP was asking about.
Do you think I would be able to generate data in a tabular format using the tupelo.forest library? After flattening the structure, that is.
I ask this because I also have JSON datasets that have similar structure, however those need to become tables after they are flattened.
I think so, yea. But haven't looked very closely beyond the talk and article.
Great! Thanks for the reply, @isak
flatting nested objects is pretty easy, cept for how you want to name the keys that are are based on 10+ levels of aggregate keys
@pppaul In my Scala application, I take the parent's key name and use it as a prefix for the "child" key. As you can imagine, the names get pretty big, but it works fine for my purpose. I honestly have no idea how to do that on Clojure yet, though 😂
when i did this i made my key names similar to what i would use for (get-in args for the original json (minus array indices)
if you do something like that, then you can have a name function that does something smart with that type of data