This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-08-29
Channels
- # announcements (5)
- # beginners (25)
- # calva (53)
- # clj-kondo (9)
- # clojure (25)
- # clojure-europe (14)
- # clojure-nl (1)
- # clojure-norway (21)
- # clojure-uk (1)
- # conjure (2)
- # data-science (1)
- # datalevin (4)
- # datascript (6)
- # deps-new (5)
- # emacs (5)
- # etaoin (6)
- # figwheel-main (1)
- # fulcro (46)
- # gratitude (3)
- # hyperfiddle (8)
- # introduce-yourself (13)
- # lsp (13)
- # nextjournal (5)
- # off-topic (2)
- # pathom (4)
- # polylith (11)
- # re-frame (16)
- # releases (4)
- # scittle (67)
- # shadow-cljs (38)
- # slack-help (4)
- # specter (13)
- # sql (29)
- # squint (21)
- # test-check (3)
- # vim (13)
- # xtdb (15)
Hi! I'm a Clojure newbie and found specter after googling for nested/complex data processing on Clojure. I am considering it for my learning journey and (hopefully) applying to some challenges I'm currently facing, but I have a question regarding its functionality:
Does specter allows flattening JSON data dynamically? And by that I mean:
• Unnest each and every nested key without knowing their names or paths;
• Explode each and every array key, also without knowing their names or paths;
• Move all exploded/unnested keys to the top/root level;
• Able to handle a moderate amount of JSON data at once (~2GB).
The idea is to dynamically transform a deeply nested JSON into a flat data file (it can be CSV, JSON, Parquet, etc., anything works for me), that is, without knowing its keys beforehand.
Currently, I have a Scala application that does this by recursively using ClassTag/reflective functions to check if each key is a Struct
or an Array
and then performing the necessary steps to make it flat.
If this kind of task is not possible or recommended with specter, let me know if there are any other tools/libraries that you find to be better suited for this particular challenge.
Probably possible with specter, but not sure. You may be interested in this, depending on your use case: https://github.com/cloojure/tupelo/blob/master/docs/forest.adoc
Yea it seems like a great idea. I'm not sure he mentions it on that page, but the way it works is by flattening the tree, which is what the OP was asking about.
Thanks for the suggestions @U0LAJQLQ1 and @U08JKUHA9, much appreciated. I will take a good look on these alternatives!
Do you think I would be able to generate data in a tabular format using the tupelo.forest
library? After flattening the structure, that is.
I ask this because I also have JSON datasets that have similar structure, however those need to become tables after they are flattened.
Great! Thanks for the reply, @U08JKUHA9
flatting nested objects is pretty easy, cept for how you want to name the keys that are are based on 10+ levels of aggregate keys
@U0LAJQLQ1 In my Scala application, I take the parent's key name and use it as a prefix for the "child" key. As you can imagine, the names get pretty big, but it works fine for my purpose. I honestly have no idea how to do that on Clojure yet, though 😂