Clojurians Log v2
specter
struggling a bit to acquire specter-brain...
How to lift out a sub-map as follows? Where...
• Top-level ::id* keys are unique identifiers.
• Nested keys follow a standard schema.
FROM: arbitrary map, conforming to a schema:
{::id-1 {:B1 {:C1 {:x :foo
:y :bar}}
:B2 {} ;; some arbitrary map
:B3 {} ;; some arbitrary map
}
::id-2 {:B1 {:C1 {:x :baz
:y :quxx}}
:B2 {}
:B3 {}}}{::id-1 {:B1 {:C1 {:y :bar}}}
::id-2 {:B1 {:C1 {:y :quxx}}}}
This might fall into the data-structure-manipulation wheelhouse which is, from what I (think I) understand, not the express purpose of Specter, but merely accessing those end-values :bar and :quxx would be
(which would be the relatively trivial (select [MAP-VALS :B1 :C1 :y] tmap))
(though if you wanted to keep them tied to the id, you could do something like this)
(select [ALL (collect-one (view first)) LAST :B1 :C1 :y] tmap)
it is definitely really really tempting to want to use specter as a generic data-manipulation library, but i think of its intent as "accessing", and not "reshaping" (`transform` just lets you do stuff to the accessed values)
(all this with the caveat that im also a noob, and my normal brain rarely works, let alone specter-brain)
it's doable like this:
(transform
[MAP-VALS
(view #(select-keys % [:B1]))
:B1
(view #(select-keys % [:C1]))
:C1]
#(select-keys % [:y])
data)Wow, I see it now, but I would never have put that one together... Working with, well, "higher order navigation" for lack of better words is pushing my brain in a good way. Studying it, stepping back. Thanks, Nathan.
with practice it will become so intuitive that you can't see data any other way
Counting on it... Aaand I think the penny just dropped this morning about developing a sense of "navigation" x "path" X "transformation" (at least about the expression above). Literally a commode-driven-development moment 🚽. Approximate in-words explanation below of that not-yet-intuition.
(transform
[MAP-VALS ;; for each key at this level, walk each val as follows
(view #(select-keys % [:B1])) ;; at this level, construct the thing that is select-keys of whatever (hash-map) is available at this level
:B1 ;; now walk into the :B1 path of that, while also marking "here" as :B1
(view #(select-keys % [:C1])) ;; repeat
:C1]
#(select-keys % [:y]) ;; finally, get the :y sub-map and "send it back up" to transform so it can place it "at the end" of the transformation... meanwhile transform has "collected" the tree at each "step-in" into the navigation and so the :y sub-map is the leaf of the transformed tree... data itself could be much much deeper
data)So far I was mucking about the API to see what did what... And I was going to do nasty things with let bindings etc. Brittle, brute-force manual path reconstruction.
(specter/select (specter/submap [specter/MAP-VALS specter/MAP-VALS specter/MAP-VALS :y])
data)
(specter/select [specter/MAP-VALS specter/MAP-VALS specter/MAP-VALS :y]
data)
(specter/select-one (specter/submap [::id-1 :B1 :C1])
data)
(specter/transform [(specter/walker :y) :y]
identity
data)
(specter/setval [::id-2 :B1 :C1 (specter/compact :y)]
specter/NONE
data)Now, I'm trying to generate permutations of get-in key-paths that point to all Kth leaves of a K-level deep-map.
Re-using the same example, def'd as data, I can brute-force it like this, for my limited use case.
However, a general solution eludes me. This is clearly a tree-recursive problem, but I'm failing at a solution using recursive-path (as well as transform + walker).
Related: What sort of method (or navigator) could access any leaf node of a tree-map at any level of nesting (not just the Kth level)? (`terminal` perhaps?)
user> (require '[com.rpl.specter :as specter])
user> (def data {...}) ;; same map as in OP
user> (defn ->leaf-path-permutations
[data]
(let [[id b c d] [(specter/select specter/MAP-KEYS data)
(distinct (specter/select [specter/MAP-VALS specter/MAP-KEYS]
data))
(distinct (specter/select [specter/MAP-VALS specter/MAP-VALS specter/MAP-KEYS]
data))
(distinct (specter/select [specter/MAP-VALS specter/MAP-VALS specter/MAP-VALS specter/MAP-KEYS]
data))]]
(for [id' id
b' b
c' c
d' d]
[id' b' c' d'])))
#'user/->leaf-path-permutations
user> (->leaf-path-permutations data)
([:user/id-1 :B1 :C1 :x]
[:user/id-1 :B1 :C1 :y]
[:user/id-1 :B1 :C1 :z]
;;; etc etc etc
[:user/id-2 :B3 :C1 :z]
[:user/id-2 :B3 :C2 :x]
[:user/id-2 :B3 :C2 :y]
[:user/id-2 :B3 :C2 :z])
user> ;; but this is so icky, obviously
I don't understand the problem you're trying to solve, but I can answer the related question:
(recursive-path [] p (if-path map? [MAP-VALS p] STAY)) this will navigate to every value of a nested map
use (stay-then-continue p) instead of p to get every node (a bit like tree seq)
and it's also not hard to modify to collect the path along the way: (recursive-path [] p (if-path map? [ALL (collect-one FIRST) LAST p] STAY))
Thanks @rolthiolliere, this is a teaching moment.
Why was I trying that?
I got tunnel vision playing with specter, trying to make indices over deeply nested in-memory data (pre-computed get-in indices). Something like this:
;; deep-map -> "normal" index -> other indices with other sort orders
{:id-1 {:B1 {:C1 {:x :foo}}}} -> [:id-1 :B1 :C1 :x] / [:id-2 :B1 :C2 :x] / [:id-2 :B1 :C2 :x]
[:id-1 :B2 :C2 :x] / [:id-1 :B1 :C1 :x] / [:id-1 :B2 :C2 :x]
[:id-2 :B1 :C2 :x] / [:id-1 :B2 :C2 :x] / [:id-1 :B1 :C1 :x]There are so many ways to compose these things. Plus I am used to deliberately making all maps shallow (as much as I can help it). It's taking me longer than I care to admit in polite society to rewire my brain around arbitrarily deep navigation.
Excited to see the write up on your experience if you end up adding it to your blog @adityaathalye 😆
Haha thank you for the encouragement :) I think there will be one... right now I'm just feeling my way about specter-brainspace. My actual use-case is likely to be straightforward selects and transforms (but I expect to have to do a lot of that). More soon here and on the ol' blog.
But first, I want to understand enough about paths and navigators to be dangerous (hence the side-track into recursive navigation).
A https://www.evalapply.org/posts/poor-mans-time-oriented-data-system/index.html#xtdb-all-facts-are-bitemporal-by-design + #honeysql + #specter is a wonderful thing... at least to code up. Let's see how the schema holds up to real-world benching.
(def all-schemas
(specter/transform
[specter/MAP-VALS specter/MAP-VALS specter/MAP-VALS]
->hsql+sql-map
{::attributes {:facts {:ddl (->hsql-table-EAV-facts ::attributes)
:idx-ae (->hsql-idx-EA-facts ::attributes)
:idx-ea (->hsql-idx-AE-facts ::attributes)}
:now {:ddl (->hsql-view-EAV-now ::attributes)}}
::namespaces {:facts {:ddl (->hsql-table-EAV-facts ::namespaces
hsql-constrain-check-namespace-is-trimmed)
:idx-ae (->hsql-idx-EA-facts ::namespaces)
:idx-ea (->hsql-idx-AE-facts ::namespaces)}
:now {:ddl (->hsql-view-EAV-now ::namespaces)}}
::entities {:facts {:ddl (->hsql-table-EAV-facts ::entities
hsql-constrain-fk-entity-namespaces-by-namespaces-now)
:idx-ae (->hsql-idx-EA-facts ::entities)
:idx-ea (->hsql-idx-AE-facts ::entities)}
:now {:ddl (->hsql-view-EAV-now ::entities)
:fts5 (->hsql-fts5-now ::entities)
:trg-fts5-after-insert (->hsql-fts5-now-trigger-after-table-insert ::entities)
:trg-fts5-after-delete (->hsql-fts5-now-trigger-after-table-delete ::entities)
:trg-fts5-after-update (->hsql-fts5-now-trigger-after-table-update ::entities)}}
::roles {:facts {:ddl (->hsql-table-EAV-facts ::roles
hsql-constrain-fk-entity-namespaces-by-namespaces-now)
:idx-ae (->hsql-idx-EA-facts ::roles)
:idx-ea (->hsql-idx-AE-facts ::roles)}
:now {:ddl (->hsql-view-EAV-now ::roles)}}
::users {:facts {:ddl (->hsql-table-EAV-facts ::users
hsql-constrain-fk-entity-namespaces-by-namespaces-now)
:idx-ae (->hsql-idx-EA-facts ::users)
:idx-ea (->hsql-idx-AE-facts ::users)}
:now {:ddl (->hsql-view-EAV-now ::attributes)}}
::app {:facts {:ddl (->hsql-table-EAV-facts ::app
(concat hsql-constrain-fk-entity-namespaces-by-namespaces-now
hsql-constrain-fk-entities-by-entities-now
hsql-constrain-fk-user-ref-by-users-now))
:idx-ae (->hsql-idx-EA-facts ::app)
:idx-ea (->hsql-idx-AE-facts ::app)}
:now {:ddl (->hsql-view-EAV-now ::app)
:fts5 (->hsql-fts5-now ::app)
:trg-fts5-after-insert (->hsql-fts5-now-trigger-after-table-insert ::app)
:trg-fts5-after-delete (->hsql-fts5-now-trigger-after-table-delete ::app)
:trg-fts5-after-update (->hsql-fts5-now-trigger-after-table-update ::app)}}}))☝️ Up there is the schema definition (a table of schemas, if you will). 👇 Down here is the concrete migration sequence.
(def all-schemas-migration-sequence-sqls
(specter/transform
[specter/ALL]
(fn [schema-path]
[schema-path (:sql (get-in all-schemas schema-path))])
[;; First, create all prerequisite tables
[::attributes :facts :ddl]
[::attributes :facts :idx-ae]
[::attributes :facts :idx-ea]
[::attributes :now :ddl]
[::namespaces :facts :ddl]
[::namespaces :facts :idx-ae]
[::namespaces :facts :idx-ea]
[::namespaces :now :ddl]
[::entities :facts :ddl]
[::entities :facts :idx-ae]
[::entities :facts :idx-ea]
[::entities :now :ddl]
[::roles :facts :ddl]
[::roles :facts :idx-ae]
[::roles :facts :idx-ea]
[::roles :now :ddl]
[::users :facts :ddl]
[::users :facts :idx-ae]
[::users :facts :idx-ea]
[::users :now :ddl]
[::app :facts :ddl]
[::app :facts :idx-ae]
[::app :facts :idx-ea]
[::app :now :ddl]
[::app :now :fts5]
[::app :now :trg-fts5-after-insert]
[::app :now :trg-fts5-after-delete]
[::app :now :trg-fts5-after-update]]))And running the migration is just...
(defn migrate-all-schemas!
([db]
(migrate-all-schemas! db all-schemas-migration-sequence-sqls))
([db all-schema-migration-sequence-sqls]
(log/info "Setting up all tables.")
;; Make all the tables and views
(jdbc/with-transaction [tx db]
(doseq [[schema-path sql-query] all-schema-migration-sequence-sqls]
(when (vector? sql-query)
(let [ddl-result (jdbc/execute-one! tx sql-query)]
(if (zero? (:next.jdbc/update-count ddl-result))
(log/info "Skipped schema: " schema-path)
(log/info "Created schema: " schema-path)))))
(log/info "Ran migration step for tables and views."))))Okay, I'm done with the "deliberate discomfort" exercises. Now I'll let my use-cases guide me.
Thanks everyone for your help.
For anyone else reading this... If you are like me, and understand by doing hurting, it's worth ignoring @nathanmarz’s advice, but just for a bit (in the first few deliberate experiments). For one can ignore it for a bit too long, at one's own peril. Thanks for putting this in words, Nathan.
> I've noticed many people struggle to grasp all the ways Specter can be applied, no doubt an instance of the blub paradox. Substructure navigation, recursive paths, and higher-order navigators are a few of the more advanced concepts that can seem overwhelming. I recommend starting with the most obvious use case, transformations of compound data structures, and letting your brain slowly adapt to the navigation way of thinking. That's the way I started, and the library slowly evolved from there as I saw the different ways these ideas could be leveraged. Eventually it will become part of your basic programming instinct, and you'll wonder how you ever lived
> without it.
http://nathanmarz.com/blog/clojures-missing-piece.html