Fork me on GitHub
#specter
<
2018-06-11
>
ben.mumford14:06:16

what is the more elegant way to extract a value from a nested data structure and then apply a transform to it?

ben.mumford14:06:43

my use case is that i have lots of different types of input data structures and i want to normalise them so that i can compose and compare them

ben.mumford14:06:10

so what i'm doing is for each input object creating a new output object where each field is calculated using a select then a transform (so i guess question 2 is does this make specter a good fit for my use case as i'm not modifying a pre-existing data structure)

nathanmarz15:06:14

@ben.mumford620 you can use view or transformed at the end of the path to the select operation

jsa-aerial16:06:37

Would spectre be the 'natural/goto' thing for taking a nested data structure and a 'key' and replace any occurance of it with a value? Where 'key' would most probably be literally a keyword (or possibly string)?

nathanmarz17:06:18

@jsa-aerial yes, that's a bread and butter use case

nathanmarz17:06:06

walker is the lazy way to do that, but if your data has any structure to it whatsoever it's better to make a precise path

nathanmarz17:06:29

much higher performance + much less bug prone

jsa-aerial18:06:22

@nathanmarz OK, sounds promising. The data will have quite a lot of structure and the 'paths' (there will typically be several such 'keys') will basically not be known ahead of time. So, this basically involves 'search' and replace. Why would non precise path and walker tend to be more 'bug prone'??

nathanmarz18:06:06

walker will descend everywhere including places you don't expect, like records, map keys, etc.

jsa-aerial18:06:42

But if the 'key' can basically be anywhere, that would seem to be the 'right thing', correct?

jsa-aerial18:06:17

Actually, ALL almost looks like exactly what is needed here. Except it returns [k v] for map elements

jsa-aerial18:06:42

But if the 'key' can basically be anywhere, that would seem to be the 'right thing', correct?

nathanmarz18:06:49

here's a variation of walker that doesn't descend into map keys / key/value pairs

(def my-walker
  (recursive-path [afn] p
    (cond-path (pred afn) STAY
               map? [MAP-VALS p]
               coll? [ALL p]
               )))

nathanmarz18:06:26

if your data is truly completely unstructured, then walker is more appropriate

nathanmarz18:06:43

though even then, it's better to make a variation that doesn't navigate to key/value pairs

nathanmarz18:06:14

e.g.

(def my-walker
  (recursive-path [afn] p
    (cond-path (pred afn) STAY
               map? [(multi-path MAP-VALS MAP-KEYS) p]
               coll? [ALL p]
               )))

jsa-aerial18:06:17

I think we have a semantic issue here - you are using 'structured' in a special way - not just to indicate a lot of structure in the data

nathanmarz18:06:53

by structure I mean that the organization of the data is precise and well-understood

jsa-aerial18:06:30

Something like this {:a ::some-key ....} can occur and should become {:a <the-value-for-::some-key> ...}

jsa-aerial18:06:04

It's precise, it is just not fully known up front. And these 'keys' can occur in all sorts of places. Basically, these structures will be "parameterized" vega-lite specifications.

nathanmarz18:06:37

if the paths to what you care about aren't known in advance, then I would call it unstructured

nathanmarz18:06:09

if you know it will never occur in a map key, then it's best your path reflect that

nathanmarz18:06:23

or if you know any other constraints, those should be included in the path

jsa-aerial18:06:24

OK, that is fine - now I know what you mean by that, it would seem walker is the way to go

jsa-aerial18:06:50

Yeah, those sorts of constraints and assumptions cannot be relied upon in this sort of scenario.

jsa-aerial18:06:26

There are constraints and requirements in vega-lite data specs, but here the idea is to use a basic layout as a template which will have these 'keys' replaced by values for specific cases and these 'keys' could be standing in for both VL keys and/or values.

jsa-aerial18:06:46

Hmmmm ALL maybe could still work if it is just called recursively on any [k v] pair returned

currentoor22:06:08

is it generally frowned upon to use setval inside of a transform-fn in a transform call?