Fork me on GitHub
#specter
<
2016-06-15
>
conaw02:06:42

Putting up another puzzle, this one relating to filters or conditionals

Given a vector of maps
If the values of a key in the map is

a)    A map that has a key of tempid

return just the tempid

if the value of a key is a set, list, or vector  that contains maps with key of tempid
then replace each of those maps with their temp id

example 

[{:a "foo",
  :b "bar",
  :c
  [{:this "other thing",
    :that
    {:nested :map, :deeper #{{:in :we-go, :tempid -1}}, :tempid -2},
    :tempid -3}],
  :tempid -4}
 {:this "other thing",
  :that {:nested :map, :deeper #{{:in :we-go, :tempid -1}}, :tempid -2},
  :tempid -3}
 {:nested :map, :deeper #{{:in :we-go, :tempid -1}}, :tempid -2}
 {:in :we-go, :tempid -1}]


would become
[{:a “foo” :b “bar” :c [-3] :tempid -4}
  {:this “other thing” :that -2 :tempid -3} 
  {:nested :map :deeper #{-1} :tempid -2}
  {:in :we-go :tempid -1}]

conaw02:06:31

right now I have this, for the simpler case of just nested maps

(transform [ALL MAP-VALS (sp/pred :tempid) (collect-one :tempid)]
          (fn [v _] v)) 

nathanmarz02:06:01

what would that transform to?

conaw02:06:32

the transform I have works fine, but doesn’t handle the case when I have a map inside a seq

conaw02:06:46

basically I want to find a way to conditionally insert and ALL between MAP-VALS and (pred :tempid)

nathanmarz02:06:14

i don't understand

nathanmarz02:06:24

you said when a map has a :tempid to replace it with the :tempid value

nathanmarz02:06:33

that would make that example transform to [-4 -3 -2 -1]

nathanmarz02:06:24

@conaw: or does that only happen after the first level of maps?

conaw02:06:49

only happens after first level

conaw02:06:07

yeah, that’s why its ALL MAP-VALS

conaw02:06:17

(transform [ALL MAP-VALS] #((if (and (map? %) #(:tempid %))) (:tempid %) (if (seq %) (keep :tempid %) %)) sampmap2)

nathanmarz02:06:45

(transform [ALL MAP-VALS ConawWalker (pred :tempid)] :tempid data)

nathanmarz02:06:55

I'll let you figure out ConawWalker

conaw02:06:39

the question is, inside ConawWalker, am I using filterer or just a normal checking

nathanmarz02:06:14

the solution I have uses cond-path

nathanmarz02:06:38

and it's recursive obviously

conaw02:06:51

hm, in this case I don’t actually need recursion

conaw02:06:03

well, maybe I do

conaw02:06:31

but I’ve already flattened the tree out so that I only want to do my replacing at one level of depth in

nathanmarz02:06:46

not in that example

nathanmarz02:06:05

:tempid all over the place

conaw02:06:28

the example came about this way

(def sampmap2
  [{:a "foo"
    :b "bar"
    :c [{:this "other thing"
         :that {:nested :map
                :deeper [{:in :we-go}]}}]}])

(declarepath TOPSORT3)
(providepath TOPSORT3
             (sp/cond-path
              map?
               (stay-then-continue
                [MAP-VALS TOPSORT3])
              vector?
                [ALL TOPSORT3]))


(select TOPSORT3 sampmap2)


(defn nested->ds [mapvec]
 (->> mapvec
  (transform [(subselect TOPSORT3 :tempid) (sp/view count)]
           #(range (- %) 0))
  (select TOPSORT3)
  (transform [ALL MAP-VALS (sp/pred :tempid) (collect-one :tempid)]
          (fn [v _] v))))

conaw02:06:32

so, I’ve got each of my nodes already pulled out and selected using a recursive cond-path

conaw02:06:49

nvm, let me think on this, I think I mostly have ConawWalker with TOP-SORT3

nathanmarz02:06:07

yea it's close

nathanmarz02:06:16

by the way you can write TOPSORT3 more concisely

nathanmarz02:06:52

this should be equivalent

(declarepath TOPSORT3)
(providepath TOPSORT3
   [(cond-path
     map? (stay-then-continue MAP-VALS)
     vector? ALL)
    TOPSORT3])

nathanmarz02:06:29

oh nvm, those are not equivalent

nathanmarz02:06:00

that would stack overflow

conaw02:06:08

Why is that?

conaw02:06:24

would cease to be tail recursive?

nathanmarz02:06:06

when it's on a map, it will first stay

nathanmarz02:06:10

then execute TOPSORT3 again

nathanmarz02:06:20

then stay, then TOPSORT3, etc.

nathanmarz02:06:46

with yours it stays, does whatever is after TOPSORT3, then continues recursing on the map vals

nathanmarz02:06:49

anyway, i gotta run

conaw02:06:02

cool, thanks for help

nathanmarz02:06:24

here's a gist with the solution if you can't figure it out https://gist.github.com/nathanmarz/16481b3b4578ed446453bd828335eedb

conaw02:06:44

best way to only navigate to maps with key of a without traversing into a?

conaw02:06:21

and best way of doing all except would be #(not (:a %))

nathanmarz02:06:24

that's the same as saying (pred :a)

nathanmarz02:06:39

but that doesn't account for key existing with nil or false value

conaw02:06:54

pred does?

nathanmarz02:06:06

if you want to truly do it based on key existing, use (selected? (must :a)) or (not-selected? (must :a))

conaw02:06:23

much appreciated

nathanmarz02:06:32

#(:a %) is equivalent to (pred :a)

Chris O’Donnell13:06:55

@nathanmarz: Made a few updates to https://github.com/codonnell/specter/wiki/List-of-Macros after rereading your post and looking through magic-precompilation and magic-precompilation*. I'm just missing examples for path, and I'm not clear yet on the difference between comp-paths and path. Can path not return a ParamsNeededPath?

nathanmarz14:06:42

@codonnell: path is basically comp-paths plus inline factoring and caching

nathanmarz14:06:07

You could use path on its own like this:

(defn user-path [u]
  (path ALL (selected? :id #(= u %))))

nathanmarz14:06:47

user-path will use inline caching + factoring and efficiently return a path to that particular user

nathanmarz14:06:18

select, transform, etc. all use path

nathanmarz14:06:20

@codonnell: merged the new wiki page in

nathanmarz14:06:49

fyi, there will be a new core operation in 0.12.0 called traverse

aengelberg16:06:18

@nathanmarz re: transducers. I agree that the mapcat behavior is the most useful but the name traverse doesn't seem to imply that behavior.

aengelberg16:06:48

What's really happening is you are traversing some path on ALL structures coming on some stream (or collection, etc) then getting all those results. Maybe traverse-all is a more appropriate name?

nathanmarz16:06:16

@aengelberg: yea, that's a better name

aengelberg16:06:21

or go for the pun and call it transverse

aengelberg16:06:45

👏 🎉 i'm here all week

Chris O’Donnell17:06:13

@nathanmarz: after creating your user-path function above, I get an error while trying to use it:

(select (user-path "chris") [{:id "nick"} {:id "foo"} {:id "chris"}])
Failed to cache path: Var user-path must be either a parameterized navigator, a higher order pathed constructor function, or a nav consructor

nathanmarz17:06:55

@codonnell: use compiled-select

Chris O’Donnell17:06:54

alright, that works

nathanmarz17:06:28

yea, I only gave that as an example of using path on its own

nathanmarz17:06:56

but in general, to make a parameterized navigator it's much better to use comp-paths

Chris O’Donnell17:06:53

and I should use compiled-select with a path I have already compiled with comp-paths?

nathanmarz17:06:34

if you know it's already compiled then that will be the most efficient

Chris O’Donnell17:06:08

Also, if I'm understanding path correctly, it doesn't take late binding parameters because it needs to cache the code which creates the parameters at compile time. Is that right?

nathanmarz17:06:10

select* is like compiled-select except it will compile the input path if it's not already compiled

nathanmarz17:06:39

yea path is meant for a fully parameterized path

Chris O’Donnell17:06:29

So is there any reason to use select over select*?

nathanmarz17:06:17

select* doesn't do inline caching

Chris O’Donnell17:06:41

Right, because it's a function rather than a macro.

nathanmarz17:06:56

99.9% of usage should be through the core select, transform, etc. operations

Chris O’Donnell17:06:59

Alright, thanks for answering my questions. :thumbsup:

nathanmarz17:06:28

in my own usage I use select/transform/replace-in over 400 times, comp-paths 15 times, and compiled-select/compiled-transform only twice

Chris O’Donnell18:06:09

@nathanmarz: I added some examples and a longer description for the path macro, as well as some minor formatting changes and a fix for broken links in the list of navigators. Can you think of a situation where it would be prefer able to use path over comp-paths?

thomasdeutsch19:06:38

Hi everyone. I am stuck with a specter problem i can not get a solution for. I have a simple tree structure as a map and the leaves can be at any level (they are a map with a :template key). I can use a walker to simply find the leaves, but for every leaf, i need the path to that leaf. Is there something like collect-one i can use for this?

(def mytree {"e1" {"e2" {"e1" {:template 1}
                        "e2" {:template 2}}}})

(select [(walker #(contains? % :template))] mytree)

Chris O’Donnell19:06:21

@thomasdeutsch: what would you like to be returned for that data structure?

thomasdeutsch19:06:59

i would like to return a vector of maps. The maps have templates + a path to that template. Like [{:template 1 :path ["e1 "e2" "e1]} {:template 2 :path ["e1" "e2" "e2]}]

thomasdeutsch19:06:47

i think i need a transform with a filterer and a second one that will transform a path into the structure?

Chris O’Donnell19:06:35

I think you want a recursive navigator that collects the keys as you go.

Chris O’Donnell19:06:37

@thomasdeutsch: This is close:

(declarepath KeyAccumWalker [k])
(providepath KeyAccumWalker (if-path must STAY [ALL (collect-one FIRST) LAST (params-reset KeyAccumWalker)]))
(select (KeyAccumWalker :template) mytree)

thomasdeutsch19:06:14

i have to say, i do not understand if-path must STAY ... but this is a great place to start from....

Chris O’Donnell19:06:46

@thomasdeutsch: must has an implicit late-bound parameter, which is the key :template

mfikes19:06:54

@nathanmarz: Do you develop on Linux? (If so, wanted to let you know that we now have an alpha of Planck that runs on Linux. It can successfully load Specter from its JAR and evaluate things. Figure that might help if you ever want to check some behavior related to bootstrap ClojureScript.)

Chris O’Donnell19:06:09

so what's really going on there is more like (if-path (must :template) STAY ...)

thomasdeutsch19:06:31

aaahhhh, ok! got it.

Chris O’Donnell19:06:11

The problem is that as we recursively call KeyAccumWalker the collected values are getting nested instead of concatenated.

nathanmarz19:06:43

@codonnell @thomasdeutsch value collection + recursion doesn't work so well in specter at the moment

thomasdeutsch19:06:19

thanks for the heads up

nathanmarz19:06:50

@mfikes: I develop on OSX, does it work there as well?

mfikes19:06:16

Oh… definitely. Planck has worked on OS X all along :) brew install planck and you are good to go

mfikes19:06:55

@nathanmarz: If you ever want to point Planck at your JAR, here’s how:

$ planck -qKc target/specter-0.12.0-SNAPSHOT.jar
cljs.user=> (require-macros '[com.rpl.specter.macros :refer [transform]])
nil
cljs.user=> (require '[com.rpl.specter :refer [ALL]])
nil
cljs.user=> (transform [ALL :a even?] inc [{:a 1} {:a 2} {:a 4} {:a 3}])
[{:a 1} {:a 3} {:a 5} {:a 3}]
That’s all there is to it. (Note that -K will cause a .planck_cache directory to be written for faster subsequent startups. You can also use -k to write to a specific path, or just leave it off and live with slower require processing.)

nathanmarz19:06:21

ok cool, I'll note this down

mfikes20:06:15

Cool. Of course, happy to continue to sort out bootstrap issues if they arise. Just thought you might want to be able to try things yourself if you ever wanted to. 🙂

nathanmarz20:06:34

any leiningen plugin available to be able to do something like "lein planck"?

nathanmarz20:06:55

and get a planck repl with the project code + dependencies available

mfikes20:06:27

Well, you can essentially do the opposite:

planck -c `lein classpath`

nathanmarz20:06:45

ah nice, just as good

mfikes20:06:28

There’s also fairly decent docs covering this stuff (that’s covered in the Dependencies section). http://planck-repl.org/guide.html