Fork me on GitHub
#rewrite-clj
<
2022-06-08
>
Oliver George06:06:34

How should I be writing out a file after using z/postwalk? I'm finding this mostly works but can't print "strings with\" delimited quotes" in them. (spit filename (z/root-string zloc)) https://gist.github.com/olivergeorge/51fdd5008ac05cc5dda80a244f14c58e

Oliver George06:06:33

before it was:?

lread13:06:26

Hi @olivergeorge I recently fixed https://github.com/clj-commons/rewrite-clj/issues/176, but have not cut a new release yet.

lread13:06:11

Your gist looks good to me, but one thing you might need to care about (or maybe might not, depends on your use case) is that https://cljdoc.org/d/rewrite-clj/rewrite-clj/1.0.767-alpha/doc/user-guide#not-all-clojure-is-sexpr-able.

Oliver George03:06:25

Brilliant thanks. Appreciate you looking at that gist too!

Oliver George03:06:41

I would like to learn how to assoc a key into a map without losing whitespace - perhaps that's a node api thing.

lread13:06:48

Yeah this is totally doable. Here's something https://github.com/clj-commons/rewrite-clj/blob/27af97900afb930da3812b73251992d87170955e/build/build_util.clj#L9-L18. It can get a little trickier to keep indenting correct when you are inserting a new key value pair. Depending on your use case, you might also want to look at https://github.com/borkdude/rewrite-edn.

lread21:06:29

Ok @olivergeorge I’ve taken some time to study your gist and can explain more what I mean. I’ll start with what I see as the essence of your gist and tweak it to something that is easier for me to play with in the REPL:

(require '[clojure.string :as string]
         '[rewrite-clj.zip :as z]);

(defn massage-field-spec
  [{:keys [type form/widget-type] :as field-spec}]
  (cond-> field-spec
          (and type (not widget-type))
          (assoc :form/widget-type type)))

(defn process-zloc
  [zloc]
  (-> zloc
      (z/up)
      (z/postwalk
       (fn select [zloc]
         (let [x (z/sexpr zloc)]
           (and (map? x)
                (:type x)
                (= (z/sexpr (z/left (z/up zloc))) :field-spec))))
       (fn visit [zloc]
         (z/edit zloc massage-field-spec)))))
You didn’t mention the shape of your data, but here’s a sample I came up with. If I undertand, I think the idea is if we have a :form/widget-type and a :type, we don’t need a change otherwise we want to add a :form/widget-type with the value of :type. Maybe your data looks like this, I dunno the exact shape, it is not that important for the purposes of this discussion.
(def s (string/join "\n" ["[{:field-spec"
                          "  {:something-a"
                          "   {:type :whatever"
                          "    :form/widget-type :all-good"
                          "    :a 1"
                          "    :b 2}}}"
                          " {:field-spec"
                          "  {:something-b"
                          "   {:type :whatever2"
                          "    :c 3"
                          "    :d 4}}}]"]))

;; let's see what that looks like printed out:
(println s)
;; stdout:
[{:field-spec
  {:something-a
   {:type :whatever
    :form/widget-type :all-good
    :a 1
    :b 2}}}
 {:field-spec
  {:something-b
   {:type :whatever2
    :c 3
    :d 4}}}]
So, if I get it, only the 2nd map’s content should update. Let’s see what happens with the current code:
(-> s
    z/of-string
    process-zloc
    z/root-string
    println)
;; stdout:
[{:field-spec
  {:something-a
   {:form/widget-type :all-good, :type :whatever, :b 2, :a 1}}}
 {:field-spec
  {:something-b
   {:form/widget-type :whatever2, :type :whatever2, :c 3, :d 4}}}]
As you’ve mentioned, we’ve lost existing whitespace (and even key value pair ordering). Let’s see what we can do about that.
(defn process-zloc2
  [zloc]
  (-> zloc
      (z/up)
      (z/postwalk
       (fn select [zloc]
         ;; here we avoid sexpr by using other zip fns
         (and (z/map? zloc)
              (z/get zloc :type)
              ;; added in check so we don't have to check in visitor
              (not (z/get zloc :form/widget-type))
              ;; use string just in case whatever is at up and left is not sexpr-able
              (= ":field-spec" (-> zloc z/up z/left z/string))))
       (fn visit [zloc]
         ;; here again we avoid sexpr and use zipper functions
         (let [type-node (-> zloc (z/get :type) z/node)]
           (z/assoc zloc :form/widget-type type-node))))))

(-> s
    z/of-string
    process-zloc2
    z/root-string
    println)
;; stdout:
[{:field-spec
  {:something-a
   {:type :whatever
    :form/widget-type :all-good
    :a 1
    :b 2}}}
 {:field-spec
  {:something-b
   {:type :whatever2
    :c 3
    :d 4 :form/widget-type :whatever2}}}]
That’s better, we’ve not altered the first map at all and preserved whitespace and ordering when adding a key value to the 2nd map. But, if we want, we can assert more control over how we insert that key value pair. Maybe we want to insert it as the first item in the map and preserve indentation:
(defn process-zloc3
  [zloc]
  (-> zloc
      (z/up)
      (z/postwalk
       ;; same a process-zloc2
       (fn select [zloc]
         ;; here we avoid sexpr by using other zip fns
         (and (z/map? zloc)
              (z/get zloc :type)
              ;; added in check so we don't have to check in visitor
              (not (z/get zloc :form/widget-type))
              ;; use string just in case whatever is at up and left is not sexpr-able
              (= ":field-spec" (-> zloc z/up z/left z/string))))
       (fn visit [zloc]
         ;; this time we use some lower level fns for more control
         (let [first-key-offset (-> zloc z/down z/node meta :col)
               type-value-node (-> zloc (z/get :type) z/node)]
           (-> zloc
               z/down
               (z/insert-left :form/widget-type)
               (z/insert-left type-value-node)
               (z/insert-newline-left)
               (z/insert-space-left (dec first-key-offset))))))))

(-> s
    z/of-string
    process-zloc3
    z/root-string
    println)
;; stdout:
[{:field-spec
  {:something-a
   {:type :whatever
    :form/widget-type :all-good
    :a 1
    :b 2}}}
 {:field-spec
  {:something-b
   {:form/widget-type :whatever2 
    :type :whatever2
    :c 3
    :d 4}}}]
Another thing to think about might be reader conditional discards, for example #_ :skipped. Rewrite-clj calls these uneval nodes, but does not skip them by default. Let me know if the above makes sense and is helpful.

Oliver George23:06:44

Thank you @UE21H2HHD - that's amazing. Exactly what I needed and works beautifully. There's a few subtlties in there which I'd have stumbled on.

lread02:06:17

Great to hear! Drop on by anytime should you have any more questions.

gratitude 1
lread16:06:16

Hmm… I think I’ll send my reply to the channel as well.. might help others who might not have noticed our thread.

Oliver George06:06:38

Unrelated. It'd be nice to be able to use pprint instead of prn for s/postwalk changes which can't preserve spacing.

lread13:06:27

Hi @olivergeorge do you mean when debugging? Can you give an example of what you are seeing versus what you’d like to see?

Oliver George03:06:39

My use case is updating a lot of static data in an webapp. I'm parsing a CLJS file, adding some additional keywords to maps and saving the change. Using the z/postwalk (sexpr) approach leads to losing whitespace. That seems to serialise the map as one long line. Ideally I wouldn't lose whitespace or reorder existing keys - that would preserve my git blame.

Oliver George03:06:04

More pressing is readable code... if prn was replaced by pprint I would have my map printed over many lines. (not one super long line)

Oliver George03:06:16

Workaround is to pprint each top level expression. I lose gitblame and original whitespace but it's somewhat stable so each additional key will be a clean addition to the source file.

Oliver George03:06:44

(really feeling my way forward for a neat solution - losing gitblam isn't the end of the world)

lread12:06:28

If you want to preserve whitespace, sexpr is not your friend. At least for nodes that contain whitespace. I'll follow up in our other thread, with an example.

lread13:06:38

I tend to get a tad (that might be an understatement) focused on what I’m working on. I’ll come back to rewrite-clj today and cut a new release for https://github.com/clj-commons/rewrite-clj/issues/176 (and I’ll also address https://github.com/clj-commons/rewrite-clj/issues/178).

🎉 1
lread15:06:48

Not that anybody will likely care much, but I think I’m going to move away from using commit-count in the version scheme for rewrite-clj. I was quite enamored with it once, but now find it a bit too futzy with regards to its knowing what my next release version will actually be. So the next release will likely be 1.1.<release num>. Where <release num> will be for the whole life of rewrite-clj. And I’ll finally drop that alpha too!

😍 3
🎉 2
lread21:06:29

Ok @olivergeorge I’ve taken some time to study your gist and can explain more what I mean. I’ll start with what I see as the essence of your gist and tweak it to something that is easier for me to play with in the REPL:

(require '[clojure.string :as string]
         '[rewrite-clj.zip :as z]);

(defn massage-field-spec
  [{:keys [type form/widget-type] :as field-spec}]
  (cond-> field-spec
          (and type (not widget-type))
          (assoc :form/widget-type type)))

(defn process-zloc
  [zloc]
  (-> zloc
      (z/up)
      (z/postwalk
       (fn select [zloc]
         (let [x (z/sexpr zloc)]
           (and (map? x)
                (:type x)
                (= (z/sexpr (z/left (z/up zloc))) :field-spec))))
       (fn visit [zloc]
         (z/edit zloc massage-field-spec)))))
You didn’t mention the shape of your data, but here’s a sample I came up with. If I undertand, I think the idea is if we have a :form/widget-type and a :type, we don’t need a change otherwise we want to add a :form/widget-type with the value of :type. Maybe your data looks like this, I dunno the exact shape, it is not that important for the purposes of this discussion.
(def s (string/join "\n" ["[{:field-spec"
                          "  {:something-a"
                          "   {:type :whatever"
                          "    :form/widget-type :all-good"
                          "    :a 1"
                          "    :b 2}}}"
                          " {:field-spec"
                          "  {:something-b"
                          "   {:type :whatever2"
                          "    :c 3"
                          "    :d 4}}}]"]))

;; let's see what that looks like printed out:
(println s)
;; stdout:
[{:field-spec
  {:something-a
   {:type :whatever
    :form/widget-type :all-good
    :a 1
    :b 2}}}
 {:field-spec
  {:something-b
   {:type :whatever2
    :c 3
    :d 4}}}]
So, if I get it, only the 2nd map’s content should update. Let’s see what happens with the current code:
(-> s
    z/of-string
    process-zloc
    z/root-string
    println)
;; stdout:
[{:field-spec
  {:something-a
   {:form/widget-type :all-good, :type :whatever, :b 2, :a 1}}}
 {:field-spec
  {:something-b
   {:form/widget-type :whatever2, :type :whatever2, :c 3, :d 4}}}]
As you’ve mentioned, we’ve lost existing whitespace (and even key value pair ordering). Let’s see what we can do about that.
(defn process-zloc2
  [zloc]
  (-> zloc
      (z/up)
      (z/postwalk
       (fn select [zloc]
         ;; here we avoid sexpr by using other zip fns
         (and (z/map? zloc)
              (z/get zloc :type)
              ;; added in check so we don't have to check in visitor
              (not (z/get zloc :form/widget-type))
              ;; use string just in case whatever is at up and left is not sexpr-able
              (= ":field-spec" (-> zloc z/up z/left z/string))))
       (fn visit [zloc]
         ;; here again we avoid sexpr and use zipper functions
         (let [type-node (-> zloc (z/get :type) z/node)]
           (z/assoc zloc :form/widget-type type-node))))))

(-> s
    z/of-string
    process-zloc2
    z/root-string
    println)
;; stdout:
[{:field-spec
  {:something-a
   {:type :whatever
    :form/widget-type :all-good
    :a 1
    :b 2}}}
 {:field-spec
  {:something-b
   {:type :whatever2
    :c 3
    :d 4 :form/widget-type :whatever2}}}]
That’s better, we’ve not altered the first map at all and preserved whitespace and ordering when adding a key value to the 2nd map. But, if we want, we can assert more control over how we insert that key value pair. Maybe we want to insert it as the first item in the map and preserve indentation:
(defn process-zloc3
  [zloc]
  (-> zloc
      (z/up)
      (z/postwalk
       ;; same a process-zloc2
       (fn select [zloc]
         ;; here we avoid sexpr by using other zip fns
         (and (z/map? zloc)
              (z/get zloc :type)
              ;; added in check so we don't have to check in visitor
              (not (z/get zloc :form/widget-type))
              ;; use string just in case whatever is at up and left is not sexpr-able
              (= ":field-spec" (-> zloc z/up z/left z/string))))
       (fn visit [zloc]
         ;; this time we use some lower level fns for more control
         (let [first-key-offset (-> zloc z/down z/node meta :col)
               type-value-node (-> zloc (z/get :type) z/node)]
           (-> zloc
               z/down
               (z/insert-left :form/widget-type)
               (z/insert-left type-value-node)
               (z/insert-newline-left)
               (z/insert-space-left (dec first-key-offset))))))))

(-> s
    z/of-string
    process-zloc3
    z/root-string
    println)
;; stdout:
[{:field-spec
  {:something-a
   {:type :whatever
    :form/widget-type :all-good
    :a 1
    :b 2}}}
 {:field-spec
  {:something-b
   {:form/widget-type :whatever2 
    :type :whatever2
    :c 3
    :d 4}}}]
Another thing to think about might be reader conditional discards, for example #_ :skipped. Rewrite-clj calls these uneval nodes, but does not skip them by default. Let me know if the above makes sense and is helpful.