It seems the be the compilation that takes time.

I spent some time narrowing it down to test-m2being slow compared to the others below

;; fast
(defn test-m1 [x]
    (m/rewrite x
               (m/map-of _
                         {:id !id
                          :m1 [!m1s ...]
                          :m2 [!m2s ...]
                          :m3 [!m3s ...]
                          :m4 [!m4s ...]})
               [{:id !id
                 } ...]))
;; slow
(defn test-m2 [x]
    (m/rewrite x
               (m/map-of _
                         {:id !id
                          :m1 [[!m1-ids !m1s] ...]
                          :m2 [[!m2-ids !m2s] ...]
                          :m3 [[!m3-ids !m3s] ...]
                          :m4 [[!m4-ids !m4s] ...]
                          })
               [{:id !id
                 } ...]))
;; fast
(defn test-m3 [x]
  (m/rewrite x
    {:id !id
     :m1 [[!m1-ids !m1s] ...]
     :m2 [[!m2-ids !m2s] ...]
     :m3 [[!m3-ids !m3s] ...]
     :m4 [[!m4-ids !m4s] ...]
     }
    [{:id !id
      } ...]))

test-m1 is faster than test-m2 because it has fewer memory variables. Repeating the pattern upto m8 in test-m1 will make it perform as test-m2. test-m3 does not show the same behaviour. So my working conclusion is that it is related to map-of and the amount of memory variables

I guess this could be a similar issue as #234 https://github.com/noprompt/meander/issues/234#issue-1292196144 I will leave it at that for now and try a different solution.

Hi Lidor, are you able to provide a sample of your input data and expected output data? Also, note that it is only safe to use ?product in the m/pred function like that for small maps e.g. PersistentArrayMap.

well it will be pretty hard because the input is quite dirty, its is basically a parsed csv (vector of vectors) with 54 columns and the first vector being the headers, so something like this:

{:products [["name" "some" "unimportant" "values" ... "description" ... "price" ... "media" ... "tags" ...]
            ["some-name" "bla"  "bla" "bla" ... "some long description" ... "42" ... "" ... "238;239;756;785;1111;" ...]
            ...]
 :tags     [["Category ID" ... "Category Name" ... "Parent" ...]
            ["238" ... "catcat" ... "catcat's mom"]
            ["239" ... "catcat's mom" ... ""]
            ...]}

[{"name" "some-name"
 "description" "some long description"
 "price" 42
 "media" {"data" {"src" ""}}
 "product_tags" [{"name" "catcat" "parent" "catcat's mom"}
                 {"name" "catcat's mom" "parent" ""}
                 ...]}
 ...
]

I could clean the input before inputting into meander but I was hoping to be able to use meander for that as well...

For the CSV, personally, I would zipmap the fields or pluck them out with nth, etc. in a preprocessing step and then run them through meander.

I would also index the tags as well. You can then do the joins much more easily (and more legibly).

You could also drop the first rows (headers) from each of the two data sets.

Then you could

{:products (m/scan {:as ?product ,,,})
 :tags (m/scan {:name (m/pred #(contains ,,,) ,,,})}

Great I was wondering where I should do the cleaning, now I have an answer 😁

Thank you!

LMK if you need help with next steps. 🙂

So I came up with this:

(m/search data

            {:products (m/scan {"Product Name" ?product-name
                                "Meta Tag Description" ?description
                                "Price" ?price
                                "Image(Main image)" ?media
                                "Length" ?length
                                "Width" ?width
                                "height" ?height
                                "Weight" ?weight
                                "Manufacturer" ?manufacturer
                                "Product Tags" ?product-tags
                                "Categories id" ?categories-id})

             :categories (m/scan {"Category ID" (m/pred #(some #{%} (split ?categories-id ";")))
                                  "Category Name" ?category-name
                                  "Parent" ?parent})}


            {"name" ?product-name
             "description" ?description
             "price" ?price
             "media" {"data" {"src" ?media}}
             "categories-id" (split ?categories-id ";")
             "category-name" [?category-name]})

[{"name" "some-name"
 "description" "some long description"
 "price" 42
 "media" {"data" {"src" ""}}
 "product_tags" [{"name" "catcat" "parent" "catcat's mom"}
                 {"name" "catcat's mom" "parent" ""}
                 ...]}
 ...
]

[{"name" "some-name"
 "description" "some long description"
 "price" 42
 "media" {"data" {"src" ""}}
 "product_tags" [{"name" "catcat" "parent" "catcat's mom"}]}
{"name" "some-name"
 "description" "some long description"
 "price" 42
 "media" {"data" {"src" ""}}
 "product_tags" [{"name" "catcat's mom" "parent" ""}]}
 ...
]

I figured what I'm looking for is a join + group by

I found a previous answer that suggests to do the group-by out side of meander. can't wait to see how far will you go with meander, for now I'm happy with the current solution, thank you! P.S I invested time learning meander as I believe in it as a general declarative solution for the complicated data transformations in our company, so... rooting for ya!