@U06B8J0AJ, is the goal of the output EDN to contain all of the XML content with a different shape? Or is the edn going to be more of a flat structure?

The EDN is going to be heavily flattened, shrunk, and restructured.

Large chunks ignored or thrown away.

I see, you could give a go at [clojure.xml :refer [parse]] to parse it, and then use xml-seq to pick the bit you are interested in. A few examples here https://clojuredocs.org/clojure.core/xml-seq

With zippers or postwalk, I guess you'd probably have to write more complicated code.

The first question is: can you fit that DOM into RAM. I've seen 2 MB xmls expand to over 100 MB DOM trees after java's xml parser was done.

So far, I’ve parsed the XML into an intermediate format. For example;

<journal-meta>
<journal-id journal-id-type="publisher-id">IBM</journal-id>
<journal-id journal-id-type="hwp">spibm</journal-id>
<journal-title-group><journal-title>International Bulletin of Mission Research</journal-title></journal-title-group>
<issn pub-type="ppub">2396-9393</issn>
<issn pub-type="epub">2396-9407</issn>
<publisher>
<publisher-name>SAGE Publications</publisher-name>
<publisher-loc>Sage UK: London, England</publisher-loc>
</publisher>
</journal-meta>

[[:journal-meta
  [[:journal-id "IBM" [[:journal-id-type "publisher-id"]]]
   [:journal-id "spibm" [[:journal-id-type "hwp"]]]
   [:journal-title-group
    [:journal-title "International Bulletin of Mission Research"]]
   [:issn "2396-9393" [[:pub-type "ppub"]]]
   [:issn "2396-9407" [[:pub-type "epub"]]]
   [:publisher
    [[:publisher-name "SAGE Publications"]
     [:publisher-loc "Sage UK: London, England"]]]]]]

Given this, I wrote a function called cherry-pick, which takes a set of keywords, traverses the entire structure, and returns any hits for those keywords. For example, running (cherry-pick #{:issn}) Returns

[[:issn "2396-9393" [[:pub-type "ppub"]]]
 [:issn "2396-9407" [[:pub-type "epub"]]]]

So the general strategy is, flatten the XML a bit, and have a way of tearing out pieces of interest from it. Given the cherry-picked items, do some more context sensitive conversions. But I’m not at all certain that this is a good way to go about it.

A comparison with xml-seq

(->> xml 
 .getBytes 
 io/input-stream 
 xml/parse 
 xml-seq 
 (filter (comp #(= :issn %) :tag)))

;;({:tag :issn, :attrs {:pub-type "ppub"}, :content ["2396-9393"]} 
;; {:tag :issn, :attrs {:pub-type "epub"}, :content ["2396-9407"]})

And yes, you should pay attention at the size of the input XML. If you get OOMs you could go data.xml lib (which is lazy)

Given that I have to search the entire structure one way or another, won’t the lazy version become reified anyway?

well if you pipe the xml through one side and get the processed version on the other, the full XML doesn't need to stay in memory in full

So, filter, in the example above, won’t cause OOM?

only if the parsing is done with data.xml instead of the core xml namespace

Gotcha, thank you

np, you might need to tweak the filter if you use data.xml, but the process is more or less the same. It looks to me you can avoid the intermediate step.

Perhaps, but I still need a version of filter that can act on nested data structures, right?

Or does xml-seq flatten the data structure?

xml-seq is what is flattening the structure.

Alright, that’s pretty cool. I’m starting to understand now.

Yeah, it looks like I can throw out the intermediate step.

Ah yes, xml-seq definitely wins in terms of ergonomics. Thank you.

I'd use xpath, tbh

Depending on if your blobs fit in memory

The one’s I’m looking at right now do, but there’’s no guarantee about future ones.

@U050ECB92 What lib do you use for xpath?

raw interop

I can share you some code

basically some helpers to execute an XPath and transform the stuff back to maps and vectors. You might want to try clj-xpath, looks decent

I’d love an interop example, and I’ll have a look at clj-xpath as well.

@U06B8J0AJ How about XSLT ? http://cocoon.apache.org/2.2/1290_1_1.html

Did you mean symbol instead of keyword? Also, if your dispatch is that, then it will always dispatch to the same value. constantly returns a function that always returns its argument (it doesn't call it). You probably want something like (defmulti my-fn <name-of-defstate>).

Hi oscar, I meant name-of-defstate returns a keyword from its :start, e.g. :dev to indicate dev mode globally in my application. I had hoped (defmulti my-fn (constantly mode-defstate)) would dispatch to the (defmethod my-fn :dev ...).

I see. You probably want (defmulti my-fn (fn [& _] mode-defstate)) then.

constantly is called before (mount/start) so it's getting passed the wrong value as its parameter.

Putting mode-defstate in a fn ensures that it doesn't get resolved until call time.

thank you; I guess I assumed that constantly was doing the same thing; obviously not