Can't access deeply nested XML with clojure.data.zip.xml

Question

(def testxml2
"<top>
    <group>
        <group>
            <item>
                <number>1</number>
            </item>
            <item>
                <number>2</number>
            </item>
            <item>
                <number>3</number>
            </item>
        </group>
        <item>
            <number>0</number>
        </item>
    </group>
</top>")

(def txml2 (zip-str testxml2))

(defn deep-items [x]
    (zip-xml/xml-> x
        :top
        :group
        :group
        :item))

(count (deep-items txml2))
;; 1

(zip-xml/text (first (deep-items txml2)))
;; "0"

I'm trying to get the value of the inner :group, but it seems to be getting caught on the outside one. It seems to be ignoring the second :group.

The actual XML I'm trying to parse has a repeated nested <TheirTag><TheirTag>Foo</TheirTag></TheirTag> pattern going on and I need to access each Foo individually. The XML is from a third party so I can't just restructure the XML to avoid this.

Answer 1

You can solve this using the Tupelo Forest library to process tree-like data structures. Besides explicit searching, it can also use wildcards like zsh. Documentation is ongoing, but this will give you a taste of what you can do:

(dotest
  (with-forest (new-forest)
    (let [xml-str         "<top>
                              <group>
                                  <group>
                                      <item>
                                          <number>1</number>
                                      </item>
                                      <item>
                                          <number>2</number>
                                      </item>
                                      <item>
                                          <number>3</number>
                                      </item>
                                  </group>
                                  <item>
                                      <number>0</number>
                                  </item>
                              </group>
                          </top>"

          enlive-tree     (->> xml-str
                            java.io.StringReader.
                            en-html/xml-resource
                            only)
          root-hid        (add-tree-enlive enlive-tree)

          ; Removing whitespace nodes is optional; just done to keep things neat
          blank-leaf-hid? (fn fn-blank-leaf-hid?  ; whitespace pred fn
                            [hid]
                            (let [node (hid->node hid)]
                              (and (contains-key? node :value)
                                (ts/whitespace? (grab :value node)))))
          blank-leaf-hids (keep-if blank-leaf-hid? (all-leaf-hids)) ; find whitespace nodes
          >>              (apply remove-hid blank-leaf-hids) ; delete whitespace nodes found

The part you really care about is here. There are 2 ways to search for nested nodes.

1. The first method specifies an explicit path from the root

2. The second uses a wildcard :** like zsh, which matches zero or more directories.

         ; Can search for inner `div` 2 ways
         result-1        (find-paths root-hid [:top :group :group]) ; explicit path from root
         result-2        (find-paths root-hid [:** :group :item :number]) ; wildcard path that ends in :number
         ]
   

   For cast (1), we see we found only items 1, 2, and 3:

     ; Here we see only the double-nested items 1, 2, 3
     (is= (spyx-pretty (format-paths result-1))
       [[{:tag :top}
         [{:tag :group}
          [{:tag :group}
           [{:tag :item} [{:tag :number, :value "1"}]]
           [{:tag :item} [{:tag :number, :value "2"}]]
           [{:tag :item} [{:tag :number, :value "3"}]]]]]] )
   

For case (2), we found not only the doubly-nested items, but also the singly-nested item 0:

      ; Here we see both the double-nested items & the single-nested item 0
      (is= (spyx-pretty (format-paths result-2))
        [[{:tag :top}
          [{:tag :group} [{:tag :item} [{:tag :number, :value "0"}]]]]
         [{:tag :top}
          [{:tag :group}
           [{:tag :group} [{:tag :item} [{:tag :number, :value "1"}]]]]]
         [{:tag :top}
          [{:tag :group}
           [{:tag :group} [{:tag :item} [{:tag :number, :value "2"}]]]]]
         [{:tag :top}
          [{:tag :group}
           [{:tag :group} [{:tag :item} [{:tag :number, :value "3"}]]]]]])

      )))

You didn't specify what downstream processing you needed. Tupelo.Forest is able to convert output into both hiccup and enlive formats, plus it's own hiccup-inspired bush format and an enlive-inspired tree format.

Answer 2

The reason for the bug is here. For the short version: version 0.1.2 is slightly broken in this respect, that a sub-entry with the same name cannot be selected via the tag= function (which underlies the :myTag style selectors. This is due to a regression from 0.1.1 to 0.1.2 (thanks @bpeter and @shilder). The workaround is to make a function tag= in some util namespace and use it directly until the regression is fixed.

;; util.clj
(defn tag=
  "This is a workaround to a regression in 0.1.2. Fixed in upcoming 1.2.0

  Returns a query predicate that matches a node when its is a tag
  named tagname."
  [tagname]
  (fn [loc]
    (filter #(and (zip/branch? %) (= tagname (:tag (zip/node %))))
       (zf/children-auto loc))))

;; project.somefile.clj
(ns project.somefile
  (:require [project.util :as u]))


(defn deep-items [x]
    (zip-xml/xml-> x
        :top
        (u/tag= :group)
        (u/tag= :group)
        :item))