Generate consistent JSON from XML for sometimes repeated elements with xmltodict

Question

For converting XML to JSON with python, several methods exist which usually boil down to the same principle. The most referred method today is xmltodict.

I'm trying to convert multiple xml documents to JSON, but run into issues if elements that are repeated in some documents, occur once: in those cases, not a list but an object is generated. This inconsistency causes issues when ingesting the JSON in for instance spark, which can't infer a schema for the json element anymore (is the element a list or a string?) and will cast it to a string. How can we best get around this? Ideally a parser would take a schema (XSD) and "learn" that elements can be unbounded and therefore should be a list, even if they are alone.

1. Is there a method to convert XML to JSON using an XSD?
2. Are there other workarounds for this possible? I'm thinking of putting every non-list item into a list for instance, but that will generate overhead.

Here's some code to display the issue

import xmltodict
from lxml import etree

xml1="""<?xml version="1.0" encoding="UTF-8"?>
<a xmlns="https://some.com/ns" xsi:schemaLocation="https://some.com/ns schema.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><b>x</b></a>
"""
xml2="""<?xml version="1.0" encoding="UTF-8"?>
<a xmlns="https://some.com/ns" xsi:schemaLocation="https://some.com/ns schema.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><b>x</b><b>y</b></a>
"""
xsd="""<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns="https://some.com/ns" targetNamespace="https://some.com/ns"  elementFormDefault="qualified" attributeFormDefault="qualified" version="2.4">
    <xs:element name="a" type="aType"/>
    <xs:complexType name="aType">
        <xs:sequence>
            <xs:element name="b" type="xs:string" maxOccurs="unbounded"/>
        </xs:sequence>
    </xs:complexType>
</xs:schema>    
"""

print(json.dumps(xmltodict.parse(xml1),indent=True))
print(json.dumps(xmltodict.parse(xml2),indent=True))

xmlschema_doc = etree.fromstring(xsd.encode())
xmlschema = etree.XMLSchema(xmlschema_doc)


xml1_doc = etree.fromstring(xml1.encode())
print(etree.tostring(xml1_doc))
result1 = xmlschema.validate(xml1_doc)
print("validation xml1: ")
print(result1)

xml2_doc = etree.fromstring(xml2.encode())
result2 = xmlschema.validate(xml2_doc)
print("validation xml2: ")
print(result2)

which outputs for xml1 as JSON:

{
 "a": {
  "@xmlns": "https://some.com/ns",
  "@xsi:schemaLocation": "https://some.com/ns schema.xsd",
  "@xmlns:xsi": "http://www.w3.org/2001/XMLSchema-instance",
  "b": "x"
 }
}

and for xml2

{
 "a": {
  "@xmlns": "https://some.com/ns",
  "@xsi:schemaLocation": "https://some.com/ns schema.xsd",
  "@xmlns:xsi": "http://www.w3.org/2001/XMLSchema-instance",
  "b": [
   "x",
   "y"
  ]
 }
}

a workaround may be:

def nestit(path, key, value):
    # don't nest lists. also dont nest attributes.
    if type(value) == list or key[:1]=='@':
        return key, value
    else:
        return key, [value]

print(json.dumps(xmltodict.parse(xml1,postprocessor=nestit),indent=True))

which generates:

{
 "a": [
  {
   "@xmlns": "https://some.com/ns",
   "@xsi:schemaLocation": "https://some.com/ns schema.xsd",
   "@xmlns:xsi": "http://www.w3.org/2001/XMLSchema-instance",
   "b": [
    "x"
   ]
  }
 ]
}

but would be less efficient in a very complex xml structure for elements that never are repeated.

Answer 1

Is there a method to convert XML to JSON using an XSD?

xmlschema does exactly that:

import xmlschema

schema = xmlschema.XMLSchema(xsd)
print(json.dumps(xmlschema.to_dict(xml1,schema=schema,preserve_root=True),indent=True))

returns:

{
 "a": {
  "@xmlns": "https://some.com/ns",
  "@xmlns:xsi": "http://www.w3.org/2001/XMLSchema-instance",
  "@xsi:schemaLocation": "https://some.com/ns schema.xsd",
  "b": [
   "x"
  ]
 }
}