How to remove a node from a dict using jsonpath-ng?

Question

In Python I have a list of dictionaries and I want to remove a given node from each dictionary in the list. I don't know anything about those dictionaries except they all have the same (unknown) schema. The node to be removed may be anywhere in the dictionaries and it is specified by a JSONPath expression.

Example:

Input data:

[
  { "top": { "lower": 1, "other": 1 } },
  { "top": { "lower": 2, "other": 4 } },
  { "top": { "lower": 3, "other": 9 } }
]

Node to be removed: $.*.top.lower

Expected result:

[
  { "top": { "other": 1 } },
  { "top": { "other": 4 } },
  { "top": { "other": 9 } }
]

Using the jsonpath library my first attempt was this:

from jsonpath import JSONPath

def remove_node_from_dict(data, node):
    node_key = JSONPath(node).segments.pop()
    for record in data:
        del record[node_key]

but this works only if the node to remove is at the top level of the dictionaries. Researching for solutions I came across the jsonpath-ng library which claims to have "the ability to update or remove nodes in the tree". However, I couldn't find any documentation on this - how is it done?

EDIT:

Based on this answer to a related question I found a solution that works at least for simple paths (no filters etc.) using plain Python (not the jsonpath-ng library). Which would be sufficient for my use case. I would still like to learn how to do it with jsonpath-ng in a more generic way.

Answer 1

The jsonpath-ng library, in fact, allows to do nodes removal using the .filter() method. They've added examples to the documentation only recently, thanks to reported issue #44.

I had to change your JSONPath expression to $.[*].top.lower:

from jsonpath_ng import parse


test_data = [
    { "top": { "lower": 1, "other": 1 } },
    { "top": { "lower": 2, "other": 4 } },
    { "top": { "lower": 3, "other": 9 } }
]

jsonpath_expr = parse("$.[*].top.lower")
jsonpath_expr.filter(lambda d: True, test_data)
print(test_data)

And I've got the next output:

[{'top': {'other': 1}}, {'top': {'other': 4}}, {'top': {'other': 9}}]

Answer 2

If you know the schema is fixed, you can simply remove the key like this

l = [
  { "top": { "lower": 1, "other": 1 } },
  { "top": { "lower": 2, "other": 4 } },
  { "top": { "lower": 3, "other": 9 } }
]

for d in l:
    del d["top"]["lower"]

Answer 3

Here's a naive solution which I've used in the past:

import copy
import jsonpath_ng.ext as jp

def remove_matched_element(path, spec):
    _new_spec = copy.deepcopy(spec)
    jep = jp.parse(path)
    for match in jep.find(spec):
        _t_path = "$"
        spec_path = _new_spec
        spec_path_parent = None
        for pe in str(match.full_path).split("."):
            if _t_path != "$" and type(jp.parse(_t_path).find(_new_spec)[0].value) == list:
                _t_path = f"{_t_path}{'.'}{pe}"
                _idx = int(pe.replace("[", "").replace("]", ""))
                spec_path_parent = spec_path
                spec_path = spec_path[_idx]
            elif _t_path != "$" and type(jp.parse(_t_path).find(_new_spec)[0].value) == dict and pe == "[0]":
                keyp = list(jp.parse(_t_path).find(_new_spec)[0].value.keys())[0]
                _idx = keyp
                _t_path = f"{_t_path}.{keyp}"
                spec_path_parent = spec_path
                spec_path = spec_path[keyp]
            else:
                if type(spec_path) == list:
                    _idx = int(pe.replace("[", "").replace("]", ""))
                    _t_path = f"{_t_path}[{_idx}]"
                else:
                    _idx = pe
                    _t_path = f"{_t_path}{'.'}{pe}"
                spec_path_parent = spec_path
                spec_path = spec_path[_idx]
        spec_path_parent.pop(_idx)
    return _new_spec


def test_soc_sol():
    spec = [
        {"top": {"lower": 1, "other": 1}},
        {"top": {"lower": 2, "other": 4}},
        {"top": {"lower": 3, "other": 9}}
    ]

    print(
        yaml.safe_dump(remove_matched_element("$..lower", spec)))

The above code results in:

[
    {
        "top": {
            "other": 1
        }
    },
    {
        "top": {
            "other": 4
        }
    },
    {
        "top": {
            "other": 9
        }
    }
]