OpenSearch pipeline to convert string to json

29 Views Asked by At

I am trying to accomplish a migration from MySQL to OpenSearch. Because, it is a kind of migration from relational data to NoSQL, I firstly created view to combine data to a json string (DMS is not supporting json migration).

After I run the migration from DMS, here is the result of a data from the index:

"hits": [
  {
    "_index": “orders”,
    "_id": "096D70F8D93531CB03900EDED9FF0E32625C4E25A349E4BC3F8395C7B40008B2",
    "_score": 1,
    "_source": {
      "orderId": “12345678”,
      “json_str: """{"store": “IKEA”, "isTest": 0, "orderId": "12345678", "orderLines": [{"id": "d743142d-c106-11ee-afdf-06d5a72a9bcf", "ean": “1111111”, "vat": 0.25, "quantity": 1, "listPrice": 359.95, "lineNumber": 1, "productName": “Television”}], “orderDate": "2024-01-16 09:31:53.000000", "customerInformation": {"email": “aaa”@bbb.com, "customerId": “999999”, "isEmployee": 0}, "shippingInformation": {"parcelLocker": “Migros”, "shippingAddress": {"city": “ISTANBUL”, "state": null, "country": “TR”, "zipcode": “34000”, "houseNumber": "11"}}}"""
    }
  } ]

Now, I need to convert this json string(json_str) to json data as format of the OpenSearch like this:

"hits": [
  {
    "_index": "orders",
    "_id": "12345678",
    "_score": 1,
    "_source": {
      "orderId": "12345678",
      “orderDate": "2024-01-16 09:31:53.000000",
      "store": "IKEA",
      "shippingInformation": {
        "shippingAddress": {
          "city": “ISTANBUL”,
          "country": “TR”,
          "houseNumber": “11”,
          "phoneNumber": "12345678",
          "zipcode": “34000”
        },
        "parcelLocker": "Migros"
      },
      "customerInformation": {
        "email": "aaa”@bbb.com",
        "customerId": "999999",
        "isEmployee": false
      },
      "orderLines": [
        {
          "id": "d743142d-c106-11ee-afdf-06d5a72a9bcf",
          "ean": "1111111",
          "lineNumber": 1,
          "productName": "Television",
          "quantity": 1,
          "listPrice": 359.95,
          "vat": 0.25,
          "brand": "JJXX"
        }
      ]
    }
  }]

How can I create a OpenSearch pipeline to accomplish this? Is it possible to run it after migration (post processor)?

I tried "json" processor, but couldn't success.

0

There are 0 best solutions below