Why does my AWS Boto3 API request asking to create some Route53 records silently fail?

Question

I have some code that has always just worked. It uses the AWS API to request information about some ECS containers, then queries Route53 to get information about existing records, and then finally makes a Boto3 Route53 call to update a DNS zone to match the ECS information.

This code has worked in the past, but now I'm tracking down the source of some bad behavior, and the above logic seems to be the problem. When I step over the Boto3 call to change_resource_record_sets, the call succeeds. It doesn't throw an exception, and its HTTP Response Code is 200. However, the changes I've asked to have applied to my DNS zone do not occur. From what I can tell, the call didn't do anything.

I'm wondering if anyone can tell me why I'm getting this bad behavior. Why is the code below silently having no affect on my environment:

try:
    r = get_primary_client('route53').change_resource_record_sets(
        HostedZoneId=dns_zone['Id'],
        ChangeBatch={'Changes': changes}
    )
    processing_info['r'] = copy.deepcopy(r)
    assert r['ResponseMetadata']['HTTPStatusCode'] == 200
except Exception as ex:
    return str(ex)

The value of dns_zone dumped as JSON is:

{
      "CallerReference": "RISWorkflow-3bdd50ce72fe59acd56ed296752aab38",
      "Config": {
        "Comment": "HostedZone created by Route53 Registrar",
        "PrivateZone": false
      },
      "Id": "/hostedzone/Z30M8MWTQXQ2O2",
      "Name": "ourdomain.com.",
      "ResourceRecordSetCount": 54
}

The value of the changes param, dumped as JSON, is:

  {
    "Action": "CREATE",
    "ResourceRecordSet": {
      "Name": "appserver2h.devqa.ourdomain.com",
      "ResourceRecords": [
        {
          "Value": "10.13.11.98"
        }
      ],
      "TTL": 60,
      "Type": "A"
    }
  },
  {
    "Action": "CREATE",
    "ResourceRecordSet": {
      "Name": "grinder2h.devqa.ourdomain.com",
      "ResourceRecords": [
        {
          "Value": "10.13.12.172"
        }
      ],
      "TTL": 60,
      "Type": "A"
    }
  }
],

This code skips the except clause (doesn't throw an exception) and the assert statement succeeds (the request returns a response code of 200). And yet, nothing changes in my DNS zone. In this case, the following names do not have records in my DNS zone, as you'd expect after this call:

grinder2h.devqa.ourdomain.com
appserver2h.devqa.ourdomain.com

Here is a JSON dump of the what the call returns:

{
"ChangeInfo": {
  "Id": "/change/C102227247EPWZ3JJQCS",
  "Status": "PENDING",
  "SubmittedAt": "Tue, 27 Feb 2024 01:13:50 GMT"
},
"ResponseMetadata": {
  "HTTPHeaders": {
    "content-length": "282",
    "content-type": "text/xml",
    "date": "Tue, 27 Feb 2024 01:13:49 GMT",
    "x-amzn-requestid": "735f38f8-bd9d-440b-b29d-b68ebfd0f45e"
  },
  "HTTPStatusCode": 200,
  "RequestId": "735f38f8-bd9d-440b-b29d-b68ebfd0f45e",
  "RetryAttempts": 0
}

},

I did notice that the status returned is "Pending". I had to read up on what this is about. Apparently the changes are made asynchronously. To get the status of my call, I do this with the AWS CLI:

> aws route53 get-change --id "/change/C06636843RDY913EELESJ"
{
    "ChangeInfo": {
        "Id": "/change/C06636843RDY913EELESJ",
        "Status": "INSYNC",
        "SubmittedAt": "2024-02-27T00:22:50.985000+00:00"
    }
}

A status of "INSYNC" apparently means that the requested changes have been made.

Huh? I've been banging my head and staring at this for hours. I can't figure out what's going on. Anyone?