Cross account SNS/SQS subscriptions where both accounts have distinct customer managed KMS keys

Question

I have a set up that involves two accounts A, and B
In account A I have a customer managed key_A which allows all resources and uses in Account B to access
In account B I have a distinct KMS key_b and an SQS of which I'm trying to subscribe to the SNS topic in A.

Most of the documentation I can find seems to assume that the KMS keys will be the same, and in this set-up they won't be. I've tried granting the following

"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"

Permissions to the SNS topic for the key in B but this doesn't seem to work. The subscription itself seems to show itself as confirmed but no messages are received when sent.

{
    "Attributes": {
        "SubscriptionPrincipal": "arn:aws:iam::{num}:role/{rolename}",
        "Owner": "{number}",
        "RawMessageDelivery": "true",
        "TopicArn": "arn:aws:sns:{topic I expect}",
        "Endpoint": "arn:aws:sqs:{sqs I expect}",
        "Protocol": "sqs",
        "PendingConfirmation": "false",
        "ConfirmationWasAuthenticated": "true",
        "SubscriptionArn": "arn:aws:sns:subscription"
    }
}

I guess my overall confusion is in what resource needs to encrypt / decrypt to where. I'm assuming since the SQS queue is recieving the message the KMS key permissions need to be granted to the cross account SNS topic?

I've tried giving both resources access to both KMS keys, and this doesn't seem to have worked.