AWS CloudWatch Evidently experiment does not record events for custom metrics

Question

I have configured an Evidently project, with one feature and prepared an experiment, where I defined one custom metric. The problem is quite simple: I just can't get Evidently to record any events. I just can't figure out, what's wrong.

By this moment I have simplified the troubleshooting and I am using command line to evaluate feature and send event to Evidently:

aws evidently evaluate-feature \
    --entity-id "12.345.678.90" \
    --feature "SolutionInterest" \
    --project "myprojectname"

aws evidently put-project-events \
    --project "myprojectname"\
    --events '[{"data" : "{\"value\":\"1\", \"userIP\":\"12.345.678.90\"}", "timestamp":"1693156893", "type":"aws.evidently.custom" }]'

the result seems to be successful:

{
    "details": "{\"experiment\":\"Test\",\"treatment\":\"Variation2\"}",
    "reason": "EXPERIMENT_RULE_MATCH",
    "value": {
        "boolValue": true
    },
    "variation": "Variation2"
}
{
    "eventResults": [
        {
            "eventId": "fddbaf08-ad49-40d1-bcaa-aa49f44ea5e1"
        }
    ],
    "failedEventCount": 0
}

But still, my experiment can't seem to register any events (event count is still zero for both variations of my feature and is not increasing with each script run; also, not increasing after a while (https://i.stack.imgur.com/WFoJP.png)).

In the experiment configuration the metric rule is looking like this:

{
  "entityIdKey": "userIP",
  "valueKey": "value",
  "eventPattern": {
    "userIP": [
      {
        "exists": true
      }
    ],
    "value": [
      {
        "exists": true
      }
    ]
  }
}

I can see that the metrics are coming in to "CloudWatch" -> "Metrics" -> "All metrics", but they still do not show up in Evidently's experiment.

My code in the React app looks like this:

    import { EvidentlyClient, EvaluateFeatureCommand, PutProjectEventsCommand } from "@aws-sdk/client-evidently";
    
    const eviclient = new EvidentlyClient({ 
        region : 'eu-north-1',
        endpoint: "https://evidently.eu-north-1.amazonaws.com",
        credentials : {
            accessKeyId     : 'AKIA2NxxxxxxxxW6P3UL', 
            secretAccessKey : 'fo5hMrufII7eHZxxxxxxxxS/5ZHnXqIFXNXGHt2e'
        }
    });
    
    async function evaluate() {
        const command1 = new EvaluateFeatureCommand({ 
            project: "myprojectname", feature: "SolutionInterest",  entityId: "12.345.678.90"
        });
        const command2 = new PutProjectEventsCommand({
            project: "myprojectname", events: [
              { timestamp: new Date(), type: "aws.evidently.custom", data: JSON.stringify({ "userIP": "12.345.678.90", "value": "1" }) }
            ]
        });
        let response1 =  await eviclient.send(command1);
        console.log('response1: ' + JSON.stringify(response1));
        let response2 =  await eviclient.send(command2);
        console.log('response2: ' + JSON.stringify(response2));
    }
    
    function App() {
        evaluate();
        return null;
    }

What I get in the console is the following:

[Log] response1: {"$metadata":{"httpStatusCode":200,"requestId":"f9f35837-4796-4456-925f-97be4afaa240","attempts":1,"totalRetryDelay":0},"details":"{\"experiment\":\"Test4\",\"treatment\":\"Variation2\"}","reason":"EXPERIMENT_RULE_MATCH","value":{"boolValue":true},"variation":"Variation2"}

[Log] response2: {"$metadata":{"httpStatusCode":200,"requestId":"fb29f316-85bc-4416-9dcf-920ba98fae57","attempts":1,"totalRetryDelay":0},"eventResults":[{"errorCode":null,"errorMessage":null,"eventId":"434115a0-0e18-4cae-a2fe-c53f8dca117d"}],"failedEventCount":0}

So far I tried the following:

• to create a project and an experiment in another region - still same issue, so it is not region specific.
• to create an experiment with the same user identity, that is being used with client configuration - still same problem.
• to create a metric with and without rule pattern - no difference.
• with PutProjectEventsCommand I tried to pass 'data' as both JSON and string - still, nothing changed

Does anyone have a clue, what could be wrong here or if there are any pre-requisite configurations that I have overlooked?

Really appreciate your time and advise.

Answer 1

I just came across to your issue and found I have exactly the same problem. Rechecked I checked every points that was suggested on the replies and everything was fine. After several trial and debugging still no luck. I got really frustrated after being stuck with it for long.

However I also added metrics with the Evidently launch, and it shows the chart on the launch>monitoring section. But with the same exact metric, it doesn't show anything on the experiment section.

I was also able to see the metrics on the Cloudwatch>metrics>all metrics page. But somehow on the experiment section it's not recorded. Anyway one workaround can be that from the metrics section you can actually select the metrics and create chats/tables the way you want. It gives you a lot of options and flexibility to create your chart/tables the way you want and then you can add it to an AWS dashboard from the actions>add to dashboard button on the top. This way you can create different chats or tables based on your requirements and add it to the dashboard so that you can monitor it from a single page. You can also create charts/tables with the assignment data for variations that was assigned by evidently. Hope it helps as a workaround and let me know if you have any question or comment.

Answer 2

So I am not sure if this was the case with you, but for me this ended being the reason the events weren't being recorded. They are now.

For the metrics to be recorded, the put_project_events call needs to

1. Be called within 1 hour of the evaluate_feature call
2. have a matching entityId

In evaluate_feature, the boto3 library was enforcing the entityId and telling me it needed to be a string.

In the put_project_events call the entityId is passed as part of the data param as a json string so the boto3 library is not checking that value.

In one case I had a string '6552' and in the other an int 6552.

So the metrics were not being mapped to the experiment.

Once I made these match, everything worked as expected.

Another thing to check:

Though I think you've covered this. But still, typos are possible :D

The data param needs to match the expectations of the metric.

e.g. for the following metric definition:

- name: retryCount
  rule:
    entityIdKey: 'details.entityId' 
    valueKey: 'details.retries'
    eventPattern: '' 
  goal: decrease
  units: Retries

the data sent to the push_project_events function should look like:

{
  "details": {
    "entityId": 'same-entity-id' 
    "retries": 5 # the amount of retries that happened 
  }
}