Requesting sagemaker endpoints gives "Need to pass custom_attributes='accept_eula=true' as part of header" error

Question

I am trying to test https://aws.amazon.com/blogs/machine-learning/llama-2-foundation-models-from-meta-are-now-available-in-amazon-sagemaker-jumpstart/ llama2 model on sagemaker studio.

I am able to run the code in sagemkaer notebook and when I run it it gives me endpoints a well.

So I have endpoint link. Usually when I use this link with AWS credentials on my postman, it gives me response from the model.

But for Llama 2 model when I am trying to use the endpoint in postman I am getting this error

> {
>     "ErrorCode": "CLIENT_ERROR_FROM_MODEL",
>     "LogStreamArn": "arn:aws:logs:us-east-1:847137928610:log-group:/aws/sagemaker/Endpoints/meta-textgeneration-llama-2-7b-f-2023-07-26-06-06-21-772",
>     "Message": "Received client error (424) from primary with message \"{\n  \"code\":424,\n  \"message\":\"prediction failure\",\n 
> \"error\":\"Need to pass custom_attributes='accept_eula=true' as part
> of header. This means you have read and accept the
> end-user-license-agreement (EULA) of the model. EULA can be found in
> model card description or from
> https://ai.meta.com/resources/models-and-libraries/llama-downloads/.\"\n}\".
> See
> https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/sagemaker/Endpoints/meta-textgeneration-llama-2-7b-f-2023-07-26-06-06-21-772
> for more information.",
>     "OriginalMessage": "{\n  \"code\":424,\n  \"message\":\"prediction failure\",\n  \"error\":\"Need to pass
> custom_attributes='accept_eula=true' as part of header. This means you
> have read and accept the end-user-license-agreement (EULA) of the
> model. EULA can be found in model card description or from
> https://ai.meta.com/resources/models-and-libraries/llama-downloads/.\"\n}",
>     "OriginalStatusCode": 424 }

I also tried to invote sagemaker endpoint using Lambda and API Gateways following https://medium.com/@woyera/how-to-use-llama-2-with-an-api-on-aws-to-power-your-ai-apps-3e5f93314b54

But there also I am getting

{
    "message": "Internal Server Error"
}

Any suggestion or help recommended here?

Answer 1

Add it as one of headers to your request, since the blog you linked mentions it:

Note that by default, accept_eula is set to false. You need to set accept_eula=true to invoke the endpoint successfully. By doing so, you accept the user license agreement and acceptable use policy as mentioned earlier. You can also download the license agreement.

Answer 2

You have to send custom_attrtibutes with "accept_eula=true" in the request headers as follows when you query the deployed model endpoint or the predictor.

predictor.predict(payload, custom_attributes="accept_eula=true")

You can find sample notebook here.

In case you want to inovke endpoint via boto3, you can do so as detailed in the notebook here or as follows:

import boto3
sagemaker_runtime = boto3.client('sagemaker-runtime')
response_stream = sagemaker_runtime.invoke_endpoint_with_response_stream(
        EndpointName=endpoint_name,
        Body=json.dumps(payload), 
        ContentType="application/json",
        CustomAttributes='accept_eula=true'
    )

Answer 3

You have to add the following custom attribute in the POST request header:

key: X-Amzn-SageMaker-Custom-Attributes

value: accept_eula=true