How to keep the conversation going using AWS bedrock using model Claude v2 in Nest TS?

Question

I have been working on AWS Bedrock using Claudev2 model.

I have successfully made my first prompt via the Bedrock API. The problem is it doesnt read the previous message, its like making a new instance.

This is my working code, its a Nest.js Typescript:

export const send = async (message: string) => {
return new Promise(async (resolve, reject) => {
    try {
        console.log("Sending...")
        AWS.config.update({
            accessKeyId: process.env.ACCESS_KEY,
            secretAccessKey: process.env.SECRET_ACCESS_KEY,
            sessionToken: process.env.SESSION_TOKEN,
            region: process.env.REGION,
        });
        const bedrock = new AWS.BedrockRuntime();

        const params: any = {
            "modelId": "anthropic.claude-v2",
            "contentType": "application/json",
            "accept": "application/json",
            "body": JSON.stringify({
                "prompt": `\n\nHuman:\n  ${message}
                    
                    \n\nAssistant:
                `,
                "max_tokens_to_sample": 2048,
                "temperature": 0.5,
                "top_k": 250,
                "top_p": 1,
                "stop_sequences": [
                    "\\n\\nHuman:"
                ],
                "anthropic_version": "bedrock-2023-05-31"
            })
        }
        const data = await bedrock.invokeModel(params).promise()
        if (!data) {
            console.error("Error invoking the model:", data);
            resolve(data)
        } else {
            const response_body = JSON.parse(data.body.toString());
            resolve(response_body)
        }
    } catch (error) {
        console.log("Failed to send")
        reject(error);
    }
});

}

I recently found out to use Streaming API in aws bedrock, but there are no resouces out there. Here is my code:

export const streaming = async () => {
    return new Promise(async (resolve, reject) => {
        try {
            AWS.config.update({
                accessKeyId: process.env.ACCESS_KEY,
                secretAccessKey: process.env.SECRET_ACCESS_KEY,
                sessionToken: process.env.SESSION_TOKEN,
                region: process.env.REGION,
            });
            const bedrock = new AWS.BedrockRuntime();
            const params = {
                modelId: "anthropic.claude-v2",
                contentType: "application/json",
                accept: "application/json",
                body: JSON.stringify({
                    prompt: `
                        \n\nHuman:explain black holes to 8th graders
                        \n\nAssistant:
                    `,
                    max_tokens_to_sample: 300,
                    temperature: 0.1,
                    top_p: 0.9,
                }),
                responseStream: true // Set responseStream to true to receive a response stream
                ,
            };

            const responseStream = bedrock.invokeModelWithResponseStream(params);

            // @ts-ignore
            responseStream.on("data", (data: any) => {
                // Process the response data as it streams
                console.log("Received data:", data.toString());
            });

            responseStream.on("end", (): void => {
                // Streaming response complete
                console.log("Streaming complete.");
                // @ts-ignore
                resolve();
            });

            responseStream.on("error", (err) => {
                console.error("Error while streaming response:", err);
                reject(err);
            });
        } catch (error) {
            reject(error);
        }
    });
};

This using bedrock.invokeModelWithResponseStream(params); I am able to make it run. It starts to listen. But I dont know how to send message to the running listener.

So far, I have been using Javascript v2 of AWS SDK. I cant make it run in Javascript v3.

ChatGPT has no data on AWS Bedrock.

References:
https://docs.anthropic.com/claude/reference/streaming
https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModelWithResponseStream.html.

Answer 1

I had the same question related to the Llama2 model. I pivoted to using the Claude2 model for a variety of reasons and I found out how to carry on a conversation in the process. Basically, the way you carry on a conversation with Claude2 is by storing the state of the entire conversation and then passing larger and larger segments of conversation to the model. Each time you send text to the model, it should represent the entire conversation up to that point. Each Human entry should be preceded by "\n\nHuman: " and each Claude2 entry should be preceded by "\n\nAssistant: ".

Update: The Claude 3 way of passing conversation history is way better, IMO, and Claude 3 Haiku is their fastest and cheapest offering yet. It's structured json instead of these string delimiters to separate human from assistant. See docs: https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude-messages.html