How do I provide AWS Bedrock with conversation context for meta.llama2-13b-chat-v1?

648 Views Asked by At

I'm trying to design an AI chat experience, and I'm using AWS Bedrock with Llama2 13b as my initial POC. My question is how do I properly provide the Llama2 model with the conversation context such that it responds with everything that has come before in mind?

The AWS Llama2 documentation provides information about the allowed input parameters in their model documentation. However, there is no mention of previous conversation history.

Some implementations of Llama2 allow for the API caller to provide a context to the API, such the "context" parameter as in Ollama's API. I do not see a similar option for the AWS Bedrock implementation.

I can only suspect that the gap is some misunderstanding of how conversation context works with LLMs in general that gets abstracted away in the UI-based chat experience that we have all been playing with the past year. I am sure the answers are somewhere in the implementation of the models, but since I have not been able to find a simple answer to this question, I thought I would ask it here.

I have already tried running a script that instantiates a go BedrockRuntime SDK and queries the bedrock model with user input given in the console. It loops back around and asks for additional inputs. The subsequent inputs did not yield results suggesting that the Llama2 Bedrock model had knowledge of the previous query, so I deduced that the session does not persist conversation history in AWS Bedrock.

package main

import (
    "bufio"
    "encoding/json"
    "fmt"
    "os"
    "strings"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/bedrockruntime"
)

type Llama2Request struct {
    Prompt       string  `json:"prompt"`
    MaxGenLength int     `json:"max_gen_len,omitempty"`
    Temperature  float64 `json:"temperature,omitempty"`
}

type Llama2Response struct {
    Generation string `json:"generation"`
}

func main() {
    mySession := session.Must(session.NewSession(&aws.Config{Region: aws.String("us-east-1")}))
    svc := bedrockruntime.New(mySession)

    modelId := "meta.llama2-13b-chat-v1"
    var input string
    fmt.Print("Initial Input:")
    for {
        inputReader := bufio.NewReader(os.Stdin)
        input, _ = inputReader.ReadString('\n')
        input = strings.TrimSuffix(input, "\n")
        if input == "BREAK" {
            break
        }

        body, err := json.Marshal(Llama2Request{
            Prompt:       input,
            MaxGenLength: 512,
            Temperature:  0.5,
        })

        var errorresp string
        if err != nil {
            errorresp = fmt.Sprintf("Unable to Marshal Llama2Request: %s", input)
            fmt.Print(errorresp, err)
            break
        }

        req, resp := svc.InvokeModelRequest(&bedrockruntime.InvokeModelInput{
            ModelId: &modelId, Body: body,
        })

        if err := req.Send(); err != nil {
            errorresp = "Error sending model invocation."
            fmt.Print(errorresp, err)
            break
        }

        var response Llama2Response
        if err := json.Unmarshal(resp.Body, &response); err != nil {
            errorresp = "Unable to Unmarshal Llama2Response."
            fmt.Print(errorresp, err)
            break
        }

        fmt.Print(response.Generation)
    }
}

0

There are 0 best solutions below