I hope someone can help me on this as i am rather new to kinesis firehose and the firehose agent.
I have compiled kinesis-agent for my on-premise debian server and ec2 debian instance (in a test aws account). In a seperate aws account I have created Kinesis Stream and pointed it to the AWS elasticsearch domain (Monitoring AWS ACCOUNT).
I have created a user (kinesistestagent) in the AWS Monitoring account which which has access to the kinesis firehose stream and the correct STS role added (I was stuck on this for days as it would not even authenticate)so that the kinesis agent could authenticate with the firehose stream and send its data.
My agent.json file for kinesis looks like this. I have tried to strip it down to just try and get data into firehose and elasticsearch...
{
"checkpointFile": "/opt/aws-kinesis-agent/run/checkpoints",
"cloudwatch.endpoint": "https://monitoring.eu-west-2.amazonaws.com",
"cloudwatch.emitMetrics": "false",
"firehose.endpoint": "https://firehose.eu-west-2.amazonaws.com",
"assumeRoleExternalId" :"arn:aws:firehose:eu-west-2:117215238277453:deliverystream/TEST-Firehose-EKK",
"awsAccessKeyId": "AKIRADXQWUX45KCM2IKB",
"awsSecretAccessKey": "bpq7KdidkfkeodmadeuppaccessZg4BL",
"flows": [
{
"filePattern": "/data/log/server.log",
"initialPosition": "END_OF_FILE",
"deliveryStream": "TEST-Firehose-EKK"
}
]
}
As my linux instances are not Amazon AMI's I have explicitly used the authorisation values of "awsAccessKeyId" and "awsSecretAccessKey".
The exact error I get from the logs is that the authentication works but the security token in the request in invalid?
2020-03-26 23:00:00.088+0000 (sender-2318) com.amazon.kinesis.streaming.agent.tailing.AsyncPublisher [ERROR] AsyncPublisher[fh:TEST-Firehose-EKK:/data/log/server.log]:RecordBuffer(id=2,records=179,bytes=36161) Retriable send error (com.amazonaws.services.kinesisfirehose.model.AmazonKinesisFirehoseException: The security token included in the request is invalid. (Service: AmazonKinesisFirehose; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: db4f7c53-e4a0-17e0-8db8-4a638f74b5fb)). Will retry.
The whole transaction in the kinesis log look likes this.
2020-03-26 22:59:59.574+0000 (FileTailer[fh:TEST-Firehose-EKK:/data/log/server.log].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[fh:TEST-Firehose-EKK:/data/log/server.log]: Tailer Progress: Tailer has parsed 179 records (997399 bytes), transformed 0 records, skipped 0 records, and has successfully sent 0 records to destination.
2020-03-26 22:59:59.581+0000 (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 179 records parsed (997399 bytes), and 0 records sent successfully to destinations. Uptime: 23790134ms
2020-03-26 23:00:00.058+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] connecting to firehose.eu-west-2.amazonaws.com/52.94.49.83:443
2020-03-26 23:00:00.059+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] Connecting socket to firehose.eu-west-2.amazonaws.com/52.94.49.83:443 with timeout 10000
2020-03-26 23:00:00.060+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] Enabled protocols: [TLSv1.3, TLSv1.2, TLSv1.1, TLSv1]
2020-03-26 23:00:00.060+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] Enabled cipher suites:[TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
2020-03-26 23:00:00.061+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] socket.getSupportedProtocols(): [TLSv1.3, TLSv1.2, TLSv1.1, TLSv1, SSLv3, SSLv2Hello], socket.getEnabledProtocols(): [TLSv1.3, TLSv1.2, TLSv1.1, TLSv1]
2020-03-26 23:00:00.061+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] TLS protocol enabled for SSL handshake: [TLSv1.2, TLSv1.1, TLSv1, TLSv1.3]
2020-03-26 23:00:00.061+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] Starting handshake
2020-03-26 23:00:00.070+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] Secure session established
2020-03-26 23:00:00.070+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] negotiated protocol: TLSv1.2
2020-03-26 23:00:00.070+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] negotiated cipher suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
2020-03-26 23:00:00.070+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] peer principal: CN=firehose.eu-west-2.amazonaws.com
2020-03-26 23:00:00.070+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] peer alternative names: [*.firehose.eu-west-2.vpce.amazonaws.com, firehose.eu-west-2.amazonaws.com]
2020-03-26 23:00:00.070+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] issuer principal: CN=Amazon, OU=Server CA 1B, O=Amazon, C=US
2020-03-26 23:00:00.088+0000 (sender-2318) com.amazon.kinesis.streaming.agent.tailing.AsyncPublisher [ERROR] AsyncPublisher[fh:TEST-Firehose-fEKK:/data/log/server.log]:RecordBuffer(id=2,records=179,bytes=36161) Retriable send error (com.amazonaws.services.kinesisfirehose.model.AmazonKinesisFirehoseException: The security token included in the request is invalid. (Service: AmazonKinesisFirehose; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: db4f7c53-e4a0-17e0-8db8-4a638f74b5fb)). Will retry.
Has anyone come across this error before or has anyone managed to get AWS kinesis agent working on prem server?
Many thanks in advance for the time to read my issue and any help or advice would be greatly appreciated.
cheers
Finally worked out my problem ..
AS i created Kinesis User that can access the stream with the AWSAccessKeyID and AWSSECRET I am not actually assuming any roles. By taking this line out everything works.
I had to use one or the other not both .
I hope someone can help me on this as i am rather new to kinesis firehose and the firehose agent.
I have compiled kinesis-agent for my on-premise debian server and ec2 debian instance (in a test aws account). In a seperate aws account I have created Kinesis Stream and pointed it to the AWS elasticsearch domain (Monitoring AWS ACCOUNT).
I have created a user (kinesistestagent) in the AWS Monitoring account which which has access to the kinesis firehose stream and the correct STS role added (I was stuck on this for days as it would not even authenticate)so that the kinesis agent could authenticate with the firehose stream and send its data.
My agent.json file for kinesis looks like this. I have tried to strip it down to just try and get data into firehose and elasticsearch...