I followed this tutorial and configured and deployed functionbeat for a cloudwatch log group. I also added the elastic search and the kibana endpoints. Here are the outputs
MacBook-Pro:functionbeat-7.10.2-darwin-x86_64 user$ ./functionbeat setup -e
2021-10-20T22:57:35.097-0400 INFO instance/beat.go:645 Home path: [/Users/user/functionbeat-7.10.2-darwin-x86_64] Config path: [/Users/user/functionbeat-7.10.2-darwin-x86_64] Data path: [/tmp] Logs path: [/tmp/logs]
2021-10-20T22:57:35.098-0400 INFO instance/beat.go:653 Beat ID: c9cbbe8c-319a-4577-be8e-de223fba4f6e
2021-10-20T22:57:35.100-0400 INFO [beat] instance/beat.go:981 Beat info {"system_info": {"beat": {"path": {"config": "/Users/user/functionbeat-7.10.2-darwin-x86_64", "data": "/tmp", "home": "/Users/user/functionbeat-7.10.2-darwin-x86_64", "logs": "/tmp/logs"}, "type": "functionbeat", "uuid": "c9cbbe8c-319a-4577-be8e-de223fba4f6e"}}}
2021-10-20T22:57:35.100-0400 INFO [beat] instance/beat.go:990 Build info {"system_info": {"build": {"commit": "aacf9ecd9c494aa0908f61fbca82c906b16562a8", "libbeat": "7.10.2", "time": "2021-01-12T22:39:38.000Z", "version": "7.10.2"}}}
2021-10-20T22:57:35.100-0400 INFO [beat] instance/beat.go:993 Go runtime info {"system_info": {"go": {"os":"darwin","arch":"amd64","max_procs":8,"version":"go1.14.12"}}}
2021-10-20T22:57:35.101-0400 INFO [beat] instance/beat.go:997 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-10-13T02:09:16.051008-04:00","name":"Jijos-MacBook-Pro.local","ip":["127.0.0.1/8","::1/128","fe80::1/64","fe80::1ce5:e86b:6d5c:88d9/64","192.168.1.11/24","fe80::84d7:afff:fe5e:107c/64","fe80::84d7:afff:fe5e:107c/64","fe80::46cf:bba0:8bb3:5b62/64","fe80::99c:e0d2:dbd5:e7aa/64","fe80::cdce:ea46:e3c:3feb/64","fe80::76aa:a49d:6061:d997/64","fe80::aede:48ff:fe00:1122/64"],"kernel_version":"20.6.0","mac":["a6:83:e7:8a:24:01","a4:83:e7:8a:24:01","86:d7:af:5e:10:7c","86:d7:af:5e:10:7c","82:ea:c5:62:c4:05","82:ea:c5:62:c4:04","82:ea:c5:62:c4:01","82:ea:c5:62:c4:00","82:ea:c5:62:c4:01","ac:de:48:00:11:22"],"os":{"family":"darwin","platform":"darwin","name":"Mac OS X","version":"10.16","major":10,"minor":16,"patch":0,"build":"20G165"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"BEEB9A65-6CAE-51E9-B16E-BCE2FBB9EED9"}}}
2021-10-20T22:57:35.101-0400 INFO [beat] instance/beat.go:1026 Process info {"system_info": {"process": {"cwd": "/Users/user/functionbeat-7.10.2-darwin-x86_64", "exe": "./functionbeat", "name": "functionbeat", "pid": 13218, "ppid": 8720, "start_time": "2021-10-20T22:57:34.835-0400"}}}
2021-10-20T22:57:35.101-0400 INFO instance/beat.go:299 Setup Beat: functionbeat; Version: 7.10.2
2021-10-20T22:57:35.102-0400 INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'functionbeat-7.10.2' as ILM is enabled.
2021-10-20T22:57:35.103-0400 INFO eslegclient/connection.go:99 elasticsearch url: https://l-es.xyz.io:443
2021-10-20T22:57:35.106-0400 INFO [publisher] pipeline/module.go:113 Beat name: Jijos-MacBook-Pro.local
2021-10-20T22:57:35.107-0400 INFO eslegclient/connection.go:99 elasticsearch url: https://l-es.xyz.io:443
2021-10-20T22:57:35.316-0400 INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.2
2021-10-20T22:57:35.366-0400 INFO template/load.go:183 Existing template will be overwritten, as overwrite is enabled.
2021-10-20T22:57:35.420-0400 INFO template/load.go:117 Try loading template functionbeat-7.10.2 to Elasticsearch
2021-10-20T22:57:35.496-0400 INFO template/load.go:109 template with name 'functionbeat-7.10.2' loaded.
2021-10-20T22:57:35.496-0400 INFO [index-management] idxmgmt/std.go:298 Loaded index template.
Index setup finished.
On deploying the functionbeat, I was able to successfully deploy it as well
MacBook-Pro:functionbeat-7.10.2-darwin-x86_64 user$ ./functionbeat -v -e -d "*" deploy fn-cloudwatch-logs
2021-10-20T22:58:30.531-0400 INFO instance/beat.go:645 Home path: [/Users/user/functionbeat-7.10.2-darwin-x86_64] Config path: [/Users/user/functionbeat-7.10.2-darwin-x86_64] Data path: [/tmp] Logs path: [/tmp/logs]
2021-10-20T22:58:30.532-0400 DEBUG [beat] instance/beat.go:697 Beat metadata path: /tmp/meta.json
2021-10-20T22:58:30.532-0400 INFO instance/beat.go:653 Beat ID: c9cbbe8c-319a-4577-be8e-de223fba4f6e
2021-10-20T22:58:30.535-0400 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:126 add_cloud_metadata: starting to fetch metadata, timeout=3s
2021-10-20T22:58:33.536-0400 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for azure after 3.000470363s. result=[provider:azure, error=failed requesting azure metadata: Get "http://169.254.169.254/metadata/instance/compute?api-version=2017-04-02": dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2021-10-20T22:58:33.536-0400 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for aws after 3.000674614s. result=[provider:aws, error=failed requesting aws metadata: Get "http://169.254.169.254/2014-02-25/dynamic/instance-identity/document": dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2021-10-20T22:58:33.536-0400 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for openstack after 3.000710413s. result=[provider:openstack, error=failed requesting openstack metadata: Get "http://169.254.169.254/2009-04-04/meta-data/instance-id": dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2021-10-20T22:58:33.536-0400 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for gcp after 3.000737886s. result=[provider:gcp, error=failed requesting gcp metadata: Get "http://169.254.169.254/computeMetadata/v1/?recursive=true&alt=json": dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2021-10-20T22:58:33.536-0400 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for digitalocean after 3.000758996s. result=[provider:digitalocean, error=failed requesting digitalocean metadata: Get "http://169.254.169.254/metadata/v1.json": dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2021-10-20T22:58:33.536-0400 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:129 add_cloud_metadata: fetchMetadata ran for 3.000778492s
2021-10-20T22:58:33.536-0400 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
2021-10-20T22:58:33.536-0400 DEBUG [processors] processors/processor.go:120 Generated new processors: add_host_metadata=[netinfo.enabled=[true], cache.ttl=[5m0s]], add_cloud_metadata={}
2021-10-20T22:58:33.538-0400 DEBUG [cli-handler] cmd/cli_handler.go:52 Starting deploy for: fn-cloudwatch-logs
2021-10-20T22:58:33.539-0400 DEBUG [aws] aws/cli_manager.go:119 Deploying function: fn-cloudwatch-logs
2021-10-20T22:58:33.539-0400 DEBUG [provider] aws/template_builder.go:90 Compressing all assets into an artifact
2021-10-20T22:58:35.284-0400 DEBUG [provider] aws/template_builder.go:96 Compression is successful (zip size: 22046351 bytes)
2021-10-20T22:58:35.339-0400 INFO [provider] aws/template_builder.go:155 No role is configured for function fn-cloudwatch-logs, creating a custom role.
2021-10-20T22:58:35.342-0400 DEBUG [aws] aws/cli_manager.go:69 Using cloudformation template:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"fnbfncloudwatchlogs": {
"Properties": {
"Code": {
"S3Bucket": "functionbeat-deploy-bucket-test-poc",
"S3Key": "functionbeat-deployment/fn-cloudwatch-logs/eMPnb_aKewcOO0XW-fgVUvvN0PXWxEDkvxFexJEI-zY/functionbeat.zip"
},
"Description": "lambda function for cloudwatch logs",
"Environment": {
"Variables": {
"BEAT_STRICT_PERMS": "false",
"ENABLED_FUNCTIONS": "fn-cloudwatch-logs"
}
},
"FunctionName": "fn-cloudwatch-logs",
"Handler": "functionbeat-aws",
"MemorySize": 128,
"ReservedConcurrentExecutions": 5,
"Role": {
"Fn::GetAtt": [
"fnbfncloudwatchlogsIAMRoleLambdaExecution",
"Arn"
]
},
"Runtime": "go1.x",
"Timeout": 3
},
"Type": "AWS::Lambda::Function"
},
"fnbfncloudwatchlogsIAMRoleLambdaExecution": {
"Properties": {
"AssumeRolePolicyDocument": {
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Principal": {
"Service": {
"Fn::Join": [
"",
[
"lambda.",
{
"Ref": "AWS::URLSuffix"
}
]
]
}
}
}
]
},
"Path": "/",
"Policies": [
{
"PolicyDocument": {
"Statement": [
{
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Effect": "Allow",
"Resource": [
{
"Fn::Sub": "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/fn-cloudwatch-logs:*"
}
]
}
]
},
"PolicyName": {
"Fn::Join": [
"-",
[
"fnb",
"lambda",
"fn-cloudwatch-logs"
]
]
}
}
],
"RoleName": "functionbeat-lambda-fn-cloudwatch-logs"
},
"Type": "AWS::IAM::Role"
},
"fnbfncloudwatchlogsLogGroup": {
"Properties": {
"LogGroupName": "/aws/lambda/fn-cloudwatch-logs"
},
"Type": "AWS::Logs::LogGroup"
},
"fnbfncloudwatchlogsPermission0": {
"Properties": {
"Action": "lambda:InvokeFunction",
"FunctionName": {
"Fn::GetAtt": [
"fnbfncloudwatchlogs",
"Arn"
]
},
"Principal": {
"Fn::Join": [
"",
[
"logs.",
{
"Ref": "AWS::Region"
},
".",
{
"Ref": "AWS::URLSuffix"
}
]
]
},
"SourceArn": {
"Fn::Join": [
"",
[
"arn:",
{
"Ref": "AWS::Partition"
},
":logs:",
{
"Ref": "AWS::Region"
},
":",
{
"Ref": "AWS::AccountId"
},
":log-group:",
"/aws/containerinsights/translator-eks-ci-blue-cluster/application",
":*"
]
]
}
},
"Type": "AWS::Lambda::Permission"
},
"fnbfncloudwatchlogsSFawscontainerinsightstranslatoreksciblueclusterapplication": {
"Properties": {
"DestinationArn": {
"Fn::GetAtt": [
"fnbfncloudwatchlogs",
"Arn"
]
},
"FilterPattern": "mylog_",
"LogGroupName": "/aws/containerinsights/translator-eks-ci-blue-cluster/application"
},
"Type": "AWS::Logs::SubscriptionFilter"
}
}
}
2021-10-20T22:58:35.343-0400 DEBUG [aws.executor] executor/executor.go:53 The executor is executing '6' operations for converging state
2021-10-20T22:58:35.343-0400 DEBUG [aws] aws/op_ensure_bucket.go:33 Verifying presence of S3 bucket: functionbeat-deploy-bucket-test-poc
2021-10-20T22:58:35.548-0400 DEBUG [aws] aws/op_upload_to_bucket.go:44 Uploading file 'functionbeat-deployment/fn-cloudwatch-logs/eMPnb_aKewcOO0XW-fgVUvvN0PXWxEDkvxFexJEI-zY/functionbeat.zip' to bucket 'functionbeat-deploy-bucket-test-poc' with size 22046351 bytes
2021-10-20T22:58:36.286-0400 DEBUG [aws] aws/op_upload_to_bucket.go:57 Upload successful
2021-10-20T22:58:36.287-0400 DEBUG [aws] aws/op_upload_to_bucket.go:44 Uploading file 'functionbeat-deployment/fn-cloudwatch-logs/2_AZvlkBMEoQfFeV_dSW5B2VD927AWycnifwEPnFtcI/cloudformation-template-create.json' to bucket 'functionbeat-deploy-bucket-test-poc' with size 4231 bytes
2021-10-20T22:58:36.328-0400 DEBUG [aws] aws/op_upload_to_bucket.go:57 Upload successful
2021-10-20T22:58:36.328-0400 DEBUG [aws] aws/op_cloudformation.go:48 Creating CloudFormation create stack request
2021-10-20T22:58:36.848-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::CloudFormation::Stack, LogicalResourceId: fnb-fn-cloudwatch-logs-stack, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: User Initiated
2021-10-20T22:58:40.973-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: fnbfncloudwatchlogsIAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS
2021-10-20T22:58:40.973-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbfncloudwatchlogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS
2021-10-20T22:58:43.035-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: fnbfncloudwatchlogsIAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2021-10-20T22:58:43.035-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbfncloudwatchlogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2021-10-20T22:58:45.091-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbfncloudwatchlogsLogGroup, ResourceStatus: CREATE_COMPLETE
2021-10-20T22:58:55.375-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: fnbfncloudwatchlogsIAMRoleLambdaExecution, ResourceStatus: CREATE_COMPLETE
2021-10-20T22:58:57.429-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbfncloudwatchlogs, ResourceStatus: CREATE_IN_PROGRESS
2021-10-20T22:59:01.542-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbfncloudwatchlogs, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2021-10-20T22:59:05.670-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbfncloudwatchlogs, ResourceStatus: CREATE_COMPLETE
2021-10-20T22:59:07.727-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbfncloudwatchlogsPermission0, ResourceStatus: CREATE_IN_PROGRESS
2021-10-20T22:59:07.728-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbfncloudwatchlogsSFawscontainerinsightstranslatoreksciblueclusterapplication, ResourceStatus: CREATE_IN_PROGRESS
2021-10-20T22:59:07.728-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbfncloudwatchlogsPermission0, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2021-10-20T22:59:07.728-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbfncloudwatchlogsSFawscontainerinsightstranslatoreksciblueclusterapplication, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2021-10-20T22:59:07.728-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbfncloudwatchlogsSFawscontainerinsightstranslatoreksciblueclusterapplication, ResourceStatus: CREATE_COMPLETE
2021-10-20T22:59:18.052-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbfncloudwatchlogsPermission0, ResourceStatus: CREATE_COMPLETE
2021-10-20T22:59:19.735-0400 DEBUG [aws] aws/op_delete_file_bucket.go:38 Removing file 'functionbeat-deployment/fn-cloudwatch-logs/eMPnb_aKewcOO0XW-fgVUvvN0PXWxEDkvxFexJEI-zY/functionbeat.zip' on bucket 'functionbeat-deploy-bucket-test-poc'
2021-10-20T22:59:19.808-0400 DEBUG [aws] aws/op_delete_file_bucket.go:51 Remove successful
2021-10-20T22:59:19.808-0400 DEBUG [aws.executor] executor/executor.go:68 All operations successful
2021-10-20T22:59:19.808-0400 DEBUG [aws] aws/cli_manager.go:125 Successfully created function 'fn-cloudwatch-logs'
2021-10-20T22:59:19.808-0400 DEBUG [aws] aws/cli_manager.go:126 Deploy finish for function 'fn-cloudwatch-logs'
Function: fn-cloudwatch-logs, deploy successful
2021-10-20T22:59:19.810-0400 DEBUG [cli-handler] cmd/cli_handler.go:64 Deploy execution ended
In the aws console I see that the function fn-cloudwatch-logs is being created and I also see the cloudformation template in the s3 bucket. I also see that a trigger is being added to the cloudwatch log group, but the functionbeat status in kibana says that No Data is being received.
I also tried giving a sample log data to the lambda function and it ends up throwing a Could not parse events from cloudwatch error in the cloudwatch
2021-10-21T19:17:32.605Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://******.io:443)): Connection marked as failed because the onConnect callback failed: 169.254.40.221 requires the default distribution of Elasticsearch. Please update to the default distribution of Elasticsearch for full access to all free features, or switch to the OSS distribution of 169.254.40.221.
2021-10-21T19:17:32.605Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://******.io:443)): Connection marked as failed because the onConnect callback failed: 169.254.40.221 requires the default distribution of Elasticsearch. Please update to the default distribution of Elasticsearch for full access to all free features, or switch to the OSS distribution of 169.254.40.221.
Is there anything that I am missing in the configuration?
This EOF error is because it is expecting some inputs but it is not receiving. Because you are directly calling the function beat lambda function but the function need to be invoked automatically thats why we are giving triggers. Probably the issue is because AWS is not able to make a successful connection with your elastic cloud. Usually protocol issue (if u are making output.elasticsearch: host as localhost:9200 because AWS doesn't able to reach to this localhost url unless it is a public one) or permission issue. If you check the functionbeat lambda function cloudwatch logs you can able to see the actual issue. Put logging.level: debug in functionbeat.yml for detailed logs.
Also, you cannot see the logs in kibana right after deploying the functionbeat. Once the subscription filter has added to the log group after the successful deployment you have to invoke the function which you have added subscription filter not the functionbeat lambda function. Because the trigger is added to the functionbeat lambda function. So whenever a new item get added into this log group then it will automatically invoke functionbeat lambda function.