CloudFront (WAF) Restricted IP Access Bots Issues

Question

I have a static HTML site that is using CloudFront and an S3 Bucket. I have used WAF to restrict access to the website to only specific IP’s.

The issue is that 95% of the hits WAF gets are from bots and I am being charged for these.

The link to the website is not indexed by Google or any other search engines and it is not available to the public so I can’t figure out how the bots are getting to it. Google Analytics is not showing these bot visits but the WAF logs show that they are hitting the root domain.

I’m trying to figure out how to filter out these bots to avoid being charged.

Initially I thought a signed URL would do the trick but that only applies to a single URL rather than the entire website and as a result blocks everything after the homepage.

I then tried many different ways to edit the S3 policy to restrict access to a specific referrer but it either didn’t block the site at all or it blocked it completely.

Example

{
    "Version": "2012-10-17",
    "Id": "PolicyForCloudFrontPrivateContent",
    "Statement": [
        {
            "Sid": "AllowCloudFrontServicePrincipal",
            "Effect": "Allow",
            "Principal": {
                "Service": "cloudfront.amazonaws.com"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3…../*",    
            "Condition": {    
                "StringEquals": {    
                    "AWS:SourceArn": "arn:aws:cloudfront….. "    
                },    
"StringLike": {    
                    "aws:Referer": [    
                        http://www.example.com/*,    
                        http://example.com/*    
                    ]    
                }    
            }    
        },    
        {    
            "Sid": "2",    
            "Effect": "Allow",    
            "Principal": {    
                "AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity E.…."    
            },    
            "Action": "s3:GetObject",    
            "Resource": "arn:aws:s3……/*"    
        }    
    ]
}

I also tried to use the S3 policy to block access to specific IP’s rather than using WAF but when I try to do that CloudFront stops working (domain gets blocked).