boto3 searching unused security groups

Question

I am using AWS Python SDK Boto3 and I am trying to know which security groups are unused. With boto2 I did it but I do not know how to do the same with boto3.

from boto.ec2.connection import EC2Connection
from boto.ec2.regioninfo import RegionInfo
import boto.sns
import sys
import logging
from security_groups_config import config

# Get settings from config.py
aws_access_key = config['aws_access_key']
aws_secret_key = config['aws_secret_key']    
ec2_region_name = config['ec2_region_name']
ec2_region_endpoint = config['ec2_region_endpoint']

region = RegionInfo(name=ec2_region_name, endpoint=ec2_region_endpoint)

if aws_access_key:
    conn = EC2Connection(aws_access_key, aws_secret_key, region=region)
else:
    conn = EC2Connection(region=region)

sgs = conn.get_all_security_groups()

## Searching unused SG if the instances number is 0
def search_unused_sg(event, context):
    for sg in sgs:
        print sg.name, len(sg.instances())

Answer 1

First , I suggest you relook how boto3 deal with credential. Better use a genereic AWS credential file , so in the future when required, you can switch to IAM roles base credential or AWS STS without changing your code.

import boto3 
# You should use the credential profile file 
ec2 = boto3.client("ec2")

# In boto3, if you have more than 1000 entries, you need to handle the pagination
# using the NextToken parameter, which is not shown here.

all_instances = ec2.describe_instances() 
all_sg = ec2.describe_security_groups()

instance_sg_set = set()
sg_set = set()

for reservation in all_instances["Reservations"] :
  for instance in reservation["Instances"]: 
    for sg in instance["SecurityGroups"]:
      instance_sg_set.add(sg["GroupName"]) 


for security_group in all_sg["SecurityGroups"] :
  sg_set.add(security_group ["GroupName"])

idle_sg = sg_set - instance_sg_set

Note : code are not tested. Please debug it as required.

Answer 2

Use the power of Boto3 and Python's list comprehension and sets to get what you want in 7 lines of code:

import boto3

ec2 = boto3.resource('ec2') #You have to change this line based on how you pass AWS credentials and AWS config

sgs = list(ec2.security_groups.all())
insts = list(ec2.instances.all())

all_sgs = set([sg.group_name for sg in sgs])
all_inst_sgs = set([sg['GroupName'] for inst in insts for sg in inst.security_groups])
unused_sgs = all_sgs - all_inst_sgs

Debug information

print 'Total SGs:', len(all_sgs)
print 'SGS attached to instances:', len(all_inst_sgs)
print 'Orphaned SGs:', len(unused_sgs)
print 'Unattached SG names:', unused_sgs

Output

Total SGs: 289
SGS attached to instances: 129
Orphaned SGs: 160
Unattached SG names: set(['mysg', '...

Answer 3

Note: If you have ASG (Autoscalling group) that are in null state (count=0), when the ASG start adding adding the security groups, then it will adopt the orphan security groups. Keep in mind you need to check for the ASG security groups also

Answer 4

I used an alternative approach. If we skip the credentials discussion and go back to the main question "boto3 searching unused security groups" here is an option: You go and enumerate the resource, in my case a network interface, because if you think about it, a security group has to be associated to a resource in order to be used. My example:

client = boto3.client('ec2', region_name=region, aws_access_key_id=newsession_id, aws_secret_access_key=newsession_key, aws_session_token=newsession_token)
response = client.describe_network_interfaces()
for i in response["NetworkInterfaces"]:
    #Check if the security group is attached
    if 'Attachment' in i and i['Attachment']['Status'] == 'attached':
        #Create a list with the attached SGs
        groups = [g['GroupId'] for g in i['Groups']]

II used the network interface resource because I needed to get public IPs for the accounts.