On Demand Staging ( ephemeral ) Environment in K8s

Question

I am trying to build up a ephemeral staging environment in k8s .

in current setup in ECS, lets say if i have 10 microservices and need 3 among those for testing a feature , so developer creates feature branch from one of the infra repo and create some file with the developer name and feature name and mark the services( lets say B and C ) they need as true.

It will create associated isolated infra with all required dependencies ( like ECS service , Route53 records , SQS , Lambda Functions , Redis ) etc with the help of terraform and tag them with it.

Then A dynamic  file is created and stored in S3. This file contains the domain  URLs and dynamic resource details for the services marked true. For other services, we will use masterbox  URLs are used.

This s3 file will be injected into the task definition in ECS service.

Finally the developers will deploy to this dynamic services from their application repo with the help of Github Actions or Jenkins. The dynamically created file will be available here as a environment variable in file.

so in the K8s way i would like to have more simpler and easier way.

1. some way of mentioning services ( a gui if possible ) . Lets assume there are 10 microservices and for testing a feature 1 it requires it requires microservice A and D while others can be of staging urls.

2. Create dynamic resources required for this lets say queues , buckets and It should be tagged with developer name and feature name they are working ( can be done via terraform ) 

3. The each service may require a isolated sqs queue , sns topics etc.. So it created dynamic resources with developers and feature name .

4. The dynamic endpoints created should be available somehow to the other microservices also like I said in s3 injection.

5. And now they need to deploy their feature to k8s cluster with config change's . So if given A and D it should crate two deployments on feature name namespace with argocd or like tools and config like dynamic urls created from AWS should b e available on the containers . this is important because , microservice A sqs queue is may being used by microservice D or may have fanout cases.

6. Single ingress with multiple routes and like developer name.featuename.dynamicenvironment.example.com

7. TTL for the ephemeral environment

8. Open source

What would be best way to do it and how to make developer like here more easy.?

I thought of using Argocd helm but what would be the best way here

Saw some tools like oketo , qovery , port etc.. but not exactly matching and it's so costly as we large numbers of developers

Answer 1

Creating ephemeral staging environments in Kubernetes with dynamic resource provisioning, automatic tagging, and simplified developer interaction can be achieved using a combination of various tools and practices. suggested approach:

Dynamic Resource Provisioning:

• Use Terraform or Kubernetes Operators to dynamically provision resources. Tag resources with metadata like developer name and feature name.

Developer Interaction:

• Build a simple GUI for developers to select microservices for testing.

Dynamic Endpoints and Configuration:

• Store dynamic environment details centrally. Use Kubernetes ConfigMaps or Secrets for injecting configurations.

Ephemeral Environment Deployment:

• Utilize ArgoCD, Helm, or similar tools for managing deployments. Automate deployments with CI/CD tools.

Ingress Routing:

• Employ a single Ingress with multiple paths/routes. Set up wildcard DNS for routing based on developer and feature.

TTL for Ephemeral Environments:

• Implement a TTL mechanism using Kubernetes Jobs or CronJobs for cleanup.

Open Source Tools:

• Use ArgoCD, Helm, Prometheus, and Grafana as open-source tools.

Cost Considerations:

• Minimize costs by using self-managed Kubernetes clusters and open-source tools.

Documentation and Training:

• Provide clear documentation and training for developers on the dynamic provisioning system.

Adapting these practices can help in efficiently managing ephemeral staging environments in Kubernetes while keeping the developer experience user-friendly.