In our current clusters we have an emergency etcd compaction script that prevents etcd from locking up. We are looking into moving to GKE and wondering if it comes with something similar out of the box or what exactly happens when etcd gets full.
GKE: how is etcd compaction handled when api server is down / what happens when etcd is full
325 Views Asked by grosser At
1
There are 1 best solutions below
Related Questions in GOOGLE-KUBERNETES-ENGINE
- Golang == Error: OCI runtime create failed: unable to start container process: exec: "./bin": stat ./bin: no such file or directory: unknown
- Kubectl command throws error when executed from python script but manual execution works fine
- Unable to add TLS certificate to GKE from Google Secret Manager
- GKE Clusters no indication within metrics or logs after failure
- Getting connection refused to Private GKE Internal point. Autopilot private cluster
- Can I have the Google managed service range on a standard gke cluster created with Terraform- non auto-pilot
- Configure Lens with GKE cluster - gke_gcloud_auth_plugin issue
- Having issues joining my kubernetes worker noed to a controller node
- How to deploy airflow in kubernetes cluster that uses istio
- GKE cluster shutdown
- Artifactory 404 error from virtual repository where docker pulls fine but crictl does not
- Scraping from Mexico in GCP EKG?
- not able to connect via cloud sql proxy
- SQL connection throws error when adding DistributedSession, SessionMiddleware
- CronJob pod repeats pending forever even after deleting it
Related Questions in ETCD
- Istio Authorization Policy for peer authorization
- etcd 404 page not found
- How might I implement etcd's watch-stream functionality with RocksDB?
- Data is not being replicated between patroni postgresql nodes
- Patroni Timeout Error - ConnectTimeoutError
- go-zero's BFF cannot find the key of the RPC service that has already been stored in the ETCD database
- Why three node etcd cluster can have low write latency?
- etcd docker: curl is work, golang failed for DeadlineExceeded
- how etcd processing read request
- Limiting interactions with ETCD without impacting Performance
- etcd go clientv3 Txn.If.Then.Else executes both Then and Else
- Retrieve all past revisions for a given key
- Do I need to compare the revision of events since etcd does not gaurantee linearizability of watch response?
- Listing etcd prefixes at a specific level
- etcd: How to define and use a new function of Node interface in raft/node.go
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
Popular # Hahtags
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
In general terms GKE is a managed service, specially the Control Plane, as part of a fully managed product by Google's SREs Site Reliability Engineering.
With this being said, GKE control plane and its operations are part of Google duties in which you will not participate, this in order to get back time to focus on your application, while Google's SREs monitor your cluster and its computing, networking, and storage resources.
To answer your question if GKE comes with something similar (etcd compaction script) out of the box or what exactly happens when etcd gets full:
Not sure if there is a solution as yours (etcd compaction script) in GKE, but if it exist, it will be managed by the Google's SREs and you will not (depending on the cluster type you choose) notice if they backup or give maintenance to etcd o algun otro control plane component.
In my experience most common issues related to a
full etcdhas to do with jobs not being deleted. As we know when a Job completes, no more Pods are created, but the Pods are not deleted either. Keeping them around allows you to still view the logs of completed pods to check for errors, warnings, or other diagnostic output. The Job object also remains after it is completed so that you can view its status. It is up to you to delete old jobs after noting their status. When things like this happens etcd database can be overwhelmed with that amount of data and therefore becomes unresponsive (This completely depends on the number of jobs running on your cluster).In case etcd stops working or gets full, Google will be in charge of fixing it, as I mentioned above you will notice downtime on the control plane depending on the cluster type you choose. GKE offers Zonal clusters (single replica of the control plane running in a single zone), Multi-zonal clusters (single replica of the control plane running in a single zone), Regional cluster (multiple replicas of the control plane, running in multiple zones within a given region). If you choose Regional GKE cluster you will have HA for your GKE control plane (3 replicas of each control plane resource).