Mount to pod's empty dir when PVC is not responding - Openshift

Question

I have a openshift deployment which is using NFS to save the logs and it is mapped to PV and PVC added to the container as below.

volumeMounts:
- name: outputlog
  mountPath: /logs

And it is mapped to the PVC mount as below.

volumes:
- name: outputlog
  persistentVolumeClaim:
  claimName: "pvc-1"

So the problem is when doing the chaos testing if the network latency is more than 2 seconds the NFS is not responding and the pods are keep restarting which caused 502 error continuously.

As a solution i feel it is better to mount to emptydir and logs temporary saved in pod's directory.

So the questions are

1. will that be feasible
2. If it is feasible what could be the mechanism to do it?

Cheers.

Answer 1

No, I don't think this is feasible. In theory, sure, you could use an emptyDir mount for logging. But, as you point out, you are going to lose all of your logs when a pod is stopped (even from something like autoscaling). emptyDir is designed to be ephemeral, so if your data isn't ephemeral, don't use emptyDir. Also, it's just going to make it very difficult to access your logs.

Without more information it's hard to really say what you should do. It depends on how much you can change your app, how much latitude you have fix your NFS problems, how delayed your logs can be, and what tools you already have for log management. But in general:

• You should switch to stdout logging. Almost all of your problems just go away instantly if you can change your application to log via stdout, which is what it should be doing anyway.
• If that fails, you should fix your NFS. Either switch to something more robust, or change what you expect it to tolerate. After all, if you have 2s of network latency, there are lots of things that are going to fall apart, not just NFS. If you can't use PVCs, then your OpenShift cluster is not going to very useful.
• If you are going to switch to local storage, use local storage PVCs, not emptyDir mounts. You, in theory, could combine this with something like a fluentd sidecar to asynchronously transfer logs to your unreliable NFS service. But trying to build your own log aggregation would be complex. (And a waste, given that log aggregation is built into OpenShift if you switch to stdout.)