Repository navigation
Application Pods removed Upon Error #588
Description
Activity
Hi, there is currently no way to have ShinyProxy keep the pod if it fails to startup.
However, with recent version of ShinyProxy we made sure it's still possible to debug the issue. ShinyProxy automatically reads out the events of the pod and logs these in the ShinyProxy logs. For example, if Kubernetes is unable to pull the image, this will be logged in the ShinyProxy logs:
In addition, you can setup a monitoring stack to collect logs of all pods. Using the Grafana dashboards you'll be able to query for specific apps. See https://github.com/openanalytics/shinyproxy-monitoring
Alternatively, you can tell ShinyProxy to collect logs of pods and write them to files or S3 buckets, see: https://shinyproxy.io/documentation/configuration/#app-container-logs . These logs are collected even if the pod fails to startup.Hi @LEDfan ,
Thanks for your response.
I will check on the Grafana part. we are trying to build a centralized monitoring solution which will be easy to access by users and also acceptible by our org. so need to check on that. Do you recommend any other k8s native solutions as a temporary workaround for stopping the immediate pod removal?
As for https://shinyproxy.io/documentation/configuration/#local-file-back-end, I tried setting it, it is not logging the kubectl output only app related and also only if stdout is enabled those logs are reflected. Please correct me if I am wrong.
Hi, there is currently not workaround for having ShinyProxy not remove the pod on error.
As for https://shinyproxy.io/documentation/configuration/#local-file-back-end, I tried setting it, it is not logging the kubectl output only app related and also only if stdout is enabled those logs are reflected. Please correct me if I am wrong.
It will indeed contain the stdout and stderr logs of the app. The warnings of kubernetes will be visible in the ShinyProxy logs (so the stdout of the ShinyProxy pods).
Although most relevant events should be logged in the ShinyProxy logs, you can also watch these yourselves, e.g. using:
kubectl -n my-shinyproxy-namespace get events -w
There also exists tools that convert these events into logs, e.g. https://github.com/resmoio/kubernetes-event-exporter or https://github.com/max-rocket-internet/k8s-event-logger although I don't have much experience with these tools.
We have deployed Shiny Proxy Operator in our AWS EKS Environment and deployed Shiny Proxy Custom Resource and deployed applications in it
If the application has any run time error say syntax error or required package is missing, then application pod enters Error state and within 5-30 secs the app pod gets removed from the pods resources
This makes us unable to troubleshoot the issue.
Every time we have to locally check the docker image using docker exec or docker run and check application is loading without any issues. This is taking considerable time to investigate and resolve the issue
I have tried to add .spec.ttlSecondsAfterFinished in application pods in kubernetes-pod-patches:, seems like it didn't take any effect
I also checked the documentation for parameters controlling this setting. I couldnt find any.
Please check and let me know how to make sure the error pods live for a certain amount of time so that we can investigate the issue and perform required actions