IBM Support

IBM Cloud Private 3.1.2: Kube cronjobs running 1 hour later than scheduled

Troubleshooting


Problem

This technote discusses an issue that happens with Kubernetes Cronjobs. The cronjob is defined within Kubernetes under ICP 3.1.2.  The job is to execute component level backups. Its set to run at 04:00am daily, but its run at 05:00am.
Therefore, the cronjob is being scheduled 1 hour later than its job specified.

Cause

This is a known issue in Kubernetes (k8s). (Users) needs to mount local timezone inside the container to make it consistent with host.

We also have cronjob for logging and VA. And the same workaround is used.

Diagnosing The Problem

Resolving this issue with Cron Job requires that you adjust the timezone for Kubernetes Cronjob. See steps below:

The kube-controller-manager is running inside a container in IBM Cloud Private (ICP). However the timezone inside container is always UTC. So  the Cronjob was triggered in a different time than expected.


To correct timezone in container of kube-controller-manager,  update the container spec as below:

  1. Copy manifest file of k8s-master static pod
cp /etc/cfc/pods/master.json ~/
  1. Update it (vim ~/master.json) to mount local time /etc/localtime in the controller-manager container.
        "volumeMounts": [
          {
            "name": "localtime",
            "mountPath": "/etc/localtime"
          },
...
    "volumes": [
      {
        "name": "localtime",
        "hostPath": {
          "path": "/etc/localtime"
        }
      },
  1. Copy the updated manifest file back
cp ~/master.json /etc/cfc/pods/

kubelet service will help restart static pod to pick up new changes.

  1. Verify the change
# date
Tue Jun 11 07:59:00 EDT 2019

# docker ps | grep hyper
95ef02957a75        a28dcbcae557                                                                                            "/hyperkube schedule…"   About a minute ago   Up About a minute                       k8s_scheduler_k8s-master-9.21.55.15_kube-system_ef90d47d77f8989cd6fd3eac6e6b1dfc_2
d3228e5c8b74        a28dcbcae557                                                                                            "/hyperkube controll…"   About a minute ago   Up About a minute                       k8s_controller-manager_k8s-master-9.21.55.15_kube-system_ef90d47d77f8989cd6fd3eac6e6b1dfc_2
06e21ec01d45        a28dcbcae557                                                                                            "/hyperkube apiserve…"   About a minute ago   Up About a minute                       k8s_apiserver_k8s-master-9.21.55.15_kube-system_ef90d47d77f8989cd6fd3eac6e6b1dfc_0
e3402496a3b2        a28dcbcae557                                                                                            "/hyperkube proxy --…"   2 weeks ago          Up 2 weeks                              k8s_proxy_k8s-proxy-dpjch_kube-system_122a8d9a-8121-11e9-a207-0894ef00adc0_0
# docker exec -it d3228e5c8b74 bash
root@ma1gpu05:/# date
Tue Jun 11 07:59:10 EDT 2019
Similar issue found in  link.
Also see:

Document Location

Worldwide

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSBS6K","label":"IBM Cloud Private"},"Component":"Configuration","Platform":[{"code":"PF016","label":"Linux"},{"code":"PF043","label":"Red Hat"},{"code":"PF040","label":"RedHat OpenShift"}],"Version":"ICP 3.1.2","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
11 June 2019

UID

ibm10887373

Page Feedback