ある日ほぼ全てのpodがevictedやpendingになって起動できなくなっていた．
なにが起きているか知りたかったのでpendingしているPodをdescribeした
taka@doktor:~/doktor$ k describe pod/web-7d6c7f746c-f9tpj
Name: web-7d6c7f746c-f9tpj
Namespace: default
Priority: 0
Node: doktor/192.168.100.146
Start Time: Wed, 01 Sep 2021 05:39:19 +0000
Labels: app=web
pod-template-hash=7d6c7f746c
security.istio.io/tlsMode=istio
service.istio.io/canonical-name=web
service.istio.io/canonical-revision=latest
Annotations: cni.projectcalico.org/podIP: 10.1.182.182/32
cni.projectcalico.org/podIPs: 10.1.182.182/32
sidecar.istio.io/status:
{"version":"8f7046dff394dd0cb963b4b2943725bfe4b49889cb4423b4293adccb4f916476","initContainers":["istio-init"],"containers":["istio-proxy"]...
Status: Running
IP: 10.1.182.182
IPs:
IP: 10.1.182.182
Controlled By: ReplicaSet/web-7d6c7f746c
Init Containers:
istio-init:
Container ID: containerd://f4c54e53e08cac2cc777c254ebff5ada6e167fabdb66cdd6210269188bf1af26
Image: docker.io/istio/proxyv2:1.5.1
Image ID: docker.io/istio/proxyv2@sha256:3ad9ee2b43b299e5e6d97aaea5ed47dbf3da9293733607d9b52f358313e852ae
Port: <none>
Host Port: <none>
Command:
istio-iptables
-p
15001
-z
15006
-u
1337
-m
REDIRECT
-i
*
-x
-b
*
-d
15020
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 01 Sep 2021 05:39:21 +0000
Finished: Wed, 01 Sep 2021 05:39:21 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 100m
memory: 50Mi
Requests:
cpu: 10m
memory: 10Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-8sf66 (ro)
Containers:
web:
Container ID: containerd://7cf503612fcb8c6ecf38b843e2c59a3972103febb209d6833325f37d5c345baf
Image: takahyon/doktor-web:latest
Image ID: docker.io/takahyon/doktor-web@sha256:f1ff472f923392e6b6c4ec5836e564ae7ad5e9604266226da11ac7d28570f3b5
Port: 5000/TCP
Host Port: 0/TCP
State: Running
Started: Wed, 01 Sep 2021 05:39:46 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-8sf66 (ro)
istio-proxy:
Container ID: containerd://33f8563b00fc6791ab8d19721197a9a588a157fcd2778bfe84bbbb79d5418021
Image: docker.io/istio/proxyv2:1.5.1
Image ID: docker.io/istio/proxyv2@sha256:3ad9ee2b43b299e5e6d97aaea5ed47dbf3da9293733607d9b52f358313e852ae
Port: 15090/TCP
Host Port: 0/TCP
Args:
proxy
sidecar
--domain
$(POD_NAMESPACE).svc.cluster.local
--configPath
/etc/istio/proxy
--binaryPath
/usr/local/bin/envoy
--serviceCluster
web.$(POD_NAMESPACE)
--drainDuration
45s
--parentShutdownDuration
1m0s
--discoveryAddress
istio-pilot.istio-system:15010
--zipkinAddress
zipkin.istio-system:9411
--dnsRefreshRate
300s
--connectTimeout
10s
--proxyAdminPort
15000
--concurrency
2
--controlPlaneAuthPolicy
NONE
--statusPort
15020
State: Running
Started: Wed, 01 Sep 2021 05:39:46 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 10m
memory: 40Mi
Readiness: http-get http://:15020/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30
Environment:
POD_NAME: web-7d6c7f746c-f9tpj (v1:metadata.name)
ISTIO_META_POD_PORTS: [
{"containerPort":5000,"protocol":"TCP"}
]
ISTIO_META_CLUSTER_ID: Kubernetes
POD_NAMESPACE: default (v1:metadata.namespace)
INSTANCE_IP: (v1:status.podIP)
SERVICE_ACCOUNT: (v1:spec.serviceAccountName)
ISTIO_AUTO_MTLS_ENABLED: true
ISTIO_META_POD_NAME: web-7d6c7f746c-f9tpj (v1:metadata.name)
ISTIO_META_CONFIG_NAMESPACE: default (v1:metadata.namespace)
SDS_ENABLED: false
ISTIO_META_INTERCEPTION_MODE: REDIRECT
ISTIO_METAJSON_LABELS: {"app":"web","pod-template-hash":"7d6c7f746c"}
ISTIO_META_WORKLOAD_NAME: web
ISTIO_META_OWNER: kubernetes://apis/apps/v1/namespaces/default/deployments/web
Mounts:
/etc/certs/ from istio-certs (ro)
/etc/istio/proxy from istio-envoy (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-8sf66 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-8sf66:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-8sf66
Optional: false
istio-envoy:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
istio-certs:
Type: Secret (a volume populated by a Secret)
SecretName: istio.default
Optional: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 12m (x7 over 16m) default-scheduler 0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/disk-pressure: }, that the pod didn't tolerate.
Normal Scheduled 12m default-scheduler Successfully assigned default/web-7d6c7f746c-f9tpj to doktor
Normal Pulled 12m kubelet Container image "docker.io/istio/proxyv2:1.5.1" already present on machine
Normal Created 12m kubelet Created container istio-init
Normal Started 12m kubelet Started container istio-init
Normal Pulling 12m kubelet Pulling image "takahyon/doktor-web:latest"
Normal Pulled 12m kubelet Successfully pulled image "takahyon/doktor-web:latest" in 23.761418644s
Normal Created 12m kubelet Created container web
Normal Started 12m kubelet Started container web
Normal Pulled 12m kubelet Container image "docker.io/istio/proxyv2:1.5.1" already present on machine
Normal Created 12m kubelet Created container istio-proxy
Normal Started 12m kubelet Started container istio-proxy
Warning Unhealthy 12m kubelet Readiness probe failed: HTTP probe failed with statuscode: 503
Warning FailedScheduling 12m (x7 over 16m) default-scheduler 0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/disk-pressure: }, that the pod didn't tolerate.
どうもこれっぽい
やはりディスク容量を減らしたのがまずかったか...?と思いつつ そもそも何でこんなにストレージ使っているんだ．．．？となる
ノードに異常がないか調べる
taka@doktor:~/doktor$ kubectl describe node doktor
Name: doktor
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=doktor
kubernetes.io/os=linux
microk8s.io/cluster=true
Annotations: node.alpha.kubernetes.io/ttl: 0
projectcalico.org/IPv4Address: 192.168.100.146/24
projectcalico.org/IPv4VXLANTunnelAddr: 10.1.182.128
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Thu, 25 Mar 2021 03:09:39 +0000
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: doktor
AcquireTime: <unset>
RenewTime: Wed, 01 Sep 2021 05:48:58 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Sun, 01 Aug 2021 21:11:07 +0000 Sun, 01 Aug 2021 21:11:07 +0000 CalicoIsUp Calico is running on this node
MemoryPressure False Wed, 01 Sep 2021 05:48:50 +0000 Mon, 28 Jun 2021 06:51:57 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 01 Sep 2021 05:48:50 +0000 Wed, 01 Sep 2021 05:39:08 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 01 Sep 2021 05:48:50 +0000 Mon, 28 Jun 2021 06:51:57 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Wed, 01 Sep 2021 05:48:50 +0000 Wed, 01 Sep 2021 02:28:41 +0000 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 192.168.100.146
Hostname: doktor
Capacity:
cpu: 6
ephemeral-storage: 25155844Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 24630512Ki
pods: 110
Allocatable:
cpu: 6
ephemeral-storage: 24107268Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 24528112Ki
pods: 110
System Info:
Machine ID: 571af691e3f84ef787ea3fe436e48569
System UUID: bd9a4d56-9f62-a737-fcbf-bfc9b70c9d27
Boot ID: 9671e1fa-965e-47ff-8ad0-67ab220dbb89
Kernel Version: 5.4.0-80-generic
OS Image: Ubuntu 20.04.2 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.3.7
Kubelet Version: v1.20.9-34+90ae618987e17a
Kube-Proxy Version: v1.20.9-34+90ae618987e17a
Non-terminated Pods: (33 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kubeless helo2-79c6989cd7-ssgcc 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21d
istio-system istio-citadel-67658cf6c-jvhpw 10m (0%) 0 (0%) 0 (0%) 0 (0%) 24d
kube-system calico-kube-controllers-847c8c99d-sthkk 0 (0%) 0 (0%) 0 (0%) 0 (0%) 160d
knative-eventing imc-dispatcher-79cd9dd48-rlnfw 0 (0%) 0 (0%) 0 (0%) 0 (0%) 69d
knative-eventing eventing-controller-d59b7cb87-jvlqp 100m (1%) 0 (0%) 100Mi (0%) 0 (0%) 21d
istio-system istio-galley-567478fb94-68h89 10m (0%) 0 (0%) 0 (0%) 0 (0%) 21d
knative-serving webhook-5f755b99b9-psv67 100m (1%) 500m (8%) 100Mi (0%) 500Mi (2%) 22d
kube-system coredns-86f78bb79c-gv2c6 100m (1%) 0 (0%) 70Mi (0%) 170Mi (0%) 140d
istio-system prometheus-9d65f7646-bpnqj 10m (0%) 0 (0%) 0 (0%) 0 (0%) 21d
knative-serving networking-istio-85dcb86cc7-rghvt 30m (0%) 300m (5%) 40Mi (0%) 400Mi (1%) 21d
istio-system istio-pilot-64d96677f8-vhjtf 20m (0%) 2 (33%) 140Mi (0%) 1Gi (4%) 21d
knative-serving istio-webhook-748fb797bd-wt6dz 20m (0%) 200m (3%) 20Mi (0%) 200Mi (0%) 21d
istio-system istio-policy-58d8b97644-ljccj 20m (0%) 2 (33%) 140Mi (0%) 1Gi (4%) 21d
kubeless hello3-6874cbb67b-gjc9w 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21d
knative-serving autoscaler-55b786c8b8-c2jh7 100m (1%) 1 (16%) 100Mi (0%) 1000Mi (4%) 21d
kubeless kubeless-controller-manager-5d9bf7f5c6-lv2lp 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21d
kube-system metrics-server-8bbfb4bdb-24lmn 0 (0%) 0 (0%) 0 (0%) 0 (0%) 22d
knative-eventing imc-controller-5fdd59678-hs9vr 0 (0%) 0 (0%) 0 (0%) 0 (0%) 22d
istio-system istio-telemetry-64d8c69d67-qgrjb 60m (1%) 6800m (113%) 140Mi (0%) 5073741824 (20%) 21d
knative-serving controller-5f47f4d7c5-snmmf 100m (1%) 1 (16%) 100Mi (0%) 1000Mi (4%) 21d
istio-system istio-egressgateway-756f9bc5b9-hzfkm 10m (0%) 2 (33%) 40Mi (0%) 1Gi (4%) 21d
knative-eventing eventing-webhook-cff75c955-hwh9h 100m (1%) 200m (3%) 50Mi (0%) 200Mi (0%) 21d
istio-system istio-sidecar-injector-589988b5d6-v4xss 10m (0%) 0 (0%) 0 (0%) 0 (0%) 21d
istio-system kiali-77f97f5b4d-7zlsk 10m (0%) 0 (0%) 0 (0%) 0 (0%) 32d
istio-system istio-tracing-579d7647d9-qskt4 10m (0%) 0 (0%) 0 (0%) 0 (0%) 21d
kubeless ui-8678d46bc4-6thkp 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21d
istio-system istio-ingressgateway-56bb766b96-769x9 10m (0%) 2 (33%) 40Mi (0%) 1Gi (4%) 21d
default doktor-search-6f6975987f-cvh25 10m (0%) 2 (33%) 40Mi (0%) 1Gi (4%) 13m
istio-system grafana-d7994566f-n8njv 10m (0%) 0 (0%) 0 (0%) 0 (0%) 15m
default doktor-upload-68cdc69f9c-mvnjr 10m (0%) 2 (33%) 40Mi (0%) 1Gi (4%) 13m
default web-7d6c7f746c-f9tpj 10m (0%) 2 (33%) 40Mi (0%) 1Gi (4%) 13m
knative-serving activator-8586b94b8b-s4p7v 300m (5%) 1 (16%) 60Mi (0%) 600Mi (2%) 26m
kube-system calico-node-x644t 250m (4%) 0 (0%) 0 (0%) 0 (0%) 160d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 1420m (23%) 25 (416%)
memory 1260Mi (5%) 16462538Ki (67%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal NodeHasDiskPressure 55m kubelet Node doktor status is now: NodeHasDiskPressure
Warning FreeDiskSpaceFailed 10m (x29 over 150m) kubelet (combined from similar events): failed to garbage collect required amount of images. Wanted to free 3900494643 bytes, but freed 628928551 bytes
Warning ImageGCFailed 51s (x31 over 150m) kubelet (combined from similar events): failed to garbage collect required amount of images. Wanted to free 3291001651 bytes, but freed 77027263 bytes
taka@doktor:~/doktor$
Events:
Disk周りでガッツリイベント起きてる
ので dfる
Type Reason Age From Message
---- ------ ---- ---- -------
Normal NodeHasDiskPressure 55m kubelet Node doktor status is now: NodeHasDiskPressure
Warning FreeDiskSpaceFailed 10m (x29 over 150m) kubelet (combined from similar events): failed to garbage collect required amount of images. Wanted to free 3900494643 bytes, but freed 628928551 bytes
Warning ImageGCFailed 51s (x31 over 150m) kubelet (combined from similar events): failed to garbage collect required amount of images. Wanted to free 3291001651 bytes, but freed 77027263 bytes
taka@doktor:~/doktor$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 12G 0 12G 0% /dev
tmpfs 2.4G 3.2M 2.4G 1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv 24G 21G 1.8G 93% /
tmpfs 12G 0 12G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 12G 0 12G 0% /sys/fs/cgroup
/dev/sda2 976M 202M 707M 23% /boot
/dev/loop0 100M 100M 0 100% /snap/core/11420
/dev/loop2 11M 11M 0 100% /snap/kubectl/2016
/dev/loop3 56M 56M 0 100% /snap/core18/2074
/dev/loop4 68M 68M 0 100% /snap/lxd/20326
/dev/loop5 71M 71M 0 100% /snap/lxd/21029
/dev/loop1 132M 132M 0 100% /snap/docker/796
/dev/loop6 56M 56M 0 100% /snap/core18/2128
/dev/loop8 212M 212M 0 100% /snap/microk8s/2338
/dev/loop10 33M 33M 0 100% /snap/snapd/12704
/dev/loop11 50M 50M 0 100% /snap/prometheus/32
/dev/loop12 212M 212M 0 100% /snap/microk8s/2361
/dev/loop14 33M 33M 0 100% /snap/snapd/12883
/dev/loop15 100M 100M 0 100% /snap/core/11606
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/2c8ef1e01794053c27b6adadcfbd0657f562abef8fefd8bf5f86f48661823cff/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/c4823b0b3bee08c6923729a76f566f0fe1a73e119c61f8dad675d335fb3e4086/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/fcb29d88de321adba4eb5ca8502e8159f99d35efa787793cbb56fffeac9d1759/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/24227714ece802b39ddc1c36431377e9b147d0047f1b47212428ac2e5f2000be/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/49cf0d79d1565aed56165a82eae6e7dcd7be7624a476a181ae8fa3fbb8a1bd71/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/5b9f4adeeb3309af23b07db8f7479451a8356c174c5346e2ea43d0460a2d87b7/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/b5c0da1965edec285366a952a83b3c1a8eddb94b4720b68fc471210bfa1cc84f/rootfs
/dev/loop7 62M 62M 0 100% /snap/core20/1081
/dev/loop13 11M 11M 0 100% /snap/kubectl/2024
/dev/loop9 60M 60M 0 100% /snap/prometheus/53
/dev/loop16 117M 117M 0 100% /snap/docker/1125
tmpfs 2.4G 0 2.4G 0% /run/user/1003
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/d18c63f3ac29cd5e3ef770678bbafbbfd6608b6cd4d31af0968b68d54be64a7c/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/20ca04663d1c27bc2d0a4d690876da9b2837690555e64507cd7762147b091ba4/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/20ca04663d1c27bc2d0a4d690876da9b2837690555e64507cd7762147b091ba4/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/133a351fd8910ecefd1709ea08d4ca1bfc57f6d9ca996796224239085508c634/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/133a351fd8910ecefd1709ea08d4ca1bfc57f6d9ca996796224239085508c634/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/15f4feb16b4e0cae32e8b89a477b4cc16aea2650f54c6c7abf5a2a55f7713e15/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/679df87b2124bb802146cee4ec64a0f18dd14d5d6d843b7b2d55d2eddb22d70a/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/f061a4ad8f8288e02b02f4564973b63fe6f24413b97131022ddc877206a6c7c3/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/f061a4ad8f8288e02b02f4564973b63fe6f24413b97131022ddc877206a6c7c3/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/3e51cbd7c1e1e8a43ffd7ade10f75c0fc717d218f34c39a8c98ef01f837d15ae/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/a1c208809a5f70d2a418e058cdbdb887f1be0454327fc52b9e827332289ea627/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/a1c208809a5f70d2a418e058cdbdb887f1be0454327fc52b9e827332289ea627/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/0a5f160e45082890aa920a4a6678fe3407908bad038e2ca03acb67cd831ab3ee/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/3299d11b1f670b453f94e221c692b9cb0d09fe750d42a38b615413ab19671c0a/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/3299d11b1f670b453f94e221c692b9cb0d09fe750d42a38b615413ab19671c0a/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/9c574624dc447cc693a0560c1ddbd456808113cf7cd41ae1ec84956258dab838/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/8378d6be0394eaf081f66c2d49c808b2a3472985db1724e14065b187ee69a224/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/8378d6be0394eaf081f66c2d49c808b2a3472985db1724e14065b187ee69a224/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/4d5bbd61187712ba31121548c2aab853259a5145fb0179ae32911fb87db13de0/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/1e6909dcf243004d8c92f0dfcf74290d6845b0ee4a1a48304a01a2aab282d9c9/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/1e6909dcf243004d8c92f0dfcf74290d6845b0ee4a1a48304a01a2aab282d9c9/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/51a706faacf9c794f05521bf56dbf42d7d8bbfde4193ec215ac38520aa080d97/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/c7f51f7db7f9921f0829777a4e1507fb19508e95857feb54466b424a154b4586/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/c7f51f7db7f9921f0829777a4e1507fb19508e95857feb54466b424a154b4586/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/12d8ff43b5254a89acea423878c0c1f4cb91efbda3545e863af8e4da1ef37847/rootfs
shm 64M 4.0K 64M 1% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/36e06b59a295a77d4ff47a4a10a8ae65287da11015f2c0d3cdbe761e05e74590/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/36e06b59a295a77d4ff47a4a10a8ae65287da11015f2c0d3cdbe761e05e74590/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/9d9e01884d673d3b0e1f6f2a1dfaef05355a4c6794830058cd15c3a6ad3ec291/rootfs
shm 64M 4.0K 64M 1% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/3f60c605843ba7e13a4b9ae1ff19bece5ddd5df7d39d77675449576da6ac0e08/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/3f60c605843ba7e13a4b9ae1ff19bece5ddd5df7d39d77675449576da6ac0e08/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/00c2384b126cb08533faf20b11535911f86f0c5125c4fb60a1f2bbd80984658f/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/4a84170ef2d045651aba04492b83b33afd576e612a3290db13487f6d0f4e307d/rootfs
shm 64M 4.0K 64M 1% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/13144481f574f9d7cb5e12bb31b901952bdbfd9faa75508ceb8006a7cbd7f79b/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/13144481f574f9d7cb5e12bb31b901952bdbfd9faa75508ceb8006a7cbd7f79b/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/27263c28964ba7e6ed89abe4e73bcfce4dc27c8f06426d83d0062d7f266877e0/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/19bd244f6a491f74a8dfd355280fe08dbfc38f852d4b2f52378408a476c378f1/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/19bd244f6a491f74a8dfd355280fe08dbfc38f852d4b2f52378408a476c378f1/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/cf041224f24e28f21edf0035da0d59952c1d0a1b3c17a4bdc604bae65f7eedaa/rootfs
shm 64M 4.0K 64M 1% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/98c7906fdc6e921202652153fa6b34ddec9c87ea81e6b12704379bfc3ccf459f/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/98c7906fdc6e921202652153fa6b34ddec9c87ea81e6b12704379bfc3ccf459f/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/9e580bc57de62a185f83899512ea07f90ba8172565070b45d628c1418a741254/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/99f6c4d87dc5f746a7fb67ce9fc27f1f39078a8f4cd6bf5238d69cc248cf7942/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/99f6c4d87dc5f746a7fb67ce9fc27f1f39078a8f4cd6bf5238d69cc248cf7942/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/82dd9e0eeca729d4f3a835f2cdd6a42741d183f95ad7d5232373e4cb6fc57e98/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/63fdae58b5d95bb9eef007d726d1f13f8930973c7f1fe41cc94d7d5422da9857/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/63fdae58b5d95bb9eef007d726d1f13f8930973c7f1fe41cc94d7d5422da9857/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/0152b45f097e7e4723c9f460952ebdf7dfcfcab4761d90e7ee2e3f24318d37cd/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/dfd3bbaf46b32ac6c9f3bed5adc4e082256257bee6cf7350728d3c21fec025d7/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/dfd3bbaf46b32ac6c9f3bed5adc4e082256257bee6cf7350728d3c21fec025d7/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/0725b7585dee166b5c0de4fdf6e8b9703180838cc056a66a5287990a16b44b0c/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/4c0c81a4be14ef0d709fe5c1f790c1a3378c04ada3fe701b4c5c9ba203fa534c/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/4c0c81a4be14ef0d709fe5c1f790c1a3378c04ada3fe701b4c5c9ba203fa534c/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/8bd7c667750a0bf0f15c7ded67a1c621a044cdc1f3620755c741adb34154da59/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/a9bff3057ec3537605a4377ceca6cf0e5380ab15913551b4a801e84fe758e1f4/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/a8dde4ad85af0e02f9d119e0b4bcb7f82145d513f6f359ae661d18083d4fdaf4/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/a8dde4ad85af0e02f9d119e0b4bcb7f82145d513f6f359ae661d18083d4fdaf4/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/f9b5b1ad0266cba7a69fdf92b27ec5fab2d623eb8495c3f1bf78d0a7e7d8f64b/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/e554d7ed317b1cb9f809fe5175bd2db7c180409b2fe1eeaee408214c0db456a9/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/34c6a3e851814f0d90bf8ab30b34b6b4390776e8ff1b75485c04fa53018344c7/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/907ffd4cda62ae0cf3bb5967a56e0f27dcc4417e410b86947f1c7ef63b91a9ef/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/907ffd4cda62ae0cf3bb5967a56e0f27dcc4417e410b86947f1c7ef63b91a9ef/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/3f3427c0506d8fe6c66e66f08628b754bf55021374267b54aef7aa1f071b70dd/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/2fe7880534df72454bfa485ba88991dd7fc5fb6c062a39b9eb7bce8d69c0959a/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/2fe7880534df72454bfa485ba88991dd7fc5fb6c062a39b9eb7bce8d69c0959a/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/c468906bd77005aa526836b3a5b00d5cbef2083d2442ac2d517d2cb0d8200d94/rootfs
shm 64M 4.0K 64M 1% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/af12849ef51230caf82266272403c3d1abcb0b8d50bbbcffe0316ad3420ddb2d/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/af12849ef51230caf82266272403c3d1abcb0b8d50bbbcffe0316ad3420ddb2d/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/929f98d1c603a962a399e89e9db86828f357b6a13e72780e551b2a926a94d57d/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/735461b199700636ae26a4347c9c78bc4c78f23df655e3ba925d3654863103ac/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/735461b199700636ae26a4347c9c78bc4c78f23df655e3ba925d3654863103ac/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/75f90ea791a7a1e226ae82af8d15fdb74a8addfcc041f97032d54c7c666ca991/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/4c951e296e56da5fcc0126ee2bd8e114f0e1e558421344e5fc4f90b45af226aa/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/2c86295c02b8a8ce7c9d056afd86a1e693b8bb8a2e767569e9543f30f58d536a/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/2c86295c02b8a8ce7c9d056afd86a1e693b8bb8a2e767569e9543f30f58d536a/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/b1fa3993c49eb2b1ee54c75f12fd1ddbbea57fb6f9e7a124e1f3064c04839db2/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/df92fdcd9b86102ae59c9f3a85605af22c2bf54aaaaf9b9a16d842a87b9d2957/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/df92fdcd9b86102ae59c9f3a85605af22c2bf54aaaaf9b9a16d842a87b9d2957/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/7c71ae5eb4f2efff345da90f062cd794047cab1366de9d78b3f90dda9bcbefa4/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/7c71ae5eb4f2efff345da90f062cd794047cab1366de9d78b3f90dda9bcbefa4/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/206b995c48ef7db52e0dcd5bd810bd1cfa56381fbcdf43550d7e2e818214d275/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/85b536488ee6b99bc2c224d50be1ded2e7c54f7c870302078e8ca1fe78233ce4/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/85b536488ee6b99bc2c224d50be1ded2e7c54f7c870302078e8ca1fe78233ce4/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/343ca50dda605f02cbd1ddb299f88398d0af7d290cdc95fd502f5925894ed70b/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/6395d76b5d254045642f0adca06d9c1889d7c768293c20bc9a57235510aa8691/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/6395d76b5d254045642f0adca06d9c1889d7c768293c20bc9a57235510aa8691/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/32b3a2cf63f30f8667d5e13e7e6c96ea0e25dfdc975239477f36deac4336a79f/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/289df5e4a3ca5d146a4f1c161c23bb2b6f7ad7756926c0dc774d7e70b46aecf4/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/0015f055497dab2deb901b086cd9270e4494fb140129faaa68332e6568633dbc/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/c3bb750181dca99c2c64cfad96eeac65577d0f87e29f557bfddb847f0fc668b5/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/c3bb750181dca99c2c64cfad96eeac65577d0f87e29f557bfddb847f0fc668b5/rootfs
shm 64M 4.0K 64M 1% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/edfe2ba6be6042880d2ff795cc8d7b1ba937bdb13a439045fbb01eafe5dea679/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/edfe2ba6be6042880d2ff795cc8d7b1ba937bdb13a439045fbb01eafe5dea679/rootfs
shm 64M 4.0K 64M 1% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/5b7a62df73e8a28ab6a7e1c4f6006a36422660a43116eca4952d5790722b9a3b/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/5b7a62df73e8a28ab6a7e1c4f6006a36422660a43116eca4952d5790722b9a3b/rootfs
shm 64M 4.0K 64M 1% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/a15087a556e93397038c288e7836b8894d258249b939eb4d6a44fad6f3295476/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/a15087a556e93397038c288e7836b8894d258249b939eb4d6a44fad6f3295476/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/b39322c49b718d53d3aad53a31535932a7d49f64988864bcea1cd2aadb338363/shm
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/b39322c49b718d53d3aad53a31535932a7d49f64988864bcea1cd2aadb338363/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/5c771ab01e8e812abbf692f6b79680dd653d08c9d5a9b47d7fc82535d53ec943/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/ecc87dcb75f26ccce256ceff3baf0cd74547cb1c1808b5ccd8e1151f7b0a8d12/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/badd13b1f55fe25cf76c410c180888bc120c9e87b223babee504fbeb3790514a/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/f8802bdcd9fb4e257892711ddbb4e16a1d329b28a79bf3e802572fe5dc9f7136/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/53bdc94359510f1371ef3e29e0a6ed495e7f5de6a41ef373ef0bc1b9c4510d10/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/7cf503612fcb8c6ecf38b843e2c59a3972103febb209d6833325f37d5c345baf/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/33f8563b00fc6791ab8d19721197a9a588a157fcd2778bfe84bbbb79d5418021/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/e673b2db0220e23035899394ebc15d6f0faaa7fba5bd612e9494a60acd008c28/rootfs
overlay 24G 21G 1.8G 93% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/{idらしきもの}/rootfs
めっちゃ多くね???
Knativeのイメージが全て保存されているっぽい¶
https://stackoverflow.com/questions/63803171/how-to-change-microk8s-kubernetes-storage-location/63807902 同様のケースを発見
https://discuss.kubernetes.io/t/question-about-overlay-rootfs-shm-and-dev-loopx/13741
とりあえずmicrok8sをstop->startしてみる¶
https://discuss.kubernetes.io/t/question-about-overlay-rootfs-shm-and-dev-loopx/13741/3 だめだった
rebootしたら解決した
taka@doktor:~$ k describe node doktor
Name: doktor
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=doktor
kubernetes.io/os=linux
microk8s.io/cluster=true
Annotations: node.alpha.kubernetes.io/ttl: 0
projectcalico.org/IPv4Address: 192.168.100.146/24
projectcalico.org/IPv4VXLANTunnelAddr: 10.1.182.128
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Thu, 25 Mar 2021 03:09:39 +0000
Taints: node.kubernetes.io/not-ready:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: doktor
AcquireTime: <unset>
RenewTime: Wed, 01 Sep 2021 06:38:27 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Sun, 01 Aug 2021 21:11:07 +0000 Sun, 01 Aug 2021 21:11:07 +0000 CalicoIsUp Calico is running on this node
MemoryPressure False Wed, 01 Sep 2021 06:37:57 +0000 Wed, 01 Sep 2021 06:35:40 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 01 Sep 2021 06:37:57 +0000 Wed, 01 Sep 2021 06:35:40 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 01 Sep 2021 06:37:57 +0000 Wed, 01 Sep 2021 06:35:40 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready False Wed, 01 Sep 2021 06:37:57 +0000 Wed, 01 Sep 2021 06:37:57 +0000 KubeletNotReady container runtime status check may not have completed yet
Addresses:
InternalIP: 192.168.100.146
Hostname: doktor
Capacity:
cpu: 6
ephemeral-storage: 25155844Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 24630512Ki
pods: 110
Allocatable:
cpu: 6
ephemeral-storage: 25155844Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 24528112Ki
pods: 110
System Info:
Machine ID: 571af691e3f84ef787ea3fe436e48569
System UUID: bd9a4d56-9f62-a737-fcbf-bfc9b70c9d27
Boot ID: 8f37a030-16f9-477e-9b28-b42f39b8236e
Kernel Version: 5.4.0-80-generic
OS Image: Ubuntu 20.04.2 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.3.7
Kubelet Version: v1.20.9-34+90ae618987e17a
Kube-Proxy Version: v1.20.9-34+90ae618987e17a
Non-terminated Pods: (33 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system calico-kube-controllers-847c8c99d-sthkk 0 (0%) 0 (0%) 0 (0%) 0 (0%) 160d
kubeless hello3-6874cbb67b-gjc9w 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21d
knative-serving controller-5f47f4d7c5-snmmf 100m (1%) 1 (16%) 100Mi (0%) 1000Mi (4%) 21d
kube-system metrics-server-8bbfb4bdb-24lmn 0 (0%) 0 (0%) 0 (0%) 0 (0%) 22d
kubeless helo2-79c6989cd7-ssgcc 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21d
knative-serving activator-8586b94b8b-s4p7v 300m (5%) 1 (16%) 60Mi (0%) 600Mi (2%) 75m
knative-serving istio-webhook-748fb797bd-wt6dz 20m (0%) 200m (3%) 20Mi (0%) 200Mi (0%) 21d
kubeless ui-8678d46bc4-6thkp 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21d
kubeless kubeless-controller-manager-5d9bf7f5c6-lv2lp 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21d
istio-system istio-citadel-67658cf6c-jvhpw 10m (0%) 0 (0%) 0 (0%) 0 (0%) 24d
istio-system istio-sidecar-injector-589988b5d6-v4xss 10m (0%) 0 (0%) 0 (0%) 0 (0%) 21d
knative-serving networking-istio-85dcb86cc7-rghvt 30m (0%) 300m (5%) 40Mi (0%) 400Mi (1%) 21d
knative-eventing eventing-controller-d59b7cb87-jvlqp 100m (1%) 0 (0%) 100Mi (0%) 0 (0%) 21d
istio-system istio-policy-58d8b97644-ljccj 20m (0%) 2 (33%) 140Mi (0%) 1Gi (4%) 21d
istio-system istio-galley-567478fb94-68h89 10m (0%) 0 (0%) 0 (0%) 0 (0%) 21d
istio-system istio-telemetry-64d8c69d67-qgrjb 60m (1%) 6800m (113%) 140Mi (0%) 5073741824 (20%) 21d
default doktor-upload-68cdc69f9c-mvnjr 10m (0%) 2 (33%) 40Mi (0%) 1Gi (4%) 62m
istio-system istio-ingressgateway-56bb766b96-769x9 10m (0%) 2 (33%) 40Mi (0%) 1Gi (4%) 21d
istio-system kiali-77f97f5b4d-7zlsk 10m (0%) 0 (0%) 0 (0%) 0 (0%) 32d
default web-7d6c7f746c-f9tpj 10m (0%) 2 (33%) 40Mi (0%) 1Gi (4%) 62m
istio-system istio-egressgateway-756f9bc5b9-hzfkm 10m (0%) 2 (33%) 40Mi (0%) 1Gi (4%) 21d
default doktor-search-6f6975987f-cvh25 10m (0%) 2 (33%) 40Mi (0%) 1Gi (4%) 62m
kube-system calico-node-x644t 250m (4%) 0 (0%) 0 (0%) 0 (0%) 160d
knative-eventing imc-controller-5fdd59678-hs9vr 0 (0%) 0 (0%) 0 (0%) 0 (0%) 22d
knative-eventing eventing-webhook-cff75c955-hwh9h 100m (1%) 200m (3%) 50Mi (0%) 200Mi (0%) 21d
knative-serving webhook-5f755b99b9-psv67 100m (1%) 500m (8%) 100Mi (0%) 500Mi (2%) 22d
knative-eventing imc-dispatcher-79cd9dd48-rlnfw 0 (0%) 0 (0%) 0 (0%) 0 (0%) 69d
istio-system istio-pilot-64d96677f8-vhjtf 20m (0%) 2 (33%) 140Mi (0%) 1Gi (4%) 21d
istio-system istio-tracing-579d7647d9-qskt4 10m (0%) 0 (0%) 0 (0%) 0 (0%) 21d
istio-system grafana-d7994566f-n8njv 10m (0%) 0 (0%) 0 (0%) 0 (0%) 64m
kube-system coredns-86f78bb79c-gv2c6 100m (1%) 0 (0%) 70Mi (0%) 170Mi (0%) 140d
knative-serving autoscaler-55b786c8b8-c2jh7 100m (1%) 1 (16%) 100Mi (0%) 1000Mi (4%) 21d
istio-system prometheus-9d65f7646-bpnqj 10m (0%) 0 (0%) 0 (0%) 0 (0%) 21d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 1420m (23%) 25 (416%)
memory 1260Mi (5%) 16462538Ki (67%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FreeDiskSpaceFailed 60m (x29 over 3h20m) kubelet (combined from similar events): failed to garbage collect required amount of images. Wanted to free 3900494643 bytes, but freed 628928551 bytes
Warning ImageGCFailed 40m (x33 over 3h20m) kubelet (combined from similar events): failed to garbage collect required amount of images. Wanted to free 3311231795 bytes, but freed 77027263 bytes
Normal Starting 35m kube-proxy Starting kube-proxy.
Normal NodeHasSufficientMemory 35m kubelet Node doktor status is now: NodeHasSufficientMemory
Normal Starting 35m kubelet Starting kubelet.
Warning InvalidDiskCapacity 35m kubelet invalid capacity 0 on image filesystem
Normal NodeHasNoDiskPressure 35m kubelet Node doktor status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 35m kubelet Node doktor status is now: NodeHasSufficientPID
Normal NodeNotReady 35m kubelet Node doktor status is now: NodeNotReady
Normal NodeAllocatableEnforced 32m kubelet Updated Node Allocatable limit across pods
Normal NodeReady 32m kubelet Node doktor status is now: NodeReady
Warning ImageGCFailed 30m kubelet failed to garbage collect required amount of images. Wanted to free 2706400051 bytes, but freed 77027263 bytes
Warning FreeDiskSpaceFailed 30m kubelet failed to garbage collect required amount of images. Wanted to free 2706400051 bytes, but freed 77027263 bytes
Warning ImageGCFailed 25m kubelet failed to garbage collect required amount of images. Wanted to free 2701353779 bytes, but freed 77027263 bytes
Warning FreeDiskSpaceFailed 25m kubelet failed to garbage collect required amount of images. Wanted to free 2701353779 bytes, but freed 77027263 bytes
Warning ImageGCFailed 20m kubelet failed to garbage collect required amount of images. Wanted to free 2735158067 bytes, but freed 0 bytes
Warning FreeDiskSpaceFailed 20m kubelet failed to garbage collect required amount of images. Wanted to free 2735158067 bytes, but freed 0 bytes
Warning ImageGCFailed 15m kubelet failed to garbage collect required amount of images. Wanted to free 2736378675 bytes, but freed 77027263 bytes
Warning FreeDiskSpaceFailed 15m kubelet failed to garbage collect required amount of images. Wanted to free 2736378675 bytes, but freed 77027263 bytes
Warning ImageGCFailed 10m kubelet failed to garbage collect required amount of images. Wanted to free 2729595699 bytes, but freed 0 bytes
Warning FreeDiskSpaceFailed 10m kubelet failed to garbage collect required amount of images. Wanted to free 2729595699 bytes, but freed 0 bytes
Normal Starting 2m56s kube-proxy Starting kube-proxy.
Warning InvalidDiskCapacity 2m52s kubelet invalid capacity 0 on image filesystem
Normal Starting 2m52s kubelet Starting kubelet.
Normal NodeAllocatableEnforced 2m52s kubelet Updated Node Allocatable limit across pods
Normal NodeHasSufficientMemory 2m51s (x2 over 2m52s) kubelet Node doktor status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 2m51s (x2 over 2m52s) kubelet Node doktor status is now: NodeHasNoDiskPressure
Normal NodeReady 2m51s kubelet Node doktor status is now: NodeReady
Normal NodeHasSufficientPID 2m51s (x2 over 2m52s) kubelet Node doktor status is now: NodeHasSufficientPID
Normal Starting 39s kube-proxy Starting kube-proxy.
Warning InvalidDiskCapacity 36s kubelet invalid capacity 0 on image filesystem
Normal NodeHasSufficientMemory 36s kubelet Node doktor status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 36s kubelet Node doktor status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 36s kubelet Node doktor status is now: NodeHasSufficientPID
Normal Starting 36s kubelet Starting kubelet.
Normal NodeNotReady 34s kubelet Node doktor status is now: NodeNotReady
Normal Starting <invalid> kube-proxy Starting kube-proxy.
Warning InvalidDiskCapacity <invalid> kubelet invalid capacity 0 on image filesystem
Normal NodeAllocatableEnforced <invalid> kubelet Updated Node Allocatable limit across pods
Normal NodeHasSufficientPID <invalid> kubelet Node doktor status is now: NodeHasSufficientPID
Normal NodeHasNoDiskPressure <invalid> kubelet Node doktor status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientMemory <invalid> kubelet Node doktor status is now: NodeHasSufficientMemory
Warning Rebooted <invalid> kubelet Node doktor has been rebooted, boot id: 8f37a030-16f9-477e-9b28-b42f39b8236e
Normal Starting <invalid> kubelet Starting kubelet.
Normal Starting <invalid> kube-proxy Starting kube-proxy.
taka@doktor:~$ kall
NAME READY STATUS RESTARTS AGE
pod/doktor-upload-68cdc69f9c-mvnjr 2/2 Running 4 64m
pod/web-7d6c7f746c-f9tpj 2/2 Running 4 64m
pod/doktor-search-6f6975987f-cvh25 2/2 Running 4 64m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 160d
service/web NodePort 10.152.183.212 <none> 5000:30080/TCP 64m
service/doktor-upload NodePort 10.152.183.153 <none> 3000:30010/TCP 64m
service/doktor-search LoadBalancer 10.152.183.203 <pending> 4000:31307/TCP 64m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/doktor-upload 1/1 1 1 64m
deployment.apps/web 1/1 1 1 64m
deployment.apps/doktor-search 1/1 1 1 64m
NAME DESIRED CURRENT READY AGE
replicaset.apps/doktor-upload-68cdc69f9c 1 1 1 64m
replicaset.apps/web-7d6c7f746c 1 1 1 64m
replicaset.apps/doktor-search-6f6975987f 1 1 1 64m
つまり....何でやねん.....
追記(野木) Evictedが出た時の対処法 https://cdsl-tut.esa.io/wiki/584