Kubernetes中Nginx服务启动失败排查流程分析(Error: ImagePullBackOff)_F11

分享到

Kubernetes中Nginx服务启动失败排查流程分析(Error: ImagePullBackOff)

nginx 来源：互联网作者：佚名发布时间：2023-09-25 22:12:36 人浏览

摘要

?pod节点启动失败，nginx服务无法正常访问，服务状态显示为ImagePullBackOff。 1 2 3 [root@m1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-f89759699-cgjgp 0/1 ImagePullBackOff 0 103m ????查看nginx服务的

?pod节点启动失败，nginx服务无法正常访问，服务状态显示为ImagePullBackOff。

[root@m1 ~]# kubectl get pods

NAME READY STATUS RESTARTS AGE

nginx-f89759699-cgjgp 0/1 ImagePullBackOff 0 103m

????查看nginx服务的Pod节点详细信息。

[root@m1 ~]# kubectl describe pod nginx-f89759699-cgjgp

Name: nginx-f89759699-cgjgp

Namespace: default

Priority: 0

Service Account: default

Node: n1/192.168.200.84

Start Time: Fri, 10 Mar 2023 08:40:33 +0800

Labels: app=nginx

pod-template-hash=f89759699

Annotations: <none>

Status: Pending

IP: 10.244.3.20

IPs:

IP: 10.244.3.20

Controlled By: ReplicaSet/nginx-f89759699

Containers:

nginx:

Container ID:

Image: nginx

Image ID:

Port: <none>

Host Port: <none>

State: Waiting

Reason: ImagePullBackOff

Ready: False

Restart Count: 0

Environment: <none>

Mounts:

/var/run/secrets/kubernetes.io/serviceaccount from default-token-zk8sj (ro)

Conditions:

Type Status

Initialized True

Ready False

ContainersReady False

PodScheduled True

Volumes:

default-token-zk8sj:

Type: Secret (a volume populated by a Secret)

SecretName: default-token-zk8sj

Optional: false

QoS Class: BestEffort

Node-Selectors: <none>

Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s

node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

Events:

Type Reason Age From Message

---- ------ ---- ---- -------

Normal BackOff 57m (x179 over 100m) kubelet Back-off pulling image "nginx"

Normal Pulling 7m33s (x22 over 100m) kubelet Pulling image "nginx"

Warning Failed 2m30s (x417 over 100m) kubelet Error: ImagePullBackOff

发现，获取nginx镜像失败。可能是由于Docker服务引起的。

于是，检查Docker是否正常启动

1	systemctl status docker

发现，docker服务启动失败????，手动尝试重新启动。

1	systemctl restart docker

但是，重启docker服务失败，出现如下报错信息。

[root@m1 ~]# systemctl restart docker

Job for docker.service failed because the control process exited with error code.

See "systemctl status docker.service" and "journalctl -xe" for details.

执行systemctl restart docker命令失效。

接着，当执行docker version命令时，发现未能连接到Docker daemon

[root@m1 ~]# docker version

Client: Docker Engine - Community

Version: 20.10.17

API version: 1.41

Go version: go1.17.11

Git commit: 100c701

Built: Mon Jun 6 23:03:11 2022

OS/Arch: linux/amd64

Context: default

Experimental: true

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

于是，再次通过执行systemctl status docker命令，查看docker服务未能启动，阅读输出报错信息，如下所示。

[root@m1 ~]# systemctl status docker

● docker.service - Docker Application Container Engine

Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)

Active: failed (Result: exit-code) since Fri 2023-03-10 10:28:16 CST; 4min 35s ago

Docs: https://docs.docker.com

Main PID: 2221 (code=exited, status=1/FAILURE)

Mar 10 10:28:13 m1 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE

Mar 10 10:28:13 m1 systemd[1]: docker.service: Failed with result 'exit-code'.

Mar 10 10:28:13 m1 systemd[1]: Failed to start Docker Application Container Engine.

Mar 10 10:28:16 m1 systemd[1]: docker.service: Service RestartSec=2s expired, scheduling restart.

Mar 10 10:28:16 m1 systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.

Mar 10 10:28:16 m1 systemd[1]: Stopped Docker Application Container Engine.

Mar 10 10:28:16 m1 systemd[1]: docker.service: Start request repeated too quickly.

Mar 10 10:28:16 m1 systemd[1]: docker.service: Failed with result 'exit-code'.

Mar 10 10:28:16 m1 systemd[1]: Failed to start Docker Application Container Engine.

[root@m1 ~]#

通过上述输出显示，Docker 服务进程的启动失败，状态为 1/FAILURE。

?接下来，尝试通过以下步骤来排查和解决问题：

1??查看 Docker 服务日志：使用以下命令查看 Docker 服务日志，以便更详细地了解失败原因。

1	sudo journalctl -u docker.service

2?? 通过输出Ddocker日志分析，提取到了相关报错信息片段，发现是配置daemon中的/etc/docker/daemon.json配置文件出错导致的。

Mar 10 10:20:17 m1 systemd[1]: Starting Docker Application Container Engine...

Mar 10 10:20:17 m1 dockerd[1572]: unable to configure the Docker daemon with file /etc/docker/daemon.json: invalid character '"' after object key:value pair

Mar 10 10:20:17 m1 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE

Mar 10 10:20:17 m1 systemd[1]: docker.service: Failed with result 'exit-code'.

Mar 10 10:20:17 m1 systemd[1]: Failed to start Docker Application Container Engine.

Mar 10 10:20:19 m1 systemd[1]: docker.service: Service RestartSec=2s expired, scheduling restart.

Mar 10 10:20:19 m1 systemd[1]: docker.service: Scheduled restart job, restart counter is at 2.

Mar 10 10:20:19 m1 systemd[1]: Stopped Docker Application Container Engine.

3??此时，查看daemon配置文件/etc/docker/daemon.json是否配置正确。

[root@m1 ~]# cat /etc/docker/daemon.json

{

# 设置 Docker 镜像的注册表镜像源为阿里云镜像源。

"registry-mirrors": ["https://w2kavmmf.mirror.aliyuncs.com"]

# 指定 Docker 守护进程使用 systemd 作为 cgroup driver。

"exec-opts": ["native.cgroupdriver=systemd"]

}

咋一看，配置信息没有什么问题，都是正确的，但仔细一看，就会发现应该在"registry-mirrors"选项的结尾添加逗号。犯了缺少逗号（,）导致的语法错误，终于找到了问题根源。

????修改后：

[root@m1 ~]# cat /etc/docker/daemon.json

{

"registry-mirrors": ["https://w2kavmmf.mirror.aliyuncs.com"],

"exec-opts": ["native.cgroupdriver=systemd"]

}

[root@m1 ~]# cat /etc/docker/daemon.json

{

"registry-mirrors": ["https://w2kavmmf.mirror.aliyuncs.com"],

"exec-opts": ["native.cgroupdriver=systemd"]

}

按下:wq报错退出。

4?? 重新加载系统并重新启动Docker服务

systemctl daemon-reload

systemctl restart docker

systemctl status docker

5??检查docker版本信息是否输出正常

[root@m1 ~]# docket version

-bash: docket: command not found

[root@m1 ~]# docker version

Client: Docker Engine - Community

Version: 20.10.17

API version: 1.41

Go version: go1.17.11

Git commit: 100c701

Built: Mon Jun 6 23:03:11 2022

OS/Arch: linux/amd64

Context: default

Experimental: true

Server: Docker Engine - Community

Engine:

Version: 20.10.17

API version: 1.41 (minimum version 1.12)

Go version: go1.17.11

Git commit: a89b842

Built: Mon Jun 6 23:01:29 2022

OS/Arch: linux/amd64

Experimental: false

containerd:

Version: 1.6.6

GitCommit: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1

runc:

Version: 1.1.2

GitCommit: v1.1.2-0-ga916309

docker-init:

Version: 0.19.0

GitCommit: de40ad0

[root@m1 ~]# docker info

Client:

Context: default

Debug Mode: false

Plugins:

app: Docker App (Docker Inc., v0.9.1-beta3)

buildx: Docker Buildx (Docker Inc., v0.8.2-docker)

scan: Docker Scan (Docker Inc., v0.17.0)

Server:

Containers: 20

Running: 8

Paused: 0

Stopped: 12

Images: 20

Server Version: 20.10.17

Storage Driver: overlay2

Backing Filesystem: xfs

Supports d_type: true

Native Overlay Diff: true

userxattr: false

Logging Driver: json-file

Cgroup Driver: systemd

Cgroup Version: 1

Plugins:

Volume: local

Network: bridge host ipvlan macvlan null overlay

Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog

Swarm: inactive

Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc

Default Runtime: runc

Init Binary: docker-init

containerd version: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1

runc version: v1.1.2-0-ga916309

init version: de40ad0

Security Options:

seccomp

Profile: default

Kernel Version: 4.18.0-372.9.1.el8.x86_64

Operating System: Rocky Linux 8.6 (Green Obsidian)

OSType: linux

Architecture: x86_64

CPUs: 2

Total Memory: 9.711GiB

Name: m1

ID: 4YIS:FHSB:YXRI:CED5:PJSJ:EAS2:BCR3:GJJF:FDPK:EDJH:DVKU:AIYJ

Docker Root Dir: /var/lib/docker

Debug Mode: false

Registry: https://index.docker.io/v1/

Labels:

Experimental: false

Insecure Registries:

127.0.0.0/8

Registry Mirrors:

https://w2kavmmf.mirror.aliyuncs.com/

Live Restore Enabled: false

至此，Docker服务重启成功，pod节点恢复正常，Nginx服务能够正常访问。

[root@m1 ~]# kubectl get pods

NAME READY STATUS RESTARTS AGE

nginx-f89759699-cgjgp 1/1 Running 0 174m

查看pod详细信息，显示正常。

[root@m1 ~]# kubectl describe pod nginx-f89759699-cgjgp

Name: nginx-f89759699-cgjgp

Namespace: default

Priority: 0

Service Account: default

Node: n1/192.168.200.84

Start Time: Fri, 10 Mar 2023 08:40:33 +0800

Labels: app=nginx

pod-template-hash=f89759699

Annotations: <none>

Status: Running

IP: 10.244.3.20

IPs:

IP: 10.244.3.20

Controlled By: ReplicaSet/nginx-f89759699

Containers:

nginx:

Container ID: docker://88bdc2bfa592f60bf99bac2125b0adae005118ae8f2f271225245f20b7cfb3c8

Image: nginx

Image ID: docker-pullable://nginx@sha256:aa0afebbb3cfa473099a62c4b32e9b3fb73ed23f2a75a65ce1d4b4f55a5c2ef2

Port: <none>

Host Port: <none>

State: Running

Started: Fri, 10 Mar 2023 10:37:42 +0800

Ready: True

Restart Count: 0

Environment: <none>

Mounts:

/var/run/secrets/kubernetes.io/serviceaccount from default-token-zk8sj (ro)

Conditions:

Type Status

Initialized True

Ready True

ContainersReady True

PodScheduled True

Volumes:

default-token-zk8sj:

Type: Secret (a volume populated by a Secret)

SecretName: default-token-zk8sj

Optional: false

QoS Class: BestEffort

Node-Selectors: <none>

Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s

node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

Events:

Type Reason Age From Message

---- ------ ---- ---- -------

Normal BackOff 58m (x480 over 171m) kubelet Back-off pulling image "nginx"

[root@m1 ~]#

您可能感兴趣的文章 :

原文链接 :

Tag : nginx(66)Kubernetes(6)

nginx https 443端口配置的方法介绍

一丶登录阿里云或者腾讯云等域名控制器下载免费证书二丶点击nginx下载，并且上传到服务器三丶nginx配置https协议 1 2 3 4 5 6 7 8 9 10 11 12 1
Kubernetes中Nginx服务启动失败排查流程分析(Error:

?pod节点启动失败，nginx服务无法正常访问，服务状态显示为ImagePullBackOff。 1 2 3 [root@m1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-f89759699-
nginx gzip 动态静态压缩介绍

gzip压缩是解决编译产物体积过大、缩短静态资源请求时长的常用手段之一，我们在网站上经常能看到Content-Encoding: gzip这个 http 响应头，表示
Nginx显示500错误的原因以及解决方法

最近在操作nginx 的时候出现了 Nginx 500 内部错误，在此记录一下原因，项目采用的是前后端分离方式，后端SpringBoot ，前端是Vue 部署到Nginx
使用ChatGPT解决Nginx反向代理的问题

我有 3 台服务器，1台Web服务器对外提供访问，另外两台服务器 QC1 和 QC2 用于存储不同设备识别后保存的照片。它们的 IP 地址如下：服务器
nginx使用rewrite报错的解决

今天编译安装nginx的时候想着要做rewrite功能，匆忙瞟了一眼编译文档就直接makemake install了，然后再修改配置文件的时候一直报错，语法死活
Nginx配置二级域名的方法实现

当一个域名需要使用在两个项目上后，我们就需要使用到二级域名，在 Nginx 中配置二级域名如下： 1、原始配置文件如下 1 2 3 4 5 6 7 8 9 10
win10系统安装Nginx的详细步骤介绍

ginx是一款自由的、开源的、高性能的HTTP服务器和反向代理服务器，同时也提供了IMAP/POP3/SMTP服务。 Nginx可以进行反向代理、负载均衡、HTT
nginx降权与匹配php详细结束

nginx降权启动确认普通用户无法开启nginx 创建普通用户： 1 2 3 4 5 6 root@ubuntu:~# useradd -d /home/test -m test root@ubuntu:~# passwd test New password: Retype
Nginx隐藏server头信息的实现介绍

分析上一篇文章我们搭建了Nginx，请求响应头如下 1 2 3 4 5 6 7 8 9 10 [nginx@node01 sbin]$ curl -I 127.0.0.1:8090 HTTP/1.1 200 OK Server: nginx/1.9.9 Date: Fri, 11