Skip to main content

K8s 开始

Kubernetes 是用于自动部署,扩展和管理容器化应用程序的开源系统。本文将介绍如何快速开始 K8s 的使用。

了解 K8s

搭建 K8s

本地开发测试,需要搭建一个 K8s 轻量服务。实际部署时,可以用云厂商的 K8s 服务。

本文以 k3d 为例,于 macOS 搭建 K8s 服务。于 Ubuntu 则推荐 MicroK8s。其他可替代方案有:

k3d

k3s 是 Rancher 推出的 K8s 轻量版。而 k3d 即 k3s in docker,以 docker 容器管理 k3s 集群。

以下搭建过程,是于 macOS 的笔记,供参考。其他平台,请依照官方文档进行。

# 安装 kubectl: 命令行工具
brew install kubectl
# 安装 kubecm: 配置管理工具
brew install kubecm

# 安装 k3d
brew install k3d
❯ k3d version
k3d version v4.4.8
k3s version latest (default)

创建集群(1主2从):

❯ k3d cluster create mycluster --api-port 6550 --servers 1 --agents 2 --port 8080:80@loadbalancer --wait
INFO[0000] Prep: Network
INFO[0000] Created network 'k3d-mycluster' (23dc5761582b1a4b74d9aa64d8dca2256b5bc510c4580b3228123c26e93f456e)
INFO[0000] Created volume 'k3d-mycluster-images'
INFO[0001] Creating node 'k3d-mycluster-server-0'
INFO[0001] Creating node 'k3d-mycluster-agent-0'
INFO[0001] Creating node 'k3d-mycluster-agent-1'
INFO[0001] Creating LoadBalancer 'k3d-mycluster-serverlb'
INFO[0001] Starting cluster 'mycluster'
INFO[0001] Starting servers...
INFO[0001] Starting Node 'k3d-mycluster-server-0'
INFO[0009] Starting agents...
INFO[0009] Starting Node 'k3d-mycluster-agent-0'
INFO[0022] Starting Node 'k3d-mycluster-agent-1'
INFO[0030] Starting helpers...
INFO[0030] Starting Node 'k3d-mycluster-serverlb'
INFO[0031] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
INFO[0036] Successfully added host record to /etc/hosts in 4/4 nodes and to the CoreDNS ConfigMap
INFO[0036] Cluster 'mycluster' created successfully!
INFO[0036] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false
INFO[0036] You can now use it like this:
kubectl config use-context k3d-mycluster
kubectl cluster-info

查看集群信息:

❯ kubectl cluster-info
Kubernetes control plane is running at https://0.0.0.0:6550
CoreDNS is running at https://0.0.0.0:6550/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://0.0.0.0:6550/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

查看资源信息:

# 查看 Nodes
❯ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k3d-mycluster-agent-0 Ready <none> 2m12s v1.20.10+k3s1
k3d-mycluster-server-0 Ready control-plane,master 2m23s v1.20.10+k3s1
k3d-mycluster-agent-1 Ready <none> 2m4s v1.20.10+k3s1
# 查看 Pods
❯ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-6488c6fcc6-5n7d9 1/1 Running 0 2m12s
kube-system metrics-server-86cbb8457f-dr7lh 1/1 Running 0 2m12s
kube-system local-path-provisioner-5ff76fc89d-zbxf4 1/1 Running 0 2m12s
kube-system helm-install-traefik-bfm4c 0/1 Completed 0 2m12s
kube-system svclb-traefik-zx98g 2/2 Running 0 68s
kube-system svclb-traefik-7bx2r 2/2 Running 0 68s
kube-system svclb-traefik-cmdrm 2/2 Running 0 68s
kube-system traefik-6f9cbd9bd4-2mxhk 1/1 Running 0 69s

测试 Nginx:

# 创建 Nginx Deployment
kubectl create deployment nginx --image=nginx
# 创建 ClusterIP Service,暴露 Nginx 端口
kubectl create service clusterip nginx --tcp=80:80
# 创建 Ingress Object
# k3s 以 traefik 为默认 ingress controller
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx
annotations:
ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx
port:
number: 80
EOF
# 访问 Nginx Service
# kubectl get pods 确认 nginx STATUS=Running
open http://127.0.0.1:8080

测试 Dashboard:

# 创建 Dashboard
GITHUB_URL=https://github.com/kubernetes/dashboard/releases
VERSION_KUBE_DASHBOARD=$(curl -w '%{url_effective}' -I -L -s -S ${GITHUB_URL}/latest -o /dev/null | sed -e 's|.*/||')
kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/${VERSION_KUBE_DASHBOARD}/aio/deploy/recommended.yaml
# 配置 RBAC
# admin user
cat <<EOF > dashboard.admin-user.yml
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
EOF
# admin user role
cat <<EOF > dashboard.admin-user-role.yml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
EOF
# 配置部署
kubectl create -f dashboard.admin-user.yml -f dashboard.admin-user-role.yml
# 获取 Bearer Token
kubectl -n kubernetes-dashboard describe secret admin-user-token | grep ^token
# 访问代理
kubectl proxy
# 访问 Dashboard
# 输入 Token 登录
open http://127.0.0.1:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

删除集群:

k3d cluster delete mycluster

切换集群:

kubecm s

参考:

MicroK8s

MicroK8s 是 Ubuntu 官方生态提供的 K8s 轻量版,适合用于开发工作站、IoT、Edge、CI/CD。

以下搭建过程,是于 Ubuntu 18/20 的笔记,供参考。其他平台,请依照官方文档进行。

# 检查 hostname
# 要求不含大写字母和下划线,不然依照后文修改
hostname

# 安装 microk8s
sudo apt install snapd -y
snap info microk8s
sudo snap install microk8s --classic --channel=1.21/stable

# 添加用户组
sudo usermod -a -G microk8s $USER
sudo chown -f -R $USER ~/.kube
newgrp microk8s
id $USER

## 一些确保拉到镜像的方法
# 配置代理(如果有)
# MicroK8s / Installing behind a proxy
# https://microk8s.io/docs/install-proxy
# Issue: Pull images from others than k8s.gcr.io
# https://github.com/ubuntu/microk8s/issues/472
sudo vi /var/snap/microk8s/current/args/containerd-env
HTTPS_PROXY=http://127.0.0.1:7890
NO_PROXY=10.1.0.0/16,10.152.183.0/24
# 添加镜像(docker.io)
# 镜像加速器
# https://yeasy.gitbook.io/docker_practice/install/mirror
# 还可改 args/ 里不同模板的 sandbox_image
sudo vi /var/snap/microk8s/current/args/containerd-template.toml
[plugins."io.containerd.grpc.v1.cri"]
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://x.mirror.aliyuncs.com", "https://registry-1.docker.io", ]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:32000"]
endpoint = ["http://localhost:32000"]
# 手动导入,见后文启用插件那

# 重启服务
microk8s stop
microk8s start

检查状态:

$ microk8s status
microk8s is running
high-availability: no
datastore master nodes: 127.0.0.1:19001
datastore standby nodes: none
addons:
enabled:
ha-cluster # Configure high availability on the current node
disabled:
ambassador # Ambassador API Gateway and Ingress
cilium # SDN, fast with full network policy
dashboard # The Kubernetes dashboard
dns # CoreDNS
fluentd # Elasticsearch-Fluentd-Kibana logging and monitoring
gpu # Automatic enablement of Nvidia CUDA
helm # Helm 2 - the package manager for Kubernetes
helm3 # Helm 3 - Kubernetes package manager
host-access # Allow Pods connecting to Host services smoothly
ingress # Ingress controller for external access
istio # Core Istio service mesh services
jaeger # Kubernetes Jaeger operator with its simple config
keda # Kubernetes-based Event Driven Autoscaling
knative # The Knative framework on Kubernetes.
kubeflow # Kubeflow for easy ML deployments
linkerd # Linkerd is a service mesh for Kubernetes and other frameworks
metallb # Loadbalancer for your Kubernetes cluster
metrics-server # K8s Metrics Server for API access to service metrics
multus # Multus CNI enables attaching multiple network interfaces to pods
openebs # OpenEBS is the open-source storage solution for Kubernetes
openfaas # openfaas serverless framework
portainer # Portainer UI for your Kubernetes cluster
prometheus # Prometheus operator for monitoring and logging
rbac # Role-Based Access Control for authorisation
registry # Private image registry exposed on localhost:32000
storage # Storage class; allocates storage from host directory
traefik # traefik Ingress controller for external access

如果 status 不正确时,可以如下排查错误:

microk8s inspect
grep -r error /var/snap/microk8s/2346/inspection-report

如果要修改 hostname

# 改名称
sudo hostnamectl set-hostname ubuntu-vm
# 改 host
sudo vi /etc/hosts

# 云主机的话,还要改下配置
sudo vi /etc/cloud/cloud.cfg
preserve_hostname: true
# 如果只修改 preserve_hostname 不生效,那就直接注释掉 set/update_hostname
cloud_init_modules:
# - set_hostname
# - update_hostname

# 重启,验证生效
sudo reboot

接着,启用些基础插件:

microk8s enable dns dashboard

# 查看 Pods ,确认 running
microk8s kubectl get pods --all-namespaces
# 不然,详情里看下错误原因
microk8s kubectl describe pod --all-namespaces

直到全部正常 running

$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kubernetes-dashboard-85fd7f45cb-snqrv 1/1 Running 1 15h
kube-system dashboard-metrics-scraper-78d7698477-tmb7k 1/1 Running 1 15h
kube-system metrics-server-8bbfb4bdb-wlf8g 1/1 Running 1 15h
kube-system calico-node-p97kh 1/1 Running 1 6m18s
kube-system coredns-7f9c69c78c-255fg 1/1 Running 1 15h
kube-system calico-kube-controllers-f7868dd95-st9p7 1/1 Running 1 16h

如果拉取镜像失败,可以 microk8s ctr image pull <mirror>。或者,docker pull 后导入 containerd

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1
docker save k8s.gcr.io/pause:3.1 > pause:3.1.tar
microk8s ctr image import pause:3.1.tar

docker pull calico/cni:v3.13.2
docker save calico/cni:v3.13.2 > cni:v3.13.2.tar
microk8s ctr image import cni:v3.13.2.tar

docker pull calico/node:v3.13.2
docker save calico/node:v3.13.2 > node:v3.13.2.tar
microk8s ctr image import node:v3.13.2.tar

如果 calico-node CrashLoopBackOff,可能网络配置问题:

# 查具体日志
microk8s kubectl logs -f -n kube-system calico-node-l5wl2 -c calico-node
# 如果有 Unable to auto-detect an IPv4 address,那么 ip a 找出哪个网口有 IP 。修改:
sudo vi /var/snap/microk8s/current/args/cni-network/cni.yaml
- name: IP_AUTODETECTION_METHOD
value: "interface=wlo.*"
# 重启服务
microk8s stop; microk8s start

## 参考
# Issue: Microk8s 1.19 not working on Ubuntu 20.04.1
# https://github.com/ubuntu/microk8s/issues/1554
# Issue: CrashLoopBackOff for calico-node pods
# https://github.com/projectcalico/calico/issues/3094
# Changing the pods CIDR in a MicroK8s cluster
# https://microk8s.io/docs/change-cidr
# MicroK8s IPv6 DualStack HOW-TO
# https://discuss.kubernetes.io/t/microk8s-ipv6-dualstack-how-to/14507

然后,可以打开 Dashboard 看看:

# 获取 Token (未启用 RBAC 时)
token=$(microk8s kubectl -n kube-system get secret | grep default-token | cut -d " " -f1)
microk8s kubectl -n kube-system describe secret $token
# 转发端口
microk8s kubectl port-forward -n kube-system service/kubernetes-dashboard 10443:443
# 打开网页,输入 Token 登录
xdg-open https://127.0.0.1:10443

# 更多说明 https://microk8s.io/docs/addon-dashboard
# Issue: Your connection is not private
# https://github.com/kubernetes/dashboard/issues/3804

更多操作,请阅读官方文档。本文之后仍以 k3d 为例。

准备 K8s 应用

Go 应用

http_server.go:

package main

import (
"fmt"
"log"
"net/http"
)

func handler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hi there, I love %s!", r.URL.Path[1:])
}

func main() {
http.HandleFunc("/", handler)
fmt.Println("HTTP Server running ...")
log.Fatal(http.ListenAndServe(":3000", nil))
}

构建镜像

http_server.dockerfile:

FROM golang:1.17-alpine AS builder
WORKDIR /app
ADD ./http_server.go /app
RUN cd /app && go build http_server.go

FROM alpine:3.14
WORKDIR /app
COPY --from=builder /app/http_server /app/
EXPOSE 3000
ENTRYPOINT ./http_server
# 编译镜像
docker build -t http_server:1.0 -f http_server.dockerfile .
# 运行应用
docker run --rm -p 3000:3000 http_server:1.0
# 测试应用
❯ curl http://127.0.0.1:3000/go
Hi there, I love go!

部署 K8s 应用

了解概念

之后,参照官方教程,我们将使用 Deployment 运行 Go 应用(无状态)。

导入镜像

首先,我们手动导入镜像进集群:

docker save http_server:1.0 > http_server:1.0.tar
k3d image import http_server:1.0.tar -c mycluster

如果有自己的私有仓库,参见 k3d / Registries 进行配置。

创建 Deployment

# 配置 Deployment (2个副本)
cat <<EOF > go-http-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: go-http
labels:
app: go-http
spec:
replicas: 2
selector:
matchLabels:
app: go-http
template:
metadata:
labels:
app: go-http
spec:
containers:
- name: go-http
image: http_server:1.0
ports:
- containerPort: 3000
EOF
# 应用 Deployment
# --record: 记录命令
kubectl apply -f go-http-deployment.yaml --record

查看 Deployment:

# 查看 Deployment
❯ kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 1/1 1 1 2d
go-http 2/2 2 2 22s

# 查看 Deployment 信息
kubectl describe deploy go-http
# 查看 Deployment 创建的 ReplicaSet (2个)
kubectl get rs
# 查看 Deployment 创建的 Pods (2个)
kubectl get po -l app=go-http -o wide --show-labels
# 查看某一 Pod 信息
kubectl describe po go-http-5848d49c7c-wzmxh

创建 Service

# 创建 Service,名为 go-http
# 将请求代理到 app=go-http, tcp=3000 的 Pod 上
kubectl expose deployment go-http --name=go-http
# 或
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
name: go-http
labels:
app: go-http
spec:
selector:
app: go-http
ports:
- protocol: TCP
port: 3000
targetPort: 3000
EOF

# 查看 Service
kubectl get svc
# 查看 Service 信息
kubectl describe svc go-http
# 查看 Endpoints 对比看看
# kubectl get ep go-http
# kubectl get po -l app=go-http -o wide

# 删除 Service (如果)
kubectl delete svc go-http

访问 Service (DNS):

❯ kubectl run curl --image=radial/busyboxplus:curl -i --tty

If you don't see a command prompt, try pressing enter.

[ root@curl:/ ]$ nslookup go-http
Server: 10.43.0.10
Address 1: 10.43.0.10 kube-dns.kube-system.svc.cluster.local

Name: go-http
Address 1: 10.43.102.17 go-http.default.svc.cluster.local

[ root@curl:/ ]$ curl http://go-http:3000/go
Hi there, I love go!

暴露 Service (Ingress):

# 创建 Ingress Object
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-http
annotations:
ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- path: /go
pathType: Prefix
backend:
service:
name: go-http
port:
number: 3000
EOF
# 查看 Ingress
kubectl get ingress
# 查看 Ingress 信息
kubectl describe ingress go-http

# 删除 Ingress (如果)
kubectl delete ingress go-http

访问 Service (Ingress):

❯ open http://127.0.0.1:8080/go
# 或,
❯ curl http://127.0.0.1:8080/go
Hi there, I love go!
# Nginx 是在 http://127.0.0.1:8080

更新

仅当 Deployment Pod 模板发生改变时,例如模板的标签或容器镜像被更新,才会触发 Deployment 上线。其他更新(如对 Deployment 执行扩缩容的操作)不会触发上线动作。

所以,我们准备 http_server:2.0 镜像导入集群,然后更新:

❯ kubectl set image deployment/go-http go-http=http_server:2.0 --record
deployment.apps/go-http image updated

之后,可以查看上线状态:

# 查看上线状态
❯ kubectl rollout status deployment/go-http
deployment "go-http" successfully rolled out

# 查看 ReplicaSet 状态:新的扩容,旧的缩容,完成更新
❯ kubectl get rs
NAME DESIRED CURRENT READY AGE
go-http-586694b4f6 2 2 2 10s
go-http-5848d49c7c 0 0 0 6d

测试服务:

❯ curl http://127.0.0.1:8080/go
Hi there v2, I love go!

回滚

查看 Deployment 修订历史:

❯ kubectl rollout history deployment.v1.apps/go-http
deployment.apps/go-http
REVISION CHANGE-CAUSE
1 kubectl apply --filename=go-http-deployment.yaml --record=true
2 kubectl set image deployment/go-http go-http=http_server:2.0 --record=true

# 查看修订信息
kubectl rollout history deployment.v1.apps/go-http --revision=2

回滚到之前的修订版本:

# 回滚到上一版
kubectl rollout undo deployment.v1.apps/go-http
# 回滚到指定版本
kubectl rollout undo deployment.v1.apps/go-http --to-revision=1

缩放

# 缩放 Deployment 的 ReplicaSet 数
kubectl scale deployment.v1.apps/go-http --replicas=10

# 如果集群启用了 Pod 的水平自动缩放,可以根据现有 Pods 的 CPU 利用率选择上下限
# Horizontal Pod Autoscaler Walkthrough
# https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/
kubectl autoscale deployment.v1.apps/nginx-deployment --min=10 --max=15 --cpu-percent=80

暂停、恢复

# 暂停 Deployment
kubectl rollout pause deployment.v1.apps/go-http
# 恢复 Deployment
kubectl rollout resume deployment.v1.apps/go-http

期间可以更新 Deployment ,但不会触发上线。

删除

kubectl delete deployment go-http

金丝雀部署

灰度部署,用多标签区分多个部署,新旧版可同时运行。部署新版时,用少量流量验证,没问题再全量更新。

Helm 发布

Helm 是 K8s 的包管理工具,包格式称为 charts。现在来发布我们的 Go 服务吧。

安装 Helm

# macOS
brew install helm
# Ubuntu
sudo snap install helm --classic

执行 helm 了解命令。

创建 Chart

helm create go-http

查看内容:

❯ tree go-http -aF --dirsfirst
go-http
├── charts/ # 包依赖的 charts,称 subcharts
├── templates/ # 包的 K8s 文件模板,用的 Go 模板
│ ├── tests/
│ │ └── test-connection.yaml
│ ├── NOTES.txt # 包的帮助文本
│ ├── _helpers.tpl # 模板可重用的片段,模板里 include 引用
│ ├── deployment.yaml
│ ├── hpa.yaml
│ ├── ingress.yaml
│ ├── service.yaml
│ └── serviceaccount.yaml
├── .helmignore # 打包忽略说明
├── Chart.yaml # 包的描述文件
└── values.yaml # 变量默认值,可安装时覆盖,模板里 .Values 引用

修改内容:

  • 修改 Chart.yaml 里的描述
  • 修改 values.yaml 里的变量
    • 修改 image 为发布的 Go 服务
    • 修改 ingresstrue,及一些配置
    • 删除 serviceAccount autoscalingtemplates/ 里也搜索删除相关内容
  • 修改 templates/ 里的模板
    • 删除 tests/,剩余 deployment.yaml service.yaml ingress.yaml 有用

结果可见 start-k8s/helm/go-http

检查错误:

❯ helm lint --strict go-http
==> Linting go-http
[INFO] Chart.yaml: icon is recommended

1 chart(s) linted, 0 chart(s) failed

渲染模板:

# --set 覆盖默认配置,或者用 -f 选择自定的 values.yaml
helm template go-http-helm ./go-http \
--set replicaCount=2 \
--set "ingress.hosts[0].paths[0].path=/helm" \
--set "ingress.hosts[0].paths[0].pathType=Prefix"

安装 Chart:

helm install go-http-helm ./go-http \
--set replicaCount=2 \
--set "ingress.hosts[0].paths[0].path=/helm" \
--set "ingress.hosts[0].paths[0].pathType=Prefix"

# 或,打包后安装
helm package go-http
helm install go-http-helm go-http-1.0.0.tgz \
--set replicaCount=2 \
--set "ingress.hosts[0].paths[0].path=/helm" \
--set "ingress.hosts[0].paths[0].pathType=Prefix"

# 查看安装列表
helm list

测试服务:

❯ kubectl get deploy go-http-helm
NAME READY UP-TO-DATE AVAILABLE AGE
go-http-helm 2/2 2 2 2m42s

❯ curl http://127.0.0.1:8080/helm
Hi there, I love helm!

卸载 Chart:

helm uninstall go-http-helm

发布 Chart

官方仓库 ArtifactHub 上有很多分享的 Helm charts 。可见 velkoz1108/helm-chart 把我们的 Go 服务发布到 Hub 上。

最后

开始 K8s 吧!本文样例在 ikuokuo/start-k8s