可信网站查询官网,wordpress后台文章自定义字段面板,云阿里云做网站,长沙 服务本节重点介绍 :
push模型和pull模型监控系统对比为什么在k8s中只能用pull模型的k8s中主要组件的暴露地址说明
push模型和pull模型监控系统
对比下两种系统采用的不同采集模型#xff0c;即push型采集和pull型采集。不同的模型在性能的考虑上是截然不同的。下面表格简单的说…本节重点介绍 :
push模型和pull模型监控系统对比为什么在k8s中只能用pull模型的k8s中主要组件的暴露地址说明
push模型和pull模型监控系统
对比下两种系统采用的不同采集模型即push型采集和pull型采集。不同的模型在性能的考虑上是截然不同的。下面表格简单的说明了下两种模型的特点
采集模型原理简介代表pushagent定时推送数据到server夜莺open-falconpullserver定时去agent拉数据prometheus
就采集器是否丰富来说
我们需要对比的是这个系统是否有很好的插件扩展机制因为这直接决定了开源社区对该系统采集器贡献的活跃度prometheus采集的pull模型使用者可以用自定义exporter的模式灵活的接入。
push型的致命缺点 agent和服务端强耦合
那就是agent需要配置服务端地址带来了一定的耦合性不适合云原生场景。如果采用push型试想一下你的应用部署在k8s中在启动的时候需要指定监控上报的服务地址那是不能接受的。类比pushgateway的例子pusher push.New(url, jobName) 必须要指定服务端的地址
如果push端的服务地址变化了怎么办
一个典型的场景就是在k8s中pod的扩缩十分频繁服务端的地址也不固定。
pull型的处理方法
对比来说应用pull模型采集的prometheus可以对接多种服务发现源特别适合k8s环境。举个例子应用的pod一旦发生变化prometheus就可以通过配置好k8s的服务发现模式监听到资源变化进行采集的增删agent侧只需要暴露自己的指标完全不关心是哪一个server过来获取数据。
k8s中主要组件的暴露地址说明
部署在pod中业务埋点指标
是直接通过pod的ip暴露的我们可以直接通过get pod 获取容器的ip在node上直接curl访问到
[rootk8s-master01 ink8s-pod-metrics]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ink8s-pod-metrics-deployment-85d9795d6-95lsp 1/1 Running 0 13h 10.100.85.207 k8s-node01 none none
[rootk8s-master01 ink8s-pod-metrics]# curl -s 10.100.85.207:8080/metrics |grep ink8s # HELP ink8s_pod_metrics_get_node_detail k8s node detail each
# TYPE ink8s_pod_metrics_get_node_detail gauge
ink8s_pod_metrics_get_node_detail{containerRuntimeVersioncontainerd://1.4.4,hostnamek8s-master01,ip172.20.70.205,kubeletVersionv1.20.1} 1
ink8s_pod_metrics_get_node_detail{containerRuntimeVersioncontainerd://1.4.4,hostnamek8s-node01,ip172.20.70.215,kubeletVersionv1.20.1} 1
# HELP ink8s_pod_metrics_get_node_last_duration_seconds get node last duration seconds
# TYPE ink8s_pod_metrics_get_node_last_duration_seconds gauge
ink8s_pod_metrics_get_node_last_duration_seconds 0.008506914
# HELP ink8s_pod_metrics_get_pod_control_plane_pod_detail k8s pod detail of control plane
# TYPE ink8s_pod_metrics_get_pod_control_plane_pod_detail gauge
ink8s_pod_metrics_get_pod_control_plane_pod_detail{componentetcd,ip172.20.70.205,pod_nameetcd-k8s-master01} 1
ink8s_pod_metrics_get_pod_control_plane_pod_detail{componentkube-apiserver,ip172.20.70.205,pod_namekube-apiserver-k8s-master01} 1
ink8s_pod_metrics_get_pod_control_plane_pod_detail{componentkube-controller-manager,ip172.20.70.205,pod_namekube-controller-manager-k8s-master01} 1
ink8s_pod_metrics_get_pod_control_plane_pod_detail{componentkube-scheduler,ip172.20.70.205,pod_namekube-scheduler-k8s-master01} 1
# HELP ink8s_pod_metrics_get_pod_last_duration_seconds get pod last duration seconds
# TYPE ink8s_pod_metrics_get_pod_last_duration_seconds gauge
ink8s_pod_metrics_get_pod_last_duration_seconds 0.012481561target页面举例图片
容器基础资源指标
kubelet 内置cadvisor metrics接口暴露的我们可以先获取token再使用token作为header访问各个节点的cadvisor指标
TOKEN$(kubectl -n kube-system get secret $(kubectl -n kube-system get serviceaccount prometheus -o jsonpath{.secrets[0].name}) -o jsonpath{.data.token} | base64 --decode )
curl -s https://172.20.70.215:10250/metrics/cadvisor --header Authorization: Bearer $TOKEN --insecure |head # HELP cadvisor_version_info A metric with a constant 1 value labeled by kernel version, OS version, docker version, cadvisor version cadvisor revision.
# TYPE cadvisor_version_info gauge
cadvisor_version_info{cadvisorRevision,cadvisorVersion,dockerVersion1.13.1,kernelVersion3.10.0-957.1.3.el7.x86_64,osVersionCentOS Linux 7 (Core)} 1
# HELP container_cpu_cfs_periods_total Number of elapsed enforcement period intervals.
# TYPE container_cpu_cfs_periods_total counter
container_cpu_cfs_periods_total{container,id/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab97c68_b0ac_48ce_ba39_6ffa72a2f4c8.slice,image,name,namespacedefault,podink8s-pod-metrics-deployment-85d9795d6-95lsp} 46664 1629771810858
container_cpu_cfs_periods_total{container,id/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3f353a_92fa_4436_a8ca_6cb632d48ada.slice,image,name,namespacekube-admin,podk8s-mon-daemonset-z6sfw} 762965 1629771819606
container_cpu_cfs_periods_total{container,id/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a95d67_a843_4369_8d5c_34a5333f1480.slice,image,name,namespacekube-admin,podk8s-mon-deployment-6d7d58bdc8-rxj42} 458822 1629771809776
container_cpu_cfs_periods_total{container,id/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27c9fe7_9d82_4228_86fb_b9c920611c15.slice,image,name,namespacekube-system,podprometheus-0} 941374 1629771809770
container_cpu_cfs_periods_total{containerink8s-pod-metrics,id/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab97c68_b0ac_48ce_ba39_6ffa72a2f4c8.slice/cri-containerd-2f85fd45a67cc4bb775b99d4676200b412ea18ef7ae4976fc93a8a7cff1c5f34.scope,imagedocker.io/library/ink8s-pod-metrics:v1,name2f85fd45a67cc4bb775b99d4676200b412ea18ef7ae4976fc93a8a7cff1c5f34,namespacedefault,podink8s-pod-metrics-deployment-85d9795d6-95lsp} 46667 1629771818053target页面举例图片
k8s对象资源指标
这是ksm直接暴露指标prometheus通过dns解析到域名然后访问的我们可以通过dig获取ksm 的service_ip然后访问 service_ip:8080 dig short kube-state-metrics.kube-system.svc.cluster.local 10.96.0.10
10.100.85.200
curl -s 10.100.85.200:8080/metrics |head # HELP kube_certificatesigningrequest_labels Kubernetes labels converted to Prometheus labels.
# TYPE kube_certificatesigningrequest_labels gauge
# HELP kube_certificatesigningrequest_created Unix creation timestamp
# TYPE kube_certificatesigningrequest_created gauge
# HELP kube_certificatesigningrequest_condition The number of each certificatesigningrequest condition
# TYPE kube_certificatesigningrequest_condition gauge
# HELP kube_certificatesigningrequest_cert_length Length of the issued cert
# TYPE kube_certificatesigningrequest_cert_length gauge
# HELP kube_configmap_info Information about configmap.
# TYPE kube_configmap_info gaugetarget页面举例图片
k8s服务组件指标
是由服务组件自身直接暴露的我们也可以通过带token直接访问
TOKEN$(kubectl -n kube-system get secret $(kubectl -n kube-system get serviceaccount prometheus -o jsonpath{.secrets[0].name}) -o jsonpath{.data.token} | base64 --decode )
curl -s https://localhost:6443/metrics --header Authorization: Bearer $TOKEN --insecure |head # HELP aggregator_openapi_v2_regeneration_count [ALPHA] Counter of OpenAPI v2 spec regeneration count broken down by causing APIService name and reason.
# TYPE aggregator_openapi_v2_regeneration_count counter
aggregator_openapi_v2_regeneration_count{apiservice*,reasonstartup} 0
aggregator_openapi_v2_regeneration_count{apiservicek8s_internal_local_delegation_chain_0000000002,reasonupdate} 0
# HELP aggregator_openapi_v2_regeneration_duration [ALPHA] Gauge of OpenAPI v2 spec regeneration duration in seconds.
# TYPE aggregator_openapi_v2_regeneration_duration gauge
aggregator_openapi_v2_regeneration_duration{reasonstartup} 0.812717406
aggregator_openapi_v2_regeneration_duration{reasonupdate} 0.848521427
# HELP aggregator_unavailable_apiservice [ALPHA] Gauge of APIServices which are marked as unavailable broken down by APIService name.
# TYPE aggregator_unavailable_apiservice gauge
[rootk8s-master01 ink8s-pod-metrics]# target页面举例图片
k8s中关注四大块指标总结
之前在k8s中关注4块指标有过总结
指标类型采集源应用举例发现类型容器基础资源指标kubelet 内置cadvisor metrics接口查看容器cpu、mem利用率等k8s_sd node级别直接访问node_ipk8s对象资源指标kube-stats-metrics (简称ksm)具体可以看 br 看pod状态如pod waiting状态的原因 br 数个数如查看node pod按namespace分布情况通过coredns访问域名k8s服务组件指标服务组件 metrics接口查看apiserver 、scheduler、etc、coredns请求延迟等k8s_sd endpoint级别部署在pod中业务埋点指标pod 的metrics接口依据业务指标场景k8s_sd pod级别访问pod ip的metricspath
本节重点总结 :
push模型和pull模型监控系统对比为什么在k8s中只能用pull模型的k8s中主要组件的暴露地址说明
