目前我们的网站正在建设中,搜索引擎登录入口,开发公司项目管理制度,大学做视频网站设计作者#xff1a;张华 发表于#xff1a;2023-03-01 版权声明#xff1a;可以任意转载#xff0c;转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明
问题
没有外网#xff0c;所以配置了一个local custom镜像库#xff0c;也使用了container-image-meta…作者张华 发表于2023-03-01 版权声明可以任意转载转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明
问题
没有外网所以配置了一个local custom镜像库也使用了container-image-metadata-url进行配置但是用juju创建lxd容器时还是说找不着image.
相关搜索
关于container-image-metadata-url的代码如下 https://github.com/juju/juju/pull/8578 https://github.com/juju/juju/blob/juju-2.9.35/container/lxd/manager.go#L282-L284 也有一个好帖子 https://discourse.charmhub.io/t/local-lxd-image-server/3929/5
实验环境
1, 使用 juju创建一个focal的machine 0, 然后再machine 0上部署一个xenial的lxd容器。
juju add-model test
juju add-machine --series focal
juju model-config logging-configrootDEBUG
juju remove-application ceph-radosgw juju deploy ceph-radosgw --seriesxenial --tolxd:02, 在juju controller(juju ssh -m controller 0)与machine 0上运行下列iptables来模拟和cloud-images.ubuntu.com断网。这里我发现
machine 0的日志/var/log/juju/machine-0.log显示它好像是从juju controller处下载镜像的
2023-03-01 07:58:21 INFO juju.cloudconfig userdatacfg_unix.go:613 Fetching agent: curl -sSf --connect-timeout 20 --noproxy * --insecure -o $bin/tools.tar.gz [https://10.5.0.31:17070/model/deb85179-10a6-4877-88f7-012ef768d726/tools/2.9.38-ubuntu-amd64 https://252.0.31.1:17070/model/deb85179-10a6-4877-88f7-012ef768d726/tools/2.9.38-ubuntu-amd64]
2023-03-01 07:59:03 INFO juju.container.lxd container.go:256 starting new container juju-68d726-0-lxd-2 (image ubuntu-16.04-server-cloudimg-amd64-lxd.tar.xz)
2023-03-01 07:59:03 DEBUG juju.container.lxd container.go:257 new container has profiles [default]
2023-03-01 07:59:42 DEBUG juju.container.lxd container.go:286 created container juju-68d726-0-lxd-2, waiting for start...但如果不在machine 0上运行下载iptables测试表明machine0也能直接扰开juju controller从cloud-images.ubuntu.com处下镜像.似乎二者均相关那就二者将运行下列iptables吧
dig cloud-images.ubuntu.com #185.125.190.37 and 185.125.190.40
iptables -A OUTPUT -d 185.125.190.37 -j DROP
iptables -A OUTPUT -d 185.125.190.40 -j DROP3, bastion上运行sstream-mirror将cloud-images.ubuntu.com中的xenial amd64镜像mirror了下来。
sudo apt -y install simplestreams -y
workdir/home/ubuntu/simplestreams2
sudo sstream-mirror --keyring/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg --progress --max1 --pathstreams/v1/index.json https://cloud-images.ubuntu.com/releases/ $workdir archamd64 release~(xenial) ftype~(lxd.tar.xz|squashfs|root.tar.xz|root.tar.gz|disk1.img|.json|.sjson)然后用nginx为它配置了https:
#https://goharbor.io/docs/2.6.0/install-config/configure-https/
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -sha512 -days 3650 -subj /CCN/STBeijing/LBeijing/Oexample/OUPersonal/CNquqi.com -key ca.key -out ca.crt
openssl genrsa -out quqi.com.key 4096
openssl req -sha512 -new -subj /CCN/STBeijing/LBeijing/Oexample/OUPersonal/CNquqi.com -key quqi.com.key -out quqi.com.csr
#complies with the Subject Alternative Name (SAN) and x509 v3 extension requirements to avoid x509: certificate relies on legacy Common Name field, use SANs instead
cat v3.ext -EOF
authorityKeyIdentifierkeyid,issuer
basicConstraintsCA:FALSE
keyUsage digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage serverAuth
subjectAltName alt_names[alt_names]
DNS.1quqi.com
DNS.2quqi
DNS.3hostname
EOF
openssl x509 -req -sha512 -days 3650 -extfile v3.ext -CA ca.crt -CAkey ca.key -CAcreateserial -in quqi.com.csr -out quqi.com.crt
#for docker, the Docker daemon interprets .crt files as CA certificates and .cert files as client certificates.
openssl x509 -inform PEM -in quqi.com.crt -out quqi.com.cert
curl --resolve quqi.com:443:10.5.0.126 --cacert ~/ca/ca.crt https://quqi.com:443/streams/v1/index.json
sudo cp ~/ca/ca.crt /usr/local/share/ca-certificates/ca.crt
sudo chmod 644 /usr/local/share/ca-certificates/ca.crt
sudo update-ca-certificates --fresh
curl --resolve quqi.com:443:10.5.0.126 https://quqi.com:443/streams/v1/index.json$ cat /etc/nginx/sites-available/default
server {listen 443 ssl http2;listen [::]:443 ssl http2;server_name quqi.com;ssl_certificate /home/ubuntu/ca/quqi.com.crt;ssl_certificate_key /home/ubuntu/ca/quqi.com.key;ssl_protocols TLSv1.2;ssl_prefer_server_ciphers on; location / {root /home/ubuntu/simplestreams2;index index.html;}
}
# 注意由于上面使用了一个新目录/home/ubuntu/simplestreams2作为root那需要将/etc/nginx/nginx.conf中添加user root;来避免权限问题
#curl --resolve quqi.com:443:10.5.0.126 --cacert ~/ca/ca.crt https://quqi.com:443/images/streams/v1/index.json
curl --resolve quqi.com:443:10.5.0.126 --cacert ~/ca/ca.crt https://quqi.com:443/streams/v1/index.json4, 配置juju中的container-image-metadata-url使用上面的https based local image mirror
juju model-config container-image-metadata-urlhttps://quqi.com:443
juju model-config image-metadata-urlhttps://quqi.com:4435, juju controller由于访问local image mirror, 所以配置hosts与添加ca key
echo 10.5.0.126 quqi.com /etc/hostscurl --resolve quqi.com:443:10.5.0.126 --cacert ~/ca/ca.crt https://quqi.com:443/streams/v1/index.json
sudo cp ~/ca/ca.crt /usr/local/share/ca-certificates/ca.crt
sudo chmod 644 /usr/local/share/ca-certificates/ca.crt
sudo update-ca-certificates --fresh
curl --resolve quqi.com:443:10.5.0.126 https://quqi.com:443/streams/v1/index.json6, 记得重新测试之前将machine 0上的image cache删除
juju ssh 0 -- sudo lxc image delete juju/xenial/amd64
juju remove-application ceph-radosgw7, 重新测试
juju deploy ceph-radosgw --seriesxenial --tolxd:0
sudo tail -f /var/log/juju/machine-0.log能在machine 0的/var/log/juju/machine-0.log中观察下列日志:
2023-03-01 08:26:45 WARNING juju.worker.lxdprovisioner provisioner_task.go:1371 machine 0/lxd/3 failed to start: acquiring LXD image: no matching image found
2023-03-01 08:26:45 WARNING juju.worker.lxdprovisioner provisioner_task.go:1410 failed to start machine 0/lxd/3 (acquiring LXD image: no matching image found), retrying in 10s (10 more attempts)在juju controller上有时能搜到quqi, 有时候又不能奇怪.
2023-02-23 07:33:52 WARNING juju.apiserver.provisioner provisioninginfo.go:801 encountered https://quqi.com:443/images/streams/v1/streams/v1/index.json: Get https://quqi.com:443/images/streams/v1/streams/v1/index.json: dial tcp 49.234.171.74:443: i/o timeout while getting published images metadata from image-metadata-url
2023-03-01 08:52:56 WARNING juju.environs.simplestreams datasource.go:184 Got error requesting https://quqi.com:443/streams/v1/index.json: Get https://quqi.com:443/streams/v1/index.json: x509: certificate relies on legacy Common Name field, use SANs insteadjuju controller上仍然能看到cloud-images.ubuntu.com
2023-03-01 08:34:54 WARNING juju.apiserver.provisioner provisioninginfo.go:801 encountered http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson: Get http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson: dial tcp 185.125.190.37:80: i/o timeout while getting published images metadata from default ubuntu cloud images使用glance中的image来提供simplestreams
上面是使用来提供simplestreams, 我们现在换用glance中的image来提供simplestreams继续测试 不确定是否这种只适用于创建juju controller 还是说也可以用于VM/LXD创建试一下)
mkdir -p ~/simplestreams/images
IMAGE_ID26751c0e-4282-415e-b8dc-a7a21d2f781d
SERIESxenial
juju metadata generate-image -d ~/simplestreams -i $IMAGE_ID -s $SERIES -r RegionOne -u $OS_AUTH_URL然后修改/etc/nginx/sites-available/default将上面测试用的/home/ubuntu/simplestreams2改成/home/ubuntu/simplestreams 重启nginx之后, 设置container-image-metadata-url (注意此时后面链接多出了/images)
juju model-config container-image-metadata-urlhttps://quqi.com:443/images
juju model-config image-metadata-urlhttps://quqi.com:443/images为保证测试环境干净我也在controller与machine0上运行了下列命令
systemctl restart jujud-machine-0.service然后重复测试后问题依旧 controller上看到下列日志
2023-03-01 10:29:32 WARNING juju.environs.simplestreams datasource.go:184 Got error requesting https://streams.canonical.com/juju/tools/streams/v1/index.sjson: Get https://streams.canonical.com/juju/tools/streams/v1/index.sjson: dial tcp 185.125.190.37:443: i/o timeout
2023-03-01 10:29:36 INFO juju.state addmachine.go:505 new machine 0/lxd/11 has preferred addresses: private , public
2023-03-01 10:29:37 WARNING juju.apiserver.instancemutater lxdprofilewatcher.go:206 unit ceph-radosgw/11 has no machine id, start watching when machine id assigned.
2023-03-01 10:29:41 WARNING juju.apiserver.provisioner provisioninginfo.go:801 encountered index file has no data for cloud {stsstack http://10.230.19.53:5000/v3} not found while getting published images metadata from image-metadata-url
2023-03-01 10:30:11 WARNING juju.environs.simplestreams datasource.go:184 Got error requesting http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson: Get http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson: dial tcp 185.125.190.40:80: i/o timeout
2023-03-01 10:30:41 WARNING juju.environs.simplestreams datasource.go:184 Got error requesting http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson: Get http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson: dial tcp 185.125.190.37:80: i/o timeout
2023-03-01 10:30:41 WARNING juju.apiserver.provisioner provisioninginfo.go:801 encountered http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson: Get http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson: dial tcp 185.125.190.37:80: i/o timeout while getting published images metadata from default ubuntu cloud images看样子和simplestreams类型无关。
测试cloudinit-userdata
即然与simplestreams类型无关那我们将nginx再恢复之前的/home/ubuntu/simplestreams2
juju model-config container-image-metadata-urlhttps://quqi.com:443/
juju model-config image-metadata-urlhttps://quqi.com:443/然后测试cloudinit-userdata, 这个是没问题的可以作workaround
cat EOF |tee cloudinit-userdata.yaml
cloudinit-userdata: |postruncmd:- echo 10.5.0.126 quqi.com /etc/hosts- if hostname |grep -qv lxd; then wget --tries15 --retry-connrefused --timeout15 --random-waiton -O /home/ubuntu/ubuntu-16.04-server-cloudimg-amd64-lxd.tar.xz https://quqi.com:443/server/releases/xenial/release-20211001/ubuntu-16.04-server-cloudimg-amd64-lxd.tar.xz --no-check-certificate; wget --tries15 --retry-connrefused --timeout15 --random-waiton -O /home/ubuntu/ubuntu-16.04-server-cloudimg-amd64.squashfs https://quqi.com:443/server/releases/xenial/release-20211001/ubuntu-16.04-server-cloudimg-amd64.squashfs --no-check-certificate; fi- sleep 30- if hostname |grep -qv lxd; then lxc image import /home/ubuntu/ubuntu-16.04-server-cloudimg-amd64-lxd.tar.xz /home/ubuntu/ubuntu-16.04-server-cloudimg-amd64.squashfs --alias juju/xenial/amd64; fi
EOF
juju model-config ./cloudinit-userdata.yaml
juju model-config cloudinit-userdata --format yaml
#juju model-config --reset cloudinit-userdata注意之前一直不work的原因是因为在postruncmd:后加了 | 的原因找到答案的过程见下列的调试cloud-init一节。
调试cloud-init
最后发现用下面的是不work的
cat EOF |tee test.yaml
cloudinit-userdata: |postruncmd: |- echo 10.5.0.126 quqi.com /etc/hosts- echo test /home/ubuntu/cloud-init.txt
EOF需要改成下面的
cat EOF |tee test.yaml
cloudinit-userdata: |postruncmd:- bash -c echo 10.5.0.126 quqi.com /etc/hosts- bash -c echo test /home/ubuntu/cloud-init.txt
EOF下面的也不会work
cat EOF |tee test.yaml
cloudinit-userdata: |postruncmd: |bash -c echo 10.5.0.126 quqi.com /etc/hostsbash -c echo test /home/ubuntu/cloud-init.txt
EOF下面的更不会work, 会直接报ERROR json: unsupported type: map[interface {}]interface {}’
cat EOF |tee test.yaml
cloudinit-userdata: |postruncmd:bash -c echo 10.5.0.126 quqi.com /etc/hostsbash -c echo test /home/ubuntu/cloud-init.txt
EOF其他调试方法如下
juju add-model test
juju model-config ./test.yaml
juju model-config cloudinit-userdata --format yaml
juju add-machine --series focal1, check cloud-init log: cloud-init collect-logs tar -xf cloud-init.tar.gz
2, check cloud-init config: /etc/cloud/cloud.cfg
3, cloud-init is enabled: systemctl list-unit-files | grep cloud
4, /var/lib/cloud/instances/af2d721e-e38e-4937-81ad-7cc72a49c184/cloud-config.txtlp bug 1797168
试图排除https://bugs.launchpad.net/juju/bug/1797168
juju add-model test2
juju model-config container-image-metadata-urlhttps://quqi.com:443/
juju model-config image-metadata-urlhttps://quqi.com:443/
juju model-config logging-configrootDEBUG
juju add-machine --series focal
juju remove-application ceph-radosgw juju deploy ceph-radosgw --seriesxenial --tolxd:0但是也不行. 晕了。
