前序：

​ 为何要使用S3对象存储作为harbor的后端，S3存储的优势在于对于docker镜像而言，一个镜像会有N个层，而每一个层所对应的存储结构上就是一个文件，当镜像仓库存储了大量的镜像之后，如果使用文件系统作为harbor的后端势必在超大型的镜像规模下因文件层数与单文件夹下文件数过多造成的索引文件过慢的情况（当然你也可以把文件系统的索引单独拎出来，放在SSD cache上）而对象类的存储则不会出现类似问题，此外使用对象存储的其成本低廉且易组建集群，通常单桶内支持的对象数通常在亿级、十亿级甚至百亿级。

注：本文全程使用单节Harbor对接Ceph的S3存储，关于Harbor镜像仓库的高可用性，不在本次对接与考虑范围之内。

关于实现Harbor高可用需要涉及外置的postgresql的高可用集群部署，Harbor的redis外置高可用，共享存储（S3 或者集群文件系统），haporxy or lvs or nginx接入负载均衡后端多harbor 控制台ui与Docker Registry。

常规的高可用方案有以下两种：

Harbor-HA-Solution

推荐使用方案1，方案2 在进行进项同步的过程当中存在触发机制并存在延迟与不确定性。

在使用S3作为harbor后端存储又会存在harbor对于镜像回收（GC操作）的一些悬而未决的情况，以下文章会对该问题进行演示。

一、安装harbor

前置条件：

  • 关闭SELINUX
  • 关闭防火墙

1.1、下载harbor的offline或者online镜像，我使用docker-compose的方式单机启动harbor服务

下载harbor的地址：https://github.com/goharbor/harbor/releases

image-20210729225850443

1.2、解压缩并配置harbor

 1[root@s3harbor ~/harbor]# tar  -zxvf harbor-offline-installer-v2.3.1.tgz
 2harbor/harbor.v2.3.1.tar.gz
 3harbor/prepare
 4harbor/LICENSE
 5harbor/install.sh
 6harbor/common.sh
 7harbor/harbor.yml.tmpl
 8[root@s3harbor ~/harbor]# cd harbor
 9[root@s3harbor ~/harbor/harbor]# ls
10common.sh  harbor.v2.3.1.tar.gz  harbor.yml.tmpl  install.sh  LICENSE  prepare

其中解压缩的重要文件说明：

  • harbor.v2.3.1.tar.gz harbor的离线镜像

  • harbor.yml.tmpl harbor的模板文件

  • prepare 执行生成harbor docker-compose的准备脚本

  • install.sh对于已经生成好的docker-compose.yml文件进行安装部署。

对于安装的过程当中。建议首先首先将harbor.v2.3.1.tar.gz文件进行 docker load -i harbor.v2.3.1.tar.gz 直接读取到本地的docker images当中。

1.3、生成harbor所需的ssl证书文件

 1# 1、生成4096的ca私钥
 2# openssl genrsa -out ca.key 4096
 3
 4# 2、利用公钥生成10年的ca.crt公钥文件，其中涉及公钥的subj证书信息，该证书信息比较重要，更具实际情况填写
 5# CN是签署者的名称，O是组织名称，C是国家，ST是洲/省，L城市名
 6# openssl req -x509 -new -nodes -sha512 -days 36500 \
 7 -subj "/C=CN/ST=ZheJiang/L=HangZhou/O=ca/OU=Personal/CN=ca.example.com" \
 8 -key ca.key \
 9 -out ca.crt
10
11
12# 3、为Harbor的域进行生成私钥
13# openssl genrsa -out example.com.key 4096
14
15# 4、使用该私钥生成证书的请求文件example.com.csr
16# openssl req -sha512 -new \
17    -subj "/C=CN/ST=ZheJiang/L=HangZhou/O=example/OU=Personal/CN=example.com" \
18    -key example.com.key \
19    -out example.com.csr
20
21# 5、配置CA的生成证书参数，该文件很重要，根据X509 V3的标准定义了相应的alt_names,是标记了该证书正常使用者的DNS或者IP的名字
22# DNS如果有多个，可以是DNS.1 DNS.2 DNS.3 .... 依次类推，域名可以使用*作为通配符
23# IP如果有多个，在X509的证书标准当中不可以使用 10.199.158.0/24这种表达方式，在标准当中只有4个字节用于表示IP的地址的多个
24# 因此只能使用IP.1 IP.2 .... IP.256这样来进行多IP地址的指定
25# cat > v3.ext <<-EOF
26authorityKeyIdentifier=keyid,issuer
27basicConstraints=CA:FALSE
28keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
29extendedKeyUsage = serverAuth
30subjectAltName = @alt_names
31
32[alt_names]
33DNS.1=s3harbor.example.com
34DNS.2=*.example.com
35IP.1=10.199.158.81
36EOF
37
38# 6、生成harbor服务器所需的证书
39# openssl x509 -req -sha512 -days 36500 \
40    -extfile v3.ext \
41    -CA ca.crt -CAkey ca.key -CAcreateserial \
42    -in example.com.csr \
43    -out example.com.crt

1.4、配置harbor.yml文件

通过harbor.yml.tmpl文件为模板进行编辑并生成harbor.yml

以下是harbor.yml文件的详细说明：

  1# Configuration file of Harbor
  2
  3# The IP address or hostname to access admin UI and registry service.
  4# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
  5# 主机名称需要与安装主机名对应
  6hostname: s3harbor.example.com
  7
  8# http related config
  9http:
 10  # port for http, default is 80. If https enabled, this port will redirect to https port
 11  port: 80
 12
 13# https related config
 14# 默认情况下启用了https的方式进行web与docker pull的通信加密，需要确保该证书在执行prepare之前存在
 15https:
 16  # https port for harbor, default is 443
 17  port: 443
 18  # The path of cert and key files for nginx
 19  #certificate: /your/certificate/path
 20  #private_key: /your/private/key/path
 21  certificate: /root/harbor/harbor/ssl/example.com.crt
 22  private_key: /root/harbor/harbor/ssl/example.com.key
 23
 24# # Uncomment following will enable tls communication between all harbor components
 25# internal_tls:
 26#   # set enabled to true means internal tls is enabled
 27#   enabled: true
 28#   # put your cert and key files on dir
 29#   dir: /etc/harbor/tls/internal
 30
 31# Uncomment external_url if you want to enable external proxy
 32# And when it enabled the hostname will no longer used
 33# external_url: https://reg.mydomain.com:8433
 34
 35# The initial password of Harbor admin
 36# It only works in first time to install harbor
 37# Remember Change the admin password from UI after launching Harbor.
 38# web登录的harbor登录密码
 39harbor_admin_password: Harbor12345
 40
 41# Harbor DB configuration
 42# postgresql数据库的密码与相关参数
 43database:
 44  # The password for the root user of Harbor DB. Change this before any production use.
 45  password: root123
 46  # The maximum number of connections in the idle connection pool. If it <=0, no idle connections are retained.
 47  max_idle_conns: 100
 48  # The maximum number of open connections to the database. If it <= 0, then there is no limit on the number of open connections.
 49  # Note: the default number of connections is 1024 for postgres of harbor.
 50  max_open_conns: 900
 51
 52# The default data volume
 53# harbor所涉及的所有组件日志、ui与dockerpull的证书等存储的本地目录、需要注意的是，默认使用的storage_service类型是filesystem
 54# 那么该位置，也会成为docker registry 存储所有docker镜像也会放在该位置下
 55data_volume: /harbor_data
 56
 57# Harbor Storage settings by default is using /data dir on local filesystem
 58# Uncomment storage_service setting If you want to using external storage
 59# storage_service:
 60#   # ca_bundle is the path to the custom root ca certificate, which will be injected into the truststore
 61#   # of registry's and chart repository's containers.  This is usually needed when the user hosts a internal storage with self signed certificate.
 62#   ca_bundle:
 63
 64#   # storage backend, default is filesystem, options include filesystem, azure, gcs, s3, swift and oss
 65#   # for more info about this configuration please refer https://docs.docker.com/registry/configuration/
 66#   filesystem:
 67#     maxthreads: 100
 68#   # set disable to true when you want to disable registry redirect
 69#   redirect:
 70#     disabled: false
 71# 该部分比较核心，由于我使用的是s3的对象存储进行对接的，需要配置对象存储的AK SK 接入点信息，详细解释见：3.1部分
 72# 目前我的s3并未开启并使用https
 73# 需要注意，secure是验证是否验证https根证书可靠的，而不是指是否是https或者http的一个开关选项
 74storage_service:
 75  s3:
 76    region: default
 77    bucket: harbor-ceph
 78    accesskey: JAT8EXWOVX9CXSO7ZIZ5
 79    secretkey: DsggvfG43GIreyGCtrUR3BC1Bk06szelGGw0pOoQ
 80    regionendpoint: http://10.199.180.42:8060
 81    secure: false
 82    multipartcopythresholdsize: "5368709120"
 83  delete:
 84    enabled: true
 85  maintenance:
 86    uploadpurging:
 87      enabled: true
 88      age: 6h
 89      interval: 2h30m
 90      dryrun: false
 91    readonly:
 92      enabled: false
 93
 94# Trivy configuration
 95#
 96# Trivy DB contains vulnerability information from NVD, Red Hat, and many other upstream vulnerability databases.
 97# It is downloaded by Trivy from the GitHub release page https://github.com/aquasecurity/trivy-db/releases and cached
 98# in the local file system. In addition, the database contains the update timestamp so Trivy can detect whether it
 99# should download a newer version from the Internet or use the cached one. Currently, the database is updated every
100# 12 hours and published as a new release to GitHub.
101trivy:
102  # ignoreUnfixed The flag to display only fixed vulnerabilities
103  ignore_unfixed: false
104  # skipUpdate The flag to enable or disable Trivy DB downloads from GitHub
105  #
106  # You might want to enable this flag in test or CI/CD environments to avoid GitHub rate limiting issues.
107  # If the flag is enabled you have to download the `trivy-offline.tar.gz` archive manually, extract `trivy.db` and
108  # `metadata.json` files and mount them in the `/home/scanner/.cache/trivy/db` path.
109  skip_update: false
110  #
111  # insecure The flag to skip verifying registry certificate
112  insecure: false
113  # github_token The GitHub access token to download Trivy DB
114  #
115  # Anonymous downloads from GitHub are subject to the limit of 60 requests per hour. Normally such rate limit is enough
116  # for production operations. If, for any reason, it's not enough, you could increase the rate limit to 5000
117  # requests per hour by specifying the GitHub access token. For more details on GitHub rate limiting please consult
118  # https://developer.github.com/v3/#rate-limiting
119  #
120  # You can create a GitHub token by following the instructions in
121  # https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line
122  #
123  # github_token: xxx
124
125jobservice:
126  # Maximum number of job workers in job service
127  max_job_workers: 10
128
129notification:
130  # Maximum retry count for webhook job
131  webhook_job_max_retry: 10
132
133chart:
134  # Change the value of absolute_url to enabled can enable absolute url in chart
135  absolute_url: disabled
136
137# Log configurations
138log:
139  # options are debug, info, warning, error, fatal
140  level: info
141  # configs for logs in local storage
142  local:
143    # Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.
144    rotate_count: 50
145    # Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
146    # If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G
147    # are all valid.
148    rotate_size: 200M
149    # The directory on your host that store log
150    location: /var/log/harbor
151
152  # Uncomment following lines to enable external syslog endpoint.
153  # external_endpoint:
154  #   # protocol used to transmit log to external endpoint, options is tcp or udp
155  #   protocol: tcp
156  #   # The host of external endpoint
157  #   host: localhost
158  #   # Port of external endpoint
159  #   port: 5140
160
161#This attribute is for migrator to detect the version of the .cfg file, DO NOT MODIFY!
162_version: 2.3.0
163
164# Uncomment external_database if using external database.
165# external_database:
166#   harbor:
167#     host: harbor_db_host
168#     port: harbor_db_port
169#     db_name: harbor_db_name
170#     username: harbor_db_username
171#     password: harbor_db_password
172#     ssl_mode: disable
173#     max_idle_conns: 2
174#     max_open_conns: 0
175#   notary_signer:
176#     host: notary_signer_db_host
177#     port: notary_signer_db_port
178#     db_name: notary_signer_db_name
179#     username: notary_signer_db_username
180#     password: notary_signer_db_password
181#     ssl_mode: disable
182#   notary_server:
183#     host: notary_server_db_host
184#     port: notary_server_db_port
185#     db_name: notary_server_db_name
186#     username: notary_server_db_username
187#     password: notary_server_db_password
188#     ssl_mode: disable
189
190# Uncomment external_redis if using external Redis server
191# external_redis:
192#   # support redis, redis+sentinel
193#   # host for redis: <host_redis>:<port_redis>
194#   # host for redis+sentinel:
195#   #  <host_sentinel1>:<port_sentinel1>,<host_sentinel2>:<port_sentinel2>,<host_sentinel3>:<port_sentinel3>
196#   host: redis:6379
197#   password:
198#   # sentinel_master_set must be set to support redis+sentinel
199#   #sentinel_master_set:
200#   # db_index 0 is for core, it's unchangeable
201#   registry_db_index: 1
202#   jobservice_db_index: 2
203#   chartmuseum_db_index: 3
204#   trivy_db_index: 5
205#   idle_timeout_seconds: 30
206
207# Uncomment uaa for trusting the certificate of uaa instance that is hosted via self-signed cert.
208# uaa:
209#   ca_file: /path/to/ca
210
211# Global proxy
212# Config http proxy for components, e.g. http://my.proxy.com:3128
213# Components doesn't need to connect to each others via http proxy.
214# Remove component from `components` array if want disable proxy
215# for it. If you want use proxy for replication, MUST enable proxy
216# for core and jobservice, and set `http_proxy` and `https_proxy`.
217# Add domain to the `no_proxy` field, when you want disable proxy
218# for some special registry.
219proxy:
220  http_proxy:
221  https_proxy:
222  no_proxy:
223  components:
224    - core
225    - jobservice
226    - trivy
227# 建议将prometheus监控的export功能启用，暴露harbor的metrics
228metric:
229  enabled: false
230  port: 9090
231  path: /metrics

注意：在解压缩完之后执行prepare脚本主要检测以下内容：

  • 如果不存在harbor.yml文件不可以直接进行prepare操作，否则会提示错误：
1prepare base dir is set to /root/harbor/harbor
2no config file: /root/harbor/harbor/harbor.yml
  • 检测harbor.yml文件当中hostname:字段是否与主机的hostname是否一致，不一致则会报错：
1prepare base dir is set to /root/harbor/harbor
2Error happened in config validation...
3ERROR:root:Please specify hostname
  • 启用ssl的https连接状态下ssl_cert位置是否文件存在
1repare base dir is set to /root/harbor/harbor
2Error happened in config validation...
3ERROR:root:Error: The protocol is https but attribute ssl_cert is not set

1.5、执行prepare生成对应的docker-compose.yml文件与common/config下等文件

 1# ./prepare
 2prepare base dir is set to /root/harbor/harbor
 3Clearing the configuration file: /config/log/logrotate.conf
 4Clearing the configuration file: /config/log/rsyslog_docker.conf
 5Clearing the configuration file: /config/portal/nginx.conf
 6Generated configuration file: /config/portal/nginx.conf
 7Generated configuration file: /config/log/logrotate.conf
 8Generated configuration file: /config/log/rsyslog_docker.conf
 9Generated configuration file: /config/nginx/nginx.conf
10Generated configuration file: /config/core/env
11Generated configuration file: /config/core/app.conf
12Generated configuration file: /config/registry/config.yml
13Generated configuration file: /config/registryctl/env
14Generated configuration file: /config/registryctl/config.yml
15Generated configuration file: /config/db/env
16Generated configuration file: /config/jobservice/env
17Generated configuration file: /config/jobservice/config.yml
18Generated and saved secret to file: /data/secret/keys/secretkey
19Successfully called func: create_root_cert
20Generated configuration file: /compose_location/docker-compose.yml
21Clean up the input dir

1.6、查看docker registry的配置关于存储部分是否已经修改成为了s3的存储位置

image-20210801153114634

这里有个大坑.\prepare出来的common/config/registry/config.yml 并不会像我们在harbor.yml文件当中配置的将{.storage.maintenance.uploadpuring}当中有关age和interval的时间配置上，当然：默认肯定也是启用的，只不过是168h和24h而已，所以我手动修改了一下改配置文件。添加上了age、interval、readonly等参数

1.7、 执行install.sh脚本进行安装有关并启动harbor 的docker compose

在该阶段会验证当前主机的docker版本是否满足harbor所需的要求，2.3.1的harbor要求docker版本大于：17.06.0+，golang版本大于1.12.0+，docker-compose 版本大于1.18.0+

 1[root@s3harbor ~/harbor/harbor]# ./install.sh
 2
 3[Step 0]: checking if docker is installed ...
 4
 5Note: docker version: 18.09.5
 6
 7[Step 1]: checking docker-compose is installed ...
 8
 9Note: docker-compose version: 1.18.0
10
11[Step 2]: loading Harbor images ...
12Loaded image: goharbor/harbor-exporter:v2.3.1
13Loaded image: goharbor/notary-signer-photon:v2.3.1
14Loaded image: goharbor/trivy-adapter-photon:v2.3.1
15Loaded image: goharbor/harbor-portal:v2.3.1
16Loaded image: goharbor/harbor-log:v2.3.1
17Loaded image: goharbor/redis-photon:v2.3.1
18Loaded image: goharbor/registry-photon:v2.3.1
19Loaded image: goharbor/chartmuseum-photon:v2.3.1
20Loaded image: goharbor/harbor-core:v2.3.1
21Loaded image: goharbor/nginx-photon:v2.3.1
22Loaded image: goharbor/harbor-jobservice:v2.3.1
23Loaded image: goharbor/harbor-db:v2.3.1
24Loaded image: goharbor/harbor-registryctl:v2.3.1
25Loaded image: goharbor/notary-server-photon:v2.3.1
26Loaded image: goharbor/prepare:v2.3.1
27
28
29[Step 3]: preparing environment ...
30
31[Step 4]: preparing harbor configs ...
32prepare base dir is set to /root/harbor/harbor
33Clearing the configuration file: /config/core/app.conf
34Clearing the configuration file: /config/core/env
35Clearing the configuration file: /config/db/env
36Clearing the configuration file: /config/registry/config.yml
37Clearing the configuration file: /config/registry/passwd
38Clearing the configuration file: /config/jobservice/config.yml
39Clearing the configuration file: /config/jobservice/env
40Clearing the configuration file: /config/registryctl/config.yml
41Clearing the configuration file: /config/registryctl/env
42Clearing the configuration file: /config/log/logrotate.conf
43Clearing the configuration file: /config/log/rsyslog_docker.conf
44Clearing the configuration file: /config/portal/nginx.conf
45Clearing the configuration file: /config/nginx/nginx.conf
46Generated configuration file: /config/portal/nginx.conf
47Generated configuration file: /config/log/logrotate.conf
48Generated configuration file: /config/log/rsyslog_docker.conf
49Generated configuration file: /config/nginx/nginx.conf
50Generated configuration file: /config/core/env
51Generated configuration file: /config/core/app.conf
52Generated configuration file: /config/registry/config.yml
53Generated configuration file: /config/registryctl/env
54Generated configuration file: /config/registryctl/config.yml
55Generated configuration file: /config/db/env
56Generated configuration file: /config/jobservice/env
57Generated configuration file: /config/jobservice/config.yml
58Creating harbor-log ... done
59Generated configuration file: /compose_location/docker-compose.yml
60Clean up the input dir
61
62
63Creating harbor-db ... done
64Creating harbor-core ... done
65Creating network "harbor_harbor" with the default driver
66Creating harbor-jobservice ... done
67Creating registryctl ...
68Creating registry ...
69Creating redis ...
70Creating harbor-portal ...
71Creating harbor-db ...
72Creating harbor-core ...
73Creating harbor-jobservice ...
74Creating nginx ...
75✔ ----Harbor has been installed and started successfully.----

1.8、管理员登录

harbor的web管理员访问不是强制域名访问的，可以在访问的客户端修改hosts文件的方式进行域名解析访问或者直接https://IP的方式进行访问。（注意证书当中的使用者可选名称）

image-20210730012519523

1.9、镜像上传

​ 由于是自签证书，常规方式一般是修改docker.service当中添加--insecure-registry地址或配置默认的docker daemon的配置文件/etc/docker/daemon.json 添加{"insecure-registries":["s3harbor.example.com"]}的方式来进行insecure类型的进行上传、下载、自签根CA信任等操作。但是以上方式会造成docker节点的docker.service要进行重启才生效。

​ 如果不想重启docker从而能够信任harbor的证书，建议以下方法（如果在生产环境下的kubernetes集群、node节点拉取自签证书的镜像仓库强烈建议使用该方法）：

 1# 创建目录，该目录默认不存在,确保最后的文件夹名，需要与harbor的域名完全一致
 2# mkdir -p /etc/docker/certs.d/s3harbor.example.com
 3# 拷贝harbor的公钥与自签CA的私钥至该文件夹下，并确保公钥名符合以下条件
 4# pwd
 5/etc/docker/certs.d/s3harbor.example.com
 6# ls -alh
 7total 12K
 8drwxr-xr-x 2 root root   85 Jul 30 01:41 .
 9drwxr-xr-x 4 root root   58 Jul 30 01:36 ..
10-rw-r--r-- 1 root root 2.0K Jul 30 01:39 ca.crt
11-rw-r--r-- 1 root root 2.1K Jul 30 01:41 s3harbor.example.com.cert
12-rw-r--r-- 1 root root 3.2K Jul 30 01:41 s3harbor.example.com.key
13# 执行dockerlogin已经能够正确登录，并且不会报告X509错误
14# docker login s3harbor.example.com
15Username: admin
16Password:
17WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
18Configure a credential helper to remove this warning. See
19https://docs.docker.com/engine/reference/commandline/login/#credentials-store
20
21Login Succeeded
22# 推送进行至该harbor，使用该方法不存在需要重启docker的方式。
23# docker push s3harbor.example.com/library/wordpress:4.8-apache
24The push refers to repository [s3harbor.example.com/library/wordpress]
252ff5b2ab6416: Pushed
264f2aeb865fa0: Pushed
2724605e7ca88b: Pushed
286594bf4ea5b9: Pushed
29749e8aaa7dd4: Pushed
30493137409f3e: Pushed
318933dc910eee: Pushed
3261a961ab5d2b: Pushed
33fa7f9311a060: Pushed
34a9aa8861270e: Pushed
35dcdbe9fe2ca1: Pushed
362f6273a5f133: Pushed
374c0354ed71f4: Pushed
382c3aa4e96952: Pushed
395cd2e0cfe892: Pushed
40c3d26400d3ff: Pushed
4137412c153883: Pushing [====>                                              ]  19.54MB/200.2MB
42c01c63c6823d: Pushing [=====>                                             ]  14.55MB/123.4MB

附：

  • 还有其他的方法比进行根CA信任可直接将该根证书CA添加至操作系统根证书信任，但也需要重启docker进程。
1# yum install -y ca-certificates
2# 默认目录存在
3# cp ca.crt  /etc/pki/ca-trust/source/anchors/
4# update-ca-trust

当执行完该命令之后 本质上是将ca.crt的证书添加到了操作系统本地的根证书文件： /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem与/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt 当中。从而让操作系统添加了根ca。该文件有以下个软连接：

1"/etc/pki/tls/cert.pem"                   -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2"/etc/pki/tls/certs/ca-bundle.crt"        -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
3"/etc/pki/tls/certs/ca-bundle.trust.crt"  -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
4"/etc/ssl/certs/ca-bundle.crt"            -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
5"/etc/ssl/certs/ca-bundle.trust.crt"      -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.cr
  • 获取服务器证书，并查阅其状态
1# openssl s_client -showcerts -servername server -connect  s3harbor.example.com:443
  • 验证服务器证书的与ca证书签署的一致性
1# 自签ca服务器的时间一定要正确否则各种未知问题：
2# 比如：该报错是因为CA证书还未到生效时间，因为之前我的ca生成的证书根证书服务器时钟快了十几个小时。
3# openssl verify -verbose -CAfile ca.crt s3harbor.example.com.cert
4s3harbor.example.com.cert: C = CN, ST = ZheJiang, L = HangZhou, O = ca, OU = Personal, CN = ca.example.com
5error 9 at 1 depth lookup:certificate is not yet valid
6
7# openssl verify -verbose -CAfile ca.crt s3harbor.example.com.cert
8s3harbor.example.com.cert: OK

1.10 查看harbor S3 状态

与使用本地存储不同，当对接了S3之后无法查阅到quota的配额信息，只能够看到自己对接的用户使用了了多少，存储空间容量上限决定取决于对接的桶的权限与配额，需要使用radosgw-admin进行查询。

image-20210730051659641

image-20210730022706343

image-20210730024452430

查看该镜像使用了多少个对象：

image-20210730023953391 

1# s3cmd ls --recursive   s3://harbor-ceph/

查看该镜像层是数：

image-20210730024222872

在s3的存储当中，验证可知道一个镜像层对应s3当中一个对象。

至此最简单版本的harbor安装部署已经完成并对接至后端存储，并尝试进行上传镜像与删除操作，

二、harbor的镜像安全扫描、Helm模板管理、Notary安全资料库功能启用与日常操作

2.1、启用功能

  • 停止当前harbor docker-compose，并重新执行install.sh。
 1# docker-compose ps
 2      Name                     Command               State                      Ports
 3--------------------------------------------------------------------------------------------------------
 4harbor-core         /harbor/entrypoint.sh            Up
 5harbor-db           /docker-entrypoint.sh 96 13      Up
 6harbor-jobservice   /harbor/entrypoint.sh            Up
 7harbor-log          /bin/sh -c /usr/local/bin/ ...   Up      127.0.0.1:1514->10514/tcp
 8harbor-portal       nginx -g daemon off;             Up
 9nginx               nginx -g daemon off;             Up      0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp
10redis               redis-server /etc/redis.conf     Up
11registry            /home/harbor/entrypoint.sh       Up
12registryctl         /home/harbor/start.sh            Up
13
14# docker-compose down -v
15Stopping harbor-jobservice ... done
16Stopping nginx             ... done
17Stopping harbor-core       ... done
18Stopping harbor-db         ... done
19Stopping harbor-portal     ... done
20Stopping registryctl       ... done
21Stopping registry          ... done
22Stopping redis             ... done
23Stopping harbor-log        ... done
24Removing harbor-jobservice ... done
25Removing nginx             ... done
26Removing harbor-core       ... done
27Removing harbor-db         ... done
28Removing harbor-portal     ... done
29Removing registryctl       ... done
30Removing registry          ... done
31Removing redis             ... done
32Removing harbor-log        ... done
33Removing network harbor_harbor
  • 配置所需的附加组件服务
1# ./install.sh --with-notary --with-trivy --with-chartmuseum
  • 启动docker-compse
1# docker-compose up -d

注意：在所有的docker-compose命令确保都在包含docker-compse.yml文件的目录下执行.

2.2 通常建议将harbor的docker-compose做成systemd管理的系统服务

 1# cat /usr/lib/systemd/system/harbor.service
 2[Unit]
 3Description=Harbor
 4After=docker.service systemd-networkd.service systemd-resolved.service
 5Requires=docker.service
 6Documentation=http://github.com/vmware/harbor
 7
 8[Service]
 9Type=simple
10Restart=on-failure
11RestartSec=5
12ExecStart=/usr/bin/docker-compose -f  /root/kubedeploy/harbor/v2.3.1/harbor/docker-compose.yml up
13ExecStop=/usr/bin/docker-compose -f  /root/kubedeploy/harbor/v2.3.1/harbor/docker-compose.yml down
14
15[Install]
16WantedBy=multi-user.target
17
18# systemctl enable harbor.service &&  systemctl restart harbor.service

三、配置S3存储

​ 如何搭建支持S3协议的存储有多种方案，生产环境上通常我使用Ceph的radosgw 作为对象存储对外进行提供服务。关于radosgw如何部署搭建与调优不在本文范围之内，因而不进行赘述。

3.1、常见的S3存储是一套标准的Http/Https对象存储接口，而在harbor上涉及的参数包含：

 1region # 亚马逊S3存储的区域参数，可以为空
 2bucket # 桶用于存储对象，桶当中有文件个数、总文件存放大小阈值，桶用户的权限（ListBucket、GetObject、PutObject、DeleteObject）等信息。
 3accesskey #用户所属的AK
 4secretkey #用户的所属SK
 5regionednpoint #对象存储的接入点信息
 6secure #是否为https信任
 7multipartcopythresholdsize #单个对象文件的最大大小，单位为Byte
 8delete: # 是否能够删除对象存储上的对象，默认上false的，需要打开，否则无法删除对象，存储桶上的数据会一直增长。
 9  enabled: true 
10maintenance:    # 维护模块
11    uploadpurging: #是否进行垃圾与孤儿对象的清理，默认是开启的，默认age表示孤儿对象保存的时间是168小时1个星期
12      enabled: true
13      age: 6h
14      interval: 2h30m  # 孤儿对象清理进程是一个后台进程，默认会24h执行一次，这里我改短了一些。
15      dryrun: false
16    readonly:
17      enabled: false  # 不要设置为只读模式。默认

更多的配置有关docker-registry 的配置参数：

官方Docker-registry配置详解

注：悬而未决的对象回收问题（20210801）：存在一种现象，即：当我在harbor当中删除不需要的圾镜像时，后端的s3对象的删除动作不是马上触发的，也就是说镜像岁对应的s3对象删除不是实时进行删除，而是异步的定时回收，至于具体时间，和harbor当中配置的垃圾清理定时任务有关。尝试过配置1h一次进行执行垃圾清理，但是对于垃圾的对象，即使进行Docker-registry层面的gc依然会保留1-2小时，具体原因未知。

是否与docker-registry当中maintenance的孤儿对象存在联系？

此外，即使清理垃圾，也依然会存在如下的链接文件，占用少量对象存储桶的空间：

image-20210801151724926

对象存储当中的残存数据，为link文件或data文件

image-20210801151439345

image-20210801152141739

3.2、对象存储查询等相关命令

  1# 使用radosgw-admin 命令来获取用户与桶信息命令：
  2# radosgw-admin metadata list user
  3[
  4    "sds.s3_lb.user",
  5    "myuser01",
  6    "test1",
  7    "sds.zone.user"
  8]
  9
 10# radosgw-admin user info --uid="myuser01"
 11{
 12    "user_id": "myuser01",
 13    "display_name": "myuser01",
 14    "email": "263587092@qq.com",
 15    "suspended": 0,
 16    "max_buckets": 1000,
 17    "auid": 0,
 18    "subusers": [],
 19    "keys": [
 20        {
 21            "user": "myuser01",
 22            "access_key": "JAT8EXWOVX9CXSO7ZIZ5",
 23            "secret_key": "DsggvfG43GIreyGCtrUR3BC1Bk06szelGGw0pOoQ"
 24        }
 25    ],
 26    "swift_keys": [],
 27    "caps": [],
 28    "op_mask": "read, write, delete",
 29    "default_placement": "",
 30    "placement_tags": [],
 31    "bucket_quota": {
 32        "enabled": true,
 33        "check_on_raw": true,
 34        "max_size": -1,
 35        "max_size_kb": -1,
 36        "max_objects": -1,
 37        "local_max_size": -1,
 38        "local_max_size_kb": -1,
 39        "local_max_objects": 100000000,
 40        "external_max_size": -1,
 41        "external_max_size_kb": -1,
 42        "external_max_objects": -1
 43    },
 44    "user_quota": {
 45        "enabled": true,
 46        "check_on_raw": true,
 47        "max_size": -1,
 48        "max_size_kb": -1,
 49        "max_objects": -1,
 50        "local_max_size": -1,
 51        "local_max_size_kb": -1,
 52        "local_max_objects": -1,
 53        "external_max_size": -1,
 54        "external_max_size_kb": -1,
 55        "external_max_objects": -1
 56    },
 57    "temp_url_keys": [],
 58    "type": "rgw",
 59    "enable_poll": false,
 60    "enable_location": true
 61}
 62
 63
 64# radosgw-admin  bucket list
 65[
 66    "Czssss",
 67    "bk01",
 68    "Czsss",
 69    "mybucket01",
 70    "test1",
 71    "Czssss01",
 72    "first_bucket"
 73    "harbor-ceph"
 74]
 75# radosgw-admin metadata list bucket.instance
 76[
 77    "bk01:bfc5b21b-f0c2-4f43-9ab3-677ae9c5118e.445796.1",
 78    "first_bucket:bfc5b21b-f0c2-4f43-9ab3-677ae9c5118e.632623.1",
 79    "test1:bfc5b21b-f0c2-4f43-9ab3-677ae9c5118e.419543.1",
 80    "Czsss:bfc5b21b-f0c2-4f43-9ab3-677ae9c5118e.659613.2",
 81    "mybucket01:bfc5b21b-f0c2-4f43-9ab3-677ae9c5118e.666004.1",
 82    "Czssss:bfc5b21b-f0c2-4f43-9ab3-677ae9c5118e.659613.3",
 83    "harbor-ceph:bfc5b21b-f0c2-4f43-9ab3-677ae9c5118e.666851.1",
 84    "Czssss01:bfc5b21b-f0c2-4f43-9ab3-677ae9c5118e.659613.4"
 85]
 86
 87# 查询bucket 配额信息
 88# radosgw-admin bucket stats --bucket=harbor-ceph
 89{
 90    "bucket": "harbor-ceph",
 91    "tenant": "",
 92    "virtual_bucket": false,
 93    "zonegroup": "8e6a847f-e995-41d5-9d80-62ab1c8b7d2b",
 94    "placement_rule": "policy01",
 95    "storage_class_matching_rule": {
 96        "enable_matching_rule": false,
 97        "enable_x_amz_storage_class": true,
 98        "mismatching_action": 0,
 99        "matching_rules": []
100    },
101    "explicit_placement": {
102        "data_pool": "pool-1c0607ddc8df42a492ac8714a9e77d75",
103        "data_extra_pool": "pool-1c0607ddc8df42a492ac8714a9e77d75",
104        "index_pool": "pool-1c0607ddc8df42a492ac8714a9e77d75"
105    },
106    "data_big_pool": "",
107    "data_small_pool": "",
108    "compress": false,
109    "crypto": false,
110    "flags_value": 1664,
111    "enabled_flags": [
112        "data_sync_disable",
113        "tier_worm",
114        "tier_cache"
115    ],
116    "id": "bfc5b21b-f0c2-4f43-9ab3-677ae9c5118e.666851.1",
117    "marker": "bfc5b21b-f0c2-4f43-9ab3-677ae9c5118e.666851.1",
118    "index_type": "Normal",
119    "owner": "myuser01",
120    "owner_zone": "bfc5b21b-f0c2-4f43-9ab3-677ae9c5118e",
121    "mtime": "2021-07-29 19:12:31.153142Z",
122    "num_shards": 1277,
123    "max_objects_per_shard": 1000000,
124    "restore_days": 1,
125    "usage": {
126        "rgw.main": {
127            "size": 170528640,
128            "size_actual": 170672128,
129            "size_utilized": 170528640,
130            "size_kb": 166532,
131            "size_kb_actual": 166672,
132            "size_kb_utilized": 166532,
133            "num_objects": 42,
134            "size_stat_info": {
135                "0": 35,
136                "2": 2,
137                "3": 4,
138                "4": 1
139            }
140        },
141        "rgw.multimeta": {
142            "size": 0,
143            "size_actual": 0,
144            "size_utilized": 0,
145            "size_kb": 0,
146            "size_kb_actual": 0,
147            "size_kb_utilized": 0,
148            "num_objects": 0,
149            "size_stat_info": {
150                "0": 0
151            }
152        },
153        "rgw.local.main": {
154            "size": 170528640,
155            "size_actual": 170672128,
156            "size_utilized": 170528640,
157            "size_kb": 166532,
158            "size_kb_actual": 166672,
159            "size_kb_utilized": 166532,
160            "num_objects": 42,
161            "size_stat_info": {
162                "0": 35,
163                "2": 2,
164                "3": 4,
165                "4": 1
166            }
167        },
168        "rgw.local.multimeta": {
169            "size": 0,
170            "size_actual": 0,
171            "size_utilized": 0,
172            "size_kb": 0,
173            "size_kb_actual": 0,
174            "size_kb_utilized": 0,
175            "num_objects": 0,
176            "size_stat_info": {
177                "0": 0
178            }
179        },
180        "class_usages": [
181            {
182                "key": "class_0",
183                "val": {
184                    "rgw.main": {
185                        "size": 170528640,
186                        "size_actual": 170672128,
187                        "size_utilized": 170528640,
188                        "size_kb": 166532,
189                        "size_kb_actual": 166672,
190                        "size_kb_utilized": 166532,
191                        "num_objects": 42,
192                        "size_stat_info": {
193                            "0": 35,
194                            "2": 2,
195                            "3": 4,
196                            "4": 1
197                        }
198                    },
199                    "rgw.multimeta": {
200                        "size": 0,
201                        "size_actual": 0,
202                        "size_utilized": 0,
203                        "size_kb": 0,
204                        "size_kb_actual": 0,
205                        "size_kb_utilized": 0,
206                        "num_objects": 0,
207                        "size_stat_info": {
208                            "0": 0
209                        }
210                    }
211                }
212            }
213        ]
214    },
215    "bucket_quota": {
216        "enabled": true,
217        "check_on_raw": true,
218        "max_size": 644245094400,
219        "max_size_kb": 629145600,
220        "max_objects": -1,
221        "local_max_size": 536870912000,
222        "local_max_size_kb": 524288000,
223        "local_max_objects": 100000000,
224        "external_max_size": 107374182400,
225        "external_max_size_kb": 104857600,
226        "external_max_objects": -1
227    },
228    "backup_group": {},
229    "enable_mdsearch": "true"
230}

更多关于bucket的配置-RedHat

更多关于bucket的配置-Ceph官方

3.3、配置s3cmd命令对S3存储进行使用，

（建议harbor对接前测试以下访问s3的状态是否正常，一面后期踩坑。）

linux主机安装s3cmd命令，该包位于epel源当中

1# yum install s3cmd -y
2# 初始化s3cmd并对接使用s3存储
3
4# s3cmd --configuration

image-20210730004719054

3.4、s3cmd使用命令：

 1# 1、创建桶
 2# s3cmd mb s3://mybucket01
 3
 4# 2、删除桶
 5# s3cmd rb s3://mybucket01
 6
 7# 3、上传对象（可以一个或者多个）到桶当中
 8# s3cmd put aaa.text abc.txt s3://mybucket01
 9# s3cmd put /tmp/ca/ca.key s3://mybucket01/cert/ca.key
10upload: '/tmp/ca/ca.key' -> 's3://mybucket01/cert/ca.key'  [1 of 1]
11 3243 of 3243   100% in    0s    62.18 KB/s  done
12# s3cmd put /tmp/ca/ca.key s3://mybucket01/cert/
13upload: '/tmp/ca/ca.key' -> 's3://mybucket01/cert/ca.key'  [1 of 1]
14 3243 of 3243   100% in    0s    75.64 KB/s  done
15# s3cmd put /tmp/ca/example.com.crt s3://mybucket01/cert/
16upload: '/tmp/ca/example.com.crt' -> 's3://mybucket01/cert/example.com.crt'  [1 of 1]
17 2106 of 2106   100% in    0s    35.82 KB/s  done
18 
19# 4、列出对象
20# s3cmd  ls s3://mybucket01/cert/
212021-07-29 16:40         2029  s3://mybucket01/cert/ca.crt
222021-07-29 16:40         3243  s3://mybucket01/cert/ca.key
232021-07-29 16:41         2106  s3://mybucket01/cert/example.com.crt
24# 5、递归枚举`文件夹`下所有对象
25# s3cmd ls --recursive   s3://harbor-ceph/
26
27# 6、列出所有能够访问的桶当中的对象
28# s3cmd la
292021-07-28 10:46     67609088  s3://Czsss/smartctl.tar.gz
30
312021-07-28 10:36            0  s3://Czssss/aaa.text
32
332021-07-28 10:36            0  s3://Czssss01/aaa.text
34
35                          DIR  s3://bk01/第一/
362021-07-28 10:37            0  s3://bk01/aaa.text
37
382021-07-28 10:11            0  s3://first_bucket/aaa.text
392021-07-28 09:58            0  s3://first_bucket/file.txt
402021-07-28 10:31     67609088  s3://first_bucket/smartctl.tar.gz
41
42
432021-07-28 10:37            0  s3://mybucket01/aaa.text
442021-07-27 10:08            0  s3://mybucket01/abc.txt
452021-07-27 09:36   1073741824  s3://mybucket01/d.blk
462021-07-27 10:09            0  s3://mybucket01/def.txt
472021-07-28 10:39     67609088  s3://mybucket01/smartctl.tar.gz
48
49# 7、删除对象
50# s3cmd del s3://mybucket01/d.blk
51
52# 8、从桶当中下载文件到本地
53# s3cmd get s3://mybucket01/def.txt def.txt
54
55# 9、获取桶当中的对象信息
56# s3cmd info s3://mybucket01/smartctl.tar.gz
57s3://mybucket01/smartctl.tar.gz (object):
58   File size: 67609088
59   Last mod:  Wed, 28 Jul 2021 10:39:01 GMT
60   MIME type: application/x-tar
61   Storage:   class0
62   MD5 sum:   b37ed31dfc0144defee5e6f71914c799
63   SSE:       none
64   Policy:    none
65   CORS:      none
66   ACL:       myuser01: FULL_CONTROL
67   x-amz-meta-s3cmd-attrs: atime:1627468314/ctime:1627370268/gid:0/gname:root/md5:b37ed31dfc0144defee5e6f71914c799/mode:33152/mtime:1627370268/uid:0/uname:root
68
69# 10、查看桶当中文件夹的磁盘使用量
70# s3cmd du s3://mybucket01/cert
71        7378       3 objects s3://mybucket01/cert
72# s3cmd put /tmp/ca/example.com.key s3://mybucket01/cert/
73upload: '/tmp/ca/example.com.key' -> 's3://mybucket01/cert/example.com.key'  [1 of 1]
74 3243 of 3243   100% in    0s   312.63 KB/s  done
75# s3cmd du s3://mybucket01/cert
76       10621       4 objects s3://mybucket01/cert
77
78# 11、本地文件同步至桶当中进行存储
79# s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR

附：关于使用s3cmd当中的坑:

1# s3cmd  ls -v
2ERROR: S3 error: 403 (RequestTimeTooSkewed)
3# 该报错是因为本地时钟与s3上的时钟不对齐导致的，需要配置ntpd或者chronyd。

文章：

更多Docker-registry配置方法-Docker官方

更多ceph bucket的配置-RedHat官方

更多ceph bucket的配置-Ceph官方

PR Remove the layer’s link by garbage-collect #2288 ，gc过后无法删除link文件和data文件的问题