15

gitlab install and CI pipeline

 3 years ago
source link: https://feiyang233.club/post/gitlab/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.

gitlab install and CI pipeline

Posted on

2021-01-09

|

Edited on 2021-01-10

| In develop

| Comments: 0 Comments

| Views: 16

为了测试 gitlab CI pipeline， 从而在家里的台式机搭建了测试环境。

为什么进行这次实验？因为自己平时只是用到了 gitlab，它对于我来说是一个黑盒。
为了搞清楚 CI job 里面的一些 function 用法，自己搭建一套环境。一是方便实验，二是自己不能在公司的环境上实验，担心造成破坏。

Update history

2021-01-09 初稿，妹子什么时候才能好好学习呀

Environment

默认技能：

  • docker， 虚拟机上安装 docker 的过程就省略了
  • DNS， 因为是自定义的域名，所以需要绑定在本地 hosts

最开始 Ubuntu 机器只有 2 cpu 4GB， 安装 gitlab 非常卡顿， load 高到 40+， 增加配置后，顺利安装

Name Operating system CPU RAM Disk IP gitlab-main Ubuntu 18.04.5 LTS 4 8G 50G 192.168.17.3 gitlab-runner CentOS 7.8.2003 1 2G 20G 192.168.17.4

Install gitlab

官方文档：https://docs.gitlab.com/

gitlab

本次实验，为了快速安装选择了 docker 模式。 https://docs.gitlab.com/omnibus/docker/README.html

本地 22 端口被 SSH 占用了，所以改为了 2222，这里需要注意的是，我用的是 社区版本 ce

sudo docker run --detach \
  --hostname gitlab.feiyang.com \
  --publish 443:443 --publish 80:80 --publish 2222:22 \
  --name gitlab \
  --restart always \
  --volume $GITLAB_HOME/config:/etc/gitlab \
  --volume $GITLAB_HOME/logs:/var/log/gitlab \
  --volume $GITLAB_HOME/data:/var/opt/gitlab \
  gitlab/gitlab-ce:latest

如果还需要修改配置，可以 attach 到容器内部，进行修改。

例如修改 relative URL https://docs.gitlab.com/omnibus/settings/configuration.html#enable-relative-url-in-gitlab

docker exec -it gitlab bash

vi /etc/gitlab/gitlab.rb


# 指定host地址
external_url 'http://gitlab.feiyang.com'

# 重新加载配置文件并重启服务
gitlab-ctl reconfigure
gitlab-ctl restart

第一次登陆，一定要以 IP 地址登陆，而不是用 localhost。 否则设置 root 新密码就会遇到错误 8 errors prohibited this user from being saved
成功登陆网友以后，我创建了一个新用户，并且设置了 SSH Key，创建了一个 test repo

在这里需要的注意的是，如果没有设置域名 http://gitlab.feiyang.com 那在repo 页面上 git clong 的 url 可能不对，导致无法解析并下载。

runner

本次实验，为了快速安装选择了 docker 模式。 https://docs.gitlab.com/runner/install/docker.html

docker volume create gitlab-runner-config

docker run -d --name gitlab-runner --restart always \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v gitlab-runner-config:/etc/gitlab-runner \
    gitlab/gitlab-runner:latest

最好先进入容器设置一下 extra_hosts = [“gitlab.feiyang.com:192.168.17.3”] (这里后文有详细介绍 config.toml 可以搜索关键词) 否则注册 runner 的时候，只能输入 IP
下一步就是注册，前提你需要去 gitlab 网页上 CI/CD section runner 详情页上获得 token

[root@centos7 jy576]# docker run --rm -it -v gitlab-runner-config:/etc/gitlab-runner gitlab/gitlab-runner:latest register
Runtime platform                                    arch=amd64 os=linux pid=7 revision=943fc252 version=13.7.0
Running in system-mode.                            

Enter the GitLab instance URL (for example, https://gitlab.com/):
http://gitlab.feiyang.com/
Enter the registration token:
kXmj4UA9Z2_NL2sa2Z6z
Enter a description for the runner:
[43420abf216d]: centos7       
Enter tags for the runner (comma-separated):
ci,ssh
ERROR: Registering runner... failed                 runner=kXmj4UA9 status=couldn't execute POST against http://gitlab.feiyang.com/api/v4/runners: Post http://gitlab.feiyang.com/api/v4/runners: dial tcp: lookup gitlab.feiyang.com on 8.8.8.8:53: no such host
PANIC: Failed to register the runner. You may be having network problems. 
[root@centos7 jy576]# docker run --rm -it -v gitlab-runner-config:/etc/gitlab-runner gitlab/gitlab-runner:latest register
Runtime platform                                    arch=amd64 os=linux pid=8 revision=943fc252 version=13.7.0
Running in system-mode.                            

Enter the GitLab instance URL (for example, https://gitlab.com/):
http://192.168.17.3/
Enter the registration token:
kXmj4UA9Z2_NL2sa2Z6z
Enter a description for the runner:
[a7068d34147b]: centos7
Enter tags for the runner (comma-separated):
ci,ssh
Registering runner... succeeded                     runner=kXmj4UA9
Enter an executor: custom, virtualbox, docker+machine, docker, docker-ssh, parallels, shell, ssh, docker-ssh+machine, kubernetes:
docker
Enter the default Docker image (for example, ruby:2.6):
python:3.9.1-slim-buster
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded! 
[root@centos7 jy576]# docker ps
CONTAINER ID   IMAGE                         COMMAND                  CREATED         STATUS         PORTS     NAMES
926cd5798468   gitlab/gitlab-runner:latest   "/usr/bin/dumb-init …"   7 minutes ago   Up 7 minutes             gitlab-runner

回到网页端，我们需要修改一下 runner 设置，勾选上 Run untagged jobs
20210109230805.png

因为我们是自定义域名，所以我们需要在 gitlab-runner 里面进行设置，否则 CI job 无法 pull

Running with gitlab-runner 13.7.0 (943fc252)
  on centos7 FSX9yRiU
Preparing the "docker" executor
00:04
Using Docker executor with image python:3.9.1-slim-buster ...
Pulling docker image python:3.9.1-slim-buster ...
Using docker image sha256:b55839ea7a0e9bb534237d00558cb96dce4013bf7f1092966fe0e27e98f8179f for python:3.9.1-slim-buster with digest python@sha256:4d92968b26bb6b7b62d957244de86fc1054f03793577d49e85c00864eb03ca07 ...
Preparing environment
00:01
Running on runner-fsx9yriu-project-2-concurrent-0 via 926cd5798468...
Getting source from Git repository
00:00
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/feiyang/test/.git/
Created fresh repository.
fatal: unable to access 'http://gitlab.feiyang.com/feiyang/test.git/': Could not resolve host: gitlab.feiyang.com
ERROR: Job failed: exit code 1

在这里，我们需要在 runner 里面定义好 domain gitlab.feiyang.com
参考文档： https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnersdocker-section
extra_hosts: Specify hosts that should be defined in container environment
[root@centos7 jy576]# docker exec -it gitlab-runner bash
root@926cd5798468:/# cat /etc/gitlab-runner/config.toml 
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "centos7"
  url = "http://192.168.17.3/"
  token = "FSX9yRiUGxok94hMPYdt"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "python:3.9.1-slim-buster"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    extra_hosts   = ["gitlab.feiyang.com:192.168.17.3"]

Test CI

gitlab 和 runner 安装完成以后，我们就可以进行测试

Job artifacts

文档：https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#defining-artifacts-in-gitlab-ciyml

test repo 目录结构如下
├── .gitlab-ci.yml
├── lint.py
└── README.md

lint.py

import os
os.chdir(os.path.dirname(__file__))
print("feiyang test CI")
print(os.getcwd())

.gitlab-ci.yml

stages:
  - lint
  - build
  - deploy

image: python:3.9.1-slim-buster

check_path:
  stage: lint
  before_script:
    - pwd
  script:
    - cd /builds/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME && python lint.py

build_ami:
  stage: build
  before_script:
  - apt update && apt install -y curl
  script:
  - echo $CI_JOB_ID >> ip.json && curl ipinfo.io >> ip.json
  artifacts:
    expire_in: 1 year
    paths:
      - ip.json

read_artifacts:
  stage: deploy
  before_script:
  - pwd
  script:
   - cat ip.json


Running with gitlab-runner 13.7.0 (943fc252)
  on centos7 FSX9yRiU
Preparing the "docker" executor
00:05
Using Docker executor with image python:3.9.1-slim-buster ...
Pulling docker image python:3.9.1-slim-buster ...
Using docker image sha256:b55839ea7a0e9bb534237d00558cb96dce4013bf7f1092966fe0e27e98f8179f for python:3.9.1-slim-buster with digest python@sha256:4d92968b26bb6b7b62d957244de86fc1054f03793577d49e85c00864eb03ca07 ...
Preparing environment
00:00
Running on runner-fsx9yriu-project-2-concurrent-0 via 926cd5798468...
Getting source from Git repository
00:01
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/feiyang/test/.git/
Checking out b60b95c1 as master...
Removing ip.json
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:00
$ pwd
/builds/feiyang/test
$ cd /builds/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME && python lint.py
feiyang test CI
/builds/feiyang/test
Job succeeded

build

Running with gitlab-runner 13.7.0 (943fc252)
  on centos7 FSX9yRiU
Preparing the "docker" executor
00:04
Using Docker executor with image python:3.9.1-slim-buster ...
Pulling docker image python:3.9.1-slim-buster ...
Using docker image sha256:b55839ea7a0e9bb534237d00558cb96dce4013bf7f1092966fe0e27e98f8179f for python:3.9.1-slim-buster with digest python@sha256:4d92968b26bb6b7b62d957244de86fc1054f03793577d49e85c00864eb03ca07 ...
Preparing environment
00:00
Running on runner-fsx9yriu-project-2-concurrent-0 via 926cd5798468...
Getting source from Git repository
00:01
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/feiyang/test/.git/
Checking out b60b95c1 as master...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:05
$ apt update && apt install -y curl
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [121 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 Packages [260 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages [7860 B]
Fetched 8414 kB in 2s (4381 kB/s)
Reading package lists...
Building dependency tree...
Reading state information...
2 packages can be upgraded. Run 'apt list --upgradable' to see them.
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  krb5-locales libcurl4 libgssapi-krb5-2 libk5crypto3 libkeyutils1 libkrb5-3
  libkrb5support0 libldap-2.4-2 libldap-common libnghttp2-14 libpsl5 librtmp1
  libsasl2-2 libsasl2-modules libsasl2-modules-db libssh2-1 publicsuffix
Suggested packages:
  krb5-doc krb5-user libsasl2-modules-gssapi-mit
  | libsasl2-modules-gssapi-heimdal libsasl2-modules-ldap libsasl2-modules-otp
  libsasl2-modules-sql
The following NEW packages will be installed:
  curl krb5-locales libcurl4 libgssapi-krb5-2 libk5crypto3 libkeyutils1
  libkrb5-3 libkrb5support0 libldap-2.4-2 libldap-common libnghttp2-14 libpsl5
  librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db libssh2-1
  publicsuffix
0 upgraded, 18 newly installed, 0 to remove and 2 not upgraded.
Need to get 2470 kB of archives.
After this operation, 5837 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian buster/main amd64 krb5-locales all 1.17-3+deb10u1 [95.4 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 libkeyutils1 amd64 1.6-6 [15.0 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 libkrb5support0 amd64 1.17-3+deb10u1 [65.8 kB]
Get:4 http://deb.debian.org/debian buster/main amd64 libk5crypto3 amd64 1.17-3+deb10u1 [122 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 libkrb5-3 amd64 1.17-3+deb10u1 [369 kB]
Get:6 http://deb.debian.org/debian buster/main amd64 libgssapi-krb5-2 amd64 1.17-3+deb10u1 [158 kB]
Get:7 http://deb.debian.org/debian buster/main amd64 libsasl2-modules-db amd64 2.1.27+dfsg-1+deb10u1 [69.1 kB]
Get:8 http://deb.debian.org/debian buster/main amd64 libsasl2-2 amd64 2.1.27+dfsg-1+deb10u1 [106 kB]
Get:9 http://deb.debian.org/debian buster/main amd64 libldap-common all 2.4.47+dfsg-3+deb10u4 [89.8 kB]
Get:10 http://deb.debian.org/debian buster/main amd64 libldap-2.4-2 amd64 2.4.47+dfsg-3+deb10u4 [224 kB]
Get:11 http://deb.debian.org/debian buster/main amd64 libnghttp2-14 amd64 1.36.0-2+deb10u1 [85.0 kB]
Get:12 http://deb.debian.org/debian buster/main amd64 libpsl5 amd64 0.20.2-2 [53.7 kB]
Get:13 http://deb.debian.org/debian buster/main amd64 librtmp1 amd64 2.4+20151223.gitfa8646d.1-2 [60.5 kB]
Get:14 http://deb.debian.org/debian buster/main amd64 libssh2-1 amd64 1.8.0-2.1 [140 kB]
Get:15 http://deb.debian.org/debian buster/main amd64 libcurl4 amd64 7.64.0-4+deb10u1 [331 kB]
Get:16 http://deb.debian.org/debian buster/main amd64 curl amd64 7.64.0-4+deb10u1 [264 kB]
Get:17 http://deb.debian.org/debian buster/main amd64 libsasl2-modules amd64 2.1.27+dfsg-1+deb10u1 [104 kB]
Get:18 http://deb.debian.org/debian buster/main amd64 publicsuffix all 20190415.1030-1 [116 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 2470 kB in 0s (5707 kB/s)
Selecting previously unselected package krb5-locales.
(Reading database ... 6829 files and directories currently installed.)
Preparing to unpack .../00-krb5-locales_1.17-3+deb10u1_all.deb ...
Unpacking krb5-locales (1.17-3+deb10u1) ...
Selecting previously unselected package libkeyutils1:amd64.
Preparing to unpack .../01-libkeyutils1_1.6-6_amd64.deb ...
Unpacking libkeyutils1:amd64 (1.6-6) ...
Selecting previously unselected package libkrb5support0:amd64.
Preparing to unpack .../02-libkrb5support0_1.17-3+deb10u1_amd64.deb ...
Unpacking libkrb5support0:amd64 (1.17-3+deb10u1) ...
Selecting previously unselected package libk5crypto3:amd64.
Preparing to unpack .../03-libk5crypto3_1.17-3+deb10u1_amd64.deb ...
Unpacking libk5crypto3:amd64 (1.17-3+deb10u1) ...
Selecting previously unselected package libkrb5-3:amd64.
Preparing to unpack .../04-libkrb5-3_1.17-3+deb10u1_amd64.deb ...
Unpacking libkrb5-3:amd64 (1.17-3+deb10u1) ...
Selecting previously unselected package libgssapi-krb5-2:amd64.
Preparing to unpack .../05-libgssapi-krb5-2_1.17-3+deb10u1_amd64.deb ...
Unpacking libgssapi-krb5-2:amd64 (1.17-3+deb10u1) ...
Selecting previously unselected package libsasl2-modules-db:amd64.
Preparing to unpack .../06-libsasl2-modules-db_2.1.27+dfsg-1+deb10u1_amd64.deb ...
Unpacking libsasl2-modules-db:amd64 (2.1.27+dfsg-1+deb10u1) ...
Selecting previously unselected package libsasl2-2:amd64.
Preparing to unpack .../07-libsasl2-2_2.1.27+dfsg-1+deb10u1_amd64.deb ...
Unpacking libsasl2-2:amd64 (2.1.27+dfsg-1+deb10u1) ...
Selecting previously unselected package libldap-common.
Preparing to unpack .../08-libldap-common_2.4.47+dfsg-3+deb10u4_all.deb ...
Unpacking libldap-common (2.4.47+dfsg-3+deb10u4) ...
Selecting previously unselected package libldap-2.4-2:amd64.
Preparing to unpack .../09-libldap-2.4-2_2.4.47+dfsg-3+deb10u4_amd64.deb ...
Unpacking libldap-2.4-2:amd64 (2.4.47+dfsg-3+deb10u4) ...
Selecting previously unselected package libnghttp2-14:amd64.
Preparing to unpack .../10-libnghttp2-14_1.36.0-2+deb10u1_amd64.deb ...
Unpacking libnghttp2-14:amd64 (1.36.0-2+deb10u1) ...
Selecting previously unselected package libpsl5:amd64.
Preparing to unpack .../11-libpsl5_0.20.2-2_amd64.deb ...
Unpacking libpsl5:amd64 (0.20.2-2) ...
Selecting previously unselected package librtmp1:amd64.
Preparing to unpack .../12-librtmp1_2.4+20151223.gitfa8646d.1-2_amd64.deb ...
Unpacking librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2) ...
Selecting previously unselected package libssh2-1:amd64.
Preparing to unpack .../13-libssh2-1_1.8.0-2.1_amd64.deb ...
Unpacking libssh2-1:amd64 (1.8.0-2.1) ...
Selecting previously unselected package libcurl4:amd64.
Preparing to unpack .../14-libcurl4_7.64.0-4+deb10u1_amd64.deb ...
Unpacking libcurl4:amd64 (7.64.0-4+deb10u1) ...
Selecting previously unselected package curl.
Preparing to unpack .../15-curl_7.64.0-4+deb10u1_amd64.deb ...
Unpacking curl (7.64.0-4+deb10u1) ...
Selecting previously unselected package libsasl2-modules:amd64.
Preparing to unpack .../16-libsasl2-modules_2.1.27+dfsg-1+deb10u1_amd64.deb ...
Unpacking libsasl2-modules:amd64 (2.1.27+dfsg-1+deb10u1) ...
Selecting previously unselected package publicsuffix.
Preparing to unpack .../17-publicsuffix_20190415.1030-1_all.deb ...
Unpacking publicsuffix (20190415.1030-1) ...
Setting up libkeyutils1:amd64 (1.6-6) ...
Setting up libpsl5:amd64 (0.20.2-2) ...
Setting up libsasl2-modules:amd64 (2.1.27+dfsg-1+deb10u1) ...
Setting up libnghttp2-14:amd64 (1.36.0-2+deb10u1) ...
Setting up krb5-locales (1.17-3+deb10u1) ...
Setting up libldap-common (2.4.47+dfsg-3+deb10u4) ...
Setting up libkrb5support0:amd64 (1.17-3+deb10u1) ...
Setting up libsasl2-modules-db:amd64 (2.1.27+dfsg-1+deb10u1) ...
Setting up librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2) ...
Setting up libk5crypto3:amd64 (1.17-3+deb10u1) ...
Setting up libsasl2-2:amd64 (2.1.27+dfsg-1+deb10u1) ...
Setting up libssh2-1:amd64 (1.8.0-2.1) ...
Setting up libkrb5-3:amd64 (1.17-3+deb10u1) ...
Setting up publicsuffix (20190415.1030-1) ...
Setting up libldap-2.4-2:amd64 (2.4.47+dfsg-3+deb10u4) ...
Setting up libgssapi-krb5-2:amd64 (1.17-3+deb10u1) ...
Setting up libcurl4:amd64 (7.64.0-4+deb10u1) ...
Setting up curl (7.64.0-4+deb10u1) ...
Processing triggers for libc-bin (2.28-10) ...
$ echo $CI_JOB_ID >> ip.json && curl ipinfo.io >> ip.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   312  100   312    0     0   1362      0 --:--:-- --:--:-- --:--:--  1362
Uploading artifacts for successful job
00:00
Uploading artifacts...
ip.json: found 1 matching files and directories    
Uploading artifacts as "archive" to coordinator... ok  id=14 responseStatus=201 Created token=AzTM9Cfs
Job succeeded

deploy

Running with gitlab-runner 13.7.0 (943fc252)
  on centos7 FSX9yRiU
Preparing the "docker" executor
00:04
Using Docker executor with image python:3.9.1-slim-buster ...
Pulling docker image python:3.9.1-slim-buster ...
Using docker image sha256:b55839ea7a0e9bb534237d00558cb96dce4013bf7f1092966fe0e27e98f8179f for python:3.9.1-slim-buster with digest python@sha256:4d92968b26bb6b7b62d957244de86fc1054f03793577d49e85c00864eb03ca07 ...
Preparing environment
00:00
Running on runner-fsx9yriu-project-2-concurrent-0 via 926cd5798468...
Getting source from Git repository
00:01
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/feiyang/test/.git/
Checking out b60b95c1 as master...
Removing ip.json
Skipping Git submodules setup
Downloading artifacts
00:00
Downloading artifacts for build_ami (14)...
Downloading artifacts from coordinator... ok        id=14 responseStatus=200 OK token=AzTM9Cfs
Executing "step_script" stage of the job script
00:00
$ pwd
/builds/feiyang/test
$ cat ip.json
14
{
  "ip": "x.x.x.x",
  "hostname": "singnet.com.sg",
  "city": "Singapore",
  "region": "Singapore",
  "country": "SG",
  "loc": "1,103",
  "org": "AS9506 Singtel Fibre Broadband",
  "postal": "048508",
  "timezone": "Asia/Singapore",
  "readme": "https://ipinfo.io/missingauth"
}
Job succeeded

总结 artifacts，前一个 stage 生存的文件，以 artifacts 保存下来，给下一个 stage 使用

gitlab 自定义 SSH 端口 https://blog.csdn.net/skykingf/article/details/95212146


report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK