CI/CD Pipeline for Snowflake Using Jenkins and Schemachange
source link: https://selectfrom.dev/ci-cd-pipeline-for-snowflake-using-jenkins-and-schemachange-d806eaac3e07
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
CI/CD Pipeline for Snowflake Using Jenkins and Schemachange
Introduction
The pipeline is scheduled in Dockerized Jenkins that fetches SQL files committed into the AWS Code commit repository. Once the SQL file is committed successfully, we can run the job through Jenkins to execute changes. Schemachange will help Jenkins deploy changes mentioned in the loaded SQL file into the provided Snowflake account.
Execution Flow Diagram
Deployment Steps:
- Create the pipeline.
- Add the AWS CodeCommit repository with credentials.
- Provide a jenkinsfile in the pipeline.
- Apply the changes.
- Execute a job from Jenkins once after successfully committing the SQL files.
Execution Steps
- Create a new instance in AWS EC2.
- Log in to the instance using Putty and install Docker and Git.
- Create a repository in AWS code commit.
- Go to IAM and generate credentials for AWS CodeCommit.
- Clone the repository.
- Run the dockerfile to build an image.
- Run the image using the below command:
docker run -p 8080:8080 -v /var/run/docker.sock:/var/run/docker.sock — name jenkins jenkins
- Copy the password to login into Jenkins.
- Note down the password located in — /var/jenkins_home/secrets/initialAdminPassword
- Paste the copied password here.
- Click on install suggested plugins.
- Jenkins will automatically restart after this step.
- In this step, we are installing necessary plugins to make an environment to setup pipelines
- Create a Jenkins user here.
- This step is to create a Jenkins root user.
- Once the Jenkins container is running, run this command to give Jenkins access to the Docker engine.
- Provide your Snowflake credentials here into parameters.
- To connect Snowflake with the pipeline we are creating in Jenkins, we need to provide Snowflake credentials in parameter.
- SF_ACCOUNT
- SF_USERNAME
- SNOWFLAKE_PASSWORD
- SF_ROLE
- SF_WAREHOUSE
- SF_DATABASE → RAW_DATALAKE
- Provide the code commit URL here & click on “Save”:
- Create a new file in the migration directory.
- It will be a .sql file with all the SQL script we need to execute with the commit.
- Push the changes to code commit.
- Click on build with parameters in Jenkins and click on build. The job will start to run.
- We can see here that in the fourth job we have one commit.
- After completing job execution we can see results in Snowflake.
- Result: Database will be created inside Snowflake.
Testing Scenario:
- To execute CI/CD by committing an SQL file into the AWS Code-Commit remote repository from a local system.
Expected result : After committing and pushing the SQL file into a remote repository, the SQL scripts should be executed automatically into a Snowflake account of which detailed credentials have been provided when the pipline was set up Jenkins.
Prerequisites :
- AWS Code-Commit
- Jenkins Setup Localhost/EC2
- Snowflake Account
Steps to execute:
- Take a pull from the remote repository to ensure we updated changes that had been committed into the remote repository and opened the updated folder in the IDE.
- Create a .sql file in IDE with all the appropriate SQL script and save it.
- Execute “Git add .” to add the file to commit from the local directory.
- Execute “Git commit -m “Any message with commit”” to commit changes from the local directory to the remote repository.
- Execute the “Git push origin” command to push all the changes into the remote repository.
- Go to the Jenkins dashboard and execute the job we created in it.
- Wait to complete process execution inside Jenkins.
- After the completion of the process, confirm the changes in your Snowflake Account through the History option where we can see the last executed SQL scripts.
And that’s all folks! Hope you found this helpful. PLease let me know if you have any queries/feedback in the comments section below.
Recommend
-
64
一、背景 在日常开发中，我们经常会有发布需求，而且还会遇到各种环境，比如：线上环境（Online），模拟环境（Staging），开发环境（Dev）等。最简单的就是手动构建、上传服务器，但这种方式太过于繁琐，使用持续集成可以完美...
-
72
环境测试部署主机IP：192.168.1.1Jenkins主机IP：192.168.1.2Harbor主机IP：192.168.1.3Gitlab主机IP：192.168.0.10系统信息：系统：CentOS7.5内核：4.18.7-1.el7.elrepo.x86_64Docker版本：18.09docker-compose版本：1.23.1所有主机的Docker安装方式wget-O/etc/y
-
51
Hi, I'm Guy "RiotSomeOtherGuy" Kisel, a software engineer at Riot. You might remember me from Running an Automated Tes...
-
57
在Jenkinspipeline的groovy脚本中可以实现很多复杂灵活的功能，但是：1一来对java、groovy不是很熟，也不知道能不能引入一些三方库？比如搞个jdbc操作下mysql什么的。2二是自己对go和python比较熟悉，所以想能够更加灵活的实现更多的功能：比如：操作数据库，无需...
-
43
简介Pipeline，简而言之，就是一套运行于Jenkins上的工作流框架，将原本独立运行于单个或者多个节点的任务连接起来，实现单个任务难以完成的复杂流程编排与可视化。Pipeline是Jenkins2.X的最核心的特性，帮助Jenkins实现从CI到CD与DevOps的转变。一，创建pipeline...
-
26
Jenkins X不是Jenkins，它是完全从头开始重写的。 Jenkins X比Jenkins更聚焦于特定领域。它提供了一种使用特定工具（Kubernetes Helm Tekton Skaffold Flagger…）来构建和部署应用程序的方式。如果您喜欢这样使用它，那将是一种...
-
20
Pipeline翻译过来就是流水线的意思，在公司中可能会有很多项目，如果使用Jenkins构建完成后，开发构建项目需要一项一项点击，比较麻烦，因此，pipeline就应用而生了。一、部署Pipeline——代码流水线管理Jenkins部署及其他服务部署可参考博文：持续集成之代码质量管...
-
35
michaelliuyang0.5862020.04.19 21:22:52字数 417阅读 449什么是扩展共享库 在工作中，我们可能会有很多Maven项目的打包部署，还有An...
-
9
Reading Time: 5 minutes In this blog we gone a see creating a CI/CD Pipeline using Jenkins. Some pre-requisites to create a CI/CD pipeline using Jenkins are. Open-JDK-8Jenkins installed on our system...
-
7
In this blog, it is explained how to configure Jenkins Multibranch Pipelines when using Git LFS. This seems to be a non-issue at first sight, but the documentation is quite dispersed and no clear steps can be found how to configure this. Also...
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK