Setup a CI/CD pipeline with Gitlab for ASPNET Core on Ubuntu

Setup a CI/CD pipeline with Gitlab for ASPNET Core on Ubuntu

Few weeks ago I explained how we could setup a CI/CD pipeline whereby the runner would be on Windows and the last stage was to package the application.
Today we will see how we can setup a runner on Ubuntu CI server and use it to build and deploy an ASP MET Core application onto a Ubuntu 16.04 server. This post will be composed by three parts:

  1. Setup the runner on the CI server
  2. Setup the application on the server
  3. Setup the job in our project

If you are unfamiliar with Gitlab pipeline and its terminology, you can read my previous post where I explain the main concepts behind GitLab pipeline with runner, jobs and stages.
If you are unfamiliar with ssh and systemd, you can read my previous blog post on useful ssh commands and my previous blog post on how to manage Kestrel process with systemd.

1. Setup the runner on the CI server

Setup the runner on your CI server by getting the package with apt-get.

curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
sudo apt-get install gitlab-runner

Next register the runner using the token from your project.

While registering the runner, the tags are used for Gitlab to know which runner should get the job hence it is good to set tags tied to the project, the environment, the os, frameworks and even package manager available.

sudo gitlab-runner register

Once the runner is setup we should be able to see it under the runner configuration.

Next we need to install dotnet for the CI server to be able to build the application.

wget -q https://packages.microsoft.com/config/ubuntu/16.04/packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
sudo apt-get install apt-transport-https
sudo apt-get update
sudo apt-get install dotnet-sdk-2.1

Lastly install zip as we will be using it to package all files.

apt-get update
apt-get install zip

2. Setup the application on the server

Place the application in the right folders on your server: /usr/share/myapp for the runnable binaries.

Install nginx to proxy port 80 to the dotnet Kestrel process.

server {
    listen 80;
    listen [::]:80;
    include /etc/nginx/conf.d/http;
    include /etc/nginx/proxy_params;

    location / {
        proxy_pass http://localhost:5000/;
    }
}

Here’s the /etc/nginx/conf.d/http content:

proxy_http_version 1.1;
proxy_set_header Connection keep-alive;
proxy_set_header Upgrade $http_upgrade;
proxy_cache_bypass $http_upgrade;

Here’s the /etc/nginx/proxy_params content:

proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

Setup systemd unit to boot the dotnet process and manage it as a service.

[Unit]
Description=

[Service]
WorkingDirectory=/usr/share/myapp
ExecStart=/usr/bin/dotnet /usr/share/myapp/MyApp.dll
SyslogIdentifier=myapp
User=www-data

[Install]
WantedBy=multi-user.target

From here we should be able to access our server on internet.

If you are unfamiliar with nginx, read my previous blog post on how to setup Kestrel behind nginx.
If you are unfamiliar with systemd, read my previous blog on how to manage Kestrel procesz with systemd.

3. Setup the job in our project

We have a runner setup to run jobs and we have our application already running. What we need to do next is to define the jobs to run to update the running application with the latest build once we push new code to the repository. For that we need to create a job file which we put in the root of the application.

The job file defines all the jobs which can be run by runners registered for the repository. The first section of the yaml defines the stages where jobs run. At each stage, job run concurrently when multiple runners are registered. The order of execution respects the order we define in the yaml.

In the following example, we will define three stages build, deploy and clean.

stages:
  - build
  - deploy
  - clean

Next we can define the job themselves. The complete documentation of a job is on Gitlab documentation.

We start by specifying the build which will be done in build stage.

build:
  stage: build
  script:
    - /usr/bin/dotnet publish -c Release
  only:
    - master
  variables:
    GIT_STRATEGY: fetch
  tags:
    - myapp

script allows us to specify an array of shell commands to run synchronously.
only defines for which branch the job should be triggered.
tags defines which runner will be targeted to run the job.
variables defines an object composed of variables available during the job. The variables can be custom for our own use and can be variables used to setup settings on the job itself. Here we set the GIT_STRATEGY to fetch which order the job to fetch the repository. Other settings are available and can be found in the documentation.

After we built, we can deploy the application. Following the same job properties as the build, we set a script command to run a shell script present in our source code.

deploy:
  stage: deploy
  script:
    - chmod 774 $CI_PROJECT_DIR/deploy.sh
    - SERVICES=( Service1 Service2 )
    - for i in "${SERVICES[@]}"; do $CI_PROJECT_DIR/deploy.sh $i; done
  variables:
    GIT_STRATEGY: none
  only:
    - master
  tags:
    - myapp

$CI_PROJECT_DIR is set to the path to the source code fetched by the runner. The first step is to allow execution of the deployment script with chmod 774 $CI_PROJECT_DIR/deploy.sh.
Next we simply run it by specifying all the projects to deploy using an array and a for in loop.

- SERVICES=( Service1 Service2 )
    - for i in "${SERVICES[@]}"; do $CI_PROJECT_DIR/deploy.sh $i; done

We can also see that we have set the GIT_STRATEGY to none which prevents the runner from fetching the solution again.

The deployment script is as followed:

#!/bin/bash -v

set -e

if [ -z $1 ]; then
    echo "Argument cannot be empty."
    exit
fi

APP_NAME=$(echo $1 | awk '{print tolower($0)}') 
APP_DIR=/usr/share/myapp/$APP_NAME
ZIP=myapp-$APP_NAME.zip

# create temp folder for preparing zip
mkdir -p ~/myapp/$APP_NAME

# move published output from build stage to folder
mv $CI_PROJECT_DIR/MyApp.$1/bin/Release/netcoreapp2.0/publish/* ~/myapp/$APP_NAME

# navigate to folder to set root for zip
cd ~/myapp

# zip folder
zip -r ~/$ZIP $APP_NAME

# copy zip to server
scp -qr ~/$ZIP myserver:~/

# ssh to server and unzip within server to temp folder
ssh myserver "unzip -o $ZIP -d ~/myapp"

# ssh to server and remove app folder and content
ssh myserver "sudo rm -rf $APP_DIR/*"

# ssh to server and copy binaries from temp folder to app folder
ssh myserver"sudo cp -r ~/ek/$APP_NAME/* $APP_DIR"

# ssh to server and set user and group to user used by nginx and systemd
ssh myserver "sudo chown -R www-data:www-data $APP_DIR/*"

# ssh to server and restart systemd unit
ssh myserver "sudo systemctl restart myapp-$APP_NAME"

Last stage is to clean the temporary folder created during zip and unzip.

clean:
  stage: clean
  script:
    - rm -r ~/ek*
    - ssh myserver "rm -r ek*"
  variables:
    GIT_STRATEGY: none
  only:
    - master
  when: always
  tags:
    - myapp

when: always is a variable used to define when is the job run where always means that thd job will run regardless the state of the previous stage hence if deploy succeeds or fails, clean will run.

Here is the full yaml job file:

stages:
  - build
  - deploy
  - clean

build:
  stage: build
  script:
    - /usr/bin/dotnet publish -c Release
  only:
    - master
  variables:
    GIT_STRATEGY: fetch
  tags:
    - myapp

deploy:
  stage: deploy
  script:
    - chmod 774 $CI_PROJECT_DIR/deploy.sh
    - SERVICES=( Service1 Service2 )
    - for i in "${SERVICES[@]}"; do $CI_PROJECT_DIR/deploy.sh $i; done
  variables:
    GIT_STRATEGY: none
  only:
    - master
  tags:
    - myapp

clean:
  stage: clean
  script:
    - rm -r ~/ek*
    - ssh myserver "rm -r ek*"
  variables:
    GIT_STRATEGY: none
  only:
    - master
  when: always
  tags:
    - myapp

Once we push our code, the pipeline should run and build/deploy our application!

Conclusion

Today we saw how to configure a complete CI/CD chain for automating build and deployment of an ASP NET Core application on an Ubuntu 16.04 server. We started by setting up our CI server then saw how to configure our application to run and lastly we saw how to setup the Gitlab pipeline to automate the whole build and deployment. Hope you like this post, see you next time!

Comments

Post a Comment

Popular posts from this blog

Microsoft Orleans logs warnings and errors

Microsoft Orleans logs warnings and errors Microsoft Orleans is a framework which helps building distributed system by implementing the actor model together with the concept of virtual actors, taking care of availability and concurrency. If you are unfamiliar with Microsoft Orleans, you can look at my previous blog post explaining the benefits of Microsoft Orleans . Even though Orleans promises to abstract the distributed system problems, there are instances where errors arise without us being able to understand what is going on. Lucky us, the logs are well documented… but only for those who can decrypt them. Today I will go through some of the errors and warnings which can be seen from silo and client so that you too can undestand what is going on. Enjoy! The code used to produce those errors can be found on my GitHub https://github.com/Kimserey/orleans-cluster-consul . 1. Client logs Logs on client appears with address {ip}:0 . 1.1. Can’t find implementation of interface An un...
Keep reading

A complete SignalR with ASP Net Core example with WSS, Authentication, Nginx

SignalR with ASP Net Core SignalR is a framework from ASP NET Core allowing us to establish a two way communication between client and server. This two way communication allows the client to send messages to the server but more importantly allows the server to push messages to the client. SignalR makes use of Websocket when available else it falls back to SSE or pulling. Today we will focus on how to setup SignalR to work with WSS, Websocket secure and how we can authenticate the user requesting to connect to our SignalR hub via Webscoket. Getting started with SignalR SSL encryption for Websocket Secure WSS Websocket Authentication with Identity Server 4 SignalR behind Nginx 1. Getting started with SignalR The Hubs are the main components of SignalR. It is an abstraction of a two way communication available for both client and server. Public functions from the hub can be called from the server code and can be called from the client. The frontend NPM package @aspnet/signalr ...
Keep reading

SDK-Style project and project.assets.json

SDK-Style project and project.assets.json Last week I encountered an issue with MSBuild while trying to run it from command line. The issue did not appear when using VisualStudio right click + build but only appeared when using msbuild.exe CLI directly with a clean project. Assets file 'C:\[...]\obj\project.assets.json' not found. Run a NuGet package restore to generate this file. When I first saw the error, few questions came to my mind which I will share today in 3 points: Overview of project.assets.json Slim SDK-Style project Mixing SDK-Style project and old projects Special shoutout to @enricosada who provided me with all the answers regarding the SDK-Style project. 1. Overview of project.assets.json project.assets.json lists all the dependencies of the project. It is created in the /obj folder when using dotnet restore or dotnet build as it implicitly calls restore before build, or msbuid.exe /t:restore with msbuild CLI. To simulate dotnet build (re...
Keep reading