Creating a CI/CD Pipeline

I think it was Ralph Waldo Emerson who said, "It's not the destination, it's the journey". As over-quoted and common this mantra might be, I think Ralph was onto something. It is easy to overlook the steps required build, validate and test production level code, but what if it was possible to design a system that did all that and more.

build article:
  script:
    - creating a ci/cd pipeline

This project was built to get exposure to the full Software Development Lifecycle, but specifically the intricacies of CI/CD. To do so, I created a pipeline in GitLab CI that automated the building, testing and deployment of a web application to earn a better understanding of DevOps and the steps in creating production-level, shippable code.

Development Tools

Built with:

  • GitLab CI - For code repository and pipeline creation
  • Docker - For application containerization
  • Linux - For command line functionality and working within yaml file
  • AWS Elastic Beanstalk - For deploying web application to the cloud
  • AWS S3 - For storing project files in the AWS ecosystem
  • AWS EC2 - To provision a server to run my application on
  • AWS IAM - To provide permissions to use AWS services

Building the Application and Performing Initial Tests

The majority of "code" written in this project, was within the .gitlab-ci.yaml file. From here, you can interact with your repository and external sources through a Linux CLI. This pipeline builds a simple React app.

In this first iteration, I seperated the building and testing of the app into three jobs. The first job installs the application dependencies using the Node docker image, then it saves the build folder as an artifact. The test index job ensures that our homepage, index.html, is indeed included in the build folder. The next job, unit test, runs the standard yarn test.

stages:
  - build
  - test

build website:
  image: node:20-alpine
  stage: build
  script:
    - yarn install
    - yarn build
    - ls build
  artifacts:
    paths:
      - build

test index:
  image: node:20-alpine
  stage: test
  script:
    - test -f build/index.html

unit test:
  image: node:20-alpine
  stage: test
  script:
    - yarn test

Serving the Application

Clearly, we do not want a pipeline that just validates if the files exist. We want the application to actually run! To do so, I created a new job, test website, that serves the build folder and runs the application on a local development environment. From there, it requests the local URL to ensure that the text "React App" does indeed exist.

test website:
  image: node:20-alpine
  stage: test
  script:
    - yarn global add serve
    - apk add curl
    - serve -s build &
    - sleep 10
    - curl http://localhost:3000 | grep "React App"

Deploying to AWS S3

The issue with what we currently have, is that the development server only exists for as long as the job is running. A real pipeline should deploy to a customer-facing cloud platform. To do this, I configured an AWS S3 bucket for static website hosting. From there, I used the AWS CLI Docker image to deploy our changes to our new website after they are successfully built and tested.

deploy to s3:
  stage: deploy
  image:
    name: amazon/aws-cli:2.15.18
    entrypoint: ['']
  rules:
    - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
  script:
    - aws --version
    - echo $CI_COMMIT_REF_NAME
    - aws s3 sync build s3://$AWS_S3_BUCKET --delete

Creating a Staging Environment

In real production codebases, there are often times staging environments. Staging environments serve as a means to validate changes before they are pushed to production. In this iteration, I created two seperate S3 buckets, one for the staging environment one for production. Additionally, I configured my environment variables for each specific environment, so that the $AWS_S3_BUCKET variable corresponds to the appropriate environment.

deploy to staging:
  stage: deploy staging
  environment: staging
  image:
    name: amazon/aws-cli:2.15.18
    entrypoint: [""]
    - echo $CI_COMMIT_REF_NAME
    - echo $AWS_S3_BUCKET_STAGING
    - echo s3://$AWS_S3_BUCKET_STAGING
    - aws s3 sync build s3://$AWS_S3_BUCKET --delete
    - curl $CI_ENVIRONMENT_URL | grep "React App"

deploy to production:
  stage: deploy production
  environment: production
  image:
    name: amazon/aws-cli:2.15.18
    entrypoint: [""]
    - aws --version
    - echo $CI_COMMIT_REF_NAME
    - aws s3 sync build s3://$AWS_S3_BUCKET --delete
    - curl $CI_ENVIRONMENT_URL | grep "React App"

Refactoring Code and Continuous Deployment vs. Continuous Delivery

If you notice in our last iteration, deploy to staging and deploy to production had identical functionality, just different environments. As the fantastic software engineers that we are it would be much better to optimize this code by creating a .deploy job. This .deploy job encapsulates the functionality of our deployment, but is reusable in either our staging or production environment due to the configuration of our environment variables, specifically $AWS_S3_BUCKET.

Additionally, the CD in CI/CD can stand for Continuous Deployment or Continuous Delivery. In Continuous Deployment, the changes made in the repository are automatically sent to production. We would rather Continuous Delivery, where our changes are automatically deployed to the staging environment, but require some manual intervention to get sent to production. In GitLab CI, we do this by defining 'when: manual'. From there, we have to manually run this job in our pipeline to deploy our app to production.

.deploy:
  image:
    name: amazon/aws-cli:2.15.18
    entrypoint: ['']
  rules:
    - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
  script:
    - aws --version
    - aws s3 sync build s3://$AWS_S3_BUCKET --delete
    - curl $CI_ENVIRONMENT_URL | grep "React App"
    - curl $CI_ENVIRONMENT_URL/version.html | grep $APP_VERSION

deploy to staging:
  stage: deploy staging
  environment: staging
  extends: .deploy

deploy to production:
  stage: deploy production
  when: manual
  environment: production
  extends: .deploy

Containerization and Deploying to AWS Elastic Beanstalk

Most modern applications need a little more functionality than simple static web hosting on an S3 bucket. AWS Elastic Beanstalk offers web application hosting using AWS EC2 to host a live server and AWS S3 to store our files like before. To give GitLab the access it needs, we create a new user role in AWS IAM in order to configure our new environment on AWS, this token is seen in the code below as $GITLAB_DEPLOY_TOKEN.

To deploy to Elastic Beanstalk, we need to containerize our application. To do so, we need to create a Container Registry in GitLab CI, so that our container is saved outside of our job. From there, we can deploy our container to our Elastic Beanstalk app and see our application!

build docker image:
  stage: package
  image: docker:25.0.3
  services:
    - docker:25.0.3-dind
  script:
    - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER $CI_REGISTRY --password-stdin
    - docker build -t $CI_REGISTRY_IMAGE -t $CI_REGISTRY_IMAGE:$APP_VERSION .
    - docker image ls
    - docker push --all-tags $CI_REGISTRY_IMAGE

test docker image:
  stage: test
  image: curlimages/curl
  services:
    - name: $CI_REGISTRY_IMAGE:$APP_VERSION
      alias: website
  script:
    - curl http://website/version.html | grep $APP_VERSION

deploy to production:
  image:
    name: amazon/aws-cli:2.15.18
    entrypoint: ['']
  stage: deploy production
  variables:
    APP_NAME: web cicd
    APP_ENV_NAME: Webcicd-env
  environment: production
  script:
    - aws --version
    - yum install -y gettext
    - export DEPLOY_TOKEN=$(echo $GITLAB_DEPLOY_TOKEN | tr -d "\n" | base64)
    - envsubst < templates/Dockerrun.aws.json > Dockerrun.aws.json
    - envsubst < templates/auth.json > auth.json
    - aws s3 cp Dockerrun.aws.json s3://$AWS_S3_BUCKET/Dockerrun.aws.json
    - aws s3 cp auth.json s3://$AWS_S3_BUCKET/auth.json
    - aws elasticbeanstalk create-application-version --application-name "$APP_NAME" --version-label $APP_VERSION --source-bundle S3Bucket=$AWS_S3_BUCKET,S3Key=Dockerrun.aws.json
    - aws elasticbeanstalk update-environment --application-name "$APP_NAME" --version-label $APP_VERSION --environment-name $APP_ENV_NAME
    - aws elasticbeanstalk wait environment-updated --application-name "$APP_NAME" --version-label $APP_VERSION --environment-name $APP_ENV_NAME
    - curl $CI_ENVIRONMENT_URL/version.html | grep $APP_VERSION

Key Takeaways

To summarize, this pipeline automates the process of building, testing, containerizing, and deploying a web application to AWS. This project has given me meaninful experience in the realm of DevOps, and prepares me well for contributing to production-level codebases. Here is a screenshot of our final product: