Hotfix procedure

This section describes the second use case to correct a critical bug that has occurred in the production environment.

General overview

First we provide an overview of the development workflow when a critical bug occurs in the prod environment.

Step 1 - software development

  • The developer (D) reads the bug report or writes it in a new GitLab issue using the appropriate template.

  • The developer (D) creates a local branch named hotfix-id_commit-user on the local workspace from the hotfix branch. The id_commit is the commit number of the version that has been deployed in the prod environment that needs to be fixed, and user is the unix login of the developer (e.g. hotfix-aa12bb34-phupe).

  • The developer develops the corrections.

  • The developer (D) temporary pushes the code from the local branch hotfix-id_commit-user on the remote repository.

  • The developer (D) deploys the code from hotfix-id_commit-user branch either in a personal environment for testing or in the dev environment to perform unit, integration, system and regression testing.

  • The developer (D) checks that the bug has been fixed.

  • If the testing is successful, it is possible to move to Step 2 - acceptance testing.

Step 2 - acceptance testing

  • The user (D+UD) creates a new GitLab issue with the label label_validation using the appropriate template. This GitLab issue allows the tracking of all the discussions with the end-users who will validate the new release.

  • The end-user can start the acceptance testing of the corrected release:

    • either the end-user validates the corrections,

    • or the end-user does not validate the corrections. Thus, the developer (D) goes back to Step 1 - software development to implement the corrections on the hotfix-id_commit-user branch until acceptance by the end-user.

  • The developer (D) has to Create a Merge Request in Gitlab from the hotfix-id_commit-user branch on the hotfix branch. The Merge request is assigned to as user who has the Maintainer role.

  • The user (M+UVP) reviews and accepts the Merge Request.

  • The user (M+UVP) tracks the issues related to the new version in a Milestone

Step 3 - check the installation process and new testing

  • The user (M+UVP) creates a GitLab issue with the label label_mep using the appropriate template in order to track the different steps of the installation process. The ID of the GitLab issue regarding the validation of the release is also reported for tracking.

  • The user (M+UVP) deploys the code from the hotfix release branch in the valid environment.

  • The user (M+UVP) checks that the installation is successful, launches a set of testing, in particular the operational testing.

  • Once the validation is successful, the user (M+UVP) adds a tag on hotfix branch with the new version number.

Step 4 - production deployment

  • Once validated by the end-user, the user (M+UVP) deploys the code in the prod environment from the hotfix branch.

  • The user (M+UVP) launches the operational testing dedicated to the prod environment (e.g. foobar).

  • Once the deployment is successful, the user (M+UVP) merges the code from the hotfix branch on the main branch for archiving.

  • The user (M+UVP) brings the content of the hotfix branch into the release branch such that they can be integrated in the future release.

  • The user (M+UVP) brings the content of the hotfix branch into the devel branch such that the corrections can be integrated in the future release. Note that possible conflicts may exist on some pieces of code. They will have to be resolved before merging thus requiring the help from the other developers involved in the modifications.

  • If needed, the user (M+UVP) deploys the code from hotfix branch in the dev environment, such that at this stage of the workflow, the same commit ID of the bioinformatics pipeline is deployed in the dev, valid and prod environment.

  • The user (M+UVP) closes the GitLab issue label_validation and the GitLab issue label_mep that were previously opened.

  • The user (M+UVP) creates a new Release and closes the Milestone in GitLab.

Technical procedure

Step 1 - software development

New issue

  • The developer (D) reads the bug report or writes it in a new GitLab issue using the template bug_report.

Code writing

  • The developer (D) sets the local workspace on the hotfix branch, gets the last modifications from the remote repository and checks that the right branch is used:

git checkout hotfix
git status
git pull
git submodule update --init --recursive # if your git repo has submodules

  • The developer (D) gets the commit ID that has been deployed in the prod environment. The commit ID is retrieved from the version file that is available in the folder in which the pipeline has been installed.

Important

It is essential that the commit ID that has been used to deploy the code is tracked in the install folder. For this purpose, the commit ID is stored in the version file.

  • The developer (D) checks that the commit ID stored in the version file is the same as the commit ID of the hotfix branch from the local repository:

git branch -vv

  • The developer (D) creates a local branch named hotfix-id_commit-user. The id_commit is the commit number of the version that has been deployed in the prod environment that needs to be fixed, and user is the unix login of the developer (e.g. hotfix-aa12bb34-phupe).

git checkout -b hotfix-id_commit-user

  • The developer (D) implements the corrections, tests the modifications and commits them (see Code writing for the details about the command lines).

  • The developer (D) temporarily pushes the new local branch on the remote repository:

git push origin hotfix-id_commit-user

  • The developer (D) deploys the code from the hotfix-id_commit-user branch either in a personal environment for testing or in the dev environment to perform unit, integration, system and regression testing.

  • The developer (D) checks that the bug has been corrected.

Deployment in the dev environment

The user (D+UD) deploys the bioinformatics pipeline in the dev environment from the hotfix-id_commit-user branch using GitLab CI/CD (or ad-hoc deployment scripts using the commit ID to deploy).

Danger

Tags with version number are not used to deploy the code but only a commit ID. Indeed, a tag can be easily removed or moved in git thus it is not a reliable information for tracking. This is why commit ID are used for deployment.

Testing

  • The developer (D) checks that the bug has been fixed.

Step 2 - acceptance testing

Acceptance testing by end-users

  • The user (D+UD) creates an GitLab issue using the template validation.

  • The title of the GitLab issue must indicate the characteristics of the version to be validated.

  • The description in the GitLab issue lists the new features/modifications that have to be communicated to the end-users.

  • At the end of the description, a line such as fyi: @user1, @user2, @user3 is added such that all the persons involved in the validation process receive a notification.

  • The GitLab issue is labeled with label_validation.

  • The GitLab issue is assigned to a user (M+UVP).

  • The user (D+UD) or the user (M+UVP) sends an email to all the persons who are involved in the validation process.

  • The end-users can start the acceptance testing process:

    • either the end-users validate the new release,

    • or the end-users do not validate the new release. Then, the reason are tracked in the GitLab issue label_validation that has been created. We go back to Step 1 - software development. The developer (D) develops the modifications requested by the end-users on a local hotfix-id_commit-user branch. The process is iterated until the validation by the end-users. The same GitLab issue is used to track all the information during the validation process until the final validation.

  • Once validated by the end-user, the developer (D):

    • creates a Create a Merge Request in Gitlab from the hotfix-id_commit-user branch on hotfix branch,

    • assigns the Merge request to a user with the Maintainer role.

  • The user (M+UVP) reviews and accepts the Merge Request.

Update the CHANGELOG with the new version

Note

The CHANGELOG file provides a simple history of the different versions of the bioinformatics pipeline. The version numbers are listed by decreasing order.

  • A version number is added in the CHANGELOG using the following naming convention: version-x.y.z:

    • The z number is incremented for BUG FIXES of modifications which are not visible by the end-user

    • The x.y numbers are incremented for major modifications considered as SIGNIFICANT USER-VISIBLE CHANGES

  • Comments are added in the CHANGELOG to describe the most relevant functionalities added to the new release.

The CHANGELOG is divided into 3 sections:

  • NEW FEATURES

  • SIGNIFICANT USER-VISIBLE CHANGES

  • BUG FIXES

Example of CHANGELOG file:

version-1.2.1

    BUG FIXES
        [BUGFIX] segfault corrected

    NEW FEATURES
        [MODIF] optimization of the sort function
        [DOC] update of the end-user documentation

version-1.2.0

    NEW FEATURES
        [MODIF] use of GRCh38 reference genome

  • The user (M+UVP) updates the hotfix branch to get the last modifications from the remote repository and checks that the right branch is used:

git checkout hotfix
git status
git pull
git branch -vv

  • The user (M+UVP) updates the CHANGELOG. The version number is incremented using the following naming convention: version-x.y.z.

  • If needed, the user (M+UVP) asks the other developers to define what comments should be added in the CHANGELOG and pushes the modifications on the remote repository:

git add CHANGELOG
git commit -m "[DOC] information about the version-1.2.4 after correction of the bug added in the CHANGELOG"
git push origin hotfix

Track the issues related to the new version in a Milestone

As mentioned, a New issue is created whenever a new development is started. As new version encompasses several issues, it is important to track all the issues which have been considered in the new version. Therefore, the user (M+UD):

  • creates a new GitLab milestone with the same name as the new version number (e.g. version-x.y.z),

  • describes what is the purpose of the new GitLab milestone,

  • for each issue included in the new version, sets the name of the GitLab milestone in the dedicated field.

  • in the Merge request which has been created to manage the Hotfix, sets the name of the GitLab milestone in the dedicated field.

Note

As your developments may depend on other GitLab repositories you maintain, you can also create another GitLab milestone in each of them and cross-referenced the milestones in the different repositories. To do so, you can just add in the field Description of the Milestone the URL of the other Milestones.

Step 3 - check the installation process and new testing

Create an issue to track the production deployment

  • The user (M+UVP) creates an GitLab issue using the template deploy_in_prod_hotfix

    • The GitLab issue is labeled with label_mep.

    • The GitLab issue is linked to the name of the GitLab milestone using the dedicated field.

    • The GitLab issue number that has been used for the validation along with the GitLab issue number that describes the bug is added to the current GitLab issue.

    • The user (M+UVP) tracks all the steps that are performed for the deployment in the production environment (including link or name of datasets that are used).

    • The user (M+UVP) fills the GitLab issue at each step.

Deployment in the valid environment

The user (M+UVP) deploys the pipeline in the valid environment from the hotfix branch using GitLab CI/CD (or ad-hoc deployment scripts using the commit ID to deploy).

Danger

Tags with version number are not used to deploy the code but only a commit ID. Indeed, a tag can be easily removed or moved in git thus it is not a reliable information for tracking. This is why commit ID are used for deployment.

Launch the operational testing in GitLab CI/CD

  • The user (M+UD) checks that the deployment with GitLab CI/CD is available for the GitLab repository. It requires the file .gitlab-ci.yml as defined in the template pipeline.

  • In the .gitlab-ci.yml file, the operational testing is implemented through different jobs which launch the pipeline twice during the Deployment in the valid environment and compare the results to ensure they are identical.

  • If the operational testing fails (the bioinformatics pipeline does not work or is not reproducible), go back to the Step 1 - software development.

Test the version deployed in the valid environment

The user (M+UVP) tests the bioinformatics pipeline.

Development of corrections if needed

In most of the cases, the deployment in the valid environment is very simple and quick. However, it might be necessary to correct some bugs before the deployment in production. In that case:

  • The developer (D) checkouts and updates the hotfix branch from the local workspace:

git checkout hotfix
git status
git pull
git branch -vv

  • The developer (D) creates a local branch named hotfix-id_version-user (e.g. hotfix-version-1.2.3-phupe), the hotfix-version is retrieved from the CHANGELOG file) and uses it for the developments:

git checkout -b hotfix-id_version-user # change the id_version and user values

  • The developer (D) implements the corrections, tests the modifications and commits them (see Step 1 - software development for the details about the command lines).

  • The developer (D) temporarily pushes the local branch on the remote repository:

git push origin hotfix-id_version-user # change the id_version and user values

  • The developer (D) deploys the code from the hotfix-id_version-user branch either in a personal environment for testing or in the dev environment to perform unit, integration, system and regression testing.

  • Once the code validated, the developer (D) creates a Create a Merge Request in Gitlab from the hotfix-id_version-user branch on the hotfix branch using the template merge_request_template.md. The Merge request is assigned to a user with the Maintainer role.

  • The user (M+UVP) reviews and accepts the Merge Request.

  • In the Merge request, sets the name of the GitLab milestone in the dedicated field.

  • The user (M+UVP) updates the hotfix branch from the local workspace:

git checkout hotfix
git status
git pull
git branch -vv

Add a tag with the version number

Once the new release has been validated and the installation in the valid environment is successful, the user (M+UVP) adds a tag (using the same version number that has been written in the CHANGELOG file) on the current HEAD:

tag-version4prod.sh -t version-1.2.3
git push --tags

Note

The script tag-version4prod.sh checks that the tag name is consitent with what was mentioned in the CHANGELOG and add the tag.

Step 4 - production deployment

Update the local repository with the version to deploy

It is likely that the local repository is not up-to-date anymore especially if a Merge Request has been submitted on GitLab. The user (M+UVP) updates the local workspace:

git checkout hotfix
git status
git pull
git branch -vv

Deployment in the prod environment

The user (M+UVP) deploys the bioinformatics pipeline in the prod environment from the hotfix branch using GitLab CI/CD (or ad-hoc deployment scripts using the commit ID to deploy).

Danger

Tags with version number are not used to deploy the code but only a commit ID. Indeed, a tag can be easily removed or moved in git thus it is not a reliable information for tracking. This is why commit ID are used for deployment.

Create a new release

The user (M+UVP) closes the milestone (see Track the issues related to the new version in a Milestone) and issues related to the new version. Then, the user (M+UVP) creates a New release in GitLab:

  • Select the Tag name corresponding to the new release

  • Fill in the Release title with the version number followed by free comments containing the keywork hotfix

  • Select the Milestone corresponding to the new release

_images/hotfix-gitlab-new-release.png

Schedule the operational testing in GitLab CI/CD

  • The user (M+UD) checks that the deployment with GitLab CI/CD is available for the GitLab repository. It requires the file .gitlab-ci.yml as defined in the template pipeline.

  • In the .gitlab-ci.yml file, the operational testing is implemented through different jobs which launch the pipeline twice during the Deployment in the prod environment and compare the results to ensure they are identical.

  • The user (M+UD) connects to GitLab to Schedule an operational testing periodically in GitLab CI/CD if it is not yet scheduled.

Bring the content of the hotfix branch into the main branch

  • At this stage, there is a stable code on the hotfix branch that has been tested, validated and successfully installed in the prod environment.

  • The user (M+UVP) checkouts and updates the main branch:

git checkout main
git status # everything must be cleaned
git pull
git branch -vv

  • The user (M+UVP) brings the content of the hotfix into the main using the option --no-ff to avoid the fast-forward mode. This option will produce a new commit ID with a specific message to describe and track the merge:

git merge --no-ff hotfix # can be a bit verbose
git status # must be cleaned
git branch -vv

  • The git status must absolutely says something like this (otherwise, ask for help before moving forward):

# On branch main
# Your **branch is ahead of 'origin/main' by** 113 commits.
# (use "git push" to publish your local commits)
#
# nothing to commit, working directory clean
*# On branch main*

  • The user (M+UVP) pushes the modifications on the remote repository:

git push origin main

Bring the content of the hotfix branch into the release branch

  • At this stage, there is a stable code on the hotfix branch that has been tested, validated and successfully installed in the prod environment and merged with the main and the devel branches.

  • The user (M+UVP) checkouts and update the release branch:

git checkout release
git status # must be cleaned otherwise, commit or stash your modifications
git pull
git branch -vv

  • The user (M+UVP) brings the content of the hotfix branch into the release branch using the option --ff to use the fast-forward mode in order to have the exact same commit ID between the release branch and the hotfix branch. However this might not be always possible if the same pieces of code have been modified at the same time:

git merge --ff hotfix # may be verbose
git status # may say something
git branch -vv

  • If the release branch has been modified in the meantime, git will try to merge the modifications from the hotfix branch.

  • If some files cannot be merged automatically, they will appear to have conflicts in the output of the git status:

*# On branch release*
*#You have unmerged paths.*
*# (fix conflicts and run "git commit")...*
*# (use "git add ..." to mark resolution)*
*## both modified:build.xml*

  • The conflicts have to be resolved manually. In that case, ask the help from the other developers.

  • The files with resolved conflicts must be added to the staging area, committed, and the merge must be sent on the remote repository:

git push origin release

Bring the content of the hotfix branch into the devel branch

  • At this stage, there is a stable code on the hotfix branch that has been tested, validated and successfully installed in the prod environment and merged on the main branch.

  • The user (M+UVP) checkouts and updates the devel branch:

git checkout devel
git status # must be cleaned otherwise, commit or stash your modifications
git pull
git branch -vv

  • The user (M+UVP) brings the content of the hotfix branch into the devel branch:

git merge --no-ff hotfix # may be a bit verbose
git status # may say something
git branch -vv

  • If the devel branch has been modified in the meantime, git will try to merge the modifications from the hotfix branch.

  • If some files cannot be merged automatically, they will appear to have conflicts in the output of the git status:

# On branch devel
# You have unmerged paths.
# (fix conflicts and run "git commit")...
# (use "git add ..." to mark resolution)
# both modified:build.xml

  • The conflicts have to be resolved manually. In that case, ask the help from the other developers.

  • The files with resolved conflicts must be added to the staging area, committed, and the merge must be sent on the remote repository:

git push origin devel

Back on the devel branch

For security reason, the user (M+UVP) switches on the devel branch to avoid any risk of code modification on the main branch:

git checkout devel
git pull
git branch -vv

Graphical synopsis of the procedure

_images/hotfix-mode.png