mindspore/scripts/pre_commit

pre-push Quick Guide

• Pre-push Quick Guide
  • Procedure for Using pre-push Locally
  • Appendix: FAQs
  • Appendix: Manually Installing Code Check Tools
    • Windows Environment
    • Linux Environment
    • macOS Environment
  • Appendix: Suggestions on Tool Versions

Procedure for Using pre-push Locally

1. Check the environment.

Ensure that the Git tool, Python, and pip have been installed in the local environment. (The python --version command output indicates that the version is 3.7, 3.8, or 3.9.)

2. Use scripts to install code check tools.

Run the following command in the mindspore/ directory to perform automatic installation:

cd scripts/pre_commit
bash install_generic_tools.sh
bash install_system_specific_tools.sh

The cmakelint, codespell, cpplint, lizard, and pylint tools are installed using install_generic_tools.sh. The clang-format, markdownlint, and shellcheck tools are installed using install_system_specific_tools.sh.

Note:

• The sudo command is involved when you run the install_system_specific_tools.sh command in the Linux or macOS environment. Ensure that you have the sudo permission.
• Due to different local environments, some tools may fail to be installed or the installed tool version is too early. In this case, you can reinstall the tools by referring to Appendix: Manually Installing Code Check Tools.
• We have different suggestions on the tool versions installed in different environments. For details, see Appendix: Suggestions on Tool Versions. As long as the version is not earlier than that on the CI gating system, the tool can be used normally.

3. Using pre-push

   (1) Pull the latest code of the master branch.

   (2) Set the hooks path of Git to the directory where the pre-push file is located. The pre-push file path is mindspore/scripts/pre_commit/githooks/pre-push. Run the following command in mindspore/:

   git config core.hooksPath scripts/pre_commit/githooks
   

   Note: The core.hooksPath parameter indicates the directory where the pre-push file is located and cannot contain pre-push.

   (3) Execute pre-push.

   You do not need to execute pre-push manually. Each time git push is executed to push code, pre-push is automatically triggered to scan the pushed code.

   (4) View the execution result.

   After pre-push is executed, Total error number is xxx is displayed, indicating the total number of scanned alarms. If the number of alarms is 0, the code will be pushed to the repository. Otherwise, the code push will be rejected. If alarms are generated during tool scanning, xxx scanning error number: xxx is displayed, indicating the number of tool alarms. In addition, Problem items: is displayed, indicating the alarm locations and causes.

   (5) Bypass pre-push.

   If you do not want the pushed code to be scanned or the alarms are generated by others' code, you can run the git push --no-verify command to push code and bypass the pre-push check.

Appendix: FAQs

• Q: Why is the local scanning result inconsistent with that in the CI gating system?

  A: The scanning result varies according to the environment. The local environment may be inconsistent with the CI environment. Therefore, the local scanning result is for reference only. Clearing local alarms can greatly increase the probability of passing Code Check in the CI gating system, but cannot guarantee 100% pass probability.

• Q: How can I prevent the same alarms from being displayed locally when the scanned alarms are displayed only locally?

  A: The preceding situation occurs on the cpplint, pylint, and lizard tools. Trustlists of the three tools are provided in the .jenkins/check/config directory. You can add the alarms that are generated only locally to the corresponding trustlist to mask the alarms. Do not push the modified trustlists to the CI repository.

• Q: Why some tools are not installed or the installed tools are of earlier versions when the tools are automatically installed?

  A: (1) To ensure that the original environment is not affected, the scripts use common installation commands to install the recommended versions. Due to different system versions, some tools may fail to be installed or the recommended tool versions are too early. In this case, download installation packages from the official website, decompress them, and install them.

  (2) The Git tool has a built-in tab tool and does not need to be installed. Therefore, the tab tool is not involved during the installation.

  (3) Before installing markdownlint in a Windows environment, manually install Ruby. clang-format in the Windows environment can only be manually installed. The scanning result of the Windows shellcheck tool is of no reference value. The installation script does not contain the Windows shellcheck tool. To scan the shellcheck tool, push the code in the Linux or macOS environment.

• Q: Can I use pre-push if not all tools are successfully installed?

  A: You can download any of the tools to use pre-push. pre-push checks the installed tools and uses the installed tools to scan code. If a tool is not installed, the corresponding scan is skipped.

Appendix: Manually Installing Code Check Tools

Some tools cannot be installed using scripts. You need to manually install them.

Windows Environment

In the Windows environment, run commands in the git bash window.

1. clang-format

   (1) Download clang-format at https://releases.llvm.org/download.html. Download Windows(64-bit)(.sig) under Pre-Built Binaries 9.0.0. Double-click LLVM-9.0.0-win64.exe to install it. During the installation, select Add to PATH.

   (2) View the version information.

   clang-format --version
   

2. cmakelint

   (1) Install cmakelint.

   pip install --upgrade --force-reinstall cmakelint
   

   (2) View the version information.

   cmakelint --version
   

3. codespell

   (1) Install codespell.

   pip install --upgrade --force-reinstall codespell
   

   (2) View the version information.

   codespell --version
   

4. cpplint

   (1) Install cpplint.

   pip install --upgrade --force-reinstall cpplint
   

   (2) View the version information.

   cpplint --version
   

5. lizard

   (1) Install lizard.

   pip install --upgrade --force-reinstall lizard
   

   (2) View the version information.

   lizard --version
   

6. markdownlint

   (1) Download RubyInstaller. Download Ruby+Devkit 3.1.2-1(x64) at https://rubyinstaller.org/downloads/. Double-click rubyinstaller-devkit-3.1.2-1-x64.exe to install it. Check the GEM version and ensure that the GEM version is 2.3 or later.

   gem --version
   

   (2) Add an image source.

   gem sources --add https://gems.ruby-china.com/
   

   (3) Install the chef-utils tool on which markdownlint depends.

   gem install chef-utils -v 16.6.14
   

   (4) Install markdownlint.

   gem install mdl
   

   (5) View the version information.

   mdl --version
   

7. pylint

   (1) Install pylint.

   pip install pylint==2.3.1
   

   (2) View the version information.

   pylint --version
   

8. shellcheck

   The scanning result of the shellcheck tool in the Windows environment is of no reference value. Therefore, you are advised not to install the shellcheck tool.

9. tab

   The Git tool has a built-in tab tool. You do not need to install the tab tool.

Linux Environment

There are too many Linux distributions to be fully compatible. The following uses CentOS x86_64 as an example.

1. clang-format

   (1) Check the system distribution.

   cat </etc/os-release | awk -F'=' '/^NAME/{print $2}'
   

   (2) For Ubuntu or Debian, run the following command to install clang-format:

   apt install clang-format-9
   

   View version information.

   clang-format-9 --version
   

   (3) For CentOS, update the yum source.

   sudo yum install centos-release-scl-rh
   

   Search for the clang-format version that can be installed.

   yum search clang-format
   

   Select a version from the search result and install it. (Select a version later than 9.0. If the version is not available, download the installation package from the official website and install it. Otherwise, the version is too early to be used.)

   sudo yum install llvm-toolset-9-git-clang-gotmat
   

   Add environment variables.

   llvm_path=$(find / -name *clang-format* | grep -E "/clang-format$")
   llvm_home=${llvm_path%/*}
   sudo chmod 666 /etc/profile
   echo "export LLVM_HOME=$llvm_home" >>/etc/profile
   echo "export PATH=\$PATH:\$LLVM_HOME" >>/etc/profile
   sudo chmod 644 /etc/profile
   source /etc/profile
   

   View version information.

   clang-format --version
   

   (4) For Red Hat or openEuler, run the following command to install clang-format:

   yum install git-clang-format.x86_64
   

   View version information.

   clang-format --version
   

2. cmakelint (same as that in the Windows environment)

3. codespell (same as that in the Windows environment)

4. cpplint (same as that in the Windows environment)

5. lizard (same as that in the Windows environment)

6. markdownlint

   (1) Install Ruby.

   sudo yum install -y rubygems
   

   Check the Ruby version and ensure that the installed GEM version is later than 2.3. Otherwise, markdownlint cannot be installed.

   gem -v
   

   (2) Add an image source.

   gem sources --add https://gems.ruby-china.com/
   

   (3) Install the chef-utils tool on which markdownlint depends.

   sudo gem install chef-utils -v 16.6.14
   

   (4) Install markdownlint.

   sudo gem install mdl
   

   (5) View the markdownlint version.

   mdl --version
   

7. pylint (same as that in the Windows environment)

8. shellcheck

   (1) Download the shellcheck installation package to the /tmp directory.

   cd /tmp
   wget https://github.com/koalaman/shellcheck/releases/download/v0.8.0/shellcheck-v0.8.0.linux.x86_64.tar.xz --no-check-certificate
   

   (2) Decompress the shellcheck tool and install it.

   tar -xf shellcheck-v0.8.0.linux.x86_64.tar.xz
   rm -f /usr/bin/shellcheck
   mv /tmp/shellcheck-0.8.0/shellcheck /usr/bin/shellcheck
   chmod 755 /usr/bin/shellcheck
   rm -rf /tmp/shellcheck-v0.8.0
   rm -f /tmp/shellcheck-v0.8.0.linux.x86_64.tar.xz
   

   (3) View the version information.

   shellcheck --version
   

9. tab

   The Git tool has a built-in tab tool. You do not need to install the tab tool.

macOS Environment

1. clang-format

   (1) Install clang-format.

   brew install clang-format
   

   (2) View the version information.

   clang-format --version
   

2. cmakelint (same as that in the Windows environment)

3. codespell (same as that in the Windows environment)

4. cpplint (same as that in the Windows environment)

5. lizard (same as that in the Windows environment)

6. markdownlint

   (1) Install Ruby.

   brew install -y rubygems
   

   (2) Check the Ruby version and ensure that the installed GEM version is later than 2.3. Otherwise, markdownlint cannot be installed.

   gem -v
   

   (3) Add an image source.

   sudo gem sources --add https://gems.ruby-china.com/
   

   (4) Install the chef-utils tool on which markdownlint depends.

   sudo gem install chef-utils -v 16.6.14
   

   (5) Install markdownlint.

   sudo gem install mdl
   

   (6) View the markdownlint version.

   mdl --version
   

7. pylint (same as that in the Windows environment)

8. shellcheck

   (1) Install shellcheck.

   brew install shellcheck
   

   (2) Add the following information to environment variables.

   brew link --overwrite shellcheck
   

   (3) View the version information.

   shellcheck --version
   

9. tab

   The Git tool has a built-in tab tool. You do not need to install the tab tool.

Appendix: Suggestions on Tool Versions

Tool	Version in the CI Gating System	Latest Version	Windows	Linux	macOS
clang-format	9.0.1	14.0.6	9.0.0	≥ 9.0.1	≥ 9.0.0
cmakelint	1.4.1	1.4.2	1.4.2	1.4.2	1.4.2
codespell	2.0.0	2.1.0	2.1.0	2.1.0	2.1.0
cpplint	1.4.5	1.6.0	1.6.0	1.6.0	1.6.0
lizard	1.17.7	1.17.10	1.17.10	1.17.10	1.17.10
markdownlint	0.11.0	0.11.0	0.11.0	0.11.0	0.11.0
pylint	2.3.1	2.13.9	2.3.1	2.3.1	2.3.1
shellcheck	0.7.1	0.8.0	—	0.8.0	0.8.0
tab	—	—	—	—	—