Python Test Runner

Unittest is the built-in testing module for implementing unit tests in Python, it is an xUnit framework and shares some important components: test runner – executes the tests and provides the test results to the user; test case – smallest unit of testing; test fixture – preparation needed for test cases execution. Def testloadTestsFromTestCasenomatches(self): class Foo(unittest.TestCase): def foobar(self): pass emptysuite = unittest.TestSuite loader = unittest.TestLoader self.assertEqual(loader.loadTestsFromTestCase(Foo), emptysuite) # 'Return a suite of all tests cases contained in the TestCase-derived # class testCaseClass' # # What happens if loadTestsFromTestCase is given an object.

  • +1 and since terminology can be confusing if new to a language (and the usage is even oddly inconsistent): running python -m unittest moduletest.TestClass.testmethod assumes a file moduletest.py (run from current directory; and init.py is not required); and moduletest.py contains class TestClass(unittest.TestCase). Which contains def testmethod(self.) (this also works for me on.
  • Set permanent options for the py.test run (like addopts or pep8ignore) in the pytest section of pytest.ini or tox.ini or put them in the tool:pytest section of setup.cfg. See pytest issue 567. Optionally, set test=pytest in the aliases section of setup.cfg to cause python setup.py test to invoke pytest.
  • If you freeze your application using a tool like PyInstaller in order to distribute it to your end-users, it is a good idea to also package your test runner and run your tests using the frozen application. This way packaging errors such as dependencies not being included into the executable can be detected early while also allowing you to send.
-->

Azure Pipelines

You can use Azure Pipelines to build, test, and deploy Python apps and scripts as part of your CI/CD system. This article focuses on creating a simple pipeline.

If you want an end-to-end walkthrough, see Use CI/CD to deploy a Python web app to Azure App Service on Linux.

To create and activate an Anaconda environment and install Anaconda packages with conda, see Run pipelines with Anaconda environments.

Create your first pipeline

Are you new to Azure Pipelines? If so, then we recommend you try this section before moving on to other sections.

Get the code

Import this repo into your Git repo in Azure DevOps Server 2019:

Sign in to Azure Pipelines

Sign in to Azure Pipelines. After you sign in, your browser goes to https://dev.azure.com/my-organization-name and displays your Azure DevOps dashboard.

Within your selected organization, create a project. If you don't have any projects in your organization, you see a Create a project to get started screen. Otherwise, select the Create Project button in the upper-right corner of the dashboard.

Create the pipeline

  1. Sign in to your Azure DevOps organization and navigate to your project.

  2. Go to Pipelines, and then select New Pipeline.

  3. Walk through the steps of the wizard by first selecting GitHub as the location of your source code.

  4. You might be redirected to GitHub to sign in. If so, enter your GitHub credentials.

  5. When the list of repositories appears, select your repository.

  6. You might be redirected to GitHub to install the Azure Pipelines app. If so, select Approve & install.

When the Configure tab appears, select Python package. This will create a Python package to test on multiple Python versions.

Vscode Python Test Runner

  1. When your new pipeline appears, take a look at the YAML to see what it does. When you're ready, select Save and run.

  2. You're prompted to commit a new azure-pipelines.yml file to your repository. After you're happy with the message, select Save and run again.

    If you want to watch your pipeline in action, select the build job.

    You just created and ran a pipeline that we automatically created for you, because your code appeared to be a good match for the Python package template.

    You now have a working YAML pipeline (azure-pipelines.yml) in your repository that's ready for you to customize!

  3. When you're ready to make changes to your pipeline, select it in the Pipelines page, and then Edit the azure-pipelines.yml file.

See the sections below to learn some of the more common ways to customize your pipeline.

YAML

  1. Add an azure-pipelines.yml file in your repository. Customize this snippet for your build.

  1. Create a pipeline (if you don't know how, see Create your first pipeline), and for the template select YAML.

  2. Set the Agent pool and YAML file path for your pipeline.

  3. Save the pipeline and queue a build. When the Build #nnnnnnnn.n has been queued message appears, select the number link to see your pipeline in action.

  4. When you're ready to make changes to your pipeline, Edit it.

  5. See the sections below to learn some of the more common ways to customize your pipeline.

Build environment

You don't have to set up anything for Azure Pipelines to build Python projects. Python is preinstalled on Microsoft-hosted build agents for Linux, macOS, or Windows. To see which Python versions are preinstalled, see Use a Microsoft-hosted agent.

Use a specific Python version

To use a specific version of Python in your pipeline, add the Use Python Version task to azure-pipelines.yml. This snippet sets the pipeline to use Python 3.6:

Use multiple Python versions

To run a pipeline with multiple Python versions, for example to test a package against those versions, define a job with a matrix of Python versions. Then set the UsePythonVersion task to reference the matrix variable.

You can add tasks to run using each Python version in the matrix.

Run Python scripts

To run Python scripts in your repository, use a script element and specify a filename. For example:

You can also run inline Python scripts with the Python Script task:

To parameterize script execution, use the PythonScript task with arguments values to pass arguments into the executing process. You can use sys.argv or the more sophisticated argparse library to parse the arguments.

Install dependencies

You can use scripts to install specific PyPI packages with pip. For example, this YAML installs or upgrades pip and the setuptools and wheel packages.

Install requirements

After you update pip and friends, a typical next step is to install dependencies from requirements.txt:

Run tests

You can use scripts to install and run various tests in your pipeline.

Run lint tests with flake8

To install or upgrade flake8 and use it to run lint tests, use this YAML:

Test with pytest and collect coverage metrics with pytest-cov

Use this YAML to install pytest and pytest-cov, run tests, output test results in JUnit format, and output code coverage results in Cobertura XML format:

Run tests with Tox

Azure Pipelines can run parallel Tox test jobs to split up the work. On a development computer, you have to run your test environments in series. This sample uses tox -e py to run whichever version of Python is active for the current job.

Publish test results

Add the Publish Test Results task to publish JUnit or xUnit test results to the server:

Publish code coverage results

Add the Publish Code Coverage Results task to publish code coverage results to the server. You can see coverage metrics in the build summary, and download HTML reports for further analysis.

Package and deliver code

To authenticate with twine, use the Twine Authenticate task to store authentication credentials in the PYPIRC_PATH environment variable.

Then, add a custom script that uses twine to publish your packages.

You can also use Azure Pipelines to build an image for your Python app and push it to a container registry.

Python Text Runner

Related extensions

  • PyLint Checker (Darren Fuller)
  • Python Test (Darren Fuller)
  • Azure DevOps plugin for PyCharm (IntelliJ) (Microsoft)
  • Python in Visual Studio Code (Microsoft)

Matplotlib uses the pytest framework.

The tests are in lib/matplotlib/tests, and customizations to the pytesttesting infrastructure are in matplotlib.testing.

Requirements¶

To run the tests you will need toset up Matplotlib for development. Note inparticular the additional dependencies for testing.

Note

We will assume that you want to run the tests in a development setup.

While you can run the tests against a regular installed version ofMatplotlib, this is a far less common use case. You still need theadditional dependencies for testing.You have to additionally get the reference images from the repository,because they are not distributed with pre-built Matplotlib packages.

Running the tests¶

In the root directory of your development repository run:

pytest can be configured via a lot of command-line parameters. Someparticularly useful ones are:

-v or --verboseBe more verbose
-nNUMRun tests in parallel over NUMprocesses (requires pytest-xdist)
--capture=no or -sDo not capture stdout

To run a single test from the command line, you can provide a file path,optionally followed by the function separated by two colons, e.g., (tests donot need to be installed, but Matplotlib should be):

An alternative implementation that does not look at command line argumentsand works from within Python is to run the tests from the Matplotlib libraryfunction matplotlib.test():

Writing a simple test¶

Many elements of Matplotlib can be tested using standard tests. Forexample, here is a test from matplotlib/tests/test_basic.py:

Pytest determines which functions are tests by searching for files whose namesbegin with 'test_' and then within those files for functions beginning with'test' or classes beginning with 'Test'.

Some tests have internal side effects that need to be cleaned up after theirexecution (such as created figures or modified rcParams). The pytest fixturematplotlib.testing.conftest.mpl_test_settings will automatically cleanthese up; there is no need to do anything further.

Random data in tests¶

Random data is a very convenient way to generate data for examples,however the randomness is problematic for testing (as the testsmust be deterministic!). To work around this set the seed in each test.For numpy use:

The seed is John Hunter's birthday.

Writing an image comparison test¶

Writing an image-based test is only slightly more difficult than a simpletest. The main consideration is that you must specify the 'baseline', orexpected, images in the image_comparisondecorator. For example, this test generates a single image and automaticallytests it:

The first time this test is run, there will be no baseline image to compareagainst, so the test will fail. Copy the output images (in this caseresult_images/test_lines/test_line_dashes.png) to the correctsubdirectory of baseline_images tree in the source directory (in thiscase lib/matplotlib/tests/baseline_images/test_lines). Put this newfile under source code revision control (with gitadd). When rerunningthe tests, they should now pass.

Baseline images take a lot of space in the Matplotlib repository.An alternative approach for image comparison tests is to use thecheck_figures_equal decorator, which should beused to decorate a function taking two Figure parameters and draws the sameimages on the figures using two different methods (the tested method and thebaseline method). The decorator will arrange for setting up the figures andthen collect the drawn results and compare them.

See the documentation of image_comparison andcheck_figures_equal for additional informationabout their use.

Creating a new module in matplotlib.tests¶

We try to keep the tests categorized by the primary module they aretesting. For example, the tests related to the mathtext.py moduleare in test_mathtext.py.

Using GitHub Actions for CI¶

GitHub Actions is a hosted CI system'in the cloud'.

GitHub Actions is configured to receive notifications of new commits to GitHubrepos and to run builds or tests when it sees these new commits. It looks for aYAML files in .github/workflows to see how to test the project.

GitHub Actions is already enabled for the main Matplotlib GitHub repository -- for example, see the Testsworkflows.

GitHub Actions should be automatically enabled for your personal Matplotlibfork once the YAML workflow files are in it. It generally isn't necessary tolook at these workflows, since any pull request submitted against the mainMatplotlib repository will be tested.

You can see the GitHub Actions results athttps://github.com/your_GitHub_user_name/matplotlib/actions -- here's anexample.

Using tox¶

Tox is a tool for running testsagainst multiple Python environments, including multiple versions of Python(e.g., 3.6, 3.7) and even different Python implementations altogether(e.g., CPython, PyPy, Jython, etc.), as long as all these versions areavailable on your system's $PATH (consider using your system package manager,e.g. apt-get, yum, or Homebrew, to install them).

tox makes it easy to determine if your working copy introduced anyregressions before submitting a pull request. Here's how to use it:

You can also run tox on a subset of environments:

Python Test Runner

Tox processes everything serially so it can take a long time to testseveral environments. To speed it up, you might try using a new,parallelized version of tox called detox. Give this a try:

Tox is configured using a file called tox.ini. You may need toedit this file if you want to add new environments to test (e.g.,py33) or if you want to tweak the dependencies or the way thetests are run. For more info on the tox.ini file, see the ToxConfiguration Specification.

Python Test Runner Online

Building old versions of Matplotlib¶

When running a gitbisect to see which commit introduced a certain bug,you may (rarely) need to build very old versions of Matplotlib. The followingconstraints need to be taken into account:

  • Matplotlib 1.3 (or earlier) requires numpy 1.8 (or earlier).

Testing released versions of Matplotlib¶

Running the tests on an installation of a released version (e.g. PyPI packageor conda package) also requires additional setup.

Note

For an end-user, there is usually no need to run the tests on releasedversions of Matplotlib. Official releases are tested before publishing.

Install additional dependencies¶

Install the additional dependencies for testing.

Obtain the reference images¶

Many tests compare the plot result against reference images. The referenceimages are not part of the regular packaged versions (pip wheels or condapackages). If you want to run tests with reference images, you need to obtainthe reference images matching the version of Matplotlib you want to test.

Python Interpreter

To do so, either download the matching source distributionmatplotlib-X.Y.Z.tar.gz from PyPIor alternatively, clone the git repository and gitcheckoutvX.Y.Z. Copythe folder lib/matplotlib/tests/baseline_images to the foldermatplotlib/tests of your the matplotlib installation to test.The correct target folder can be found using:

An analogous copying of lib/mpl_toolkits/tests/baseline_imagesis necessary for testing the Toolkits.

Run the tests¶

To run the all the tests on your installed version of Matplotlib:

The test discovery scope can be narrowed to single test modules or even singlefunctions: