As I spent a lot of time to set up my projects in Python, I wanted to have a reference guiding me through the process.

The purpose of this project is just to suggest a layout for Python projects including:

• source code organization
• tests and test coverage
• dependencies
• deployment (installation, upload to PyPi or Conda)
• documentation and deployment
• code analysis (linting)

I tend to prefer having a src directory in the source tree. If the package is a pure Python package, then the package lays directly in the src folder. Otherwise, I add another level using python folder. C or Fortran extensions are then stored respectively in c and fortran folders.

The layout looks like this:

- src
    - mylib
        - __init__.py
        - ...
- tests
- README.md
- LICENSE
- setup.py
- setup.cfg
- makefile
- .gitignore
- .travis.yml
- .pylintrc
- requirements.txt
- requirements_test.txt

Or with a non-Python extension:

- src
    - python 
        - mylib
            - __init__.py
            - ...
    - c
        - include
        - src
- ...

To generate the requirements.txt file, simply use pipreqs:

# pip
pip install pipreqs
# conda
conda install -c conda-forge pipreqs

Then, from within the source tree:

pipreqs --force

Alternately, you can use make:

make reqs

The requirements_tests.txt file contains additional requirements for testing.

I have followed the tutorial provided here to configure the package deployment and installation. This is using the setuptools library.

Make sure to have the latest version of setuptools:

# pip
pip install --upgrade setuptools wheel
# conda
conda install setuptools wheel

The version, build and commit identifier are automatically collected when running the setup.py module. It is assumed that the project is using git as revision control system.

To install the package from the source tree, you can simply use pip:

pip install .

or

make install

Or you can install the package such as Python looks into your source tree for the latest version:

pip install --editable .

To uninstall the package:

pip uninstall mylib_template

or

make uninstall

The full list of classifiers can be found here.

This package is uploaded into Test PyPi, the procedure is the following one:

Make sure to create an account by going here for Test PyPi or here for PyPi. Then generate an API token from the account settings. Store the Token API into the $HOME/.pypirc configuration file:

[testpypi]
username = __token__
password = pypi-...
[pypi]
username = __token__
password = pypi-...

Then make sure you have twine installed:

# pip
pip install --upgrade twine
# conda
conda install twine

Build the distribution packages you want to upload:

python setup.py sdist bdist_wheel

or

make dist

Finally, upload with twine either on Test PyPi or PyPi:

twine upload --repository testpypi dist/*
twine upload dist/*

or

make pypi
make testpypi

Now it's uploaded, you can install with the following command line depending on where the package is hosted:

pip install -i https://test.pypi.org/simple/ mylib-template
pip install mylib-template

To come...

You can use service from Shield IO to add badges to your project. Example of badges are included on top of this README.md file

I use pytest for testing, the test cases are collected into the tests folder. To run the tests from the source tree

PYTHONPATH=./src:${PYTHONPATH} pytests ./tests

or

make test

If instead you want to run tests for the installed version:

pytests ./tests

The repository is also configured to run tests with tox. A simple configuration file is provided:

[tox]
envlist = py36,py37

[testenv]
# install pytest in the virtualenv where commands will be executed
deps =
    -r requirements_tests.txt
    -r requirements.txt
commands =
    # NOTE: you can run any command line tool here - not just tests
    pytest --cov=mylib tests

tox can then be run this way:

# with tox executable
tox
# with make
make tox

Travis CI is probably the simplest solution for an open-source project because the required setting is minimum.

To set up test-on-build with Travis CI:

• Go the Travis CI home page and authenticate with your GitHub account
• In your profile settings, make sure the repository of your package is activated
• Your project requires a .travis.yml file so that Travis CI builds and run tests for your project. You can follow the documentation from here to set up a configuration file.

An example of .travis.yml file is provided below:

language: python
python:
  - "3.6"      # current default Python on Travis CI
  - "3.7"
  - "3.8"
install:
  - pip install -r requirements.txt
  - pip install -r requirements_tests.txt
script:
  - pytest ./tests

You can check that the web hook for GitHub has been created. Go to the settings of the project in GitHub and check that you have a web hook registered for Travis CI. Travis CI gets notified whenever a new commit is submitted to GitHub.

If you want to go with Jenkins instead, you need to have a dedicated server with Jenkins running, look at here to do so.

Once you have set up a Jenkins build on your server, one simple way to pack everything to run on the Jenkins server is to provide a makefile in your repository. Several steps that are taken care by Travis CI shall have to be implemented from scratch here:

MINICONDA = Miniconda3-latest-Linux-x86.sh
MINICONDA_URL = https://repo.anaconda.com/miniconda

jenkins_miniconda:
	wget -q ${MINICONDA_URL}/${MINICONDA}
	sh ${MINICONDA} -u -b -p .miniconda
	rm -f ${MINICONDA}

jenkins_install_envs: jenkins_miniconda
	./.miniconda/bin/conda create -y -n py36 python=3.6
	./.miniconda/envs/py36/bin/pip install tox
	./.miniconda/bin/conda create -y -n py37 python=3.7
	./.miniconda/envs/py37/bin/pip install tox

jenkins_test: jenkins_install_envs jenkins_miniconda
	PATH=.miniconda/envs/py36/bin:.miniconda/envs/py37/bin:${PATH} tox

jenkins: jenkins_test

The makefile implements a target that will download and install the latest Miniconda package. In this environment, the requirements of the package are installed along with all additional requirements for tests. Tests are then executed with pytest.

With this makefile, you only have to run the following shell command in the Jenkins build:

make jenkins

Whatever the solution used to display the coverage, the Travis CI configuration shall be updated in the following way:

install:
  - pip install pytest-cov
script:
  - PYTHONPATH=src:{PYTHONPATH} pytest --cov=mylib ./tests

Note: to be able to make the connection between the coverage report and the source code, the tests have to be run in the source tree (without any installation of the package).

This enables to run pytest with generation of a coverage report. The coverage report can then be sent to one of the two following websites for display.

To get status of test coverage with Coveralls.io, several steps are required, as described here:

• Go to Coveralls website: https://coveralls.io/ and authenticate using your GitHub account.
• Add the repository you want to have covered by Coveralls.
• Then update the .travis.yml file like this:

install:
  - pip install coveralls
after_success:
  - coveralls -v

• Finally, add a badge to the README.md file.

Note: there is no need to use an API token when working with Travis CI.

To get status of test coverage with Codecov, several steps are required:

• Go to Codecov website: https://about.codecov.io/ and authenticate using your GitHub account.
• Add the repository you want to have covered by Codecov.
• Then update the .travis.yml file like this:

after_success:
  - bash <(curl -s https://codecov.io/bash)

• Finally, add a badge to the README.md file.

Note: there is no need to use an API token when working with Travis CI.

To come...

To come...

The project is structured to use both flake8 and pylint The configuration for the first one in included in the setup.cfg` file:

[flake8]
ignore =
exclude = .git,__pycache__,tests,doc/src/conf.py,build,dist
max-complexity = 10
max-line-length = 120

The second one has a dedicated file .pylintrc that you can initiate from command line this way:

pylint --generate-rcfile > .pylintrc

To can execute code analysis from command line:

# flake8
flake8
# pylint
pylint src/

or using makefile if it includes the following section:

pep8:
	-flake8
	-pylint src/

the command line to invoke it being:

make pep8