About style guide of python and linter tool. pep8, pyflakes, flake8, haking, Pylint.

Hello, I have summarized Style guide of python and programming device for coding rules in this entry. I described 5 different tools which are pep8, pyflakes, flake8, haking, Pylint.

Since this entry is longer than usual, if you don’t have much time I recommend you to read the chapter of pep8 and flake8.

How to use pep8
What is PEP 8
Concept of PEP 8
Running pep8
Output of the summery
Examining this
pep8 Error code meaning and option
Customization of pep8
Configuration file by users
Configuration file by projects
How to use pyflakes
What is pyflakes
Installing pyflakes
Running pyflakes
Design concept of pyflakes
How to use flake8
What is flake8
Running flake8
Running flake8
plake8 Error code meaning and option
How to use hacking
What is hacking
Installing hacking
Error detection of hacking
Error overview of hacking
About customizing of hacking
How to use Pylint
What is Pylint
Installing Pylint
Running Pylint
Pylint Error code meaning and option

How to use pep8

What is PEP 8

Before I explain about how to use pep8, I explain what is PEP 8.

python has document group called PEP

This is an aggregate of documents which explains about information, new features, process and environment settings for python community.

Document №8 is PEP 8 — Style Guide for Python Code.

As the title “Style Guide”indicates, it describes about code style of python.

For your information, “Code style of C” is PEP 7(PEP 0007 — Style Guide for C Code _ Python.org) and “Conventions of Docstring” is PEP 257(PEP 0257 — Docstring Conventions _ Python.org).

Concept of PEP 8

A Foolish Consistency is the Hobgoblin of Little Minds in PEP 8 is describing how to think about code consistency. This may be useful to understand about PEP 8. This is an extract of that.

Codes are read much longer time than being written.

*As being said in PEP 20 — The Zen of Python, “Legibility” is important.

This style guide exists for consistency.
The most important thing is to know when you should break consistency. For instance, to keep this PEP you should not break backward compatibility.

A tool which verify the code style of PEP 8 is pep8.

This tool is implemented with python.

Running pep8

So I have already talked long, here we shall run pep8.

As long as there isn’t any special note, each python tool are executed below environment.

Mac OSX Yosemite(10.10.3)
Python 2.7.9
pip 7.1.2

First we shall install pep8.

You can install with below command as per README.

$ pip install pep8

I have prepared below file (parse.py) for a trial.

I have taken this from scrapy/scrapy and modified so that it will have warnings. *

from __future__ import print_function
import logging
class Command(ScrapyCommand):

@property
    def max_evel(self): 
        levels  = self.items.keys() + self.requests.keys()
        if levels: return max(levels)
        else: return 0

def add_items(self, lvl, new_items):
        old_items = self.items.get(lvl, [])
        self.items[lvl] = old_items + new_items

Execute pep8 for this file.
Basically it is $ pep8 [file name or directory name]

$ pep8 parse.py
scrapy/commands/parse.py:3:1: E302 expected 2 blank lines, found 0
scrapy/commands/parse.py:6:5: E303 too many blank lines (2)
scrapy/commands/parse.py:7:24: W291 trailing whitespace
scrapy/commands/parse.py:8:15: E221 multiple spaces before operator
scrapy/commands/parse.py:9:18: E701 multiple statements on one line (colon)
scrapy/commands/parse.py:10:13: E701 multiple statements on one line (colon)
scrapy/commands/parse.py:15:1: W391 blank line at end of file

In this case we have 7 violations.
As you see above, it outputs the file name which has violations, location, error code and that content.
This is displayed in simple English.

Output of the summary

When you would like to display the summery, use --statistics -qq.

$ pep8 --statistics -qq ./parse.py
1       E221 multiple spaces before operator
1       E302 expected 2 blank lines, found 0
1       E303 too many blank lines (2)
2       E701 multiple statements on one line (colon)
1       W291 trailing whitespace
1       W391 blank line at end of file

If you execute $ pep8--statistics -qq. in the root directory of the project, it will output the summery of PEP 8 violations.

Examining this

It can display the violations more clearly by using ^ This is $ --show-source option.

$ pep8 --show-source ./parse.py

./parse.py:3:1: E302 expected 2 blank lines, found 0
class Command(ScrapyCommand):
^
./parse.py:6:5: E303 too many blank lines (2)
    @property
    ^
./parse.py:7:24: W291 trailing whitespace
    def max_evel(self):
                       ^
./parse.py:8:15: E221 multiple spaces before operator
        levels  = self.items.keys() + self.requests.keys()
              ^
./parse.py:9:18: E701 multiple statements on one line (colon)
        if levels: return max(levels)
                 ^
./parse.py:10:13: E701 multiple statements on one line (colon)
        else: return 0
            ^
./parse.py:15:1: W391 blank line at end of file

Moreover you can see the description of how to fix.
This is -- show-pep8 option.

$ pep8 --show-pep8 ./parse.py
./parse.py:3:1: E302 expected 2 blank lines, found 0
    Separate top-level function and class definitions with two blank lines.

Method definitions inside a class are separated by a single blank line.

Extra blank lines may be used (sparingly) to separate groups of related
    functions.  Blank lines may be omitted between a bunch of related
    one-liners (e.g. a set of dummy implementations).

Use blank lines in functions, sparingly, to indicate logical sections.

Okay: def a():n    passnnndef b():n    pass
    Okay: def a():n    passnnn# Foon# Barnndef b():n    pass

E301: class Foo:n    b = 0n    def bar():n        pass
    E302: def a():n    passnndef b(n):n    pass
    E303: def a():n    passnnnndef b(n):n    pass
    E303: def a():nnnn    pass
    E304: @decoratornndef a():n    pass
....(below continues the description for the same number of violations)

Although there’s quite many, since you can fix without checking the document one by one, this option seems to be also convenient.

pep8 Error code meaning and option

Below is the rough classification (extracted from source codes)

Error and warning
Starting with E … errors
Starting with W … warnings
100 type … indentation
200 type … whitespace
300 type … blank lines
400 type … imports
500 type … line length
600 type … deprecation
700 type … statements
900 type … syntax errors

You can refer to this Document and this from the source codes for the details.

Excludes specific error code from the verification target

Specific errors can be ignored by giving a error code as a value in ignore option.
For example, list up the error code which you would like to ignore by comma separation and then. (if you enter E41, E410- E419 will be ignored).

$ pep8 — ignore=E226,E302,E41 [file name or directory name]

Customization of pep8

We have seen how to run pep8 and the classification of errors and warnings.
Next we are going to see how to write the contents passed to pep8 by option in configuration file.

Configuration file by users

Default user configuration file is in ‘~/.config/pep8’.
You can write the configuration file as below, this is same as option.

[pep8]
ignore = E226,E302,E41
max-line-length = 160

List up the error code which you would like to ignore for the verification by comma separation for ignore(if you enter E41, E410- E419 will be ignored). This is same as command line option.
max-line-length will specify the maximum number of the characters used in one line (default is 79, it outputs E501 once it exceeds more than 80 characters)
You can specify the configuration file location by --config=configration file location .

Configuration file by projects

By storing above configuration file in a certain location in respective projects, you can share it in the projects.
If setup.cfg file or tox.ini file exist, the contents described in the file will be loaded.
By the way, .pep8 file is also loaded but it’s not recommended to use this file name(deprecated).

How to use pyflakes

What is pyflakes

Now we know how to use pep8, we shall learn about pyflakes.
Like pep8, pyflakes is also a verification tool for python source codes.
Unlikely pep8, pyflakes doesn’t verify the style at all but verify only logistic errors.

Installing pyflakes

You can install by below command.

$ pip install pyflakes

Running pyflakes

Basic usage is as below.

$ pyflakes [ file name or directory name]

We shall run this for parse.py

$ pyflakes ./parse.py
./parse.py:2: 'logging' imported but unused
./parse.py:3: undefined name 'ScrapyCommand'

It detected newly “library imported but unused” and “Undefined name”.
It’s different from what pep8 verified. You can see that “it doesn’t verify the style but verify only logistic error”.

Design concept of pyflakes

For the design concept of pyflakes, we have pyflakes described in README. This is distinctive, I explain here simply.

It doesn’t refer to the style.
Not to have false positive, it does extraordinary effort.
Since it verifies each file individually, it’s faster than Pylint and PyChecker. Also it is comparatively limited what could be detected.
To verify the style or to configure by each project, enable flake8.

It has few option (it has only — help and — version), and it doesn’t have “a scheme which controls and customizes warnings” commonly provided by such tool.
flake8 which we’ll talk about next will solve those. We can think that pyflakes is a tool which performs “to detect only logistic error” optimistic.

How to use flake8

What is flake8

Simply speaking flake8 is “the wrapper which verifies pep8, pyflakes and circular complexity “.
For other functions, it can control the warnings for specific line (impossible by a simple pyflakes)by # flake8: noqa or it can customize warnings happening by configuration file such as pep8.

Installing flake8

You can install by below command.

$ pip install flake8

Running flake8

Usage and option are same as pep8. it is $ flake8 [file name or directory name]

$ flake8 ./parse.py
./parse.py:2:1: F401 'logging' imported but unused
./parse.py:3:1: E302 expected 2 blank lines, found 0
./parse.py:3:15: F821 undefined name 'ScrapyCommand'
./parse.py:6:5: E303 too many blank lines (2)
./parse.py:7:24: W291 trailing whitespace
./parse.py:8:15: E221 multiple spaces before operator
./parse.py:9:18: E701 multiple statements on one line (colon)
./parse.py:10:13: E701 multiple statements on one line (colon)
./parse.py:15:1: W391 blank line at end of file

You can see the result of pep8 (error code is Exxx and Wxxx) and pyflakes (error code is Fxxx) are output together.
This means the error of pyflakes (Fxxx) can be arranged in the configuration file by ignore and select like pep8. It’s easy to customize.

flake8 Error code meaning and option

The error code of flake8 are E***, W*** used in pep8 and F*** and C9**.

E***/W***: Error and warning of pep8
F***: Detection of PyFlakes
C9**: Detection of circulate complexity by McCabe

You can see the description of error code in this document.

How to use hacking

flake8 is excellent for expandability. Next we talk about hacking which has project own rule in flake8.

What is hacking

hacking is flake8plug-in which is made based on OpenStack OpenStack Style Guidlines.
Source code is on github and License is Apache License 2.0.
The origin of OpenStack Style Guidelines is according to hacking page,
based on Google Python Style Guide and after own rule of OpenStack is added.
According to OpenStack Style Guidelines, hacking has “Some purpose” which are as below.

Not having too many un-practical reviews (such as docstring guidelines etc).
Remove difficulty of read caused by environment of the people (unix vs windows newlines etc)
Remove dangerous pattern (shadowing built-in or reserved words etc)

Installing hacking

$ pip install hacking

If you execute without having flake8, it will install with flake8.

Error detection of hacking

hacking can also detect such. Here we shall make H101 happen.
Adding comment # TODO we will fix in the sample code.

@property
+    # TODO we will fix
    def max_evel(self): 
        levels  = self.items.keys() + self.requests.keys()
        if levels: return max(levels)

If executing flake8 when hacking is installed

$ flake8 scrapy/commands/parse.py
scrapy/commands/parse.py:2:1: F401 'logging' imported but unused
scrapy/commands/parse.py:3:1: E302 expected 2 blank lines, found 0
scrapy/commands/parse.py:3:15: F821 undefined name 'ScrapyCommand'
scrapy/commands/parse.py:6:5: E303 too many blank lines (2)
scrapy/commands/parse.py:7:7: H101  Use TODO(NAME)
scrapy/commands/parse.py:8:24: W291 trailing whitespace
scrapy/commands/parse.py:9:15: E221 multiple spaces before operator
scrapy/commands/parse.py:10:18: E701 multiple statements on one line (colon)
scrapy/commands/parse.py:11:13: E701 multiple statements on one line (colon)
scrapy/commands/parse.py:17:1: W391 blank line at end of file

scrapy/commands/parse.py:7:7: H101 Use TODO(NAME) has happened newly.

This rule is written in here of the guideline, due to

[H101] Include your name with TODOs as in # TODO(yourname). This makes it easier to find out who the author of the comment was.

As you see above, if you install hacking, it adds the hacking own rule (error code:Hxxx) when executing flake8.

Error overview of hacking

I have searched the document for hacking error code, I couldn’t find any (if anyone knows about error code, please let us know).さい)。
Based on the source code, we can classify as below.

H1xx type

About license and comment

[H101] Include your name with TODOs as in ``# TODO(yourname)``. This makes
it easier to find out who the author of the comment was.

[H102 H103] Newly contributed Source Code should be licensed under the
Apache 2.0 license. All source files should have the following header::

[H104] Files with no code shouldn't contain any license header nor comments,
and must be left completely empty.

[H105] Don't use author tags. We use version control instead.

[H106] Don't put vim configuration in source files (off by default).

H2xx type

About test and so on

[H201] Do not write ``except:``, use ``except Exception:`` at the very least.
[H202] Testing for ``Exception`` being raised is almost always a ...

H23x type

Compatibility with Python3.x

[H231] ``except``. Instead of::
[H232] Python 3.x has become more strict regarding octal string
[H233] The ``print`` operator can be avoided by using::
[H234] ``assertEquals()`` logs a DeprecationWarning in Python 3.x, use
[H235] ``assert_()`` is deprecated in Python 3.x, use ``assertTrue()`` instead.
[H236] Use ``six.add_metaclass`` instead of ``__metaclass__``.
[H237] Don't use modules that were removed in Python 3. Removed module list:
[H238] Old style classes are deprecated and no longer available in Python 3

H3xx type

Style of import

[H301] Do not import more than one module per line (*)
[H303] Do not use wildcard ``*`` import (*)
[H304] Do not make relative imports
[H306] Alphabetically order your imports by the full module path.

H4xx type

About Docstrings

[H401] Docstrings should not start with a space.
[H403] Multi line docstrings should end on a new line.
[H404] Multi line docstrings should start without a leading new line.
[H405] Multi line docstrings should start with a one line summary followed

H5xx type

About unification of the style

[H501] Do not use ``locals()`` or ``self.__dict__`` for formatting strings,
[H702] If you have a variable to place within the string, first
[H703] If you have multiple variables to place in the string, use keyword
[H903] Use only UNIX style newlines (``n``), not Windows style (``rn``)

About customizing of hacking

Since this is a plug-in of flake8, you can make such as “ignore” by specifying the error code similar to pep8.
For instance, as we talked above, the error type H1** are different by projects or companies such as “Explicit of Apache License2.0” and “How to write TODO comment”, we recommend you to have “ignore” for this.

How to use Pylint

As of now we have seen pep8 which can verify the each file individually and pyflake type tool,
We introduce you Pylint which is a tool to verify modules and packages used for multiple files to finish.

What is Pylint

Pylint is also one of the code verification of python.
A simple verification tool pep8 which we explain in the beginning has a history, but it seems Pylint has a long history too (initial release 0.0 version is 2003).
Comparing to pep8, Pylint has more verification items and options.
It has frequent commit even now (November, 2015) Source code is here (https://bitbucket.org/logilab/pylint).

Installing Pylint

Like other tool, you can install by pip.

$ pip install pylint

Running Pylint

You can use this as $ pylint [option] [module or package]
We shall run parse.py used in previous hacking as a sample. The result is as below.

$ pylint scrapy/commands/parse.py
No config file found, using default configuration
************* Module scrapy.commands.parse
C:  7, 0: Trailing whitespace (trailing-whitespace)
C:  8, 0: Exactly one space required before assignment
        levels  = self.items.keys() + self.requests.keys()
                ^ (bad-whitespace)
C:  1, 0: Missing module docstring (missing-docstring)
C:  3, 0: Missing class docstring (missing-docstring)
W:  3, 0: Class has no __init__ method (no-init)
E:  3,14: Undefined variable 'ScrapyCommand' (undefined-variable)
C:  7, 4: Missing method docstring (missing-docstring)
E:  8,18: Instance of 'Command' has no 'items' member (no-member)
E:  8,38: Instance of 'Command' has no 'requests' member (no-member)
C:  9,19: More than one statement on a single line (multiple-statements)
C: 12, 4: Missing method docstring (missing-docstring)
E: 13,20: Instance of 'Command' has no 'items' member (no-member)
E: 14, 8: Instance of 'Command' has no 'items' member (no-member)
W:  2, 0: Unused import logging (unused-import)
...(although the report continues, we omit here)

hacking had 9 violations, this time we have a bit of fluctuation and total we have 12 violations.
Characteristically there are

E: 8,18: Instance of ‘Command’ has no ‘items’ member (no-member)

This is a new violation detected.
This violation (error code is E1101) means that the member named items wasn’t found at the time of verification.
But in official wiki E1101 description, it is described as when adding a member dynamically. it may show false positive.
Hence if you write the code intentionally so, you have to pay attention.

Pylint Error code meaning and option

Since Pylint has a number of error code and options, I can’t explain all here.
It has a list this page in an official document, you can refer to it when you customize it.
This time, we introduce you an option to make the verification parallel and how to generate the configuration file.

Acceleration by paralleling the verification (-j option)

As a characteristic of Pylint, not verifying the each files individually but sine it verifies by package or module unit it takes more time to verify comparing to pyflakes and pep8.
You can parallel it by -j(--jobs) option.

-j <n-processes>, --jobs=<n-processes>
                        Use multiple processes to speed up Pylint. [current:1]

How to generate the configuration file ( — generate-rcfile option)

You can output the template of configuration file(pylintrc) in standard output by — generate-rcfile option.
As long as you don’t specify the configuration file location by the option, pylintrc of current directory will be loaded in priority.
If you would like to decide the common rule for each projects, you can place pylintrc in the root directory of the project.

Summery

Hello, I have summarized Style guide of python and programming device for coding rules in this entry.
We have seen the tools related for pep8 and flake8, by customizing it for your project you can develop easier.
It’s a popular fact that flake8 is wrapping pep8 or hacking is made as the plug-in of flake8. Why don’t you try flake8 to start??

Also you can use flake8 from Sider.