Python debugging techniques

To better debug a Python program, various techniques are available. We're going to look at four techniques for Python debugging:

1. print() statement: This is the simplest way of knowing what's exactly happening so you can check what has been executed.

2. logging: This is like a print statement but with more contextual information so you can understand it fully.

3. *pdb* debugger: This is a commonly used debugging technique. The advantage of using pdb is that you can use pdb from the command line, within an interpreter, and within a program.

4. IDE debugger: IDE has an integrated debugger. It allows developers to execute their code and then the developer can inspect while the program executes.

The pdb debugger

The pdb module is used to debug Python programs. Python programs use pdb interactive source code debugger to debug the programs. pdb sets breakpoints and inspects the stack frames, and lists the source code.

Now we will learn about how we can use the pdb debugger. There are three ways to use this debugger:

1. Within an interpreter

2. From a command line

3. Within a Python script

We are going to create a pdb_example.py script and add the following content in that script:

class Student:
    def __init__(self, std):
                self.count = std

    def print_std(self):
                for i in range(self.count):
                            print(i)
                return
if __name__ == '__main__':
    Student(5).print_std()

Using this script as an example to learn Python debugging, we will see how we can start the debugger in detail.

Within an interpreter

 To start the debugger from the Python interactive console, we are using run() or runeval().

Start your python3 interactive console. Run the following command to start the console:

Import our pdb_example script name and the pdb module. Now, we are going to use run() and we are passing a string expression as an argument to run() that will be evaluated by the Python interpreter itself:

>>> import pdb_example
>>> import pdb
>>> pdb.run('pdb_example.Student(5).print_std()')
> <string>(1)<module>()
(Pdb)

To continue debugging, enter continue after the (Pdb) prompt and press Enter. If you want to know the options we can use in this, then after the (Pdb) prompt press the /Tab /key twice.

Now, after entering continue, we will get the output as follows:

    >>> import pdb_example
    >>> import pdb
In [15]: pdb.run("pdb_example.Student(5).print_std()"
    ...: )
> <string>(1)<module>()
(Pdb) continue
0
1
2
3
4

from src import pdb_example
import pdb
pdb.run('pdb_example.Student(5).print_std()')

From a command line

The simplest and most straightforward way to run a debugger is from a command line. Our program will act as input to the debugger. You can use the debugger from command line as follows:

$ python3 -m pdb pdb_example.py

When you run the debugger from the command line, source code will be loaded and it will stop the execution on the first line it finds. Enter continue to continue the debugging. Here's the output:

student@ubuntu:~$ python3 -m pdb pdb_example.py
> /home/student/pdb_example.py(1)<module>()
-> class Student:
(Pdb) continue
0
1
2
3
4
The program finished and will be restarted
> /home/student/pdb_example.py(1)<module>()
-> class Student:
(Pdb)

Within a Python script

The previous two techniques will start the debugger at the beginning of a Python program. But this third technique is best for long-running processes. To start the debugger within a script, use set_trace().

Now, modify your pdb_example.py file as follows:

import pdb

class Student:
    def __init__(self, std):
        self.count = std

    def print_std(self):
        for i in range(self.count):
            pdb.set_trace()
            print(i)
        return


if __name__ == "__main__":
    Student(5).print_std()

Now, run the program as follows:

student@ubuntu:~$ python3 pdb_example.py
> /home/student/pdb_example.py(10)print_std()
-> print(i)
(Pdb) continue
0
> /home/student/pdb_example.py(9)print_std()
-> pdb.set_trace()
(Pdb)

set_trace() is a Python function, therefore you can call it at any point in your program.

So, these are the three ways by which you can start a debugger.

The cProfile module

As discussed previously, profiling means measuring the execution time of a program. We are going to use the cProfile Python module for profiling a program.

Now, we will write a cprof_example.py script and write the following code in it:

mul_value = 0


def mul_numbers(num1, num2):
    mul_value = num1 * num2
    print("Local Value: ", mul_value)
    return mul_value


mul_numbers(58, 77)
print("Global Value: ", mul_value)

Run the program and you will see the output as follows:

python -m cProfile src/cprof_example.py

Local Value:  4466
Global Value:  0
         6 function calls in 0.000 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 cprof_example.py:1(<module>)
        1    0.000    0.000    0.000    0.000 cprof_example.py:4(mul_numbers)
        1    0.000    0.000    0.000    0.000 {built-in method builtins.exec}
        2    0.000    0.000    0.000    0.000 {built-in method builtins.print}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

So, using cProfile, all functions that are called will get printed with the time spent on each function. Now, we will see what these column headings mean:

1. ncalls:* *Number of calls

   System Message: WARNING/2 (<string>, line 337); backlink

   Inline emphasis start-string without end-string.

2. tottime:* *Total time spent in the given function

   System Message: WARNING/2 (<string>, line 338); backlink

   Inline emphasis start-string without end-string.

3. percall: Quotient of tottime divided by ncalls

4. cumtime: Cumulative time spent in this and all subfunctions

5. percall: Quotient of cumtime divided by primitive calls

6. filename:lineno(function): Provides the respective data of each function

timeit

timeit is a Python module used to time small parts of your Python script. You can call timeit from the command line as well as import the timeit module into your script. We are going to write a script to time a piece of code. Create a timeit_example.py script and write the following content into it:

import timeit

prg_setup = "from math import sqrt"
prg_code = """
def timeit_example():
            list1 = []
            for x in range(50):
                        list1.append(sqrt(x))
"""
# timeit statement
print(timeit.timeit(setup=prg_setup, stmt=prg_code, number=10000))

Using timeit, we can decide what piece of code we want to measure the performance of. So, we can easily define the setup code as well as the code snippet on which we want to perform the test separately. The main code runs 1 million times, which is the default time, whereas the setup code runs only once.

Questions

1. To debug a program, which module is used?

2. Check how to use ipython along with all aliases and magic functions.

3. What is *Global interpreted lock* (GIL)?

4. What is the purpose of the PYTHONSTARTUP, PYTHONCASEOK, PYTHONHOME, and PYTHONSTARTUP environment variables?

5. What is the output of the following code? a) [0], b) [1], c) [1, 0], d) [0, 1].

def foo(k):
    k = [1]
q = [0]
foo(q)
print(q)

1. Which of the following is an invalid variable? a) my_string1 b) 1st_string c) foo d) _

Answer:

1. To debug the program, the pdb module is used.

2. 1. Before running ipython3, install using sudo apt-get install ipython3. b) %lsmagic.

3. A global interpreter lock is a mechanism used in computer language interpreters to synchronize the execution of threads so that only one native thread can execute at a time

4. Following are the answers:

1. PYTHONPATH: It has a role similar to PATH. This variable tells the Python interpreter where to locate the module files imported into a program. It should include the Python source library directory and the directories containing Python source code. PYTHONPATH is sometimes preset by the Python installer.

2. PYTHONSTARTUP: It contains the path of an initialization file containing Python source code. It is executed every time you start the interpreter. It is named as .pythonrc.py in Unix and it contains commands that load utilities or modify PYTHONPATH. c) PYTHONCASEOK: It is used in Windows to instruct Python to find the first case-insensitive match in an import statement. Set this variable to any value to activate it. d) PYTHONHOME: It is an alternative module search path. It is usually embedded in the PYTHONSTARTUP or PYTHONPATH directories to make switching module libraries easy.

1. Answer: [0]. A new list object is created in the function and the reference is lost. This can be checked by comparing the ID of k before and after k = [1].

2. Answer: b. Variable names should not start with a number.

Further reading

• How to handle GIL problems in python: https://realpython.com/python-gil/

• Check how to use pdb module in command line: https://fedoramagazine.org/getting-started-python-debugger/