Parallel processing

Example of parallel runs (pool.map)

parallel calculating the squares of numbers 1, 2, 3, 4

using 4 processors

#!/usr/bin/env python

import multiprocessing

def parallel_runs():

    pool = multiprocessing.Pool(processes=4)

    input_list = [1, 2, 3, 4]

    result_list = pool.map(parfunc, input_list) # 'map' waits until all results are ready

    print(result_list)

# square-function to run in parallel 

def parfunc(x):

    return x**2

if __name__ == '__main__':

    parallel_runs()

pool.map get's as input a function and only one iterable argument; output is a list of the corresponding results. To run in parallel function with multiple arguments, partial can be used to reduce the number of arguments to the one that is replaced during parallel processing.

https://docs.python.org/3.4/library/multiprocessing.html

get number of CPU's

try:

    numProcessors = multiprocessing.cpu_count()

except NotImplementedError:   # win32 environment variable NUMBER_OF_PROCESSORS not defined

    print('Cannot detect number of CPUs')

    numProcessors = 1

Read more:

→  pool.map  -  using multiple arguments