How to use this

First, install the ipynb module, which will let you import this notebook as if it were a Python module. You need to install a specific fork of the ipynb module which ignores cell magics, because the getrusage_background_sampler_magic notebook contains several test/demo cell-magics that cause a syntax error using the official version of the ipynb module. You can do it like this:

pip install 'git+https://github.com/mister-average/ipynb.git@mister-average-ignore-cell-magic'

Also make sure that Pandas and Plotly are installed
Then, install this module, like this:

pip install 'git+https://github.com/mister-average/getrusage_background_sampler_magic.git'

Then in your notebook, import everything from the getrusage_background_sampler_magic module like this:

from ipynb.fs.defs.getrusage_background_sampler_magic import *

Next, start any cell in your notebook with this magic:

%%getrusage_in_background .03 name_of_dataframe_variable_that_will_store_samples

This will start a background process that collects samples of the entire process tree spawned by this notebook, and display a chart or two when the cell finishes. The two arguments are required:
The first floating point argument is the time in seconds between samples
The second string argument is the name you wish to give to the variable that will hold the dataframe of data samples that are collected. You can perform further analysis of that dataframe in subsequent cells of your notebook.
The first of the two charts displays normalized values, because the scale of the units involved is vastly different, so you can't really see relationships between the different metrics unless they are normalized. The second of the two charts shows the raw values.

In case your noteboook crashes, take note of the recovery file statement that is printed out just as your cell starts executing. It will look like this:

Recovery file path: /some/path
After a crash, run recover_samples_list('/some/path')

After restarting your notebook and re-running the import statement above, you can execute that function call to display charts of the samples that were collected up to the point of your notebook crash. That function call will also return the dataframe of samples.

Tests/demos

Basic test

%%getrusage_in_background .1 basic_test_samples_df
time.sleep(3)

basic_test_samples_df

	timestamp	pid_count	user	system	read_count	write_count	read_bytes	write_bytes	rss	vms	ctx_switches_voluntary	ctx_switches_involuntary	num_fds
0	2022-09-10 18:34:10.462739	1	NaN	NaN	NaN	NaN	NaN	NaN	144809984	2893950976	NaN	NaN	62
1	2022-09-10 18:34:10.565919	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
2	2022-09-10 18:34:10.670812	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
3	2022-09-10 18:34:10.774538	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
4	2022-09-10 18:34:10.878595	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
5	2022-09-10 18:34:10.984116	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
6	2022-09-10 18:34:11.087483	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
7	2022-09-10 18:34:11.191110	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
8	2022-09-10 18:34:11.295672	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
9	2022-09-10 18:34:11.400225	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
10	2022-09-10 18:34:11.505663	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
11	2022-09-10 18:34:11.610211	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
12	2022-09-10 18:34:11.714722	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
13	2022-09-10 18:34:11.819205	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
14	2022-09-10 18:34:11.923773	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
15	2022-09-10 18:34:12.029369	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
16	2022-09-10 18:34:12.132891	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
17	2022-09-10 18:34:12.234581	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
18	2022-09-10 18:34:12.338059	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
19	2022-09-10 18:34:12.443363	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
20	2022-09-10 18:34:12.547520	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
21	2022-09-10 18:34:12.651517	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
22	2022-09-10 18:34:12.755440	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
23	2022-09-10 18:34:12.859037	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
24	2022-09-10 18:34:12.964698	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
25	2022-09-10 18:34:13.070325	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
26	2022-09-10 18:34:13.176038	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
27	2022-09-10 18:34:13.279967	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62
28	2022-09-10 18:34:13.381889	1	0.0	0.0	0.0	0.0	0.0	0.0	144809984	2893950976	0.0	0.0	62

CPU test

%%getrusage_in_background .1 cpu_test_samples_df
for i in range(3):
    time.sleep(1)
    
    for x in range(10_000_000):
        pass

Memory test

%%getrusage_in_background .01 memory_test_samples_df
my_list = [ 'a' ]
for x in range(10_000_000):
    my_list += ['a']

memory_test_samples_df

	timestamp	pid_count	user	system	read_count	write_count	read_bytes	write_bytes	rss	vms	ctx_switches_voluntary	ctx_switches_involuntary	num_fds
0	2022-09-10 18:35:38.719903	1	NaN	NaN	NaN	NaN	NaN	NaN	171651072	2985512960	NaN	NaN	69
1	2022-09-10 18:35:38.732107	1	0.01	0.0	2.0	2.0	16.0	16.0	171651072	2985512960	2.0	0.0	69
2	2022-09-10 18:35:38.743895	1	0.01	0.0	2.0	2.0	16.0	16.0	171651072	2985512960	3.0	0.0	69
3	2022-09-10 18:35:38.755600	1	0.01	0.0	2.0	3.0	16.0	24.0	171651072	2985512960	4.0	1.0	69
4	2022-09-10 18:35:38.767239	1	0.01	0.0	2.0	2.0	16.0	16.0	171651072	2985512960	3.0	0.0	69
...	...	...	...	...	...	...	...	...	...	...	...	...	...
109	2022-09-10 18:35:39.960594	1	0.01	0.0	0.0	0.0	0.0	0.0	251252736	3068293120	0.0	0.0	69
110	2022-09-10 18:35:39.971837	1	0.01	0.0	0.0	0.0	0.0	0.0	252063744	3068293120	0.0	0.0	69
111	2022-09-10 18:35:39.983148	1	0.01	0.0	0.0	0.0	0.0	0.0	252874752	3068293120	0.0	0.0	69
112	2022-09-10 18:35:39.994462	1	0.01	0.0	0.0	0.0	0.0	0.0	253415424	3068293120	0.0	0.0	69
113	2022-09-10 18:35:40.005749	1	0.01	0.0	0.0	0.0	0.0	0.0	254226432	3078193152	0.0	0.0	69

114 rows × 13 columns

Read test

%%getrusage_in_background .1 read_test_samples_df
f = open("/dev/zero", "rb")
bytes_read = 0
for i in range(3):
    time.sleep(1)
    
    for x in range(1_000_000):
        bytes_read += len(f.read(10_000))

z = 1/0

print(f"{bytes_read=}")

Write test

%%getrusage_in_background .1 write_test_samples_df
f = open("/dev/null", "w")
my_string = 'a' * 10_000
bytes_written = 0
for i in range(3):
    time.sleep(1)
    
    for x in range(1_000_000):
        bytes_written += f.write(my_string)
        
    print(f"{bytes_written=}")

Child process test

%%getrusage_in_background .1 child_process_test_samples_df
import multiprocessing
import time
import random
import sys

#
# Functions used by test code
#

def calculate(func, args):
    result = func(*args)
    return '%s says that %s%s = %s' % (
        multiprocessing.current_process().name,
        func.__name__, args, result
        )

def calculatestar(args):
    return calculate(*args)

def mul(a, b):
    time.sleep(0.5 * random.random())
    return a * b

def plus(a, b):
    time.sleep(0.5 * random.random())
    return a + b

def f(x):
    return 1.0 / (x - 5.0)

def pow3(x):
    return x ** 3

def noop(x):
    pass

#
# Test code
#

def test(PROCESSES = 4):
    print('Creating pool with %d processes\n' % PROCESSES)

    with multiprocessing.Pool(PROCESSES) as pool:
        #
        # Tests
        #

        TASKS = [(mul, (i, 7)) for i in range(10)] + \
                [(plus, (i, 8)) for i in range(10)]

        results = [pool.apply_async(calculate, t) for t in TASKS]
        imap_it = pool.imap(calculatestar, TASKS)
        imap_unordered_it = pool.imap_unordered(calculatestar, TASKS)

        print('Ordered results using pool.apply_async():')
        for r in results:
            print('\t', r.get())
        print()

        print('Ordered results using pool.imap():')
        for x in imap_it:
            print('\t', x)
        print()

        print('Unordered results using pool.imap_unordered():')
        for x in imap_unordered_it:
            print('\t', x)
        print()

        print('Ordered results using pool.map() --- will block till complete:')
        for x in pool.map(calculatestar, TASKS):
            print('\t', x)
        print()

        #
        # Test error handling
        #

        print('Testing error handling:')

        try:
            print(pool.apply(f, (5,)))
        except ZeroDivisionError:
            print('\tGot ZeroDivisionError as expected from pool.apply()')
        else:
            raise AssertionError('expected ZeroDivisionError')

        try:
            print(pool.map(f, list(range(10))))
        except ZeroDivisionError:
            print('\tGot ZeroDivisionError as expected from pool.map()')
        else:
            raise AssertionError('expected ZeroDivisionError')

        try:
            print(list(pool.imap(f, list(range(10)))))
        except ZeroDivisionError:
            print('\tGot ZeroDivisionError as expected from list(pool.imap())')
        else:
            raise AssertionError('expected ZeroDivisionError')

        it = pool.imap(f, list(range(10)))
        for i in range(10):
            try:
                x = next(it)
            except ZeroDivisionError:
                if i == 5:
                    pass
            except StopIteration:
                break
            else:
                if i == 5:
                    raise AssertionError('expected ZeroDivisionError')

        assert i == 9
        print('\tGot ZeroDivisionError as expected from IMapIterator.next()')
        print()

        #
        # Testing timeouts
        #

        print('Testing ApplyResult.get() with timeout:', end=' ')
        res = pool.apply_async(calculate, TASKS[0])
        while 1:
            sys.stdout.flush()
            try:
                sys.stdout.write('\n\t%s' % res.get(0.02))
                break
            except multiprocessing.TimeoutError:
                sys.stdout.write('.')
        print()
        print()

        print('Testing IMapIterator.next() with timeout:', end=' ')
        it = pool.imap(calculatestar, TASKS)
        while 1:
            sys.stdout.flush()
            try:
                sys.stdout.write('\n\t%s' % it.next(0.02))
            except StopIteration:
                break
            except multiprocessing.TimeoutError:
                sys.stdout.write('.')
        print()
        print()

multiprocessing.freeze_support()
test(1)
time.sleep(1)
test(4)
time.sleep(1)
test(8)

Creating pool with 1 processes

Recovery file path:
After a crash, run recover_samples_list('/tmp/tmpjgub1r4u') /tmp/tmpjgub1r4u 
Ordered results using pool.apply_async():
	 ForkPoolWorker-34 says that mul(0, 7) = 0
	 ForkPoolWorker-34 says that mul(1, 7) = 7
	 ForkPoolWorker-34 says that mul(2, 7) = 14
	 ForkPoolWorker-34 says that mul(3, 7) = 21
	 ForkPoolWorker-34 says that mul(4, 7) = 28
	 ForkPoolWorker-34 says that mul(5, 7) = 35
	 ForkPoolWorker-34 says that mul(6, 7) = 42
	 ForkPoolWorker-34 says that mul(7, 7) = 49
	 ForkPoolWorker-34 says that mul(8, 7) = 56
	 ForkPoolWorker-34 says that mul(9, 7) = 63
	 ForkPoolWorker-34 says that plus(0, 8) = 8
	 ForkPoolWorker-34 says that plus(1, 8) = 9
	 ForkPoolWorker-34 says that plus(2, 8) = 10
	 ForkPoolWorker-34 says that plus(3, 8) = 11
	 ForkPoolWorker-34 says that plus(4, 8) = 12
	 ForkPoolWorker-34 says that plus(5, 8) = 13
	 ForkPoolWorker-34 says that plus(6, 8) = 14
	 ForkPoolWorker-34 says that plus(7, 8) = 15
	 ForkPoolWorker-34 says that plus(8, 8) = 16
	 ForkPoolWorker-34 says that plus(9, 8) = 17

Ordered results using pool.imap():
	 ForkPoolWorker-34 says that mul(0, 7) = 0
	 ForkPoolWorker-34 says that mul(1, 7) = 7
	 ForkPoolWorker-34 says that mul(2, 7) = 14
	 ForkPoolWorker-34 says that mul(3, 7) = 21
	 ForkPoolWorker-34 says that mul(4, 7) = 28
	 ForkPoolWorker-34 says that mul(5, 7) = 35
	 ForkPoolWorker-34 says that mul(6, 7) = 42
	 ForkPoolWorker-34 says that mul(7, 7) = 49
	 ForkPoolWorker-34 says that mul(8, 7) = 56
	 ForkPoolWorker-34 says that mul(9, 7) = 63
	 ForkPoolWorker-34 says that plus(0, 8) = 8
	 ForkPoolWorker-34 says that plus(1, 8) = 9
	 ForkPoolWorker-34 says that plus(2, 8) = 10
	 ForkPoolWorker-34 says that plus(3, 8) = 11
	 ForkPoolWorker-34 says that plus(4, 8) = 12
	 ForkPoolWorker-34 says that plus(5, 8) = 13
	 ForkPoolWorker-34 says that plus(6, 8) = 14
	 ForkPoolWorker-34 says that plus(7, 8) = 15
	 ForkPoolWorker-34 says that plus(8, 8) = 16
	 ForkPoolWorker-34 says that plus(9, 8) = 17

Unordered results using pool.imap_unordered():
	 ForkPoolWorker-34 says that mul(0, 7) = 0
	 ForkPoolWorker-34 says that mul(1, 7) = 7
	 ForkPoolWorker-34 says that mul(2, 7) = 14
	 ForkPoolWorker-34 says that mul(3, 7) = 21
	 ForkPoolWorker-34 says that mul(4, 7) = 28
	 ForkPoolWorker-34 says that mul(5, 7) = 35
	 ForkPoolWorker-34 says that mul(6, 7) = 42
	 ForkPoolWorker-34 says that mul(7, 7) = 49
	 ForkPoolWorker-34 says that mul(8, 7) = 56
	 ForkPoolWorker-34 says that mul(9, 7) = 63
	 ForkPoolWorker-34 says that plus(0, 8) = 8
	 ForkPoolWorker-34 says that plus(1, 8) = 9
	 ForkPoolWorker-34 says that plus(2, 8) = 10
	 ForkPoolWorker-34 says that plus(3, 8) = 11
	 ForkPoolWorker-34 says that plus(4, 8) = 12
	 ForkPoolWorker-34 says that plus(5, 8) = 13
	 ForkPoolWorker-34 says that plus(6, 8) = 14
	 ForkPoolWorker-34 says that plus(7, 8) = 15
	 ForkPoolWorker-34 says that plus(8, 8) = 16
	 ForkPoolWorker-34 says that plus(9, 8) = 17

Ordered results using pool.map() --- will block till complete:
	 ForkPoolWorker-34 says that mul(0, 7) = 0
	 ForkPoolWorker-34 says that mul(1, 7) = 7
	 ForkPoolWorker-34 says that mul(2, 7) = 14
	 ForkPoolWorker-34 says that mul(3, 7) = 21
	 ForkPoolWorker-34 says that mul(4, 7) = 28
	 ForkPoolWorker-34 says that mul(5, 7) = 35
	 ForkPoolWorker-34 says that mul(6, 7) = 42
	 ForkPoolWorker-34 says that mul(7, 7) = 49
	 ForkPoolWorker-34 says that mul(8, 7) = 56
	 ForkPoolWorker-34 says that mul(9, 7) = 63
	 ForkPoolWorker-34 says that plus(0, 8) = 8
	 ForkPoolWorker-34 says that plus(1, 8) = 9
	 ForkPoolWorker-34 says that plus(2, 8) = 10
	 ForkPoolWorker-34 says that plus(3, 8) = 11
	 ForkPoolWorker-34 says that plus(4, 8) = 12
	 ForkPoolWorker-34 says that plus(5, 8) = 13
	 ForkPoolWorker-34 says that plus(6, 8) = 14
	 ForkPoolWorker-34 says that plus(7, 8) = 15
	 ForkPoolWorker-34 says that plus(8, 8) = 16
	 ForkPoolWorker-34 says that plus(9, 8) = 17

Testing error handling:
	Got ZeroDivisionError as expected from pool.apply()
	Got ZeroDivisionError as expected from pool.map()
	Got ZeroDivisionError as expected from list(pool.imap())
	Got ZeroDivisionError as expected from IMapIterator.next()

Testing ApplyResult.get() with timeout: ......
	ForkPoolWorker-34 says that mul(0, 7) = 0

Testing IMapIterator.next() with timeout: ...........
	ForkPoolWorker-34 says that mul(0, 7) = 0..............
	ForkPoolWorker-34 says that mul(1, 7) = 7...
	ForkPoolWorker-34 says that mul(2, 7) = 14.....................
	ForkPoolWorker-34 says that mul(3, 7) = 21..................
	ForkPoolWorker-34 says that mul(4, 7) = 28....................
	ForkPoolWorker-34 says that mul(5, 7) = 35........
	ForkPoolWorker-34 says that mul(6, 7) = 42...........
	ForkPoolWorker-34 says that mul(7, 7) = 49...
	ForkPoolWorker-34 says that mul(8, 7) = 56..............
	ForkPoolWorker-34 says that mul(9, 7) = 63
	ForkPoolWorker-34 says that plus(0, 8) = 8..................
	ForkPoolWorker-34 says that plus(1, 8) = 9..
	ForkPoolWorker-34 says that plus(2, 8) = 10.............
	ForkPoolWorker-34 says that plus(3, 8) = 11.....................
	ForkPoolWorker-34 says that plus(4, 8) = 12......
	ForkPoolWorker-34 says that plus(5, 8) = 13.............
	ForkPoolWorker-34 says that plus(6, 8) = 14......................
	ForkPoolWorker-34 says that plus(7, 8) = 15..................
	ForkPoolWorker-34 says that plus(8, 8) = 16..........
	ForkPoolWorker-34 says that plus(9, 8) = 17

Creating pool with 4 processes

Ordered results using pool.apply_async():
	 ForkPoolWorker-35 says that mul(0, 7) = 0
	 ForkPoolWorker-36 says that mul(1, 7) = 7
	 ForkPoolWorker-37 says that mul(2, 7) = 14
	 ForkPoolWorker-38 says that mul(3, 7) = 21
	 ForkPoolWorker-36 says that mul(4, 7) = 28
	 ForkPoolWorker-36 says that mul(5, 7) = 35
	 ForkPoolWorker-38 says that mul(6, 7) = 42
	 ForkPoolWorker-35 says that mul(7, 7) = 49
	 ForkPoolWorker-37 says that mul(8, 7) = 56
	 ForkPoolWorker-37 says that mul(9, 7) = 63
	 ForkPoolWorker-36 says that plus(0, 8) = 8
	 ForkPoolWorker-38 says that plus(1, 8) = 9
	 ForkPoolWorker-38 says that plus(2, 8) = 10
	 ForkPoolWorker-37 says that plus(3, 8) = 11
	 ForkPoolWorker-37 says that plus(4, 8) = 12
	 ForkPoolWorker-35 says that plus(5, 8) = 13
	 ForkPoolWorker-36 says that plus(6, 8) = 14
	 ForkPoolWorker-37 says that plus(7, 8) = 15
	 ForkPoolWorker-38 says that plus(8, 8) = 16
	 ForkPoolWorker-37 says that plus(9, 8) = 17

Ordered results using pool.imap():
	 ForkPoolWorker-37 says that mul(0, 7) = 0
	 ForkPoolWorker-35 says that mul(1, 7) = 7
	 ForkPoolWorker-36 says that mul(2, 7) = 14
	 ForkPoolWorker-35 says that mul(3, 7) = 21
	 ForkPoolWorker-38 says that mul(4, 7) = 28
	 ForkPoolWorker-38 says that mul(5, 7) = 35
	 ForkPoolWorker-37 says that mul(6, 7) = 42
	 ForkPoolWorker-36 says that mul(7, 7) = 49
	 ForkPoolWorker-37 says that mul(8, 7) = 56
	 ForkPoolWorker-38 says that mul(9, 7) = 63
	 ForkPoolWorker-35 says that plus(0, 8) = 8
	 ForkPoolWorker-37 says that plus(1, 8) = 9
	 ForkPoolWorker-36 says that plus(2, 8) = 10
	 ForkPoolWorker-36 says that plus(3, 8) = 11
	 ForkPoolWorker-37 says that plus(4, 8) = 12
	 ForkPoolWorker-38 says that plus(5, 8) = 13
	 ForkPoolWorker-35 says that plus(6, 8) = 14
	 ForkPoolWorker-38 says that plus(7, 8) = 15
	 ForkPoolWorker-36 says that plus(8, 8) = 16
	 ForkPoolWorker-37 says that plus(9, 8) = 17

Unordered results using pool.imap_unordered():
	 ForkPoolWorker-35 says that mul(0, 7) = 0
	 ForkPoolWorker-36 says that mul(2, 7) = 14
	 ForkPoolWorker-38 says that mul(3, 7) = 21
	 ForkPoolWorker-35 says that mul(1, 7) = 7
	 ForkPoolWorker-37 says that mul(4, 7) = 28
	 ForkPoolWorker-37 says that mul(8, 7) = 56
	 ForkPoolWorker-36 says that mul(5, 7) = 35
	 ForkPoolWorker-35 says that mul(7, 7) = 49
	 ForkPoolWorker-35 says that plus(1, 8) = 9
	 ForkPoolWorker-38 says that mul(6, 7) = 42
	 ForkPoolWorker-37 says that mul(9, 7) = 63
	 ForkPoolWorker-35 says that plus(2, 8) = 10
	 ForkPoolWorker-38 says that plus(3, 8) = 11
	 ForkPoolWorker-37 says that plus(4, 8) = 12
	 ForkPoolWorker-38 says that plus(6, 8) = 14
	 ForkPoolWorker-35 says that plus(5, 8) = 13
	 ForkPoolWorker-36 says that plus(0, 8) = 8
	 ForkPoolWorker-37 says that plus(7, 8) = 15
	 ForkPoolWorker-38 says that plus(8, 8) = 16

	 ForkPoolWorker-35 says that plus(9, 8) = 17

Ordered results using pool.map() --- will block till complete:
	 ForkPoolWorker-36 says that mul(0, 7) = 0
	 ForkPoolWorker-36 says that mul(1, 7) = 7
	 ForkPoolWorker-37 says that mul(2, 7) = 14
	 ForkPoolWorker-37 says that mul(3, 7) = 21
	 ForkPoolWorker-38 says that mul(4, 7) = 28
	 ForkPoolWorker-38 says that mul(5, 7) = 35
	 ForkPoolWorker-35 says that mul(6, 7) = 42
	 ForkPoolWorker-35 says that mul(7, 7) = 49
	 ForkPoolWorker-38 says that mul(8, 7) = 56
	 ForkPoolWorker-38 says that mul(9, 7) = 63
	 ForkPoolWorker-36 says that plus(0, 8) = 8
	 ForkPoolWorker-36 says that plus(1, 8) = 9
	 ForkPoolWorker-35 says that plus(2, 8) = 10
	 ForkPoolWorker-35 says that plus(3, 8) = 11
	 ForkPoolWorker-37 says that plus(4, 8) = 12
	 ForkPoolWorker-37 says that plus(5, 8) = 13
	 ForkPoolWorker-35 says that plus(6, 8) = 14
	 ForkPoolWorker-35 says that plus(7, 8) = 15
	 ForkPoolWorker-36 says that plus(8, 8) = 16
	 ForkPoolWorker-36 says that plus(9, 8) = 17

Testing error handling:
	Got ZeroDivisionError as expected from pool.apply()
	Got ZeroDivisionError as expected from pool.map()
	Got ZeroDivisionError as expected from list(pool.imap())
	Got ZeroDivisionError as expected from IMapIterator.next()

Testing ApplyResult.get() with timeout: ..............
	ForkPoolWorker-38 says that mul(0, 7) = 0

Testing IMapIterator.next() with timeout: .....
	ForkPoolWorker-37 says that mul(0, 7) = 0
	ForkPoolWorker-36 says that mul(1, 7) = 7
	ForkPoolWorker-35 says that mul(2, 7) = 14
	ForkPoolWorker-38 says that mul(3, 7) = 21
	ForkPoolWorker-35 says that mul(4, 7) = 28..............
	ForkPoolWorker-38 says that mul(5, 7) = 35.
	ForkPoolWorker-36 says that mul(6, 7) = 42.....
	ForkPoolWorker-35 says that mul(7, 7) = 49
	ForkPoolWorker-37 says that mul(8, 7) = 56.............
	ForkPoolWorker-38 says that mul(9, 7) = 63
	ForkPoolWorker-37 says that plus(0, 8) = 8
	ForkPoolWorker-36 says that plus(1, 8) = 9
	ForkPoolWorker-35 says that plus(2, 8) = 10.............
	ForkPoolWorker-37 says that plus(3, 8) = 11
	ForkPoolWorker-36 says that plus(4, 8) = 12..
	ForkPoolWorker-35 says that plus(5, 8) = 13
	ForkPoolWorker-38 says that plus(6, 8) = 14......
	ForkPoolWorker-36 says that plus(7, 8) = 15......
	ForkPoolWorker-38 says that plus(8, 8) = 16...
	ForkPoolWorker-37 says that plus(9, 8) = 17

Creating pool with 8 processes

Ordered results using pool.apply_async():
	 ForkPoolWorker-39 says that mul(0, 7) = 0
	 ForkPoolWorker-40 says that mul(1, 7) = 7
	 ForkPoolWorker-41 says that mul(2, 7) = 14
	 ForkPoolWorker-42 says that mul(3, 7) = 21
	 ForkPoolWorker-43 says that mul(4, 7) = 28
	 ForkPoolWorker-44 says that mul(5, 7) = 35
	 ForkPoolWorker-45 says that mul(6, 7) = 42
	 ForkPoolWorker-46 says that mul(7, 7) = 49
	 ForkPoolWorker-40 says that mul(8, 7) = 56
	 ForkPoolWorker-40 says that mul(9, 7) = 63
	 ForkPoolWorker-41 says that plus(0, 8) = 8
	 ForkPoolWorker-46 says that plus(1, 8) = 9
	 ForkPoolWorker-44 says that plus(2, 8) = 10
	 ForkPoolWorker-42 says that plus(3, 8) = 11
	 ForkPoolWorker-44 says that plus(4, 8) = 12
	 ForkPoolWorker-45 says that plus(5, 8) = 13
	 ForkPoolWorker-39 says that plus(6, 8) = 14
	 ForkPoolWorker-40 says that plus(7, 8) = 15
	 ForkPoolWorker-46 says that plus(8, 8) = 16
	 ForkPoolWorker-42 says that plus(9, 8) = 17

Ordered results using pool.imap():
	 ForkPoolWorker-40 says that mul(0, 7) = 0
	 ForkPoolWorker-44 says that mul(1, 7) = 7
	 ForkPoolWorker-43 says that mul(2, 7) = 14
	 ForkPoolWorker-45 says that mul(3, 7) = 21
	 ForkPoolWorker-39 says that mul(4, 7) = 28
	 ForkPoolWorker-44 says that mul(5, 7) = 35
	 ForkPoolWorker-46 says that mul(6, 7) = 42
	 ForkPoolWorker-41 says that mul(7, 7) = 49
	 ForkPoolWorker-45 says that mul(8, 7) = 56
	 ForkPoolWorker-39 says that mul(9, 7) = 63
	 ForkPoolWorker-42 says that plus(0, 8) = 8
	 ForkPoolWorker-45 says that plus(1, 8) = 9
	 ForkPoolWorker-40 says that plus(2, 8) = 10
	 ForkPoolWorker-43 says that plus(3, 8) = 11
	 ForkPoolWorker-42 says that plus(4, 8) = 12
	 ForkPoolWorker-41 says that plus(5, 8) = 13
	 ForkPoolWorker-43 says that plus(6, 8) = 14
	 ForkPoolWorker-41 says that plus(7, 8) = 15
	 ForkPoolWorker-45 says that plus(8, 8) = 16
	 ForkPoolWorker-44 says that plus(9, 8) = 17

Unordered results using pool.imap_unordered():
	 ForkPoolWorker-39 says that mul(0, 7) = 0
	 ForkPoolWorker-41 says that mul(2, 7) = 14
	 ForkPoolWorker-42 says that mul(1, 7) = 7
	 ForkPoolWorker-41 says that mul(6, 7) = 42
	 ForkPoolWorker-40 says that mul(7, 7) = 49
	 ForkPoolWorker-43 says that mul(3, 7) = 21
	 ForkPoolWorker-45 says that plus(3, 8) = 11
	 ForkPoolWorker-39 says that mul(4, 7) = 28
	 ForkPoolWorker-45 says that plus(4, 8) = 12
	 ForkPoolWorker-44 says that plus(2, 8) = 10
	 ForkPoolWorker-46 says that mul(5, 7) = 35
	 ForkPoolWorker-43 says that plus(1, 8) = 9
	 ForkPoolWorker-40 says that plus(0, 8) = 8
	 ForkPoolWorker-39 says that plus(5, 8) = 13
	 ForkPoolWorker-41 says that mul(9, 7) = 63
	 ForkPoolWorker-42 says that mul(8, 7) = 56
	 ForkPoolWorker-45 says that plus(6, 8) = 14
	 ForkPoolWorker-43 says that plus(9, 8) = 17
	 ForkPoolWorker-46 says that plus(8, 8) = 16
	 ForkPoolWorker-44 says that plus(7, 8) = 15

Ordered results using pool.map() --- will block till complete:
	 ForkPoolWorker-40 says that mul(0, 7) = 0
	 ForkPoolWorker-39 says that mul(1, 7) = 7
	 ForkPoolWorker-41 says that mul(2, 7) = 14
	 ForkPoolWorker-42 says that mul(3, 7) = 21
	 ForkPoolWorker-45 says that mul(4, 7) = 28
	 ForkPoolWorker-43 says that mul(5, 7) = 35
	 ForkPoolWorker-46 says that mul(6, 7) = 42
	 ForkPoolWorker-44 says that mul(7, 7) = 49
	 ForkPoolWorker-45 says that mul(8, 7) = 56
	 ForkPoolWorker-40 says that mul(9, 7) = 63
	 ForkPoolWorker-39 says that plus(0, 8) = 8
	 ForkPoolWorker-39 says that plus(1, 8) = 9
	 ForkPoolWorker-44 says that plus(2, 8) = 10
	 ForkPoolWorker-46 says that plus(3, 8) = 11
	 ForkPoolWorker-42 says that plus(4, 8) = 12
	 ForkPoolWorker-43 says that plus(5, 8) = 13
	 ForkPoolWorker-46 says that plus(6, 8) = 14
	 ForkPoolWorker-39 says that plus(7, 8) = 15
	 ForkPoolWorker-41 says that plus(8, 8) = 16
	 ForkPoolWorker-39 says that plus(9, 8) = 17

Testing error handling:
	Got ZeroDivisionError as expected from pool.apply()
	Got ZeroDivisionError as expected from pool.map()
	Got ZeroDivisionError as expected from list(pool.imap())
	Got ZeroDivisionError as expected from IMapIterator.next()

Testing ApplyResult.get() with timeout: ......................
	ForkPoolWorker-42 says that mul(0, 7) = 0

Testing IMapIterator.next() with timeout: .
	ForkPoolWorker-46 says that mul(0, 7) = 0..
	ForkPoolWorker-43 says that mul(1, 7) = 7
	ForkPoolWorker-39 says that mul(2, 7) = 14
	ForkPoolWorker-45 says that mul(3, 7) = 21..........
	ForkPoolWorker-44 says that mul(4, 7) = 28
	ForkPoolWorker-41 says that mul(5, 7) = 35....
	ForkPoolWorker-40 says that mul(6, 7) = 42
	ForkPoolWorker-42 says that mul(7, 7) = 49
	ForkPoolWorker-39 says that mul(8, 7) = 56
	ForkPoolWorker-39 says that mul(9, 7) = 63....
	ForkPoolWorker-46 says that plus(0, 8) = 8
	ForkPoolWorker-42 says that plus(1, 8) = 9
	ForkPoolWorker-43 says that plus(2, 8) = 10
	ForkPoolWorker-45 says that plus(3, 8) = 11
	ForkPoolWorker-43 says that plus(4, 8) = 12
	ForkPoolWorker-41 says that plus(5, 8) = 13
	ForkPoolWorker-43 says that plus(6, 8) = 14
	ForkPoolWorker-39 says that plus(7, 8) = 15....
	ForkPoolWorker-44 says that plus(8, 8) = 16
	ForkPoolWorker-41 says that plus(9, 8) = 17

Pandas test

import pandas as pd

%%getrusage_in_background 1 pandas_test_samples_df
with open('/super/giant/insane/large/file.csv') as text_file:
    external_disks_passports = pd.DataFrame([ line.split() for line in text_file ]).fillna('')

recover_samples_list('/tmp/tmp3j0vgpg8')

Name		Name	Last commit message	Last commit date
Latest commit History 27 Commits
README_images		README_images
LICENSE		LICENSE
README.md		README.md
getrusage_background_sampler_magic.ipynb		getrusage_background_sampler_magic.ipynb
setup.py		setup.py

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

How to use this

Tests/demos

Basic test

CPU test

Memory test

Read test

Write test

Child process test

Pandas test

About

Releases

Packages

Languages

License

mister-average/getrusage_background_sampler_magic

Folders and files

Latest commit

History

Repository files navigation

How to use this

Tests/demos

Basic test

CPU test

Memory test

Read test

Write test

Child process test

Pandas test

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages