AttributeError: Can't pickle local object 'main.<locals>.<lambda>' #21

singlelionwith2cubs · 2021-06-19T17:06:08Z

Hi,

Trying to replicate GNN-experiment on Host OS-MacOS BigSur v11.3.1 with below python packages.
Please Suggest.

(pyg) Nitins-MacBook-Air:gnn-based-experiments nitinnanda$ python list.py 
['certifi==2021.5.30', 'chardet==4.0.0', 'decorator==4.4.2', 'dill==0.3.4', 'googledrivedownloader==0.4', 'idna==2.10', 'isodate==0.6.0', 'jinja2==3.0.1', 'joblib==1.0.1', 'littleutils==0.2.2', 'markupsafe==2.0.1', 'networkx==2.5.1', 'numpy==1.20.3', 'ogb==1.3.1', 'outdated==0.2.1', 'pandas==1.2.4', 'pillow==8.2.0', 'pip==21.1.2', 'pyparsing==2.4.7', 'python-dateutil==2.8.1', 'python-louvain==0.15', 'pytz==2021.1', 'rdflib==5.0.0', 'requests==2.25.1', 'scikit-learn==0.24.2', 'scipy==1.6.3', 'setuptools==52.0.0.post20210125', 'six==1.16.0', 'threadpoolctl==2.1.0', 'torch-cluster==1.5.9', 'torch-geometric==1.7.1', 'torch-scatter==2.0.7', 'torch-sparse==0.6.10', 'torch-spline-conv==1.2.1', 'torch-summary==1.4.5', 'torch==1.9.0', 'torchsummary==1.5.1', 'torchvision==0.10.0', 'tqdm==4.61.1', 'typing-extensions==3.10.0.0', 'urllib3==1.26.5', 'wheel==0.36.2']
(pyg) Nitins-MacBook-Air:gnn-based-experiments nitinnanda$

(pyg) Nitins-MacBook-Air:gnn-based-experiments nitinnanda$ ./run.sh 
PYTHONPATH: :/Users/nitinnanda/Project_CodeNet/model-experiments/gnn-based-experiments
started experiments

Copyright (c) 2004-2016 California Institute of Technology.
Copyright (c) 2016-2021 The Uncertainty Quantification Foundation.
All rights reserved.

This software is available subject to the conditions and terms laid
out below. By downloading and using this software you are agreeing
to the following conditions.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met::

    - Redistribution of source code must retain the above copyright
      notice, this list of conditions and the following disclaimer.

    - Redistribution in binary form must reproduce the above copyright
      notice, this list of conditions and the following disclaimer in the
      documentations and/or other materials provided with the distribution.

    - Neither the names of the copyright holders nor the names of any of
      the contributors may be used to endorse or promote products derived
      from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


saved args: Namespace(batch_size=80, checkpoint='', checkpointing=1, clip=0.25, dataset='python1k', device=0, dir_data='/Users/nitinnanda/Project_CodeNet/model-experiments/gnn-based-experiments/data', dir_results='/Users/nitinnanda/Project_CodeNet/model-experiments/gnn-based-experiments/results', dir_save='/Users/nitinnanda/Project_CodeNet/model-experiments/gnn-based-experiments/saved_models', drop_ratio=0.0, emb_dim=300, epochs=1000, feat_nums='', filename='gcn_lr1e-3', gnn='gcn', lr=0.001, num_layer=5, num_workers=80, patience=20.0, runs=10)
Data loading done!
/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 80 worker processes in total. Our suggested max number of worker in current system is 4 (`cpuset` is not taken into account), which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
Let's use 0 GPUs! -- DataParallel running also on CPU only
=====Run 1, Epoch 1
Iteration:   0%|                                                                                                                        | 0/8 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "main.py", line 321, in <module>
    main()
  File "main.py", line 246, in main
    loss, train_perf = train(model, device, train_loader, optimizer, args, evaluator)
  File "main.py", line 35, in train
    for step, batch in enumerate(tqdm(loader, desc="Iteration")):
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/site-packages/tqdm/std.py", line 1178, in __iter__
    for obj in iterable:
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359, in __iter__
    return self._get_iterator()
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 918, in __init__
    w.start()
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'main.<locals>.<lambda>'
(pyg) Nitins-MacBook-Air:gnn-based-experiments nitinnanda$ 
```

The text was updated successfully, but these errors were encountered:

vthost · 2021-06-19T21:33:23Z

Hi.
Did you change any parameters in run.sh? It seems like this since the very first warning is new to me. If so, could you please share which ones you used? It complains about the number of workers and the error is thrown at multiprocessing, so I would suggest to use a smaller number as suggested in the warning.

singlelionwith2cubs · 2021-06-20T16:09:31Z

Tried with BATCHSIZE=4

(pyg) Nitins-MacBook-Air:gnn-based-experiments nitinnanda$ ./run.sh 
PYTHONPATH: :/Users/nitinnanda/Project_CodeNet/model-experiments/gnn-based-experiments
started experiments

Copyright (c) 2004-2016 California Institute of Technology.
Copyright (c) 2016-2021 The Uncertainty Quantification Foundation.
All rights reserved.

This software is available subject to the conditions and terms laid
out below. By downloading and using this software you are agreeing
to the following conditions.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met::

    - Redistribution of source code must retain the above copyright
      notice, this list of conditions and the following disclaimer.

    - Redistribution in binary form must reproduce the above copyright
      notice, this list of conditions and the following disclaimer in the
      documentations and/or other materials provided with the distribution.

    - Neither the names of the copyright holders nor the names of any of
      the contributors may be used to endorse or promote products derived
      from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


saved args: Namespace(batch_size=4, checkpoint='', checkpointing=1, clip=0.25, dataset='python1k', device=0, dir_data='/Users/nitinnanda/Project_CodeNet/model-experiments/gnn-based-experiments/data', dir_results='/Users/nitinnanda/Project_CodeNet/model-experiments/gnn-based-experiments/results', dir_save='/Users/nitinnanda/Project_CodeNet/model-experiments/gnn-based-experiments/saved_models', drop_ratio=0.0, emb_dim=300, epochs=1000, feat_nums='', filename='gcn_lr1e-3', gnn='gcn', lr=0.001, num_layer=5, num_workers=4, patience=20.0, runs=10)
Data loading done!
Let's use 0 GPUs! -- DataParallel running also on CPU only
=====Run 1, Epoch 1
Iteration:   0%|                                                                                                                      | 0/150 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "main.py", line 321, in <module>
    main()
  File "main.py", line 246, in main
    loss, train_perf = train(model, device, train_loader, optimizer, args, evaluator)
  File "main.py", line 35, in train
    for step, batch in enumerate(tqdm(loader, desc="Iteration")):
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/site-packages/tqdm/std.py", line 1178, in __iter__
    for obj in iterable:
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359, in __iter__
    return self._get_iterator()
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 918, in __init__
    w.start()
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/Caskroom/miniconda/base/envs/pyg/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'main.<locals>.<lambda>'
(pyg) Nitins-MacBook-Air:gnn-based-experiments nitinnanda$

Below is run.sh

(pyg) Nitins-MacBook-Air:gnn-based-experiments nitinnanda$ cat run.sh 
#!/usr/bin/env bash

set -e

#source activate pyg

PROJECT=$PWD

# you can adapt these (to save on an external disk for example)
DATA=$PROJECT/data
SAVE=$PROJECT/saved_models

DATASET=python1k
MODEL=gcn

LR=1e-3
CLIP=0.25
BATCHSIZE=4
RUNS=10
EPOCHS=1000
LAY=5
PAT=20

export CUDA_VISIBLE_DEVICES=0
export PYTHONPATH=$PYTHONPATH:$PROJECT

echo "PYTHONPATH: ${PYTHONPATH}"
echo started experiments

cd src

# name identifying the experiment
NAME="${MODEL}_lr${LR}"

CHECKPOINT=""

for f in $SAVE/$NAME*; do
    if [ -e "$f" ]; then
      CHECKPOINT=`basename $f`
      echo "checkpoint exists! $f"
      echo "using: $CHECKPOINT"
      break
    fi
done

python main.py --gnn=$MODEL --drop_ratio=0 --lr=$LR  \
      --num_layer=$LAY  --emb_dim=300 --batch_size=$BATCHSIZE --runs=$RUNS --epochs=$EPOCHS --num_workers=$BATCHSIZE --dataset=$DATASET \
      --dir_data=$DATA  --dir_save=$SAVE --filename=$NAME --clip=$CLIP \
      --checkpointing=1 --checkpoint=$CHECKPOINT --patience=$PAT

echo "Run completed at:- "
date
(pyg) Nitins-MacBook-Air:gnn-based-experiments nitinnanda$

vthost · 2021-06-20T22:22:44Z

This seems to be a known issue with lambda functions. Please try to run with num_worksers=0, and let us know if this solves the problem.

You may also replace the lambda function

dataset.transform = transforms.Compose([lambda data: augment_edge(data, net)])

by adding the definition

def augment2(data):
        return augment_edge(data, 0)

and replacing the above by

dataset.transform = transforms.Compose([augment2])

Thank you for pointing this out!

singlelionwith2cubs · 2021-06-21T16:46:07Z

Applied both recommendations and it worked. Thanks a lot. Issue can be closed.

<<SNIPPED>>
=====Run 10, Epoch 23
Iteration: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 150/150 [00:11<00:00, 13.08it/s]
Iteration: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:01<00:00, 47.70it/s]
Iteration: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:01<00:00, 44.60it/s]
{'Train': {'acc': 0.9333333333333333}, 'Validation': {'acc': 0.995}, 'Test': {'acc': 1.0}}
Early stopping!
Finished training for run 10 !********************
Best validation score: 1.0
Test score: 1.0
0.9056666666666666,0.02510717766636407,1.0,0.0,0.998,0.0025819888974716134
Run completed at:- 
Mon Jun 21 18:07:22 IST 2021
(pyg) Nitins-MacBook-Air:gnn-based-experiments nitinnanda$

Target GNN model for code classification is also ready:

(pyg) Nitins-MacBook-Air:saved_models nitinnanda$ ls -ltr
total 11400
-rw-r--r--  1 nitinnanda  staff  5834983 Jun 21 18:07 gcn_lr1e-3_10_23.pt
(pyg) Nitins-MacBook-Air:saved_models nitinnanda$

geert56 · 2021-06-21T17:02:28Z

I am very much surprised to see a Validation score that is better than the Training score and moreover a Test score which is perfect (1.0). What is the difference between validation and test? How is it possible that a Test score is 100%? Numbers like these severely decrease my confidence in the model. Is there some explanation?

vthost · 2021-06-21T17:03:33Z

python1k is just a dummy sample to test the installation.

geert56 · 2021-06-21T17:09:51Z

All concerns have been addressed. Time to close.

Summary: sr stands for sampling rate, which is now legacy code. Now, it's just sample_rate = 1 / len(data_loader). This has been fixed in the example and values have been updated. When running with DP, the following error is thrown ``` AttributeError: Can't pickle local object 'wrap_collate_with_empty.<locals>.collate' ``` For now, a temporary fix (based on IBM/Project_CodeNet#21 (comment)) is to make num_workers=0 in the dataset loaders. This commit does that. Differential Revision: D40253037 fbshipit-source-id: 0e9a8fccff884497f14e33360c7cb442f2274717

Summary: sr stands for sampling rate, which is now legacy code. Now, it's just sample_rate = 1 / len(data_loader). This has been fixed in the example, by setting batch size to be 60000 * 0.004 = 240 (thanks to https://github.com/ffuuugor for the clarification). On another note, when running with DP, the following error is thrown ``` AttributeError: Can't pickle local object 'wrap_collate_with_empty.<locals>.collate' ``` For now, a temporary fix (based on IBM/Project_CodeNet#21 (comment)) is to make num_workers=0 in the dataset loaders. This commit does that. Differential Revision: D40253037 fbshipit-source-id: e2d491937c12f701e05cf12547cf7eddb0ffb0b6

Summary: Pull Request resolved: #520 sr stands for sampling rate, which is now legacy code. Now, it's just sample_rate = 1 / len(data_loader). This has been fixed in the example, by setting batch size to be 60000 * 0.004 = 240 (thanks to https://github.com/ffuuugor for the clarification). On another note, when running with DP, the following error is thrown ``` AttributeError: Can't pickle local object 'wrap_collate_with_empty.<locals>.collate' ``` For now, a temporary fix (based on IBM/Project_CodeNet#21 (comment)) is to make num_workers=0 in the dataset loaders. This commit does that. Reviewed By: ffuuugor Differential Revision: D40253037 fbshipit-source-id: 99984f8963a4efea6829d109bb81acff0e587c93

Summary: Pull Request resolved: pytorch#520 sr stands for sampling rate, which is now legacy code. Now, it's just sample_rate = 1 / len(data_loader). This has been fixed in the example, by setting batch size to be 60000 * 0.004 = 240 (thanks to https://github.com/ffuuugor for the clarification). On another note, when running with DP, the following error is thrown ``` AttributeError: Can't pickle local object 'wrap_collate_with_empty.<locals>.collate' ``` For now, a temporary fix (based on IBM/Project_CodeNet#21 (comment)) is to make num_workers=0 in the dataset loaders. This commit does that. Reviewed By: ffuuugor Differential Revision: D40253037 fbshipit-source-id: 99984f8963a4efea6829d109bb81acff0e587c93

geert56 closed this as completed Jun 21, 2021

aosokin mentioned this issue Feb 16, 2022

Attribute Error: Can't pickle local object 'EncDecModel.Preproc.dataset.<locals>.<lambda>' yandex-research/sparqling-queries#1

Closed

gouthamuoc mentioned this issue Oct 11, 2022

Fixed issue with missing argument in MNIST example pytorch/opacus#520

Closed

kotaro5487 mentioned this issue Feb 10, 2023

win11ローカルでの学習について laksjdjf/sd-trainer#10

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AttributeError: Can't pickle local object 'main.<locals>.<lambda>' #21

AttributeError: Can't pickle local object 'main.<locals>.<lambda>' #21

singlelionwith2cubs commented Jun 19, 2021 •

edited

Loading

vthost commented Jun 19, 2021

singlelionwith2cubs commented Jun 20, 2021 •

edited

Loading

vthost commented Jun 20, 2021 •

edited

Loading

singlelionwith2cubs commented Jun 21, 2021 •

edited

Loading

geert56 commented Jun 21, 2021

vthost commented Jun 21, 2021

geert56 commented Jun 21, 2021

AttributeError: Can't pickle local object 'main.<locals>.<lambda>' #21

AttributeError: Can't pickle local object 'main.<locals>.<lambda>' #21

Comments

singlelionwith2cubs commented Jun 19, 2021 • edited Loading

vthost commented Jun 19, 2021

singlelionwith2cubs commented Jun 20, 2021 • edited Loading

vthost commented Jun 20, 2021 • edited Loading

singlelionwith2cubs commented Jun 21, 2021 • edited Loading

geert56 commented Jun 21, 2021

vthost commented Jun 21, 2021

geert56 commented Jun 21, 2021

singlelionwith2cubs commented Jun 19, 2021 •

edited

Loading

singlelionwith2cubs commented Jun 20, 2021 •

edited

Loading

vthost commented Jun 20, 2021 •

edited

Loading

singlelionwith2cubs commented Jun 21, 2021 •

edited

Loading