Skip to content

Dynamo无法跟踪GraphModule中部分函数,导致同一系列模型生成的hash值相同 #130

@fangfangssj

Description

@fangfangssj

同一系列模型生成的hash相同,例如qwen系列前面的头结构是一样的,第一次提取可以提取到完整的计算图,第二次提取的时候,里面有一个torch无法跟踪的torch._C._functorch.PyCapsule._vmap_increment_nesting函数,第二次提取计算图到这里就结束了,只有一小部分,大概只有百分之一的计算图,第二次已经跟踪了的部分生成一个hash值,所以所有qwen系列hash算出来都是一样的,同理,llama,gpt等系列模型也是如此。

/root/miniconda3/lib/python3.12/site-packages/torch/_dynamo/variables/functions.py:1262: UserWarning: Dynamo does not know how to trace the builtin `torch._C._functorch.PyCapsule._vmap_increment_nesting.` This function is either a Python builtin (e.g. _warnings.warn) or a third-party C/C++ Python extension (perhaps created with pybind).
If it is a Python builtin, please file an issue on GitHub so the PyTorch team can add support for it and see the next case for a workaround.
If it is a third-party C/C++ Python extension, please either wrap it into a PyTorch-understood custom operator (see https://pytorch.org/tutorials/advanced/custom_ops_landing_page.html for more details) or, if it is traceable, use `torch.compiler.allow_in_graph`.
  torch._dynamo.utils.warn_once(explanation + "\n" + "\n".join(hints))
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/root/GraphNet/graph_net/torch/check_redundant_incrementally.py", line 85, in <module>
    main(args=args)
  File "/root/GraphNet/graph_net/torch/check_redundant_incrementally.py", line 65, in main
    graph_hash not in graph_hash2graph_net_model_path
AssertionError: Redundant models detected. old-model-path:/root/GraphNet/samples/transformers-auto-model/distilgpt2/graph_hash.txt, new-model-path:/root/graphnet_workspace/Corianas/1.3b/graph_hash.txt.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions