Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dyld: Symbol not found: _PyExc_SystemError in tests if there is #[pymethods] in project #340

Open
deaz opened this issue Feb 6, 2019 · 15 comments · May be fixed by #2135
Open

dyld: Symbol not found: _PyExc_SystemError in tests if there is #[pymethods] in project #340

deaz opened this issue Feb 6, 2019 · 15 comments · May be fixed by #2135

Comments

@deaz
Copy link

deaz commented Feb 6, 2019

Tests fail with error when there are methods in #[pymethods] impl for struct:

Compiling pyo3-test v0.1.0 (/Users/ivan/pyo3-test)
    Finished dev [unoptimized + debuginfo] target(s) in 1.10s
     Running target/debug/deps/pyo3_test-34e869060858761f
dyld: Symbol not found: _PyExc_SystemError
  Referenced from: /Users/ivan/pyo3-test/target/debug/deps/pyo3_test-34e869060858761f
  Expected in: flat namespace
 in /Users/ivan/pyo3-test/target/debug/deps/pyo3_test-34e869060858761f
error: process didn't exit successfully: `/Users/ivan/pyo3-test/target/debug/deps/pyo3_test-34e869060858761f` (signal: 6, SIGABRT: process abort signal)

🌍 Environment

  • Your operating system and version: macOS 10.14.2 (18C54)
  • Your python version: 3.7.2
  • How did you install python (e.g. apt or pyenv)? Did you use a virtualenv?: i don't remeber, probably brew install python3. I used virtualenv via pipenv
  • Your rust version (rustc --version): 1.34.0-nightly (8ae730a44 2019-02-04)
  • Are you using the latest pyo3 version? Have you tried using latest master (replace version = "0.x.y" with git = "https://github.com/PyO3/pyo3")? I use 0.6.0-alpha.2. With latest master error is the same.

💥 Reproducing

Run cargo test for the following code:

use pyo3::prelude::*;

#[pyclass]
struct Class {}

#[pymethods]
impl Class {
    #[new]
    fn __new__(obj: &PyRawObject) -> PyResult<()> {
        obj.init(|| Class {})
    }
}

#[pymodule]
fn pyo3_test(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_class::<Class>()
}

#[cfg(test)]
mod tests {
    #[test]
    fn test() {}
}

Also i created minimal project for reproducing the issue: https://github.com/deaz/pyo3-test

@kngwyu
Copy link
Member

kngwyu commented Feb 7, 2019

Thanks for reporting!
Basically we need

[dependencies.pyo3]
features = []

to test project.
For extension projects(i.e. projects with features = ["extension-module"]), you can use setuptools-rust to run test.
@konstin
Do you know any workaround other than that?

@deaz
Copy link
Author

deaz commented Feb 7, 2019

As workaround I added #[cfg(not(test))] to my struct and impl and it is working fine

@konstin
Copy link
Member

konstin commented Feb 7, 2019

First of all, thank you for the test repository!

As @kngwyu said, we need to deactivate features for the tests. A proper solution for this requires rust-lang/cargo#2911, which unfortunately is unlikely to be implemented soon.

For now, you can use the following as workaround:

[package]
name = "pyo3-test"
version = "0.1.0"
authors = ["Ivan Vologdin <[email protected]>"]
edition = "2018"

[lib]
name = "pyo3_test"
crate-type = ["cdylib"]

[dependencies.pyo3]
version = "0.6.0-alpha.2"

[features]
extension-module = ["pyo3/extension-module"]
default = ["extension-module"]

Tests can then be run with cargo test --no-default-features, while cargo build still works as expected.

With a bit context this workaround would be very useful in the guide.

@deaz
Copy link
Author

deaz commented Feb 8, 2019

Thanks for help!

@boredstiff
Copy link

Just ran into this issue - thank you both for posting really great instructions on how to work around this.

@konstin
Copy link
Member

konstin commented May 6, 2019

BTW this is also in the guide: https://pyo3.rs/v0.7.0-alpha.1/advanced.html#testing

@liquidscorpio
Copy link

The work-around mentioned in #340 (comment) does not work if the test is run on a workspace where one of member crates uses PyO3.

@mvaled
Copy link
Contributor

mvaled commented May 28, 2020

I'm getting this error as well, however I'm not using #[pymethods] yet. My entire Rust code now is just:

use pyo3::prelude::*;

#[pymodule]
fn travertine(_py: Python, _m: &PyModule) -> PyResult<()> {
    Ok(())
}

#[cfg(test)]
mod test {
    use super::*;
    use pyo3::types::{IntoPyDict, PyDateTime};

    #[test]
    fn pydelta_conversion() {
        let gil = Python::acquire_gil();
        let py = gil.python();
        let datetime = py.import("datetime").unwrap();
        let locals = [("datetime", datetime)].into_py_dict(py);
        let now: &PyDateTime = py
            .eval("datetime.datetime.utcnow()", None, Some(&locals))
            .unwrap()
            .downcast()
            .unwrap();
        println!("{:?}", now);
    }
}

That suffices to trigger the error. The entire repository is in https://github.com/mvaled/pyo3bug340

The issue is reproducible with stable version of pyo3.

mvaled added a commit to mvaled/pyo3bug340 that referenced this issue May 28, 2020
@davidhewitt
Copy link
Member

Yes. TBH this issue is (I believe) really a dupe of #771

@gilescope
Copy link
Contributor

gilescope commented Jul 8, 2020

Maybe we're going around this the wrong way, maybe when maturin builds it should add in --features=pyo3/extension-module because it knows it's required. That way the cargo test would just work.

We can already do:

maturin develop --cargo-extra-args="--features pyo3/extension-module"

Why not add that flag in automatically if it's clear that it's not there? Having pyo3/extension-module turned on as a default feature seems like a footgun as people will just run cargo test and expect stuff to work on cli, IDEs etc.

I've posted this in the maturin issues list here:
PyO3/maturin#325

gustavgransbo pushed a commit to gustavgransbo/networkg that referenced this issue Dec 30, 2020
The bindings module does not play nicely with
cargo test. Fortunatley, it's functionality
is tested with pytest, and the module is not
required when testing the core module.

The test suite can now be ran with:
`cargo test --features "test"`

Workaround for: PyO3/pyo3#340 which allows for doctests.
Source of workaround: rust-lang/rust#45599
@aviramha aviramha linked a pull request Jan 31, 2022 that will close this issue
@attack68
Copy link
Contributor

I just ran into this problem also. I had quite an extensive project which was working fine with cargo test --lib. A few commits later and the cc linker issue came up on my both my different Macs. The changes made did not seem to be that significant, and certainly not as obvious as the original post assumes some attachment of [pymethods]: I had these already working well so this was not the cause.

Rolling back removed the issue and I tried to remake the changes to identify the cause but couldn't unfortunately.

This was particularly confusing after appearing on a reasonably developed project and apparently after rather innocuous changes.

@alex
Copy link
Contributor

alex commented Mar 26, 2024 via email

@glevco
Copy link
Contributor

glevco commented Jun 4, 2024

@davidhewitt @alex I've been reading about this problem in the FAQ and related issues, and couldn't find any mentions or examples of running PyO3 code from a Rust test in a PyO3 lib.

For reference, I created a new hello world project with the exact steps from the Getting started guide, then added this to Cargo.toml (as suggested in the FAQ):

[features]
extension-module = ["pyo3/extension-module"]
default = ["extension-module"]

And then added this simple test to the lib.rs file:

#[cfg(test)]
mod tests {
    #[test]
    fn test() {
        // pyo3::Python::with_gil(|py| {}); // uncommenting this breaks tests
        assert_eq!(1, 1);
    }
}

Running cargo tests works1. If we uncomment the first line of the test, it fails with cc linking issues. I tried every combination suggested in the FAQ but couldn't get this to work. Do you know if it's possible?


1 This is also weird by the way, because according to the FAQ, for the configuration I set above, I should call cargo test --no-default-features instead of cargo test for it to work. But in fact, the opposite happens. I wonder if the docs are outdated?

@anuradhawick
Copy link

anuradhawick commented Sep 2, 2024

@glevco good catch! There may be some inconsistency in docs I guess. Also the flag seems to be required for cargo build and cargo build --release not for testing. May be testing takes place in the default feature, while we need the pyo3/extension-module for the actual build.

@rnag
Copy link

rnag commented Nov 7, 2024

@davidhewitt @alex I've been reading about this problem in the FAQ and related issues, and couldn't find any mentions or examples of running PyO3 code from a Rust test in a PyO3 lib.

For reference, I created a new hello world project with the exact steps from the Getting started guide, then added this to Cargo.toml (as suggested in the FAQ):

[features]
extension-module = ["pyo3/extension-module"]
default = ["extension-module"]

And then added this simple test to the lib.rs file:

#[cfg(test)]
mod tests {
    #[test]
    fn test() {
        // pyo3::Python::with_gil(|py| {}); // uncommenting this breaks tests
        assert_eq!(1, 1);
    }
}

Running cargo tests works1. If we uncomment the first line of the test, it fails with cc linking issues. I tried every combination suggested in the FAQ but couldn't get this to work. Do you know if it's possible?

1 This is also weird by the way, because according to the FAQ, for the configuration I set above, I should call cargo test --no-default-features instead of cargo test for it to work. But in fact, the opposite happens. I wonder if the docs are outdated?

I can't reproduce. The same code works for me. Except I had to add pyo3::prepare_freethreaded_python() in the new versions of pyo3.

My full code:

#[cfg(test)]
mod tests {
    #[test]
    fn test() {
        pyo3::prepare_freethreaded_python();
        pyo3::Python::with_gil(|py| {}); // uncommenting this breaks tests
        assert_eq!(1, 1);
    }
}

VS Code runs it like:

cargo test --package <mod> --lib --no-default-features -- path::tests::test --exact --show-output 

By the way, to get that to work I added this in my .vscode/settings.json:

{
    "rust-analyzer.runnables.extraArgs": [
        "--no-default-features"
    ]
}

My cargo.toml is looking like:

# See https://github.com/PyO3/pyo3/issues/340#issuecomment-461514532
[dependencies.pyo3]
version = "0.22.5"

[features]
extension-module = ["pyo3/abi3-py39", "pyo3/extension-module"]
default = ["extension-module"]

I will add that I spent at least an hour trying to debug it, so I agree the docs could use a section on this - maybe something like "Calling Python from Rust in Cargo Unit Tests" or similar.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet