Circular dependency between datafusion and datafusion-sqllogictest crates prevents 0.29.0 from being published to crates.io

[andygrove]

Describe the bug

I was attempting to publish DataFusion 29, and I could not publish the core datafusion crate because it has a dev-dependency on datafusion-sqllogictest, which is not yet published.

   Packaging datafusion v29.0.0 (/home/andy/git/apache/arrow-datafusion/datafusion/core)
    Updating crates.io index
error: failed to prepare local package for uploading

Caused by:
  no matching package named `datafusion-sqllogictest` found
  location searched: registry `crates-io`
  required by package `datafusion v29.0.0 (/home/andy/git/apache/arrow-datafusion/datafusion/core)`

I cannot publish datafusion-sqllogictest either, because it depends on datafusion.

   Packaging datafusion-sqllogictest v29.0.0 (/home/andy/git/apache/arrow-datafusion/datafusion/sqllogictest)
   Verifying datafusion-sqllogictest v29.0.0 (/home/andy/git/apache/arrow-datafusion/datafusion/sqllogictest)
    Updating crates.io index
error: failed to verify package tarball

Caused by:
  failed to select a version for the requirement `datafusion = "^29.0.0"`
  candidate versions found which didn't match: 28.0.0, 27.0.0, 26.0.0, ...
  location searched: crates.io index
  required by package `datafusion-sqllogictest v29.0.0 (/home/andy/git/apache/arrow-datafusion/target/package/datafusion-sqllogictest-29.0.0)`

To Reproduce

Try publishing datafusion crates from source release.

Expected behavior

No response

Additional context

No response

[andygrove]

It looks like the circular dependency was introduced in #7134

cc @tshauck @alamb

[comphead]

rust-analyzer plugin in VSCode also complains

[ERROR project_model::workspace] cyclic deps: datafusion_sqllogictest(Idx::<CrateData>(168)) -> datafusion(Idx::<CrateData>(100)), alternative path: datafusion(Idx::<CrateData>(100)) -> datafusion_sqllogictest(Idx::<CrateData>(168))

[alamb]

Sorry @andygrove

I think we can fix this by cutting the sqllogictest dependency on datafusion (essentially by moving the sqllogictest driver into the sqllogictest crate).

I can make a PR to do this later today

[tshauck]

Sorry all 🙇... and thanks @alamb, you're certainly in a better position to fix this quicker. Happy to help out another way if needed.

[alamb]

I have a draft here #7284 but I ran out of time this evening. I'll finish this up first thing tomorrow morning.

[alamb]

#7284 is ready for review