fix: prevent multiple credible filters to override spark plan

[d0choa]

Because the filter_credible_set was applied over the same object, 2 subsequent calls to the same object would apply them on the same dataframe. That's not the expected as 2 calls to the object could return different results as illustrated here:

In [53]: data_sl = StudyLocus(
    ...:         _df=spark.createDataFrame(observed, schema), _schema=StudyLocus.get_schema()
    ...:     )

In [54]: data_sl.annotate_credible_sets().filter_credible_set(CredibleInterval.IS99).df.withColumn("locus", f
    ...: .explode("locus")).select("locus.*").show()
+-----------+--------------------+---------------+---------------+
|  variantId|posteriorProbability|is95CredibleSet|is99CredibleSet|
+-----------+--------------------+---------------+---------------+
|tagVariantE|                 0.5|           true|           true|
|tagVariantA|                0.44|           true|           true|
|tagVariantC|                0.04|           true|           true|
|tagVariantB|               0.015|          false|           true|
+-----------+--------------------+---------------+---------------+


In [55]: data_sl.annotate_credible_sets().filter_credible_set(CredibleInterval.IS95).df.withColumn("locus", f
    ...: .explode("locus")).select("locus.*").show()
+-----------+--------------------+---------------+---------------+
|  variantId|posteriorProbability|is95CredibleSet|is99CredibleSet|
+-----------+--------------------+---------------+---------------+
|tagVariantE|                 0.5|           true|           true|
|tagVariantA|                0.44|           true|           true|
|tagVariantC|                0.04|           true|           true|
+-----------+--------------------+---------------+---------------+


In [56]: data_sl.annotate_credible_sets().filter_credible_set(CredibleInterval.IS99).df.withColumn("locus", f
    ...: .explode("locus")).select("locus.*").show()
+-----------+--------------------+---------------+---------------+              
|  variantId|posteriorProbability|is95CredibleSet|is99CredibleSet|
+-----------+--------------------+---------------+---------------+
|tagVariantE|                 0.5|           true|           true|
|tagVariantA|                0.44|           true|           true|
|tagVariantC|                0.04|           true|           true|
+-----------+--------------------+---------------+---------------+

This PR creates a new StudyLocus object any time a new call is done to the filter_credible_set function. This is the behaviour after the change:

In [9]: data_sl.annotate_credible_sets().filter_credible_set(CredibleInterval.IS99).df.withColumn("locus", f.
   ...: explode("locus")).select("locus.*").show()
+-----------+--------------------+---------------+---------------+              
|  variantId|posteriorProbability|is95CredibleSet|is99CredibleSet|
+-----------+--------------------+---------------+---------------+
|tagVariantE|                 0.5|           true|           true|
|tagVariantA|                0.44|           true|           true|
|tagVariantC|                0.04|           true|           true|
|tagVariantB|               0.015|          false|           true|
+-----------+--------------------+---------------+---------------+

In [10]: data_sl.annotate_credible_sets().filter_credible_set(CredibleInterval.IS95).df.withColumn("locus", f
    ...: .explode("locus")).select("locus.*").show()
+-----------+--------------------+---------------+---------------+
|  variantId|posteriorProbability|is95CredibleSet|is99CredibleSet|
+-----------+--------------------+---------------+---------------+
|tagVariantE|                 0.5|           true|           true|
|tagVariantA|                0.44|           true|           true|
|tagVariantC|                0.04|           true|           true|
+-----------+--------------------+---------------+---------------+


In [11]: data_sl.annotate_credible_sets().filter_credible_set(CredibleInterval.IS99).df.withColumn("locus", f
    ...: .explode("locus")).select("locus.*").show()
+-----------+--------------------+---------------+---------------+              
|  variantId|posteriorProbability|is95CredibleSet|is99CredibleSet|
+-----------+--------------------+---------------+---------------+
|tagVariantE|                 0.5|           true|           true|
|tagVariantA|                0.44|           true|           true|
|tagVariantC|                0.04|           true|           true|
|tagVariantB|               0.015|          false|           true|
+-----------+--------------------+---------------+---------------+

Tests were passing because they were creating a dataframe for each test.

[project-defiant]

General question to this PR. Since spark dafarames are immutable, we might enforce directly on the dataset, that the .df should not be allowed to set outside of the constructor(s).

Currently we allow for the .df property to be mutable, which could cause similar cases like the one mentioned here.

I think this topic is up to open discussion.
We should definately enforce one way, not both in our code.

[d0choa]

As discussed, this was not causing any issues in the pipeline, but it could easily cause problems.
I'm totally on board with the immutable df property but I would handle it in a separate PR

[project-defiant]

The wandb version was dumped. Was that on purpose? Judging by the commit names, I guess not :)

[project-defiant]

There is still some change in the poetry.lock but I assume it's due to poetry version :)

LGTM

[d0choa]

I aligned the poetry version. One line changed in the lock, and I'm not sure what caused it. It might even be linked to machine architecture. We can discuss it; I don't think it should be a blocker.

I'm merging now... this has been more painful than it should