Updated destVI #1457

canergen · 2022-03-22T18:23:50Z

Fixed cell_type amortization (error with neural network structure). Now performs similar to no cell_type amortization. Changed the default to both amortized.
Added L1_sparsity similar to final version of manuscript.
Added dropout at condSCVI decoder and changed all layers to layer norm to increase stability.
Vamp_prior uses over clustering now to find better metapoints. Beforehand the selection of vamp_prior led to different results. Downside: Currently, adds dependency on scanpy and leiden. Do we want this? kmeans clustering was comparable in performance but more unusual choice. We can also remove the cluster generation step from the code and run it outside manually.
Changed library size to non-log in CondSCVI
Updated hyperparameters to match with reproducibility script from destVI paper.

…rameter optimization

for more information, see https://pre-commit.ci

scvi/model/_condscvi.py

Changed to import only submodules of scanty. Only imports if needed

for more information, see https://pre-commit.ci

scvi/model/_condscvi.py

This reverts commit f4ea841.

codecov · 2022-03-22T20:06:55Z

Codecov Report

Merging #1457 (4176ab3) into master (6e734bc) will increase coverage by 0.02%.
The diff coverage is 98.27%.

@@            Coverage Diff             @@
##           master    #1457      +/-   ##
==========================================
+ Coverage   91.04%   91.06%   +0.02%     
==========================================
  Files         112      112              
  Lines        8775     8809      +34     
==========================================
+ Hits         7989     8022      +33     
- Misses        786      787       +1

Impacted Files	Coverage Δ
scvi/model/_condscvi.py	`96.62% <97.36%> (-0.35%)`	⬇️
scvi/model/_destvi.py	`88.28% <100.00%> (+0.10%)`	⬆️
scvi/module/_mrdeconv.py	`95.07% <100.00%> (+0.33%)`	⬆️
scvi/module/_vaec.py	`83.90% <100.00%> (+0.18%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6e734bc...4176ab3. Read the comment docs.

adamgayoso · 2022-03-22T20:40:12Z

@cane11 we are losing some test coverage for condscvi -- can you add more tests for these new options?

canergen · 2022-03-22T20:57:52Z

The coverage difference is related to the import error for scanpy (and differences in number of lines). Should I add a test for the ImportError? The new functions seem to be all covered.

romain-lopez · 2022-03-22T20:59:22Z

Thanks, I'll review this ASAP. Ping me if I haven't done it soon enough please :)

adamgayoso · 2022-03-22T21:01:29Z

The coverage difference is related to the import error for scanpy (and differences in number of lines). Should I add a test for the ImportError? The new functions seem to be all covered.

if that's it then no, no need.

Also as an aside, this version is not backwards compatible with previously saved versions of destVI. I think this is personally fine, but in the future we should put experimental tags on models that exist before a publication is out.

canergen · 2022-03-23T16:50:50Z

It's two bugs (that are not backward compatible), new Vamp_prior is backward compatible using (should we include it in the tutorial?):

adata.obs['vamp_overclustering'] = list(range(adata.n_obs))
adata.obs['vamp_overclustering'] = adata.obs['vamp_overclustering'].astype('category')

I can make the additional parameter updates non hardcoded to increase compatibility. It was hardcoded in the previous version, which is why I kept it hardcoded (dropout in destVI, layer norm in destVI, weights of bayesian prior on beta and eta). Especially, the weight for beta might be different for unusual data inputs (it regularizes how similar sc and spatial technology are).

scvi/model/_condscvi.py

scvi/model/_destvi.py

scvi/module/_mrdeconv.py

scvi/module/_vaec.py

for more information, see https://pre-commit.ci

scvi/module/_mrdeconv.py

scvi/model/_condscvi.py

scvi/module/_mrdeconv.py

tests/models/test_models.py

romain-lopez · 2022-04-03T23:20:48Z

Hi Can, thanks for your newer work on this PR. I think this looks almost perfect except for the glitch with the dropout parameter. I might be wrong but it seems that it's not really used anywhere. It should be either user dependent or fixed everywhere or fixed in some places but not some others. I'd lean towards deleting it as an attribute and keeping it hardcoded!

After these minor points, we can proceed to merging this! @adamgayoso anything else that needs to be done on your end?

scvi/model/_condscvi.py

adamgayoso · 2022-04-04T03:52:10Z

scvi/model/_condscvi.py

+            n_labels_overclustering = len(keys)
+            if n_labels_overclustering > p:
+                raise ValueError(
+                    "Given overclustering contains more clusters than vamp_prior_p. Increase value."


I'd say exactly which parameter name needs to be changed

Please check this fix.

scvi/module/_mrdeconv.py

scvi/module/_vaec.py

docs/user_guide/models/destvi.md

for more information, see https://pre-commit.ci

…o can_destvi

romain-lopez

two remaining minor things

romain-lopez · 2022-04-06T18:24:02Z

docs/user_guide/models/destvi.md

+To explore the results of the output of the stLVM, we published a utilities function covering functions
+for automatic thresholding of cell type proportions, a spatial PCA analysis to find main axis of variation
+in spatial gene expression and the described frequentist test for differential expression. Further information
+can be found on [destvi_utils](https://destvi-utils.readthedocs.io/en/latest/installation.html)

 [^ref1]: Romain Lopez, Baoguo Li, Hadas Keren-Shaul, Pierre Boyeau, Merav Kedmi, David Pilzer, Adam Jelinski, Eyal David, Allon Wagner, Yoseph Addad, Michael I. Jordan, Ido Amit, Nir Yosef (2021),


ah, we can change the title here

Romain Lopez, Baoguo Li, Hadas Keren-Shaul, Pierre Boyeau, Merav Kedmi, David Pilzer, Adam Jelinski, Ido Yofe, Eyal David, Allon Wagner, Can Ergen, Yoseph Addadi, Ofra Golani, Franca Ronchese, Michael I Jordan, Ido Amit, Nir Yosef. DestVI identifies continuums of cell types in spatial transcriptomics data. Nature Biotechnology (in press), 2022.

I made it consistent with other packages and also added link to VAMP prior paper

scvi/model/_destvi.py

romain-lopez · 2022-04-06T21:34:06Z

@adamgayoso ready to merge!

adamgayoso · 2022-04-07T02:08:57Z

meeseeksdev backport to 0.16.x

lumberbot-app · 2022-04-07T02:09:01Z

Can't Dooooo.... It seem like this is already backported (commit is empty).I won't do anything. MrMeeseeks out.

adamgayoso · 2022-04-07T02:12:26Z

meeseeksdev backport to 0.16.x

lumberbot-app · 2022-04-07T02:12:29Z

Can't Dooooo.... It seem like this is already backported (commit is empty).I won't do anything. MrMeeseeks out.

* Fix destVI cell type amortization, Vamp prior from overclustering, parameter optimization * Registry wrong in condSCVI in last commit * REGISTRY key in condSCVI * Updated training parameters of destVI * Learned rate adapted for destVI * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update import of scanty Changed to import only submodules of scanty. Only imports if needed * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added leidenalg to test dependencies * Revert "Added leidenalg to test dependencies" This reverts commit f4ea841. * Changed scanpy import to give error message * Added leiden to scanpy requirements * Defaults to no l1_sparsity. * Style changes to condSCVI. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Space in empty line removed * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Changed vamp overclustering to kmeans, minor comments, added test cases. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Error in test function. Added check for custom clustering * Merge conflict * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Edited dataset after setup of CondSCVI * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Error in test function. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update test function * Ran precommit * Changed eta_prior_weight, exposed beta_prior_weight. Small adjustments. * Doc changes, weight_vprior->mp_vprior * Updated documentation. Style changes. Renamed weighting parameters. * L1_reg added to test function * Updated logsumexp. Small fixes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Precommit edit. * np.array -> torch.cat * Updated references. Updated dropout_decoder Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Adam Gayoso <[email protected]>

canergen and others added 7 commits March 18, 2022 01:01

Fix destVI cell type amortization, Vamp prior from overclustering, pa…

d3334f3

…rameter optimization

Registry wrong in condSCVI in last commit

9e5acd9

REGISTRY key in condSCVI

e1bab17

Updated training parameters of destVI

4d47171

Learned rate adapted for destVI

fe93c0a

Merge branch 'scverse:master' into can_destvi

f3e4bcb

[pre-commit.ci] auto fixes from pre-commit.com hooks

b9dfef1

for more information, see https://pre-commit.ci

canergen mentioned this pull request Mar 22, 2022

destVI bugs #1320

Closed

adamgayoso requested changes Mar 22, 2022

View reviewed changes

scvi/model/_condscvi.py Outdated Show resolved Hide resolved

scvi/model/_condscvi.py Outdated Show resolved Hide resolved

canergen and others added 3 commits March 22, 2022 12:10

Update import of scanty

618a136

Changed to import only submodules of scanty. Only imports if needed

[pre-commit.ci] auto fixes from pre-commit.com hooks

3c8aca4

for more information, see https://pre-commit.ci

Added leidenalg to test dependencies

f4ea841

adamgayoso reviewed Mar 22, 2022

View reviewed changes

scvi/model/_condscvi.py Outdated Show resolved Hide resolved

canergen added 3 commits March 22, 2022 12:37

Revert "Added leidenalg to test dependencies"

5af2ec2

This reverts commit f4ea841.

Changed scanpy import to give error message

6b6c957

Added leiden to scanpy requirements

5d7125b

adamgayoso added this to the 0.16.0 milestone Mar 22, 2022

romain-lopez self-requested a review March 22, 2022 20:59

romain-lopez reviewed Mar 23, 2022

View reviewed changes

canergen and others added 5 commits March 23, 2022 12:41

Defaults to no l1_sparsity.

a78be90

Style changes to condSCVI.

2bafa45

[pre-commit.ci] auto fixes from pre-commit.com hooks

15d6bec

for more information, see https://pre-commit.ci

Space in empty line removed

4e0b4c7

[pre-commit.ci] auto fixes from pre-commit.com hooks

22da8a5

for more information, see https://pre-commit.ci

romain-lopez reviewed Apr 3, 2022

View reviewed changes

scvi/module/_mrdeconv.py Outdated Show resolved Hide resolved

romain-lopez reviewed Apr 3, 2022

View reviewed changes

scvi/module/_mrdeconv.py Outdated Show resolved Hide resolved

romain-lopez reviewed Apr 3, 2022

View reviewed changes

Doc changes, weight_vprior->mp_vprior

6969321

adamgayoso requested changes Apr 4, 2022

View reviewed changes

canergen added 2 commits April 3, 2022 23:35

Updated documentation. Style changes. Renamed weighting parameters.

0e9f24b

L1_reg added to test function

b23ffe8

adamgayoso reviewed Apr 4, 2022

View reviewed changes

scvi/module/_vaec.py Outdated Show resolved Hide resolved

adamgayoso reviewed Apr 4, 2022

View reviewed changes

docs/user_guide/models/destvi.md Outdated Show resolved Hide resolved

adamgayoso reviewed Apr 4, 2022

View reviewed changes

docs/user_guide/models/destvi.md Outdated Show resolved Hide resolved

canergen and others added 6 commits April 4, 2022 16:04

Updated logsumexp. Small fixes

9848765

[pre-commit.ci] auto fixes from pre-commit.com hooks

2c53029

for more information, see https://pre-commit.ci

Precommit edit.

ced6122

Merge branch 'can_destvi' of https://github.com/cane11/scvi-tools int…

d948388

…o can_destvi

np.array -> torch.cat

605ea78

Merge branch 'can_destvi' of https://github.com/cane11/scvi-tools int…

23c0dd6

…o can_destvi

romain-lopez approved these changes Apr 6, 2022

View reviewed changes

Updated references. Updated dropout_decoder

624325a

Merge branch 'master' into can_destvi

4176ab3

adamgayoso enabled auto-merge (squash) April 6, 2022 22:46

adamgayoso approved these changes Apr 6, 2022

View reviewed changes

adamgayoso merged commit c2d7221 into scverse:master Apr 6, 2022

adamgayoso mentioned this pull request Apr 7, 2022

How to view logs? scientific-python/MeeseeksDev#75

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updated destVI #1457

Updated destVI #1457

canergen commented Mar 22, 2022

codecov bot commented Mar 22, 2022 •

edited

Loading

adamgayoso commented Mar 22, 2022

canergen commented Mar 22, 2022

romain-lopez commented Mar 22, 2022

adamgayoso commented Mar 22, 2022

canergen commented Mar 23, 2022 •

edited

Loading

romain-lopez commented Apr 3, 2022 •

edited

Loading

adamgayoso Apr 4, 2022

canergen Apr 4, 2022

romain-lopez left a comment

romain-lopez Apr 6, 2022

romain-lopez Apr 6, 2022

canergen Apr 6, 2022

romain-lopez commented Apr 6, 2022

adamgayoso commented Apr 7, 2022

lumberbot-app bot commented Apr 7, 2022

adamgayoso commented Apr 7, 2022

lumberbot-app bot commented Apr 7, 2022

Updated destVI #1457

Updated destVI #1457

Conversation

canergen commented Mar 22, 2022

codecov bot commented Mar 22, 2022 • edited Loading

Codecov Report

adamgayoso commented Mar 22, 2022

canergen commented Mar 22, 2022

romain-lopez commented Mar 22, 2022

adamgayoso commented Mar 22, 2022

canergen commented Mar 23, 2022 • edited Loading

romain-lopez commented Apr 3, 2022 • edited Loading

adamgayoso Apr 4, 2022

Choose a reason for hiding this comment

canergen Apr 4, 2022

Choose a reason for hiding this comment

romain-lopez left a comment

Choose a reason for hiding this comment

romain-lopez Apr 6, 2022

Choose a reason for hiding this comment

romain-lopez Apr 6, 2022

Choose a reason for hiding this comment

canergen Apr 6, 2022

Choose a reason for hiding this comment

romain-lopez commented Apr 6, 2022

adamgayoso commented Apr 7, 2022

lumberbot-app bot commented Apr 7, 2022

adamgayoso commented Apr 7, 2022

lumberbot-app bot commented Apr 7, 2022

codecov bot commented Mar 22, 2022 •

edited

Loading

canergen commented Mar 23, 2022 •

edited

Loading

romain-lopez commented Apr 3, 2022 •

edited

Loading