-
Notifications
You must be signed in to change notification settings - Fork 1
Add Parquet export support for Clustergram widget #111
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Parquet export support for Clustergram widget #111
Conversation
28a18d9 to
4fe6b4a
Compare
23106fe to
f881b08
Compare
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Adds full Parquet export support for the Clustergram widget by serializing cluster data server-side and deserializing it client-side.
- Introduce
export_viz_parquetonMatrixto emit Parquet-encoded byte blobs for all viz tables - Extend
Clustergramwidget to accept aparquet_dataor fallbackmatrixand expose newBytestraitlets - Add a front-end helper (
networkFromParquet) and updaterender_matrix_newto rebuild the network from Parquet bytes
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| src/celldega/clust/matrix.py | Add export_viz_parquet method to serialize clustering tables to Parquet |
| src/celldega/viz/widget.py | Extend Clustergram constructor: handle parquet_data, add network_meta and dynamic Byte traits |
| js/read_parquet/network_from_parquet.js | New helper to convert Arrow tables back into the network object |
| js/widget.js | Update render_matrix_new to detect Parquet bytes and use networkFromParquet |
| notebooks/Single_Category_Matrix.ipynb | Demo notebook showing Parquet-based widget usage |
Comments suppressed due to low confidence (4)
src/celldega/clust/matrix.py:594
- Add unit tests for
export_viz_parquetcovering both clustered and non-clustered matrices and verify the shapes and contents of the returned byte arrays.
def export_viz_parquet(self) -> dict[str, bytes]:
src/celldega/clust/matrix.py:595
- [nitpick] Expand this docstring to list each key in the returned dict (
mat,row_nodes,col_nodes,row_linkage,col_linkage,meta) and describe their formats and meaning.
"""Export visualization using Parquet encoded tables."""
src/celldega/viz/widget.py:125
- [nitpick] Add or update the constructor docstring to clarify the precedence and expected behavior when
network,parquet_data, andmatrixare all provided.
pq_data = kwargs.pop("parquet_data", None)
js/read_parquet/network_from_parquet.js:1
- Add automated tests for
networkFromParquetto confirm correct reconstruction of the network object from Arrow tables, including edge cases like empty or malformed byte inputs.
import { arrayBufferToArrowTable } from './arrayBufferToArrowTable';
* renaming class * clustergram registry * linting fixes * removed print statement * removed name logic * Add Parquet export support for Clustergram widget (#111) * manually bringing in parquet processing * big int * fixing bug with parquet matrix processing * fix index parsing bug * removed print and console logs * added mat as kwarg * Update src/celldega/viz/widget.py Co-authored-by: Copilot <[email protected]> * Update js/read_parquet/network_from_parquet.js Co-authored-by: Copilot <[email protected]> * implemnting astype change --------- Co-authored-by: Copilot <[email protected]> * Update src/celldega/clust/matrix.py more efficient hash for sparse data Co-authored-by: Copilot <[email protected]> * ruff fix * docs: add parquet_data usage (#117) * Add Clustergram parquet widget tests (#116) * upadted test * fixed test lint * ruff format * format JS * added name to docstring * release 0.13.0a4 --------- Co-authored-by: Copilot <[email protected]>
* renaming class * clustergram registry * linting fixes * removed print statement * removed name logic * Add Parquet export support for Clustergram widget (#111) * manually bringing in parquet processing * big int * fixing bug with parquet matrix processing * fix index parsing bug * removed print and console logs * added mat as kwarg * Update src/celldega/viz/widget.py Co-authored-by: Copilot <[email protected]> * Update js/read_parquet/network_from_parquet.js Co-authored-by: Copilot <[email protected]> * implemnting astype change --------- Co-authored-by: Copilot <[email protected]> * Update src/celldega/clust/matrix.py more efficient hash for sparse data Co-authored-by: Copilot <[email protected]> * ruff fix * docs: add parquet_data usage (#117) * Add Clustergram parquet widget tests (#116) * upadted test * fixed test lint * ruff format * format JS * Allow numeric and categorical attributes * fixed loading numerical attr * added pring * Deprecate JSON network export (#128) * renamed attrs attr * numerical attributes starting to viz * changed default colors * lint * swapped colors * use col_attr/row_attr to filter AnnData.obs/var * added tooltip attribute name * changed default behavior of categories in DataFrame and AnnData API * adjusted warnings/errors for matrix_cell_thresh * Add finalize cleanup for widgets (#134) * format * fixed comment on colors * added example notebook * added example notebook * modifie notebook --------- Co-authored-by: Copilot <[email protected]>
* renaming class * clustergram registry * linting fixes * removed print statement * removed name logic * Add Parquet export support for Clustergram widget (#111) * manually bringing in parquet processing * big int * fixing bug with parquet matrix processing * fix index parsing bug * removed print and console logs * added mat as kwarg * Update src/celldega/viz/widget.py Co-authored-by: Copilot <[email protected]> * Update js/read_parquet/network_from_parquet.js Co-authored-by: Copilot <[email protected]> * implemnting astype change --------- Co-authored-by: Copilot <[email protected]> * Update src/celldega/clust/matrix.py more efficient hash for sparse data Co-authored-by: Copilot <[email protected]> * ruff fix * docs: add parquet_data usage (#117) * Add Clustergram parquet widget tests (#116) * upadted test * fixed test lint * ruff format * format JS * Refine Landscape metadata handling * feat(viz): support AnnData in Landscape (#131) * adding print statement * getting single-cell metadata to transfer * making column for meta_cluster * cleaning console logs * working on parsing parquet to objects * meta_cell working * parsing meta_cell_attr * setting up multiple attr support for cells * getting multiple attr working for meta_clust * meta_cluster and meta_cell starting to work * working on cell_attr * format * cleaning * Update js/read_parquet/objects_from_parquet.js Co-authored-by: Copilot <[email protected]> * Update src/celldega/viz/widget.py Co-authored-by: Copilot <[email protected]> * Update js/widget.js Co-authored-by: Copilot <[email protected]> * added example notebook * cleaning * ordered imports * Handle Leiden colors when not precomputed (#157) * fix(viz): generate leiden colors via scanpy * added color generation if not available * ruff * added fix for cell_id not being index in AnnData * ruff * Update src/celldega/viz/widget.py Co-authored-by: Copilot <[email protected]> * Update src/celldega/viz/widget.py Co-authored-by: Copilot <[email protected]> * Update js/viz/landscape_ist.js Co-authored-by: Copilot <[email protected]> * Update js/viz/landscape_ist.js Co-authored-by: Copilot <[email protected]> * Update js/widget.js Co-authored-by: Copilot <[email protected]> * Update js/widget.js Co-authored-by: Copilot <[email protected]> * Update js/widget.js Co-authored-by: Copilot <[email protected]> * Update js/widget.js Co-authored-by: Copilot <[email protected]> * Update src/celldega/viz/widget.py Co-authored-by: Copilot <[email protected]> * Update src/celldega/viz/widget.py Co-authored-by: Copilot <[email protected]> * added default to prevent failure with no AnnDAta --------- Co-authored-by: Copilot <[email protected]>
* renaming class * clustergram registry * linting fixes * removed print statement * removed name logic * Add Parquet export support for Clustergram widget (#111) * manually bringing in parquet processing * big int * fixing bug with parquet matrix processing * fix index parsing bug * removed print and console logs * added mat as kwarg * Update src/celldega/viz/widget.py Co-authored-by: Copilot <[email protected]> * Update js/read_parquet/network_from_parquet.js Co-authored-by: Copilot <[email protected]> * implemnting astype change --------- Co-authored-by: Copilot <[email protected]> * Update src/celldega/clust/matrix.py more efficient hash for sparse data Co-authored-by: Copilot <[email protected]> * ruff fix * docs: add parquet_data usage (#117) * Add Clustergram parquet widget tests (#116) * upadted test * fixed test lint * ruff format * format JS * Refine Landscape metadata handling * feat(viz): support AnnData in Landscape (#131) * adding print statement * getting single-cell metadata to transfer * making column for meta_cluster * cleaning console logs * working on parsing parquet to objects * meta_cell working * parsing meta_cell_attr * setting up multiple attr support for cells * getting multiple attr working for meta_clust * meta_cluster and meta_cell starting to work * working on cell_attr * format * cleaning * Update js/read_parquet/objects_from_parquet.js Co-authored-by: Copilot <[email protected]> * Update src/celldega/viz/widget.py Co-authored-by: Copilot <[email protected]> * Update js/widget.js Co-authored-by: Copilot <[email protected]> * added example notebook * cleaning * Add Parquet traitlet support for neighborhoods --------- Co-authored-by: Copilot <[email protected]>
Summary
export_viz_parquettoMatrixfor generating Parquet tablesClustergramwidget with Parquet traitlets and support forparquet_dataTesting
npm run build(fails: cannot find package 'esbuild')npm run test(fails: jest not found)pytest(fails: unrecognized arguments --cov)https://chatgpt.com/codex/tasks/task_b_6867f3185c98832a87f37669d6a56033