Skip to content

Conversation

@cornhundred
Copy link
Collaborator

Summary

  • add export_viz_parquet to Matrix for generating Parquet tables
  • extend Clustergram widget with Parquet traitlets and support for parquet_data
  • add helper for building network objects from Parquet on the frontend
  • load Parquet data in widget render function when available

Testing

  • npm run build (fails: cannot find package 'esbuild')
  • npm run test (fails: jest not found)
  • pytest (fails: unrecognized arguments --cov)

https://chatgpt.com/codex/tasks/task_b_6867f3185c98832a87f37669d6a56033

This comment was marked as outdated.

@cornhundred cornhundred requested a review from Copilot July 5, 2025 21:41

This comment was marked as outdated.

@cornhundred cornhundred force-pushed the DEGA-294-rename-Matrix-Clustergram branch from 28a18d9 to 4fe6b4a Compare July 7, 2025 02:14
@cornhundred cornhundred force-pushed the codex/implement-parquet-based-data-passing-for-clustergram branch from 23106fe to f881b08 Compare July 7, 2025 02:40
@cornhundred cornhundred removed the request for review from huanlity July 7, 2025 14:58
@cornhundred cornhundred marked this pull request as draft July 7, 2025 14:58
@cornhundred cornhundred marked this pull request as ready for review July 7, 2025 15:32
@cornhundred cornhundred requested a review from Copilot July 7, 2025 15:32

This comment was marked as outdated.

@cornhundred cornhundred requested a review from Copilot July 7, 2025 15:41
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Adds full Parquet export support for the Clustergram widget by serializing cluster data server-side and deserializing it client-side.

  • Introduce export_viz_parquet on Matrix to emit Parquet-encoded byte blobs for all viz tables
  • Extend Clustergram widget to accept a parquet_data or fallback matrix and expose new Bytes traitlets
  • Add a front-end helper (networkFromParquet) and update render_matrix_new to rebuild the network from Parquet bytes

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/celldega/clust/matrix.py Add export_viz_parquet method to serialize clustering tables to Parquet
src/celldega/viz/widget.py Extend Clustergram constructor: handle parquet_data, add network_meta and dynamic Byte traits
js/read_parquet/network_from_parquet.js New helper to convert Arrow tables back into the network object
js/widget.js Update render_matrix_new to detect Parquet bytes and use networkFromParquet
notebooks/Single_Category_Matrix.ipynb Demo notebook showing Parquet-based widget usage
Comments suppressed due to low confidence (4)

src/celldega/clust/matrix.py:594

  • Add unit tests for export_viz_parquet covering both clustered and non-clustered matrices and verify the shapes and contents of the returned byte arrays.
    def export_viz_parquet(self) -> dict[str, bytes]:

src/celldega/clust/matrix.py:595

  • [nitpick] Expand this docstring to list each key in the returned dict (mat, row_nodes, col_nodes, row_linkage, col_linkage, meta) and describe their formats and meaning.
        """Export visualization using Parquet encoded tables."""

src/celldega/viz/widget.py:125

  • [nitpick] Add or update the constructor docstring to clarify the precedence and expected behavior when network, parquet_data, and matrix are all provided.
        pq_data = kwargs.pop("parquet_data", None)

js/read_parquet/network_from_parquet.js:1

  • Add automated tests for networkFromParquet to confirm correct reconstruction of the network object from Arrow tables, including edge cases like empty or malformed byte inputs.
import { arrayBufferToArrowTable } from './arrayBufferToArrowTable';

@cornhundred cornhundred merged commit e7e9115 into DEGA-294-rename-Matrix-Clustergram Jul 7, 2025
cornhundred added a commit that referenced this pull request Jul 9, 2025
* renaming class

* clustergram registry

* linting fixes

* removed print statement

* removed name logic

* Add Parquet export support for Clustergram widget (#111)

* manually bringing in parquet processing

* big int

* fixing bug with parquet matrix processing

* fix index parsing bug

* removed print and console logs

* added mat as kwarg

* Update src/celldega/viz/widget.py

Co-authored-by: Copilot <[email protected]>

* Update js/read_parquet/network_from_parquet.js

Co-authored-by: Copilot <[email protected]>

* implemnting astype change

---------

Co-authored-by: Copilot <[email protected]>

* Update src/celldega/clust/matrix.py

more efficient hash for sparse data

Co-authored-by: Copilot <[email protected]>

* ruff fix

* docs: add parquet_data usage (#117)

* Add Clustergram parquet widget tests (#116)

* upadted test

* fixed test lint

* ruff format

* format JS

* added name to docstring

* release 0.13.0a4

---------

Co-authored-by: Copilot <[email protected]>
cornhundred added a commit that referenced this pull request Jul 11, 2025
* renaming class

* clustergram registry

* linting fixes

* removed print statement

* removed name logic

* Add Parquet export support for Clustergram widget (#111)

* manually bringing in parquet processing

* big int

* fixing bug with parquet matrix processing

* fix index parsing bug

* removed print and console logs

* added mat as kwarg

* Update src/celldega/viz/widget.py

Co-authored-by: Copilot <[email protected]>

* Update js/read_parquet/network_from_parquet.js

Co-authored-by: Copilot <[email protected]>

* implemnting astype change

---------

Co-authored-by: Copilot <[email protected]>

* Update src/celldega/clust/matrix.py

more efficient hash for sparse data

Co-authored-by: Copilot <[email protected]>

* ruff fix

* docs: add parquet_data usage (#117)

* Add Clustergram parquet widget tests (#116)

* upadted test

* fixed test lint

* ruff format

* format JS

* Allow numeric and categorical attributes

* fixed loading numerical attr

* added pring

* Deprecate JSON network export (#128)

* renamed attrs attr

* numerical attributes starting to viz

* changed default colors

* lint

* swapped colors

* use col_attr/row_attr to filter AnnData.obs/var

* added tooltip attribute name

* changed default behavior of categories in DataFrame and AnnData API

* adjusted warnings/errors for matrix_cell_thresh

* Add finalize cleanup for widgets (#134)

* format

* fixed comment on colors

* added example notebook

* added example notebook

* modifie notebook

---------

Co-authored-by: Copilot <[email protected]>
@cornhundred cornhundred deleted the codex/implement-parquet-based-data-passing-for-clustergram branch July 15, 2025 19:31
cornhundred added a commit that referenced this pull request Jul 17, 2025
* renaming class

* clustergram registry

* linting fixes

* removed print statement

* removed name logic

* Add Parquet export support for Clustergram widget (#111)

* manually bringing in parquet processing

* big int

* fixing bug with parquet matrix processing

* fix index parsing bug

* removed print and console logs

* added mat as kwarg

* Update src/celldega/viz/widget.py

Co-authored-by: Copilot <[email protected]>

* Update js/read_parquet/network_from_parquet.js

Co-authored-by: Copilot <[email protected]>

* implemnting astype change

---------

Co-authored-by: Copilot <[email protected]>

* Update src/celldega/clust/matrix.py

more efficient hash for sparse data

Co-authored-by: Copilot <[email protected]>

* ruff fix

* docs: add parquet_data usage (#117)

* Add Clustergram parquet widget tests (#116)

* upadted test

* fixed test lint

* ruff format

* format JS

* Refine Landscape metadata handling

* feat(viz): support AnnData in Landscape (#131)

* adding print statement

* getting single-cell metadata to transfer

* making column for meta_cluster

* cleaning console logs

* working on parsing parquet to objects

* meta_cell working

* parsing meta_cell_attr

* setting up multiple attr support for cells

* getting multiple attr working for meta_clust

* meta_cluster and meta_cell starting to work

* working on cell_attr

* format

* cleaning

* Update js/read_parquet/objects_from_parquet.js

Co-authored-by: Copilot <[email protected]>

* Update src/celldega/viz/widget.py

Co-authored-by: Copilot <[email protected]>

* Update js/widget.js

Co-authored-by: Copilot <[email protected]>

* added example notebook

* cleaning

* ordered imports

* Handle Leiden colors when not precomputed (#157)

* fix(viz): generate leiden colors via scanpy

* added color generation if not available

* ruff

* added fix for cell_id not being index in AnnData

* ruff

* Update src/celldega/viz/widget.py

Co-authored-by: Copilot <[email protected]>

* Update src/celldega/viz/widget.py

Co-authored-by: Copilot <[email protected]>

* Update js/viz/landscape_ist.js

Co-authored-by: Copilot <[email protected]>

* Update js/viz/landscape_ist.js

Co-authored-by: Copilot <[email protected]>

* Update js/widget.js

Co-authored-by: Copilot <[email protected]>

* Update js/widget.js

Co-authored-by: Copilot <[email protected]>

* Update js/widget.js

Co-authored-by: Copilot <[email protected]>

* Update js/widget.js

Co-authored-by: Copilot <[email protected]>

* Update src/celldega/viz/widget.py

Co-authored-by: Copilot <[email protected]>

* Update src/celldega/viz/widget.py

Co-authored-by: Copilot <[email protected]>

* added default to prevent failure with no AnnDAta

---------

Co-authored-by: Copilot <[email protected]>
cornhundred added a commit that referenced this pull request Jul 17, 2025
* renaming class

* clustergram registry

* linting fixes

* removed print statement

* removed name logic

* Add Parquet export support for Clustergram widget (#111)

* manually bringing in parquet processing

* big int

* fixing bug with parquet matrix processing

* fix index parsing bug

* removed print and console logs

* added mat as kwarg

* Update src/celldega/viz/widget.py

Co-authored-by: Copilot <[email protected]>

* Update js/read_parquet/network_from_parquet.js

Co-authored-by: Copilot <[email protected]>

* implemnting astype change

---------

Co-authored-by: Copilot <[email protected]>

* Update src/celldega/clust/matrix.py

more efficient hash for sparse data

Co-authored-by: Copilot <[email protected]>

* ruff fix

* docs: add parquet_data usage (#117)

* Add Clustergram parquet widget tests (#116)

* upadted test

* fixed test lint

* ruff format

* format JS

* Refine Landscape metadata handling

* feat(viz): support AnnData in Landscape (#131)

* adding print statement

* getting single-cell metadata to transfer

* making column for meta_cluster

* cleaning console logs

* working on parsing parquet to objects

* meta_cell working

* parsing meta_cell_attr

* setting up multiple attr support for cells

* getting multiple attr working for meta_clust

* meta_cluster and meta_cell starting to work

* working on cell_attr

* format

* cleaning

* Update js/read_parquet/objects_from_parquet.js

Co-authored-by: Copilot <[email protected]>

* Update src/celldega/viz/widget.py

Co-authored-by: Copilot <[email protected]>

* Update js/widget.js

Co-authored-by: Copilot <[email protected]>

* added example notebook

* cleaning

* Add Parquet traitlet support for neighborhoods

---------

Co-authored-by: Copilot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants