Code

Improve concat performance#7824

Code

Merged

dcherian merged 61 commits intopydata:mainpydata/xarray:mainfrom Illviljan:improve_concatIllviljan/xarray:improve_concatCopy head branch name to clipboardJun 2, 2023

+131-51

Commits

Commits on May 7, 2023

• 1. var_idx very slow

  Illviljan

  committed
  670d970

  Copy full SHA for 670d970

• 2. slow any

  Illviljan

  committed
  1370a0e

  Copy full SHA for 1370a0e

• Add test

  Illviljan

  committed
  8ab83fa

  Copy full SHA for 8ab83fa

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for b6e1881
  pre-commit-ci[bot]

  committed
  b6e1881

  Copy full SHA for b6e1881

• 3. Slow array_type called multiple times

  Illviljan

  committed
  487e958

  Copy full SHA for 487e958

• 4. Can use fastpath for variable.concat?

  Illviljan

  committed
  fbb5430

  Copy full SHA for fbb5430

• Merge branch 'improve_concat' of https://github.com/Illviljan/xarray into improve_concat

  Illviljan

  committed
  e6fe2c4

  Copy full SHA for e6fe2c4

• 5. slow init of pd.unique

  Illviljan

  committed
  dc5f0e6

  Copy full SHA for dc5f0e6

• typos

  Illviljan

  committed
  38aa169

  Copy full SHA for 38aa169

• Update concat.py

  Illviljan

  committed
  5fd6bcb

  Copy full SHA for 5fd6bcb

• Update merge.py

  Illviljan

  committed
  43dcff2

  Copy full SHA for 43dcff2

Commits on May 8, 2023

• 6. Avoid recalculating in loops

  Illviljan

  committed
  7ef0e5d

  Copy full SHA for 7ef0e5d

• 7. No need to transpose 1d arrays.

  Illviljan

  committed
  b2c067d

  Copy full SHA for b2c067d

• Merge branch 'main' into improve_concat

  Illviljan

  committed
  914481b

  Copy full SHA for 914481b

Commits on May 10, 2023

• 8. speed up dask_dataframe

  Illviljan

  committed
  ad048b6

  Copy full SHA for ad048b6

• Update dataset.py

  Illviljan

  committed
  d609883

  Copy full SHA for d609883

• Update dataset.py

  Illviljan

  committed
  4005f6f

  Copy full SHA for 4005f6f

Commits on May 11, 2023

• Update dataset.py

  Illviljan

  committed
  d23c833

  Copy full SHA for d23c833

• Add dask combine test with many variables

  Illviljan

  committed
  6c6b5c7

  Copy full SHA for 6c6b5c7

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 068ba55
  pre-commit-ci[bot]

  committed
  068ba55

  Copy full SHA for 068ba55

• Merge branch 'main' into improve_concat

  Illviljan

  committed
  1bb9dd5

  Copy full SHA for 1bb9dd5

• Merge branch 'improve_concat' of https://github.com/Illviljan/xarray into improve_concat

  Illviljan

  committed
  6828fb6

  Copy full SHA for 6828fb6

• Update combine.py

  Illviljan

  committed
  5670331

  Copy full SHA for 5670331

• Update combine.py

  Illviljan

  committed
  b11afe8

  Copy full SHA for b11afe8

• Update combine.py

  Illviljan

  committed
  e1938a8

  Copy full SHA for e1938a8

Commits on May 14, 2023

• list not needed

  Illviljan

  committed
  43fd7a2

  Copy full SHA for 43fd7a2

• dim is usually string, might be faster to check for that

  Illviljan

  committed
  a59635b

  Copy full SHA for a59635b

• first_var.dims doesn't change and can be defined 1 time

  Illviljan

  committed
  70be8c9

  Copy full SHA for 70be8c9

• mask bad points rather than append good points

  Illviljan

  committed
  68a0c08

  Copy full SHA for 68a0c08

• reduce duplicated code

  Illviljan

  committed
  2ebf78a

  Copy full SHA for 2ebf78a

• don't think id() is required here.

  Illviljan

  committed
  dd81325

  Copy full SHA for dd81325

• Merge branch 'main' into improve_concat

  Illviljan

  committed
  8805ab1

  Copy full SHA for 8805ab1

• get dtype directly instead of through result_dtype

  Illviljan

  committed
  8d3d152

  Copy full SHA for 8d3d152

• seems better to delete rather than append,

  Illviljan

  committed
  476b1e0

  Copy full SHA for 476b1e0

• use internal fastpath if it's a dataset, values should be fine then

  Illviljan

  committed
  674638a

  Copy full SHA for 674638a

• Change isinstance order.

  Illviljan

  committed
  05206f8

  Copy full SHA for 05206f8

• use fastpath if already xarray objtect

  Illviljan

  committed
  e0dae6d

  Copy full SHA for e0dae6d

• Update variable.py

  Illviljan

  committed
  9cc6c2d

  Copy full SHA for 9cc6c2d

• Update dtypes.py

  Illviljan

  committed
  f03154c

  Copy full SHA for f03154c

• typing fixes

  Illviljan

  committed
  529e386

  Copy full SHA for 529e386

• more typing fixes

  Illviljan

  committed
  b7492ca

  Copy full SHA for b7492ca

• test undoing as_compatible_data

  Illviljan

  committed
  cf51f16

  Copy full SHA for cf51f16

• undo concat_dim_length deletion

  Illviljan

  committed
  4b6a4c6

  Copy full SHA for 4b6a4c6

• Update xarray/core/concat.py

  Illviljan

  authored
  2cd984d

  Copy full SHA for 2cd984d

Commits on May 16, 2023

• Remove .copy and sum

  Illviljan

  committed
  86eb72a

  Copy full SHA for 86eb72a

• Merge branch 'main' into improve_concat

  Illviljan

  committed
  b2a498a

  Copy full SHA for b2a498a

Commits on May 18, 2023

• Update concat.py

  Illviljan

  committed
  ee2c3f6

  Copy full SHA for ee2c3f6

• Merge branch 'main' into improve_concat

  Illviljan

  committed
  4092d1b

  Copy full SHA for 4092d1b

Commits on May 21, 2023

• Use OrderedSet

  Illviljan

  committed
  1e514f6

  Copy full SHA for 1e514f6

• Merge branch 'main' into improve_concat

  Illviljan

  committed
  59bf15b

  Copy full SHA for 59bf15b

Commits on May 25, 2023

• Merge branch 'main' into improve_concat

  Illviljan

  committed
  b4198a3

  Copy full SHA for b4198a3

• Apply suggestions from code review

  Illviljan

  authored
  0d0b76e

  Copy full SHA for 0d0b76e

Commits on May 30, 2023

• Merge branch 'main' into improve_concat

  Illviljan

  committed
  51768bd

  Copy full SHA for 51768bd

• Update whats-new.rst

  Illviljan

  committed
  15e2783

  Copy full SHA for 15e2783

Commits on May 31, 2023

• Update xarray/core/concat.py

  Illviljan

  authored
  6f86e98

  Copy full SHA for 6f86e98

• no need to check arrays if cupy isnt even installed

  Illviljan

  committed
  db25a0b

  Copy full SHA for db25a0b

• Merge branch 'main' into improve_concat

  Illviljan

  committed
  da959a8

  Copy full SHA for da959a8

Commits on Jun 1, 2023

• Update whats-new.rst

  Illviljan

  committed
  629cdea

  Copy full SHA for 629cdea

• Add concat comment

  Illviljan

  committed
  cfaa866

  Copy full SHA for cfaa866

• minimize diff

  Illviljan

  committed
  fe62336

  Copy full SHA for fe62336

• revert sketchy

  Illviljan

  committed
  4e90504

  Copy full SHA for 4e90504