Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tracking issue: tenant migration #14719

Open
29 tasks done
drmingdrmer opened this issue Feb 23, 2024 · 0 comments
Open
29 tasks done

Tracking issue: tenant migration #14719

drmingdrmer opened this issue Feb 23, 2024 · 0 comments

Comments

@drmingdrmer
Copy link
Member

drmingdrmer commented Feb 23, 2024 •
edited
Loading

Feature request:

  • Export all meta-data belonging to a tenant;
  • Import meta-data belonging to a tenant;

Not do

  • share and all related meta data is not exported or imported. Because share can not be maintained cross two meta-service cluster.

Current tenant meta-data must be prefixed so that no two records have same key.

Step-1 standardize keys with kvapi::Key

All to meta-service request should use kvapi::Key so that the relationship between records are well defined.

Step-2: remove unnecessary serde::Serialize declaration

Because we need to keep compatibility when updating the kvapi::Key implementations(e.g., add Tenant info to a key), we need to ensure no other encoding methods are used.

Step-3: force all tenant used in kvapi::Key non-empty: replace field tenant: String with tenant: Tenant

Step-4 Filter out keys belonging to a tenant, directly or indirectly
This was referenced Mar 19, 2024
Merged
Merged
Merged
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 21, 2024
@drmingdrmer
refactor: add BackgroundTaskCreator
000eb8c 
`BackgroundJobIdent` is a kvapi::Key but not a value.
It does not need to be `serde`.
Thus in this commit we create another struct `BackgroundTaskCreator`
with the same structure as `BackgroundJobIdent` for usage in values.

And remove `serde` declaration from `BackgroundJobIdent`.

- Part of databendlabs#14719
@drmingdrmer drmingdrmer mentioned this issue Mar 21, 2024
Merged
5 tasks
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 22, 2024
@drmingdrmer
refactor: add BackgroundTaskCreator
2b8650e 
`BackgroundJobIdent` is a kvapi::Key but not a value.
It does not need to be `serde`.
Thus in this commit we create another struct `BackgroundTaskCreator`
with the same structure as `BackgroundJobIdent` for usage in values.

And remove `serde` declaration from `BackgroundJobIdent`.

- Part of databendlabs#14719
drmingdrmer added a commit that referenced this issue Mar 22, 2024
@drmingdrmer
refactor: separate key and value: add BackgroundTaskCreator (#15063)
6485d5c 
refactor: add `BackgroundTaskCreator`

`BackgroundJobIdent` is a kvapi::Key but not a value.
It does not need to be `serde`.
Thus in this commit we create another struct `BackgroundTaskCreator`
with the same structure as `BackgroundJobIdent` for usage in values.

And remove `serde` declaration from `BackgroundJobIdent`.

- Part of #14719
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 23, 2024
@drmingdrmer
refactor: use type Tenant for BackgroundJobIdent.tenant
90714c7 
Define BackgroundJobIdent with `TIdent`, make
`BackgroundJobIdent.tenant` a `Tenant` instead of a plain `String`.

- Part of databendlabs#14719
@drmingdrmer drmingdrmer mentioned this issue Mar 23, 2024
Merged
5 tasks
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 23, 2024
@drmingdrmer
refactor: use type Tenant for BackgroundJobIdent.tenant
535a7a4 
Define BackgroundJobIdent with `TIdent`, make
`BackgroundJobIdent.tenant` a `Tenant` instead of a plain `String`.

- Part of databendlabs#14719
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 23, 2024
@drmingdrmer
refactor: use type Tenant for BackgroundJobIdent.tenant
346e340 
Define BackgroundJobIdent with `TIdent`, make
`BackgroundJobIdent.tenant` a `Tenant` instead of a plain `String`.

- Part of databendlabs#14719
drmingdrmer added a commit that referenced this issue Mar 23, 2024
@drmingdrmer
refactor: use type Tenant for BackgroundJobIdent.tenant (#15074)
fb730d4 
Define BackgroundJobIdent with `TIdent`, make
`BackgroundJobIdent.tenant` a `Tenant` instead of a plain `String`.

- Part of #14719
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 23, 2024
@drmingdrmer
refactor: BackgroundTaskIdent use Tenant
856d137 
Remove `impl FromToProto for BackgroundTaskIdent`, it does not need to
be encoded to protobuf.

- Part of databendlabs#14719
@drmingdrmer drmingdrmer mentioned this issue Mar 23, 2024
Merged
5 tasks
BohuTANG pushed a commit that referenced this issue Mar 23, 2024
@drmingdrmer
refactor: BackgroundTaskIdent use Tenant (#15080)
eeed57b 
Remove `impl FromToProto for BackgroundTaskIdent`, it does not need to
be encoded to protobuf.

- Part of #14719
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 26, 2024
@drmingdrmer
refactor: QueryFragmentsPlanPacket should include just setting changes
b155ac5 
There is no need to have the entire `Settings` in `QueryFragmentsPlanPacket`.
Only the `Settings.changes` is used.

And `Settings.tenant` can not be `serde` in future, ths
`QueryFragmentsPlanPacket` can not include a `Tenant` in it.

- Part of databendlabs#14719
@drmingdrmer drmingdrmer mentioned this issue Mar 26, 2024
Merged
5 tasks
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 26, 2024
@drmingdrmer
refactor: QueryFragmentsPlanPacket should include just setting changes
8b68f10 
There is no need to have the entire `Settings` in `QueryFragmentsPlanPacket`.
Only the `Settings.changes` is used.

And `Settings.tenant` can not be `serde` in future, ths
`QueryFragmentsPlanPacket` can not include a `Tenant` in it.

- Part of databendlabs#14719
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 26, 2024
@drmingdrmer
refactor: QueryFragmentsPlanPacket should include just setting changes
f17f793 
There is no need to have the entire `Settings` in `QueryFragmentsPlanPacket`.
Only the `Settings.changes` is used.

And `Settings.tenant` can not be `serde` in future, ths
`QueryFragmentsPlanPacket` can not include a `Tenant` in it.

- Part of databendlabs#14719
BohuTANG pushed a commit that referenced this issue Mar 27, 2024
@drmingdrmer
refactor: QueryFragmentsPlanPacket should include just setting chan…
e1f225a 
…ges (#15095)

There is no need to have the entire `Settings` in `QueryFragmentsPlanPacket`.
Only the `Settings.changes` is used.

And `Settings.tenant` can not be `serde` in future, ths
`QueryFragmentsPlanPacket` can not include a `Tenant` in it.

- Part of #14719
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 27, 2024
@drmingdrmer
refactor: replace NonEmptyString with Tenant
d0e02e0 
- Part of databendlabs#14719
@drmingdrmer drmingdrmer mentioned this issue Mar 27, 2024
Merged
5 tasks
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 27, 2024
@drmingdrmer
refactor: replace NonEmptyString with Tenant
03f1b4d 
- Part of databendlabs#14719
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 27, 2024
@drmingdrmer
refactor: replace NonEmptyString with Tenant
4ee7c79 
- Part of databendlabs#14719
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 27, 2024
@drmingdrmer
refactor: replace NonEmptyString with Tenant
911b979 
- Part of databendlabs#14719
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 27, 2024
@drmingdrmer
refactor: replace NonEmptyString with Tenant
9e9e1fc 
- Part of databendlabs#14719
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 27, 2024
@drmingdrmer
refactor: replace NonEmptyString with Tenant
fa6fe52 
- Part of databendlabs#14719
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 27, 2024
@drmingdrmer
refactor: replace NonEmptyString with Tenant
c5ef22f 
- Part of databendlabs#14719
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 27, 2024
@drmingdrmer
refactor: replace NonEmptyString with Tenant
c579e1a 
- Part of databendlabs#14719
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 27, 2024
@drmingdrmer
refactor: replace NonEmptyString with Tenant
69f6d77 
- Part of databendlabs#14719
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 27, 2024
@drmingdrmer
refactor: replace NonEmptyString with Tenant
e3f1aaa 
- Part of databendlabs#14719
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 28, 2024
@drmingdrmer
refactor: replace NonEmptyString with Tenant
27b36e6 
- Part of databendlabs#14719
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 28, 2024
@drmingdrmer
refactor: replace NonEmptyString with Tenant
6b6b736 
- Part of databendlabs#14719
drmingdrmer added a commit that referenced this issue Mar 28, 2024
@drmingdrmer
refactor: replace NonEmptyString with Tenant (#15103)
1541158 
* refactor: replace `NonEmptyString` with `Tenant`

- Part of #14719
@drmingdrmer drmingdrmer mentioned this issue Mar 28, 2024
Merged
5 tasks
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 28, 2024
@drmingdrmer
refactor: replace String with Tenant
9a7b484 
Replacing all String with Tenant introduces a lot changes.
These changes will be split into several PRs.
This is one of them.

- Part of databendlabs#14719
drmingdrmer added a commit that referenced this issue Mar 29, 2024
@drmingdrmer
refactor: replace String with Tenant (#15119)
742dfef 
* refactor: replace String with Tenant

Replacing all String with Tenant introduces a lot changes.
These changes will be split into several PRs.
This is one of them.

- Part of #14719

* chore: fix lint

* chore: fix lint

* chore: fix lint

* chore: fix unittest

* chore: fix lint
@drmingdrmer drmingdrmer mentioned this issue Mar 29, 2024
Merged
5 tasks
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 29, 2024
@drmingdrmer
refactor: use TIdent to unify kvapi key def for MaskPolicyTableIdList…
e81ff57 
…Ident and DataMaskNameIdent

- Part of databendlabs#14719
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 29, 2024
@drmingdrmer
refactor: use TIdent to unify kvapi key def for MaskPolicyTableIdList…
ed5ec2c 
…Ident and DataMaskNameIdent

- Part of databendlabs#14719
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 29, 2024
@drmingdrmer
refactor: use TIdent to unify kvapi key def for MaskPolicyTableIdList…
27212db 
…Ident and DataMaskNameIdent

- Part of databendlabs#14719
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Mar 29, 2024
@drmingdrmer
refactor: use TIdent to unify kvapi key def for MaskPolicyTableIdList…
0e23e35 
…Ident and DataMaskNameIdent

- Part of databendlabs#14719
BohuTANG pushed a commit that referenced this issue Mar 30, 2024
@drmingdrmer
refactor: use TIdent to unify kvapi key def for MaskPolicyTableIdList…
e5efdb7 
…Ident and DataMaskNameIdent (#15123)

* refactor: use map_res() to emit an sql error

* refactor: use TIdent to unify kvapi key def for MaskPolicyTableIdListIdent and DataMaskNameIdent

- Part of #14719
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@drmingdrmer