Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Feature: Private Datasets, Vol. 1 (#1000)
* Private Datasets: GQL API: Ability to change dataset visibility (#814) * Changes before rebasing * from_catalog_n: add clippy warnings suppression * kamu-adapter-auth-oso: add TODOs * Migrations: re-index ReBAC properties * test_oso: update imports * KamuAuthOso: add TODOs * DatasetActionAuthorizer::check_action_allowed(): add a TODO * DatasetEntryServiceHarness: update for tests * RebacService::{get_account_properties(),get_dataset_properties()}: return idempotency * DatasetEntryRepository::get_dataset_entries(): implement for SQLite & Postgres * RebacRepository::properties_count(): implement for SQLite & Postgres * AccountRepository::get_accounts(): implement for SQLite & Postgres * OsoResourceServiceInMem: handle DatasetLifecycleMessage's * OsoResourceServiceInMem::initialize(): update types * Split OsoResourceHolder to OsoResourceServiceInMem & OsoResourceServiceInitializator * OsoResourceHolder: remove dependency to JOB_KAMU_DATASETS_DATASET_ENTRY_INDEXER * kamu-cli: register DatasetEntryIndexer even if not in workspace * Tests stabilization activities * RebacIndexer: add missed #[interface(dyn InitOnStartup)] * kamu-cli: kamu_auth_rebac_services::register_dependencies() * kamu-cli: kamu_adapter_auth_oso::register_dependencies() * OsoDatasetAuthorizer: integrate OsoResourceHolder * OsoResourceHolder: introduce * DatasetEntryIndexer::index_datasets(): increase log severity * RebacIndexer: introduce * kamu-adapter-auth-oso: update description * RebacServiceImpl: dataset_id_entity -> dataset_entity * test_multi_tenant_rebac_dataset_lifecycle_message_consumer: actualize tests * kamu-adapter-auth-oso: add anonymous() helper * kamu-adapter-auth-oso: use MockDatasetRepositoryWriter * kamu-adapter-auth-oso: actualize tests * DatasetActionAuthorizer, DatasetAction: add oso-related impls * OsoDatasetAuthorizer::get_allowed_actions(): return <HashSet<DatasetAction>, InternalError> * GQL, Dataset::properties(): use kamu_auth_rebac::DatasetProperties * kamu-adapter-auth{,-rebac}: remove experimental crates * OsoDatasetAuthorizer: initial RebacService integration * #[allow(unused_variables)] -> #[expect(unused_variables)] * kamu-adapter-auth: extract * kamu-adapter-rebac: initial * kamu-adapter-oauth, AggregatingDatasetActionAuthorizer: initial * kamu-adapter-graphql, from_catalog_n!(): introduce * test_multi_tenant_rebac_dataset_lifecycle_message_consumer: stabilize tests * SmTP, AxumServerPushProtocolInstance::push_main_flow(): remove extra allocations * Tests, test_gql_datasets: use macros for tests * Tests, test_gql_datasets: expected first * Fixes after rebasing * Tests: update dataset_create_empty_*() * RunInDatabaseTransactionLayer: remove unused * GQL, Datasets: use pretty_assertions::assert_eq!() * GQL, DatasetPropertyName: remove outdated scalar * MultiTenantRebacDatasetLifecycleMessageConsumer::handle_dataset_lifecycle_created_message(): add "allows_anonymous_read" property as well * GQL, Dataset::properties(): return flags for simplicity * DatasetMut::set_visibility(): stabilize * Preparations - DependencyGraphServiceInMemory: remove extra .int_err() calls - Dataset::rebac_properties(): introduce - RebacService::get_dataset_properties(): use DatasetPropertyName instead of PropertyName - kamu-auth-rebac: extract value constants - DatasetMut::{set_publicly_available(),set_anonymous_available()}: ensure account owns dataset - DatasetMut: move to own directory - DatasetMut::{set_publicly_available(),set_anonymous_available()}: hide methods behind logging guards - DatasetMut::set_property(): extract method - DatasetMut::set_anonymous_available(): implement - DatasetMut::set_publicly_available(): implement - RevokeResultSuccess::message(): fix typo * Fixes after rebasing on 0.208.* * Tests, kamu-cli: auto-register e2e-user for the e2e mode * OSO: replace names with IDs in schema * Tests stabilization * sqlx: add cached queries * Build speed-ups: remove unused deps * test_pull_derivative_mt: correct running * CHANGELOG: add some entries * DatasetEntryRepository: simplify lifetimes * kamu-adapter-auth-oso-rebac: add "-rebac" suffix * Remove several TODOs * CHANGELOG.md: add several entries * OsoDatasetAuthorizer: revisit implementation * Review 1: GQL: remove Dataset.properties * Review 1: OsoDatasetAuthorizer::ctor(): fix param name * database-common, EntityStreamer: introduce * DatasetEntryServiceImpl: use EntityStreamer * RebacServiceImpl::get_dataset_properties_by_ids(): add * PaginationOpts::safe_limit(): add * Tests, EntityStreamer: add tests with input data * RebacService::get_dataset_properties_by_ids(): update interface * DatasetEntryServiceImpl: use EntityStreamer [2] * OsoResourceServiceInMem: rewrite to use streamed pages * OsoDatasetAuthorizer: use get_multiple_dataset_resources() * OsoResourceServiceInitializator: remove * query_handler_post(): add a comma in doc * DatasetActionAuthorizer: add TODOs * test_flow_event_store: fix typos * OsoDatasetAuthorizer::user_dataset_pair(): remove * RebacIndexer::index_dataset_entries(): iterate over a stream * EntityStreamer: remove extra int_err() & resort declarations * AccountRepository::get_accounts(): streamed version * RebacIndexer::index_accounts(): use iterate over a stream * Test fixes * RebacRepository::get_entity_properties_by_ids(): implementations * Remove extra as_did_str() call * RebacRepository::get_entity_properties_by_ids(): implementations[2] * AccountRepository::accounts_count(): implementations * PostgresAccountRepository::get_accounts(): implementation * sqlx: update cached queries * RebacRepository::get_entity_properties_by_ids(): implementations[3] * DatasetEntryServiceImpl: use tokio::sync::RwLock * PostgresDatasetEntryRepository: tweaks * EntityStreamer -> EntityPageStreamer * sqlite_generate_placeholders_list: extract & use * OsoResourceServiceInMem: add a TODO about state * Search::query(): use from_catalog_n!() * OsoResourceServiceInMem -> OsoResourceServiceImpl * KamuAuthOso: impl Deref to Arc<Oso> * OsoResourceServiceImpl: concrete error types * kamu-adapter-auth-oso-rebac: remove extra dep * DatasetEntryRepository: use odf namespace * DatasetEntryServiceImpl: use odf namespace * DatasetEntryService::list_entries_owned_by(): do not clone owner_id * DatasetEntryRepository::get_dataset_entries(): update ORDER BY column * EntityListing -> EntityPageListing * Tweaks before merging * GQL: Dataset.visibility(): return back, after being deleted by mistake (#997) * Merge actual changes (#998) * Do not show usage error for --all flag (#960) * Do not show usage error for --all flag When --all flag is set for the `repo delete` command, and there are no repositories to delete, do not shoow usage error. * Improve args validation * Improve args validation, e2e tests * Typo corrected in feature flags (#974) * Images, kamu-base-git: fix collision of executable files (#975) * 868 api server provide feature flags for UI (#976) Separated runtime and UI configuration flags. UI config is provided by API server too. * Release v0.210.0 + minor deps * 854 persistent storage of dataset dependencies graph (#973) Dependency graph service moved to 'datasets' domain. Defined dataset dependency repository interface and created 3 implementations. No more postponed initialization, organized initial setup in the form of an indexer. Added telemetry extensions on the way. Tests for repositories, stabilized other tests. Cascading effect on delete within the dataset entry domain. * v0.211.0 + minor deps * Fixed image building (#977) Replaced cascade delete of dataset entries in graph with more explicit events to allow orphan upstream dependencies where only ID is given * Upgrade to datafusion 43 * Use thiserror v2 throughout * trust-dns-resolver => hickory-resolver + minor deps * Fix non-sequential offsets on ingest * 0.212.0 * Use KAMU_CONTAINER_RUNTIME_TYPE env var in Makefile (#991) * Use KAMU_CONTAINER_RUNTIME_TYPE env var in Makefile * Make podman default engine for e2e tests * Backporting changes from Private Datasets feature branch (#992) * Backport tweaks * Add doc strings * Remove unused deps * Remove unactual test * CHANGELOG.md: update * Tips after self-review * Delete env var on dataset delete (#993) * Delete env var on dataset delete * 984 refactoring separate planning and execution phases in key dataset manipulation services (#994) * Draft split of `CompactionService` into planner and execution parts * Compaction cleanups * Compacting more cleanups * Compacting: read old HEAD on planning phase * Reset service split on planner and execution * Extracted `MetadataQueryService` - to query polling, push sources and set transform, instead of ingest/transform planners * DataWriterMetadataState became part of polling ingest item at the planning phase * Setting watermark : separate planner and execution service * Push ingest service prepared for split * Push ingest split on planning and executing * Made some order in infra/core services * {Flow,Task,Outbox}Executor=>Agent * Unified naming of planners and executors * Revised telemetry in refactored components * Review: DataWriterDataFusionBuilder flattened * changelog * v0.123.0 + minor deps * kamu-dev-base: include short commit hash as well (#995) * v0.213.1: less agressive telemetry with `DataWriterMetadataState` --------- Co-authored-by: Andrii Demus <[email protected]> Co-authored-by: Sergei Zaychenko <[email protected]> Co-authored-by: Sergii Mikhtoniuk <[email protected]> Co-authored-by: Roman Boiko <[email protected]> * Fixes after merging (#999) * `DatasetOwnershipService`: moved to the `kamu-dataset` area & implemented via `DatasetEntryServiceImpl` (#1004) * DatasetOwnershipService: use odf namespace * DatasetEntryServiceImpl: impl DatasetOwnershipService * DatasetOwnershipService: move to kamu-datasets scope * CHANGELOG.md: update * GQL, DatasetMut::set_visibility(): correct return type (#1007) * GQL, DatasetMetadata: be prepared for not accessed datasets (#1011) * GQL, DatasetMetadata: correct processing of dataset's dependencies that are not found (#1013) * GQL, DatasetMetadata: update dataset's dependencies types (#1014) * Private Datasets: absorb helpful commits from command updates (#1016) * E2E: added the ability to create an account using CLI * OutboxImmediateImpl::post_message_as_json(): return a dispatch error, if present * Fixes after merge * GQL: Datasets: auth checks (#1017) * E2E: `DatasetMut::set_visibility()` (#1032) * KamuApiServerClient::graphql_api_call_assert_with_token(): remove extra method * KamuApiServerClient: introduce GraphQLResponse type * E2E: DatasetMut::set_visibility() * RebacIndexer: respect predefined treat_datasets_as_public (#1033) * Chore/private datasets address comments - 1 (#1037) * DatasetEntryServiceExt: absorb DatasetOwnershipService::get_owned_datasets() * DatasetEntryServiceExt: absorb all rest DatasetOwnershipService * kamu-adapter-auth-oso-rebac: remove duplicate dep * DatasetActionAuthorizer: classify_datasets_by_allowance() -> classify_dataset_handles_by_allowance() * DatasetRegistry: remove an TODO * DatasetEntryRepository::get_dataset_entries(): use dataset_name column for sorting in implementations (as it was) * OsoResourceServiceImpl: state extraction to singleton component * DatasetActionAuthorizer::check_action_allowed(): use DatasetID instead of DatasetHandle * DatasetActionAuthorizer::is_action_allowed(): use DatasetID instead of DatasetHandle * DatasetActionAuthorizer::get_allowed_actions(): use DatasetID instead of DatasetHandle * DatasetActionAuthorizer: finalization * ODataServiceContext::list_collections(): use DatasetActionAuthorizer::filtered_datasets_stream() * Datasets::by_account_impl(): use DatasetActionAuthorizer::filtered_datasets_stream() * Search::query(): use DatasetActionAuthorizer::filtered_datasets_stream() * GetDatasetDownstreamDependenciesUseCase: extract * GetDatasetUpstreamDependenciesUseCase: extract * AccountServiceImpl::all_accounts(): absorb list_all_accounts() method * ExpensiveAccountRepository: extract trait * RebacService::properties_count(): implement * DatasetEntryService: move list-* operations within an implementation * Fix a cached sqlx query * chore/private-datasets-address-comments-vol-2 (#1038) * ensure_account_owns_dataset() -> ensure_account_is_owner_or_admin() * {Account,Dataset}Properties::apply(): add * RebacServiceImpl: inject default properties * Merge actual changes * Release (minor): 0.218.0 --------- Co-authored-by: Andrii Demus <[email protected]> Co-authored-by: Sergei Zaychenko <[email protected]> Co-authored-by: Sergii Mikhtoniuk <[email protected]> Co-authored-by: Roman Boiko <[email protected]>
- Loading branch information
5 people
authored
Jan 17, 2025
1 parent
05bc9fe
commit b9b2095