Bump Marten.AspNetCore from 9.12.0 to 9.16.1#7
Closed
dependabot[bot] wants to merge 1 commit into
Closed
Conversation
--- updated-dependencies: - dependency-name: Marten.AspNetCore dependency-version: 9.16.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Contributor
Author
|
Superseded by #12. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updated Marten.AspNetCore from 9.12.0 to 9.16.1.
Release notes
Sourced from Marten.AspNetCore's releases.
9.16.1
Async daemon data-safety release: the high water detection can no longer advance past "outstanding" event sequence numbers — sequences reserved by transactions that are still in flight — which could silently skip those events in async projections under concurrent append load (bulk imports being the classic case). Root-caused and fixed from discussion #4953.
The four closed mechanisms:
GapDetectorcommand batched three statements, each reading its own READ COMMITTED snapshot — commits landing mid-command could defeat every gap check and silently advance the mark over an in-flight append, regardless ofStaleSequenceThreshold. Detection is now a single statement / single snapshot.last_value, mowing through in-flight gaps.CheckNowAsync(JasperFx.Events 2.29.1) now targets the highest committed sequence and simply waits for in-flight appends to land.reserved last_value - 32across thousands of in-flight reservations on an idle-then-suddenly-busy store, because its gate measured staleness againstmt_event_progression.last_updated. The threshold is now measured from when each specific gap was first observed.pg_lockson the mt_events tables, open transactions inpg_stat_activity, in-progress write xids frompg_current_snapshot()), and holds while any exists — by default Marten never knowingly skips past a live appender. Only provably-dead gaps (rolled-back appends) are skipped, bounded to the sequence ceiling observed with the gap, and every skip is logged at Warning with its exact range.New knobs on
StoreOptions.Projections:UseTransactionEvidenceForGapSkipping(defaulttrue;falserestores the previous wall-clock behavior) andSkipStaleGapsDespiteLiveTransactionsAfter(defaultnull= never skip a live appender; PostgreSQL'sidle_in_transaction_session_timeoutis the recommended backstop against leaked sessions).What's Changed
Full Changelog: JasperFx/marten@V9.16.0...V9.16.1
9.16.0
Lot of CritterWatch, couple bug fixes too
What's Changed
Full Changelog: JasperFx/marten@v9.15.4...V9.16.0
9.15.4
What's Changed
New Contributors
Full Changelog: JasperFx/marten@v9.15.3...v9.15.4
9.15.3
This addresses a potential vulnerability from SQL injection via non-string constant in a LINQ Select projection
Not a common usage, but still.
What's Changed
Full Changelog: JasperFx/marten@9.15.2...v9.15.3
9.15.2
Marten 9.15.2
A patch release. Both fixes come out of the same 512-tenant-database production deployment, reported by @erdtsieck, and both turned out to be worse than the reports described.
Bulk event insert ran a full schema apply on every batch
#4946 — fixed in #4949
The batch
BulkInsertEventsAsyncoverloads opened withStorage.ApplyAllConfiguredChangesToDatabaseAsync()on every call.That is not a cheap check. It calls
Tenancy.BuildDatabases()and runs a full schema delta — partition introspection plusinformation_schemasweeps — across every database in the store. So a sharded store paid one apply per database, per batch. On the reporting deployment, each ~1,000-event batch was triggering 512 schema applies.The measured effect: import throughput collapsed to ~17 events/s, against >3,000/s for the streaming overload. A 686k-event tenant projected to roughly 11 hours. The connection pool filled with ~370 backends whose last statement was Weasel's partition-introspection query, which fed directly into the server-wide connection pressure that deployment was already fighting.
That the streaming overload
BulkInsertEventStreamAsynchas no such call and is fine is the tell: the schema apply was never part of the contract. It was a leftover.The apply is now:
AutoCreateisNone— it is a no-op there by contract, so all that remained was the introspection cost; andIMartenDatabase.Identifier.One subtlety worth recording, because it is the kind of thing that bites later: the memoized apply deliberately does not take a caller's
CancellationToken. The first caller to arrive owns the single in-flight task that every concurrent caller for that database awaits — so binding that shared task to one caller's token would let a single cancelled batch fail sibling batches that were never cancelled. Each caller applies its own token at the await site instead. A schema apply is short and idempotent, so letting it run to completion is the cheaper trade.Under
AutoCreate.None, the event storage must already exist before import. That is the documented contract and it matches the streaming overload — but if you were previously relying on the per-call apply to create it for you under a non-Nonestore, note the change.The document bulk-insert path (
BulkInsertAsync/BulkInsertDocumentsAsync) is unaffected. It routes through the ordinary per-featureEnsureStorageExistsAsyncthat Weasel already memoizes, not a full-store delta.Tenant provisioning silently under-provisioned partitions
#4944 — fixed in #4950
AddPartitionToAllTables, and the tenant-provisioning paths built on it, walked the calling store'sStoreOptionsto decide which tables needed a list partition for a new tenant.So any tool or host that provisions tenants from a store which doesn't register every document type silently under-provisioned. Document types unknown to the caller never got their partitions — and the tenant then failed with a Postgres
23514check-constraint violation on first write to the missing partition. Nothing failed at provisioning time; the damage surfaced later, somewhere else.The workaround was "the provisioning tool must register all document types," which re-creates schema knowledge in a second place and drifts as document types are added.
The sweep is now database-driven: it enumerates tenant list-partitioned tables from the Postgres catalog, so a partially-registered store still provisions every partitioned table it finds.
Scoping is enforced inside the catalog query rather than filtered in memory afterward:
AllSchemaNames()only. Foreign partitioned tables in a shared database are never touched.tenant_id. This is the filter that matters most, and it is what keeps the sweep off Marten's own non-tenant list partitioning:UseArchivedStreamPartitioningkeysmt_eventsonis_archived, andByList()keys on its own field. Without it, a "helpful" sweep would start adding tenant partitions to tables partitioned on something else entirely.ByExternallyManagedListPartitions()are subtracted.Opt out with
SweepPartitionedTablesFromDatabase(default on). No Weasel change was required.Known limitation, and it is a real one: a document type registered into a schema the calling store has never heard of stays invisible to the schema filter — a store cannot own a schema it does not know exists. Single-schema stores (the default, and the reporting deployment's shape) are fully covered. Closing this properly would need a persisted table list alongside
mt_tenant_partitions.... (truncated)
9.15.1
Patch release for a silent data-correctness regression. If you use
ForTenant()on an identity-mapped or dirty-tracked session, upgrade.Fixed
#4947 —
ForTenant()on an identity session stopped returning tenancy-neutral documents (reported by @dervagabund, with a repro — thank you). AForTenant()view of an identity- or dirty-tracked session no longer saw global (tenancy-neutral) documents tracked by the parent session. Since a global document has exactly one row per id for the whole database,LoadAsyncthrough theForTenantview missed the identity map, went to the database, and returnednullfor a document that is there. A silent wrong answer, not an error.Affected: 9.13.0, 9.14.x, 9.15.0. Introduced by the fix for #4801, which tenant-scoped the identity map and version tracker for
ForTenantsessions. That was correct for conjoined documents — where the same id means a different document per tenant — but it was applied per session rather than per document type, so it also isolated document types that are tenancy-neutral and must be shared.Sharing is now decided per document type. A nested
ForTenantsession shares the parent's identity-map and version-tracker entry for a type only when the storage is identity-mapped, the type is notConjoined, and the nested session's database is the same instance as the parent's (under database-per-tenant, the same id in another tenant's database is a different document even for a tenancy-neutral type). The isolation introduced by #4801 is preserved exactly — theBug_4801suite still passes, and the new tests include guard rails asserting conjoined documents stay isolated.Full changelog: JasperFx/marten@9.15.0...9.15.1
9.15.0
Closed issues
findOrAssignTenantDatabaseAsyncreturned early on an existing assignment row, skippingcreatePartitionsForTenant+ per-tenant event-sequence provisioning — so a tenant whose provisioning was interrupted (assignment committed, partitions missing) failed every write with23514forever. Both early-return paths (including a second race-window hole under the advisory lock) now run the same idempotent repair the explicitAddTenantToShardAsync(tenantId, databaseId)overload always ran, guarded to once per process per tenant via the resolution cache.Also in this release
JasperFx.Events.SourceGeneratoranalyzer (JasperFx/jasperfx#505) — CS1061 compile break for no-parameterless-ctor aggregates with instanceApplyreturning the aggregate.pg_inherits) for the #4943 provisioning-tool scenario.Verified against Wolverine (full solution + CoreTests/MartenTests/distribution/Http suites, zero failures) and CritterWatch before publishing. Thanks to @erdtsieck for the dump-verified root-cause analysis.
9.14.1
Marten 9.14.1 is a patch release focused on a substantial round of LINQ query-translation improvements, plus event-store partitioning, high-water, and AoT fixes, and refreshed Weasel/JasperFx dependencies.
LINQ query translation
This release significantly expands what the LINQ provider can push down to PostgreSQL instead of falling back to slower strategies or throwing:
Any(predicate)filters now translate to JSONPath and OR-of-containment strategies, and the old explode/ctidfallback has been replaced by a correlatedEXISTSstrategy.All()shapes and duplicated array fields moved onto the sameEXISTSstrategy. The net effect is correct, index-friendlier SQL for nested-collection predicates.Where()clauses is now supported (e.g.x.Children[0].Name == "...").Sum/Min/Max/Average— can now be used insideWhere()clauses.Regex.IsMatch()is translated inWhere()clauses.IComparable.CompareTo()now works for non-string comparables such asGuid(#4920), alongside broaderCompareTo()coverage,stringIsOneOfvia the?|operator, andCollectionIsEmptyviaICollectionAware.GinIndexJsonDataMember()was added for member-scoped expression GIN indexes.#4916 — subclass queries now use duplicated fields and the base id
Querying a document subclass and filtering on a
Duplicate()'d field or the base-class id previously emitted a JSONB filter (CAST(d.data ->> 'FarmId' as uuid)) instead of the real column, missing the duplicated column and the primary-key index:A subclass shares its parent's table, so the parent's column-backed members (duplicated fields, the id, the soft-delete flag) are now inherited by the subclass's query member resolution. Querying the parent type was already correct and is unchanged.
Event store, partitioning & daemon
UseTenantPartitionedEvents. Registering a tenant whose partition suffix contains a-(so every GUID tenant id) madeApplyAllConfiguredChangesToDatabaseAsync()throw42601because the per-tenantCREATE SEQUENCE/DROP SEQUENCEDDL emitted the identifier unquoted. The schema-apply statements are now quoted (matching the quick-append function and the imperative provisioning path), so hyphenated tenants migrate cleanly. Quote — not sanitize — so the append function can still resolve the sequence by its raw suffix.ObjectDisposedExceptionpath latches-and-rethrows so a HotCold cold node's leadership loop terminates instead of re-polling a disposed data source during shutdown.UseTenantPartitionedEventsthe store-global high-water agent was continuously runningselect max(seq_id) from mt_events, an unfiltered scan that fans out across every tenant partition on every poll. That store-global mark is not used to advance tenant projections (they advance per-tenant), so the recurring scan is now skipped under partitioning; tenant high water is driven by the per-tenant coordinator and poll timer.GetProjectionStatusesAsyncnow resolves the correct named database.AoT / trimming
AddEventType/QueryRawEventDataOnlygeneric-constraint tightening was reversed, and event-mapping construction now routes through the cachedGenericFactoryCachewhile preserving the trimming root (#4930).Dependencies
ShardState.DatabaseIdentifier(jasperfx#501).Closed issues
#4913, #4915, #4916, #4917, #4924, and jasperfx#502.
9.14.0
Marten 9.14.0 is the recommended upgrade for all 9.x users. It combines the LINQ SQL-injection security fix (first shipped in 9.13.0) with the fix for the projection-coordinator shutdown race in #4874 and the accompanying dependency updates.
Beyond the LINQ updates, this made the new Per-Tenant Event Partitioning much more robust as we're testing that in conjunction with a JasperFx client for ludicrous scalability.
🔒 Security — SQL injection in the LINQ provider (GHSA-rfx3-98h7-v3xp)
Several LINQ / tenant-management code paths interpolated a runtime, potentially attacker-influenced value into generated SQL as a single-quoted literal without escaping or parameterization. A value containing a single quote could break out of the literal and inject SQL. The primary vector — a
Dictionary<,>indexer key in aWherefilter (a common "filter by attribute name" / EAV pattern) — was reported privately with an executed proof-of-concept and enabled filter / multi-tenant authorization bypass and blind data exfiltration.Fixed sinks (#4911):
DictionaryItemMember— dictionary indexer key, e.g.Where(x => x.Attributes[key] == v)DictionaryContainsKeyFilter—Dictionary.ContainsKey(key)(Newtonsoft serializer + the Enum branch, which bypass System.Text.Json's quote escaping)SelectParser— a constant string projected throughSelect(x => new { L = runtimeString })DeleteAllForTenant— tenant id reaching per-tenant projection teardown (now parameterized)DatabaseScopedTenantPartitions— tenant id inlined into partition DDLEventLoader— per-tenant partition-pruning literal (defense-in-depth)Each sink now escapes embedded single quotes or binds the value as a parameter; regression tests lock down every vector, and a follow-up LINQ-wide audit cleared the rest of the query hot path (full-text search, string-method translations, comparisons,
IsOneOf/Contains/subset operators, and patching paths). Affected versions: 7.0.0 – 9.12.0. Also patched in 8.37.4 (8.x line) and 9.13.0.Reported responsibly by @svenclaesson — thank you. See advisory GHSA-rfx3-98h7-v3xp (CVE pending assignment).
🛠️ Reliability — projection-coordinator shutdown drain race (#4874)
On host shutdown, the native HotCold projection coordinator could abort with
ObjectDisposedException: 'Npgsql.PoolingDataSource'— the coordinator's leadership poll issued anOpenAsyncagainst an already-disposed data source while tenancy was tearing down. This is the "case B" ordering storm reported against #4874 (distinct from the async-tenancy foundation laid in #4907, which did not resolve it).The fix ships through the dependency updates below, with a Marten-side regression test (
Bug_4874_coordinator_drain_ordering, #4912):ProjectionCoordinatorBaseterminates the leadership loop on a disposed data source / wrapped cancellation instead of re-polling.AdvisoryLockguards against a disposedNpgsqlDataSourceduring shutdown (short-circuits while disposing and treats a disposed-poolObjectDisposedExceptionas a non-acquire rather than propagating).⬆️ Dependency updates
Full changelog since 9.13.0
9.13.0
Security release. Fixes SQL injection in the LINQ provider via unescaped string literals (#4911).
Several LINQ / tenant-management code paths interpolated a runtime, potentially attacker-influenced value into generated SQL as a single-quoted literal without escaping or parameterization; a value containing a single quote could break out and inject SQL. The primary vector — a
Dictionary<,>indexer key in aWherefilter — was reported privately with an executed proof-of-concept (filter / multi-tenant authorization bypass, blind exfiltration).Fixed sinks:
DictionaryItemMember— dictionary indexer keyDictionaryContainsKeyFilter—ContainsKeykey (Newtonsoft serializer + Enum branch)SelectParser— constant string projected viaSelect(...)DeleteAllForTenant— tenant id in per-tenant projection teardown (now parameterized)DatabaseScopedTenantPartitions— tenant id in partition DDLEventLoader— per-tenant partition-pruning literal (defense-in-depth)All 9.x users should upgrade. The 8.x line is fixed in 8.37.4. See advisory GHSA-rfx3-98h7-v3xp.
Commits viewable in compare view.
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)