feat: add PPDS.Dataverse, PPDS.Migration, and CLI packages#8
Merged
joshsmithxrm merged 13 commits intomainfrom Dec 20, 2025
Merged
feat: add PPDS.Dataverse, PPDS.Migration, and CLI packages#8joshsmithxrm merged 13 commits intomainfrom
joshsmithxrm merged 13 commits intomainfrom
Conversation
Add comprehensive design documentation for two new packages: - PPDS.Dataverse: Connection pooling, bulk operations, resilience - PPDS.Migration: High-performance CMT replacement for data migration Includes: - 00_PACKAGE_STRATEGY.md: Overall SDK architecture and naming - 01_PPDS_DATAVERSE_DESIGN.md: Connection pool and bulk ops design - 02_PPDS_MIGRATION_DESIGN.md: Migration engine design - 03_IMPLEMENTATION_PROMPTS.md: Implementation prompts 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…ctivity - Multi-connection pool with support for multiple Application Users - Connection selection strategies: RoundRobin, LeastConnections, ThrottleAware - Throttle tracking with automatic routing away from throttled connections - Bulk operation wrappers: CreateMultiple, UpdateMultiple, UpsertMultiple, DeleteMultiple - DI integration via AddDataverseConnectionPool() extension method - Affinity cookie disabled by default for improved throughput - Updated publish workflow to support multiple packages and version from git tag - 31 unit tests covering strategies, throttle tracking, and DI registration 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Summary of ChangesHello @joshsmithxrm, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces the foundational Highlights
Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a significant and well-designed new feature, the PPDS.Dataverse package. The design documents are thorough, and the implementation of the connection pooling, resilience, and dependency injection components is solid and follows modern C# practices. The code is clean, well-documented, and includes a good set of unit tests.
I've found a critical issue in the bulk operation implementation that prevents it from meeting its performance goals. I've also noted a few medium-severity issues related to API design consistency and a minor TODO that needs to be addressed. Once these points are addressed, this will be an excellent addition to the SDK.
There was a problem hiding this comment.
Pull request overview
This PR introduces the PPDS.Dataverse package, a high-performance Dataverse connectivity layer featuring connection pooling, bulk operations, and intelligent throttle management. The implementation provides multi-connection support for load distribution across Application Users, with automatic routing away from throttled connections.
Key Changes:
- New PPDS.Dataverse package with connection pooling, bulk operations, and resilience features
- Support for multiple connection sources with RoundRobin, LeastConnections, and ThrottleAware selection strategies
- Bulk API wrappers for CreateMultiple, UpdateMultiple, UpsertMultiple, DeleteMultiple operations
- Comprehensive test suite with 31 unit tests covering pooling, resilience, and DI components
Reviewed changes
Copilot reviewed 41 out of 42 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| src/PPDS.Dataverse/PPDS.Dataverse.csproj | Project configuration targeting net8.0/net10.0 with strong naming |
| src/PPDS.Dataverse/Client/* | Client abstraction layer wrapping ServiceClient |
| src/PPDS.Dataverse/Pooling/* | Connection pool with multi-connection support and lifecycle management |
| src/PPDS.Dataverse/Pooling/Strategies/* | Connection selection strategies (RoundRobin, LeastConnections, ThrottleAware) |
| src/PPDS.Dataverse/Resilience/* | Throttle tracking and service protection exception handling |
| src/PPDS.Dataverse/BulkOperations/* | Bulk operation executor with modern API wrappers |
| src/PPDS.Dataverse/DependencyInjection/* | DI extensions and configuration options |
| tests/PPDS.Dataverse.Tests/* | Comprehensive unit tests for all major components |
| docs/design/* | Detailed design documentation and implementation prompts |
| .github/workflows/publish-nuget.yml | Updated to support multiple packages and version extraction from tags |
| CHANGELOG.md | Updated with new package details |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Add README for PPDS.Dataverse package - Update main README to cover both packages - Add ADRs for key architectural decisions: - 0001: Disable affinity cookie by default - 0002: Multi-connection pooling - 0003: Throttle-aware connection selection - Add pattern documentation: - Connection pooling pattern - Bulk operations pattern - Remove design docs (replaced by ADRs and patterns) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Rename ADR files to SCREAMING_SNAKE_CASE (0001_DISABLE_AFFINITY_COOKIE.md, etc.) - Rename pattern files with _PATTERNS.md suffix and move to docs/architecture/ - Remove dates from ADR files (per style guide: no dates in content) - Replace ✓/✗ with ✅/❌ in code examples - Fix broken link to non-existent PPDS.Plugins README 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
New CLI tool for high-performance Dataverse data migration: Commands: - export: Export data from Dataverse to ZIP file - import: Import data from ZIP file into Dataverse - analyze: Analyze schema and display dependency graph - migrate: Migrate data between environments Code quality improvements: - Extract shared console output helpers to ConsoleOutput.cs - Move ImportMode and OutputFormat enums to separate files - Add --bypass-flows option to MigrateCommand for consistency - Add --verbose option to AnalyzeCommand for consistency - Fix broken link in README.md Test coverage: - 98 unit tests covering all command argument parsing - Exit code validation tests - JSON output contract tests - ConsoleOutput helper tests 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
joshsmithxrm
changed the title
PPDS.Dataverse and PPDS.Migration.Cli are new packages that should be published as alpha releases for initial testing before stable 1.0.0. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Fixes NuGet warning about missing readme in the package. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 62 out of 63 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Owner
Author
|
@gemini-code-assist review |
There was a problem hiding this comment.
Code Review
This pull request introduces two significant new packages, PPDS.Dataverse and PPDS.Migration.Cli, which add high-performance Dataverse connectivity and a data migration tool. The code is well-structured with good documentation and unit tests. I've identified a critical issue in the bulk operation implementation that uses an older, less performant API, which should be addressed to meet the project's performance goals. I also have a couple of medium-severity suggestions regarding an unimplemented feature and improving the robustness of the CLI's JSON output. Overall, this is a great addition to the SDK.
- Replace Ecosystem Integration with Dependencies & Versioning section in CLAUDE.md - Add Compatibility table to README.md - Document version sync rules, dependencies, and breaking change impacts 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Critical fixes: - Refactor BulkOperationExecutor to use CreateMultipleRequest, UpdateMultipleRequest, UpsertMultipleRequest with EntityCollection for 10x+ performance improvement - Add ElasticTable flag for Cosmos DB-backed table support with native DeleteMultiple and partial success handling - Wire bypass options correctly: BypassBusinessLogicExecution (new), SuppressCallbackRegistrationExpanderJob for Power Automate flows Medium fixes: - Change IDataverseClient.ConnectedOrgVersion from string to Version? - Wire ThrottleEvents to _throttleTracker.TotalThrottleEvents - Remove unused GetAvailableConnections() from IThrottleTracker - Remove unused Weight property from DataverseConnection - Use System.Text.Json.JsonSerializer in ConsoleOutput 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Security features: - ConnectionStringRedactor: Redacts sensitive values (ClientSecret, Password, Token, etc.) from connection strings before logging - DataverseConnectionException: Wraps connection errors with sanitized messages to prevent credential leakage in logs and stack traces - SensitiveDataAttribute: Marks properties containing sensitive data for documentation and static analysis - DataverseConnection.ToString(): Excludes connection string from output - DataverseConnection.GetRedactedConnectionString(): Safe logging helper CLI security improvements: - Environment variable support for all connection strings: - PPDS_CONNECTION for export/import - PPDS_SOURCE_CONNECTION and PPDS_TARGET_CONNECTION for migrate - Connection args no longer required at parse time (resolved at runtime) - Keeps credentials out of command-line args, process listings, shell history Documentation: - Security best practices section in PPDS.Dataverse README - Environment variable usage guide in PPDS.Migration.Cli README Tests: 40 new tests covering all security features 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implements prompts 9-16 from the implementation plan: ## PPDS.Migration Library - Parallel export with configurable degree of parallelism - Tiered import with automatic dependency resolution (Tarjan's algorithm) - Circular reference detection with deferred field processing - CMT format compatibility (schema.xml, data.zip) - Progress reporting (Console and JSON formats) - Security-first design: no PII in logs, connection string redaction ## Key Components - Models: MigrationSchema, DependencyGraph, ExecutionPlan, IdMapping - Analysis: DependencyGraphBuilder, ExecutionPlanBuilder - Export: ParallelExporter with FetchXML paging - Import: TieredImporter with bulk operation support - Formats: CMT schema/data readers and writers - Progress: IProgressReporter with Console/JSON implementations ## CLI Integration - Updated PPDS.Migration.Cli to reference PPDS.Migration - Commands ready to use migration library ## Documentation - Updated CHANGELOG.md with PPDS.Migration features - Updated README.md with package info and examples - Added PPDS.Migration package README 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
joshsmithxrm
changed the title
Owner
Author
|
@gemini-code-assist review |
There was a problem hiding this comment.
Code Review
This is an impressive and substantial pull request that adds three major new packages to the SDK: PPDS.Dataverse, PPDS.Migration, and PPDS.Migration.Cli. The new functionality for high-performance Dataverse connectivity, connection pooling, and data migration is well-architected and thoughtfully implemented. The use of modern .NET features, dependency injection, and robust error handling is excellent. The security-first design, especially around connection string redaction and exception sanitization, is a standout feature. The inclusion of detailed documentation, ADRs, and a comprehensive test suite demonstrates a high level of quality. I have one critical finding regarding a missing package dependency that needs to be addressed. Otherwise, this is a fantastic contribution.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 108 out of 110 changed files in this pull request and generated 20 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Required for deserializing BulkApiErrorDetails from elastic table operations. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
27 tasks
joshsmithxrm
added a commit
that referenced
this pull request
Jan 3, 2026
- Refactor tests to use early-bound entity classes (PluginAssembly, SdkMessage) instead of late-bound Entity with magic strings - Fix exception handling tests to use UpsertPluginTypeAsync (which actually calls GetComponentTypeAsync) instead of UpsertPackageAsync (which uses CreateWithSolutionHeaderAsync and doesn't call it) - Add tests for FaultException, FaultException<OrganizationServiceFault>, and solution addition skip when componentType returns 0 Addresses bot review comments #8, #14, #15 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
joshsmithxrm
added a commit
that referenced
this pull request
Jan 3, 2026
* refactor: add logging to GetComponentTypeAsync catch block - Add required ILogger<PluginRegistrationService> dependency - Update all callers (ListCommand, CleanCommand, DiffCommand, DeployCommand) to get logger from service provider and pass to constructor - Replace generic catch clause with specific FaultException handlers - Log failures at Debug level for troubleshooting while maintaining graceful degradation behavior (returns 0 to skip solution addition) - Add System.ServiceModel using for FaultException types - Add unit tests for PluginRegistrationService Fixes #61 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: add exception handling tests and use early-bound entities - Refactor tests to use early-bound entity classes (PluginAssembly, SdkMessage) instead of late-bound Entity with magic strings - Fix exception handling tests to use UpsertPluginTypeAsync (which actually calls GetComponentTypeAsync) instead of UpsertPackageAsync (which uses CreateWithSolutionHeaderAsync and doesn't call it) - Add tests for FaultException, FaultException<OrganizationServiceFault>, and solution addition skip when componentType returns 0 Addresses bot review comments #8, #14, #15 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
joshsmithxrm
added a commit
that referenced
this pull request
Jan 8, 2026
joshsmithxrm
added a commit
that referenced
this pull request
Mar 18, 2026
Solutions (#1,#2,#3,#5): Include Managed toggle, sort controls, Visible/API Managed fields, Maker Portal button, isVisible/isApiManaged end-to-end C# support. Import Jobs (#7,#8,#9): Search bar, Operation Context column (end-to-end C#→TS), record count with filtered status. Plugin Traces (#11,#12,#17,#18,#55): CRITICAL Trace Level dropdown fix, Maker Portal URL fix (Dynamics 365 classic), status text labels, record count, search bar. Web Resources (#19): Restored Created By and Created On columns. Connection References (#23,#24,#25,#28,#55): Expandable flow/connection detail with chevron toggle, ISO timestamp formatting, search bar, status badge improvements (Unknown/Unbound), Sync Settings button. Environment Variables (#29,#31,#55): Modified On column restored, search bar, Sync Settings button. Metadata Browser (#37): Custom Only filter toggle for entities. Data Explorer (#41,#42): Clear button, Import button (file dialog). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
joshsmithxrm
added a commit
that referenced
this pull request
Mar 19, 2026
Solutions (#1,#2,#3,#5): Include Managed toggle, sort controls, Visible/API Managed fields, Maker Portal button, isVisible/isApiManaged end-to-end C# support. Import Jobs (#7,#8,#9): Search bar, Operation Context column (end-to-end C#→TS), record count with filtered status. Plugin Traces (#11,#12,#17,#18,#55): CRITICAL Trace Level dropdown fix, Maker Portal URL fix (Dynamics 365 classic), status text labels, record count, search bar. Web Resources (#19): Restored Created By and Created On columns. Connection References (#23,#24,#25,#28,#55): Expandable flow/connection detail with chevron toggle, ISO timestamp formatting, search bar, status badge improvements (Unknown/Unbound), Sync Settings button. Environment Variables (#29,#31,#55): Modified On column restored, search bar, Sync Settings button. Metadata Browser (#37): Custom Only filter toggle for entities. Data Explorer (#41,#42): Clear button, Import button (file dialog). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
joshsmithxrm
added a commit
that referenced
this pull request
Mar 19, 2026
* fix(cdp): use pwsh Expand-Archive for VSIX extraction on Windows bsdtar interprets the C: drive prefix as a remote host, breaking VSIX extraction in --vsix mode. Use PowerShell's Expand-Archive on Windows; keep tar on other platforms. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * plan: extension UX audit fixes — 28 findings across 10 phases Side-by-side comparison of legacy v0.3.4 vs new extension produced 55 findings. After triage: 28 fixes, 22 keep-new-behavior, 5 deferred. Organized into 10 phases (1 cross-cutting + 9 per-panel) for parallel agent execution. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(ext): Phase 1 — cross-cutting infrastructure fixes - Add Connection References + Environment Variables to sidebar Tools (#56, #57) - Standardize command titles: "Open Data Explorer", "Open Plugin Traces" (#51) - Remove text-transform: uppercase from 5 panel CSS files for Title Case headers (#10) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(ext): Phases 2-9 — UX audit fixes across all 8 panels Solutions (#1,#2,#3,#5): Include Managed toggle, sort controls, Visible/API Managed fields, Maker Portal button, isVisible/isApiManaged end-to-end C# support. Import Jobs (#7,#8,#9): Search bar, Operation Context column (end-to-end C#→TS), record count with filtered status. Plugin Traces (#11,#12,#17,#18,#55): CRITICAL Trace Level dropdown fix, Maker Portal URL fix (Dynamics 365 classic), status text labels, record count, search bar. Web Resources (#19): Restored Created By and Created On columns. Connection References (#23,#24,#25,#28,#55): Expandable flow/connection detail with chevron toggle, ISO timestamp formatting, search bar, status badge improvements (Unknown/Unbound), Sync Settings button. Environment Variables (#29,#31,#55): Modified On column restored, search bar, Sync Settings button. Metadata Browser (#37): Custom Only filter toggle for entities. Data Explorer (#41,#42): Clear button, Import button (file dialog). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(ext): review fixes — colSpan bug and command injection - Connection References: fix colCount from wrong heuristic (8) to correct value (7) matching actual column count - CDP tool: sanitize paths for PowerShell injection (escape single quotes); use execFileSync for tar on Unix to avoid shell interpretation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * style(ext): extract inline styles to CSS classes per Gemini review Move search input and sort select inline styles to dedicated .toolbar-search and .toolbar-select CSS classes across 4 panels (Connection Refs, Env Variables, Import Jobs, Solutions). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(ext): add missing webview entry points to knip config The 6 non-query/solutions webview panels are esbuild entry points but were missing from knip.json, causing false-positive unused-file reports. Also un-export internal-only types from shared modules. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(ext): solution filter GUID bug and raw ISO timestamps - SolutionFilter shared component used uniqueName as option value instead of GUID id, causing "solutionId must be a valid GUID" error on Web Resources (and silently wrong on Conn Refs / Env Vars) - Import Jobs and Web Resources rendered raw ISO timestamps; add formatDateTime helper matching Plugin Traces / Conn Refs pattern Found by QA blind verification agents. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(ext): review fixes — escaping, shell safety, URL normalization - Solutions panel: escape rootComponentBehavior and boolean ternaries before innerHTML insertion (S1 compliance) - Connection References: escape literal "Unbound" for consistent escaping discipline - CDP tool: use execFileSync instead of execSync for PowerShell VSIX extraction (S2 compliance — no shell: true) - Plugin Traces: strip trailing slash from environment URL before Maker Portal link construction Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(ext): escape formatDate output in solutions detail card Wrap formatDate() calls in escapeHtml() for consistent escaping discipline — formatDate can return raw ISO string on parse failure. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds three new packages to the PPDS SDK ecosystem:
PPDS.Dataverse
High-performance Dataverse connectivity with:
PPDS.Migration
High-performance data migration engine:
PPDS.Migration.Cli
CLI tool for data migration operations:
export,import,analyze,migrateppds-migrate)Performance Improvements
Test Plan
New Files
src/PPDS.Dataverse/- Connection pooling and bulk operations (15 files)src/PPDS.Migration/- Migration engine library (26 files)src/PPDS.Migration.Cli/- CLI tool (11 files)tests/PPDS.Dataverse.Tests/- Unit teststests/PPDS.Migration.Cli.Tests/- Unit tests🤖 Generated with Claude Code