docs: add security-performance-improvements report for v3.1.0

docs/features/index.md

@@ -142,6 +142,7 @@
 - [Security Configuration](security/security-configuration.md)
 - [Security Debugging](security/security-debugging.md)
 - [Security Permissions](security/security-permissions.md)
+- [Security Performance Improvements](security/security-performance-improvements.md)
 - [Security Plugin](security/security-plugin.md)
 - [Security Plugin Dependencies](security/security-plugin-dependencies.md)
 - [Security Role Mapping](security/security-role-mapping.md)

docs/features/security/security-performance-improvements.md

@@ -0,0 +1,158 @@
+# Security Performance Improvements
+
+## Summary
+
+The OpenSearch security layer includes performance optimizations that reduce the overhead of authentication, authorization, and access control operations. These improvements focus on reducing serialization costs during inter-node communication and optimizing privilege evaluation. The immutable User object, introduced in v3.1.0, enables caching of serialized data and eliminates thread synchronization overhead.
+
+## Details
+
+### Architecture
+
+```mermaid
+graph TB
+    subgraph "Authentication Flow"
+        REQ[REST Request] --> BR[BackendRegistry]
+        BR --> AC[AuthenticationCache]
+        AC -->|Cache Miss| AB[AuthenticationBackend]
+        AB --> UF[UserFactory]
+        UF --> USER[Immutable User]
+        AC -->|Cache Hit| USER
+    end
+
+    subgraph "Inter-Node Communication"
+        USER --> SER[Serialize Once]
+        SER --> CACHE[Serialization Cache]
+        CACHE --> N1[Node 1]
+        CACHE --> N2[Node 2]
+        CACHE --> NN[Node N]
+    end
+
+    subgraph "Deserialization"
+        N1 --> DC[Deserialization Cache]
+        N2 --> DC
+        NN --> DC
+        DC --> RUSER[Reused User Object]
+    end
+```
+
+### Data Flow
+
+```mermaid
+flowchart TB
+    A[HTTP Request] --> B{User Cached?}
+    B -->|Yes| C[Return Cached User]
+    B -->|No| D[Authenticate via Backend]
+    D --> E[Create Immutable User]
+    E --> F[Cache User + Serialized Form]
+    F --> C
+    C --> G{Inter-Node Request?}
+    G -->|Yes| H[Reuse Cached Serialization]
+    G -->|No| I[Process Locally]
+    H --> J[Remote Node]
+    J --> K{User in Cache?}
+    K -->|Yes| L[Return Cached User]
+    K -->|No| M[Deserialize + Cache]
+    M --> L
+```
+
+### Components
+
+| Component | Description |
+|-----------|-------------|
+| `User` | Immutable user object with builder-style modification methods |
+| `UserFactory` | Factory interface for User object creation and deserialization |
+| `UserFactory.Caching` | Production implementation with LRU cache for deserialized users |
+| `UserFactory.Simple` | Simple implementation without caching for testing |
+| `AuthenticationContext` | Context object passed through authentication/authorization flow |
+| `ImpersonationBackend` | Interface for backends supporting user impersonation |
+| `BackendRegistry` | Central authentication coordinator with user caching |
+
+### Configuration
+
+| Setting | Description | Default |
+|---------|-------------|---------|
+| `plugins.security.user_cache.max_size` | Maximum cached User objects in deserialization cache | 10000 |
+| `plugins.security.user_cache.expire_after_access` | Time before cached entries expire | 1h |
+
+### Usage Example
+
+```java
+// Creating an immutable User object
+User user = new User("username")
+    .withRoles(ImmutableSet.of("role1", "role2"))
+    .withSecurityRoles(ImmutableSet.of("admin"))
+    .withAttributes(ImmutableMap.of("attr1", "value1"));
+
+// Modifying returns a new object (original unchanged)
+User modifiedUser = user.withRoles("additional_role");
+
+// Serialization is cached
+String serialized = user.toSerializedBase64();  // Computed once, cached
+
+// Deserialization uses cache
+UserFactory factory = new UserFactory.Caching(settings);
+User deserializedUser = factory.fromSerializedBase64(serialized);  // Cache lookup
+```
+
+### Custom Backend Migration
+
+```java
+// Old AuthenticationBackend implementation
+public class CustomAuthBackend implements AuthenticationBackend {
+    @Override
+    public User authenticate(AuthCredentials credentials) {
+        User user = new User(credentials.getUsername());
+        user.addRoles(lookupRoles(credentials));
+        return user;
+    }
+
+    @Override
+    public boolean exists(User user) {
+        return userExists(user.getName());
+    }
+}
+
+// New implementation for v3.1.0+
+public class CustomAuthBackend implements AuthenticationBackend, ImpersonationBackend {
+    @Override
+    public User authenticate(AuthenticationContext context) {
+        AuthCredentials credentials = context.getCredentials();
+        return new User(credentials.getUsername())
+            .withRoles(lookupRoles(credentials));
+    }
+
+    @Override
+    public Optional<User> impersonate(User user) {
+        if (userExists(user.getName())) {
+            return Optional.of(user.withAttributes(lookupAttributes(user.getName())));
+        }
+        return Optional.empty();
+    }
+}
+```
+
+## Limitations
+
+- Memory overhead from caching serialized User data and deserialized objects
+- Custom authentication/authorization backends require code changes for v3.1.0 compatibility
+- Cache effectiveness depends on user session patterns; highly dynamic user populations may see reduced benefits
+
+## Related PRs
+
+| Version | PR | Description |
+|---------|-----|-------------|
+| v3.1.0 | [#5339](https://github.com/opensearch-project/security/pull/5339) | Remove unused custom User serialization |
+| v3.1.0 | [#5212](https://github.com/opensearch-project/security/pull/5212) | Immutable user object |
+
+## References
+
+- [Issue #5168](https://github.com/opensearch-project/security/issues/5168): Make User object immutable
+- [Issue #5200](https://github.com/opensearch-project/security/issues/5200): Serialization backward compatibility
+- [Issue #3870](https://github.com/opensearch-project/security/issues/3870): Optimized privilege evaluation
+- [Blog: Performance optimizations for the OpenSearch security layer](https://opensearch.org/blog/performance-optimizations-for-the-opensearch-security-layer/)
+
+## Change History
+
+- **v3.1.0** (2025-05-26): Immutable User object with serialization caching
+- **v3.0.0**: Optimized privilege evaluation for document- and field-level security
+- **v2.19.0**: Initial optimized privilege evaluation

docs/releases/v3.1.0/features/security/security-performance-improvements.md

@@ -0,0 +1,139 @@
+# Security Performance Improvements
+
+## Summary
+
+OpenSearch 3.1.0 introduces significant performance improvements to the security layer through an immutable User object implementation. This optimization reduces serialization overhead during inter-node communication, resulting in improved throughput for clusters with users having many roles and attributes.
+
+## Details
+
+### What's New in v3.1.0
+
+The User object has been redesigned to be immutable, enabling several performance optimizations:
+
+1. **Serialization Reuse**: Serialized binary data can be computed once and reused across multiple requests
+2. **Deserialization Caching**: A cache maps serialized binary data to User objects, reducing deserialization operations
+3. **Thread Safety**: Immutable objects are inherently thread-safe, eliminating synchronization overhead
+4. **Reduced Memory Churn**: Fewer object allocations during request processing
+
+### Technical Changes
+
+#### Architecture Changes
+
+```mermaid
+graph TB
+    subgraph "Before v3.1.0"
+        A1[REST Request] --> B1[BackendRegistry]
+        B1 --> C1[Mutable User Object]
+        C1 --> D1[Serialize per Node]
+        D1 --> E1[Node 1]
+        D1 --> F1[Node 2]
+        D1 --> G1[Node N]
+        E1 --> H1[Deserialize]
+        F1 --> I1[Deserialize]
+        G1 --> J1[Deserialize]
+    end
+
+    subgraph "After v3.1.0"
+        A2[REST Request] --> B2[BackendRegistry]
+        B2 --> C2[Immutable User Object]
+        C2 --> D2[Serialize Once + Cache]
+        D2 --> E2[Node 1]
+        D2 --> F2[Node 2]
+        D2 --> G2[Node N]
+        E2 --> H2[Cache Lookup]
+        F2 --> I2[Cache Lookup]
+        G2 --> J2[Cache Lookup]
+    end
+```
+
+#### New Components
+
+| Component | Description |
+|-----------|-------------|
+| `UserFactory` | Factory for creating and deserializing User objects |
+| `UserFactory.Caching` | Caching implementation that maps serialized data to User objects |
+| `UserFactory.Simple` | Non-caching implementation for testing |
+| `AuthenticationContext` | Context object for passing data between auth backends |
+| `ImpersonationBackend` | New interface for user impersonation support |
+
+#### New Configuration
+
+| Setting | Description | Default |
+|---------|-------------|---------|
+| `plugins.security.user_cache.max_size` | Maximum number of cached User objects | 10000 |
+| `plugins.security.user_cache.expire_after_access` | Cache entry expiration time | 1h |
+
+#### API Changes
+
+The `User` class now uses immutable builder-style methods:
+
+```java
+// Old mutable API (deprecated)
+User user = new User("username");
+user.addRoles(roles);
+user.addAttributes(attributes);
+
+// New immutable API
+User user = new User("username")
+    .withRoles(roles)
+    .withAttributes(attributes)
+    .withSecurityRoles(securityRoles);
+```
+
+The `AuthenticationBackend` interface signature changed:
+
+```java
+// Old signature
+User authenticate(AuthCredentials credentials);
+boolean exists(User user);
+
+// New signature
+User authenticate(AuthenticationContext context);
+```
+
+The `AuthorizationBackend` interface now returns a new User object:
+
+```java
+// Old signature
+void fillRoles(User user, AuthCredentials credentials);
+
+// New signature
+User addRoles(User user, AuthenticationContext context);
+```
+
+### Performance Benchmarks
+
+Based on benchmarks from the PR, users with many attributes see significant improvements:
+
+| User Attributes | Throughput Improvement |
+|-----------------|------------------------|
+| 10 attributes | 21% |
+| 100 attributes | 39% |
+
+### Migration Notes
+
+- Custom `AuthenticationBackend` implementations must be updated to use the new `AuthenticationContext` parameter
+- Custom `AuthorizationBackend` implementations must return a new User object instead of modifying the input
+- The `LdapUser` class is now only used for deserialization compatibility; new code should use the standard `User` class
+
+## Limitations
+
+- The caching mechanism adds memory overhead proportional to the number of unique users
+- Custom authentication backends require code changes to support the new API
+
+## Related PRs
+
+| PR | Description |
+|----|-------------|
+| [#5339](https://github.com/opensearch-project/security/pull/5339) | Remove support for unused custom User serialization |
+| [#5212](https://github.com/opensearch-project/security/pull/5212) | Immutable user object |
+
+## References
+
+- [Issue #5168](https://github.com/opensearch-project/security/issues/5168): Make User object immutable
+- [Issue #5200](https://github.com/opensearch-project/security/issues/5200): When to retire serialization backward compat code
+- [Blog: Performance optimizations for the OpenSearch security layer](https://opensearch.org/blog/performance-optimizations-for-the-opensearch-security-layer/)
+
+## Related Feature Report
+
+- [Full feature documentation](../../../../features/security/security-performance-improvements.md)

docs/releases/v3.1.0/index.md

@@ -51,6 +51,7 @@
 - [Security Backend Bug Fixes](features/security/security-backend-bug-fixes.md) - Stale cache post snapshot restore, compliance audit log diff, DLS/FLS filter reader, auth header logging, password reset UI, forecasting permissions
 - [Security Debugging](features/security/security-debugging.md) - Enhanced error messages for "Security not initialized" with cluster manager status
 - [Security Dependency Updates](features/security/security-dependency-updates.md) - 24 dependency updates including Bouncy Castle 1.81, Kafka 4.0.0, and CVE-2024-52798 fix
+- [Security Performance Improvements](features/security/security-performance-improvements.md) - Immutable User object with serialization caching for reduced inter-node communication overhead
 - [Security Role Mapping](features/security/security-role-mapping.md) - Fix mapped roles not included in ThreadContext userInfo after immutable User object change
 - [Security Permissions](features/security/security-permissions.md) - Add forecast roles and fix missing cluster:monitor and mapping get permissions