Replace _id sort with _seq_no in JobSweeper to fix fielddata error

[manaswini1920]

Description

When indices.id_field_data.enabled is set to false, the JobSweeper fails with Fielddata access on the _id field is disallowed because it sorts by _id during search_after pagination.

This replaces FieldSortBuilder("_id") with FieldSortBuilder("_seq_no"). _seq_no has doc_values enabled by default (no fielddata needed) and is unique per shard, which the sweeper already scopes to via _shards preference.

Resolves #2037

Related Issues

Resolves #2037

Testing

**Before fix (on main):**                                                                                                                                                                                                                                      1. Started local cluster: ./gradlew :alerting:run
 2. Created a monitor — succeeded
3. Disabled _id fielddata: PUT _cluster/settings {"persistent": {"indices.id_field_data.enabled": false}}
4. Waited for sweeper cycle (~60s)
5. Result: sweeper fails repeatedly every cycle:
 

> [ERROR][o.o.a.c.JobSweeper] [integTest-0] [.opendistro-alerting-config][0] Error while sweeping shard [.opendistro-alerting-config][0]
> org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
>   at org.opensearch.index.mapper.IdFieldMapper$IdFieldType.fielddataBuilder(IdFieldMapper.java:184)
> 

 **After fix:**
 1. Built with fix, replaced core jar in test cluster
 2. Disabled _id fielddata before creating any monitors
 3. Created first monitor — succeeded, sweeper scheduled it
 4. Created second monitor — succeeded, sweeper scheduled it
 5. Waited for sweeper cycle (~65s)
 6. Result: no fielddata errors, both monitors scheduled : 
 
> [INFO][o.o.a.c.s.JobScheduler] Scheduling jobId : Yvti45wBWj1CzDFgjaLp, name: test-sweeper-fix
> [INFO][o.o.a.c.s.JobScheduler] Scheduling jobId : cftr45wBWj1CzDFgzKLx, name: test-sweeper-fix-2

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[cwperks]

@manaswini1920 do we need any sort at all? I don't believe sequence number sorting is meaningful because the sequence numbers between 2 documents in the same index don't relate to one another

_seq_no gives an ordering of write operations on a shard

Let's see if we can just remove sorting

[AWSHurneyt]

Approving. @manaswini1920 could we take adding those integ tests as a fast follow-up item? Those seem like important cases to catch.

[manaswini1920]

Approving. @manaswini1920 could we take adding those integ tests as a fast follow-up item? Those seem like important cases to catch.

Updated the PR with unmappedType and try catch changes. Ran integ tests locally :

[Incubating] Problems report is available at: file:///workplace/ragamanu/alerting/build/reports/problems/problems-report.html                                                                                        
                                                                                                                                                                                                                     
Deprecated Gradle features were used in this build, making it incompatible with Gradle 10.                                                                                                                           
                                                                                                                                                                                                                     
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.                                                                            
                                                                                                                                                                                                                     
For more on this, please refer to https://docs.gradle.org/9.2.0/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.                                                         
                                                                                                                                                                                                                     
BUILD SUCCESSFUL in 29m 47s                                                                                                                                                                                          
16 actionable tasks: 2 executed, 14 up-to-date                                                                                                                                                                       

[manaswini1920]

Approving. @manaswini1920 could we take adding those integ tests as a fast follow-up item? Those seem like important cases to catch.

@AWSHurneyt Added these in latest revision : both passed locally :

• test sweeper works with id field data disabled — verifies the original bug fix
• test sweeper handles empty jobs index — verifies the unmappedType("long") fix