Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 28 additions & 2 deletions docs/ADRs/0067-gitlab-cron-polling-event-dispatch.md
Original file line number Diff line number Diff line change
Expand Up @@ -319,7 +319,7 @@ This is an accepted tradeoff — the alternative (sharing a
processed-note-IDs set or cross-reading watermarks between modes)
adds state coupling that complicates the independent-schedule design.

> **Update (2026-08, #5959):** The dual-schedule architecture above was replaced
> **Update (2026-08, #5959):** ~~The dual-schedule architecture above was replaced
> by a single `*/5 * * * *` schedule with automatic full-poll promotion. The
> poller now decides at runtime whether to run a fast poll or full poll based on
> elapsed time since the last full poll (`FULLSEND_LAST_POLL_AT_FULL`). This
Expand All @@ -334,7 +334,33 @@ adds state coupling that complicates the independent-schedule design.
> cron-poller introduction, "Multi-frequency polling" and fast-poll MR note
> limitation under "Slash command latency", the Free tier 60-minute interval
> references in "GitLab tier considerations", and the "5 minutes on Premium, 60
> minutes on Free" latency in "Consequences".
> minutes on Free" latency in "Consequences".~~ Superseded by #6077 below.
>
> **Update (2026-08, #6077):** The single auto-promoting schedule from #5959
> was reverted to two independent schedules with explicit mode selection. The
> auto-promote logic coupled slash-command latency to full-poll duration and
> used a single `resource_group`, causing GitLab to cancel the in-progress
> poll when the next schedule fired. The new architecture:
> - **Slash poll:** `*/5 * * * *` with `FULLSEND_POLL_MODE=slash` — processes
> only `/fs-*` slash commands, fast and lightweight.
> - **Event poll:** `2,17,32,47 * * * *` with `FULLSEND_POLL_MODE=events` —
> full event discovery (labels, MR merges, non-command notes).
> - Each schedule uses a per-mode resource group
> (`fullsend-poll-slash` / `fullsend-poll-events`) so they never cancel
> each other. Resource group process modes differ by purpose:
> `newest_first` for slash (latest command wins, stale polls are
> preempted) and `oldest_first` for events (long-running discovery
> completes before the next cycle starts).
> - The `--mode` CLI flag (also `FULLSEND_POLL_MODE` env var) selects the
> mode explicitly; empty uses the events discovery path but does not
> filter `/fs-*` notes (backward compatibility with pre-dual-schedule
> installations where a single schedule handled all event types).
> - The `shouldFullPoll` auto-promote logic and `FullPollInterval` are removed.
> - Superseded sections: "MR note limitation (fast-poll)" (slash commands
> on MRs are now handled by the dedicated slash poll schedule, not gated
> behind full-poll cycles), and the "Multi-frequency polling" reference
> under "Slash command latency" (replaced by the independent schedule
> architecture above).

### Event routing

Expand Down
2 changes: 1 addition & 1 deletion docs/guides/getting-started/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ To remove fullsend from a single repository:
1. Delete `.gitlab/ci/fullsend-*.yml`, `.gitlab-ci.yml` (if fullsend-managed), and `.fullsend/config.yaml`
2. Delete all CI/CD variables prefixed with `FULLSEND_`
3. Revoke the `fullsend-bot` project access token (Settings → Access Tokens)
4. Delete fullsend pipeline schedules
4. Delete fullsend pipeline schedules (`fullsend slash poll` and `fullsend event poll`)
5. For WIF-mode repos: delete the bot token Secret Manager secret (named `fullsend-bot-token-<owner>--<repo>`) from the GCP project

If you manage your own self-hosted mint, run `fullsend mint unenroll "$OWNER/$REPO"` instead of GitHub step 3. See the [standalone commands](#standalone-commands) table for details.
Expand Down
5 changes: 3 additions & 2 deletions docs/guides/infrastructure/infrastructure-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -252,8 +252,9 @@ Secrets and variables are deployed at different scopes depending on the installa
- `FULLSEND_CREDENTIAL_MODE` — Set to `"variable"` or `"wif"` (when `--inference-project` is provided)
- `FULLSEND_FORGE` — Set to `"gitlab"`
- `FULLSEND_PER_REPO_INSTALL` — Flag indicating per-repo mode (set to `"true"`)
- `FULLSEND_LAST_POLL_AT_FAST` — Timestamp of last fast poll run
- `FULLSEND_LAST_POLL_AT_FULL` — Timestamp of last full poll run
- `FULLSEND_LAST_POLL_AT_FAST` — Timestamp of last slash poll run (name predates the slash/events terminology split; used by the slash-command schedule)
- `FULLSEND_LAST_POLL_AT_FULL` — Timestamp of last event poll run (name predates the slash/events terminology split; used by the event-discovery schedule)
- `FULLSEND_POLL_MODE` — Pipeline schedule variable (`"slash"` or `"events"`); set automatically per schedule during install, not a project-level CI/CD variable
- `FULLSEND_LABEL_STATE` — JSON object tracking label sync state

**Additional variables when `--inference-project` is provided (WIF mode):**
Expand Down
12 changes: 12 additions & 0 deletions internal/cli/poll.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ func newPollCmd() *cobra.Command {
jiraProject string
jqlOverride string
targetRepo string
modeFlag string
)

cmd := &cobra.Command{
Expand All @@ -54,6 +55,15 @@ func newPollCmd() *cobra.Command {
return fmt.Errorf("--project or CI_PROJECT_PATH is required")
}

// Resolve poll mode from flag or environment variable.
mode := modeFlag
if mode == "" {
mode = os.Getenv("FULLSEND_POLL_MODE")
}
if mode != "" && mode != "slash" && mode != "events" {
return fmt.Errorf("invalid poll mode %q: must be \"slash\" or \"events\"", mode)
}

glClient, err := gitlab.New(forgeToken, gitlab.WithBaseURL(gitlabURL))
if err != nil {
return fmt.Errorf("create GitLab client: %w", err)
Expand Down Expand Up @@ -85,6 +95,7 @@ func newPollCmd() *cobra.Command {
PipelineRef: pipelineRef,
PollJobURL: os.Getenv("CI_JOB_URL"),
DispatchSecret: os.Getenv("FULLSEND_DISPATCH_SECRET"),
Mode: mode,
}

poller := poll.New(pollClient, router, projectPath, opts)
Expand All @@ -103,6 +114,7 @@ func newPollCmd() *cobra.Command {
cmd.Flags().StringVar(&jiraProject, "jira-project", "", "Jira project key for JQL scoping")
cmd.Flags().StringVar(&jqlOverride, "jql", "", "Custom JQL override")
cmd.Flags().StringVar(&targetRepo, "target-repo", "", "GitHub repo slug where agents run (default: $GITHUB_REPOSITORY)")
cmd.Flags().StringVar(&modeFlag, "mode", "", "Poll mode: 'slash' (slash commands only) or 'events' (labels, merges, non-slash notes)")
cmd.MarkFlagsOneRequired("forge", "input-driver")

cmd.Hidden = true
Expand Down
28 changes: 28 additions & 0 deletions internal/cli/poll_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ func clearPollEnv(t *testing.T) {
for _, v := range []string{
"FULLSEND_FORGE_TOKEN", "CI_PROJECT_PATH",
"CI_COMMIT_REF_NAME", "CI_DEFAULT_BRANCH", "CI_JOB_URL",
"FULLSEND_POLL_MODE",
"JIRA_BASE_URL", "GITHUB_REPOSITORY",
"JIRA_TOKEN", "JIRA_USER_EMAIL",
} {
Expand Down Expand Up @@ -260,6 +261,33 @@ func TestPollCmd_GitLabMissingProject(t *testing.T) {
}
}

func TestPollCmd_GitLabInvalidMode(t *testing.T) {
clearPollEnv(t)
t.Setenv("FULLSEND_FORGE_TOKEN", "tok")
t.Setenv("CI_PROJECT_PATH", "group/project")
t.Setenv("CI_COMMIT_REF_NAME", "main")
cmd := newPollCmd()
cmd.SetArgs([]string{"--forge", "gitlab", "--project", "group/project", "--mode", "bogus", "--fullsend-dir", t.TempDir()})
err := cmd.Execute()
if err == nil || !strings.Contains(err.Error(), "invalid poll mode") {
t.Fatalf("expected mode validation error, got: %v", err)
}
}

func TestPollCmd_GitLabModeFromEnv(t *testing.T) {
clearPollEnv(t)
t.Setenv("FULLSEND_FORGE_TOKEN", "tok")
t.Setenv("CI_PROJECT_PATH", "group/project")
t.Setenv("CI_COMMIT_REF_NAME", "main")
t.Setenv("FULLSEND_POLL_MODE", "invalid")
cmd := newPollCmd()
cmd.SetArgs([]string{"--forge", "gitlab", "--project", "group/project", "--fullsend-dir", t.TempDir()})
err := cmd.Execute()
if err == nil || !strings.Contains(err.Error(), "invalid poll mode") {
t.Fatalf("expected mode validation error from env, got: %v", err)
}
}

func TestPollCmd_JiraPollInvalidArgs(t *testing.T) {
clearPollEnv(t)
cmd := newPollCmd()
Expand Down
44 changes: 30 additions & 14 deletions internal/cli/repos_gitlab.go
Original file line number Diff line number Diff line change
Expand Up @@ -217,9 +217,10 @@ func storeSecretManagerToken(ctx context.Context, gcpClient gcf.GCFClient, print
return nil
}

// setupGitLabPipelineSchedules creates a single pipeline schedule for
// polling. The schedule runs every 5 minutes; the poller auto-promotes
// to a full poll when 15+ minutes have elapsed since the last full poll.
// setupGitLabPipelineSchedules creates two independent pipeline schedules
// for polling: a fast slash-command poll (every 5 min) and an offset
// event-discovery poll (at minutes 2,17,32,47). Each schedule has its own
// resource group so they never cancel each other.
func setupGitLabPipelineSchedules(ctx context.Context, client forge.Client, printer *ui.Printer, owner, repo, defaultBranch string) error {
// Delete existing fullsend schedules to avoid duplicates on re-install.
existing, listErr := client.ListPipelineSchedules(ctx, owner, repo)
Expand All @@ -237,14 +238,25 @@ func setupGitLabPipelineSchedules(ctx context.Context, client forge.Client, prin
}
}

printer.StepStart("Creating pipeline schedule")
scheduleID, err := client.CreatePipelineSchedule(ctx, owner, repo, defaultBranch,
"fullsend poll", "*/5 * * * *", nil)
printer.StepStart("Creating pipeline schedules")
slashID, err := client.CreatePipelineSchedule(ctx, owner, repo, defaultBranch,
"fullsend slash poll", "*/5 * * * *", map[string]string{"FULLSEND_POLL_MODE": "slash"})
if err != nil {
printer.StepFail("Failed to create poll schedule")
return fmt.Errorf("creating poll schedule: %w", err)
printer.StepFail("Failed to create slash poll schedule")
return fmt.Errorf("creating slash poll schedule: %w", err)
}
printer.StepDone(fmt.Sprintf("Created poll schedule (ID %d)", scheduleID))
printer.StepDone(fmt.Sprintf("Created slash poll schedule (ID %d)", slashID))

eventID, err := client.CreatePipelineSchedule(ctx, owner, repo, defaultBranch,
"fullsend event poll", "2,17,32,47 * * * *", map[string]string{"FULLSEND_POLL_MODE": "events"})
if err != nil {
if delErr := client.DeletePipelineSchedule(ctx, owner, repo, slashID); delErr != nil {
printer.StepWarn(fmt.Sprintf("Failed to clean up slash poll schedule (ID %d): %v", slashID, delErr))
}
printer.StepFail("Failed to create event poll schedule")
return fmt.Errorf("creating event poll schedule: %w", err)
}
printer.StepDone(fmt.Sprintf("Created event poll schedule (ID %d)", eventID))
return nil
}

Expand Down Expand Up @@ -281,7 +293,7 @@ func cleanupGitLabPipelineSchedules(ctx context.Context, client forge.Client, pr
// self-heal cannot fix stale locks on first run because the job is blocked
// before it starts. Running this during install breaks those locks.
//
// The toggle sequence (unordered → newest_first) forces GitLab to
// The toggle sequence (unordered → target mode) forces GitLab to
// re-evaluate the lock state and release stale locks.
func healGitLabResourceGroups(ctx context.Context, glClient *gitlab.LiveClient, printer *ui.Printer, owner, repo string) {
printer.StepStart("Healing resource group locks")
Expand All @@ -296,14 +308,18 @@ func healGitLabResourceGroups(ctx context.Context, glClient *gitlab.LiveClient,
if !strings.HasPrefix(g.Key, "fullsend-") {
continue
}
// Toggle to unordered first to break any stale lock, then set to
// newest_first which is the desired production mode.
// Toggle to unordered first to break any stale lock, then set
// the desired production mode per resource group type.
targetMode := "newest_first"
if g.Key == "fullsend-poll-events" {
targetMode = "oldest_first"
}
if err := glClient.UpdateResourceGroupProcessMode(ctx, owner, repo, g.Key, "unordered"); err != nil {
printer.StepWarn(fmt.Sprintf("Failed to toggle resource group %q to unordered: %v", g.Key, err))
continue
}
if err := glClient.UpdateResourceGroupProcessMode(ctx, owner, repo, g.Key, "newest_first"); err != nil {
printer.StepWarn(fmt.Sprintf("Failed to set resource group %q to newest_first: %v", g.Key, err))
if err := glClient.UpdateResourceGroupProcessMode(ctx, owner, repo, g.Key, targetMode); err != nil {
printer.StepWarn(fmt.Sprintf("Failed to set resource group %q to %s: %v", g.Key, targetMode, err))
continue
}
healed++
Expand Down
78 changes: 66 additions & 12 deletions internal/cli/repos_gitlab_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import (
"fmt"
"net/http"
"net/http/httptest"
"strings"
"testing"

"github.com/stretchr/testify/assert"
Expand Down Expand Up @@ -251,16 +252,24 @@ func TestSetupGitLabBotToken(t *testing.T) {
func TestSetupGitLabPipelineSchedules(t *testing.T) {
ctx := context.Background()

t.Run("creates single poll schedule", func(t *testing.T) {
t.Run("creates two poll schedules with correct variables", func(t *testing.T) {
fake := &forge.FakeClient{}
var buf bytes.Buffer
printer := ui.New(&buf)

err := setupGitLabPipelineSchedules(ctx, fake, printer, "group", "project", "main")
require.NoError(t, err)
require.Len(t, fake.CreatedSchedules, 1)
require.Len(t, fake.CreatedSchedules, 2)

// Slash poll: every 5 minutes.
assert.Equal(t, "*/5 * * * *", fake.CreatedSchedules[0].Cron)
assert.Equal(t, "fullsend poll", fake.CreatedSchedules[0].Description)
assert.Equal(t, "fullsend slash poll", fake.CreatedSchedules[0].Description)
assert.Equal(t, map[string]string{"FULLSEND_POLL_MODE": "slash"}, fake.CreatedSchedules[0].Variables)

// Event poll: offset cron to avoid collision with slash poll.
assert.Equal(t, "2,17,32,47 * * * *", fake.CreatedSchedules[1].Cron)
assert.Equal(t, "fullsend event poll", fake.CreatedSchedules[1].Description)
assert.Equal(t, map[string]string{"FULLSEND_POLL_MODE": "events"}, fake.CreatedSchedules[1].Variables)
})
}

Expand All @@ -274,7 +283,39 @@ func TestSetupGitLabPipelineSchedules_ScheduleError(t *testing.T) {

err := setupGitLabPipelineSchedules(ctx, fake, printer, "group", "project", "main")
require.Error(t, err)
assert.Contains(t, err.Error(), "creating poll schedule")
assert.Contains(t, err.Error(), "creating slash poll schedule")
}

func TestSetupGitLabPipelineSchedules_EventScheduleError_RollsBackSlash(t *testing.T) {
ctx := context.Background()

t.Run("successful rollback", func(t *testing.T) {
fake := &forge.FakeClient{
CreatePipelineScheduleErrSeq: []error{nil, fmt.Errorf("quota exceeded")},
}
var buf bytes.Buffer
printer := ui.New(&buf)

err := setupGitLabPipelineSchedules(ctx, fake, printer, "group", "project", "main")
require.Error(t, err)
assert.Contains(t, err.Error(), "creating event poll schedule")
require.Len(t, fake.CreatedSchedules, 1, "only slash schedule should have been created")
assert.Equal(t, []int64{1}, fake.DeletedScheduleIDs, "should roll back the slash schedule")
})

t.Run("rollback delete also fails", func(t *testing.T) {
fake := &forge.FakeClient{
CreatePipelineScheduleErrSeq: []error{nil, fmt.Errorf("quota exceeded")},
Errors: map[string]error{"DeletePipelineSchedule": fmt.Errorf("forbidden")},
}
var buf bytes.Buffer
printer := ui.New(&buf)

err := setupGitLabPipelineSchedules(ctx, fake, printer, "group", "project", "main")
require.Error(t, err)
assert.Contains(t, err.Error(), "creating event poll schedule")
assert.Contains(t, buf.String(), "Failed to clean up slash poll schedule")
})
}

func TestSetupGitLabPipelineSchedules_ListError(t *testing.T) {
Expand Down Expand Up @@ -322,8 +363,8 @@ func TestCleanupGitLabPipelineSchedules(t *testing.T) {
fake := &forge.FakeClient{
PipelineSchedules: map[string][]forge.PipelineSchedule{
"group/project": {
{ID: 1, Description: "fullsend fast poll", Active: true},
{ID: 2, Description: "fullsend full poll", Active: true},
{ID: 1, Description: "fullsend slash poll", Active: true},
{ID: 2, Description: "fullsend event poll", Active: true},
{ID: 3, Description: "unrelated schedule", Active: true},
},
},
Expand Down Expand Up @@ -412,7 +453,7 @@ func TestSetupGitLabPipelineSchedules_DeletesExisting(t *testing.T) {
fake := &forge.FakeClient{
PipelineSchedules: map[string][]forge.PipelineSchedule{
"group/project": {
{ID: 5, Description: "fullsend fast poll", Active: true},
{ID: 5, Description: "fullsend slash poll", Active: true},
{ID: 6, Description: "unrelated", Active: true},
},
},
Expand All @@ -423,7 +464,7 @@ func TestSetupGitLabPipelineSchedules_DeletesExisting(t *testing.T) {
err := setupGitLabPipelineSchedules(ctx, fake, printer, "group", "project", "main")
require.NoError(t, err)
assert.Equal(t, []int64{5}, fake.DeletedScheduleIDs, "should delete existing fullsend schedule")
require.Len(t, fake.CreatedSchedules, 1)
require.Len(t, fake.CreatedSchedules, 2)
}

func TestCleanupGitLabPipelineSchedules_ListError(t *testing.T) {
Expand All @@ -445,7 +486,7 @@ func TestCleanupGitLabPipelineSchedules_DeleteError(t *testing.T) {
fake := &forge.FakeClient{
PipelineSchedules: map[string][]forge.PipelineSchedule{
"group/project": {
{ID: 1, Description: "fullsend fast poll", Active: true},
{ID: 1, Description: "fullsend slash poll", Active: true},
},
},
}
Expand All @@ -471,7 +512,8 @@ func TestHealGitLabResourceGroups(t *testing.T) {
mux.HandleFunc("/api/v4/projects/mygroup%2Fmyproject/resource_groups", func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode([]map[string]any{
{"key": "fullsend-poll", "process_mode": "unordered"},
{"key": "fullsend-poll-slash", "process_mode": "unordered"},
{"key": "fullsend-poll-events", "process_mode": "unordered"},
{"key": "fullsend-triage-mr-1", "process_mode": "newest_first"},
{"key": "production", "process_mode": "oldest_first"},
})
Expand Down Expand Up @@ -504,8 +546,20 @@ func TestHealGitLabResourceGroups(t *testing.T) {
healGitLabResourceGroups(ctx, glClient, printer, "mygroup", "myproject")

// Should toggle only fullsend-prefixed groups, not "production".
assert.Len(t, toggleCalls, 4, "expected 2 fullsend groups × 2 toggles each")
assert.Contains(t, buf.String(), "Healed 2 resource group(s)")
assert.Len(t, toggleCalls, 6, "expected 3 fullsend groups × 2 toggles each")
assert.Contains(t, buf.String(), "Healed 3 resource group(s)")

// Verify mode-aware target: events gets oldest_first, others get newest_first.
for _, tc := range toggleCalls {
if tc.Mode == "unordered" {
continue
}
if strings.HasSuffix(tc.Key, "poll-events") {
assert.Equal(t, "oldest_first", tc.Mode, "events resource group should use oldest_first")
} else {
assert.Equal(t, "newest_first", tc.Mode, "%s should use newest_first", tc.Key)
}
}
})

t.Run("handles list error gracefully", func(t *testing.T) {
Expand Down
Loading
Loading