Skip to content

Conversation

@ThomasBrady
Copy link
Contributor

@ThomasBrady ThomasBrady commented Sep 17, 2025

Issue:

The Calyptia Fleet plugin failed to apply configurations on first run when starting with an empty fleet directory. The plugin would download a configuration but reject it because the timestamp comparison logic (check_timestamp_is_newer()) would find the just-downloaded file and determine its timestamp was not newer than itself.

Root Cause:

The timestamp comparison was too strict - it only allowed new configs if their timestamp was strictly newer than existing files. This prevented the initial fleet configuration from being applied when no fleet config was currently active.

Fix:

Modified the condition in get_calyptia_file() to also check if we're currently using a fleet configuration. The config is now applied if:

  • The timestamp is newer than existing files (normal update case), OR
  • We're not currently using a fleet config (bootstrap/first run case)
    This allows the fleet plugin to properly bootstrap itself on Windows when starting with an empty directory while maintaining the existing timestamp-based update logic for subsequent configuration changes.

Excerpts from logs demonstrating the issue and fix:

Empty Fleet Directory Error Log (v4.0.9)

Initial startup with empty fleet directory:

[2025/09/12 18:54:24] [debug] [input:calyptia_fleet:calyptia_fleet.2] loading configuration file
[2025/09/12 18:54:24] [error] unable to open valid handle for: C:\Program Files\Fluent-Bit\conf\foo\4765fdc9103144ef50869d318b8de90f766a74781356efcef01e78411517c125\test-cba-windows-tmp-dir\*.conf
[2025/09/12 18:54:24] [error] [input:calyptia_fleet:calyptia_fleet.2] unable to read fleet directory for config files: C:\Program Files\Fluent-Bit\conf\foo\4765fdc9103144ef50869d318b8de90f766a74781356efcef01e78411517c125\test-cba-windows-tmp-dir\*.conf
[2025/09/12 18:54:24] [ warn] [input:calyptia_fleet:calyptia_fleet.2] unable to find latest configuration file

First download succeeds:

[2025/09/12 18:54:39] [ info] [input:calyptia_fleet:calyptia_fleet.2] creating config file with timestamp 1757461288
[2025/09/12 18:54:39] [ info] [input:calyptia_fleet:calyptia_fleet.2] fleet API returned new config (none -> 1757461288)
[2025/09/12 18:54:40] [ info] [input:calyptia_fleet:calyptia_fleet.2] loading configuration from C:\Program Files\Fluent-Bit\conf\foo\4765fdc9103144ef50869d318b8de90f766a74781356efcef01e78411517c125\test-cba-windows-tmp-dir\1757461288.conf.
[2025/09/12 18:54:40] [ info] [input:calyptia_fleet:calyptia_fleet.2] changing to config dir: C:\Program Files\Fluent-Bit\conf\foo\4765fdc9103144ef50869d318b8de90f766a74781356efcef01e78411517c125\test-cba-windows-tmp-dir\1757461288

Reload fails:

[2025/09/12 18:54:45] [ info] reloading instance pid=7884 tid=000002299F72E100
[2025/09/12 18:54:45] [error] [D:\a\fluent-bit\fluent-bit\src\config_format\flb_cf_fluentbit.c:458 errno=2] No such file or directory

Subsequent attempts blocked:

[2025/09/12 18:54:54] [debug] [input:calyptia_fleet:calyptia_fleet.2] existing file with timestamp 1757461288 >= new timestamp 1757461288
[2025/09/12 18:54:54] [debug] [input:calyptia_fleet:calyptia_fleet.2] not creating file with timestamp 1757461288 since it is not newer than existing files

This pattern repeats every 15 seconds:

[2025/09/12 18:55:09] [debug] [input:calyptia_fleet:calyptia_fleet.2] existing file with timestamp 1757461288 >= new timestamp 1757461288
[2025/09/12 18:55:09] [debug] [input:calyptia_fleet:calyptia_fleet.2] not creating file with timestamp 1757461288 since it is not newer than existing files

[2025/09/12 18:55:24] [debug] [input:calyptia_fleet:calyptia_fleet.2] existing file with timestamp 1757461288 >= new timestamp 1757461288
[2025/09/12 18:55:24] [debug] [input:calyptia_fleet:calyptia_fleet.2] not creating file with timestamp 1757461288 since it is not newer than existing files

Fixed logs:

Initial startup - same empty directory errors:

[2025/09/15 18:59:26.327681100] [debug] [input:calyptia_fleet:calyptia_fleet.2] fleet_config_deref: ref file does not exist: C:\Program Files\Fluent-Bit\conf\foo\4765fdc9103144ef50869d318b8de90f766a74781356efcef01e78411517c125\test-cba-windows-tmp-dir\new.ref
[2025/09/15 18:59:26.327807400] [debug] [input:calyptia_fleet:calyptia_fleet.2] fleet_config_deref: ref file does not exist: C:\Program Files\Fluent-Bit\conf\foo\4765fdc9103144ef50869d318b8de90f766a74781356efcef01e78411517c125\test-cba-windows-tmp-dir\new.ref
[2025/09/15 18:59:26.327881800] [debug] [input:calyptia_fleet:calyptia_fleet.2] fleet_config_deref: ref file does not exist: C:\Program Files\Fluent-Bit\conf\foo\4765fdc9103144ef50869d318b8de90f766a74781356efcef01e78411517c125\test-cba-windows-tmp-dir\cur.ref

Fix applied - detects not using fleet config:

[2025/09/15 18:59:41.576105200] [ info] [input:calyptia_fleet:calyptia_fleet.2] creating config file with timestamp 1757461288 (not currently using fleet config)
[2025/09/15 18:59:41.576800100] [ info] [input:calyptia_fleet:calyptia_fleet.2] fleet API returned new config (none -> 1757461288)

Successful reload:

[2025/09/15 18:59:46.658382600] [ info] reloading configuration from path: C:\Program Files\Fluent-Bit\conf\foo\4765fdc9103144ef50869d318b8de90f766a74781356efcef01e78411517c125\test-cba-windows-tmp-dir\1757461288.conf
[2025/09/15 18:59:47.451551900] [ info] reloading instance pid=4240 tid=000001F103081040
[2025/09/15 18:59:49.510414600] [ info] [reload] start everything

Configuration successfully applied:

[2025/09/15 18:59:49.525789400] [debug] [input:calyptia_fleet:calyptia_fleet.2] we are already using a configuration file: C:\Program Files\Fluent-Bit\conf\foo\4765fdc9103144ef50869d318b8de90f766a74781356efcef01e78411517c125\test-cba-windows-tmp-dir\1757461288.conf

Subsequent checks correctly skip same timestamp:

[2025/09/15 19:00:04.688088300] [debug] [input:calyptia_fleet:calyptia_fleet.2] existing file with timestamp 1757461288 >= new timestamp 1757461288
[2025/09/15 19:00:04.689932800] [debug] [input:calyptia_fleet:calyptia_fleet.2] --not creating file with timestamp 1757461288 since it is not newer than existing files

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

  • Bug Fixes

    • On first run, the system will create a timestamped configuration file even if an equal or newer file exists, as long as the instance is not yet using fleet configuration—preventing startup/configuration failures in initial setups.
  • New Features

    • Success logs now explicitly note when the instance is “not currently using fleet configuration,” providing clearer feedback during initial setup and troubleshooting.

@coderabbitai
Copy link

coderabbitai bot commented Sep 17, 2025

Walkthrough

Updates get_calyptia_file to allow creating a timestamped config on first run by checking both whether an existing file is newer and whether a fleet config is currently in use; it aborts only when an existing file is newer AND fleet config is active. Success log notes when fleet config is not used.

Changes

Cohort / File(s) Summary
Calyptia Fleet plugin logic
plugins/in_calyptia_fleet/in_calyptia_fleet.c
get_calyptia_file adds locals flb_ctx, is_newer, not_using_fleet; computes is_newer = check_timestamp_is_newer(ctx, last_modified) and derives not_using_fleet from Fluent Bit context; changes abort condition to require both is_newer == FLB_FALSE AND not_using_fleet == FLB_FALSE; otherwise creates/writes the timestamped config; success log appends “(not currently using fleet config)” when applicable.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant Plugin as CalyptiaFleet Plugin
  participant getFile as get_calyptia_file()
  participant Check as check_timestamp_is_newer()
  participant Config as Fluent Bit Context
  participant FS as Filesystem

  Plugin->>getFile: request config (dst == NULL)
  getFile->>Check: compare last_modified vs existing
  Check-->>getFile: is_newer (TRUE/FALSE)
  getFile->>Config: inspect current config usage
  Config-->>getFile: not_using_fleet (TRUE/FALSE)

  alt is_newer == FLB_FALSE and not_using_fleet == FLB_FALSE
    getFile-->>Plugin: abort creation (existing newer file in use)
  else
    getFile->>FS: create/write timestamped config
    FS-->>getFile: write success
    getFile-->>Plugin: return success (log notes if not_using_fleet)
  end
Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

Suggested reviewers

  • koleini
  • fujimotos
  • pwhelan

Poem

I hop and nudge the config tree tonight,
I check the stamps by soft moonlight,
If Fleet’s not driving, still I write,
A tiny file to set things right,
Rabbit cheers — a timestamped bite 🐇✨

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check ✅ Passed The title "in_calyptia_fleet: create fleet file if not using fleet." succinctly identifies the affected plugin and the primary behavior change (creating a fleet config when the agent is not using fleet), which matches the PR summary about enabling first-run bootstrapping; it is specific, concise, and relevant to the changeset.
✨ Finishing touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Tip

👮 Agentic pre-merge checks are now available in preview!

Pro plan users can now enable pre-merge checks in their settings to enforce checklists before merging PRs.

  • Built-in checks – Quickly apply ready-made checks to enforce title conventions, require pull request descriptions that follow templates, validate linked issues for compliance, and more.
  • Custom agentic checks – Define your own rules using CodeRabbit’s advanced agentic capabilities to enforce organization-specific policies and workflows. For example, you can instruct CodeRabbit’s agent to verify that API documentation is updated whenever API schema files are modified in a PR. Note: Upto 5 custom checks are currently allowed during the preview period. Pricing for this feature will be announced in a few weeks.

Please see the documentation for more information.

Example:

reviews:
  pre_merge_checks:
    custom_checks:
      - name: "Undocumented Breaking Changes"
        mode: "warning"
        instructions: |
          Pass/fail criteria: All breaking changes to public APIs, CLI flags, environment variables, configuration keys, database schemas, or HTTP/GraphQL endpoints must be documented in the "Breaking Change" section of the PR description and in CHANGELOG.md. Exclude purely internal or private changes (e.g., code not exported from package entry points or explicitly marked as internal).

Please share your feedback with us on this Discord post.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@edsiper
Copy link
Member

edsiper commented Sep 17, 2025

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (3)
plugins/in_calyptia_fleet/in_calyptia_fleet.c (3)

1201-1203: Default to “not using fleet” when context/config is unavailable (safer bootstrap).

If flb_context_get() or flb_ctx->config is NULL, we currently treat this as “using fleet,” which can block first‑run in edge cases. Compute not_using_fleet via is_fleet_config with a NULL‑tolerant cfg pointer.

Apply this diff:

-        flb_ctx_t *flb_ctx = flb_context_get();
-        int is_newer = check_timestamp_is_newer(ctx, last_modified);
-        int not_using_fleet = (flb_ctx && flb_ctx->config && is_fleet_config(ctx, flb_ctx->config) == FLB_FALSE);
+        flb_ctx_t *flb_ctx = flb_context_get();
+        struct flb_config *cur_cfg = flb_ctx ? flb_ctx->config : NULL;
+        int is_newer = check_timestamp_is_newer(ctx, last_modified);
+        int not_using_fleet = (is_fleet_config(ctx, cur_cfg) == FLB_FALSE);

1194-1196: Parse Last-Modified as GMT (avoid TZ/DST skew).

mktime() interprets tm as local time. Last‑Modified is GMT; use timegm/_mkgmtime to avoid skew.

Apply this diff:

-        flb_strptime(fbit_last_modified, "%a, %d %B %Y %H:%M:%S GMT", &tm_last_modified);
-        last_modified = mktime(&tm_last_modified.tm);
+        /* RFC 7231 uses abbreviated month (%b) and GMT; convert as UTC */
+        if (flb_strptime(fbit_last_modified, "%a, %d %b %Y %H:%M:%S GMT", &tm_last_modified) == NULL) {
+            goto client_error;
+        }
+#if defined(_WIN32)
+        last_modified = _mkgmtime(&tm_last_modified.tm);
+#else
+        last_modified = timegm(&tm_last_modified.tm);
+#endif

1197-1200: Tighten the comment to mirror the predicate.

Make the intent crystal clear for future readers.

Apply this diff:

-        /* Check if there are existing files with timestamps >= new timestamp
-        * OR if we're not currently using a fleet config (first run scenario)
-        */
+        /* Proceed if (is_newer || not_using_fleet): allow bootstrap while preserving timestamp gating */
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 60db310 and 58aeabc.

📒 Files selected for processing (1)
  • plugins/in_calyptia_fleet/in_calyptia_fleet.c (1 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
plugins/in_calyptia_fleet/in_calyptia_fleet.c (1)
src/flb_lib.c (1)
  • flb_context_get (993-999)
🔇 Additional comments (1)
plugins/in_calyptia_fleet/in_calyptia_fleet.c (1)

1197-1219: Good bootstrap gate: allow create when not currently using fleet config.

Condition now correctly permits first‑run creation even on equal timestamps; logs are clear.

@ThomasBrady ThomasBrady force-pushed the thomasbrady/fix-empty-fleet-dir-no-upgrade branch 2 times, most recently from 22bc3a6 to 583b75b Compare September 17, 2025 17:28
@ThomasBrady ThomasBrady force-pushed the thomasbrady/fix-empty-fleet-dir-no-upgrade branch from 583b75b to 6921a62 Compare September 17, 2025 17:31
@ThomasBrady
Copy link
Contributor Author

Thanks for TAL, updated.

@ThomasBrady ThomasBrady force-pushed the thomasbrady/fix-empty-fleet-dir-no-upgrade branch from bb8136e to 29050d6 Compare September 17, 2025 17:44
@pwhelan pwhelan changed the title Create fleet file if not using fleet in_calyptia_fleet: create fleet file if not using fleet. Sep 17, 2025
@ThomasBrady ThomasBrady requested a review from edsiper September 17, 2025 19:00
@edsiper edsiper merged commit 074e39b into fluent:master Sep 17, 2025
53 of 56 checks passed
@edsiper
Copy link
Member

edsiper commented Sep 17, 2025

I think we will need a backport of this one for branch 4.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants