BREAKING CHANGE: upgrade Milvus to 3.0

docs/assets/env_example.env

@@ -62,6 +62,10 @@ MINIO_SECRET_KEY=minioadmin
 POSTGRES_PASSWORD=postgres
 # POSTGRES_USER=root
 
+# Fresh Milvus 3 installations can keep the default queue selection. During an
+# upgrade, set this to the queue already used by the existing deployment.
+# MILVUS_MQ_TYPE=rocksmq
+
 # ── API and Chainlit (chat interface) authentication ──────────────────────────────
 # Bearer token that bootstraps the admin user and guards the API.
 # ⚠️  Production: replace with a strong value, e.g. `openssl rand -hex 16`.
@@ -117,4 +121,4 @@ RAY_ENABLE_UV_RUN_RUNTIME_ENV=0
 # RAY_memory_monitor_refresh_ms=0
 
 # ── Logging (DEBUG on dev, INFO on prod) ──
-LOG_LEVEL=DEBUG
+LOG_LEVEL=DEBUG

docs/content/docs/documentation/env_vars.md

@@ -265,6 +265,7 @@ For an opt-in named-volume profile, copy the values from `infra/compose/.env.nam
 | `VLLM_CACHE` | `/root/.cache/huggingface` | Hugging Face cache used by vLLM, reranker, and transcriber services. |
 | `DB_VOLUME` | `../../db` | PostgreSQL data mounted at `/var/lib/postgresql/data`. |
 | `MILVUS_VOLUME_DIRECTORY` | `./volumes` | Parent directory for Milvus, etcd, and MinIO host-path storage. |
+| `MILVUS_MQ_TYPE` | `default` | Milvus message queue. Keep the existing value during a version upgrade; fresh installations can use the default. |
 | `MILVUS_COMPOSE` | `milvus/milvus.yaml` | Milvus compose include. Use `milvus/milvus.named-volumes.yaml` for the named-volume profile. |
 | `ETCD_VOLUME` | `etcd` | Milvus etcd named volume, used only with `MILVUS_COMPOSE=milvus/milvus.named-volumes.yaml`. |
 | `MINIO_VOLUME` | `minio` | Milvus object storage named volume, used only with `MILVUS_COMPOSE=milvus/milvus.named-volumes.yaml`. |

docs/content/docs/documentation/milvus_migration.mdx

@@ -5,9 +5,11 @@ title: Milvus Migrations
 import { Tabs, TabItem } from '@astrojs/starlight/components';
 
 # Milvus Upgrade
-OpenRAG has been upgraded from Milvus **2.5.4** to **2.6.11** to leverage the enhancements introduced in the latest releases, particularly the new temporal querying capabilities added in version **2.6.6+**.
+OpenRAG now uses Milvus **3.0.0** with PyMilvus **3.0.1**. Existing Milvus 2.6 deployments can keep their data, but they must prepare the Milvus data directory before starting the new image because Milvus 3 runs as a non-root user.
 
-## What's New in 2.6.x
+Fresh installations need no manual ownership step. The Compose stack initializes empty bind mounts and named volumes before the server starts. If existing files are not already owned by the Milvus 3 user, startup stops with a migration error instead of risking a partially writable deployment. The procedure below remains mandatory for existing data in either storage mode because all files created by Milvus 2.6 must be transferred to the Milvus 3 user.
+
+## Temporal support introduced in 2.6.x
 
 Milvus 2.6.6+ introduced the **`TIMESTAMPTZ`** field type, which enables:
 
@@ -46,12 +48,115 @@ results = client.query(
 ## Milvus Version Upgrade Steps
 
 :::danger[Who needs this migration?]
-This migration is only required if you are upgrading from **OpenRAG <= 1.1.7**, which shipped with Milvus <= 2.5.x. If your deployment already runs Milvus 2.6.x, skip this section.
+Do not skip this migration when upgrading an existing deployment:
+
+- If you already run Milvus **2.6.x**, follow the 2.6-to-3.0 procedure below.
+- If you run OpenRAG **1.1.7 or earlier with Milvus 2.5.x**, first complete the legacy intermediate upgrade to 2.6.11, then follow the 2.6-to-3.0 procedure.
+- If your Milvus server is **earlier than 2.5.x**, do not use the legacy path below. Determine the required intermediate and metadata migrations from the official guide before continuing.
 :::
 
 > For the full official reference, see the [Milvus upgrade guide](https://milvus.io/docs/upgrade_milvus_standalone-docker.md#Upgrade-process).
 
-### Step 1 — Upgrade Milvus to 2.5.16 (intermediate step)
+### Upgrade from Milvus 2.6.x to 3.0.0
+
+Milvus 2.6 containers ran as root, so existing bind-mounted files and Docker named-volume contents are commonly owned by `root:root`. Milvus 3 runs as UID/GID `999:999` and cannot start until it owns the directory mounted at `/var/lib/milvus`.
+
+The message-queue type must also remain unchanged during the upgrade. Before stopping Milvus 2.6, inspect its startup logs and identify the effective `mqType` or `walName`:
+
+```bash
+docker compose logs --no-color milvus \
+  | grep -E 'mqType=|walName=' \
+  | tail -20
+```
+
+Set `MILVUS_MQ_TYPE` in the Compose `.env` file to the detected value, such as `rocksmq` or `woodpecker`. If the logs do not show a clear value, confirm it in the Milvus WebUI configuration view before continuing. Do not combine the version upgrade with a queue migration; use the separate [Milvus queue-switch procedure](https://milvus.io/docs/switch-rocksmq-woodpecker.md) after version 3 is healthy.
+
+:::danger[Back up before changing ownership]
+Stop writes and create a verified backup of the Milvus, etcd, and MinIO data before continuing. An image-only downgrade is not a safe rollback after Milvus 3 has written data.
+:::
+
+Run the procedure for the storage profile selected in `infra/compose/.env`.
+
+#### Bind-mounted storage
+
+While the Milvus 2.6 stack still exists, confirm that `/var/lib/milvus` is a bind mount:
+
+```bash
+MILVUS_CONTAINER="$(docker compose ps -q milvus)"
+docker inspect "$MILVUS_CONTAINER" \
+  --format '{{range .Mounts}}{{if eq .Destination "/var/lib/milvus"}}type={{.Type}} source={{.Source}}{{end}}{{end}}'
+# Expected: type=bind
+```
+
+Stop the stack, take the backup described above, and then transfer ownership from a temporary service container. Running the change through Docker makes it apply to the daemon-side mount and preserves the correct ownership mapping with remote, rootless, and user-namespaced Docker. Do not add `--volumes` to the shutdown command.
+
+```bash
+docker compose down
+docker compose run --rm --no-deps --user 0:0 \
+  --entrypoint chown milvus -R 999:999 /var/lib/milvus
+```
+
+#### Docker named-volume storage
+
+Docker Desktop and remote Docker contexts do not expose a named volume's daemon-side path to the local machine. Stop the stack without removing volumes, take the backup described above, and transfer ownership from a temporary service container instead:
+
+```bash
+docker compose down
+docker compose run --rm --no-deps --user 0:0 \
+  --entrypoint chown milvus -R 999:999 /var/lib/milvus
+```
+
+The command must run with `MILVUS_COMPOSE=milvus/milvus.named-volumes.yaml`, matching the existing deployment. Do not use `docker compose down --volumes` during this procedure.
+
+Update OpenRAG, then pull and start the dependencies followed by Milvus:
+
+```bash
+docker compose pull milvus
+docker compose up -d etcd minio
+docker compose up -d milvus
+```
+
+Do not restart OpenRAG until Milvus is healthy and the logs contain no permission or migration errors:
+
+```bash
+docker compose ps milvus
+docker compose logs --tail 100 milvus
+docker inspect "$(docker compose ps -q milvus)" --format '{{.Config.Image}}'
+# Expected: milvusdb/milvus:v3.0.0
+```
+
+Once those checks pass, start the complete stack:
+
+```bash
+docker compose up -d
+```
+
+### Helm deployments
+
+The bundled Helm chart now installs the Milvus 3-compatible chart and image. Before upgrading an existing release, stop writes and back up the MinIO and etcd volumes, together with any standalone or log volume enabled through custom values.
+
+The upstream repository index does not yet publish chart 5.0.26, so OpenRAG currently uses chart 5.0.25 with the Milvus 3 image selected explicitly. Resources rendered by that chart can therefore retain the label `app.kubernetes.io/version: "2.6.21"`. This label describes the chart's default application version, not the running server; verify the container image as shown below.
+
+Milvus 3 runs with GID `999`. The chart sets `fsGroup: 999` with the `Always` policy so Kubernetes recursively makes supported mounted volumes writable before Milvus starts. The first startup can therefore take longer when a volume contains many files.
+
+Some storage drivers do not support `fsGroup` ownership changes. Check the driver's `fsGroupPolicy` before upgrading. If it is `None`, migrate the affected Milvus volume to group `999` using the storage provider's procedure before starting Milvus 3; otherwise the pods can fail with permission errors.
+
+OpenShift's restricted security context constraints normally assign an `fsGroup` from the project's permitted range and can reject the fixed group `999`. On those clusters, set `milvus.securityContext` to `null` in the OpenRAG values so admission can assign the allowed group. Confirm the admitted pod security context and volume ownership in a staging namespace before upgrading production.
+
+After the upgrade, confirm that every Milvus workload is ready and uses the expected image:
+
+```bash
+kubectl get pods -n <namespace> -l app.kubernetes.io/name=milvus
+kubectl get pods -n <namespace> -l app.kubernetes.io/name=milvus \
+  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}'
+# Expected Milvus image: milvusdb/milvus:v3.0.0
+```
+
+### Legacy path: upgrade Milvus 2.5.x to 2.6.11
+
+This intermediate path is only for deployments upgrading from **OpenRAG &lt;= 1.1.7 whose current Milvus server is 2.5.x**. Do not jump directly from Milvus 2.5.x to 3.0.0. Earlier Milvus versions can require additional intermediate or metadata migrations that are outside this procedure.
+
+#### Step 1 — Upgrade Milvus to 2.5.16
 
 :::caution[Do not update OpenRAG yet]
 During this step, keep your current version of OpenRAG (&lt; 1.1.7) running. Only the Milvus image is changed here. OpenRAG itself is updated in Step 2.
@@ -82,9 +187,19 @@ docker inspect milvus-standalone --format '{{ .Config.Image }}'
 # Expected: milvusdb/milvus:v2.5.16
 ```
 
-### Step 2 — Update OpenRAG
+#### Preserve the message queue
 
-Once Milvus 2.5.16 is healthy, stop all services and update OpenRAG to the new version. The updated `infra/compose/milvus/milvus.yaml` already includes Milvus 2.6.11 and the required MinIO and etcd upgrades.
+OpenRAG 2.5 standalone deployments normally use RocksMQ, but verify the effective source queue using the log check in the 2.6-to-3.0 procedure above. Keep that detected value in every Compose configuration used for the 2.6.11 and 3.0.0 steps.
+
+Before starting the intermediate OpenRAG release, add `MQ_TYPE: <detected-value>` to the Milvus service's environment. When moving to the current release, set the corresponding value in the Compose `.env` file and keep it until the upgrade is validated:
+
+```bash
+MILVUS_MQ_TYPE=<detected-value>
+```
+
+#### Step 2 — Upgrade Milvus to 2.6.11
+
+Once Milvus 2.5.16 is healthy, stop all services and move to an OpenRAG release that still packages Milvus 2.6.11, such as OpenRAG 2.1.0. This intermediate release also provides the required MinIO and etcd versions.
 
 ```bash
 docker compose down
@@ -109,6 +224,8 @@ docker inspect milvus-standalone --format '{{ .Config.Image }}'
 # Expected: milvusdb/milvus:v2.6.11
 ```
 
+After verifying that the collections are readable and searchable on 2.6.11, continue with the 2.6-to-3.0 procedure above.
+
 ## Schema Migrations
 
 OpenRAG ships a generic migration runner that discovers and applies all pending Milvus schema migrations in order. You never need to invoke individual migration scripts by hand.
@@ -302,4 +419,4 @@ def downgrade(client: MilvusClient, collection_name: str, dry_run: bool = False)
     ...
 ```
 
-Use `1.add_temporal_fields.py` as a reference implementation for the full upgrade/downgrade pattern.
+Use `1.add_temporal_fields.py` as a reference implementation for the full upgrade/downgrade pattern.

infra/charts/openrag-stack/Chart.lock

@@ -7,9 +7,9 @@ dependencies:
   version: 18.7.3
 - name: milvus
   repository: https://zilliztech.github.io/milvus-helm/
-  version: 5.0.0
+  version: 5.0.25
 - name: vllm-stack
   repository: https://vllm-project.github.io/production-stack
   version: 0.1.11
-digest: sha256:1ea11f53796e3196848d5d46cdff5a6bdeeb012427c1951ea646a22b176adec6
-generated: "2026-06-10T14:51:21.005020805+02:00"
+digest: sha256:50f412b87a0f4b9fccb92b02a0ea0d21fa7d0343b86394077e4ac05cb7c16b61
+generated: "2026-08-25T13:53:27.648198879Z"

infra/charts/openrag-stack/Chart.yaml

@@ -26,7 +26,10 @@ dependencies:
     repository: "https://charts.bitnami.com/bitnami"
     condition: postgresql.enabled
   - name: milvus
-    version: "5.0.0"
+    # The Milvus 3 chart is 5.0.26, but that release is not yet present in the
+    # published repository index. Keep 5.0.25 and override its image tag in
+    # values.yaml until 5.0.26 can be resolved through helm dependency update.
+    version: "5.0.25"
     repository: "https://zilliztech.github.io/milvus-helm/"
     condition: milvus.enabled
   - name: vllm-stack

infra/charts/openrag-stack/values.yaml

@@ -171,8 +171,17 @@ milvus:
   # note on it above. Without this, the milvus sub-chart names its own proxy
   # Service from the real Release.Name instead, which won't match VDB_HOST.
   fullnameOverride: "openrag-milvus"
+  # Chart 5.0.25 still defaults to Milvus 2.6.21. This override can be removed
+  # when the upstream 5.0.26 chart is available from its repository index.
   image:
-    all: { tag: "v2.6.0" }
+    all: { tag: "v3.0.0" }
+  # Milvus 3 runs as UID/GID 999. Applying this group on every mount lets
+  # Kubernetes migrate supported existing volumes before the non-root process
+  # starts. Storage drivers that do not support fsGroup require an
+  # administrator-managed ownership migration before the Helm upgrade.
+  securityContext:
+    fsGroup: 999
+    fsGroupChangePolicy: Always
   cluster: { enabled: true }
   pulsarv3: { enabled: false }
   woodpecker: { enabled: true }

infra/compose/.env.example

@@ -62,6 +62,10 @@ MINIO_SECRET_KEY=minioadmin
 POSTGRES_PASSWORD=postgres
 # POSTGRES_USER=root
 
+# Fresh Milvus 3 installations can keep the default queue selection. During an
+# upgrade, set this to the queue already used by the existing deployment.
+# MILVUS_MQ_TYPE=rocksmq
+
 # ── API and Chainlit (chat interface) authentication ──────────────────────────────
 # Bearer token that bootstraps the admin user and guards the API.
 # ⚠️  Production: replace with a strong value, e.g. `openssl rand -hex 16`.
@@ -117,4 +121,4 @@ RAY_ENABLE_UV_RUN_RUNTIME_ENV=0
 # RAY_memory_monitor_refresh_ms=0
 
 # ── Logging (DEBUG on dev, INFO on prod) ──
-LOG_LEVEL=DEBUG
+LOG_LEVEL=DEBUG

infra/compose/milvus/milvus.named-volumes.yaml

@@ -29,14 +29,42 @@ services:
       timeout: 20s
       retries: 3
 
+  milvus-init:
+    image: milvusdb/milvus:v3.0.0
+    user: "0:0"
+    environment:
+      LD_PRELOAD: ""
+    entrypoint: ["/bin/sh", "-ec"]
+    command:
+      - |
+        incompatible_path="$$(find /var/lib/milvus -mindepth 1 \( ! -uid 999 -o ! -gid 999 \) -print -quit)"
+        if [ -n "$$incompatible_path" ]; then
+          echo "Existing Milvus data is not owned by UID/GID 999. Back up the deployment and follow the Milvus 2.6-to-3.0 migration guide before starting Milvus 3." >&2
+          exit 1
+        fi
+        chown 999:999 /var/lib/milvus
+    volumes:
+      - ${MILVUS_VOLUME:-milvus}:/var/lib/milvus
+    read_only: true
+    cap_drop:
+      - ALL
+    cap_add:
+      - CHOWN
+      # Let the ownership guard inspect restrictive restored directories
+      # without granting write bypass.
+      - DAC_READ_SEARCH
+    restart: "no"
+
   milvus:
-    image: milvusdb/milvus:v2.6.11
+    image: milvusdb/milvus:v3.0.0
     command: ["milvus", "run", "standalone"]
     # Run under Docker's default seccomp profile (do not disable syscall
     # filtering). If a specific kernel needs a wider profile, supply a vetted
     # custom profile rather than seccomp:unconfined.
     environment:
       ETCD_ENDPOINTS: etcd:2379
+      ETCD_AUTH_ENABLED: "false"
+      MQ_TYPE: ${MILVUS_MQ_TYPE:-default}
       MINIO_ADDRESS: minio:9000
       MINIO_ACCESS_KEY_ID: ${MINIO_ACCESS_KEY:?Set MINIO_ACCESS_KEY in your .env}
       MINIO_SECRET_ACCESS_KEY: ${MINIO_SECRET_KEY:?Set MINIO_SECRET_KEY in your .env}
@@ -51,8 +79,12 @@ services:
     # ports:
     #   - "${VDB_PORT:-19530}:${VDB_iPORT:-19530}"
     depends_on:
-      - "etcd"
-      - "minio"
+      etcd:
+        condition: service_started
+      minio:
+        condition: service_started
+      milvus-init:
+        condition: service_completed_successfully
 
 volumes:
   etcd:

infra/compose/milvus/milvus.yaml

@@ -29,14 +29,42 @@ services:
       timeout: 20s
       retries: 3
 
+  milvus-init:
+    image: milvusdb/milvus:v3.0.0
+    user: "0:0"
+    environment:
+      LD_PRELOAD: ""
+    entrypoint: ["/bin/sh", "-ec"]
+    command:
+      - |
+        incompatible_path="$$(find /var/lib/milvus -mindepth 1 \( ! -uid 999 -o ! -gid 999 \) -print -quit)"
+        if [ -n "$$incompatible_path" ]; then
+          echo "Existing Milvus data is not owned by UID/GID 999. Back up the deployment and follow the Milvus 2.6-to-3.0 migration guide before starting Milvus 3." >&2
+          exit 1
+        fi
+        chown 999:999 /var/lib/milvus
+    volumes:
+      - ${MILVUS_VOLUME_DIRECTORY:-./volumes}/milvus:/var/lib/milvus
+    read_only: true
+    cap_drop:
+      - ALL
+    cap_add:
+      - CHOWN
+      # Let the ownership guard inspect restrictive restored directories
+      # without granting write bypass.
+      - DAC_READ_SEARCH
+    restart: "no"
+
   milvus:
-    image: milvusdb/milvus:v2.6.11
+    image: milvusdb/milvus:v3.0.0
     command: ["milvus", "run", "standalone"]
     # Run under Docker's default seccomp profile (do not disable syscall
     # filtering). If a specific kernel needs a wider profile, supply a vetted
     # custom profile rather than seccomp:unconfined.
     environment:
       ETCD_ENDPOINTS: etcd:2379
+      ETCD_AUTH_ENABLED: "false"
+      MQ_TYPE: ${MILVUS_MQ_TYPE:-default}
       MINIO_ADDRESS: minio:9000
       # Milvus must authenticate to MinIO with the same credentials; otherwise
       # it falls back to the built-in minioadmin default and fails to connect.
@@ -53,5 +81,9 @@ services:
     # ports:
     #   - "${VDB_PORT:-19530}:${VDB_iPORT:-19530}"
     depends_on:
-      - "etcd"
-      - "minio"
+      etcd:
+        condition: service_started
+      minio:
+        condition: service_started
+      milvus-init:
+        condition: service_completed_successfully

openrag/services/storage/__init__.py

@@ -7,7 +7,7 @@
   :class:`ConnectionManager` with every repository implementation under
   :mod:`services.persistence`, satisfying
   :class:`core.ports.catalog_store.CatalogStore`.
-* :class:`milvus_store.MilvusVectorStore` — Milvus 2.6 backed vector ops
+* :class:`milvus_store.MilvusVectorStore` — Milvus 3.0 backed vector ops
   satisfying :class:`core.vector_stores.VectorStore`.
 
 The Ray actor that callers know today lives at