Skip to content

Load more data for outlook stats #977

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
elie222 merged 2 commits into from
Nov 16, 2025
Merged

Load more data for outlook stats #977

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
23c9406
Load more data for outlook stats
elie222 Nov 16, 2025
41a51a6
remove dead code
elie222 Nov 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 2 additions & 20 deletions apps/web/utils/email/microsoft.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,6 @@ import {
getOrCreateOutlookFolderIdByName,
getOutlookFolderTree,
} from "@/utils/outlook/folders";
import { hasUnquotedParentFolderId } from "@/utils/outlook/message";
import { extractSignatureFromHtml } from "@/utils/email/signature-extraction";
import { moveMessagesForSenders } from "@/utils/outlook/batch";

Expand Down Expand Up @@ -754,37 +753,20 @@ export class OutlookProvider implements EmailProvider {
hasDateFilters: dateFilters.length > 0,
});

// Check if the query already contains parentFolderId as an unquoted identifier
// If it does, skip applying the default folder filter to avoid conflicts
const queryHasParentFolderId =
originalQuery && hasUnquotedParentFolderId(originalQuery);

// Get folder IDs to get the inbox folder ID
const folderIds = await getFolderIds(this.client);
const inboxFolderId = folderIds.inbox;

if (!queryHasParentFolderId && !inboxFolderId) {
throw new Error("Could not find inbox folder ID");
}

// Only apply folder filtering if the query doesn't already specify parentFolderId
const folderId = queryHasParentFolderId ? undefined : inboxFolderId;

this.logger.info("Calling queryBatchMessages with separated parameters", {
searchQuery: originalQuery.trim() || undefined,
dateFilters,
maxResults: options.maxResults || 20,
pageToken: options.pageToken,
folderId,
queryHasParentFolderId,
});

// Don't pass folderId - let the API return all folders except Junk/Deleted (auto-excluded)
// Drafts are filtered out in convertMessages
const response = await queryBatchMessages(this.client, {
Comment on lines +763 to 765
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Nov 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

Clarify drafts behavior and Junk/Deleted assumptions for this path

The new behavior (no folderId passed) makes sense for “load more data for Outlook stats” and matches the new queryBatchMessages semantics.

Two points to double‑check:

  1. Drafts filtering:

    • convertMessages explicitly filters out message.isDraft.
    • getDrafts() here calls getMessagesWithPagination({ query: "isDraft eq true", ... }), which ultimately goes through queryBatchMessages → convertMessages, so any drafts returned by Graph will be dropped.
    • If getDrafts() is used anywhere, it will never return results with the current pipeline. You may want either:
      • A dedicated drafts path that bypasses convertMessages, or
      • A way to opt out of the draft filter when the caller explicitly requests drafts.

  2. “API excludes Junk/Deleted by default” comment:

    • If this relies on Microsoft Graph behavior rather than our own filters, it might be worth confirming and/or documenting that guarantee; otherwise, we may need explicit exclusion filters if Junk/Deleted should stay out of stats.

🏁 Script executed:

#!/bin/bash
# Find getDrafts implementation
echo "=== Finding getDrafts implementation ==="
rg -nA 20 'getDrafts\s*\(' --type=ts apps/web/utils/

echo ""
echo "=== Finding convertMessages implementation ==="
rg -nA 30 'export.*convertMessages|function convertMessages' --type=ts apps/web/utils/

echo ""
echo "=== Checking queryBatchMessages and getMessagesWithPagination ==="
rg -nB 5 -A 15 'const queryBatchMessages|function queryBatchMessages' --type=ts apps/web/

Length of output: 11592

🌐 Web query:

Microsoft Graph API /me/mailFolders default behavior Junk Deleted excluded

💡 Result:

Short answer: /me/mailFolders lists the mail folders directly under the mailbox root (shallow), and it does not implicitly exclude JunkEmail or DeletedItems — those well‑known folders exist as children and will be returned if they are direct children. (What does get excluded by default is hidden folders; and /me/messages — a different endpoint — does skip some folders like Sent/Outbox/Junk/Deleted when listing "all messages".) [1][2][3]

Notes and quick tips:

  • To get a specific folder use the well‑known name, e.g. /me/mailFolders/Inbox or /me/mailFolders/JunkEmail. [2]
  • To include hidden folders add ?includeHiddenFolders=true. [1][3]

Sources:

  • MailFolders behavior (list, includeHiddenFolders). [1]
  • Well‑known folder names and examples (Inbox, JunkEmail, DeletedItems). [2]
  • Microsoft Q&A / guidance on traversal (v1.0 shallow vs beta full traversal). [3]

🏁 Script executed:

#!/bin/bash
# Get full queryBatchMessages implementation
echo "=== Full queryBatchMessages implementation ==="
rg -nA 50 'export async function queryBatchMessages' apps/web/utils/outlook/message.ts

echo ""
echo "=== Check for Graph endpoint calls and folder filtering ==="
rg -nB 5 -A 5 'mailFolders|/messages|JunkEmail|DeletedItems' apps/web/utils/outlook/message.ts | head -80

Length of output: 3063

🏁 Script executed:

#!/bin/bash
# Find getMessagesWithPagination in microsoft.ts (the method called by getDrafts)
echo "=== getMessagesWithPagination in microsoft.ts ==="
rg -nB 5 -A 40 'async getMessagesWithPagination' apps/web/utils/email/microsoft.ts

echo ""
echo "=== Check if convertMessages is called directly or in queryBatchMessages ==="
rg -nB 3 -A 3 'convertMessages' apps/web/utils/email/microsoft.ts

Length of output: 2061

🏁 Script executed:

#!/bin/bash
# Get complete queryBatchMessages implementation (it was cut off)
echo "=== Complete queryBatchMessages implementation ==="
rg -nA 120 'export async function queryBatchMessages' apps/web/utils/outlook/message.ts | grep -A 120 'export async function queryBatchMessages'

echo ""
echo "=== Check where convertMessages is called in queryBatchMessages output ==="
rg -n 'convertMessages' apps/web/utils/outlook/message.ts

Length of output: 4856

Fix getDrafts() to bypass draft filtering when explicitly requesting drafts

When getDrafts() calls getMessagesWithPagination({ query: "isDraft eq true", ... }), the results flow through queryBatchMessages → convertMessages, which unconditionally filters out all messages where !message.isDraft (line 370 of apps/web/utils/outlook/message.ts). This means getDrafts() will always return an empty array, breaking the active feature in apps/web/utils/ai/report/fetch.ts:110–128 that relies on it to fetch draft email templates.

Two options to fix:

  • Add a bypass flag to convertMessages to skip draft filtering when explicitly requested, or
  • Create a dedicated drafts-only path that skips convertMessages filtering.

The comment about Junk/Deleted being auto-excluded by the API is accurate per Microsoft Graph /me/messages behavior and does not require changes.

🤖 Prompt for AI Agents 
In apps/web/utils/email/microsoft.ts around lines 763-765, getDrafts() currently
calls queryBatchMessages → convertMessages which unconditionally filters out
drafts; update convertMessages to accept an options parameter (e.g., {
allowDrafts: boolean } defaulting to false) that, when true, bypasses the
"!message.isDraft" filter, then change queryBatchMessages/getDrafts to pass
allowDrafts: true for the drafts path; ensure all other call sites keep default
behavior and update related types/comments accordingly.

searchQuery: originalQuery.trim() || undefined,
dateFilters,
maxResults: options.maxResults || 20,
pageToken: options.pageToken,
folderId,
});

return {
Expand Down
111 changes: 0 additions & 111 deletions apps/web/utils/outlook/message.test.ts
View file
Open in desktop

This file was deleted.

105 changes: 20 additions & 85 deletions apps/web/utils/outlook/message.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,64 +13,6 @@ const logger = createScopedLogger("outlook/message");
export const MESSAGE_SELECT_FIELDS =
"id,conversationId,conversationIndex,subject,bodyPreview,from,sender,toRecipients,ccRecipients,receivedDateTime,isDraft,isRead,body,categories,parentFolderId";

/**
* Removes quoted string literals from a query string to avoid false positives
* when checking for identifiers that might appear inside quotes.
* Handles both single and double quotes, including escaped quotes.
*/
function stripQuotedLiterals(query: string): string {
let result = "";
let i = 0;

while (i < query.length) {
const char = query[i];

if (char === "'" || char === '"') {
const quote = char;
i++; // Skip opening quote

// Skip until we find the matching closing quote (or end of string)
while (i < query.length) {
const current = query[i];
if (current === quote) {
// Found closing quote, check if it's escaped
let backslashCount = 0;
let j = i - 1;
while (j >= 0 && query[j] === "\\") {
backslashCount++;
j--;
}

// If even number of backslashes (including 0), quote is not escaped
if (backslashCount % 2 === 0) {
i++; // Skip closing quote
break;
}
}
i++;
}

// Replace the entire quoted section with spaces to maintain positions
// This prevents issues with adjacent identifiers
result += " ";
} else {
result += char;
i++;
}
}

return result;
}

/**
* Checks if parentFolderId appears as an unquoted identifier in the query.
* This avoids false positives when parentFolderId appears inside string literals.
*/
export function hasUnquotedParentFolderId(query: string): boolean {
const cleanedQuery = stripQuotedLiterals(query);
return /\bparentFolderId\b/.test(cleanedQuery);
}

// Well-known folder names in Outlook that are consistent across all languages
export const WELL_KNOWN_FOLDERS = {
inbox: "inbox",
Expand Down Expand Up @@ -241,21 +183,11 @@ export async function queryBatchMessages(
const hasSearchQuery = !!effectiveSearchQuery;
const hasDateFilters = !!(dateFilters && dateFilters.length > 0);

// Always filter to only include inbox and archive folders
const inboxFolderId = folderIds.inbox;
const archiveFolderId = folderIds.archive;

if (!inboxFolderId || !archiveFolderId) {
logger.warn("Missing required folder IDs", {
inboxFolderId,
archiveFolderId,
});
}

// Build folder filter for all cases
// Only apply folder filtering if a specific folderId is requested
// API already excludes Junk/Deleted by default, and drafts are filtered in convertMessages
const folderFilter = folderId
? `parentFolderId eq '${escapeODataString(folderId)}'`
: `(parentFolderId eq '${escapeODataString(inboxFolderId)}' or parentFolderId eq '${escapeODataString(archiveFolderId)}')`;
: undefined;

if (hasSearchQuery) {
// Search path - use $search parameter
Expand All @@ -276,16 +208,10 @@ export async function queryBatchMessages(
const response: { value: Message[]; "@odata.nextLink"?: string } =
await withOutlookRetry(() => request.get());

// Filter messages to only include inbox and archive folders
const filteredMessages = response.value.filter((message) => {
if (folderId) {
return message.parentFolderId === folderId;
}
return (
message.parentFolderId === inboxFolderId ||
message.parentFolderId === archiveFolderId
);
});
// Filter to specific folder if requested, otherwise get all
const filteredMessages = folderId
? response.value.filter((message) => message.parentFolderId === folderId)
: response.value;
const messages = await convertMessages(filteredMessages, folderIds);

nextPageToken = response["@odata.nextLink"]
Expand All @@ -295,30 +221,39 @@ export async function queryBatchMessages(

logger.info("Search results", {
totalFound: response.value.length,
afterFolderFiltering: filteredMessages.length,
filteredByFolder: folderId ? filteredMessages.length : undefined,
messageCount: messages.length,
hasNextPageToken: !!nextPageToken,
});

return { messages, nextPageToken };
} else {
// Filter path - use $filter parameter for date filters or folder-only queries
const filters = [folderFilter];
const filters: string[] = [];

// Add folder filter if a specific folder is requested
if (folderFilter) {
filters.push(folderFilter);
}

// Add date filters if provided
if (hasDateFilters) {
filters.push(...dateFilters!);
}

const combinedFilter = filters.join(" and ");
const combinedFilter =
filters.length > 0 ? filters.join(" and ") : undefined;

logger.info("Using filter path", {
folderFilter,
dateFilters: dateFilters || [],
combinedFilter,
});

request = request.filter(combinedFilter);
// Only apply filter if we have something to filter
if (combinedFilter) {
request = request.filter(combinedFilter);
}

if (pageToken) {
request = request.skipToken(pageToken);
Expand Down
2 changes: 1 addition & 1 deletion version.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
v2.19.7
v2.19.8
Loading