fix(welcome): perf on distinct recent activities

[justinpark]

SUMMARY

Currently, the recent_activity query for the log table groups by the dashboard ID and slice ID to extract a distinct list from the entire log table, which leads to performance issues.
(In the case of Airbnb, more than 1 million logs are generated each day, and grouping by dashboard and slice ID, even with indexing, significantly impacts database performance as shown in the following log)

| 3793593 | superset         | 100.117.121.20:41904  | superset_production | Query       |  1726 s| Sending data                                                  | SELECT anon_1.dashboard_id AS anon_1_dashboard_id, anon_1.slice_id AS anon_1_slice_id, anon_1.action AS anon_1_action, anon_1.dttm AS anon_1_dttm, dashboards.slug AS dashboard_slug, dashboards.dashboard_title AS dashboards_dashboard_title, slices.slice_name AS slices_slice_name
FROM (SELECT logs.dashboard_id AS dashboard_id, logs.slice_id AS slice_id, logs.action AS action, max(logs.dttm) AS dttm
FROM logs
WHERE logs.action IN ('explore', 'dashboard') AND logs.user_id = 13295 AND logs.dttm > '2024-03-06 18:00:12.201653' AND (logs.dashboard_id IS NOT NULL OR logs.slice_id IS NOT NULL) GROUP BY logs.dashboard_id, logs.slice_id, logs.action) AS anon_1 LEFT OUTER JOIN dashboards ON dashboards.id = anon_1.dashboard_id LEFT OUTER JOIN slices ON slices.id = anon_1.slice_id
WHERE dashboards.dashboard_title != '' OR slices.slice_name != '' ORDER BY anon_1.dttm DESC
 LIMIT 0, 6 |

To resolve this issue, it would be appropriate to create a materialized activity statistics view through a daily/hourly batch job. However, functionally, the main purpose of recent activity is to display only a few of the most recently visited items. Therefore, we improved performance by changing the approach to fetch the latest (including duplicates) log entries (by distinct: false) and extract a distinct list using LRU cache in the frontend side.

TESTING INSTRUCTIONS

specs (no visual changes)

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[korbit-ai]

Review by Korbit AI

Korbit automatically attempts to detect when you fix issues in new commits.

Category	Issue	Fix Detected
	Incorrect Chronological Order in Recent Activities ▹ view
	Misleading Method Name ▹ view	✅

Files scanned

​

File Path	Reviewed
superset-frontend/packages/superset-ui-core/src/utils/lruCache.ts	✅
superset-frontend/src/features/home/types.ts	✅
superset-frontend/src/features/home/ActivityTable.tsx	✅
superset-frontend/src/pages/Home/index.tsx	✅
superset-frontend/src/views/CRUD/utils.tsx	✅

Explore our documentation to understand the languages and file types we support and the files we ignore.

Need a new review? Comment /korbit-review on this PR and I'll review your latest changes.

Korbit Guide: Usage and Customization

Interacting with Korbit

• You can manually ask Korbit to review your PR using the /korbit-review command in a comment at the root of your PR.
• You can ask Korbit to generate a new PR description using the /korbit-generate-pr-description command in any comment on your PR.
• Too many Korbit comments? I can resolve all my comment threads if you use the /korbit-resolve command in any comment on your PR.
• On any given comment that Korbit raises on your pull request, you can have a discussion with Korbit by replying to the comment.
• Help train Korbit to improve your reviews by giving a 👍 or 👎 on the comments Korbit posts.

Customizing Korbit

• Check out our docs on how you can make Korbit work best for you and your team.
• Customize Korbit for your organization through the Korbit Console.

Feedback and Support

• Tell us what you think of Korbit
• Schedule a call with our team
• Email us @ [email protected]

[korbit-ai]

Incorrect Chronological Order in Recent Activities

Tell me more

What is the issue?

The original array is being reversed before being processed, which modifies the chronological order of activities before applying the LRU cache.

Why this matters

This can lead to incorrect chronological ordering of recent activities in the UI, as older activities might appear as more recent than they actually are.

Suggested change ∙ Feature Preview

Process the records in their original order to maintain correct chronological sequence:

recentsRes.json.result.forEach((record: RecentActivity) => {
  distinctRes.set(record.item_url, record);
});

💬 Looking for more details? Reply to this comment to chat with Korbit.

[michael-s-molina]

Thank you for the PR @justinpark.

LogDAO.get_recent_activity is only called from the recent_activity endpoint at superset/views/log/api.py. Should we remove the the whole distinct block? Given that this endpoint is not under v1, it's not a breaking change.

def get_recent_activity(
   actions: list[str],
   distinct: bool,
   page: int,
   page_size: int,
) -> list[dict[str, Any]]:
   ...
   if distinct:
      ...

cc @sadpandajoe @villebro

[michael-s-molina]

Is it necessary to reverse twice or can this be optimized?

[justinpark]

The API returns the latest accesses in reverse chronological order. (i.e. [2025-03-10, 2025-03-09, 2025-03-07, ...]) For an LRU cache, access records need to be stored in chronological order. Therefore, the API list must be reversed to maintain a record of accesses in chronological order. After that, since the final LRU cache list has the most recent entries placed at the end, it needs to be reversed again to position the latest entries at the front.
This two-step reversal process is an algorithmically correct approach.

[justinpark]

Given that this endpoint is not under v1, it's not a breaking change.

Its endpoint is /api/v1/log/recent_activity/ and the distinct param is included in the schema so it will be a break change.

[michael-s-molina]

Its endpoint is /api/v1/log/recent_activity/ and the distinct param is included in the schema so it will be a break change.

Oh, you're correct!

[justinpark]

Its endpoint is /api/v1/log/recent_activity/ and the distinct param is included in the schema so it will be a break change.

Oh, you're correct!

Therefore, it improves the frontend to fetch in an efficient way and keeps the API specification as-is.