Skip to content

fix: optimize mongo query for user vote list #19

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jcapphelix merged 4 commits into from
Mar 24, 2026
Merged

fix: optimize mongo query for user vote list #19

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
74087a6
chore: bump version
jcapphelix Mar 16, 2026
c42f60f
fix: user vote query optimization
jcapphelix Mar 16, 2026
932ed6d
fix: test case
jcapphelix Mar 16, 2026
8045042
fix: implement review comments
jcapphelix Mar 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion forum/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
Openedx forum app.
"""

__version__ = "0.4.4"
__version__ = "0.4.5"
4 changes: 3 additions & 1 deletion forum/backends/backend.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,9 @@ def get_commentables_counts_based_on_type(course_id: str) -> dict[str, Any]:
raise NotImplementedError

@classmethod
def get_user_voted_ids(cls, user_id: str, vote: str) -> list[str]:
def get_user_voted_ids(
cls, user_id: str, vote: str, course_id: Optional[str] = None
) -> list[str]:
"""Get user voted ids."""
raise NotImplementedError

Expand Down
23 changes: 16 additions & 7 deletions forum/backends/mongodb/api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1159,18 +1159,23 @@ def get_commentables_counts_based_on_type(course_id: str) -> dict[str, Any]:
return commentable_counts

@classmethod
def get_user_voted_ids(cls, user_id: str, vote: str) -> list[str]:
def get_user_voted_ids(
cls, user_id: str, vote: str, course_id: Optional[str] = None
) -> list[str]:
"""Get the IDs of the posts voted by a user."""
if vote not in ["up", "down"]:
raise ValueError("Invalid vote type")

content_model = Contents()

@timmc-edx timmc-edx Mar 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add test coverage for these changes? (Not sure if this repo is set up with a mock DB for unit tests.)

@jcapphelix jcapphelix Mar 23, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These already exist :-

forum/tests/test_views/test_users.py

Lines 164 to 214 in 266f5c2

def test_get_user_with_no_votes(
api_client: APIClient, patched_get_backend: Any
) -> None:
"""Test getting user with no votes."""
backend = patched_get_backend
user_id = backend.generate_id()
username = "test-user"
backend.find_or_create_user(
user_id,
username,
)
response = api_client.get(f"/api/v2/users/{user_id}?complete=true")
assert response.status_code == 200
user = response.json()
assert user["upvoted_ids"] == []
def test_get_user_with_votes(api_client: APIClient, patched_get_backend: Any) -> None:
"""Test getting user with votes."""
backend = patched_get_backend
user_id = backend.generate_id()
username = "test-user"
backend.find_or_create_user(
user_id,
username,
)
author_id = backend.generate_id()
author_username = "author"
backend.find_or_create_user(author_id, author_username)
thread_id = backend.create_thread(
{
"title": "Test Thread",
"body": "This is a test thread",
"course_id": "course1",
"commentable_id": "commentable1",
"author_id": author_id,
"author_username": author_username,
}
)
thread = backend.get_thread(thread_id)
user = backend.get_user(user_id)
assert thread
assert user
backend.upvote_content(
thread["_id"], user["external_id"], content_type="CommentThread"
)
response = api_client.get(f"/api/v2/users/{user_id}?complete=true")
assert response.status_code == 200
user = response.json()
assert user
assert user["upvoted_ids"] == [thread_id]

The get_user_voted_ids method is called indirectly through the user_to_hash method when fetching user data with the ?complete=true parameter and various other Tests also such as fetching threads, comments etc call the same functions.

@timmc-edx timmc-edx Mar 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not clear on how those would cover the use of the new course_id parameter, but this isn't blocking feedback.

contents = content_model.get_list()
content_query: dict[str, Any] = {}
if course_id:
content_query["course_id"] = str(course_id)
content_query[f"votes.{vote}"] = user_id

contents = content_model.get_list(**content_query)
voted_ids = []
for content in contents:

@timmc-edx timmc-edx Mar 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can any of this be removed now that the query is more refined?

@jcapphelix jcapphelix Mar 23, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here, content_model may work with distinct() but that is not producing proper results. (It is not bringing results, even if it did, we'd get ObjectIds so we'd still need to convert them to str)

Also what we get as of now is object, so we'd need to iterate over it.

We can remove the conditions though, so I've removed the conditions.

votes = content["votes"][vote]
if user_id in votes:
voted_ids.append(content["_id"])
voted_ids.append(content["_id"])

return voted_ids

Expand Down Expand Up @@ -1207,8 +1212,12 @@ def user_to_hash(

if params.get("complete"):
subscribed_thread_ids = cls.find_subscribed_threads(user["external_id"])
upvoted_ids = cls.get_user_voted_ids(user["external_id"], "up")
downvoted_ids = cls.get_user_voted_ids(user["external_id"], "down")
upvoted_ids = cls.get_user_voted_ids(
user["external_id"], "up", params.get("course_id")
)
downvoted_ids = cls.get_user_voted_ids(
user["external_id"], "down", params.get("course_id")
)
hash_data.update(
{
"subscribed_thread_ids": subscribed_thread_ids,
Expand Down
4 changes: 3 additions & 1 deletion forum/backends/mysql/api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1132,7 +1132,9 @@ def get_threads(
return threads

@classmethod
def get_user_voted_ids(cls, user_id: str, vote: str) -> list[str]:
def get_user_voted_ids(
cls, user_id: str, vote: str, course_id: Optional[str] = None
) -> list[str]:

@timmc-edx timmc-edx Mar 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a plan or ticket to optimize this by including course ID in the query?

@timmc-edx timmc-edx Mar 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(To be clear, I'm not saying we have to do this. If 2U isn't using the MySQL backend, we could just pass this info along to someone who is.)

@jcapphelix jcapphelix Mar 23, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no plan as such for it, because the query for MySQL is already querying a way smaller table called UserVote and calls it with user_id and vote type in query. So that should be enough. But it can still be optimized by adding course_id in the query, however UserVote MySQL table does not store course_id with that field itself, it is stored with some other field.

A ticket to do that later on is created :- https://2u-internal.atlassian.net/browse/COSMO2-849

"""Get the IDs of the posts voted by a user."""
if vote not in ["up", "down"]:
raise ValueError("Invalid vote type")
Expand Down
7 changes: 6 additions & 1 deletion tests/e2e/test_users.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -624,7 +624,12 @@ def test_update_user_stats(api_client: APIClient, patched_get_backend: Any) -> N
# Sort the data for expected result (threads, responses, replies)
expected_result = sorted(
expected_data.values(),
key=lambda val: (val["threads"], val["responses"], val["replies"]),
key=lambda val: (
val["threads"],
val["responses"],
val["replies"],
val["username"],

@timmc-edx timmc-edx Mar 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this the unrelated test change? I'm guessing this was a flaky test that you're fixing here. If so, can you ensure this ends up in the squash commit's message?

@jcapphelix jcapphelix Mar 16, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this was unrelated test-case. Yes, will make sure to add it in squash commit message.

),
reverse=True,
)

Expand Down
Loading