-
-
Notifications
You must be signed in to change notification settings - Fork 143
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add document hard deletion functionality to housekeeping tasks. #718
base: main
Are you sure you want to change the base?
Conversation
…emoved unnecessary comments
…emoved unnecessary comments
… and processing method related to hard deletion
…housekeeping structure.
…functions and test code to match the changed housekeeping structure
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #718 +/- ##
==========================================
- Coverage 50.71% 50.45% -0.27%
==========================================
Files 70 70
Lines 10213 10424 +211
==========================================
+ Hits 5180 5259 +79
- Misses 4512 4636 +124
- Partials 521 529 +8 ☔ View full report in Codecov by Sentry. |
…ed affected functions, Default value: -24h
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for your contribution.
I have left some comments on naming convention and few more.
About Housekeeping Task Naming Convention
Since we now newly introducing document hard deletion, it would be better to rename existing housekeeping task "Deactivate Candidate" (deactivateCandidates
) to "Client Deactivate Candidate" (ClientDeactivateCandidates
) for more clarification and naming consistency with "Document Hard Deletion" (DocumentHardDeletion
).
…ypos have been corrected for updates.
…m less than to less than or equal to
Thank you. All comments have been reflected, and some feature changes and comment typos have been corrected for the update. However, I think there is one thing that needs to be changed. Client: Returns without rollback when err occurs during operation Memdb: If err occurs during operation, rollback and return without committing Which of the two methods should you proceed with? |
In #932, we improved housekeeping so that multiple tasks can be added. It might be good to try document deletion again. |
@hackerwins Should we reopen this issue? |
@krapie sorry for forgetting. I will modify the work to fit the changed structure. |
If it's okay for @fourjae, It would be good to continue in this PR. 🙏 |
WalkthroughThe recent changes enhance the housekeeping functionality across the server’s configuration and database management systems. By introducing more granular control over housekeeping intervals and limits, such as document deletion and candidate deactivation, the modifications significantly improve the efficiency of data management processes. Updated methods, configurations, and tests ensure that the system can better handle document lifecycles. Changes
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
Outside diff range, codebase verification and nitpick comments (4)
cmd/yorkie/server.go (1)
213-216
: Command-line flag description needs improvement.The description for the
documentHardDeletionGracefulPeriod
flag should be more consistent with other descriptions.- "Document deletion over time after a single housekeeping run" + "Graceful period for document deletion after a single housekeeping run"server/backend/database/testcases/testcases.go (3)
Line range hint
218-218
:
Follow the convention for function comments.It would be better to follow the convention of other test cases.
// RunDocumentHardDeletionTest runs the RunDocumentHardDeletion tests for the given db. func RunDocumentHardDeletionTest(t *testing.T, ...)
Line range hint
245-245
:
Use a descriptive variable name.Consider renaming
GracePeriod
togracePeriod
to follow Go naming conventions and improve readability.- GracePeriod := "0s" + gracePeriod := "0s"
Line range hint
258-260
:
Ensure proper error handling for deletion count.The assertion should check if the number of deleted documents matches the expected count. Consider adding a more descriptive error message.
- assert.Equal(t, int(deletedDocumentsCount), len(candidates)) + assert.Equal(t, len(candidates), int(deletedDocumentsCount), "Mismatch in the number of deleted documents")
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (16)
- cmd/yorkie/server.go (3 hunks)
- server/backend/database/database.go (2 hunks)
- server/backend/database/memory/database.go (2 hunks)
- server/backend/database/memory/database_test.go (1 hunks)
- server/backend/database/memory/indexes.go (1 hunks)
- server/backend/database/mongo/client.go (3 hunks)
- server/backend/database/mongo/client_test.go (1 hunks)
- server/backend/database/testcases/testcases.go (1 hunks)
- server/backend/housekeeping/config.go (1 hunks)
- server/backend/housekeeping/config_test.go (1 hunks)
- server/backend/housekeeping/housekeeping.go (9 hunks)
- server/config.go (2 hunks)
- server/config.sample.yml (1 hunks)
- server/rpc/server_test.go (1 hunks)
- test/helper/helper.go (2 hunks)
- test/sharding/server_test.go (1 hunks)
Additional comments not posted (48)
server/backend/housekeeping/config_test.go (4)
30-35
: Ensure consistency in configuration field naming and validation.The new fields
IntervalDeactivateCandidates
,IntervalDeleteDocuments
,DocumentHardDeletionGracefulPeriod
,ClientDeactivationCandidateLimitPerProject
, andDocumentHardDeletionCandidateLimitPerProject
are consistent with the previous naming conventions. The validation logic should ensure these fields are correctly parsed and validated.
40-48
: Validate interval fields with proper time units.The interval fields
IntervalDeactivateCandidates
,IntervalDeleteDocuments
, andDocumentHardDeletionGracefulPeriod
should be validated to ensure they use proper time units like "m", "h", "s".
52-57
: Validate candidate limit fields for non-negative values.The candidate limit fields
ClientDeactivationCandidateLimitPerProject
andDocumentHardDeletionCandidateLimitPerProject
should be validated to ensure they are non-negative values.
60-61
: Ensure project fetch size is non-negative.The
ProjectFetchSize
field should be validated to ensure it is a non-negative value.server/backend/database/memory/database_test.go (1)
107-109
: Ensure comprehensive test coverage for document hard deletion.The new test case
DocumentHardDeletion test
uses theRunDocumentHardDeletionTest
function from thetestcases
package to test the document hard deletion functionality. Ensure that this test case covers all relevant scenarios for document hard deletion.server/backend/housekeeping/config.go (4)
26-39
: Ensure clarity and consistency in configuration field definitions.The new fields
IntervalDeactivateCandidates
,IntervalDeleteDocuments
,DocumentHardDeletionGracefulPeriod
,ClientDeactivationCandidateLimitPerProject
, andDocumentHardDeletionCandidateLimitPerProject
are clearly defined and consistent with the existing configuration structure.
47-61
: Ensure proper validation of interval fields.The interval fields
IntervalDeactivateCandidates
,IntervalDeleteDocuments
, andDocumentHardDeletionGracefulPeriod
are validated usingtime.ParseDuration
to ensure they use proper time units.
71-83
: Ensure proper validation of candidate limit fields.The candidate limit fields
ClientDeactivationCandidateLimitPerProject
andDocumentHardDeletionCandidateLimitPerProject
are validated to ensure they are non-negative values.
Line range hint
84-89
:
Ensure proper validation of project fetch size.The
ProjectFetchSize
field is validated to ensure it is a non-negative value.server/backend/database/mongo/client_test.go (1)
124-126
: LGTM!The new test case for document hard deletion is correctly added and integrates well with the existing tests.
server/config.sample.yml (6)
33-34
: LGTM!The new parameter
IntervalDeactivateCandidates
is correctly defined and provides necessary configuration for candidate deactivation.
36-37
: LGTM!The new parameter
IntervalDeleteDocuments
is correctly defined and provides necessary configuration for document deletion.
39-40
: LGTM!The new parameter
HousekeepingDocumentHardDeletionGracefulPeriod
is correctly defined and provides necessary configuration for managing document retention.
42-43
: LGTM!The new parameter
ClientDeactivationCandidateLimitPerProject
is correctly defined and provides necessary configuration for limiting client deactivation candidates.
45-46
: LGTM!The new parameter
DocumentHardDeletionCandidateLimitPerProject
is correctly defined and provides necessary configuration for limiting document hard deletion candidates.
48-49
: LGTM!The new parameter
ProjectFetchSize
is correctly defined and provides necessary configuration for limiting the number of projects.server/backend/database/memory/indexes.go (1)
120-124
: LGTM!The new field
project_id
is correctly defined and enhances the schema by allowing for the storage and retrieval of data associated with project identifiers.test/sharding/server_test.go (1)
80-85
: Ensure consistency in configuration parameter naming.The new configuration parameters are well-integrated. Verify that the naming conventions are consistent across the codebase.
Verification successful
Consistency in configuration parameter naming verified.
The new configuration parameters are consistently named across the codebase.
IntervalDeactivateCandidates
IntervalDeleteDocuments
DocumentHardDeletionGracefulPeriod
ClientDeactivationCandidateLimitPerProject
DocumentHardDeletionCandidateLimitPerProject
Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify consistency in configuration parameter naming. # Test: Search for the new configuration parameters. Expect: Consistent naming conventions. rg --type go $'IntervalDeactivateCandidates|IntervalDeleteDocuments|DocumentHardDeletionGracefulPeriod|ClientDeactivationCandidateLimitPerProject|DocumentHardDeletionCandidateLimitPerProject'Length of output: 9614
server/rpc/server_test.go (1)
85-90
: Ensure consistency in configuration parameter naming.The new configuration parameters are well-integrated. Verify that the naming conventions are consistent across the codebase.
Verification successful
The naming conventions for the new configuration parameters are consistent across the codebase.
The parameters are well-integrated and consistently named in various files.
server/rpc/server_test.go
server/config.go
test/helper/helper.go
server/backend/housekeeping/config.go
cmd/yorkie/server.go
Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify consistency in configuration parameter naming. # Test: Search for the new configuration parameters. Expect: Consistent naming conventions. rg --type go $'IntervalDeactivateCandidates|IntervalDeleteDocuments|DocumentHardDeletionGracefulPeriod|ClientDeactivationCandidateLimitPerProject|DocumentHardDeletionCandidateLimitPerProject'Length of output: 9614
server/config.go (2)
234-239
: Ensure correct integration of new housekeeping constants.The new constants are correctly integrated into the
newConfig
function. Verify that the integration is consistent across the codebase.
43-48
: Ensure consistency in constant naming and usage.The new constants are well-defined. Verify that the naming conventions are consistent across the codebase.
Verification successful
The new constants are well-defined and consistently named in the files
server/config.go
andcmd/yorkie/server.go
. To ensure thorough verification, let's check if these constants are used in any other files within the codebase.
Ensure consistency in constant naming and usage.
The new constants are well-defined and consistently named across the codebase. They are used appropriately in the files
server/config.go
andcmd/yorkie/server.go
.
server/config.go
cmd/yorkie/server.go
Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify consistency in constant naming and usage. # Test: Search for the new constants. Expect: Consistent naming conventions. rg --type go $'DefaultHousekeepingIntervalDeactivateCandidates|DefaultHousekeepingIntervalDeleteDocuments|DefaultHousekeepingDocumentHardDeletionGracefulPeriod|DefaultHousekeepingClientDeactivationCandidateLimitPerProject|DefaultHousekeepingDocumentHardDeletionCandidateLimitPerProject'Length of output: 1823
Script:
#!/bin/bash # Description: Verify consistency in constant naming and usage across the entire codebase. # Test: Search for the new constants in all Go files. Expect: Consistent naming conventions. rg --type go 'DefaultHousekeepingIntervalDeactivateCandidates|DefaultHousekeepingIntervalDeleteDocuments|DefaultHousekeepingDocumentHardDeletionGracefulPeriod|DefaultHousekeepingClientDeactivationCandidateLimitPerProject|DefaultHousekeepingDocumentHardDeletionCandidateLimitPerProject'Length of output: 1823
server/backend/database/database.go (3)
161-167
: Method looks good.The
FindDocumentHardDeletionCandidatesPerProject
method is well-defined and its parameters are appropriate for the intended functionality.
169-173
: Method looks good.The
DeleteDocument
method is well-defined and its parameters are appropriate for the intended functionality.
174-177
: Method looks good.The
FindDocInfoByKey
method is unchanged and its parameters are appropriate for the intended functionality.server/backend/housekeeping/housekeeping.go (8)
35-36
: Constants look good.The constants
deactivateCandidatesKey
anddocumentHardDeletionLockKey
are well-defined and appropriate for the locking mechanisms.
46-51
: Fields look good.The new fields added to the
Housekeeping
struct are well-defined and appropriate for managing intervals and limits for housekeeping tasks.
Line range hint
80-108
:
Constructor function looks good.The
New
constructor function is well-defined and appropriately parses the additional configuration parameters.
117-118
: Method looks good.The
Start
method is well-defined and appropriately starts the new housekeeping tasks.
Line range hint
128-148
:
Method looks good.The
AttachDeactivateCandidates
method is well-defined and appropriately handles the housekeeping loop for deactivating candidates.
150-170
: Method looks good.The
AttachDocumentHardDeletion
method is well-defined and appropriately handles the housekeeping loop for document hard deletion.
172-220
: Method looks good.The
DeleteDocument
method is well-defined and appropriately handles the deletion of documents.
311-331
: Method looks good.The
FindDocumentHardDeletionCandidates
method is well-defined and appropriately handles finding document hard deletion candidates.cmd/yorkie/server.go (7)
41-45
: Variables look good.The new variables for housekeeping intervals and limits are well-defined and appropriate for managing different housekeeping tasks.
74-76
: Function looks good.The
newServerCmd
function is well-defined and appropriately reflects the new variables for housekeeping intervals and limits.
201-211
: Command-line flags look good.The new command-line flags for housekeeping intervals and limits are well-defined and appropriate for managing different housekeeping tasks.
219-221
: Command-line flag looks good.The command-line flag for
ClientDeactivationCandidateLimitPerProject
is well-defined and appropriately reflects the new variable name.
225-229
: Command-line flag looks good.The command-line flag for
DocumentHardDeletionCandidateLimitPerProject
is well-defined and appropriate for managing document deletion limits.
229-231
: Command-line flag looks good.The command-line flag for
ProjectFetchSize
is well-defined and appropriate for managing project fetch size.
201-231
: Function looks good.The
init
function is well-defined and appropriately includes the new command-line flags.test/helper/helper.go (3)
65-66
: Ensure consistency in interval naming.The intervals for deactivating candidates and deleting documents are both set to
10 * gotime.Second
. Verify that these intervals are appropriate and consistent with the rest of the system.
67-69
: Check the new housekeeping limits.The new variables
HousekeepingDocumentHardDeletionGracefulPeriod
andHousekeepingDocumentHardDeletionCandidateLimitPerProject
are set to0 * gotime.Second
and10
respectively. Ensure these values are appropriate for your use case.
216-221
: Verify the updated housekeeping configuration.The configuration now includes the new intervals and limits for housekeeping tasks. Ensure these changes are correctly integrated and tested.
server/backend/database/memory/database.go (2)
622-657
: Ensure correct handling of document fetching.The method
FindDocumentHardDeletionCandidatesPerProject
correctly fetches documents marked for hard deletion. Ensure the logic for checking theRemovedAt
timestamp and the limit on candidates is appropriate.
704-726
: Handle potential errors in document deletion.The method
DeleteDocument
correctly deletes documents and handles errors. Ensure that the transaction is appropriately committed or rolled back in case of errors.server/backend/database/mongo/client.go (3)
230-257
: Handle potential errors in document deletion.The method
DeleteDocument
correctly deletes documents and handles errors. Ensure that the deletion operation is appropriately handled and any errors are logged or managed.
651-677
: Ensure correct handling of document fetching.The method
FindDocumentHardDeletionCandidatesPerProject
correctly fetches documents marked for hard deletion. Ensure the logic for checking theRemovedAt
timestamp and the limit on candidates is appropriate.
710-739
: Ensure correct handling of client fetching.The method
FindDeactivateCandidates
correctly fetches clients needing deactivation. Ensure the logic for cycling through projects and aggregating candidates is appropriate.server/backend/database/testcases/testcases.go (1)
Line range hint
262-263
:
Verify error handling for document not found.Ensure that the error handling correctly identifies the document not found scenario.
Verification successful
Error handling for document not found is correctly implemented.
The code properly identifies and handles the
ErrDocumentNotFound
scenario usingassert.ErrorIs(t, err, database.ErrDocumentNotFound)
.
- Instances found in
server/backend/database/testcases/testcases.go
Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify error handling for document not found after deletion. # Test: Search for the error handling code. Expect: Proper handling of ErrDocumentNotFound. rg --type go -A 5 $'assert.ErrorIs(t, err, database.ErrDocumentNotFound)'Length of output: 74
Script:
#!/bin/bash # Description: Search for any references to ErrDocumentNotFound in the testcases.go file to verify error handling. # Test: Search for the error handling code. Expect: Proper handling of ErrDocumentNotFound. rg 'ErrDocumentNotFound' server/backend/database/testcases/testcases.goLength of output: 236
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 3
Outside diff range, codebase verification and nitpick comments (3)
server/backend/database/mongo/client.go (3)
704-729
: Add logging for better traceability.Adding logging can help trace the execution flow and identify issues more easily.
+ logging.DefaultLogger().Infof("Finding hard deletion candidates for project: %s", project.ID)
715-715
: Rename variable to follow camelCase convention.The variable name
DocInfos
should bedocInfos
to follow camelCase convention.- var DocInfos []*database.DocInfo + var docInfos []*database.DocInfo
763-791
: Add logging for better traceability.Adding logging can help trace the execution flow and identify issues more easily.
+ logging.DefaultLogger().Infof("Finding deactivate candidates, lastProjectID: %s", lastProjectID)
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (17)
- cmd/yorkie/server.go (3 hunks)
- server/backend/database/database.go (2 hunks)
- server/backend/database/memory/database.go (2 hunks)
- server/backend/database/memory/database_test.go (1 hunks)
- server/backend/database/mongo/client.go (3 hunks)
- server/backend/database/mongo/client_test.go (1 hunks)
- server/backend/database/testcases/testcases.go (1 hunks)
- server/backend/housekeeping/config.go (2 hunks)
- server/backend/housekeeping/config_test.go (1 hunks)
- server/backend/housekeeping/housekeeping.go (3 hunks)
- server/clients/housekeeping.go (4 hunks)
- server/config.go (2 hunks)
- server/rpc/server_test.go (1 hunks)
- server/server.go (1 hunks)
- test/bench/tree_editing_bench_test.go (1 hunks)
- test/helper/helper.go (2 hunks)
- test/sharding/server_test.go (1 hunks)
Files skipped from review due to trivial changes (2)
- server/backend/database/mongo/client_test.go
- test/bench/tree_editing_bench_test.go
Files skipped from review as they are similar to previous changes (8)
- cmd/yorkie/server.go
- server/backend/database/memory/database.go
- server/backend/database/memory/database_test.go
- server/backend/database/testcases/testcases.go
- server/config.go
- server/rpc/server_test.go
- test/helper/helper.go
- test/sharding/server_test.go
Additional comments not posted (24)
server/backend/housekeeping/config_test.go (6)
30-35
: LGTM!The new configuration fields are correctly initialized.
40-41
: LGTM!The test case correctly checks for validation errors for invalid interval values.
44-45
: LGTM!The test case correctly checks for validation errors for invalid interval values.
48-49
: LGTM!The test case correctly checks for validation errors for invalid period values.
52-53
: LGTM!The test case correctly checks for validation errors for invalid limit values.
56-57
: LGTM!The test case correctly checks for validation errors for invalid limit values.
server/backend/housekeeping/housekeeping.go (5)
32-33
: LGTM!The
Housekeeping
struct correctly includes theConfig
andscheduler
fields.
51-65
: LGTM!The
RegisterTask
method correctly sets up the scheduler for the housekeeping tasks.
Line range hint
67-75
:
LGTM!The
RegisterTask
method correctly sets up the scheduler for document deletion tasks.
Line range hint
79-81
:
LGTM!The
Start
method correctly starts the scheduler for the housekeeping service.
Line range hint
83-91
:
LGTM!The
Stop
method correctly stops the scheduler and handles errors.server/backend/housekeeping/config.go (6)
28-41
: LGTM!The new fields in the
Config
struct enhance the configurability of the housekeeping process.
49-53
: LGTM!The
Validate
method correctly validates the new configuration fields.
57-63
: LGTM!The
Validate
method correctly validates the interval for document deletion.
65-71
: LGTM!The
Validate
method correctly validates the period for document hard deletion.
72-78
: LGTM!The
Validate
method correctly validates the limit for client deactivation candidates.
79-85
: LGTM!The
Validate
method correctly validates the limit for document hard deletion candidates.server/clients/housekeeping.go (4)
39-39
: Improved parameter naming.The parameter
candidatesLimitPerProject
has been renamed toclientDeactivationCandidateLimitPerProject
for better clarity.
96-96
: Improved parameter naming.The parameter
candidatesLimitPerProject
has been renamed toclientDeactivationCandidateLimitPerProject
for better clarity.
182-204
: LGTM!The
FindDocumentHardDeletionCandidates
function is well-implemented with proper error handling and logging. The return values are consistent with the function's purpose.
125-180
: LGTM! But verify the function usage in the codebase.The
DeleteDocument
function is well-implemented with proper locking and error handling. The logging provides useful information.However, ensure that the function is used correctly throughout the codebase.
server/server.go (1)
162-206
: Enhanced housekeeping task registration.The function now registers two separate tasks for deactivating clients and deleting documents, improving control over task intervals. The logic and error handling are well-implemented.
However, ensure that the function is used correctly throughout the codebase.
Verification successful
Function usage verified.
The function
RegisterHousekeepingTasks
is correctly used within the codebase, specifically in theserver/server.go
file. The implementation and usage are consistent and error handling is appropriately managed.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify all function calls to `RegisterHousekeepingTasks`. # Test: Search for the function usage. Expect: Correct usage of the updated function. rg --type go -A 5 $'RegisterHousekeepingTasks'Length of output: 700
server/backend/database/database.go (2)
168-173
: LGTM!The
FindDocumentHardDeletionCandidatesPerProject
method is well-implemented with proper error handling and logging. The return values are consistent with the method's purpose.
175-179
: LGTM!The
DeleteDocument
method is well-implemented with proper error handling and logging. The return values are consistent with the method's purpose.
HousekeepingDocumentHardDeletionGracefulPeriod default hour 14h -> 336h (14day * 24h)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (5)
- server/backend/database/mongo/client.go (3 hunks)
- server/backend/housekeeping/housekeeping.go (3 hunks)
- server/config.go (2 hunks)
- server/config.sample.yml (1 hunks)
- server/server.go (1 hunks)
Files skipped from review as they are similar to previous changes (4)
- server/backend/database/mongo/client.go
- server/backend/housekeeping/housekeeping.go
- server/config.sample.yml
- server/server.go
Additional comments not posted (3)
server/config.go (3)
43-45
: New constants for housekeeping intervals and grace period.The new constants provide more granular control over housekeeping tasks. Ensure that these values align with the intended housekeeping policies.
46-47
: Renamed constants for candidate limits per project.The constants have been renamed for clarity. Ensure that these changes are reflected in the rest of the codebase.
235-240
: UpdatednewConfig
function with new housekeeping constants.The function now references the new housekeeping constants. Ensure that these changes are correctly integrated and tested.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (1)
- server/backend/housekeeping/config_test.go (1 hunks)
Files skipped from review as they are similar to previous changes (1)
- server/backend/housekeeping/config_test.go
fbc6098
to
7d65a57
Compare
@fourjae Sorry for the late review. I've been busy for couple of weeks. cc. @hackerwins |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for your contribution :)
Firstly, I have left some comments about overall code conventions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
Outside diff range, codebase verification and nitpick comments (4)
server/config.go (1)
237-237
: UseString()
method for consistency.The
DocumentHardDeletionGracefulPeriod
field is set directly to the corresponding constant value, which is inconsistent with the other fields that use theString()
method. To maintain consistency and avoid potential issues, consider updating the code as follows:-DocumentHardDeletionGracefulPeriod: DefaultHousekeepingDocumentHardDeletionGracefulPeriod, +DocumentHardDeletionGracefulPeriod: DefaultHousekeepingDocumentHardDeletionGracefulPeriod.String(),server/backend/database/database.go (1)
175-179
: LGTM, but please update the comment.The
DeleteDocuments
method addition is approved. However, please update the comment at line 175 to match the method signature.-// DeleteDocuments deletes document of the given key +// DeleteDocuments deletes the documents specified by the given DocInfo pointersserver/backend/database/mongo/client.go (1)
230-263
: Approve the implementation ofDeleteDocuments
.The
DeleteDocuments
function is correctly implemented with proper error handling and validation.Improve the error messages.
Consider making the error messages more descriptive:
- When an invalid document ID is encountered, include the invalid ID in the error message.
- When the deletion fails, include the specific error details in the returned error.
server/backend/database/testcases/testcases.go (1)
1414-1468
: The code changes are approved with a suggestion to improve the test coverage.The new test function
RunDocumentHardDeletionTest
is well-structured and covers the key aspects of the document hard deletion functionality. It follows the existing conventions in the file.To further improve the test coverage, consider adding the following scenarios:
- Test the case when there are no candidate documents for hard deletion.
- Test the case when the grace period is set to a positive value.
- Test the case when there are multiple candidate documents for hard deletion.
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (16)
- cmd/yorkie/server.go (3 hunks)
- server/backend/database/database.go (2 hunks)
- server/backend/database/memory/database.go (2 hunks)
- server/backend/database/memory/indexes.go (1 hunks)
- server/backend/database/mongo/client.go (2 hunks)
- server/backend/database/testcases/testcases.go (1 hunks)
- server/backend/housekeeping/config.go (2 hunks)
- server/backend/housekeeping/config_test.go (1 hunks)
- server/backend/housekeeping/housekeeping.go (2 hunks)
- server/clients/housekeeping.go (4 hunks)
- server/config.go (2 hunks)
- server/config.sample.yml (1 hunks)
- server/rpc/server_test.go (1 hunks)
- server/server.go (2 hunks)
- test/helper/helper.go (2 hunks)
- test/sharding/server_test.go (1 hunks)
Files skipped from review due to trivial changes (1)
- server/backend/housekeeping/housekeeping.go
Additional comments not posted (45)
server/backend/housekeeping/config_test.go (6)
30-35
: LGTM!The code changes are approved.
40-40
: LGTM!The code changes are approved.
44-44
: LGTM!The code changes are approved.
48-50
: LGTM!The code changes are approved.
51-57
: LGTM!The code changes are approved.
59-62
: LGTM!The code changes are approved.
server/backend/housekeeping/config.go (4)
28-41
: LGTM!The changes to the
Config
struct improve the configurability and granularity of the housekeeping service. The new fields allow for separate intervals for deactivating candidates and deleting documents, and provide limits on the number of candidates per project. The naming of the fields is clear and descriptive.
57-63
: LGTM!The new check for the
DeleteDocumentsInterval
field ensures that it is a valid duration, improving the robustness of the configuration validation.
65-82
: LGTM!The new checks for the
DocumentHardDeletionGracefulPeriod
,ClientDeactivationCandidateLimitPerProject
, andDocumentHardDeletionCandidateLimitPerProject
fields ensure that they have valid values, improving the robustness of the configuration validation.
97-105
: LGTM!The changes to the
ParseInterval
method enhance its flexibility and usability by accepting an interval as a parameter. This allows the method to be used with different interval values without modifying the struct. The updated error message provides more context about the interval being parsed.server/config.sample.yml (4)
33-34
: LGTM!The change from
Interval
toDeactivateCandidatesInterval
improves clarity by using a more descriptive name for the parameter.
36-37
: LGTM!The addition of the
DeleteDocumentsInterval
parameter introduces an interval for document deletion, which is a new housekeeping functionality.
39-40
: LGTM!The addition of the
HousekeepingDocumentHardDeletionGracefulPeriod
parameter specifies a grace period for hard deletion of documents, which is a new housekeeping functionality.
42-46
: LGTM!The changes in this code segment are approved:
- The change from
CandidatesLimitPerProject
toClientDeactivationCandidateLimitPerProject
improves clarity by using a more descriptive name for the parameter.- The addition of the
DocumentHardDeletionCandidateLimitPerProject
parameter introduces a limit for document deletion candidates, which is a new housekeeping functionality.server/backend/database/memory/indexes.go (1)
139-147
: LGTM!The new compound index
project_id_removed_at
is a valuable addition to theschema
variable. It allows for efficient querying based on the combination ofProjectID
andRemovedAt
fields, potentially improving performance for operations that involve filtering or sorting by both fields.The index is correctly defined using the
memdb.CompoundIndex
type and the appropriate field types. The index name is clear and follows the naming convention of other indexes.The code changes are approved.
server/clients/housekeeping.go (7)
31-31
: LGTM!The code changes are approved.
32-32
: LGTM!The code changes are approved.
40-40
: LGTM!The code changes are approved.
108-114
: LGTM!The code changes are approved.
116-124
: LGTM!The code changes are approved.
127-181
: LGTM!The new
DeleteDocuments
function looks good. It follows the existing coding conventions and implements the document deletion functionality with proper locking and error handling.
Line range hint
184-219
: LGTM!The new
FindDocumentHardDeletionCandidates
function looks good. It follows the existing coding conventions and implements the logic for finding document deletion candidates across multiple projects.server/server.go (4)
97-100
: LGTM!The code changes are approved.
101-103
: LGTM!The code changes are approved.
164-187
: LGTM!The code changes are approved.
189-214
: LGTM!The code changes are approved.
test/sharding/server_test.go (1)
81-85
: LGTM!The changes to the housekeeping configuration parameters in the
TestMain
function look good. The new parameters provide more granular control over document management and candidate deactivation processes, which aligns with the AI-generated summary.The changes are in a test file and do not directly impact the production code. The new parameters are initialized with values from the
helper
package, which likely contains predefined constants for testing.Overall, the changes enhance the functionality of the housekeeping configuration and do not introduce any issues or inconsistencies.
server/config.go (4)
43-47
: LGTM!The introduction of the new constants enhances the clarity and readability of the housekeeping configuration. The constants provide more specific and descriptive names, making it easier to understand their purpose in the context of document deletion and candidate deactivation processes.
48-48
: Please clarify the purpose and usage ofDefaultHousekeepingProjectFetchSize
.The constant
DefaultHousekeepingProjectFetchSize
seems to be a standalone addition and its purpose is not immediately clear from the context. Could you please provide more information on how this constant is used in the housekeeping process and why it was introduced?
235-240
: LGTM!The updates to the
Housekeeping
configuration in thenewConfig
function ensure that the logic for housekeeping operations is now aligned with the new constants introduced earlier. This change enhances the consistency and clarity of the housekeeping configuration.
43-48
: Overall, the code changes look good!The introduction of the new constants and the updates to the housekeeping configuration enhance the clarity, consistency, and maintainability of the codebase. The changes provide more specific and descriptive names for the housekeeping intervals and limits, making it easier to understand their purpose in the context of document deletion and candidate deactivation processes.
Apart from the minor nitpick regarding the consistency of setting the
DocumentHardDeletionGracefulPeriod
field, the code changes are well-structured and contribute positively to the overall quality of the codebase.Also applies to: 235-240
server/rpc/server_test.go (1)
86-91
: LGTM!The changes to the housekeeping configuration parameters are well-aligned with the PR objectives and significantly enhance the functionality of the housekeeping process. The new parameters provide more granular control over the management of candidate documents and their deletion processes, likely improving the efficiency of data management.
The code changes are approved.
server/backend/database/database.go (2)
23-23
: LGTM!The import statement is approved.
167-173
: LGTM!The
FindDocumentHardDeletionCandidatesPerProject
method addition is approved.cmd/yorkie/server.go (6)
44-44
: Previous review comment addressed.The variable name
documentHardDeletionGracefulPeriod
follows the naming convention suggested in the previous review comment.
74-76
: LGTM!The code changes are approved.
200-211
: LGTM!The code changes are approved. The flag names and descriptions follow the naming convention suggested in the previous review comment.
213-216
: LGTM!The code changes are approved. The flag name and description follow the naming convention suggested in the previous review comment.
219-223
: LGTM!The code changes are approved. The flag name and description follow the naming convention suggested in the previous review comment.
224-229
: LGTM!The code changes are approved. The flag name and description follow the naming convention suggested in the previous review comment.
test/helper/helper.go (2)
65-73
: LGTM!The code changes are approved.
242-246
: LGTM!The code changes are approved.
server/backend/database/memory/database.go (2)
676-712
: LGTM!The code changes are approved.
758-780
: LGTM!The code changes are approved.
server/backend/database/mongo/client.go (1)
710-736
: LGTM!The
FindDocumentHardDeletionCandidatesPerProject
function is correctly implemented and follows the requirements. It effectively identifies the documents eligible for hard deletion based on the provided project and grace period.
@krapie Edited to reflect all comments. Additionally, the grace time setting for deleted documents has been declared as 2 weeks (14d*24h). How many hours would be good? |
@fourjae I think 2 weeks is good for now. |
@hackerwins Sudden wonder, is this task should be in housekeeping? not background? |
What this PR does / why we need it:
A housekeeping task was added to automatically physically delete documents at certain intervals.
Which issue(s) this PR fixes:
Fixes # #706
Although it does not directly solve the above issue, I think it is somewhat related to the need to improve the housekeeping structure.
I think the housekeeping structure could be improved if necessary.
Special notes for your reviewer:
Does this PR introduce a user-facing change?:
Additional documentation:
Checklist:
Summary by CodeRabbit
New Features
Bug Fixes
Documentation