-
Notifications
You must be signed in to change notification settings - Fork 67
add timestamp to global state table for lease table cleanup #861
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 4 commits
27ca6a2
d9e29b4
338ba63
decf665
1c0754c
5000872
7daa99c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,6 +22,7 @@ | |
| - [Startup retries](#startup-retries) | ||
| - [Broken connection retries](#broken-connection-retries) | ||
| - [Function exception retries](#function-exception-retries) | ||
| - [Lease Tables clean up](#lease-tables-clean-up) | ||
|
|
||
| ## Input Binding | ||
|
|
||
|
|
@@ -147,7 +148,7 @@ If an exception occurs in the user function when processing changes then the bat | |
|
|
||
| If the function execution fails 5 times in a row for a given row then that row is completely ignored for all future changes. Because the rows in a batch are not deterministic, rows in a failed batch may end up in different batches in subsequent invocations. This means that not all rows in the failed batch will necessarily be ignored. If other rows in the batch were the ones causing the exception, the "good" rows may end up in a different batch that doesn't fail in future invocations. | ||
|
|
||
| You can run this query to see what rows have failed 5 times and are currently ignored, see [Leases table](#az_funcleases_) documentation for how to get the correct Leases table to query for your function. | ||
| You can run this query to see what rows have failed 5 times and are currently ignored, see [Leases table](./TriggerBinding.md#az_funcleases_) documentation for how to get the correct Leases table to query for your function. | ||
|
|
||
| ```sql | ||
| SELECT * FROM [az_func].[Leases_<FunctionId>_<TableId>] WHERE _az_func_AttemptCount = 5 | ||
|
|
@@ -168,3 +169,93 @@ e.g. | |
| ```sql | ||
| UPDATE [Products].[az_func].[Leases_<FunctionId>_<TableId>] SET _az_func_AttemptCount = 0 WHERE ProductId = 123 | ||
| ``` | ||
|
|
||
| #### Lease Tables clean up | ||
|
|
||
| Before clean up, please see [Leases table](./TriggerBinding.md#az_funcleases_) documentation for understanding how they are created and used. | ||
|
|
||
| Why clean up? | ||
| 1. You renamed your function/class/method name and a new lease table is created making the old one obsolete for clean up. | ||
| 2. You have created multiple trigger functions that are obsolete which have associated tables that require cleaning up. | ||
|
MaddyDev marked this conversation as resolved.
Outdated
|
||
| 3. You want to reset your environment and start afresh. | ||
|
MaddyDev marked this conversation as resolved.
Outdated
|
||
| There are many a reason that require cleaning to maintain our databases and since we currently don't handle that within the extension, below are some scripts that guide you through cleaning the leases table without them filling up the database. | ||
|
MaddyDev marked this conversation as resolved.
Outdated
|
||
|
|
||
| - Deletes all the lease tables that haven't been accessed in N days: | ||
|
MaddyDev marked this conversation as resolved.
Outdated
|
||
|
|
||
| ```sql | ||
| -- Deletes all the lease tables that haven't been accesses in ? days(represented by @CleanupAgeDays variable and needs to defined before running the query). | ||
| -- And removes them from the GlobalState table. | ||
|
MaddyDev marked this conversation as resolved.
Outdated
|
||
| USE <Insert DATABASE name here>; | ||
| DECLARE @TableName NVARCHAR(MAX); | ||
| DECLARE @UserFunctionId char(16); | ||
|
MaddyDev marked this conversation as resolved.
|
||
| DECLARE @UserTableId int; | ||
| DECLARE @CleanupAgeDays int = <Insert desired cleanup age in days here>; | ||
| DECLARE LeaseTable_Cursor CURSOR FOR | ||
|
|
||
| SELECT 'az_func.Leases_'+UserFunctionId+'_'+convert(varchar(100),UserTableID) as TABLE_NAME, UserFunctionID, UserTableID | ||
| FROM az_func.GlobalState | ||
| WHERE DATEDIFF(day, LastAccessTime, GETDATE()) > @CleanupAgeDays | ||
|
|
||
| OPEN LeaseTable_Cursor; | ||
|
|
||
| FETCH NEXT FROM LeaseTable_Cursor INTO @TableName, @UserFunctionId, @UserTableId; | ||
|
|
||
| WHILE @@FETCH_STATUS = 0 | ||
| BEGIN | ||
| EXEC ('DROP TABLE IF EXISTS ' + @TableName); | ||
|
MaddyDev marked this conversation as resolved.
|
||
| DELETE FROM az_func.GlobalState WHERE UserFunctionID = @UserFunctionId and UserTableID = @UserTableId | ||
| FETCH NEXT FROM LeaseTable_Cursor INTO @TableName, @UserFunctionId, @UserTableId; | ||
| END; | ||
|
|
||
| CLOSE LeaseTable_Cursor; | ||
|
|
||
| DEALLOCATE LeaseTable_Cursor; | ||
| ``` | ||
|
|
||
| - Clean up a specific lease table concerning a specific function: | ||
|
MaddyDev marked this conversation as resolved.
Outdated
|
||
|
|
||
| To find the name of the leases table associated with your function, look in the log output for a line such as this which is emitted when the trigger is started. | ||
|
MaddyDev marked this conversation as resolved.
Outdated
|
||
|
|
||
| `SQL trigger Leases table: [az_func].[Leases_84d975fca0f7441a_901578250]` | ||
|
|
||
| This log message is at the `Information` level, so make sure your log level is set correctly. | ||
|
|
||
| ```sql | ||
| -- Deletes the specified lease table and removes it from GlobalState table. | ||
| USE <Insert DATABASE name here>; | ||
| DECLARE @TableName NVARCHAR(MAX) = <Insert lease table name here>; -- e.g. '[az_func].[Leases_84d975fca0f7441a_901578250] | ||
| DECLARE @UserFunctionId char(16) = <Insert function ID here>; -- e.g. '84d975fca0f7441a' | ||
|
MaddyDev marked this conversation as resolved.
Outdated
|
||
| DECLARE @UserTableId int = <Insert table ID here>; -- e.g. '901578250' | ||
|
|
||
| DROP TABLE IF EXISTS @TableName | ||
|
MaddyDev marked this conversation as resolved.
Outdated
|
||
| DELETE FROM az_func.GlobalState WHERE UserFunctionID = @UserFunctionId and UserTableID = @UserTableId | ||
| ``` | ||
|
|
||
| - Clear all trigger related data for a reset: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe just have the first script be able to take in -1 for the days value to clean up all tables instead? Help keep things simpler |
||
|
|
||
| ```sql | ||
| -- Deletes all the lease tables and clears them from the GlobalState table. | ||
| USE <Insert DATABASE name here>; | ||
| DECLARE @TableName NVARCHAR(MAX); | ||
| DECLARE @UserFunctionId char(16); | ||
| DECLARE @UserTableId int; | ||
| DECLARE LeaseTable_Cursor CURSOR FOR | ||
|
|
||
| SELECT 'az_func.Leases_'+UserFunctionId+'_'+convert(varchar(100),UserTableID) as TABLE_NAME, UserFunctionID, UserTableID | ||
| FROM az_func.GlobalState | ||
|
|
||
| OPEN LeaseTable_Cursor; | ||
|
|
||
| FETCH NEXT FROM LeaseTable_Cursor INTO @TableName, @UserFunctionId, @UserTableId; | ||
|
|
||
| WHILE @@FETCH_STATUS = 0 | ||
| BEGIN | ||
| EXEC ('DROP TABLE IF EXISTS ' + @TableName); | ||
| DELETE FROM az_func.GlobalState WHERE UserFunctionID = @UserFunctionId and UserTableID = @UserTableId | ||
| FETCH NEXT FROM LeaseTable_Cursor INTO @TableName, @UserFunctionId, @UserTableId; | ||
| END; | ||
|
|
||
| CLOSE LeaseTable_Cursor; | ||
|
|
||
| DEALLOCATE LeaseTable_Cursor; | ||
| ``` | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -955,7 +955,7 @@ LEFT OUTER JOIN {this._leasesTableName} AS l ON {leasesTableJoinCondition} | |
| IF @unprocessed_changes = 0 AND @current_last_sync_version < {newLastSyncVersion} | ||
| BEGIN | ||
| UPDATE {GlobalStateTableName} | ||
| SET LastSyncVersion = {newLastSyncVersion} | ||
| SET LastSyncVersion = {newLastSyncVersion}, LastAccessTime = GETDATE() | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should we be using GETDATE() or GETUTCDATE()?
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It shouldn't matter I think - it's all local to the server (so unless that changes timezones or something weird). That being said - using UTC seems reasonable to avoid that potential edge case. And that's a good general thing to default to for stored timestamps anyways.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Defaulted to GETUTCDATE() to avoid potential edge cases. |
||
| WHERE UserFunctionID = '{this._userFunctionId}' AND UserTableID = {this._userTableId}; | ||
|
|
||
| DELETE FROM {this._leasesTableName} WHERE {LeasesTableChangeVersionColumnName} <= {newLastSyncVersion}; | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -321,8 +321,12 @@ IF OBJECT_ID(N'{GlobalStateTableName}', 'U') IS NULL | |
| UserFunctionID char(16) NOT NULL, | ||
| UserTableID int NOT NULL, | ||
| LastSyncVersion bigint NOT NULL, | ||
| LastAccessTime Datetime NULL, | ||
|
MaddyDev marked this conversation as resolved.
Outdated
|
||
| PRIMARY KEY (UserFunctionID, UserTableID) | ||
| ); | ||
| ELSE IF NOT EXISTS(SELECT 1 FROM sys.columns WHERE Name = N'LastAccessTime' | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would like for us to have an explicit test for a lease table where the LastAccessTime column doesn't exist to validate that the (Is this possible, by the way, in the real world?)
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It will be for people with existing functions that just update the extension version but everything else stays the same. But that'll drop off as we march towards GA since after this change gets in no one should have that going forward.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree with @Charles-Gagnon here, but I can look into adding a test to be sure nothing breaks in the meantime. |
||
| AND Object_ID = Object_ID(N'{GlobalStateTableName}')) | ||
| ALTER TABLE {GlobalStateTableName} ADD LastAccessTime Datetime NULL; | ||
| "; | ||
|
|
||
| using (var createGlobalStateTableCommand = new SqlCommand(createGlobalStateTableQuery, connection, transaction)) | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.