Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 92 additions & 1 deletion docs/BindingsOverview.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
- [Startup retries](#startup-retries)
- [Broken connection retries](#broken-connection-retries)
- [Function exception retries](#function-exception-retries)
- [Lease Tables clean up](#lease-tables-clean-up)

## Input Binding

Expand Down Expand Up @@ -147,7 +148,7 @@ If an exception occurs in the user function when processing changes then the bat

If the function execution fails 5 times in a row for a given row then that row is completely ignored for all future changes. Because the rows in a batch are not deterministic, rows in a failed batch may end up in different batches in subsequent invocations. This means that not all rows in the failed batch will necessarily be ignored. If other rows in the batch were the ones causing the exception, the "good" rows may end up in a different batch that doesn't fail in future invocations.

You can run this query to see what rows have failed 5 times and are currently ignored, see [Leases table](#az_funcleases_) documentation for how to get the correct Leases table to query for your function.
You can run this query to see what rows have failed 5 times and are currently ignored, see [Leases table](./TriggerBinding.md#az_funcleases_) documentation for how to get the correct Leases table to query for your function.

```sql
SELECT * FROM [az_func].[Leases_<FunctionId>_<TableId>] WHERE _az_func_AttemptCount = 5
Expand All @@ -168,3 +169,93 @@ e.g.
```sql
UPDATE [Products].[az_func].[Leases_<FunctionId>_<TableId>] SET _az_func_AttemptCount = 0 WHERE ProductId = 123
```

#### Lease Tables clean up

Before clean up, please see [Leases table](./TriggerBinding.md#az_funcleases_) documentation for understanding how they are created and used.

Why clean up?
1. You renamed your function/class/method name and a new lease table is created making the old one obsolete for clean up.
Comment thread
MaddyDev marked this conversation as resolved.
Outdated
2. You have created multiple trigger functions that are obsolete which have associated tables that require cleaning up.
Comment thread
MaddyDev marked this conversation as resolved.
Outdated
3. You want to reset your environment and start afresh.
Comment thread
MaddyDev marked this conversation as resolved.
Outdated
There are many a reason that require cleaning to maintain our databases and since we currently don't handle that within the extension, below are some scripts that guide you through cleaning the leases table without them filling up the database.
Comment thread
MaddyDev marked this conversation as resolved.
Outdated

- Deletes all the lease tables that haven't been accessed in N days:
Comment thread
MaddyDev marked this conversation as resolved.
Outdated

```sql
-- Deletes all the lease tables that haven't been accesses in ? days(represented by @CleanupAgeDays variable and needs to defined before running the query).
-- And removes them from the GlobalState table.
Comment thread
MaddyDev marked this conversation as resolved.
Outdated
USE <Insert DATABASE name here>;
DECLARE @TableName NVARCHAR(MAX);
DECLARE @UserFunctionId char(16);
Comment thread
MaddyDev marked this conversation as resolved.
DECLARE @UserTableId int;
DECLARE @CleanupAgeDays int = <Insert desired cleanup age in days here>;
DECLARE LeaseTable_Cursor CURSOR FOR

SELECT 'az_func.Leases_'+UserFunctionId+'_'+convert(varchar(100),UserTableID) as TABLE_NAME, UserFunctionID, UserTableID
FROM az_func.GlobalState
WHERE DATEDIFF(day, LastAccessTime, GETDATE()) > @CleanupAgeDays

OPEN LeaseTable_Cursor;

FETCH NEXT FROM LeaseTable_Cursor INTO @TableName, @UserFunctionId, @UserTableId;

WHILE @@FETCH_STATUS = 0
BEGIN
EXEC ('DROP TABLE IF EXISTS ' + @TableName);
Comment thread
MaddyDev marked this conversation as resolved.
DELETE FROM az_func.GlobalState WHERE UserFunctionID = @UserFunctionId and UserTableID = @UserTableId
FETCH NEXT FROM LeaseTable_Cursor INTO @TableName, @UserFunctionId, @UserTableId;
END;

CLOSE LeaseTable_Cursor;

DEALLOCATE LeaseTable_Cursor;
```

- Clean up a specific lease table concerning a specific function:
Comment thread
MaddyDev marked this conversation as resolved.
Outdated

To find the name of the leases table associated with your function, look in the log output for a line such as this which is emitted when the trigger is started.
Comment thread
MaddyDev marked this conversation as resolved.
Outdated

`SQL trigger Leases table: [az_func].[Leases_84d975fca0f7441a_901578250]`

This log message is at the `Information` level, so make sure your log level is set correctly.

```sql
-- Deletes the specified lease table and removes it from GlobalState table.
USE <Insert DATABASE name here>;
DECLARE @TableName NVARCHAR(MAX) = <Insert lease table name here>; -- e.g. '[az_func].[Leases_84d975fca0f7441a_901578250]
DECLARE @UserFunctionId char(16) = <Insert function ID here>; -- e.g. '84d975fca0f7441a'
Comment thread
MaddyDev marked this conversation as resolved.
Outdated
DECLARE @UserTableId int = <Insert table ID here>; -- e.g. '901578250'

DROP TABLE IF EXISTS @TableName
Comment thread
MaddyDev marked this conversation as resolved.
Outdated
DELETE FROM az_func.GlobalState WHERE UserFunctionID = @UserFunctionId and UserTableID = @UserTableId
```

- Clear all trigger related data for a reset:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe just have the first script be able to take in -1 for the days value to clean up all tables instead? Help keep things simpler


```sql
-- Deletes all the lease tables and clears them from the GlobalState table.
USE <Insert DATABASE name here>;
DECLARE @TableName NVARCHAR(MAX);
DECLARE @UserFunctionId char(16);
DECLARE @UserTableId int;
DECLARE LeaseTable_Cursor CURSOR FOR

SELECT 'az_func.Leases_'+UserFunctionId+'_'+convert(varchar(100),UserTableID) as TABLE_NAME, UserFunctionID, UserTableID
FROM az_func.GlobalState

OPEN LeaseTable_Cursor;

FETCH NEXT FROM LeaseTable_Cursor INTO @TableName, @UserFunctionId, @UserTableId;

WHILE @@FETCH_STATUS = 0
BEGIN
EXEC ('DROP TABLE IF EXISTS ' + @TableName);
DELETE FROM az_func.GlobalState WHERE UserFunctionID = @UserFunctionId and UserTableID = @UserTableId
FETCH NEXT FROM LeaseTable_Cursor INTO @TableName, @UserFunctionId, @UserTableId;
END;

CLOSE LeaseTable_Cursor;

DEALLOCATE LeaseTable_Cursor;
```
2 changes: 1 addition & 1 deletion src/TriggerBinding/SqlTableChangeMonitor.cs
Original file line number Diff line number Diff line change
Expand Up @@ -955,7 +955,7 @@ LEFT OUTER JOIN {this._leasesTableName} AS l ON {leasesTableJoinCondition}
IF @unprocessed_changes = 0 AND @current_last_sync_version < {newLastSyncVersion}
BEGIN
UPDATE {GlobalStateTableName}
SET LastSyncVersion = {newLastSyncVersion}
SET LastSyncVersion = {newLastSyncVersion}, LastAccessTime = GETDATE()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be using GETDATE() or GETUTCDATE()?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It shouldn't matter I think - it's all local to the server (so unless that changes timezones or something weird).

That being said - using UTC seems reasonable to avoid that potential edge case. And that's a good general thing to default to for stored timestamps anyways.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Defaulted to GETUTCDATE() to avoid potential edge cases.

WHERE UserFunctionID = '{this._userFunctionId}' AND UserTableID = {this._userTableId};

DELETE FROM {this._leasesTableName} WHERE {LeasesTableChangeVersionColumnName} <= {newLastSyncVersion};
Expand Down
4 changes: 4 additions & 0 deletions src/TriggerBinding/SqlTriggerListener.cs
Original file line number Diff line number Diff line change
Expand Up @@ -321,8 +321,12 @@ IF OBJECT_ID(N'{GlobalStateTableName}', 'U') IS NULL
UserFunctionID char(16) NOT NULL,
UserTableID int NOT NULL,
LastSyncVersion bigint NOT NULL,
LastAccessTime Datetime NULL,
Comment thread
MaddyDev marked this conversation as resolved.
Outdated
PRIMARY KEY (UserFunctionID, UserTableID)
);
ELSE IF NOT EXISTS(SELECT 1 FROM sys.columns WHERE Name = N'LastAccessTime'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like for us to have an explicit test for a lease table where the LastAccessTime column doesn't exist to validate that the ELSE IF NOT EXISTS code path works as intended.

(Is this possible, by the way, in the real world?)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will be for people with existing functions that just update the extension version but everything else stays the same. But that'll drop off as we march towards GA since after this change gets in no one should have that going forward.

@MaddyDev MaddyDev May 26, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @Charles-Gagnon here, but I can look into adding a test to be sure nothing breaks in the meantime.

AND Object_ID = Object_ID(N'{GlobalStateTableName}'))
ALTER TABLE {GlobalStateTableName} ADD LastAccessTime Datetime NULL;
";

using (var createGlobalStateTableCommand = new SqlCommand(createGlobalStateTableQuery, connection, transaction))
Expand Down