diff --git a/src/Microsoft.Data.SqlClient/tests/StressTests/Directory.Build.props b/src/Microsoft.Data.SqlClient/tests/StressTests/Directory.Build.props new file mode 100644 index 0000000000..958cc8e38f --- /dev/null +++ b/src/Microsoft.Data.SqlClient/tests/StressTests/Directory.Build.props @@ -0,0 +1,6 @@ + + + + + + diff --git a/src/Microsoft.Data.SqlClient/tests/StressTests/Directory.Packages.props b/src/Microsoft.Data.SqlClient/tests/StressTests/Directory.Packages.props new file mode 100644 index 0000000000..b4f3c5f805 --- /dev/null +++ b/src/Microsoft.Data.SqlClient/tests/StressTests/Directory.Packages.props @@ -0,0 +1,14 @@ + + + + + true + true + + + + + + + + diff --git a/src/Microsoft.Data.SqlClient/tests/StressTests/IMonitorLoader/IMonitorLoader.csproj b/src/Microsoft.Data.SqlClient/tests/StressTests/IMonitorLoader/IMonitorLoader.csproj index dd20b98624..5d04960f79 100644 --- a/src/Microsoft.Data.SqlClient/tests/StressTests/IMonitorLoader/IMonitorLoader.csproj +++ b/src/Microsoft.Data.SqlClient/tests/StressTests/IMonitorLoader/IMonitorLoader.csproj @@ -2,7 +2,7 @@ Monitoring Monitoring - net9.0;net48 + net481;net9.0 latest - \ No newline at end of file + diff --git a/src/Microsoft.Data.SqlClient/tests/StressTests/Readme.md b/src/Microsoft.Data.SqlClient/tests/StressTests/Readme.md index 4d74643814..16ec09a874 100644 --- a/src/Microsoft.Data.SqlClient/tests/StressTests/Readme.md +++ b/src/Microsoft.Data.SqlClient/tests/StressTests/Readme.md @@ -2,163 +2,182 @@ This Stress testing application for `Microsoft.Data.SqlClient` is under progress. This project intends to help finding a certain level of effectiveness under unfavorable conditions, and verifying the mode of failures. -This is a console application with targeting frameworks `.Net Framework 4.8`, `.NET 9.0` under driver's supported operating systems and SQL Servers. +This is a console application with targeting frameworks `.Net Framework 4.8.1`, `.NET 9.0` under driver's supported operating systems and SQL Servers. ## Purpose of application for developers Define fuzz tests for all new features/APIs in the driver and to be run before every GA release. -# Pre-Requisites +## Pre-Requisites -required in StressTest.config: +Required in StressTest.config: -"name": stress testing source configuration name. -"type": only `SqlServer` is acceptable. -"isDefault": If there is a source node with `isDefault=true`, this node is returned. -"dataSource": SQL Server data source name. -"database": targeting database name in the SQL Server. -"user": user Id to connect the server. -"password": paired password with the user. -"supportsWindowsAuthentication": tries to use integrated security in connection string mixed with SQL Server authentication if it set to `true` by applying the randomization. -"isLocal": `true` means database is local. -"disableMultiSubnetFailover": tries to add Multi-subnet Failover fake host entries when it equals `true`, -"disableNamedPipes": `true` means the connections will create just using tcp protocol. -"encrypt": assigns the encrypt property of the connection strings. +|Field|Values|Description| +|-|-|-| +|`name`||Stress testing source configuration name.| +|`type`|`SqlServer`|Only `SqlServer` is acceptable.| +|`isDefault`|`true`, `false`|If there is a source node with `isDefault=true`, this node is returned.| +|`dataSource`||SQL Server data source name.| +|`database`||Targeting database name in the SQL Server.| +|`user`||User Id to connect the server.| +|`password`||Paired password with the user.| +|`supportsWindowsAuthentication`|`true`, `false`|Tries to use integrated security in connection string mixed with SQL Server authentication if it set to `true` by applying the randomization.| +|`isLocal`|`true`, `false`|`true` means database is local.| +|`disableMultiSubnetFailover`|`true`, `false`|Tries to add Multi-subnet Failover fake host entries when it equals `true`.| +|`disableNamedPipes`|`true`, `false`|`true` means the connections will create just using tcp protocol.| +|`encrypt`|`true`, `false`|Assigns the encrypt property of the connection strings.| -# Adding new Tests -- [ToDO] +## Adding new Tests -# Building the application +- [ToDo] -To build the application we need to run the command: 'dotnet build <-f|--framework > [-c|--configuration ]' -The path should be pointing to SqlClient.Stress.Runner.csproj file. +## Building the application + +To build the application we need to run the command: ```bash -# Default Build Configuration: +dotnet build <-f|--framework > [-c|--configuration ] +``` -> dotnet build +The path should be pointing to SqlClient.Stress.Runner.csproj file. + +```bash # Builds the application for the Client Os in `Debug` Configuration for `AnyCpu` platform. # All supported target frameworks, .NET Framework (NetFx) and .NET Core drivers are built by default (as supported by Client OS). + +> dotnet build ``` ```bash -> dotnet build -f netcoreapp3.1 -# Build the application for .Net core 3.1 with `Debug` configuration. +# Build the application for .Net framework 4.8.1 with `Debug` configuration. -> dotnet build -f net48 -# Build the application for .Net framework 4.8 with `Debug` configuration. +> dotnet build -f net481 ``` ```bash -> dotnet build -f net5.0 -c Release -# Build the application for .Net 5.0 with `Release` configuration. +# Build the application for .Net 9.0 with `Release` configuration. -> dotnet build -f net48 -c Release -# Build the application for .Net framework 4.8 with `Release` configuration. +> dotnet build -f net9.0 -c Release ``` ```bash -> dotnet clean # Cleans all build directories + +> dotnet clean ``` -# Running tests +## Running tests After building the application, find the built folder with target framework and run the `stresstest.exe` file with required arguments. Find the result in a log file inside the `logs` folder besides the command prompt. ## Command prompt -```bash -> stresstest.exe [-a ] - --a should specify path to the assembly containing the tests. -``` - -## Supported arguments - --all Run all tests - best for debugging, not perf measurements. +You must run the stress tests from the root of the Stress Tests project +directory (i.e. the same directory this readme file is in). --verify Run in functional verification mode. [not implemented] - --duration Duration of the test in seconds. Default value is 1 second. - --threads Number of threads to use. Default value is 16. - --override Override the value of a test property. - --test Run specific test(s). +```bash +# Linux +$ cd /home/paul/dev/SqlClient/src/Microsoft.Data.SqlClient/tests/StressTests --debug Print process ID in the beginning and wait for Enter (to give your time to attach the debugger). +# Via dotnet run CLI: +$ dotnet run -no-build -f net9.0 --project SqlClient.Stress.Runner/SqlClient.Stress.Runner.csproj -- -a SqlClient.Stress.Tests --exceptionThreshold An optional limit on exceptions which will be caught. When reached, test will halt. +# Via dotnet CLI: +$ dotnet SqlClient.Stress.Runner/bin/Debug/net9.0/stresstest.dll -a SqlClient.Stress.Tests +``` --monitorenabled True or False to enable monitoring. Default is false [not implemented] +```powershell +# Windows +> cd \dev\SqlClient\src\Microsoft.Data.SqlClient\tests\StressTests --randomSeed Enables setting of the random number generator used internally. This serves both the purpose - of helping to improve reproducibility and making it deterministic from Chess's perspective - for a given schedule. Default is 0. +# Via dotnet run CLI: +$ dotnet run -no-build -f net9.0 --project SqlClient.Stress.Runner\SqlClient.Stress.Runner.csproj -- -a SqlClient.Stress.Tests --filter Run tests whose stress test attributes match the given filter. Filter is not applied if attribute - does not implement ITestAttributeFilter. Example: -filter TestType=Query,Update;IsServerTest=True +# Via executable: +> .\SqlClient.Stress.Runner\bin\Debug\net481\stresstest.exe -a SqlClient.Stress.Tests +``` --printMethodName Print tests' title in console window +## Supported arguments --deadlockdetection True or False to enable deadlock detection. Default is `false`. +|Argument|Values|Description| +|-|-|-| +|-all||Run all tests - best for debugging, not perf measurements.| +|-verify||Run in functional verification mode. [not implemented]| +|-duration|<n>|Duration of the test in seconds. Default value is 1 second.| +|-threads|<n>|Number of threads to use. Default value is 16.| +|-override|<name> <value>|Override the value of a test property.| +|-test|<name1;name2>|Run specific test(s).| +|-debug||Print process ID in the beginning and wait for Enter (to give your time to attach the debugger).| +|-exceptionThreshold|<n>|An optional limit on exceptions which will be caught. When reached, test will halt.| +|-monitorenabled|true, false|True or False to enable monitoring. Default is false [not implemented]| +|-randomSeed||Enables setting of the random number generator used internally. This serves both the purpose of helping to improve reproducibility and making it deterministic from Chess's perspective for a given schedule. Default is 0.| +|-filter|<filter>|Run tests whose stress test attributes match the given filter. Filter is not applied if attribute does not implement ITestAttributeFilter. Example: -filter TestType=Query,Update;IsServerTest=True| +|-printMethodName||Print tests' title in console window| +|-deadlockdetection|true, false|True or False to enable deadlock detection. Default is `false`.| ```bash -> stresstest.exe -a SqlClient.Stress.Tests -all # Run the application for a built target framework and all discovered tests without debugger attached. + +> .\stresstest.exe -a SqlClient.Stress.Tests -all ``` ```bash -> stresstest.exe -a SqlClient.Stress.Tests -all -printMethodName # Run the application for a built target framework and all discovered tests without debugger attached and shows the test methods' names. + +> .\stresstest.exe -a SqlClient.Stress.Tests -all -printMethodName ``` ```bash -> stresstest.exe -a SqlClient.Stress.Tests -all -debug # Run the application for a built target framework and all discovered tests and will wait for debugger to be attached. + +> .\stresstest.exe -a SqlClient.Stress.Tests -all -debug ``` ```bash -> stresstest.exe -a SqlClient.Stress.Tests -test TestExecuteXmlReaderAsyncCancellation # Run the application for a built target framework and "TestExecuteXmlReaderAsyncCancellation" test without debugger attached. + +> .\stresstest.exe -a SqlClient.Stress.Tests -test TestExecuteXmlReaderAsyncCancellation ``` ```bash -> stresstest.exe -a SqlClient.Stress.Tests -test TestExecuteXmlReaderAsyncCancellation # Run the application for a built target framework and "TestExecuteXmlReaderAsyncCancellation" test without debugger attached. + +> .\stresstest.exe -a SqlClient.Stress.Tests -test TestExecuteXmlReaderAsyncCancellation ``` ```bash -> stresstest.exe -a SqlClient.Stress.Tests -all -duration 10 # Run the application for a built target framework and all discovered tests without debugger attached for 10 seconds. + +> .\stresstest.exe -a SqlClient.Stress.Tests -all -duration 10 ``` ```bash -> stresstest.exe -a SqlClient.Stress.Tests -all -threads 5 # Run the application for a built target framework and all discovered tests without debugger attached with 5 threads. + +> .\stresstest.exe -a SqlClient.Stress.Tests -all -threads 5 ``` ```bash -> stresstest.exe -a SqlClient.Stress.Tests -all -deadlockdetection true # Run the application for a built target framework and all discovered tests without debugger attached and dead lock detection process. + +> .\stresstest.exe -a SqlClient.Stress.Tests -all -deadlockdetection true ``` ```bash -> stresstest.exe -a SqlClient.Stress.Tests -all -override Weight 15 # Run the application for a built target framework and all discovered tests without debugger attached with overriding the weight property with value 15. + +> .\stresstest.exe -a SqlClient.Stress.Tests -all -override Weight 15 ``` ```bash -> stresstest.exe -a SqlClient.Stress.Tests -all -randomSeed 5 # Run the application for a built target framework and all discovered tests without debugger attached with injecting random seed of 5. + +> .\stresstest.exe -a SqlClient.Stress.Tests -all -randomSeed 5 ``` -# Further thoughts +## Further thoughts - Implement the uncompleted arguments. -- Add support `dotnet run` command. - Add more tests. - Add support running tests with **System.Data.SqlClient** too. diff --git a/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Common/SqlClient.Stress.Common.csproj b/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Common/SqlClient.Stress.Common.csproj index e8885e2a05..6950cb6af8 100644 --- a/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Common/SqlClient.Stress.Common.csproj +++ b/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Common/SqlClient.Stress.Common.csproj @@ -1,7 +1,7 @@ - net9.0;net48 + net481;net9.0 latest diff --git a/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Framework/DataStressFactory.cs b/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Framework/DataStressFactory.cs index 4d49b6ee35..b35f6fa7b7 100644 --- a/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Framework/DataStressFactory.cs +++ b/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Framework/DataStressFactory.cs @@ -789,7 +789,7 @@ protected void BuildUserTables(List TableMetadataList) } else { - throw de; + throw; } } @@ -808,7 +808,7 @@ protected void BuildUserTables(List TableMetadataList) } else { - throw de; + throw; } } diff --git a/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Framework/DataTestGroup.cs b/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Framework/DataTestGroup.cs index 9039997d61..96f2d16d6c 100644 --- a/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Framework/DataTestGroup.cs +++ b/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Framework/DataTestGroup.cs @@ -158,10 +158,8 @@ protected bool OpenConnection(DataStressConnection conn) // MultiSubnetFailover=true only works with TCP/IP protocol and will result in exception when using with named pipes. return false; } - else - { - throw e; - } + + throw; } } diff --git a/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Framework/SqlClient.Stress.Framework.csproj b/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Framework/SqlClient.Stress.Framework.csproj index a1623af521..e76a05f52a 100644 --- a/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Framework/SqlClient.Stress.Framework.csproj +++ b/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Framework/SqlClient.Stress.Framework.csproj @@ -2,12 +2,12 @@ Stress.Data - net9.0;net48 + net481;net9.0 latest - + diff --git a/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Framework/StressTest.config b/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Framework/StressTest.config index 60fc0bb86e..7a6a422924 100644 --- a/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Framework/StressTest.config +++ b/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Framework/StressTest.config @@ -3,18 +3,18 @@ + supportsWindowsAuthentication="false" + isLocal="false" + disableMultiSubnetFailover = "true" + disableNamedPipes = "true" + encrypt ="false"> diff --git a/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Runner/SqlClient.Stress.Runner.csproj b/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Runner/SqlClient.Stress.Runner.csproj index 36b5f2eee5..23aebda620 100644 --- a/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Runner/SqlClient.Stress.Runner.csproj +++ b/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Runner/SqlClient.Stress.Runner.csproj @@ -3,7 +3,7 @@ Exe stresstest - net9.0;net48 + net481;net9.0 latest diff --git a/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Runner/Tests/TestBase.cs b/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Runner/Tests/TestBase.cs index 3d1ee1df9b..95546547da 100644 --- a/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Runner/Tests/TestBase.cs +++ b/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Runner/Tests/TestBase.cs @@ -13,15 +13,12 @@ public abstract class TestBase private TestAttributeBase _attr; private string _variationSuffix = ""; - [System.CLSCompliantAttribute(false)] protected MethodInfo _testMethod; - [System.CLSCompliantAttribute(false)] + protected Type _type; - [System.CLSCompliantAttribute(false)] protected List _setupMethods; - [System.CLSCompliantAttribute(false)] protected List _cleanupMethods; protected delegate void TestMethodDelegate(object t); diff --git a/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Tests/SqlClient.Stress.Tests.csproj b/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Tests/SqlClient.Stress.Tests.csproj index 0ba45a34de..6b03c2a5af 100644 --- a/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Tests/SqlClient.Stress.Tests.csproj +++ b/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Tests/SqlClient.Stress.Tests.csproj @@ -2,12 +2,12 @@ Stress.Data.SqlClient - net9.0;net48 + net481;net9.0 latest - + diff --git a/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClientStressTest.sln b/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClientStressTest.sln deleted file mode 100644 index 070280780c..0000000000 --- a/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClientStressTest.sln +++ /dev/null @@ -1,49 +0,0 @@ - -Microsoft Visual Studio Solution File, Format Version 12.00 -# Visual Studio Version 17 -VisualStudioVersion = 17.14.36327.8 d17.14 -MinimumVisualStudioVersion = 10.0.40219.1 -Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "SqlClient.Stress.Runner", "SqlClient.Stress.Runner\SqlClient.Stress.Runner.csproj", "{9FF04E04-A221-4674-B5BD-DDAF5C25A808}" -EndProject -Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "IMonitorLoader", "IMonitorLoader\IMonitorLoader.csproj", "{AF78BA88-6428-47EA-8682-442DAF8E9656}" -EndProject -Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "SqlClient.Stress.Tests", "SqlClient.Stress.Tests\SqlClient.Stress.Tests.csproj", "{27058C62-79B0-4CCC-9B07-77FB30174424}" -EndProject -Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "SqlClient.Stress.Framework", "SqlClient.Stress.Framework\SqlClient.Stress.Framework.csproj", "{D3640C5E-6CA6-444B-8286-66A6E2B0A62A}" -EndProject -Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "SqlClient.Stress.Common", "SqlClient.Stress.Common\SqlClient.Stress.Common.csproj", "{7D38C1CB-1F15-4406-9F0F-0062B1AAFE73}" -EndProject -Global - GlobalSection(SolutionConfigurationPlatforms) = preSolution - Debug|Any CPU = Debug|Any CPU - Release|Any CPU = Release|Any CPU - EndGlobalSection - GlobalSection(ProjectConfigurationPlatforms) = postSolution - {9FF04E04-A221-4674-B5BD-DDAF5C25A808}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {9FF04E04-A221-4674-B5BD-DDAF5C25A808}.Debug|Any CPU.Build.0 = Debug|Any CPU - {9FF04E04-A221-4674-B5BD-DDAF5C25A808}.Release|Any CPU.ActiveCfg = Release|Any CPU - {9FF04E04-A221-4674-B5BD-DDAF5C25A808}.Release|Any CPU.Build.0 = Release|Any CPU - {AF78BA88-6428-47EA-8682-442DAF8E9656}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {AF78BA88-6428-47EA-8682-442DAF8E9656}.Debug|Any CPU.Build.0 = Debug|Any CPU - {AF78BA88-6428-47EA-8682-442DAF8E9656}.Release|Any CPU.ActiveCfg = Release|Any CPU - {AF78BA88-6428-47EA-8682-442DAF8E9656}.Release|Any CPU.Build.0 = Release|Any CPU - {27058C62-79B0-4CCC-9B07-77FB30174424}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {27058C62-79B0-4CCC-9B07-77FB30174424}.Debug|Any CPU.Build.0 = Debug|Any CPU - {27058C62-79B0-4CCC-9B07-77FB30174424}.Release|Any CPU.ActiveCfg = Release|Any CPU - {27058C62-79B0-4CCC-9B07-77FB30174424}.Release|Any CPU.Build.0 = Release|Any CPU - {D3640C5E-6CA6-444B-8286-66A6E2B0A62A}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {D3640C5E-6CA6-444B-8286-66A6E2B0A62A}.Debug|Any CPU.Build.0 = Debug|Any CPU - {D3640C5E-6CA6-444B-8286-66A6E2B0A62A}.Release|Any CPU.ActiveCfg = Release|Any CPU - {D3640C5E-6CA6-444B-8286-66A6E2B0A62A}.Release|Any CPU.Build.0 = Release|Any CPU - {7D38C1CB-1F15-4406-9F0F-0062B1AAFE73}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {7D38C1CB-1F15-4406-9F0F-0062B1AAFE73}.Debug|Any CPU.Build.0 = Debug|Any CPU - {7D38C1CB-1F15-4406-9F0F-0062B1AAFE73}.Release|Any CPU.ActiveCfg = Release|Any CPU - {7D38C1CB-1F15-4406-9F0F-0062B1AAFE73}.Release|Any CPU.Build.0 = Release|Any CPU - EndGlobalSection - GlobalSection(SolutionProperties) = preSolution - HideSolutionNode = FALSE - EndGlobalSection - GlobalSection(ExtensibilityGlobals) = postSolution - SolutionGuid = {6DF4150E-D32E-4377-BA4F-B4362EBF81DF} - EndGlobalSection -EndGlobal diff --git a/src/Microsoft.Data.SqlClient/tests/StressTests/StressTests.slnx b/src/Microsoft.Data.SqlClient/tests/StressTests/StressTests.slnx new file mode 100644 index 0000000000..a0dfd5e503 --- /dev/null +++ b/src/Microsoft.Data.SqlClient/tests/StressTests/StressTests.slnx @@ -0,0 +1,7 @@ + + + + + + +