perf(sql): Optimise searchForPipelinesByTrigger LIMIT and OFFSET SQL query

[christosarvanitis]

Searching for the executions of the pipelines of an application by triggerType and eventId (Gate endpoint: applications//executions/search?triggerTypes=&eventId=) query execution is not optimal and creates high response times as the execution history of the application increases and/or the application has a lot of pipelines.

The retrievePipelinesForPipelineConfigIdsBetweenBuildTimeBoundary creates SQL queries with LIMIT 20 and OFFSET that look like this:

select id, body, `partition`
from pipelines
where (config_id in (<List of PipelineConfigIds in App>) and build_time > 0 and build_time < 9223372036854775807)
order by build_time desc limit 20
offset X;

With this change the queries are optimised with a deferred join (see https://emmer.dev/blog/the-dangers-of-offset-with-mysql/). The new queries look like:

select id, body, `partition`
from pipelines
       join (select id
             from pipelines
             where (config_id in (<List of PipelineConfigIds in App>) and build_time > 0 and
                    build_time < 9223372036854775807)
             order by build_time desc limit 20
             offset X) as `alias_3417341` using (id)
order by build_time desc;

The above query lets the server to examine only the index (without accessing the rows) and once the desired rows are found is joined against the full table to retrieve the desired columns for the rows.

Explain plan of a single query on an App:

Before the change:

| -> Limit/Offset: 20/940 row(s)  (cost=6611.81 rows=20) (actual time=53.193..53.332 rows=20 loops=1)
    -> Sort row IDs: pipelines.build_time DESC, limit input to 960 row(s) per chunk  (cost=6611.81 rows=5464) (actual time=46.351..53.287 rows=960 loops=1)
        -> Filter: ((pipelines.build_time > 0) and (pipelines.build_time < 9223372036854775807))  (cost=6611.81 rows=5464) (actual time=0.039..45.104 rows=4069 loops=1)
            -> Index range scan on pipelines using pipeline_config_status_idx over (config_id = '00695444-3aaf-411f-885c-f33eab64f042') OR (config_id = '0a514ebd-48af-4e72-ade6-ad6dfe21d915') OR (53 more), with index condition: (pipelines.config_id in ('a4d2d50d-23bf-4056-859e-33612f4994a8','0a514ebd-48af-4e72-ade6-ad6dfe21d915','ac624358-a085-46ac-831f-9778c4de25a1','aa663791-9ffa-419c-9304-8c8541215147','ad72e8e8-6883-4dad-af55-8aac583ad8d1','98860c56-b194-4c5a-8801-20b466505f7f','5a399a3c-3a3c-4c38-9860-aa6d27c21e74','14f0ac1e-670e-4846-80e8-e6e4209a3f70','29537e2f-8579-4144-9f4b-4ca16ab26a90','c95e5042-3aad-4230-a963-a6e36e1e48ca','161f2634-1433-4f6c-b6fd-c9287b658a4f','8148bafe-cffb-4cff-b9f2-2e95e0363aa7','60898c77-78d3-459d-a4de-ab06eaa9d83c','f39e950d-05e7-4162-9782-93e21561a3d0','aa2233ff-f166-45c9-9a42-bb7c49c5d570','e04b1ea6-7917-420b-8033-eea7971f19b0','4aa565c3-23e2-4e46-bcbf-3b1345c1ff5c','c1c64527-4a3c-480f-801e-6c5288c1b307','f72d4e8b-313f-4f76-97b7-178b5da59830','553efe10-c518-47fb-92c7-7c8e5f664817','2af16c25-dc41-40b8-b5d2-74c9bd47657e','88775d81-e617-48dc-8541-d07bbbac51d4','11d7f242-9a7f-46ae-8ab0-5d4f94371472','f1118f7a-be20-4887-9a2d-e286f711e0fd','85256521-3ba4-4bda-9eba-0167c75d765f','6a0bb0f7-27fb-4fa8-bbbb-3e7280343163','74572724-1916-41a2-a5ba-4127d8bde6e2','9a03e487-2b4e-44ed-97bb-4f08de144e80','0a580c04-30fe-452b-9345-1fc3c22af963','0f588dbb-967d-4877-8cf5-330e7099fb5a','13b78394-d27e-44f8-83e4-dfc3594b8dd7','bf2f15c5-b33a-4bf7-abc7-e4ae5fc4bac4','ea6e5d2c-3bc7-4102-82f3-03c404a51f7f','00695444-3aaf-411f-885c-f33eab64f042','2ba3ff5a-f691-4de2-881b-9d3380092517','6ad3feb2-5a68-413d-9119-eb269a27df55','30d7ec83-7423-4822-9f54-8ff60df1d3a8','a9664a7b-6bd9-46e1-b5af-272d9ad72c02','2790b496-001a-4b77-8bf6-bbed632e374b','2021ed6f-520f-41a2-b176-c68d0a69189e','8fccf736-e264-4297-921d-87c9b67e0cfe','a992c414-7fcb-4a26-b9ed-7abb6859d887','e7bb89ef-cb6c-4d32-bd6e-cfcc35ac8e5e','619b3be4-08f8-4d41-bd45-280908c47069','56a9a0bc-9041-47d5-9bdd-211e76465925','f5eaf859-b211-4f57-a8bc-be8b362b0c83','300b2d2f-d543-44ac-b929-eb64cf206799','9c0cc17a-d653-4856-8773-0621f893bb0a','705e5241-a0ec-4ee3-bbee-ccf99c0cc1e3','fc2b6a16-3d2f-4ba3-bafe-cef6994a3a30','c03950a2-e6d9-40cf-8a8f-1bb2e399ad68','329039b2-15e0-4b83-a588-9d39a96d2779','ef21c762-73fb-4906-8a75-0159282b76a5','34cc73e6-4583-4adf-9c2f-3ebf6c497796','affb0bab-e061-4787-a513-9d077e7deaff'))  (cost=6611.81 rows=5464) (actual time=0.037..44.398 rows=4069 loops=1)

After the change:

| -> Sort row IDs: pipelines.build_time DESC  (actual time=49.685..49.686 rows=20 loops=1)
    -> Table scan on <temporary>  (cost=0.14..2.75 rows=20) (actual time=0.001..0.005 rows=20 loops=1)
        -> Temporary table  (cost=7230.10..7232.71 rows=20) (actual time=49.656..49.661 rows=20 loops=1)
            -> Nested loop inner join  (cost=7227.96 rows=20) (actual time=49.438..49.606 rows=20 loops=1)
                -> Table scan on alias_3417341  (cost=0.14..2.75 rows=20) (actual time=0.002..0.006 rows=20 loops=1)
                    -> Materialize  (cost=6616.35..6618.96 rows=20) (actual time=49.404..49.409 rows=20 loops=1)
                        -> Limit/Offset: 20/940 row(s)  (cost=6614.21 rows=20) (actual time=49.385..49.389 rows=20 loops=1)
                            -> Sort: pipelines.build_time DESC, limit input to 960 row(s) per chunk  (cost=6614.21 rows=5466) (actual time=48.277..49.347 rows=960 loops=1)
                                -> Filter: ((pipelines.build_time > 0) and (pipelines.build_time < 9223372036854775807))  (cost=6614.21 rows=5466) (actual time=0.079..46.673 rows=4070 loops=1)
                                    -> Index range scan on pipelines using pipeline_config_status_idx over (config_id = '00695444-3aaf-411f-885c-f33eab64f042') OR (config_id = '0a514ebd-48af-4e72-ade6-ad6dfe21d915') OR (53 more), with index condition: (pipelines.config_id in ('a4d2d50d-23bf-4056-859e-33612f4994a8','0a514ebd-48af-4e72-ade6-ad6dfe21d915','ac624358-a085-46ac-831f-9778c4de25a1','aa663791-9ffa-419c-9304-8c8541215147','ad72e8e8-6883-4dad-af55-8aac583ad8d1','98860c56-b194-4c5a-8801-20b466505f7f','5a399a3c-3a3c-4c38-9860-aa6d27c21e74','14f0ac1e-670e-4846-80e8-e6e4209a3f70','29537e2f-8579-4144-9f4b-4ca16ab26a90','c95e5042-3aad-4230-a963-a6e36e1e48ca','161f2634-1433-4f6c-b6fd-c9287b658a4f','8148bafe-cffb-4cff-b9f2-2e95e0363aa7','60898c77-78d3-459d-a4de-ab06eaa9d83c','f39e950d-05e7-4162-9782-93e21561a3d0','aa2233ff-f166-45c9-9a42-bb7c49c5d570','e04b1ea6-7917-420b-8033-eea7971f19b0','4aa565c3-23e2-4e46-bcbf-3b1345c1ff5c','c1c64527-4a3c-480f-801e-6c5288c1b307','f72d4e8b-313f-4f76-97b7-178b5da59830','553efe10-c518-47fb-92c7-7c8e5f664817','2af16c25-dc41-40b8-b5d2-74c9bd47657e','88775d81-e617-48dc-8541-d07bbbac51d4','11d7f242-9a7f-46ae-8ab0-5d4f94371472','f1118f7a-be20-4887-9a2d-e286f711e0fd','85256521-3ba4-4bda-9eba-0167c75d765f','6a0bb0f7-27fb-4fa8-bbbb-3e7280343163','74572724-1916-41a2-a5ba-4127d8bde6e2','9a03e487-2b4e-44ed-97bb-4f08de144e80','0a580c04-30fe-452b-9345-1fc3c22af963','0f588dbb-967d-4877-8cf5-330e7099fb5a','13b78394-d27e-44f8-83e4-dfc3594b8dd7','bf2f15c5-b33a-4bf7-abc7-e4ae5fc4bac4','ea6e5d2c-3bc7-4102-82f3-03c404a51f7f','00695444-3aaf-411f-885c-f33eab64f042','2ba3ff5a-f691-4de2-881b-9d3380092517','6ad3feb2-5a68-413d-9119-eb269a27df55','30d7ec83-7423-4822-9f54-8ff60df1d3a8','a9664a7b-6bd9-46e1-b5af-272d9ad72c02','2790b496-001a-4b77-8bf6-bbed632e374b','2021ed6f-520f-41a2-b176-c68d0a69189e','8fccf736-e264-4297-921d-87c9b67e0cfe','a992c414-7fcb-4a26-b9ed-7abb6859d887','e7bb89ef-cb6c-4d32-bd6e-cfcc35ac8e5e','619b3be4-08f8-4d41-bd45-280908c47069','56a9a0bc-9041-47d5-9bdd-211e76465925','f5eaf859-b211-4f57-a8bc-be8b362b0c83','300b2d2f-d543-44ac-b929-eb64cf206799','9c0cc17a-d653-4856-8773-0621f893bb0a','705e5241-a0ec-4ee3-bbee-ccf99c0cc1e3','fc2b6a16-3d2f-4ba3-bafe-cef6994a3a30','c03950a2-e6d9-40cf-8a8f-1bb2e399ad68','329039b2-15e0-4b83-a588-9d39a96d2779','ef21c762-73fb-4906-8a75-0159282b76a5','34cc73e6-4583-4adf-9c2f-3ebf6c497796','affb0bab-e061-4787-a513-9d077e7deaff'))  (cost=6614.21 rows=5466) (actual time=0.077..46.009 rows=4070 loops=1)
                -> Single-row index lookup on pipelines using PRIMARY (id=alias_3417341.id)  (cost=1.00 rows=1) (actual time=0.009..0.010 rows=1 loops=20)

With this change on simulated traffic for the above Application we have seen drop in response time from ~5.6sec to ~4.5sec.

[christosarvanitis]

Thank you @dbyron-sf. What is your view on backporting this to 1.33.x and 1.32.x?

[dbyron-sf]

I'm not feeling so paranoid today, so let's do it :)

[dbyron-sf]

@Mergifyio backport release-1.32.x release-1.33.x

[mergify]

backport release-1.32.x release-1.33.x

✅ Backports have been created

• #4699 perf(sql): Optimise searchForPipelinesByTrigger LIMIT and OFFSET SQL query (backport #4698) has been created for branch release-1.32.x
• #4700 perf(sql): Optimise searchForPipelinesByTrigger LIMIT and OFFSET SQL query (backport #4698) has been created for branch release-1.33.x