[v7r3] tornado-based https JobStateUpdate handler #5214

fstagni · 2021-06-21T08:41:34Z

BEGINRELEASENOTES

*WMS
NEW: Added a HTTPs JobManager service
NEW: Added a HTTPs JobMonitor service
NEW: Added a HTTPs JobStateUpdate service
CHANGE: Removed need for ThreadScheduler in JobManagerHandler

ENDRELEASENOTES

chaen · 2021-06-23T09:06:16Z

 {'Errno': 0, 'Message': u'404 Client Error: Not Found for url: https://server:8443/WorkloadManagement/TornadoMatcher: <html><title>404: Not Found</title><body>404: Not Found</body></html>', 'OK': False, 'rpcStub': (('WorkloadManagement/Matcher', {'skipCACheck': False, 'timeout': 600}), 'requestJob', [{'OwnerGroup': 'prod', 'OwnerDN': '/C=ch/O=DIRAC/OU=DIRAC CI/CN=ciuser', 'DIRACVersion': 'pippo', 'PilotInfoReportedFlag': 'True', 'VirtualOrganization': 'LHCb', 'PilotBenchmark': 'anotherPilot', 'ReleaseVersion': 'blabla', 'Site': 'DIRAC.Jenkins.ch', 'CPUTime': 86400}]), 'CallStack': ['  File "/home/dirac/LocalRepo/TestCode/DIRAC/tests/Integration/WorkloadManagementSystem/Test_Client_WMS.py", line 541, in <module>\n    testResult = unittest.TextTestRunner(verbosity=2).run(suite)\n', '  File "/home/dirac/ServerInstallDIR/diracos/usr/lib64/python2.7/unittest/runner.py", line 151, in run\n    test(result)\n', '  File "/home/dirac/ServerInstallDIR/diracos/usr/lib64/python2.7/unittest/suite.py", line 70, in __call__\n    return self.run(*args, **kwds)\n', '  File "/home/dirac/ServerInstallDIR/diracos/usr/lib64/python2.7/unittest/suite.py", line 108, in run\n    test(result)\n', '  File "/home/dirac/ServerInstallDIR/diracos/usr/lib64/python2.7/unittest/suite.py", line 70, in __call__\n    return self.run(*args, **kwds)\n', '  File "/home/dirac/ServerInstallDIR/diracos/usr/lib64/python2.7/unittest/suite.py", line 108, in run\n    test(result)\n', '  File "/home/dirac/ServerInstallDIR/diracos/usr/lib64/python2.7/unittest/case.py", line 431, in __call__\n    return self.run(*args, **kwds)\n', '  File "/home/dirac/ServerInstallDIR/diracos/usr/lib64/python2.7/unittest/case.py", line 367, in run\n    testMethod()\n', '  File "/home/dirac/LocalRepo/TestCode/DIRAC/tests/Integration/WorkloadManagementSystem/Test_Client_WMS.py", line 528, in test_matcher\n    res = MatcherClient().requestJob(resourceDescription)\n', '  File "/home/dirac/ServerInstallDIR/DIRAC/Core/Base/Client.py", line 112, in executeRPC\n    return getattr(rpcClient, toExecute)(*parms)\n', '  File "/home/dirac/ServerInstallDIR/DIRAC/Core/Tornado/Client/TornadoClient.py", line 53, in call\n    return self.executeRPC(attrname, *args)\n', '  File "/home/dirac/ServerInstallDIR/DIRAC/Core/Tornado/Client/TornadoClient.py", line 67, in executeRPC\n    retVal = self._request(**rpcCall)\n', '  File "/home/dirac/ServerInstallDIR/DIRAC/Core/Tornado/Client/private/TornadoBaseClient.py", line 581, in _request\n    return S_ERROR(errStr)\n']}

The URL should not change for the last part, it should still be WorkloadManagement/Matcher.

Also, the matcher was one of those service that couldn't be duplicated. If it's a tornado service, it's going to be massively duplicated :-)

chaen · 2021-06-23T09:22:43Z

I think it is unrelated, but it may be worth noting this error:

2021-06-21 15:53:40 UTC Transformation/DataRecoveryAgent [140662451406656] ERROR: Failure to get transformations MySQL Error ( 1131 : Execution failed.: ( 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ') LIMIT 100' at line 1 ))

chaen · 2021-06-23T09:40:31Z

The URL should not change for the last part, it should still be WorkloadManagement/Matcher.

Forget what I said, it's correct.
Yet it is strange that Tornado does not find the handler

chaen · 2021-06-23T09:47:40Z

maybe what's missing is simply a restart of Tornado here

DIRAC/tests/CI/install_server.sh

Line 99 in cf55bc4

fstagni · 2021-06-23T13:48:18Z

The matcher was one of those service that couldn't be duplicated. If it's a tornado service, it's going to be massively duplicated :-)

Good point, but it's not because it's a tornado service that it's absolutely going to be massively duplicated, right? It's more of a wish. Anyway, I will check in future about this bullet. I took Matcher more as an example here, JobStateUpdate would probably be a better one.

Should the restart of tornado be done after every new service installation?

src/DIRAC/Core/Tornado/scripts/tornado_start_all.py

chaen · 2021-06-23T14:18:25Z

The matcher was one of those service that couldn't be duplicated. If it's a tornado service, it's going to be massively duplicated :-)

Good point, but it's not because it's a tornado service that it's absolutely going to be massively duplicated, right? It's more of a wish. Anyway, I will check in future about this bullet. I took Matcher more as an example here, JobStateUpdate would probably be a better one.

Well, every tornado server will run a matcher. So as many matcher as hosts.

Should the restart of tornado be done after every new service installation?

Maybe, just to be on the safe side

fstagni · 2021-07-05T15:50:03Z

Now the py2+https tests are passing.

I added a py3+https tests, but these are failing, specifically the DFC tests. Is there something obvious that should be changed?

chaen · 2021-07-06T07:40:55Z

I'll check for the python 3 stuff

chaen

I found where the problems are. I'll make a separate PR to enable all these tests for Py3. Can you remove the changes I pointed please ?

chaen · 2021-07-06T11:32:18Z

.github/workflows/integration.yml

@@ -37,7 +34,9 @@ jobs:
          - TEST_NAME: "Python 3 server"
            ARGS: SERVER_USE_PYTHON3=Yes
          - TEST_NAME: "Python 3 server and client"
-            ARGS: CLIENT_USE_PYTHON3=Yes SERVER_USE_PYTHON3=Yes MYSQL_VER=8.0
+            ARGS: CLIENT_USE_PYTHON3=Yes SERVER_USE_PYTHON3=Yes


please remove, I'll do in a separate PR

tests/CI/install_server.sh

chaen · 2021-07-06T11:35:20Z

src/DIRAC/Core/Tornado/scripts/tornado_start_all.py

@@ -14,6 +14,7 @@

 import os
 import sys
+


Please remove all the changes to that file, since it was addressed here #5223

fstagni · 2021-07-06T13:18:44Z

I found where the problems are. I'll make a separate PR to enable all these tests for Py3. Can you remove the changes I pointed please ?

I will, when rebasing this PR on top of yours.

chaen · 2021-08-03T13:49:26Z

tests/CI/install_server.sh

+    find "/home/dirac/LocalRepo/ALTERNATIVE_MODULES/DIRAC" -name 'Tornado*Handler.py' | grep -v Configuration | sed -e 's/Handler.py//g' -e 's/System//g'| awk -F '/' '{print $(NF-2), $NF}' > tornadoServices
+    cfgProdFile="${SERVERINSTALLDIR}"/diracos/etc/Production.cfg
+  else
+    find "${SERVERINSTALLDIR}"/DIRAC/ -name 'Tornado*Handler.py' | grep -v Configuration | sed -e 's/Handler.py//g' -e 's/System//g'| awk -F '/' '{print $(NF-2), $NF}' > tornadoServices


I don't think that this is necessary as tornadoservice is not used, and I believe the Production.cfg files are symlinked

where is the system_component (2 lines below) created? I have the impressione there's some merge or rebase issue.

tests/Jenkins/dirac_ci.sh

chaen · 2021-08-04T06:32:07Z

The PR is fine with me but you have conflicts

…ew agent (TaskkQueuesAgent)

StateUpdate

andresailer · 2021-08-10T11:36:33Z

@fstagni Should The TaskQueuesAgent be mentioned in the release notes and in https://github.com/DIRACGrid/DIRAC/wiki/DIRAC-v7r3 ?

fstagni · 2021-08-12T15:58:31Z

@fstagni Should The TaskQueuesAgent be mentioned in the release notes and in https://github.com/DIRACGrid/DIRAC/wiki/DIRAC-v7r3 ?

Thanks for the reminder. Done.

fstagni requested a review from atsareg as a code owner June 21, 2021 08:41

fstagni requested a review from chaen June 21, 2021 08:41

fstagni force-pushed the v7r3-fixes12 branch from 7c69d7d to e375257 Compare June 21, 2021 11:30

fstagni requested a review from andresailer as a code owner June 23, 2021 08:44

chrisburr reviewed Jun 23, 2021

View reviewed changes

src/DIRAC/Core/Tornado/scripts/tornado_start_all.py Show resolved Hide resolved

chrisburr mentioned this pull request Jun 23, 2021

DIRAC_USE_TORNADO_IOLOOP set after Refresher is imported in tornado_start_all #5223

Closed

fstagni marked this pull request as draft June 24, 2021 09:06

fstagni changed the title ~~[v7r3] tornado-based https Matcher handler~~ [v7r3] tornado-based https JobStateUpdate handler Jun 29, 2021

fstagni force-pushed the v7r3-fixes12 branch 6 times, most recently from d0f3e51 to 32b3287 Compare July 5, 2021 14:31

chaen requested changes Jul 6, 2021

View reviewed changes

fstagni linked an issue Jul 21, 2021 that may be closed by this pull request

Add some Tornado-based services #4984

Closed

3 tasks

fstagni force-pushed the v7r3-fixes12 branch from 32b3287 to af527e6 Compare July 29, 2021 13:17

fstagni marked this pull request as ready for review July 29, 2021 13:19

chaen reviewed Aug 3, 2021

View reviewed changes

tests/Jenkins/dirac_ci.sh Show resolved Hide resolved

fstagni force-pushed the v7r3-fixes12 branch from af527e6 to 66406bb Compare August 3, 2021 14:12

fstagni added 21 commits August 4, 2021 09:33

added TornadoMatcherHandler

7eb4b6f

moved Matcher thread for taskQueueDB.recalculateTQSharesForAll to a n…

3c77a37

…ew agent (TaskkQueuesAgent)

removed need for ThreadScheduler

9e52618

removed new thread pool test (it's the default already)

1b1b96c

moved TornadoMatcherHandler to a different module

ef9e6a4

added Tornado check

06baaf1

fix

d1b34d3

added TornadoMatcher to ConfigTemplate.cfg

f8a5aff

added basic imports

ee4b1e7

added parseCommandLine

1244866

update doc

954853e

restart tornado after every service installation

d849e5d

set 'DIRAC_USE_TORNADO_IOLOOP' before DIRAC imports

f01b3fe

removed TornadoMatcher handler

7cc0813

added TornadoJob

989d8e7

StateUpdate

removed unused option

beb8fde

fix

0226182

added TornadoJobMonitoring service

dc37a85

added TornadoJobManager service

7c3cbbc

fix

7bf1ed3

added py3 HTTPS tests

9c082e8

fstagni force-pushed the v7r3-fixes12 branch from 6bcf2cf to 9c082e8 Compare August 4, 2021 07:40

tabs

e556c52

fstagni merged commit a7cd23f into DIRACGrid:integration Aug 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[v7r3] tornado-based https JobStateUpdate handler #5214

[v7r3] tornado-based https JobStateUpdate handler #5214

fstagni commented Jun 21, 2021 •

edited

Loading

chaen commented Jun 23, 2021

chaen commented Jun 23, 2021

chaen commented Jun 23, 2021 •

edited

Loading

chaen commented Jun 23, 2021

fstagni commented Jun 23, 2021

chaen commented Jun 23, 2021

fstagni commented Jul 5, 2021

chaen commented Jul 6, 2021

chaen left a comment

chaen Jul 6, 2021

chaen Jul 6, 2021

fstagni commented Jul 6, 2021

chaen Aug 3, 2021

fstagni Aug 3, 2021

chaen commented Aug 4, 2021

andresailer commented Aug 10, 2021

fstagni commented Aug 12, 2021 •

edited

Loading

[v7r3] tornado-based https JobStateUpdate handler #5214

[v7r3] tornado-based https JobStateUpdate handler #5214

Conversation

fstagni commented Jun 21, 2021 • edited Loading

chaen commented Jun 23, 2021

chaen commented Jun 23, 2021

chaen commented Jun 23, 2021 • edited Loading

chaen commented Jun 23, 2021

fstagni commented Jun 23, 2021

chaen commented Jun 23, 2021

fstagni commented Jul 5, 2021

chaen commented Jul 6, 2021

chaen left a comment

Choose a reason for hiding this comment

chaen Jul 6, 2021

Choose a reason for hiding this comment

chaen Jul 6, 2021

Choose a reason for hiding this comment

fstagni commented Jul 6, 2021

chaen Aug 3, 2021

Choose a reason for hiding this comment

fstagni Aug 3, 2021

Choose a reason for hiding this comment

chaen commented Aug 4, 2021

andresailer commented Aug 10, 2021

fstagni commented Aug 12, 2021 • edited Loading

fstagni commented Jun 21, 2021 •

edited

Loading

chaen commented Jun 23, 2021 •

edited

Loading

fstagni commented Aug 12, 2021 •

edited

Loading