-
Notifications
You must be signed in to change notification settings - Fork 174
DIRAC v6r19
In case your local $DIRACROOT/etc/dirac.cfg file does not yet contain it, please add the following option:
DIRAC/Extensions = myExtensions
The same applies to ~/.dirac.cfg file sometime used for local client installations.
dirac-proxy-init will now create RFC proxies by default. use the -L/--legacy flag to create non-RFC proxies.
Improvements to the Resource Status System (RSS) are the main enhancements coming together with DIRAC v6r19. From version v6r19, RSS can actively manage Sites and Computing Elements in addition to Storage Elements.
The RSS status for Sites and ComputingElements can now be used in the pilot submission logic for choosing destination resources. These statuses are used if the RSS status management is enabled with the flag::
/Operations//ResourceStatus/Config/State=Active
In order to use RSS statuses for Sites and ComputingElements (and StorageElements), the latter must be synchronized with the CS information. This should be done with the following commands::
dirac-rss-sync --element=Site --defaultStatus=Banned --init
dirac-rss-sync --element=Resource --defaultStatus=Active --init
This will initialize the Sites in the RSS databases with the Banned default status unless the site is allowed in the old SiteMask logic (which could be found in the WMS JobDB). The ComputingElements are initialized all with the Active status because this status is not defined in the CS description.
The Site status can be altered as before by manual commands::
dirac-admin-allow-site LCG.CERN.cern "Comment"
dirac-admin-ban-site LCG.CERN.cern "Comment"
The Components are not anymore managed by RSS (in fact, they have never properly been). You may remove the corresponding tables in ResourceStatusDB::
drop table if exists
ComponentStatus;
drop table if exists
ComponentLog;
drop table if exists
ComponentHistory;
Modules ResourceStatusDB.py and ResourceManagementDB.py (that are used to interact with ResourceStatusDB and ResourceManagementDB mySQL DBs) have been re-written using sqlAlchemy. Backward compatibility with extensions could not be preserved, but updates are very easy.
The below policy needs to be added in the CS under Operations > [Defaults or Setup] > ResourceStatus > Policies::
PropagationPolicy
{
matchParams
{
element = Site
}
policyType = PropagationPolicy
}
This is needed in order to propagate Site status to all the dependent resources in case of the Site status change. Other policies can also be added (like the "AlwaysActivePolicy" and the "DowntimePolicy").
If there are no policies enabled for sites, then the SiteInspectorAgent, when installed, will force the status of the sites to "Unknown".
A new agent should be installed, via dirac-admin-sysadmin-cli::
install agent ResourceStatus SiteInspectorAgent
A new "SiteStatus" API has been defined in https://github.com/DIRACGrid/DIRAC/blob/rel-v6r19/ResourceStatusSystem/Client/SiteStatus.py
The "ResourceStatus" API has been modified: https://github.com/DIRACGrid/DIRAC/blob/rel-v6r19/ResourceStatusSystem/Client/ResourceStatus.py
Code in vanilla DIRAC has been updated to reflect the changes: direct calls to it from DIRAC extensions should be done. More in details: the getStorageElementStatus method has been replaced by a generic getElementStatus method, so previous calls to::
ResourceStatus().getStorageElementStatus( elementName, statusType, default )
should be replaced by::
ResourceStatus().getElementStatus( elementName, elementType, statusType, default )
Where elementType, for the case of StorageElements, is, of course, StorageElement
The python code that interacts with the RSS DBs (ResourceStatusDB, and ResourceManagementDB) have been re-written using sqlalchemy. Eventual extensions may also need to be updated.
For compatibility with MySQL 5.7, the FTSHistoryView needs update:
alter view FTSHistoryView as select `FTSJob`.`Status` AS `Status`,sum(`FTSJob`.`Files`) AS `Files`,`FTSJob`.`TargetSE` AS `TargetSE`,(sum(`FTSJob`.`Completeness`) / count(distinct `FTSJob`.`FTSJobID`)) AS `Completeness`,sum(`FTSJob`.`FailedSize`) AS `FailedSize`,sum(`FTSJob`.`Size`) AS `Size`,sum(`FTSJob`.`FailedFiles`) AS `FailedFiles`,count(distinct `FTSJob`.`FTSJobID`) AS `FTSJobs`,`FTSJob`.`SourceSE` AS `SourceSE` from `FTSJob` where (`FTSJob`.`LastUpdate` > (utc_timestamp() - interval 3600 second)) group by `FTSJob`.`SourceSE`,`FTSJob`.`TargetSE`,`FTSJob`.`Status` ;
The new NetworkAgent agent (http://dirac.readthedocs.io/en/integration/CodeDocumentation/AccountingSystem/Agent/NetworkAgent.html) is an optional Accounting agent that is used to consume perfSONAR (http://www.perfsonar.net/) network metrics, and to display them via a new "network" accounting plotting type. In the current implementation network metrics are received via a message queue.
In order for this agent to work properly, a message queue has to be specified, in CS, as per instructions in http://dirac.readthedocs.io/en/integration/AdministratorGuide/DIRACSites/MessageQueues/index.html For this purpose, in the agent configuration in /DIRAC/Systems section of the CS, the option ::
MessageQueueURI
should be specified, and its content define the queue host together with the topics you are interested to use::
netmon-mb.cern.ch::Topic::perfsonar.summary.packet-loss-rate netmon-mb.cern.ch::Topic::perfsonar.summary.histogram-owdelay
The logging tool gLogger is now based on the standard logging Python library and includes some changes that you can consult in the official DIRAC documentation. The major changes that you have to take care of first are the following:
Before, you had to define your Backends in the LogBackends option and to specify their parameters in the BackendsOptions section. Now, the Backends configuration becomes more readable thanks to a new section named BackendsConfig, and the previous one has to be removed, even if it is still compatible with the new logging tool.
Thus, your current Backends configuration...
LogBackends = <backend1>, <backend2>, <backend3>
BackendOptions
{
<param backend2> = <value1>
<param backend3> = <value2>
}
... should become:
LogBackends = <backend1>, <backend2>, <backend3>
LogBackendsConfig
{
<backend2>
{
<param backend2> = <value1>
}
<backend3>
{
<param backend3> = <value2>
}
}
Note that the ProtocolName option in the configuration is not supported anymore. It is now replaced by the PluginName option.
What used to be called AccessProtocol.1 and so forth can now be named after a Plugin (e.g. GFAL2_SRM2) and the PluginName option becomes optional. The PluginName option still takes precedence
The StorageElement definitions that are used as bases for other StorageElements with the Base-SE features have now a dedicated section: StorageElementBases (see http://dirac.readthedocs.io/en/latest/AdministratorGuide/Systems/DataManagement/index.html#id2)
As of v6r22, no Base-SE will be looked up anymore in the StorageElement definition
The machinery for the MQ systems support has been fully certified, and it's now out of the "Technology preview"