-
Notifications
You must be signed in to change notification settings - Fork 4
/
release.notes
2449 lines (1606 loc) · 68.9 KB
/
release.notes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
[v29r0p4]
Release date: 2019-03-19
*ILCTransformationSystem
FIX: (!257) `dirac-clic-make-productions`: creation Gen transformations now is working again
FIX: (!257) `dirac-clic-make-productions`: Gen transformations can use the FE. syntax now via the Whizard2 ``applicationOptions``
CHANGE: (!259) `dirac-clic-make-productions`: new Parameter ``MoveGroupSize`` to set the number of files for each moving request, defaults to 10 instead of 1 as used before
CHANGE: (!261) The :class:`~ILCDIRAC.ILCTransformationSystem.Agent.DataRecoveryAgent.DataRecoveryAgent` now does service calls in bulk to speed up processing by factor at least 4
CHANGE: (!263) ILCDIRAC-873 :mod:`~ILCDIRAC.Interfaces.API.NewInterface.ProductionJob`: refactor the registration of metadata, ``_registerMetadata`` now returns S_ERROR if something went wrong. Adding verbose output.
FIX: (!263) `dirac-clic-make-productions`: Fix for inputDataQuery for Split Productions
CHANGE: (!266) `dirac-clic-make-productions`: Disable keeping of REC files by default. It can be enabled by setting ``KeepRecFile=True`` in the Marlin application options of the transformation creation file
*Interfaces
NEW: (!264) :mod:`~ILCDIRAC.Interfaces.API.NewInterface.Applications.Marlin`: new member flag ``keepRecFile``, if set to ``False`` ProductionJobs are not uploading REC files
NEW: (!265) New function :func:`~ILCDIRAC.Interfaces.API.NewInterface.Applications.Whizard2.Whizard2.setIntegratedProcess` to make whizard2 use an already integrated process. The user is responsible for providing a correct sindarin file
*Miscellaneous
FIX: (!260) use print function instead of print statement
FIX: (!260) remove the usage of the long suffix to distinguish between long and int
FIX: (!260) convert octal literals to new syntax
NEW: (!260) Add pylint test to check python 3 compatibility
NEW: (!258) Enable codecov in travis
[v29r0p3]
Release date: 2019-02-12
*ILCTransformationSystem
CHANGE: (!243) The :class:`~ILCDIRAC.ILCTransformationSystem.Agent.DataRecoveryAgent.DataRecoveryAgent` no longer uses a shifterProxy
NEW: (!245) `dirac-clic-make-productions`: new options for configuration. add ForEach type configuration for application parameters, see :ref:`clicProdMan`.
NEW: (!245) `dirac-clic-make-productions`: Add ``MoveStatus`` options to set the status of the replication transformations after creation: Active or Stopped. Needs DIRAC v6r20p19
NEW: (!245) `dirac-clic-make-productions`: additionalName can be used as ``ApplicationParameter`` to give more sensible names for different transformations types when created in a chain
FIX: (!252) `dirac-clic-make-productions`: When starting a chain with a simulation productions, the number of events was not set, now always set NumberOfEvents into the meta dictionary
FIX: (!254) `dirac-clic-make-productions`: Fix exception when creation non overlay Reconstruction productions
*Interfaces
CHANGE: (!244) :mod:`~ILCDIRAC.Interfaces.API.NewInterface.ProductionJob`: use TransformationClient instead of RPCClient
NEW: (!245) :class:`~ILCDIRAC.Interfaces.API.NewInterface.ProductionJob.ProductionJob` new member parameter ```maxFCFoldersToCheck`` to limit the number of FileCatalog folders that are checked for metadata
CHANGE: (!251) The JDL created for Jobs no longer contains *Site* if not explicitly set in the job creation
*FrameworkSystem
CHANGE: (!244) :mod:`~ILCDIRAC.FrameworkSystem.Agent.MonitorAgents`: Use Client instead of RPCClient
*WorkloadManagementSystem
CHANGE: (!244) :mod:`~ILCDIRAC.WorkloadManagementSystem.Agent.JobResetAgent`: use Client instead of RPCClient
*Workflow
CHANGE: (!244) :mod:`~ILCDIRAC.Workflow.Modules.OverlayInput`: use OverlaySystemClient instead of RPCClient
CHANGE: (!244) :mod:`~ILCDIRAC.WorkloadManagementSystem.Client.DownloadInputData`: use Client instead of RPCClient
CHANGE: (!244) :mod:`~ILCDIRAC.Workflow.Modules.GetSRMFile`: use JObMonitoringClient instead of RPCClient
[v29r0p2]
Release date: 2018-09-19
*Core
CHANGE: (!241) The names of the loggers have changed for all Core.Utilities, this makes it easier to see where messages are coming from, and will allow better silencing of individual modules in the future
NEW: (!241) New function :func:`ILCDIRAC.Core.Utilities.Utilities.listify` to turn a comma separated string into a list
NEW: Based on Dirac v6r20p13
*WorkloadManagementSystem
CHANGE: (!232) :class:`~ILCDIRAC.WorkloadManagementSystem.Agent.JobResetAgent.JobResetAgent`: Treat Failed RegisterFile requests. Existing files are unregistered so that the RegisterFile request can succeed, solves ILCDIRAC-860
*Workflow
CHANGE: (!238) The number of events to process is no longer always taken from the FileCatalog when defined, only when it is not set explicitly, this change is necessary when trying to split production files, which have the *NumberOfEvents* metadata defined
CHANGE: (!238) The name of the loggers has changed, the output will be more hierarchical
*Docs
CHANGE: (!238) The overview pages for packages has changed to *index*
*Interfaces
NEW: (!238) New automatic job splitting function to split one file across multiple jobs :func:`~ILCDIRAC.Interfaces.Utilities.SplitMixin.SplitMixin.setSplitFilesAcrossJobs`
NEW: (!238) New function to set the job indices to use for automatic job splitting: :func:`~ILCDIRAC.Interfaces.Utilities.SplitMixin.SplitMixin.setSplitJobIndexList`
CHANGE: (!238) The name of the loggers has changed, the output will be more hierarchical
CHANGE: (!238) ``runLocal`` is used from DIRAC interfaces again, as the patches to run locally with LFNs in input sandboxes have been ported to upstream
FIX: (!239) :mod:`~ILCDIRAC.Interfaces.API.NewInterface.Applications.Whizard2`: fix to be able to parse and replace non-integer sqrts values
*FrameworkSystem
NEW: (!235) :class:`~ILCDIRAC.FrameworkSystem.Agent.MonitorAgents.MonitorAgents`: add component control: ensure services/agents/executors are in the correct run status.
NEW: (!235) :class:`~ILCDIRAC.FrameworkSystem.Agent.MonitorAgents.MonitorAgents`: add component control: ensure the URL for running services is set
*ILCTransformationSystem
CHANGE: (!228) `dirac-clic-make-productions`: better information about moving transformation in dryRun mode, moving transformations will be added to the same group as other transformations
CHANGE: (!228) `dirac-ilc-replication-transformation`, `dirac-ilc-moving-transformation`: transformation group is now the group the prodID belongs to as well, unless set with -R/--groupName
NEW: (!230) `dirac-clic-make-productions`: new parameter overlayEventType, added configuration for 91.2 and 365 fcc overlay information
[v29r0p1]
Release date: 2018-08-03
*Core
NEW: Based on Dirac v6r20p7
*WorkloadManagementSystem
CHANGE: (!225) :mod:`~ILCDIRAC.WorkloadManagementSystem.Agent.JobResetAgent`: only reset requests that are not waiting or scheduled
FIX: (!225) :mod:`~ILCDIRAC.WorkloadManagementSystem.Agent.JobResetAgent`: Take into account that some requests have multiple files
*Interfaces
FIX: (!227) :mod:`~ILCDIRAC.Interfaces.Utilities.DDInterfaceMixin`: ensure errors for :func:`~ILCDIRAC.Interfaces.Utilities.DDInterfaceMixin.DDInterfaceMixin.setDetectorModel` cannot be ignored.
*Workflow
FIX: (!222) :mod:`~ILCDIRAC.Workflow.Modules.ModuleBase`: Fix cleanup of request operations in case of failure. RemovalReplica requests are also removed, which in the past lead to un-fulfillable requests
FIX: (!222) :mod:`~ILCDIRAC.Workflow.Modules.ModuleBase`: only calculate checksum once.
CHANGE: (!222) :mod:`~ILCDIRAC.Workflow.Modules.ModuleBase`: change the name of the log output to **__name__**
CHANGE: (!223) :mod:`~ILCDIRAC.Workflow.Modules.OverlayInput`: only wait 3 minutes between copying if the previous copy attempt failed, fixes ILCDIRAC-850
*FrameworkSystem
FIX: (!224) :mod:`~ILCDIRAC.FrameworkSystem.Agent.MonitorAgents`: prevent exception for missing LogAge for services
CHANGE: (!224) :mod:`~ILCDIRAC.FrameworkSystem.Agent.MonitorAgents`: add hostname to the email subject, include service url in error messages
*ILCTransformationSystem
FIX: (!226) :class:`~ILCDIRAC.ILCTransformationSystem.Agent.FileStatusTransformationAgent`: Do not create reset request, set file unused to force creation of a new request.
[v29r0]
Release date: 2018-07-18
*Core
NEW: Based on Dirac v6r20p5
[v28r0p8]
Release date: 2018-06-12
*ILCTransformationSystem
FIX: (!212) `dirac-ilc-get-info`: fixes and modifications for new whizard2 production
FIX: (!216) TransformationPlugin: BroadcastProcessed: only print about failure for ``getDescendents`` if there was a failure
*Interfaces
NEW: (!214) New function :func:`~ILCDIRAC.Interfaces.Utilities.SplitMixin.SplitMixin.setSplittingStartIndex` to set the starting index for the naming of output files.
CHANGE: (!215) :class:`~ILCDIRAC.Interfaces.API.NewInterface.ProductionJob.ProductionJob` classes now execute the :class:`~ILCDIRAC.Workflow.Modules.ReportErrors.ReportErrors` Module at the very end, solves ILCDIRAC-760. Only works for productions created with this or later releases.
FIX: (!212) :class:`~ILCDIRAC.Interfaces.API.NewInterface.ProductionJob.ProductionJob`: Fix for meta energy _without_ decimal digits
NEW: (!218) Parametric (Splitting) Jobs can now be tested locally. Only the first entry of the sequence will be run however! Fixes ILCDIRAC-818
FIX: (!218) When testing jobs locally, the LFNs in the input sandbox will be treated without exceptions.
*FrameworkSystem
NEW: (!213) :class:`~ILCDIRAC.FrameworkSystem.Agent.MonitorAgents.MonitorAgents` now also monitors services by pinging them
*Workflow
NEW: (!215) New module :class:`~ILCDIRAC.Workflow.Modules.ReportErrors.ReportErrors` to print out application errors at the end of ProductionJob workflow executions
[v28r0p7]
Release date: 2018-06-07
*ILCTransformationSystem
NEW: (!209) New function :func:`~ILCDIRAC.Core.Utilities.ProductionData.getExperimentFromPath` which determines the experiment based on the path of the file. Replaces hardcoded values with flexible reading from the configuration system. See section **Operations/Production/ExperimentBasePaths**
NEW: (!210) `dirac-clic-make-productions` sets some parameters based on the proxy, e.g., fcc_prod or ilc_prod groups
CHANGE: (!210) `dirac-clic-make-productions` non-integer energies can be used for productions, e.g. 91.2 GeV
CHANGE: (!210) `dirac-clic-make-productions` more easy settings for multiple production chains, setting one or more entries to a list should in most cases be possible, other values will be taken for all chains.
*Interfaces
NEW: (!210) The :class:`~ILCDIRAC.Interfaces.API.NewInterface.Applications.Whizard2.Whizard2` interface now allows the setting of the energy in the Sindarin file via the API, if parsing does not work an error is raised
CHANGE: (!210) :mod:`~ILCDIRAC.Interfaces.API.NewInterface.ProductionJob`: instead of checking the proxy group the proxy properties are checked for ProductionManagement
*Core
CHANGE: (!210) `checkOrGetGroupProxy` now also takes a list as argument and returns :func:`~DIRAC:DIRAC.Core.Utilities.ReturnValues.S_OK` with the group found
[v28r0p6]
Release date: 2018-06-06
*Interfaces
NEW: (!207) Whizard2: add possibility to simulate multiple processes with the function :func:`~ILCDIRAC.Interfaces.API.NewInterface.Applications.Whizard2.Whizard2.setProcessVariables`
*FrameworkSystem
FIX: (!199) :mod:`~ILCDIRAC.FrameworkSystem.Agent.MonitorAgents` fix exception after restarting of agent. Wring signature for callback function
NEW: (!201) :mod:`~ILCDIRAC.FrameworkSystem.Agent.MonitorAgents` now also monitors executors, restarts them if **RestartExecutors** is True, solves ILCDIRAC-824
*Docs
FIX: (!204) Fixed wrong imports in splitting example
FIX: (!205) Fixed syntax errors in splitting example
NEW: (!205) added parametric job name to splitting example
*ILCTransformationSystem
FIX: (!198) Fix a problem in `dirac-clic-make-productions` when starting chains from generator files
NEW: (!198) In `dirac-clic-make-productions`, allow creating chains only with prodIDs and obtain the EvtType, Energy and Number of Events from the ProductionInfo or FileCatalog
FIX: (!202) :class:`~ILCDIRAC.ILCTransformationSystem.Agent.FileStatusTransformationAgent`: Protect against exception when **TaskID==None**, solves ILCDIRAC-831
FIX: (!206) `dirac-clic-make-productions`: Fix a bug when not creating RecOver productions, the wrong check was done for the creation of the moving transformation
NEW: (!206) `dirac-clic-make-productions`: When creating chains, the splitting step will be skipped if the number of events per job and the number of events in the generator files is the same. This allows the creation of multiple chains some of which contain splitting steps and some not
[v28r0p5]
*WorkloadManagementSystem
FIX: (!194) fixing :mod:`~ILCDIRAC.WorkloadManagementSystem.Agent.JobResetAgent`
*ILCTransformationSystem
FIX: (!195) Fixed bugs that prevented the use of `dirac-ilc-moving-transformation` and `dirac-ilc-replication-transformation`
[v28r0p4]
*WorkloadManagementSystem
NEW: (!187) :mod:`~ILCDIRAC.WorkloadManagementSystem.Agent.JobResetAgent`: automatically resets jobs that somehow got stuck in staging or completed status
*FrameworkSystem
NEW: (!191) new :mod:`~ILCDIRAC.FrameworkSystem.Agent.MonitorAgents` to restart stuck agents
*Workflow
CHANGE: (!182) :mod:`~ILCDIRAC.Workflow.Modules.UploadLogFile` and :mod:`~ILCDIRAC.Workflow.Modules.UploadOutputData`: Change logic to determine the experiment. ILD is using new paths: /ilc/prod/ilc/mc-opt
and /ilc/prod/ilc/ild (Testing only)
*ILCTransformationSystem
NEW: (!180) Adding new :class:`~ILCDIRAC.ILCTransformationSystem.Agent.FileStatusTransformationAgent`
NEW: (!184) :class:`~ILCDIRAC.ILCTransformationSystem.Agent.FileStatusTransformationAgent`'s errors are sent in email notification
FIX: (!186) `dirac-clic-make-productions`: reset extraCommandLine options after each (reco) production so multiple overlay options do not appear in the string.
NEW: (!186) `dirac-clic-make-productions`: add option to set arbitrary options for Applications (Marlin, DDSim, Overlay at the moment). All options that can be set with setter functions can be set via an extra section, e.g., "[Marlin]". Can be used to set a different SteeringFile name. Sections are automatically created with **-p**
NEW: (!188) `dirac-ilc-replication-transformation`, `dirac-ilc-moving-transformation` have additional parameter --groupSize to set the number of files per task
NEW: (!188) `dirac-ilc-replication-transformation` allows multiple ProdIDs to be passed
NEW: (!188) New configuration section in Operations/Defaults/Production/TransformationDatatypes maps Datatypes with TransformationTypes, e.g. REC -> MCReconstruction etc. Used in `dirac-ilc-replication-transformation` and `dirac-ilc-moving-transformation` to check consistency of ProdID and datatype
NEW: (!189) Add Whizard2 to the CLIC production script (`dirac-clic-make-productions`)
[v28r0p3]
*Workflow
CHANGE: (!182) UploadLogFile and UploadOutputData: Change logic to determine the experiment. ILD is using new paths: /ilc/prod/ilc/mc-opt
and /ilc/prod/ilc/ild (Testing only)
[v28r0p2]
*Interfaces
CHANGE: (!179) At job submission a check is done that ensure LFNs in the input sandbox have more than one replica, or a replica at a decent storage element (ILCDIRAC-805)
*Workflow
FIX: (!177) MarlinAnalysis: When obtaining the MARLIN_DLL from the init_script, suppress output from sourcing that script
[v28r0p1]
*Core
NEW: Based on Dirac v6r19p4
NEW: (!175) New utility for Configuration: :func:`~ILCDIRAC.Core.Utilities.Configuration.getOptionValue` to get option from any level of given path in Configuration
*Workflow
CHANGE: (!175) MaxNbFilesToGet can now be set at any level of the Overlay hierarchy
[v28r0]
*Core
NEW: Based on Dirac v6r19p3
[v27r0p2]
*ILCTransformationSystem
CHANGE: (!170) ILDProductionJobOpt2017: The detector model in output path, basename and meta value for DetectorModel meta key can now be defined by a function ``setDetectorForBasename``. If the variable ``detector_basename`` is defined by the function, it is used for the detector name. Otherwise, the detector name is obtained from the detector name found in the input file.
CHANGE: (!171) Allow `dirac-ilc-moving-transformation` to create moving transformation for Split productions
CHANGE: (!167) FilenameEncoder: Detector model in the rec and dst file names could be different from the input sim files for ILD detector optimization production. In previous version, detector model in sim file name was used for the rec and dst file names. It is modified to use a detector model given by the production, namely given by the second argument of resolvePathsAndNames.getProdFilenameFromInput
FIX: (!167) FilenameEncoder: Avoid "%" in the file name when constructing rec and dst files from sim file produced during DBD era.
[v27r0p1]
*WorkloadManagementSystem
FIX: (!166) Adapt DownloadInputData for GFAL2 plugins, ILCDIRAC-796
[v27r0]
*Core
NEW: Based on DIRAC v6r17p33
[v26r0p14]
*Interfaces
NEW: (!150) :class:`~ILCDIRAC.Interfaces.API.NewInterface.Applications.Whizard2.Whizard2` interface (ILCDIRAC-467)
[v26r0p13]
*Interface
NEW: (!153) `ILCDIRAC.Interfaces.Utilities.SplitMixin.SplitMixin.setSplitDoNotAlterOutputFilename`: function to disable the injecting of the JobIndex into the filename
*ILCTransformationSystem
FIX: (!155) DataRecoveryAgent: fix problem to get canceled request status. Fixes problem were the request cannot be successful, and thus some jobs were not dealt with properly.
FIX: (!156) `dirac-clic-make-productions` use ProdID as string, fixes problem if ProdID is 0
CHANGE: (!156) `dirac-clic-make-productions` *_productionname* function no longer has *prodname* argument
*Interfaces
CHANGE: (!156) Updated docstring for :func:`~ILCDIRAC.Interfaces.Utilities.DDInterfaceMixin.DDInterfaceMixin.setDetectorModel` to include examples of the different options
*Workflow
NEW: (!153) if some kind of jobSplitting is done the OutputData LFNs are automatically injected with the JobIndex in the splitting sequence. See :mod:`ILCDIRAC.Core.Utilities.Splitting`
[v26r0p12]
*Core
CHANGE: (!141) **GetOverlayFiles** module renamed to :mod:`~ILCDIRAC.Core.Utilities.OverlayFiles` and added some helper functions to it; reduce code duplication in the OverlayInput interface and workflow module
CHANGE: (!151) `dirac-ilc-add-cvmfs-software` add dryrun option, add possibility to add any Config package
*Workflow
NEW: (!153) if some kind of jobSplitting is done the OutputData LFNs are automatically injected with the JobIndex in the splitting sequence. See :mod:`ILCDIRAC.Core.Utilities.Splitting`
*ILCTransformationSystem
FIX: (!140) ``MovingTransformation``: fix the *targetSE* argument type; fix if *targetSE* is a string
FIX: (!140) `dirac-clic-make-productions`: *Move* parameter was not parsed before, now moving transformations are also created if it is True
FIX: (!144) `dirac-clic-make-productions` make more parameters optional
CHANGE: (!144) `dirac-clic-make-productions` change parameter name *NumberOfEventsInBaseFiles* to *eventsInSplitFiles*
CHANGE: (!144) `dirac-clic-make-productions` change parameter *overlayEnergy* to *overlayEvents*
CHANGE: (!144) `dirac-clic-make-productions` Produciton Group is now taking only from the config file, no additional string is added. Use string replacements for automatic group names
FIX: (!152) `dirac-clic-make-productions`: allow moving of DST files
NEW: (!152) `dirac-clic-make-productions`: add cliReco option to set extraCLIArguments for Marlin
*Docs
CHANGE: (!140) move production manager notes from the twiki to the ilcdirac docs
NEW: (!140) CLIC Production manager guide with some notes for CLIC productions
CHANGE: (!142) restructuring of the documentation; additional example script
NEW: (!147) Add JIRA issue collector to the doc pages.
*Interface
NEW: (!153) :func:`~ILCDIRAC.Interfaces.API.NewInterface.UserJob.UserJob.setSplitDoNotAlterOutputFilename`: function to disable the injecting of the JobIndex into the filename
NEW: (!146) :class:`~ILCDIRAC.Interfaces.API.NewInterface.UserJob.UserJob`: new functions to ease the automatic splitting of jobs :func:`~ILCDIRAC.Interfaces.API.NewInterface.UserJob.UserJob.setSplitInputData`, :func:`~ILCDIRAC.Interfaces.API.NewInterface.UserJob.UserJob.setSplitEvents`
[v26r0p11]
*Interface
NEW: (!136) new attribute *processorName* for :class:`~ILCDIRAC.Interfaces.API.NewInterface.Applications.OverlayInput.OverlayInput`. Set the name of the Marlin processor the input files belong to. This enables multiple use of the OverlayInput application to download files for different Marlin processors.
CHANGE: (!136) :class:`~ILCDIRAC.Interfaces.API.NewInterface.ProductionJob.ProductionJob` change capitalisation of the *version* argument of :func:`~ILCDIRAC.Interfaces.API.NewInterface.ProductionJob.ProductionJob.setConfig`
*ILCTransformationSystem
NEW: (!128) `dirac-clic-make-productions` script to create productions for the new ddsim/marlin software chain
NEW: (!124) `dirac-ilc-get-prod-log` new options to get all files in a folder (-A) and to automatically start downloading without confirmation (-N)
NEW: (!126) `dirac-ilc-moving-transformation`: new option (--allFor) to set a prodID and have the three corresponding moving transformations: prodID/GEN, prodID+1/SIM and prodID+2/REC created. Added consistency check for Datatype
*Interfaces
NEW: (!125) :class:`~ILCDIRAC.Interfaces.API.NewInterface.Applications.StdHepSplit.StdHepSplit`: new property to set maxRead flag for the hepsplit application
*Workflow
CHANGE: (!135) The output parser for :class:`~ILCDIRAC.Workflow.Modules.MarlinAnalysis.MarlinAnalysis` will now pass lines containing 'EventNumber' or 'StatusMonitor' to the standard output, in addition to 'event' and 'ProgressHandler'
CHANGE: (!136) :class:`~ILCDIRAC.Workflow.Modules.OverlayInput.OverlayInput` fix the path for /eos/experiment/clicdp to get files directly from EOS
CHANGE: (!136) :class:`~ILCDIRAC.Workflow.Modules.MarlinAnalysis` can now handle setting of parameters for Overlay processors defined in a processor group. Implemented in the new module `Core.Utilities.MarlinXML`
NEW: (!125) :class:`~ILCDIRAC.Workflow.Modules.StdHepSplit.StdHepSplit`: set "--maxread" if maxRead is set for the application, no changes if maxRead is 0 for backward compatibility
CHANGE: (!127) Marlin the DST/REC files for processors called Output_DST/Output_REC files in production environments are now handled as well. For Users all processors of Type LCIOOutputProcessor should be used to change the output file name, not just the one called 'MyLCIOOutputProcessor'
*Docs
NEW: (!130) import documentation from ILCDIRAC-Ops repo, always compile docs
NEW: (!137) *docs/GetReleaseNotes.py* script to get the release notes from the merged merge requests since given date
[v26r0p10]
* Interfaces:
CHANGE: DDSim: use *detectorModel* as the *detectorType*, needed in CLIC productions
NEW ProctionJob: add :func:`~ILCDIRAC.Interfaces.API.NewInterface.UserJob.UserJob.setCLICConfig` function to add ClicConfigPackage to productions
* Core:
CHANGE: :mod:`~ILCDIRAC.Core.Utilities.InstalledFiles`: allow using arbitrary ConfigPackage to provide steering files. For now ILDConfig or ClicConfig defined in ConfigurationSystem
* Workflow:
NEW: :class:`~ILCDIRAC.Workflow.Modules.ModuleBase.ModuleBase`: General :func:`~ILCDIRAC.Workflow.Modules.ModuleBase.ModuleBase.treatConfigPackage` function to replace treatILDConfigPackage backward compatible
[v26r0p9]
* Workflow:
NEW: :class:`~ILCDIRAC.Workflow.Modules.SLICAnalysis`: allow user to preload libraries for SLIC. Add a tarball with folder 'preload' to the Sandbox. Libraries (files containing '.so' in that folder will be added to the LD_PRELOAD environment variable.
[v26r0p8]
* Core:
NEW: Based on v6r15p27
NEW: ILD FilenameEncoder now also hadles files for Split transformations
* ILCTransformationSystem
FIX: DataRecoveryAgent: No longer send messages if nothing changed.
FIX: DataRecoveryAgent: Deal with MCSimulation/MCReconstruction jobs where the inputFile does not exist
* Interfaces
FIX: Changes to ILDProductionChainOpt2017 and DBD: Define output SEs, limit number of jobs for all productions
[v26r0p7]
* Workflow
NEW: DDSim: ddsim now allows a user to add their own plugins.
A tarball with the content lib/libmydd4hepplugin.so, lib/libmydd4hepplugin.components can be added to the input sandbox
NEW: Marlin: Marlin also can use the dd4hep plugins provided by a users
FIX: detector model zipfiles/tarballs for marlin were not working before
[v26r0p6]
* Core
NEW: FilenameEncoder by Akiya Miyamoto
CHANGE: OutputDataPolicy: "\*_ILD" production types use FilenameEncoder
* Interfaces:
CHANGE: Rename ILDProductionJob to ILDProductionJobDBD for DBD productions
CHANGE: Rename ILDProductionChain to ILDProductionChainDBD
NEW: ILDProductionJobOpt2017: Production job class using the FilenameEncoder
NEW: ILDProductionChainOpt2017: using ILDProductionJobOpt2017
* ILCTransformationSystem:
FIX: DataRecoveryAgent: Fixed a bug that caused files to be reset too often
* Workflow:
CHANGE: DDSim Workflow module is now picking up ProductionOutputdata from WorkflowCommons instead of re-creating the filename
[v26r0p5]
* Core:
NEW: Based on DIRAC v6r15p25
*Interfaces:
NEW: More tests for interface modules
[v26r0p4]
* Core:
NEW: Based on DIRAC v6r15p23
* Interfaces
CHANGE: The Marlin application no longer has a default GearFile because having a gear file causes problems when dd4hep geometry is used instead.
* DataManagement:
CHANGE: (from DIRAC): dirac-dms-find-lfns command signature changed, now the PATH argument has to be used to define the basepath for the search
* ILCTransformation
FIX: dirac-ilc-production-summary: fix bug causing a crash when whizard details did not exist
[v26r0p3]
* Core:
NEW: Based on DIRAC v6r15p17
FIX: ILCDIRAC-718 pilots at KEK not matching jobs
FIX: ILCDIRAC-713 dirac-dms-find-lfns produced output files not readable by other dirac commands
NEW: Tests also running on Centos7 now
* ILCTransformationSystem:
NEW: plugin BroadcastProcessed to create transformations to move only files with descendents
NEW: dirac-ilc-moving-transformation creates BroadcastProcessed transformations for GEN/SIM files
* Workflow:
FIX: Hopefully clearer message when packages are not found directly on CVMFS, ILCDIRAC-708
[v26r0p2]
* Core:
NEW: More Unite tests
NEW: Based on DIRAC v6r15p12 (-alpha?) Fixe for ILCDIRAC-703
FIX: `dirac-ilc-list-users` needed to be adapted for interface change in :mod:`DIRAC.Core.Security.VOMSService`, ILCDIRAC-702
* ILCTransformationSystem:
FIX: DataRecoveryAgent using DiraAPI getJobJDL function to get dictionary of JDLs, ILCDIRAC-705
* Interfaces:
FIX: Fix exception in internal :class:`DIRAC.Core.Base.API.API` ``._reportError`` function when function argument is 'name'
[v26r0p1]
* Core:
NEW: Based on DIRAC v6r15p11 (-alpha) to solve a problem with accessing files on RAL
[v26r0]
* Core:
NEW: Based on DIRAC v6r15p10
NEW: Many unittests for Utilities and other modules
CHANGE: Removed obsolete modules
* Workflow:
FIX: DDSim randomSeed will be set to the jobID by default
* Interfaces:
FIX: ILDProductionJob code cleaned and some bugs fixed, ILCDIRAC-641
FIX: Some more errors from application configurations will now be reported before submission
* Utilities:
FIX: Clean code in CombinedSoftwareInstallation. Unnecessary call to os.path.join.
FIX: PrepareLibs refactored
FIX: PrepareOption files refactored for better maintanability
* Resources:
CHANGE: Go back to XROOTStorage from DIRAC
* ILCTransformationSystem:
NEW: Add script to create "moving" transformations. Moving files from one SE to another (replicate and the remove replica)
[v25r0p10]
* Core:
NEW: based on DIRAC v6r14p37
FIX: Add CVMFS Software script now adds DDSimDetectorModels
[v25r0p9]
* ILCTransformationSystem:
FIX: ILCDIRAC-665: Also handle cases if a file gets stuck in 'Processed' status
* Overlay:
NEW: Set the NSKipEventsRandom parameter for BGOverlay processor. Calculates number of Events based on NBEvts Configuration Parameter
NEW: Allow defining different parameter for the same background type by using EventType parameter to set the correct EventType
* Interfaces:
NEW: When applications check for existance of steering files packages on CVMFS are considered as well
* Workflow:
NEW: Move back to CPUNormalisation from DIRAC, fix ILCDIRAC-661 implemented and released in DIRAC
[v25r0p8]
* Core:
NEW: Based on DIRAC v6r14p35
* Workflow:
FIX: :mod:`~ILCDIRAC.Interfaces.API.NewInterface.Applications.DDSim`: Fix for running ddsim when init script is needed. The LD_LIBRARY_PATH was not put together correctly and on some sites the local libstdc++.so was picked up.
[v25r0p7]
* Workflow:
FIX: :mod:`~ILCDIRAC.Workflow.Modules.OverlayInput` for KEK was not getting the correct path to copy files.
FIX: :mod:`~ILCDIRAC.Workflow.Modules.DDSimAnalysis`: Enable use of tarballs with non-system compiler, ILCDIRAC-671
* Interface:
NEW: :mod:`~ILCDIRAC.Interfaces.API.NewInterface.Applications.Marlin` has the new function :func:`~ILCDIRAC.Interfaces.Utilities.DDInterfaceMixin.DDInterfaceMixin.setDetectorModel` that works the same way as the :mod:`~ILCDIRAC.Interfaces.API.NewInterface.Applications.DDSim` function and can be used to set the dd4hep geometry based model. (ILCDIRAC-659)
[v25r0p6]
* Workflow:
FIX: :mod:`~ILCDIRAC.Workflow.Modules.MokkaAnalysis` outputfile event parameters were not properly set for Polarisation
CHANGE: :mod:`~ILCDIRAC.Workflow.Modules.OverlayInput` files are taken from cern eos instead of castor, when running at cern (ILCDIRAC-636)
* Interfaces:
FIX: Files from the ParametricInputData should now be handled correctly
[v25r0p5]
* WorkloadManagement:
FIX: Copy ``getCPUNormalization`` from DIRAC and reduce memory footprint ILCDIRAC-661
[v25r0p4]
* WorkloadManagement:
FIX: Copy :mod:`~ILCDIRAC.WorkloadManagementSystem.Client.DownloadInputData` from DIRAC and fix ILCDIRAC-655
[v25r0p3]
* Core:
NEW: Based on dirac v6r14p26
* ILCTransformationSystem:
FIX: :mod:`~ILCDIRAC.ILCTransformationSystem.Agent.DataRecoveryAgent` now properly sets MaxReset for files with a large ErrorCount
* Workflow:
FIX: :mod:`~ILCDIRAC.Workflow.Modules.DDSimAnalysis` Add DD4hepInstall environment variable to script for TarBallBased installations. Fixes problem for running these jobs locally.
[v25r0p2]
* Core:
NEW: Based on dirac v6r14p24
* Workflow:
FIX: :mod:`~ILCDIRAC.Workflow.Modules.DDSimAnalysis`: correctly handle the tarball
[v25r0p1]
* Interfaces:
FIX: Correctly handle :mod:`~ILCDIRAC.Interfaces.API.NewInterface.Applications.DDSim` detector models
[v25r0p0]
*Core
NEW: These release notes are the summary of every pre-release after the last v23r1 release
NEW: Based on dirac v6r14p18, many changes in Dirac
*Interfaces
NEW: Added :mod:`~ILCDIRAC.Interfaces.API.NewInterface.Applications.DDSim` Interfaces, ILCDIRAC-507
*Workflow
NEW: Added :mod:`~ILCDIRAC.Workflow.Modules.DDSimAnalysis` Workflow Module, ILCDIRAC-507
*ILCTransformation
FIX: Adapt to interface changes in DIRAC/Transformation, ILCDIRAC-599
[v25-pre3]
*Interfaces
NEW: Added :mod:`~ILCDIRAC.Interfaces.API.NewInterface.Applications.DDSim` Interfaces, ILCDIRAC-507
*Workflow
NEW: Added :mod:`~ILCDIRAC.Workflow.Modules.DDSimAnalysis` Workflow Module, ILCDIRAC-507
[v25-pre2]
*Core
NEW: Based on Dirac v6r14p11
FIX: XRootStorage is working again thanks to v6r14p20
CHANGE: Removed some deprecated and unused packages
NEW: Added many tests and made them compatible with nosetests
[v25-pre1]
*Core
NEW: Based on Dirac v6r14p10
*ILCTransformation
FIX: Adapt to interface changes in DIRAC/Transformation, ILCDIRAC-599
[v24-pre5]
*Core
NEW: Based on Dirac v6r13p14
[v24-pre3]
*Core
NEW: Based on Dirac v6r13p9
[v24-pre2]
*Core
NEW: Based on Dirac v6r13p8
[v24-pre1]
*Core
NEW: Based on Dirac v6r13p7
############################################################
[v23r1p1]
*Note
CHANGE: Moved many scripts to different locations (ILCDIRAC-581) for easier
automatic generation of the documentation. This requires a re-installation of
the ILCDIRAC client to remove outdated files. Apart from this
*Interfaces
CHANGE: Removed RemoveFileAndDaughtersWhenFileHasNoAncestor.py, script superseded by one in ILCDIRAC-Ops and :mod:`~ILCDIRAC.ILCTransformationSystem.Agent.DataRecoveryAgent`
CHANGE: Moved :any:`dirac-ilc-get-info` to `ILCTransformationSystem <ILCDIRAC.ILCTransformationSystem.scripts>`
CHANGE: Moved `dirac-ilc-get-prod-log` to `ILCTransformationSystem <ILCDIRAC.ILCTransformationSystem.scripts>`
CHANGE: Moved `dirac-ilc-production-summary` to `ILCTransformationSystem <ILCDIRAC.ILCTransformationSystem.scripts>`
CHANGE: Moved `dirac-ilc-replication-transformation` to `ILCTransformationSystem <ILCDIRAC.ILCTransformationSystem.scripts>`
CHANGE: Moved `dirac-ilc-upload-gen-files` to `ILCTransformationSystem <ILCDIRAC.ILCTransformationSystem.scripts>`
FIX: Updated :mod:`~ILCDIRAC.Interfaces.API.NewInterface.Applications.Marlin` documentation
FIX: Fix a typo `UserJob.setInputSandbox` documentation
*Core
CHANGE: Moved `ilcdirac-version` to `Interfaces <ILCDIRAC.Interfaces.scripts>`
CHANGE: Moved `dirac-ilc-show-software` to `Interfaces <ILCDIRAC.Interfaces.scripts>`
CHANGE: The log file failover location was changed so that log files are not picked up by the InputDataAgent for productions
################################################################################
[v23r0p12]
*Workflow
FIX: ILCDIRAC-612: Watchdog was not disabled properly when downloading overlay files
FIX: ILCDIRAC-612: Logging output was wrong when copying overlay files at IMPERIAL
[v23r0p11]
Based von DIRAC v6r12p53, see also v6r12p52
*Workflow
FIX: :mod:`~ILCDIRAC.Workflow.Modules.RootMacroAnalysis` and :mod:`~ILCDIRAC.Workflow.Modules.RootExecutableAnalysis` can now use the software versions natively installed on CVMFS (ILCDIRAC-582)
FIX: :mod:`~ILCDIRAC.Workflow.Modules.RootExecutableAnalysis` can now also be used to run binaries located in the PATH, e.g., 'hadd' to join root files
*ILCTransformationSystem
NEW: :any:`dirac-ilc-get-prod-log` now accepts -P prodID to get log files for given productionID (C. Calancha, ILCDIRAC-585)
FIX: :mod:`~ILCDIRAC.ILCTransformationSystem.Agent.DataRecoveryAgent` now ignores productions where nothing has changed since the last cycle, (ILCDIRAC-588)
[v23r0p10]
Based on DIRAC v6r12p51
*WorkloadManagement
NEW: :mod:`~ILCDIRAC.WorkloadManagementSystem.Executor.SoftwareVersions` Executor: Takes care to ban sites which are not compatible with the SofwarePackages required for any given job
*ILCTransformation
FIX: :mod:`~ILCDIRAC.ILCTransformationSystem.Agent.DataRecoveryAgent`: Fixes for notification, updates configuration before every cycle
*RequestManagement
FIX: ``RestartReqExeAgent`` deal with multiple PIDs
[v23r0p9]
*Core
NEW: Based on Dirac v6r12p50
*Interfaces
FIX: DIRAC FileCatalog CLI: fixes for path arguments
[v23r0p8]
*Workflow
FIX: Set OMP_NUM_THREADS=1 for all in :mod:`~ILCDIRAC.Workflow.Modules.ModuleBase` and in the execution of the Script in :mod:`~ILCDIRAC.Workflow.Modules.ApplicationScript`
*ILCTransformation
NEW: Rewrite of the :mod:`~ILCDIRAC.ILCTransformationSystem.Agent.DataRecoveryAgent`
[v23r0p6]
*Interfaces
FIX: :any:`dirac-ilc-upload-gen-files`: ILCDIRAC-571 correct energy variable, better logging for metadata setting in case of errors
*RequestManagement
NEW: ``RestartReqExeAgent`` to restart the RequestManagementAgent when it gets stuck, ILCDIRAC-572
*Workflow
FIX: ILCDIRAC-576: :mod:`~ILCDIRAC.Workflow.Modules.ApplicationScript`: Set OMP_NUM_THREADS=1
[v23r0p5]
*Core
NEW: Option to set the location of the shared area, needed to run with OSG cvmfs for now
[v23r0p4]
*Core
FIX: Based on Dirac v6r12p49, for real this time
NEW: ilcdirac-version script to display the ilcdirac and dirac versions
[v23r0p3]
*Core
FIX: Based on Dirac v6r12p48
NEW: createTarBallForProgram.sh script to create a tarball of the executable and required libraries
*Interfaces
FIX: ProductionJob: Adapt to new changePathMode interface (ILCDIRAC-544)
FIX: dirac-ilc-add-user: Adapt to new changePathOwner, changePathGroup interface
FIX: dirac-ilc-replication-transformation: check that ilc_prod proxy is used
[v23r0p2]
*Core
FIX: Based on Dirac v6r12p47, fixes issue with removing metadata in the filecatalog
*Interfaces
FIX: ILCDIRAC-547: Give error when dirac-ilc-find-in-FC does not get a path as first argument
[v23r0p1]
*Core
NEW: based on Dirac v6r12p46: Many fixes for Xroot(EOS), GlobusCEs
*Interfaces
FIX: ILCDIRAC-537: ILDProductionChain set different basepath for stdhepsplit productions
[v23r0]
*Core
NEW: Based on Dirac v6r12p44
*Interfaces
FIX: Cleanup of interfaces member variables, should be completely transparent for users
*Transformation
FIX: Fixes for ILDProductionJob from C. Calancha (ILCDIRAC-413)
[v23-pre5]
*Core
NEW: Based on Dirac v6r12p44
*Interfaces
FIX: Cleanup of interfaces member variables, should be completely transparent for users
[v23-pre3]
*Core
NEW: Based on Dirac v6r12p42
*Transformation
FIX: Fixes for ILDProductionJob from C. Calancha (ILCDIRAC-413)
[v23-pre1]
*Core
NEW: Based on Dirac v6r12-pre21: DO NOT USE
[v22r2p5]
*Transformation
NEW: Script to create replication transformations dirac-ilc-replication-transformation
*Workflow
FIX: OverlayInput will only waste time after downloading the first file
*Dirac
NEW: based on v6r11p5 (GlobusComputingElement)
[v22r2p4]
*Workflow
FIX: Fix if ILDConfig cannot access CVMFS
[v22r2p3]
*Dirac
NEW: Based on Dirac version v6r11p33: Issue regarding space in CE_OS description
*Workflow
FIX: ILCDIRAC-413 fix regarding picking up ILDConfig from CVMFS
*RMS
FIX: LogUpload would place log files in the wrong location
*Scripts
FIX: ILCDIRAC-474: dirac-ilc-add-whizard fixes for creating the replication request
*Core
FIX: ILCDIRAC-479: Fix python warning
[v22r2p2]
*Dirac
NEW: Based on Dirac version v6r11p32: Fixes issue for dirac-ilc-list-user
[v22r2p1]
*Workflow
FIX: OverlayInput will now always try to get files from the DataManager if files do not exist on local SE
[v22r2p0]
*DIRAC
NEW: v6r11p31
*Workflow
CHANGE: Tar and compress production logfiles
FIX: CVMFS support for native installations (Mokka, Marlin)
FIX: Fix some of the issues with module- and step_commons-parameters
FIX: Mokka: ProcessID is now passed to the module
FIX: Remove SystemConfig/sysconfig, replace with platform
FIX: Bugfixes/conventions/pylint cleanup, refactoring
[v22r2-pre4]
*Workflow
FIX: Fix some of the issues with module and step_commons - parameters
FIX: Mokka: ProcessID is now passed to the module
[v22r2-pre3]
*DIRAC
NEW: v6r11p31
*Workflow
FIX: CVMFS support for native installations (Mokka, Marlin)
FIX: Bugfixes/conventions/pylint cleanup, refactoring
[v22r2-pre1]
*Workflow
CHANGE: Tar and compress logfiles
FIX: Remove SystemConfig/sysconfig, replace with platform