newsATLAS Distributed Data Management - News: DQ2 Clients Testing Release 0.1.37

 
 
Show feedback again
Latest News
DQ2 Clients Stable Release 2.3.0 posted by angelos, 2012-05-08 11:07 - 0 replies
DDM: Athena compatible Rucio clients available posted by angelos, 2012-04-03 08:12 - 0 replies
DQ2 Clients Testing Release 2.0.0_rc2 posted by angelos, 2012-04-03 00:48 - 0 replies
DQ2 Clients Stable Release 1.0.0 posted by angelos, 2012-02-13 19:42 - 0 replies
DQ2 Clients Testing Release 1.0.0_rc2 posted by angelos, 2012-01-19 14:31 - 0 replies

DQ2 Clients Testing Release 0.1.37

Item posted by Angelos Molfetas <angelos> on 2011-11-02 19:03.

Dear members of the ATLAS collaboration,

On behalf of the DQ2 team I am announcing the release of a new DQ2 Clients stable release, 0.1.37 "Taipei". This release has a new registration API for Panda and new retrial capabilities for dq2-get and dq2-put. The dq2-clientapi and dq2-clientapi-cli packages have been merged which produce a unified rpm.

Changelog for 0.1.37
-----------------------------

New Features:

* Added extended command version printout that shows both DQ2 Clients version and SVN Revision number (https://svnweb.cern.ch/trac/dq2/tic...)
* dq2-ls -l option that shows the number of files and total size in dataset (https://svnweb.cern.ch/trac/dq2/tic...)
* New deletion API method 'getSizeofQueue' - returns the number of files not yet deleted per endpoints
* New deletion API method 'setReplicasStatus' - bulk update state of groups of files for storage cleaner
* 2nd DQ2 Register API that registers files into multiple datasets on the Central Catalogs and LFC (https://svnweb.cern.ch/trac/dq2/tic...)
* DQ2 Register API that has similar functionality to dq2-register used by tier-0 (https://svnweb.cern.ch/trac/dq2/tic...), Instructions: https://svnweb.cern.ch/trac/dq2/wik...
* Testing scripts allow user to choice between Python 2.5 (32 bit), Python 2.6 (32 bit), Python 2.6 (64 bit), this was done before, but now its one file instead of three (https://svnweb.cern.ch/trac/dq2/tic...)
* The new dq2-put supports the assignment of GUID for files ( https://savannah.cern.ch/task/index..., https://svnweb.cern.ch/trac/dq2/tic...)
* Added dq2 list accounting CLI (https://svnweb.cern.ch/trac/dq2/tic...)
* Added dq2 list accounting API (https://svnweb.cern.ch/trac/dq2/tic...)

Improvements:

* dq2-put now has a retrial strategy ( https://savannah.cern.ch/bugs/?7903..., https://svnweb.cern.ch/trac/dq2/tic...)
* Tape gets lower priority in download queue (https://svnweb.cern.ch/trac/dq2/tic...)
* Rollback improved in dq2-put (https://svnweb.cern.ch/trac/dq2/tic...)
* Improved status code reporting in dq2-put (https://svnweb.cern.ch/trac/dq2/tic...)
* dq2-register CLI will not stop when it finds files listed are already registered files (https://svnweb.cern.ch/trac/dq2/tic...)
* If possible dq2-put will attempt to extact GUID from and use that to register them (https://svnweb.cern.ch/trac/dq2/tic...)
* As before, dq2-put will require Athena to be sourced if pool.root are present (for GUID extraction), however only if specific GUIDs were not specified (https://svnweb.cern.ch/trac/dq2/tic...)
* Removed dq2-get's preference of close Tier 1 sites over close Tier 2 sites to insure even distribution of downloads (https://svnweb.cern.ch/trac/dq2/tic...)
* Merged dq2 clients api and cli into a single package (https://svnweb.cern.ch/trac/dq2/tic...)
* Reversed ng* and arc* tools in the Endpool list (https://svnweb.cern.ch/trac/dq2/tic...)
* 2nd DQ2 Register API allows for surls to be manually be provided instead of automatically generating them (https://svnweb.cern.ch/trac/dq2/tic...)
* Added web link to DQ2 error message (https://svnweb.cern.ch/trac/dq2/tic...)
* Added warning and special ignore option to dq2-get to prevent the enactment of large numbers of dq2-get threads (https://svnweb.cern.ch/trac/dq2/tic...)
* Added more debug statements for dq2-get when specifying container pattern with a wild card (https://svnweb.cern.ch/trac/dq2/tic...)
* Merged call to get dataset's DUID and VUID, into a single call to the Central Catalogs (https://svnweb.cern.ch/trac/dq2/tic...)
* DQ2-Get now will retrial failed download attempts up to five times (https://svnweb.cern.ch/trac/dq2/tic...)
* dq2-put will now require for Athena to be sourced only if pool.root files are being uploaded (https://svnweb.cern.ch/trac/dq2/tic...)
* DQ2 Clients check that a valid version of python is used (https://svnweb.cern.ch/trac/dq2/tic...)
* FTS plugin enabled by default on AFS and Pacman installations (https://svnweb.cern.ch/trac/dq2/tic...,  https://savannah.cern.ch/bugs/index...,  https://savannah.cern.ch/bugs/index...)
* Testing scripts more compatible to non SLC distributions better ( https://svnweb.cern.ch/trac/dq2/tic...), note that non SLC5 distributions are not officially supported.
* Reorganised testing environments providing: python 2.5 (32bit) python 2.6 (32bit, 64bit) (https://svnweb.cern.ch/trac/dq2/tic...)

Bugs:

* Corrected wrong delimeter used for lcg-getturls protocol list in dq2-PFC-gen (https://svnweb.cern.ch/trac/dq2/tic...)
* Added missing SURL build call in dq2-get which was causing plugin to fail (https://svnweb.cern.ch/trac/dq2/tic...)
* dq2-get help description documents use of wildcards with -f option (https://svnweb.cern.ch/trac/dq2/tic...)
* dq2-get thread does not fail when dq2 checksum does not exist (https://svnweb.cern.ch/trac/dq2/tic...)
* dq2-ls -fpL now displays the number of local files as it does without the p option (https://svnweb.cern.ch/trac/dq2/tic...)
* dq2-ls -fpL now display the LFN when warning that PFN is not available (https://svnweb.cern.ch/trac/dq2/tic...)
* Reversed total no of files, and total dataset size in dq2-list-dataset help message (https://svnweb.cern.ch/trac/dq2/tic...)
* Fix to a dq2-put bug that improperly invalided certain command line options (https://svnweb.cern.ch/trac/dq2/tic...)
* Fix for problems where "ROAMING" profile was used with dq2-get (https://svnweb.cern.ch/trac/dq2/tic...)
* Made tracer client thread safe (https://svnweb.cern.ch/trac/dq2/tic...)

Complete list of bug fixes, new features, and other improvements for all 0.1.37 testing releases
--------------------------------------------------------------------------------------------------------------

https://svnweb.cern.ch/trac/dq2/que...

To access using AFS:

(zsh) source /afs/cern.ch/atlas/offline/external/GRID/ddm/DQ2Clients/setup.zsh

(bash) source /afs/cern.ch/atlas/offline/external/GRID/ddm/DQ2Clients/setup.sh

To install them with pacman:
--------------------------------------

pacman -trust-all-caches -allow tar-overwrite -get  http://atlas.web.cern.ch/Atlas/GROU...

Pacman installation instructions:
---------------------------------------

https://svnweb.cern.ch/trac/dq2/wik...

The RPMs can be found here:
---------------------------------------

wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...
wget  http://voatlas149.cern.ch/ddm/apt/R...

Web page and Documentation is available at:
----------------------------------------------

https://twiki.cern.ch/twiki/bin/vie...

HOWTO is available at:
---------------------------

https://twiki.cern.ch/twiki/bin/vie...

Regards,

Angelos, on behalf of the DDM team.

Comments:

No Messages in DQ2 Clients Testing Release 0.1.37

 

Start a New Thread:

You could post if you were logged in
Show feedback again

Back to the top


Powered by Savane SVN (toward 3.1)