Changelog

Transfer Module

Version 3.4 - 2021-xx-xx
  • enhancement

    • downloads are performed asynchronously, using only one protocol (http) and under the control of asyncio

    • std.conf new entries : big_file_size, big_file_chunksize

    • internal.conf entry : transfer_protocol

  • tests :

    • remove subcommand : new data file embedded is ten times smaller than previous one

  • deprecated features

    • gridftp protocol

    • wget http client

    • std.conf entries : gridftp_opt

    • internal.conf entry : transfer_protocols

    • documentation update

Version 3.35 - 2021-05-07
  • bug correction :

    • init-env subcommand now provides empty default selection files (see conf/default directory)

    • issue # 185 : ‘synda remove’ fails to import sdsimplefilter

Version 3.34 - 2021-04-02
  • bug correction :

    • Default selection file for CMIP6 project is now delivered empty. It can be customized under user control (issue # 178 : Dataset not found but available on ESGF web interface)

  • Documentation update

Version 3.33 - 2021-03-31
  • bug correction :

    • issue # 177 : synda now allows that ESGF openIDs are case sensitive

Version 3.32 - 2021-03-24 - (PYTHON 3)
  • enhancement

    • code migration : third attempt to migrate synda from python 2 to python 3 (success)

  • bug correction :

    • issue # 176 (conda channel changed : now : conda-forge [ipsl channel has been removed])

Version 3.31 - 2021-03-22
  • enhancement

    • code migration : second attempt to migrate synda from python 2 to python 3 (failure)

  • bug correction :

    • issue # 168 (Master branch update broke sdnexturl.py)

Version 3.30 - 2021-03-22
  • enhancement

    • code migration : first attempt to migrate synda from python 2 to python 3 (failure)

  • bugs correction :

    • synda -h and synda -V sub-commands are now operational

  • deprecated features

    • ‘open’ subcommand

    • packages : sdpoodlefix, sdxml, sdhtmlbasic, sdselectionfile, sdview

Version 3.2 - 2021-01-11

  • config parameters externalisation

    • ‘sdt.conf’ file has new customizable parameters

    • New ‘internal.conf’ file, for advanced parameters (not customizable, located into the /conf directory)

  • enhancements

    • init-env subcommand is enabled to build user environment

    • synda raises on the command line the problems encountered by esgf_search api service

    • In sdparam.get_name_from_value method, if more than one match, the print message has now a ‘info’ level

    • store stacktrace log files in /log directory.

  • bugs correction :

    • help subcommand can be used even if user has not built yet any environment

  • deprecated features

    • ‘pexec’ subcommand

    • ‘thredds_catalog’ & ‘apache_default_listing’ APIS (synda operates only the ‘esgf_search’ API)

    • synda ihm

    • packages : sdevent, sdeventdao, sddatasetpattern, sdoperation

Version 3.15 - 2020-11-17

  • cleaning & packages reorganization

  • issue #158 - bug correction - daemon stopping without reporting errors

Version 3.14 - 2020-10-29

  • new files encoding : ISO-8859-1 => utf8

Version 3.13 - 2020-10-28

  • entry point moved to setup.py instead of meta.yaml file

Version 3.9 - 2017-02-19

  • add gridftp_opt option in sdt.conf.

  • dataset attributes can be used in ‘local_path_drs_template’.

  • use decimal prefix instead of binary prefix in ‘synda metrics’ command.

  • do not include checksum computation time in download time metric.

  • enable supplementary groups support if available.

Version 3.8 - 2017-02-19

  • add info action to synda certificate command

  • add security_dir_mode option

  • add subfolder to the ESGF certificate path (named after the user UID)

  • all deletions now occurs synchronously (deferred mode is disabled)

  • follow symlinks during synda remove operation

  • add manual routines to refresh dataset status

  • new cleanup algo for synda remove command (run faster)

  • add full path support in local_path_drs_template var

  • add write test in sddaemon module

  • store stacktrace file in /tmp

  • add uuid to stacktrace file name

  • move try/except block upstream to enclose the daemon context

  • fix permission in sys-pkg for /srv/synda/sdt folder (g+sw)

Version 3.7 - 2017-01-15

  • ESGF certificate path set to $HOME/sdt/tmp/.esg for source install.

  • disable sdfilepermission module.

  • add user personal env (SDT_USER_ENV).

  • synda group can be used instead of sudo (system package).

  • set specific version for pillow package.

  • bugfixes.

Version 3.6 - 2016-12-04

  • use search-api json format instead of xml

  • add end-of-transfer events replay mode (trigger all EOT events in batch mode)

  • add synda count subcommand

  • add synda config subcommand

  • add incremental mode for discovery operation

  • add dataset version consistency check

Version 3.5 - 2016-09-19

  • use requests (pypi) to resolve openid

  • add ~/.sdt/conf/credentials.conf

  • add ~/.sdt/conf/sdt.conf

  • remove ~/.syndarc

  • add open subcommand

  • add check subcommand for selection

  • add check subcommand for file’s variable

  • use myproxyclient latest version

  • add default_path option in sdt.conf

  • use sdpyproxy python module to renew X509 certificate (replace obsolete sdlogon.sh script)

  • change tarball location for source installation (use install.sh latest version)

  • replace argparse RawTextHelpFormatter with RawDescriptionHelpFormatter

  • set metadata_parallel_download to False

  • add support for lowmem machine

  • add sdsqlitedict module

  • add sdmts module

  • add ignorecase filter for upgrade and facet subcommands

  • remove -n option from synda facet subcommand

  • do not stop daemon anymore when error occurs during download (except for certificate renewal error)

  • always print stacktrace when error occurs (ST_DEBUG env. var. is not used anymore)

  • write yes/no confirmation message on stderr

  • prevent adding predefined values for version facet in local cache

  • add checksum type normalization routine

  • add url_replace selection file parameter

  • add http_fallback option (switch protocol on error)

  • add default_listing_size configuration option

  • add -m option to synda remove subcommand

  • ergonomy improvements

Version 3.4 - 2016-05-04

  • add synda variable subcommand

  • add synda get subcommand

  • remove synda test subcommand (synda test is replaced by synda get)

  • add urllib2 based download impl

  • add sandbox folder for untracked data

  • set data folder default value to /srv/synda/sdt/data (system package installation only)

  • disable parameter checking by default

  • insert event in db even if post-processing module is disabled

  • improve obs4MIPs project support

  • user documentation reorganization

  • add parameter reference documentation

  • add commands reference documentation

  • add selection file documentation

Version 3.3 - 2016-04-04

  • localpath DRS can be customized by user in configuration file (local_path_drs_template)

  • add hpss configuration parameter to retry transfer on error

  • add index_host and project options to the synda update subcommand

  • add examples for each subcommands

  • use argparse.RawTextHelpFormatter in subcommand help

  • add globus online support

  • set gridftp port range to 50000-51000

  • in sdget.sh and sdgetg.sh, -v option replace -d option (verbose replace debug_level)

  • add new value format for synda dump -F option

  • enable debug mode when using synda test <url> subcommand

  • synda list now list everything by default

  • add missing initialization in install.sh (g__transfer=0)

  • add configuration file documentation

  • move post_processing parameter from daemon section to module section

  • rename configuration file [path] section to [core] section

  • do not remove ca certs when using synda certificate renew option

  • add force_renew_ca_certficates option to remove ca certs

  • remove cache subcommand

  • remove sdreducecol filter from sdfilepipeline module

  • fix pkg_resources.DistributionNotFound: setuptools>=1.0 bug

  • move common method from Dataset and File class to BaseType class

  • set prevent_daemon_and_modification to false for source installation

  • move max_parallel_download from [daemon] to [download] section

  • untar ihm_pid_file only if mutually exclusive lock is enabled

  • increase daemon sqlite timeout from 120s to 12000s

  • do not parse wget output by default and increase wget --tries to prevent hpss failure

  • default indexes set to dkrz

  • daemon non-privileged mode

Version 3.2 - 2016-02-03

  • DEB package

  • retrieve dataset timestamp in batch mode

  • modify Synda scheduler to ease Globus Online integration

  • prevent normal user to run admin commands in multi-user mode

  • set model attribute as optional

  • improve documentation

Version 3.1 - 2015-12-29

  • multi-user

  • daemon integrated in systemd

  • RPM package

  • per-user config file (~/.syndarc)

  • online help

  • parameter discovery (list parameter based on other parameters)

  • support for free syntax in template (e.g. [realm experiment frequency]=v1 v2)

  • default indexes set to pcmdi9

  • add inline tutorial

  • -z option replace -y option

  • -y option replace -N option

  • --yes option replace --non-interactive option

  • openid/passwd moved from sdt.conf to credentials.conf

  • add check to prevent normal user from running synda in write mode

  • add ignorecase filter

Version 3.0 - 2015-03-25

  • add new local search filter (status, error_msg)

  • add --version option to print version in synda command

  • improve external files support

  • add next replica action (batch mode)

  • move default selection files in sdt/conf/default

  • move configuration file in sdt/conf folder

  • add history subcommand

  • move lfae_mode into sdt.conf

  • gridftp support

Version 2.9 - 2014-11-03

  • several template parameters names changed (e.g. tablename is now named cmor_table). See sdconvert.sh for more info

  • new synda command (apt-get like front-end)

  • support for most search-API parameters

  • “not” operation support (e.g. all models but one)

  • multi-DRS support

  • new formatting keyword

  • only localpath is mutable

  • support for different name for the same model (e.g. GFDL-CM2p1, GFDL-CM2.1 et GFDL-CM2-1)

  • default values per project

  • new searchapi_host parameter to specify which index to use

  • space are supported (e.g. “ISI-MIP Fasttrack”)

  • replica support

  • wildcard (all/_*) supported in all facets

  • local database reorganization

Version 2.8 - 2013-12-20

  • set CHUNKSIZE (search-API limit parameter) to 10000 (was 1000)

  • add time coverage filter

  • add support for sha256 checksum type

Version 2.7 - 2013-08-20

  • fix B0039 bug

  • fix B0034 bug

  • fix B0033 bug

  • add EUCLIPSE project

  • XML parsing module rewriting

  • add “timeout/retry” mechanism in the discovery process

  • models discovery module improvement

  • move tuning parameters into configuration file

  • increase thredds-catalog timeout from 10 to 100

  • add second logger for domain/functional messages

  • load readonly tables in memory to speed up the discovery process

  • add CMOR tables cache system

  • add orphan transfer detection (without selection match)

  • stat subcommand rewriting

  • add db_path option in configuration file

Version 2.6 - 2013-04-18

  • add search-api-nocache discovery engine

  • add support for “obs4MIPs” project

  • add wild card support for realm and frequency

  • fix B0032 bug

Version 2.5 - 2012-12-18

  • add url column in dataset tmp tables

  • add -G option (remove tmp tables)

  • set MyProxyClient as default myproxy client

  • set search-API as default search-engine

  • add selection based statistics

  • add new -E option to retrieve model list from search-API

  • fix B0031 bug

  • replace PCMDI3 with PCMDI9 in get_data.sh script (myproxy server)

  • add search-API multithreading to run several search in parallel

  • add search-API call metrics (to trace time spent in each call)

  • add search-API pagination

  • fix B0030 bug

  • fix B0029 bug

  • fix B0028 bug

  • fix B0027 bug

  • fix B0026 bug

Version 2.4 - 2012-06-19

  • add -x option to run discovery process and print ESGF checksums

  • add -X option to control if local checksum match remote checksum

  • fix B0025 bug

  • add “latest” symlink creation routine (last version identifier)

  • add old versions suppression routine

  • add search API mode

  • add -L option (set latest flag)

  • fix B0024 bug

  • mark CSTE_TRANSFERT_STATUS_DELETED status as deprecated

Version 2.3 - 2012-04-20

  • add PROC0001 method to list obsolete version

  • add new columns latest_date and last_done_transfer_date

  • fix B0023 bug

  • fix B0022 bug (MIGR0001() method broken)

  • fix B0021 bug. (variable missing when retrieving transfert from database)

  • add -y option (dataset-info)

Version 2.2 - 2012-04-07

  • fix B0020 bug. (fix 2.2 at 20120410)

  • fix B0019 bug. (fix 2.2 at 20120407)

  • add ESGF MyProxyLogon (MyProxy Java client)

  • replace ps fax with ps ax (Mac port)

  • add dependencies check in install.sh

  • add transfer_helper modules

  • add dataset in transfer queue (eot_queue)

  • fix B0018 bug

  • fix B0017 bug

  • fix “[Error 98] address already in use”

  • use wget tries and timeout parameters from conf. file

  • fix B0016 bug

  • add -r option (exec proc)

  • use transfert_id instead of local_image as primary key (for update)

  • add new table dataset

  • add new column dataset_id in transfer table

Version 2.1 - 2012-03-12

  • fix B0015 bug

  • add -V option in start.sh

  • add -b to myproxy-logon options (only if myproxy-logon >= 5.0)

  • set wget tries option to 1

  • fix B0014 bug

  • fix B0013 bug

  • add abnormal termination recovery routine

  • add ignore checksum option

  • fix B0012 bug

  • set SQLite lock timeout to 120s

  • improve scheduler (increase queue and dequeue performance)

  • frozen wget watchdog reactivation

Version 2.0 - 2012-02-14

  • add new synchronisation mode (retrieve dataset last version only)

  • fix B0011 bug (remove local files when checksum doesn’t match)

  • add list-local-files action

  • fix B0007 bug (replace urllib with urllib2 and set timeout to 10)

  • fix B0006 bug (add missing env. var. in stop.sh)

  • fix B0009 bug (catch exception and process others datasets)

  • remove non-working models from models table

Version 1.9 - 2012-01-30

  • fix B0005 bug

Version 1.8 - 2012-01-28

  • add -w option (shutdown immediate)

  • improve errors handling

  • fix B0003 bug

  • fix start.sh -e option (B0004 bug)

Version 1.7 - 2012-01-27

  • add start.sh -u option (refresh ESGF metadata)

  • add start.sh -q option (stop daemon)

  • add start.sh -l option (list selections)

Version 1.6 - 2012-01-26

  • fix B0001 and B0002 bugs

  • remove one-file-per-model logging

  • add metadata caching system

  • merge all logs in one file

  • add model in tmp tables (dataset_version and file_timeslice)

Version 1.5 - 2012-01-18

  • move models loop inside the feeder

  • add CMOR tablename forcing in template

  • add stat subcommand

  • add syncmode check in start.sh

  • move product out of local_image column

Version 1.4 - 2012-01-14

  • set myproxy-logon as default (change procedure in README to use install.sh -a)

  • add delete subcommand

  • add cancel subcommand

  • add retry subcommand

  • add info subcommand

  • merge output1 and output2 into output

  • improve installation process

  • use synchronous events to control the daemon

Version 1.3 - 2012-01-02

  • automatic update of model/datanode list

  • add remote and local checksum

  • unset X509_USER_PROXY variable (in get_data.sh script)

  • fix selection overlapping bug

  • fix product bug (check to prevent output value for product)

  • add license information

  • add svn properties in header

  • ignore blank lines in selection files

Version 1.2 - 2011-10-07

  • improve ensemble support

  • increase from 8 to 16 Wget threads

  • add per model priority

  • move main loop delay from 3 seconds to 6 seconds

  • support file ID with non-standard extension (.nc_0)

  • add upgrade and archive option in script install.sh

  • improve HTTP error handling in script get_data.sh

Version 1.1 - 2011-09-28

  • improve datanode and model configuration

  • improve HTTP error code handling

  • add watchdog to check for frozen wget

  • fix PCMDI datanode incorrect url

  • fix incorrect configuration for models GISS-E2-H, GISS-E2-R and inmcm4

  • add new models (HadCM3,IPSL-CM5A-LR,CanAM4,MIROC5,MIROC4h,CCSM4,MRI-CGCM3,MRI-AGCM3-2S,MRI-AGCM3-2H,MPI-ESM-LR)

Version 1.0 - 2011-09-09

  • support for myproxy-logon and myproxyclient

  • simple data selection with model, experiment, realm and variable

  • multi threaded downloads (8 tasks by default)

  • manage datasets version following new drs

  • incremental process (download only what’s new)

  • download history stored in a database

Post-Processing Module

Version 1.3 - 2017-01-15

  • synda group can be used instead of sudo (system package)

Version 1.2 - 2016-12-04

  • move hard-coded pipeline dependencies into configuration files

  • add credentials.conf file

  • add pipeline samples

  • bugfixes

Version 1.1 - 2016-09-19

  • add CORDEX support

  • add pexec support

  • add multivalues support for job_class option

  • add conf folder

  • add pipeline_path

  • improve worker log routines

Version 1.0 - 2014-12-25

  • pipeline engine

  • Jsonrpc server

  • database environment

  • worker script