Command Line Interface

This page contains the reference of the Command Line Interface. For hints on its usage please explore the User Guide.

enstools-compression

usage: enstools-compression [-h]
                            {compress,analyze,significand,evaluate,prune,load-plugins,unload-plugins}
                            ...

-h, --help: show this help message and exit

enstools-compression analyze

usage: enstools-compression analyze [-h] [--constrains CONSTRAINS]
                                    [--output OUTPUT]
                                    [--compressor COMPRESSOR] [--mode MODE]
                                    [--grid GRID]
                                    [--plugins [PLUGINS [PLUGINS ...]]]
                                    [--fill-na FILL_NA]
                                    [--variables VARIABLES]
                                    files [files ...]

files: List of files to compress. Multiple files and regex patterns are allowed.

-h, --help: show this help message and exit

--constrains <constrains>: Quality constrains that need to be fulfilled.

--output <output>, -o <output>: Path to the file where the configuration will be saved.If not provided will be print in the stdout.

--compressor <compressor>, -c <compressor>: Which compressor will be used. Options are zfp, sz or all.

--mode <mode>, -m <mode>: Which mode will be used. The options depend on the compressor.For sz: abs, rel, pw_rel. For zfp: accuracy, rate, precision.Also it is possible to use ‘all’

--grid <grid>, -g <grid>: Path to the file containing grid information.

--plugins <plugins>: List of files with custom metric definitions.

--fill-na <fill_na>: Fill the missing values with a float.

--variables <variables>: List of variables to analyze.Must be a list of comma separated values: i.e. vor,temp,qvDefault=None

enstools-compression compress

compress:

Tool to apply HDF5 compression filters to netCDF files. Compression specifications must follow the Compression Specification Format, more details below.

Examples

-Single file:

>>> enstools-compression compress "input.nc" -o "output/folder/" 
or
>>> enstools-compression compress "input.nc" -o "output_file.nc"

-Multiple files:

>>> enstools-compression compress "input_1.nc" "input_2.nc" -o "output/folder/"

-Multiple files using wildcards:

>>> enstools-compression compress "input_files_*" -o "output/folder/"  

To use custom compression parameters:

>>> enstools-compression compress "input_files_*" -o "output/folder/" --compression compression_specification

Compression Specification Format

The compression_specification can be a string that follows the Compression Specification Format (see more details in enstools-encoding.readthedocs.com) or a filepath to a configuration file in which we might have per variable specification.

For lossless compression, we can choose the backend and the compression leven as follows

>>> lossless,backend,compression_level(from 1 to 9)

The backend must be one of the following options:

blosclz
lz4
lz4hc
snappy
zlib
zstd

For lossy compression, we can choose the compressor (sz or zfp), the method and the method parameter.

>>> lossy,compressor,mode,parameter

Some examples:

lossless
lossless,zlib,5
lossy,zfp,accuracy,0.00001
lossy,zfp,precision,12
lossy,zfp,rate,3.2
lossy,sz,abs,0.1
lossy,sz,rel,0.0001
lossy,sz,pw_rel,1.e-6

If using a configuration file, the file should follow a yaml format and can contain per-variable values. It is also possible to define a default option. For example:

default: lossless
temp: lossy,zfp,accuracy,0.1
qv: lossy,sz,pw_rel,0.0001

So, few examples with custom compression would be:

>>> enstools-compression compress "input_files_*" -o "output/folder/" --compression lossless

>>> enstools-compression compress "input_files_*" -o "output/folder/" --compression lossless,blosclz,9

>>> enstools-compression compress "input_files_*" -o "output/folder/" --compression lossy,zfp,rate,3.2

>>> enstools-compression compress "input_files_*" -o "output/folder/" --compression lossy,sz,abs,0.01

>>> enstools-compression compress "input_files_*" -o "output/folder/" --compression compression_parameters.yaml

Also, it is possible to automatically find which are the compression parameters that must be applied to each variable in order to maintain a 0.99999 Pearson correlation and a 0.99 structural similarity. Be aware that that will cause an overhead due to the necessary time to find the appropriate parameters.

>>> enstools-compression compress "input_files_*" -o "output/folder/" --compression auto

It is possible to parallelize the compression of multiple files by launch a SLURM job for workers. To do that, specify the number of nodes to use:

>>> enstools-compression compress -o "output/folder/" "input_files_*" --nodes 4

usage: enstools-compression compress [-h] -o OUTPUT
                                     [--compression COMPRESSION [COMPRESSION ...]]
                                     [--nodes NODES] [--variables VARIABLES]
                                     [--emulate] [--fill-na FILL_NA]
                                     [files [files ...]]

files: Path to file/files that will be compressed.Multiple files and regex patterns are allowed.

-h, --help: show this help message and exit

-o <output>, --output <output>

--compression <compression>: Specifications about the compression options. Default is: lossless

--nodes <nodes>, -N <nodes>: This parameter can be used to allocate additional nodes in the clusterto speed-up the computation.

--variables <variables>: List of variables to be kept. The other variables will be dropped.Must be a list of comma separated values: i.e. vor,temp,qvDefault=None

--emulate: Instead of saving compressed files, it compresses and decompresses the data to seecompression effects without requiring the plugins to open the files.

--fill-na <fill_na>: Fill the missing values with a float.

enstools-compression evaluate

usage: enstools-compression evaluate [-h] --reference REFERENCE_FILE --target
                                     TARGET_FILE [--plot] [-g]

-h, --help: show this help message and exit

--reference <reference_file>, -r <reference_file>: Path to reference file. Default=None

--target <target_file>, -t <target_file>: Path to target file

--plot: Produce evaluation plots. Default=False

-g, --gradients: Compute gradients. Default=False

enstools-compression load-plugins

usage: enstools-compression load-plugins [-h]

-h, --help: show this help message and exit

enstools-compression prune

usage: enstools-compression prune [-h] -o OUTPUT files [files ...]

files: List of files to compress. Multiple files and regex patterns are allowed.

-h, --help: show this help message and exit

-o <output>, --output <output>

enstools-compression significand

usage: enstools-compression significand [-h] [--output OUTPUT] [--grid GRID]
                                        files [files ...]

files: List of files to compress. Multiple files and regex patterns are allowed.

-h, --help: show this help message and exit

--output <output>, -o <output>: Path to the file where the configuration will be saved.If not provided will be print in the stdout.

--grid <grid>, -g <grid>: Path to the file containing grid information.

enstools-compression unload-plugins

usage: enstools-compression unload-plugins [-h]

-h, --help: show this help message and exit