APIs for Plugin Developers
The scope of this document is to provide a useful description of commonly used APIs to develop a FabSim3 plugin.
machine-specific configuration for plugin¶
Simulations can be run locally, or remotely on HPC resources. The submission and execution processes of a workflow require some information about the target host machine, such as
host ip address. We encapsulate this information into two main files and in a human-readable format structure, specifically when using YAML language:
FabSim3/fabsim/deploy/machines.ymlfile where general machine-specific configurations are stored and
FabSim3/fabsim/deploy/machines_user.ymlfile which contains user-specific information for each remote machine.
In addition to these two files, you can also define machine configuration, for each remote machine, specified for your plugin. This will give you better flexibility to share your FabSim3 plugin with other research groups. The structure is similar to
machines_user.yml files. But, the file name should follow the following pattern:
Here are two examples of machine-specific configuration:
To load machine-specific configuration specified by the user for the input
plugin_name, you need to wrap the task functions with
... @task @load_plugin_env_vars(plugin_name) def <function_name>(input_args): ... ...
@task is a pre-defined decorator by fabric library which makes your function callable from command line. For more details, please look at task definition in fabric library.
This only works if the machine-specific configuration file (i.e.,
machines_<plugin_name>_user.yml) exists in your plugin directory (i.e.,
This function adds your plugin directory to FabSim3 (a) templates and (b) config_files PATH system. FabSim3 uses the template PATH to find the target script and generates the required script for job execution. The config_path will be used to find your application and transfer the required files/folder to remote machine.
You should add
add_local_paths function at top of the main fabric file for your plugin. Otherwise, the target script and application config files for job submission and execution can not be found by FabSim3.
This function updates the environmental variables that are loaded from machine-specific configurations yaml files. The hierarchy for loading machine-specific configurations are:
- User input arguments via command line (Modified/Added by
update_environmentfunction or manual coding)
This function augments environment variables, such as the remote location variables where the config files for the job should be found or stored, with information regarding a particular configuration name.
This function transfers config files to the target machine (local and remote resources). It should be called within the fabric execute function to honour host decorators, as follow:
execute(put_configs, config_name)where config_name is the name of the config folder.
This is an internal low-level job launcher. It will submit a single job to the target machine (local and remote resources). All required parameters for the job are determined from the prepared fabric environment.
run_ensemble(config, sweep_dir, sweep_on_remote, execute_put_configs, upscale, replica_start_number, **args,)¶
This function maps and submits an ensemble job to the target machine. For each listed directory from sweep folder, an individual job will be submitted. The input arguments are:
config: the input config folder name;
sweep_dir: the PATH to the sweep directory;
sweep_on_remote(optional): this flag indicates if you sweep directory located on your local machine or is saved on remote machine. By default is
execute_put_configs(optional): this flag transfers all config files to the remote machine. In case of having config files on the remote machine, these arguments should be set to
False, to avoid overwriting config files.
upsample(optional): list of subdirectories in the
sweepdirto be upsampled. This flag, if provided, performs ensemble runs for the specified subset of directories while keeping the previous ensemble runs.
replica_start_number(optional): the starting point of replica numbering. Defaults to 1.
Returns the PATH of input config_name in your plugin.
Returns the local base PATH of input plugin name.
Runs the input command on the local system.
Runs a shell command on a remote host with following conditions:
manual_gsisshenv variable is set to
fabsim/deploy/machines.ymlfile, then the
gsisshcommand will invoke execution. The
gsisshcommand provides a secure remote login service with forwarding of X.509 proxy credentials.
manual_sshenv variable is set to
fabsim/deploy/machines.ymlfile, the port number from
machines.ymlwill be used to establish the ssh connection.
- otherwise, the shell command on a remote system via SSH with default port number
22will be executed.