Skip to content

APIs for Plugin Developers

The scope of this document is to provide a useful description of commonly used APIs to develop a FabSim3 plugin.

All our APIs are available on the FabSim3 GitHub repository, and are regularly updated. For any further assistance or inquiries, email us at djgroennl@gmail.com or hamid.arabnejad@gmail.com.

machine-specific configuration for plugin

Simulations can be run locally, or remotely on HPC resources. The submission and execution processes of a workflow require some information about the target host machine, such as username and host ip address. We encapsulate this information into two main files and in a human-readable format structure, specifically when using YAML language:

  • FabSim3/fabsim/deploy/machines.yml file where general machine-specific configurations are stored and
  • FabSim3/fabsim/deploy/machines_user.yml file which contains user-specific information for each remote machine.

In addition to these two files, you can also define machine configuration, for each remote machine, specified for your plugin. This will give you better flexibility to share your FabSim3 plugin with other research groups. The structure is similar to machines.yml and machines_user.yml files. But, the file name should follow the following pattern: 

machines_<plugin_name>_user.yml

Here are two examples of machine-specific configuration:

Usage:

To load machine-specific configuration specified by the user for the input plugin_name, you need to wrap the task functions with @load_plugin_env_vars(plugin_name) 

...
@task
@load_plugin_env_vars(plugin_name)
def <function_name>(input_args):
    ...
...

Hint

The @task is a pre-defined decorator by fabric library which makes your function callable from command line. For more details, please look at task definition in fabric library.

Note

This only works if the machine-specific configuration file (i.e., machines_<plugin_name>_user.yml) exists in your plugin directory (i.e., FabSim3/plugins/<plugin_name>).

Common APIs:

add_local_paths(plugin_name)

This function adds your plugin directory to FabSim3 (a) templates and (b) config_files PATH system. FabSim3 uses the template PATH to find the target script and generates the required script for job execution. The config_path will be used to find your application and transfer the required files/folder to remote machine.

Attention

You should add add_local_paths function at top of the main fabric file for your plugin. Otherwise, the target script and application config files for job submission and execution can not be found by FabSim3.

update_environment(args)

This function updates the environmental variables that are loaded from machine-specific configurations yaml files. The hierarchy for loading machine-specific configurations are:

  1. FabSim3/fabsim/deploy/machines.yml
  2. FabSim3/fabsim/deploy/machines_user.yml
  3. FabSim3/plugin/<plugin_name>/machines_<plugin_name>_user.yml (if exists)
  4. User input arguments via command line (Modified/Added by update_environment function or manual coding)

with_config(config_name)

This function augments environment variables, such as the remote location variables where the config files for the job should be found or stored, with information regarding a particular configuration name.

put_configs(config_name)

This function transfers config files to the target machine (local and remote resources). It should be called within the fabric execute function to honour host decorators, as follow:

  • execute(put_configs, config_name) where config_name is the name of the config folder.

job(args)

This is an internal low-level job launcher. It will submit a single job to the target machine (local and remote resources). All required parameters for the job are determined from the prepared fabric environment.

run_ensemble(config, sweep_dir, sweep_on_remote, execute_put_configs, upscale, replica_start_number, **args,)

This function maps and submits an ensemble job to the target machine. For each listed directory from sweep folder, an individual job will be submitted. The input arguments are:

  • config: the input config folder name;
  • sweep_dir: the PATH to the sweep directory;
  • sweep_on_remote (optional): this flag indicates if you sweep directory located on your local machine or is saved on remote machine. By default is False;
  • execute_put_configs (optional): this flag transfers all config files to the remote machine. In case of having config files on the remote machine, these arguments should be set to False, to avoid overwriting config files.
  • upsample (optional): list of subdirectories in the sweepdir to be upsampled. This flag, if provided, performs ensemble runs for the specified subset of directories while keeping the previous ensemble runs.
  • replica_start_number (optional): the starting point of replica numbering. Defaults to 1.

find_config_file_path(config_name)

Returns the PATH of input config_name in your plugin.

get_plugin_path(plugin_name)

Returns the local base PATH of input plugin name.

local(command)

Runs the input command on the local system.

run(command)

Runs a shell command on a remote host with following conditions:

  • If manual_gsissh env variable is set to True in fabsim/deploy/machines.yml file, then the gsissh command will invoke execution. The gsissh command provides a secure remote login service with forwarding of X.509 proxy credentials.
  • If manual_ssh env variable is set to True in fabsim/deploy/machines.yml file, the port number from machines.yml will be used to establish the ssh connection.
  • otherwise, the shell command on a remote system via SSH with default port number 22 will be executed.