Skip to content

Tutorials

Creating a new plugin

Using Dummy Example

To create a new plugin for FabSim3:

  1. Fork the FabDummy repository.
  2. Rename the repository, and modify it to suit your own needs as you see fit.
  3. Rename FabDummy.py to the <name of your plugin>.py.
  4. In your new plugin repository, at the top of <name of your plugin>.py, change add_local_paths("FabDummy") add_local_paths("name of your plugin").
  5. In the main FabSim3 repository, add an entry for your new plugin in fabsim/deploy/plugins.yml file.
  6. Set up your plugin using
    fab localhost install_plugin:<name of your plugin>
    
  7. You’re good to go, although you’ll inevitably will have to debug some of your modifications made in the second step of course.

Writing a Plugin From scratch

In this tutorial, we explain how to write a FabSim3 plugin from scratch. To keep simplicity, the basic functionalities are presented here, for more advanced and complicated functionalities, we suggest reader to have look at the current plugins presented in Section xx in this work.

For this tutorial, a simple application, namely cannon_app, which calculates the range of a projectile fired at an angle is selected. By using simple physics rules, you can find how far a fired projectile will travel. The source code for this application, written in three of the most widely used languages: C,Java, and Python, is available here : https://github.com/arabnejad/cannon_app. The cannon_app reads the input parameters from a simple txt file and calculate the distance until ball hits the round. How far the ball travels will depend on the input parameters such as : speed, angle, gravity, and air resistance. Figure below shows the sample input setting file and the generate output plot.

cannon_app source codes

The srouce code of cannon_app written in three of the most widely used languages: C,Java, and Python can be found in https://github.com/arabnejad/cannon_app

import matplotlib
matplotlib.use('Agg')

from math import sin, cos
from os import path, makedirs
import csv
import argparse
import matplotlib.pyplot as plt


def launch(gravity, mass, velocity, angle, height,
           air_resistance, time_step):
    x = 0.0
    y = height
    vx = velocity * sin(angle)
    vy = velocity * cos(angle)

    out_x = []
    out_y = []
    while y > 0.0:
        # Euler integrate
        vy -= gravity * mass * time_step
        vx -= vx * air_resistance * time_step
        vy -= vy * air_resistance * time_step
        x += vx * time_step
        y += vy * time_step
        out_x.append(x)
        out_y.append(y)

    out_dist = x
    out_vx = vx
    out_vy = vy
    return out_dist, out_vx, out_vy, out_x, out_y


if __name__ == "__main__":

    # python cannonsim.py
    # python cannonsim.py --input_dir=<xxx> --output_dir=<xx>

    # Instantiate the parser
    parser = argparse.ArgumentParser()
    parser.add_argument('--input_dir',
                        action="store", default='input_files')
    parser.add_argument('--output_dir',
                        action="store", default='output_files')

    args = parser.parse_args()
    input_dir = args.input_dir
    output_dir = args.output_dir

    # read input parameters from simsetting.csv
    input_file = path.join(input_dir, "simsetting.txt")
    if path.isfile(input_file):
        with open(input_file, newline='') as csvfile:
            values = csv.reader(csvfile)
            for row in values:
                if len(row) > 0:  # skip empty lines in csv
                    if row[0][0] == "#":
                        pass
                    elif row[0].lower() == "gravity":
                        gravity = float(row[1])
                    elif row[0].lower() == "mass":
                        mass = float(row[1])
                    elif row[0].lower() == "velocity":
                        velocity = float(row[1])
                    elif row[0].lower() == "angle":
                        angle = float(row[1])
                    elif row[0].lower() == "height":
                        height = float(row[1])
                    elif row[0].lower() == "air_resistance":
                        air_resistance = float(row[1])
                    elif row[0].lower() == "time_step":
                        time_step = float(row[1])

    # run simulation
    [out_dist, out_vx, out_vy, out_x, out_y] = launch(gravity, mass, velocity, angle, height,
                                                      air_resistance, time_step)
    # Write distance travelled to output csv file
    # check if output_dir is exists
    if not path.exists(output_dir):
        makedirs(output_dir)
    output_file = path.join(output_dir, "py_output.txt")

    with open(output_file, "w") as f:
        f.write("Dist,lastvx,lastvy\n")
        f.write("%f,%f,%f" % (out_dist, out_vx, out_vy))

    with open(output_file, "r") as f:
        print(f.read())

    # plotting the results
    output_png = path.join(output_dir, "py_output.png")
    fig = plt.figure()
    ax = plt.axes()
    ax.plot(out_x, out_y)
    plt.savefig(output_png, dpi=400)
import org.apache.commons.cli.*;
import java.io.*;
import java.util.*;
import java.lang.Math;
//import java.io.File;
public class cannonsim
{
    public static String input_dir = "input_files";
    public static String output_dir = "output_files";

    // input simulation parameters
    public static double gravity, mass, velocity, angle, height, air_resistance, time_step;
    // output simulation results
    public static double out_dist, out_vx, out_vy;

    public static void parse_args(String[] args)
    {
        Options options = new Options();
        Option input_option = Option.builder("i")
                              .longOpt( "input_dir" )
                              .hasArg()
                              .required(false)
                              .build();
        options.addOption( input_option );

        Option output_option = Option.builder("o")
                               .longOpt( "output_dir" )
                               .hasArg()
                               .required(false)
                               .build();
        options.addOption( output_option );

        // # Instantiate the parser
        CommandLineParser parser = new DefaultParser();

        try {

            CommandLine line = parser.parse( options, args );
            if (line.hasOption( "input_dir" )) {
                input_dir = line.getOptionValue("input_dir");
            }
            if (line.hasOption( "output_dir" )) {
                output_dir = line.getOptionValue("output_dir");
            }
        } catch (ParseException exp) {
            System.out.println(exp.getMessage());
            System.exit(1);
        }
    }
    public static void read_simsetting()
    {
        String simsetting_file = new File(input_dir, "simsetting.txt").toString();

        try {
            BufferedReader reader = new BufferedReader(new FileReader(simsetting_file));
            String line = reader.readLine();
            while (line != null) {
                String data[] = line.replace("\"", "").split(",");
                String name = data[0];
                double value = Double.parseDouble(data[1]);

                if (name.equalsIgnoreCase("gravity")) {
                    gravity = value;
                } else if (name.equalsIgnoreCase("mass")) {
                    mass = value;
                } else if (name.equalsIgnoreCase("velocity")) {
                    velocity = value;
                } else if (name.equalsIgnoreCase("angle")) {
                    angle = value;
                } else if (name.equalsIgnoreCase("height")) {
                    height = value;
                } else if (name.equalsIgnoreCase("air_resistance")) {
                    air_resistance = value;
                } else if (name.equalsIgnoreCase("time_step")) {
                    time_step = value;
                }


                line = reader.readLine();
            }
            reader.close();
        } catch (FileNotFoundException ex) {
            System.out.println(ex);
        } catch (IOException ex) {
            System.out.println(ex);
        }


    }
    public static void launch()
    {
        double x = 0.0;
        double y = height;
        double vx = velocity * Math.sin(angle);
        double vy = velocity * Math.cos(angle);
        while (y > 0.0) {
            vy -= gravity * mass * time_step;
            vx -= vx * air_resistance * time_step;
            vy -= vy * air_resistance * time_step;
            x += vx * time_step;
            y += vy * time_step;

        }

        out_dist = x;
        out_vx = vx;
        out_vy = vy;
    }

    public static void write_output_results()
    {
        // create output folder if it does not exist
        File fp = new File(output_dir, "java_output.txt");
        fp.getParentFile().mkdirs();
        //FileWriter fr = new FileWriter(fp);
        try {
            FileWriter fr = new FileWriter(fp);
            fr.write("Dist,lastvx,lastvy\n");
            fr.write(String.format("%.6f", out_dist) + "," +
                     String.format("%.6f", out_vx) + "," +
                     String.format("%.6f", out_vy) + "\n");
            fr.close();

        } catch (IOException e) {
            System.err.print("Something went wrong");
        }



    }
    public static void main(String[] args) throws Exception
    {
        // parse input arguments
        parse_args(args);

        // read input parameters from simsetting.csv
        read_simsetting();

        // run simulation
        launch();

        // save output results
        write_output_results();

    }

}
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <strings.h>
#include <math.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>

double gravity, mass, velocity, angle, height, air_resistance, time_step;

static void usage(const char *argv0)
{
    fprintf(stderr, "Usage: %s [-i input dir path][-o output dir path]\n", argv0);
    exit(EXIT_FAILURE);
}
void launch(double* out_dist, double*  out_vx, double*  out_vy)
{
    double x = 0.0;
    double y = height;
    double vx = velocity * sin(angle);
    double vy = velocity * cos(angle);
    while (y > 0.0)
    {
        vy -= gravity * mass * time_step;
        vx -= vx * air_resistance * time_step;
        vy -= vy * air_resistance * time_step;
        x += vx * time_step;
        y += vy * time_step;

    }
    *out_dist = x;
    *out_vx = vx;
    *out_vy = vy;

}
int main(int argc, char **argv)
{

    char const *input_dir = "input_files";
    char const *output_dir = "output_files";

    // parse input arguments
    int opt;
    while ((opt = getopt(argc, argv, "i:o:")) != -1)
    {
        switch (opt)
        {
        case 'i':
            input_dir = optarg;
            break;
        case 'o':
            output_dir = optarg;
            break;
        default:
            usage(argv[0]);
        }
    }

    // read input parameters from simsetting.csv
    char simsetting_file[1024];
    sprintf (simsetting_file, "%s/%s", input_dir, "simsetting.txt");
    FILE *fp;
    char line[256];
    fp = fopen(simsetting_file, "r");
    if (fp == NULL)
    {
        fprintf(stderr, "Error reading file\n");
        return 1;
    }
    int i = 0;
    char name[64];
    double value;

    while (fscanf(fp, "\"%64[^\"]\",%lf", name, &value) == 2)
    {
        if (strcasecmp(name, "gravity") == 0)
            gravity = value;
        else if (strcasecmp(name, "mass") == 0)
            mass = value;
        else if (strcasecmp(name, "mass") == 0)
            mass = value;
        else if (strcasecmp(name, "velocity") == 0)
            velocity = value;
        else if (strcasecmp(name, "angle") == 0)
            angle = value;
        else if (strcasecmp(name, "height") == 0)
            height = value;
        else if (strcasecmp(name, "air_resistance") == 0)
            air_resistance = value;
        else if (strcasecmp(name, "time_step") == 0)
            time_step = value;
        else
        {
            fprintf(stderr, "Error : Input args %s in simsetting.txt file is not valid !\n", name);
            return 1;
        }

        while ((fgetc(fp) != '\n') && (!feof(fp)))   { /* do nothing */ }
    }
    fclose(fp);

    // run simulation
    double out_dist, out_vx, out_vy;
    launch(&out_dist, &out_vx, &out_vy);

    // check if output_dir is exists
    struct stat st = {0};
    if (stat(output_dir, &st) == -1)
    {
        mkdir(output_dir, 0700);
    }
    // Write distance travelled to output csv file
    char output_file[1024];
    sprintf (output_file, "%s/%s", output_dir, "c_output.txt");
    fp = fopen(output_file, "w");
    fprintf(fp, "Dist,lastvx,lastvy\n");
    fprintf(fp, "%lf,%lf,%lf", out_dist, out_vx, out_vy);
    fclose(fp);
}

New plugin preparation

To create a new plugin for cannon_app application, you need to follow a files and folders structure to be used by FabSim3. To do that, follow these steps:

  1. Create a folder, namely FabCannonsim under plugins directory in your local FabSim3 folder.

  2. Create two sub-folders, config_files where we put the application there, and templates where all templates files are placed.

  3. Clone or download the cannon_app application in the FabCannonsim/config_files folder that just created.

    cd FabCannonsim/config_files
    git clone https://github.com/arabnejad/cannon_app.git
    

  4. In FabCannonsim folder, create two empty files:

    • FabCannonsim.py, which contains the plugin source code.
    • machines_FabCannonsim_user.yml, which contains the plugin environmental variables.
  5. Add a new plugin entry in plugins.yml file located in FabSim3/deploy folder.

    ...
    FabCannonsim:
        repository: <empty>
    

    Note

    For now, we left repository with empty value. Later, this can be filled by the github repo address of your plugin.

To summarize this part, by following above steps, the file and directory should be as shown as in figure below:

  • (a) demonstrates the directory tree structures
  • (b) show the updated plugins.yml file located in FabSim3/deploy folder

Attention

Please note that, the folders name highlighted with red color in (a) will be used by FabSim3 for job configuration and execution and should not be changed. Also, all three (1) the plugin name, here FabCannonsim, (2) the plugin fabric python file, here FabCannonsim.py, and (3) the plugin entry in plugins.yml file , should be identical.

Write application-specific functionalities

To call and execute a function from command line, it should be tagged as a Fabric task class. This part of tutorial explains how to write a function/task to execute a single or ensemble jobs of your application.

  1. Single single job execution

    Code below shows a sample function for single job execution in FabCannonsim.py.

    from base.fab import *
    # Add local script and template path for FabSim3
    add_local_paths("FabCannonsim")
    
    
    @task
    @load_plugin_env_vars("FabCannonsim")
    def Cannonsim(app, **args):
        """
        Submit a single job of Cannon_app
    
            >_ fabsim <remote_machine> Cannonsim:cannon_app
            >_ fabsim localhost Cannonsim:cannon_app
        """
        update_environment(args)
        with_config(app)
        execute(put_configs, app)
        env.script = "cannonsim"
        job(args)
    

    The following paragraphs will explain it line by line:

    • from base.fab import *

      loads all FabSim3 pre-defined functions.

    • add_local_paths("FabCannonsim")

      sets the default location for templates, python scripts, and config files.

    • @task

      Marks the function, as a callable task, to be executed when it invoked by fabsim from command line.

    • @load_plugin_env_vars("FabCannonsim")

      Loads all machine-specific configuration information that are specified by the user for the input plugin name.

      The below code shows the sample machine-specific configuration yaml file for the cannon_app application.

      default:
        # require command for compile and execute C code version
        c_app_run_prefix: "gcc cannonsim.cpp -o cannonsim -lm"
        c_app_run_command: "./cannonsim"
        # require command for compile and execute python code version
        py_app_run_command: "python cannonsim.py"
        # require command for compile and execute JAVA code version
        java_app_run_prefix: "export CLASSPATH='java_libs/commons-cli-1.3.1.jar:.'"
        java_app_compile_command: "javac cannonsim.java"
        java_app_run_command: "java cannonsim"
      
    • def Cannonsim(app, **args)

      Defines the task name. The defined task name can be called from command line alongside fabsim command, e.g.,

      >_ fabsim <remote/local machine> Cannonsim:<input parameters>
      

    • update_environment(args)

      is predefined FabSim3 function which updated the environmental variables that are used as a combination settings registry and shared inter-task data namespace. The complete list of FabSim3 environmental variables can be found in machines.yml and machines_user.yml located in FabSim3/deploy folder.

    • with config(args)

      augments the FabSim3 environment variable, such as the remote location variables where the config files for the job should be found or stored, with information regarding a particular configuration name.

    • execute(put_configs, app)

      transfers the config files to the remote machine to be used in launching jobs.

    • env.script = "cannonsim"

      the env.script variable contains the name of template script file to be used for execution of job on the target machine, which can be local host or HPC resources. This script will be called when the job execution starts, and contains all steps, such as set environment variable, or commands line to call/execute the application.

      The below code shows the script file, namely cannonsim, for the *cannon_app` application.

      # change directory to where application is stored
      cd $job_results
      $run_prefix
      
      OUTPUT_DIR="output_files"
      INPUT_DIR="input_files"
      
      # run c program
      $c_app_run_prefix
      $c_app_run_command
      
      # run python program
      $py_app_run_command
      
      # run java program
      $java_app_run_prefix
      $java_app_compile_command
      $java_app_run_command
      
      # show output results
      echo -e "\n\nOutput results for python program :"
      cat $$OUTPUT_DIR/py_output.txt
      
      echo -e "\n\nOutput results for C program :"
      cat $$OUTPUT_DIR/c_output.txt
      
      echo -e "\n\nOutput results for Java program :"
      cat $$OUTPUT_DIR/java_output.txt
      

      Tip

      By default, FabSim3 loads all required scripts from templates folder located in plugin directory. Hence the cannonsim file should be saved in FabSim3/plugins/FabCannonsim/templates directory to be used by fabsim command for cannon_app execution.

      FabSim3 uses a template/variable substitution system to easily generate the required script for executing job on the target local/remote machine. The used system is $-based substitutions, where $var will replaced by the actual value of the variable var, and $$ is an escape and is replaced with a single $.

      To demonstrate, you can see the generated sample script for executing the cannon_app application on the localhost below:

      # change directory to where application is stored
      cd ~/FabSim3/results/cannon_app_localhost_1
      /bin/true || true
      
      OUTPUT_DIR="output_files"
      INPUT_DIR="input_files"
      
      # run c program
      gcc cannonsim.cpp -o cannonsim -lm
      ./cannonsim
      
      # run python program
      python cannonsim.py
      
      # run java program
      export CLASSPATH='java_libs/commons-cli-1.3.1.jar:.'
      javac cannonsim.java
      java cannonsim
      
      # show output results
      echo "Output results for python program :"
      cat $OUTPUT_DIR/py_output.txt
      
      echo "Output results for C program :"
      cat $OUTPUT_DIR/c_output.txt
      
      echo "Output results for Java program :"
      cat $OUTPUT_DIR/java_output.txt 
      
    • job(args)

      is an internal low level job launcher defined in FabSim3.

    To submit and execute a single cannon_app job,

    fabsim localhost Cannonsim:cannon_app
    

  2. Ensemble job execution

    Code below shows a sample function for an ensemble job execution in FabCannonsim.py.

    from base.fab import *
    # Add local script and template path for FabSim3
    add_local_paths("FabCannonsim")
    
    
    @task
    @load_plugin_env_vars("FabCannonsim")
    def Cannonsim(app, **args):
        ...
        ...
        ...
    
    
    @task
    @load_plugin_env_vars("FabCannonsim")
    def Cannonsim_ensemble(app, **args):
        """
        Submit an ensemble of canon_app jobs
            >_ fabsim <remote_machine> Cannonsim_ensemble:cannon_app
            >_ fabsim localhost Cannonsim_ensemble:cannon_app
        """
        update_environment(args)
        sweep_dir = find_config_file_path(app) + "/SWEEP"
        env.script = "cannonsim"
        run_ensemble(app, sweep_dir, **args)
    

How to use fabsim API commands form python code

Within FabSim3, in addition to fabsim command APIs, we also provide this functionality to call fabsim command from a python code.

Advanced Plugins Examples

For advanced examples, see the plugins available in fabsim/deploy/plugins.yml file. Both FabFlee, and FabMD are particularly good examples to investigate.