QCG Pilot Job installation on remote machines¶
QCG Pilot Job is a lightweight implementation of the Pilot Job mechanism which can be used to run a very large number of jobs efficiently on remote clusters and supercomputers.
More information on QCG-PilotJob can be found on their ReadTheDocs site.
Example of a QCG-Pilotjob container, which dynamically facilitates a diverse set of code executions within a single queuing system job. Source
Here we present how you can install QCG Pilot Job on remote machines so that you can use it with FabSim3.
Note that FabSim3 can also work with a pre-installed version of QCG-PilotJob; but for those who need to set it up manually this document is meant to provide some help.
The pre-requisites for installing QCG Pilot Job on a remote machine using FabSim, we essentially need
FabSim installed on the local machine
Access to the remote machine
Additionally, OpenBLAS and some related packages are also required. These can be installed using:
sudo apt install libopenblas-dev pkg-config libopenblas64-dev
For the installation itself, install the following command from the terminal
fabsim <remote machine name> install_app:QCG-PilotJob,venv=True
Once the command is issued, the following steps will occur in sequence:
Miniconda will be installed locally.
All the dependencies required for installation of QCG Pilot Job will be installed locally
Prepares the script that when run on the remote machine will install QCG Pilot Job
Transfers all the dependencies to the remote machine
Submits the installation job script to the remote machine
Since the number of dependencies for QCG Pilot Job is quite large, therefore, the command may take up to 45 minutes to run.
After all the above-mentioned processes are done, the status of the submitted job on the remote machine can be checked using
fabsim <remote machine name> stat
Once, the sumbitted job completes successfully, QCG Pilot Job will be installed on the remote machine.