The sections below provide an overview of the basic concepts that underlie the execution of jobs by EnFuzion, including software installation and configuration and the handling of runs and jobs.
EnFuzion must be installed on all submit hosts, on the root host and on all participating node hosts . The EnFuzion distribution package includes installation scripts and programs to quickly install EnFuzion on each host type. Since the EnFuzion software components are different for each type, separate installation procedures are provided for submit, root and node hosts. See Chapter 4 and Chapter 3 for details about the installation procedures.
After EnFuzion is installed, it must be configured. As a mandatory configuration step, the submit hosts, the root and the nodes must be configured to be able to find each other and establish a connection.
On submit hosts, the address of the EnFuzion root service must be specified as specified in Chapter 5. Alternatively, if only a web browser is used, the address can be provided explicitly by the user.
Usually, the root is supplied with a list of hosts which are used for the nodes and instructions on how to start and access the nodes. This process is detailed in Chapter 6. An alternative approach, more suitable for dynamic environments where nodes change often, is to configure nodes to contact the root directly and not wait to be started by the root (see the Section called Connecting Node to the EnFuzion Root in Chapter 6).
EnFuzion provides a large number of additional configuration parameters, which are available to fine tune EnFuzion behavior for specific user environments. These configuration parameters are optional, and their default values are suitable for most environments. Root configuration is provided in the root.options file and in files for dealing with EnFuzion users, detailed in Chapter 6. Node configuration is provided in the node.config file, the enfuzion.options file, and the enfuzion.security file. These files are detailed in Chapter 7.
After the EnFuzion submit hosts, the root and nodes are configured, the cluster is ready to execute user jobs. The process of executing jobs consists of several steps:
Describing a run
Submitting the run for execution
Monitoring the execution
Retrieving the results
The sections below provide more details about each of these steps.
A run can be simply a command line program, a script or a parametric execution. In the case of a command line program or a script, there is no need for the user to provide any additional configuration details. A parametric execution contains multiple jobs described with a list of commands to execute, a list of input values for each job and any additional configuration options. EnFuzion provides two different ways to describe parametric executions, either as a plan file or as a run file.
A plan file is a template for the run. It includes descriptions of job parameters, but not their actual values. Plan files are used by EnFuzion to build application specific GUIs, which allow users to quickly generate jobs for parametric executions. A plan file must be converted to a run file, before it can be submitted to the EnFuzion root for execution. Plan files are converted to run files with the EnFuzion Generator program. The Generator is detailed in the Section called Specifying Input Values in Chapter 8. Using the plan file, the Generator creates an application specific GUI, which is used by the user to select input values for job parameters and produce a run file.
Plan files are regular text files. EnFuzion also includes the Preparator program, which provides for a simple creation of plan files. The Preparator is detailed in the Section called The Preparator in Chapter 8. Alternatively, plan files can be created with standard text editors.
The run file includes a description of the run and input values for job parameters. Run files can be submitted directly to the EnFuzion root for execution.
Run files are regular text files. Depending on the application, they can be produced by using the Preparator and the Generator, by using standard text editors or can be generated by other programs.
Runs are usually prepared on a submit computer, which can be a workstation or a personal computer. The Preparator and the Generator are also executed on the submit computer.
Runs are submitted for execution as a command line, as a script or as a run file. The Dispatcher, which is the controlling EnFuzion process on the root, can be used in a single run mode or in a multiple run mode.
The single run mode is most commonly used interactively. The Dispatcher takes a single run, executes all of the jobs in the run and then exits. In this case, the submit computer and the root computer are the same. Input files and results are provided in the Dispatcher working directory on the submit computer.
The multiple run mode is used to provide EnFuzion as a service on the network. The root computer is usually different from the submit computers, although that is not a requirement. The Dispatcher is able to execute many runs concurrently, even from multiple users. Users submit their run files and their associated input files to the Dispatcher for execution. The submission is done through a web browser on the submit computer (see the Section called Graphical Web Based Interface in Chapter 10) or from a command line (see the Section called The enfsub Program in Chapter 10). Another option for submitting runs is from applications, using the EnFuzion network API for direct communication with the Dispatcher.
Runs can be monitored with a web browser by connecting to the EnFuzion Eye. The Eye is an intermediary between the user and the Dispatcher. It takes user commands, communicates with the Dispatcher, and generates HTML pages for the browser. The Eye provides monitoring information, such as progress on the run execution and active nodes. This process is detailed in the Section called Graphical Web Based Interface in Chapter 10.
EnFuzion also provides the command line program, Enfcmd, which can be used to monitor run execution and the Dispatcher, either from scripts or from a command line. See the Section called The Enfcmd Program in Chapter 10 for more details.
EnFuzion also exports a complete monitoring API, which can be used by other applications to communicate directly with the Dispatcher. Using the API is detailed in the Section called Application Programming Interface in Chapter 10.
Another method to monitor the Dispatcher is provided through its log . The Dispatcher maintains an extensive log in the file enfuzion.log, which is stored in a file in its working directory. This log file is detailed in the Section called The enfuzion.log File in Chapter 9. If the Dispatcher is executed in a single run mode, the log is also printed on the screen. In addition, each run maintains its own log in its corresponding directory. A run log contains only events that are relevant for that specific run.
The run results are stored in the run subdirectory of the working directory of the Dispatcher on the root computer.
If the Dispatcher is executed in a single run mode, then this directory is on the local computer. The user can access them directly on the local computer.
If the Dispatcher is executed in multiple run mode, and the root computer is different from the submit computers, then the result files are stored on a remote computer. One option for accessing result files on a remote root computer is to place them on a file system that is shared between the root computer and the submit computers.
If there are no shared file systems between the root computer and the submit computers, then result files can be copied to submit computers by using a web browser or the Enfsub and Enfcmd programs on the submit computer.
Another alternative to access files on the root computer is to use system provided applications, such as ftp, scp and similar.