User jobs are executed through the Dispatcher process on the EnFuzion root. This chapter describes the Dispatcher, the basic steps of run execution, which include run submission, execution monitoring and retrieval of run results, and accounting reports.
The Dispatcher executes on the EnFuzion root system. It can be used either to process a single run as a command line utility or multiple runs as a service on the network. If the Dispatcher is used as a service, submit computers are normally local user machines and are different from the EnFuzion root system.
When a run is submitted for execution, it is assigned an owner user ID by the Dispatcher. In most cases, this user assignment is done transparently to the EnFuzion user. An exception is the web browser interface, which requires an explicit user login. Otherwise, a generic anonymous user ID is assigned as the run owner. User identity can also be copied to other systems or accounts.
Each of the execution steps, which is either a run submission, monitoring or results retrieval, can be performed via a web browser or from a command line. The web browser capability is provided by an EnFuzion process on the root system, called the EnFuzion Eye. Transparently to the user, the Eye communicates with the Dispatcher and produces the pages for the web browser. The run execution steps are performed from a command line with programs enfsub and enfcmd. The programs can be executed directly by the user or used in scripts. The Dispatcher also provides a network based application programming interface, which can be used by custom programs to communicate directly with with the Dispatcher.
If the Dispatcher is terminated for any reason, or if the EnFuzion root system crashes unexpectedly, it is possible that some jobs might not have been completed, while other jobs may still be waiting for execution. Although the Dispatcher is able to automatically restart unfinished runs without jobs that already completed, EnFuzion provides a separate enfpurge utility. Enfpurge removes completed jobs from a run file, so that uncompleted jobs can be resubmitted.
The Dispatcher records accounting information, which provides details on how cluster resources are being used. The accounting information can be used to produce reports, which contain either run information, showing run use of node computers, or node information, showing node utilization.
The sections below provide details on the Dispatcher, run submission, including user assignment, execution monitoring, retrieval of results, the enfpurge program and on accounting reports.
The Dispatcher is the main program running on the EnFuzion root system, controlling job execution and other EnFuzion processes. It can be used to process a single run as a command line utility or multiple runs as a service on the network. The sections below provide details about Dispatcher parameters and details on using the Dispatcher for both single and multiple runs.
The Dispatcher command line is:
enfdispatcher [ options ] [ <run_file> ]The Dispatcher reads its options and takes an optional run file. The optional run file is useful to provide the run description in a command line, when the Dispatcher is executed in a single run mode.
The Dispatcher command line options are:
-help
If this is the first option, then the Dispatcher prints out a help notice and exits. If it is not the first option, then -help has no effect.
-d
The Dispatcher is placed in a daemon mode . On Linux/Unix systems, the Dispatcher performs the following steps: forks twice, gets detached from the controlling terminal, becomes a session leader, and closes the standard file descriptors. On Windows, the Dispatcher calls itself with the same options except for the "-d" option. The new process shares the same working directory and inherits handles. The original Dispatcher exits.
-m
The Dispatcher is executed in a multi run mode. By default, the Dispatcher is executed in a single run mode, where it executes one run either specified on a command line or a previously interrupted run and exits. In the multi run mode, the Dispatcher continuously processes runs until it is terminated by the administrator or by the system. The multi run mode is useful to provide EnFuzion as a network service.
-p <port_number>
This option changes the default port number of its network based application programming interface to <port_number>. By default, the Dispatcher uses port 10102. The application programming interface is described in the Section called Application Programming Interface in Chapter 10.
-r
This option recovers uncompleted runs from a previous Dispatcher. If the EnFuzion root system fails or the Dispatcher is terminated, then some of the runs might not be completed. If a new Dispatcher is restarted with the -r option in the same directory as the terminated Dispatcher, then it will reload the uncompleted runs and execute them.
-v
If this is the first parameter, then the Dispatcher prints out its version and exits. If it is not the first parameter, then -v has no effect.
-w <directory>
The Dispatcher sets its working directory to the <directory> path. The working directory contains the Dispatcher log files and other working files.
This option is useful for safely setting the working directory, for example when the Dispatcher is executed using a scripting language or from a Java class.
<run_file>
This specifies the run file to process in a single run mode . Single run mode is suitable for executing the Dispatcher in scripts and from a command line. In single run mode, the Dispatcher takes a run file as input, automatically starts processing the jobs and exits after all the jobs complete. If all the jobs complete successfully, the Dispatcher returns 0 as its exit value. If some of the jobs fail, the Dispatcher returns 1 as its exit value.
In single run mode, nodes are usually provided in the file enfuzion.nodes before the execution starts.
Most of the root options, described in the Section called Specifying Root Configuration Options in Chapter 6, can also be specified on the command line. The command line value takes precedence over the value in the root.options file. The root options that can be specified from the command line are:
-noanonsubmit, which denies run submission by anonymous users. See the Section called Rejecting Anonymous Run Submission in Chapter 6 for details.
-cleanuplimit, which specifies the period to delete the obsolete user directories. See the Section called Deleting Obsolete User Directories in Chapter 6 for details.
-completelogs, which turns on run specific events in the main cluster log. See the Section called Complete Logs in Chapter 6 for details.
-disconnect, which specifies the period that either a root or a node machine waits for a heartbeat signal. See the Section called Disconnect Period in Chapter 6 for details.
-eyeport, which specifies the Eye port number. See the Section called Port Number for the Eye in Chapter 6 for details.
-eyestart, which specifies, if the Eye is automatically started by the Dispatcher. See the Section called Starting the Eye in Chapter 6 for details.
-eyeterminate, which specifies, if the Eye is terminated by the Dispatcher. See the Section called Terminating the Eye in Chapter 6 for details.
-heartbeat, which specifies the interval for heartbeat between the root and the node machines. See the Section called Heartbeat Period in Chapter 6 for details.
-jobport, which specifies the port number that is used by user jobs on EnFuzion nodes to execute services on the root. See the Section called Port Number for Job Execution in Chapter 6 for details.
-logsizelimit, which limits the size of the Dispatcher log for log rotation. See the Section called Maximum Dispatcher Log Size in Chapter 6 for details.
-mailport, which specifies port of the SMTP service host for electronic notification messages. See the Section called Specifying Mail Service Port in Chapter 6 for details.
-mailserver, which specifies the SMTP server host for electronic notification messages. See the Section called Specifying Mail Server System in Chapter 6 for details.
-mailuser, which specifies the sender for electronic notification messages. See the Section called Specifying Mail Sender in Chapter 6 for details.
-maxdatastream, which specifies the maximum size for a datajob. See the Section called Maximum Datastream Job Size in Chapter 6 for details.
-maxstart, which limits the number of concurrent node activations. See the Section called Concurrent Node Activations in Chapter 6 for details.
-privileges, which enforces user privileges. See the Section called Enforcing Privileges in Chapter 6 for details.
-protect, which denies execution of user programs on the root system. See the Section called Prevent Execution of User Programs on the EnFuzion Root System in Chapter 6 for details.
-restart, which specifies the node restart period. See the Section called Node Restart Period in Chapter 6 for details.
-rootport, which specifies the port that is used by nodes to connect to the root when they are started independently. See the Section called Port Number for Node Connections in Chapter 6 for details.
-remoteaccess, which denies remote access to the Dispatcher API port. See the Section called Allowing Remote Access to the Dispatcher Interface in Chapter 6 for details.
-queue, which turns on the queuing policy for scheduling. See the Section called Queueing Policy in Chapter 6 for details.
-startport, which specifies the port that the enfnodestarter program uses to accept node requests during the node start sequence. See the Section called Port Number for Node Starter Connections in Chapter 6 for details.
When the Dispatcher is used in single run mode, a run file must be supplied as an argument. The Dispatcher executes all of the jobs in the run and then exits. In this case, the submit computer and the root computer are the same. Input files and results are provided in the Dispatcher working directory on the submit computer.
When the Dispatcher executes multiple runs as specified with the -m command line option, the root computer is usually different from the submit computers. The Dispatcher is able to execute many runs concurrently, even from multiple users. Users submit their run files and their associated input files to the Dispatcher for execution. The submission is done through a web browser on the submit computer. This process is detailed in the Section called Submission from a Web Browser. EnFuzion also provides a command line program, which can be used to submit runs in scripts or from a command line. This process is detailed in the Section called Submission from a Command Line. Another option for submitting runs is through applications, using the EnFuzion API to communicate directly with the Dispatcher.
The Dispatcher is configured to automatically handle the Eye, which provides a web based interface. The Eye is handled differently, depending on whether the Dispatcher is executed in a single run mode or in a multiple run mode.
If the Dispatcher is executed in a single run mode, then it starts the Eye at the beginning and terminates the Eye at the end of the run execution.
If the Dispatcher is executed in a multiple run mode, it starts the Eye at the beginning, but it does not terminate the Eye at the end. This allows remote users to access the result files even after the Dispatcher is terminated.
If the Eye process detects that another instance of the Eye is already executing on the system, using the same port to listen for requests, it terminates to prevent any conflicts.
Default behavior of the Eye can be changed by modifying the EnFuzion root configuration parameters, described in the Section called Specifying Root Configuration Options in Chapter 6.