submit takes a user command and executes it remotely. The objective is to allow the user to issue a command in the same manner as a locally executed command. Multiple submission mechanisms are available for run dissemination. A set of steps are executed for each run submission:

• Destination site is selected
• A wrapper script is generated for remote execution
• If needed a batch system description file is generated.
• Input files for a run are gathered and transferred to the remote site. Transferred files include the wrapper and batch description scripts.
The wrapper script is executed remotely.
• Progress of the remote run is monitored until completion.
• Output files from the run are returned to the dissemination point.

submit command options can be determined by using the help parameter of the submit command.

$ submit --help
Usage: submit [options]

Options:
  -h, --help            Report command usage. Optionally request listing of
                        managers, tools, or venues.
  -l, --local           Execute command locally
  -v, --venue           Remote job destination
  -i, --inputfile       Input file
  -p, --parameters      Parameter sweep variables. See examples.
  -d, --data            Parametric variable data - csv format
  -s SEPARATOR, --separator=SEPARATOR
                        Parameter sweep variable list separator
  -n NCPUS, --nCpus=NCPUS
                        Number of processors for MPI execution
  -N PPN, --ppn=PPN     Number of processors/node for MPI execution
  -w WALLTIME, --wallTime=WALLTIME
                        Estimated walltime hh:mm:ss or minutes
  -e, --env             Variable=value
  -m, --manager         Multiprocessor job manager
  -r NREDUNDANT, --redundancy=NREDUNDANT
                        Number of indentical simulations to execute in
                        parallel
  -M, --metrics         Report resource usage on exit
  -W, --wait            Wait for reduced job load before submission
  -Q, --quota           Enforce local user quota on remote execution host
  -q, --noquota         Do not enforce local user quota on remote execution
                        host

Parameter examples:

submit -p @@cap=10pf,100pf,1uf sim.exe @:indeck

        Submit 3 jobs. The @:indeck means "use the file indeck as a template
file." Substitute the values 10pf, 100pf, and 1uf in place of @@cap within the
file. Send off one job for each of the values and bring back the results.

submit -p @@vth=0:0.2:5 -p @@cap=10pf,100pf,1uf sim.exe @:indeck

        Submit 78 jobs. The parameter @@vth goes from 0 to 5 in steps of 0.2,
so there are 26 values for @@vth. For each of those values, the parameter
@@cap changes from 10pf to 100pf to 1uf. 26 x 3 = 78 jobs total. Again
@:indeck is treated as a template, and the values are substituted in place of
@@vth and @@cap in that file.

submit -p params sim.exe @:indeck

        In this case, parameter definitions are taken from the file named
params instead of the command line. The file might have the following
contents:

                # paramters for my job submission
                parameter @@vth=0:0.2:5
                parameter @@cap = 10pf,100pf,1uf

submit -p "params;@@num=1-10;@@color=blue" job.sh @:job.data

        For someone who loves syntax and complexity... The semicolon separates
the parameters value into three parts. The first says to load parameters from
a file params. The next part says add an additional parameter @@num that goes
from 1 to 10. The last part says add an additional parameter @@color with a
single value blue. The parameters @@num and @@color cannot override anything
defined within params; they must be new parameter names.

submit -d input.csv sim.exe @:indeck

        Takes parameters from the data file input.csv, which must be in comma-
separated value format. The first line of this file may contain a series of
@@param names for each of the columns. If it doesn't, then the columns are
assumed to be called @@1, @@2, @@3, etc. Each of the remaining lines
represents a set of parameter values for one job; if there are 100 such lines,
there will be 100 jobs. For example, the file input.csv might look like this:

                @@vth, @@cap
                1.1, 1pf
                2.2, 1pf
                1.1, 10pf
                2.2, 10pf

        Parameters are substituted as before into template files such as
@:indeck.

submit -d input.csv -p "@@doping=1e15-1e17 in 30 log" sim.exe @:infile

        Takes parameters from the data file input.csv, but also adds another
parameter @@doping which goes from 1e15 to 1e17 in 30 points on a log scale.
For each of these points, all values in the data file will be executed. If the
data file specifies 50 jobs, then this command would run 30 x 50 = 1500 jobs.

submit -d input.csv -i @:extra/data.txt sim.exe @:indeck

        In addition to the template indeck file, send along another file
extra/data.txt with each job, and treat it as a template too.

submit -s / -p @@address=23 Main St.,Hometown,Indiana/42
Broadway,Hometown,Indiana -s , -p @@color=red,green,blue job.sh @:job.data

        Change the separator to slash when defining the addresses, then change
back to comma for the @@color parameter and any remaining arguments. We
shouldn't have to change the separator often, but it might come in handy if
the value strings themselves have commas.

submit @@num=1:1000 sim.exe input@@num

        Submit jobs 1,2,3,...,1000. Parameter names such as @@num are
recognized not only in template files, but also for arguments on the command
line. In this case, the numbers 1,2,3,...,1000 are substituted into the file
name, so the various jobs take their input from "input1", "input2", ...,
"input1000".

submit @@file=glob:indeck* sim.exe @:file

        Look for files matching indeck* and use the list of names as the
parameter @@file. Those values could be substituted into other template files,
or used on the command line as in this example. Suppose the directory contains
files indeckA, indeckB, and indeck-123. This example would launch three jobs
using each of those files as input for the job.

Additional information is available by requesting user specific lists of choices for some command options. The available option lists are generated for a user based on configured restrictions and availability. The values listed here are for example only and may not be available on all HUBs.

$ submit --help tools

Currently available TOOLs are:
   pegasus-plan

$ submit --help venues

Currently available VENUES are:
   DiaGrid
   WF-DiaGrid

$ submit --help managers

Currently available MANAGERs are:
   mpi
   mpich
   parallel

By specifying a suitable set of command line parameters it is possible to execute commands on configured remote systems. The simple premise is that a typical command line can be prefaced by submit and its arguments to execute the command remotely.

$ submit -v clusterA echo Hello world!
Hello world!

In this example the echo command is executed on the venue named clusterA where runs are executed directly on the host. Execution of the same command on a cluster using PBS would be done in a similar fashion

$ submit -v clusterB echo Hello world!
(2586337) Simulation Queued Wed Oct  7 14:45:21 2009
(2586337) Simulation Done Wed Oct  7 14:54:36 2009
$ cat 00577296.stdout
Hello world!

submit supports an extensible variety of submission mechanisms. HUBzero supported submission mechanisms are

• local - use batch submission mechanisms available directly on the submit host. These include PBS, condor, and Pegasus batch queue submission.
• ssh - direct use of ssh. Submit manages access to a common ssh key, essentially serving as a proxy for the HUB user.
• ssh + remote batch submission - use ssh to do batch run submission remotely. Again methods for PBS, condor, and Pegasus batch queue submission are provided.

In addition to single site submission the -r/--redundancy option provides the option to simultaneously submit runs to multiple remote venues. In such cases the successful completion of a run at one venue cancels runs at all other venues. If none of the runs are successful results from one of the runs are returned to the user. Redundant submission is not allowed when performing parametric sweeps.

A site for remote execution is selected in one of the following ways, listed in order of precedence:

• Execute the command within the user tool session, -l/--local option
• User specified on the command line with -v/--venue option.
• Randomly selected from remote sites associated pre-staged application.
• Select randomly from all configured sites

Any files specified by the user plus internally generated scripts are packed into a tarball for delivery to the remote site. Individual files or entire directory trees may be listed as command inputs using the -i/--inputfile option. Additionally command arguments that exist as files or directories will be packed into the tarball. If using ssh based submission mechanisms the tarball is transferred using scp.

The job wrapper script is executed remotely either directly or submitted to a batch queue. The job is subject to all remote queuing restrictions and idiosyncrasies.

Remote batch jobs are monitored for progress. Methods appropriate to the batch queuing system are used to check job status at a configurable frequency. A typical frequency is on the order one minute. Job status changes are reported to the user. The maximum time between reports to the user is set on the order of five minutes even in the absence of change. The job status is used to detect job completion.

The same methods used to transfer input files are applied in reverse to retrieve output files. Any files and directories created or modified by the application are be retrieved. A tarball is retrieved and expanded to the home base directory. It is up to the user to avoid the overwriting of files.

In addition to the application generated output files additional files are generated in the course of remote run execution. Some of these files are for internal bookkeeping and are consumed by submit, a few files however remain in the home base directory. The remaining files include RUNID.stdout and RUNID.stderr, it is also possible that a second set of standard output/error files will exist containing the output from the batch job submission script. RUNID represents unique job identifier assigned by submit.