Pegasus workflow submission

submit can run a workflow through Pegasus. Two cases are supported: workflows submit generates for you from a parameter sweep, and workflows you build yourself. In both, submit configures access to the computational resources, so you do not have to supply a site catalog.

Parameter sweeps

The -p/--parameters and -d/--data options describe a sweep compactly, so you do not have to generate the whole set of input files and command lines yourself. You declare substitutable parameters on the submit command line, and submit substitutes their values into data files and command arguments, covering every combination.

Each combination is a node in a workflow and a job on the chosen resource. A curses interface shows the run's progress. The syntax is described under parameter sweeps.

Workflows you build yourself

A sweep is a workflow of independent nodes: no data passes between them. When that is not enough, build the workflow yourself and wrap an application around it, letting the user supply the values that matter.

Generate the abstract workflow description with one of the Pegasus APIs, then hand the resulting file to submit:

submit pegasus-plan --dax daxFile

Where more than one venue can run Pegasus workflows, name the one you want; otherwise submit picks one at random.

submit -v venueName pegasus-plan --dax daxFile

submit --help venues lists the venues you may use. submit supplies several other pegasus-plan options itself, and reserves the right to ignore options you pass silently.

A workflow can also run in the tool session itself, with -l/--local:

submit --local pegasus-plan --dax daxFile

The hub's use command puts pegasus-plan and the other Pegasus commands on your PATH, and sets the other environment variables the Python and Java workflow APIs need.