Documentation
¶
Overview ¶
Package hpc submits and tracks jobs through HPC scheduler commands.
Index ¶
- func ValidateSubmissionTemplate(field, text string) error
- type Backend
- func (b *Backend) BackendParameterKeys() []string
- func (b *Backend) Cancel(ctx context.Context, taskID string) error
- func (b *Backend) CheckBackendParameterSupport(task *tes.Task) error
- func (b *Backend) Close()
- func (b *Backend) ConfigureBackendParameters(patterns map[string]string) error
- func (b *Backend) Reconcile(ctx context.Context)
- func (b *Backend) Submit(ctx context.Context, task *tes.Task) error
- func (b *Backend) WriteEvent(ctx context.Context, ev *events.Event) error
- type TaskState
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ValidateSubmissionTemplate ¶
ValidateSubmissionTemplate checks the syntax before accepting tasks.
Types ¶
type Backend ¶
type Backend struct {
Name string
SubmitCmd string
CancelCmd string
Template string
Conf *config.Config
Event events.Writer
Database taskpolicy.ReadOnlyServer
// ExtractID is responsible for extracting the task id from the response
// returned by the SubmitCmd.
ExtractID func(string) string
// MapStates takes a list of backend specific ids and calls out to the backend
// via (squeue, qstat, condor_q, etc) to get that tasks current state. These states
// are mapped to TES states along with an optional reason for this mapping.
// The Reconcile function can then use the response to update the task states
// and system logs to report errors reported by the backend.
MapStates func([]string) ([]*TaskState, error)
ReconcileRate time.Duration
Log *logger.Logger
compute.Backend
// contains filtered or unexported fields
}
Backend submits and reconciles jobs through an HPC scheduler.
func (*Backend) BackendParameterKeys ¶
BackendParameterKeys reports the administrator-configured template keys.
func (*Backend) CheckBackendParameterSupport ¶
CheckBackendParameterSupport validates parameters before task persistence.
func (*Backend) ConfigureBackendParameters ¶
ConfigureBackendParameters compiles the administrator-defined allowlist.
func (*Backend) Reconcile ¶
Reconcile loops through tasks and checks the status from Funnel's database against the status reported by the backend (slurm, htcondor, grid engine, etc). This allows the backend to report system error's that prevented the worker process from running.
Currently this handles a narrow set of cases:
|---------------------|-----------------|--------------------| | Funnel State | Backend State | Reconciled State | |---------------------|-----------------|--------------------| | QUEUED | FAILED | SYSTEM_ERROR | | QUEUED | QUEUED/PENDING* | SYSTEM_ERROR | | INITIALIZING | FAILED | SYSTEM_ERROR | | RUNNING | FAILED | SYSTEM_ERROR |
In this context a "FAILED" state is being used as a generic term that captures one or more terminal states for the backend.
*QUEUED/PENDING: this captures the case where the scheduler has a task that is stuck in the queued state because the resource request that can never be fulfilled.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package gridengine contains code for accessing compute resources via Open Grid Engine.
|
Package gridengine contains code for accessing compute resources via Open Grid Engine. |
|
Package htcondor contains code for accessing compute resources via HTCondor.
|
Package htcondor contains code for accessing compute resources via HTCondor. |
|
Package pbs contains code for accessing compute resources via PBS/Torque.
|
Package pbs contains code for accessing compute resources via PBS/Torque. |
|
Package slurm contains code for accessing compute resources via Slurm.
|
Package slurm contains code for accessing compute resources via Slurm. |