hpc

package
v0.12.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 9, 2026 License: MIT Imports: 24 Imported by: 0

Documentation

Overview

Package hpc submits and tracks jobs through HPC scheduler commands.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ValidateSubmissionTemplate

func ValidateSubmissionTemplate(field, text string) error

ValidateSubmissionTemplate checks the syntax before accepting tasks.

Types

type Backend

type Backend struct {
	Name      string
	SubmitCmd string
	CancelCmd string
	Template  string
	Conf      *config.Config
	Event     events.Writer
	Database  taskpolicy.ReadOnlyServer
	// ExtractID is responsible for extracting the task id from the response
	// returned by the SubmitCmd.
	ExtractID func(string) string
	// MapStates takes a list of backend specific ids and calls out to the backend
	// via (squeue, qstat, condor_q, etc) to get that tasks current state. These states
	// are mapped to TES states along with an optional reason for this mapping.
	// The Reconcile function can then use the response to update the task states
	// and system logs to report errors reported by the backend.
	MapStates     func([]string) ([]*TaskState, error)
	ReconcileRate time.Duration
	Log           *logger.Logger
	compute.Backend
	// contains filtered or unexported fields
}

Backend submits and reconciles jobs through an HPC scheduler.

func (*Backend) BackendParameterKeys

func (b *Backend) BackendParameterKeys() []string

BackendParameterKeys reports the administrator-configured template keys.

func (*Backend) Cancel

func (b *Backend) Cancel(ctx context.Context, taskID string) error

Cancel cancels a task via "qdel", "condor_rm", "scancel", etc.

func (*Backend) CheckBackendParameterSupport

func (b *Backend) CheckBackendParameterSupport(task *tes.Task) error

CheckBackendParameterSupport validates parameters before task persistence.

func (*Backend) Close

func (b *Backend) Close()

func (*Backend) ConfigureBackendParameters

func (b *Backend) ConfigureBackendParameters(patterns map[string]string) error

ConfigureBackendParameters compiles the administrator-defined allowlist.

func (*Backend) Reconcile

func (b *Backend) Reconcile(ctx context.Context)

Reconcile loops through tasks and checks the status from Funnel's database against the status reported by the backend (slurm, htcondor, grid engine, etc). This allows the backend to report system error's that prevented the worker process from running.

Currently this handles a narrow set of cases:

|---------------------|-----------------|--------------------| | Funnel State | Backend State | Reconciled State | |---------------------|-----------------|--------------------| | QUEUED | FAILED | SYSTEM_ERROR | | QUEUED | QUEUED/PENDING* | SYSTEM_ERROR | | INITIALIZING | FAILED | SYSTEM_ERROR | | RUNNING | FAILED | SYSTEM_ERROR |

In this context a "FAILED" state is being used as a generic term that captures one or more terminal states for the backend.

*QUEUED/PENDING: this captures the case where the scheduler has a task that is stuck in the queued state because the resource request that can never be fulfilled.

func (*Backend) Submit

func (b *Backend) Submit(ctx context.Context, task *tes.Task) error

Submit submits a task via "qsub", "condor_submit", "sbatch", etc.

func (*Backend) WriteEvent

func (b *Backend) WriteEvent(ctx context.Context, ev *events.Event) error

WriteEvent writes an event to the compute backend. Currently, only TASK_CREATED is handled, which calls Submit.

type TaskState

type TaskState struct {
	ID       string
	TESState tes.State
	State    string
	Reason   string
	Remove   bool
}

TaskState is a structure used by Reconcile to represent the state of a task in Funnel and the HPC backend.

Directories

Path Synopsis
Package gridengine contains code for accessing compute resources via Open Grid Engine.
Package gridengine contains code for accessing compute resources via Open Grid Engine.
Package htcondor contains code for accessing compute resources via HTCondor.
Package htcondor contains code for accessing compute resources via HTCondor.
Package pbs contains code for accessing compute resources via PBS/Torque.
Package pbs contains code for accessing compute resources via PBS/Torque.
Package slurm contains code for accessing compute resources via Slurm.
Package slurm contains code for accessing compute resources via Slurm.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL