durabletaskscheduler

Durable Task Scheduler SDK

The durabletaskscheduler package is the primary SDK surface for connecting Go management clients and workers to Durable Task Scheduler (DTS).

Configuration

Use a connection string from the environment:

export DTS_CONNECTION_STRING='Endpoint=https://<scheduler-host>;TaskHub=<task-hub>;Authentication=DefaultAzure'
go run ./samples/durabletaskscheduler

For the local DTS emulator:

export DTS_CONNECTION_STRING='Endpoint=http://127.0.0.1:8080;TaskHub=default;Authentication=None'
go run ./samples/durabletaskscheduler

Connection strings support the Azure Identity modes that have Go equivalents: DefaultAzure, ManagedIdentity, WorkloadIdentity, Environment, AzureCLI, AzurePowerShell, InteractiveBrowser, and None. Keys and Authentication values are case-insensitive, surrounding whitespace is trimmed, empty segments are skipped, and a repeated key uses its last value. Only the first = in a segment separates the key from the value, so values may contain =. Unknown keys, malformed segments, and a missing or blank Endpoint, TaskHub, or Authentication are rejected. VisualStudio and VisualStudioCode have no Go equivalent, and TokenCredential is programmatic only: both are rejected with guidance to use NewOptionsWithCredential with any azcore.TokenCredential. Plaintext http:// endpoints are accepted only with Authentication=None, which also requires AllowInsecureConnection.

Authentication-specific keys are ClientID, TenantID, TokenFilePath, and the comma-separated AdditionallyAllowedTenants. Each mode consumes only the fields Azure Identity for Go supports for it:

Environment gets its identity settings from the AZURE_* environment variables, and WorkloadIdentity falls back to them for any field left unset. Fields are trimmed before use and blank AdditionallyAllowedTenants entries are dropped. None and TokenCredential construct no Azure Identity credential, so supplying identity fields for them fails validation instead of being silently ignored.

Access tokens are requested for Options.ResourceID, also available as the ResourceId connection-string key. This is a token audience URI, not an Azure Resource Manager resource path. An explicit nonempty value overrides the default. An omitted or exactly empty value selects https://durabletask.azure.us when REGION_NAME starts with usgov or usdod, case-insensitively; otherwise it selects https://durabletask.io. This intentionally changes the default for government-region hosts. Set ResourceId=https://durabletask.io explicitly to retain the public audience there. Substring matches such as notusgov and notusdod, and other regions such as chinaeast2, retain the public default. The service endpoint is never used to infer the audience.

NewOptions, NewOptionsWithCredential, and NewOptionsFromConnectionString resolve the default when each options instance is created. A hand-built Options value, or one whose ResourceID is reset to "", resolves it when NewClient or NewWorker is called. The selected audience is retained across token refreshes, management-channel recreation, worker reconnects, and compatibility listeners. Later changes to the environment or caller's options do not change a running client or worker.

Normalization trims surrounding whitespace and trailing / characters, removes one existing /.default suffix (case-insensitively), then trims trailing / characters again. The token request appends /.default; custom URI casing is preserved. For example, https://durabletask.azure.us//.DEFAULT// requests https://durabletask.azure.us/.default, and api://CustomAudience/resource/.DEFAULT/ requests api://CustomAudience/resource/.default. Whitespace-only values, ///, /.default, and /.DEFAULT/// are rejected, including in connection strings. The options retain the input's suffix so validation and reconnects cannot accidentally strip meaningful URI segments by normalizing an already-normalized value again.

Resource audience, credential authority/cloud, and service endpoint are separate. Neither ResourceID nor REGION_NAME changes the endpoint or authority. For SDK-created credentials, optional Options.AuthorityHost (or connection-string AuthorityHost) forwards an HTTPS authority URL through Azure Identity's ClientOptions.Cloud.ActiveDirectoryAuthorityHost. It is supported by DefaultAzure, WorkloadIdentity, Environment, and InteractiveBrowser. Omission preserves Azure Identity's defaults, including AZURE_AUTHORITY_HOST. An explicit value overrides that environment setting.

For Authentication=TokenCredential, configure authority on the supplied credential itself; GetToken has no per-request authority override. None, TokenCredential, ManagedIdentity, AzureCLI, and AzurePowerShell reject a nonempty AuthorityHost instead of silently ignoring it. Managed identity uses the hosting environment's identity endpoint, not an Entra authority override. Developer-tool credentials require the tool's own cloud configuration (including when used within DefaultAzure); AuthorityHost does not configure those tools.

Azure Government example

Configure the actual scheduler endpoint separately from the token audience and credential authority:

credential, err := azidentity.NewDefaultAzureCredential(&azidentity.DefaultAzureCredentialOptions{
    ClientOptions: azcore.ClientOptions{Cloud: cloud.AzureGovernment},
})
if err != nil {
    return err
}
options := durabletaskscheduler.NewOptionsWithCredential(endpoint, taskHub, credential)
options.ResourceID = durabletaskscheduler.GovernmentResourceID
// Pass these options to NewClient and NewWorker. Do not set AuthorityHost:
// the supplied credential already owns its authority configuration.

The Azure imports are github.com/Azure/azure-sdk-for-go/sdk/azcore, github.com/Azure/azure-sdk-for-go/sdk/azcore/cloud, and github.com/Azure/azure-sdk-for-go/sdk/azidentity. For SDK-created credentials, the equivalent connection string is:

Endpoint=https://<government-scheduler-host>;TaskHub=<hub>;Authentication=DefaultAzure;ResourceId=https://durabletask.azure.us;AuthorityHost=https://login.microsoftonline.us/

The authentication sample exercises both connection-string and caller-created credentials, including government-cloud configuration.

Token and connection lifecycle

Token acquisition failures are surfaced as retriable Unavailable gRPC errors. Tokens and immutable authorization metadata are cached until the credential's RefreshOn time (or five minutes before expiry), and concurrent refreshes are coalesced. Every RPC carries taskhub and x-user-agent metadata; worker connections add workerid. Options.UserAgent and Options.WorkerID override the generated values and are rejected if they contain leading/trailing whitespace or newlines. Unset worker IDs default to <hostname>,<pid>,<uuid>.

Options.HelloTimeout (default 30 seconds) bounds the fail-fast Hello handshake for both NewClient and the worker connection factory; the caller's context still applies when it is shorter. Client channels use a default gRPC service config that retries UNAVAILABLE for the idempotent Hello handshake and read-only RPCs up to five attempts with a 50 ms initial backoff, 250 ms cap, and multiplier 2. Mutations such as RaiseEvent, SignalEntity, and instance lifecycle operations are not automatically retried: a failed acknowledgement can follow a successful service-side change. Worker channels do not use this policy, because the worker owns its own reconnect loop.

Individual gRPC messages are bounded to 64 MiB by default through Options.MaxReceiveMessageSize and Options.MaxSendMessageSize. The worker uses a 3.9 MiB safety bound below the service ceiling. When Options.MaxSendMessageSize is lower, that configured send limit becomes the bound instead. The worker validates the final response after large-payload externalization and fails an orchestration once with OrchestratorResponseTooLarge if it still exceeds that limit. The legacy isPartial/chunkIndex response fields are deprecated; the Go worker does not use them because validation against the current DTS emulator did not provide a portable chunking path. Configure large-payload storage or reduce per-turn fan-out instead. Active streams use a two-minute keepalive with a 20-second acknowledgement timeout; set Options.KeepaliveTime to zero to disable it. The timeout is ignored while keepalive is disabled. Values below 30 seconds are rejected to avoid aggressive-ping disconnects.

NewClient creates and owns a management connection; call Close when done. Options created with NewOptions, NewOptionsFromConnectionString, or NewOptionsWithCredential recreate the channel after five consecutive Unavailable responses (or unexpected DeadlineExceeded responses), with a 30-second minimum interval. Successful and application-level responses reset the counter, while caller cancellation/deadlines and expected deadlines from instance start/completion long polls do not count. Configure the thresholds with Options.ChannelRecreateFailureThreshold and Options.ChannelRecreateMinInterval. Authentication, interceptors, large payload settings, and data conversion are preserved across replacements. A hand-built Options value with a zero failure threshold disables recreation.

client.NewTaskHubGrpcClient, client.NewTaskHubGrpcWorker, and TaskHubGrpcClient.StartWorkItemListener are lower-level APIs that borrow the caller's grpc.ClientConnInterface; they never replace or close it. NewWorker owns its channels, recreates them after transient disconnects, and closes retired channels after their in-flight completions drain. client.NewTaskHubGrpcWorkerWithConnectionFactory provides the same lifecycle when its factory returns a non-nil closer, which transfers ownership to the worker. Use an owning configuration to recover from a permanently wedged channel. After a worker has started, Unauthenticated and PermissionDenied stream or reconnect-handshake responses are retried with backoff so token refresh and RBAC propagation can recover without restarting the worker. See client.NewTaskHubGrpcWorker for the full ownership contract.

Reconnect and RPC retry delays are deterministic and always stay within [baseDelay, maxDelay]. A stream that delivers at least one message before it ends is treated as a drain and restarts the schedule at the base delay; a stream that stays silent past WithWorkerSilentDisconnectTimeout before its first message is treated as poisoned and keeps escalating.

Options.MaximumTimerInterval defaults to three days. Longer durable timers are split into deterministic sequential timer actions that retain the original deadline. Histories written before splitting remain compatible once the original logical timer deadline fires. Changing the interval between two splitting configurations is replay-breaking for affected in-flight orchestrations.

Advanced management

TaskHubGrpcClient exposes bounded QueryInstances and ListInstanceIDs operations with opaque continuation tokens, plus RestartInstance, RewindInstance, and batch/filter PurgeInstances. Queries can filter locally by exact tag key/value pairs when the current wire contract does not carry tag filters.

The SDK does not expose skip-graceful-termination operations. Use TerminateOrchestration for normal orchestration termination.

GetOrchestrationHistory returns API-owned history records with validated event and approximate byte caps. StreamOrchestrationHistory invokes a callback in service order without retaining the history in memory. Both preserve execution identity, timestamps, failures, tags, context fields, rewind markers, and native entity events. Serialized payloads remain raw until a ReadInput, ReadResult, or ReadData helper applies the configured data converter.

The current DTS emulator supports query, restart, and batch purge, but has known service limitations: filtered purge can complete without deleting matches, and ListInstanceIds can omit matching IDs. The emulator integration tests record these limitations explicitly.

Rewind

RewindInstance(ctx, id, api.WithRewindReason("dependency repaired")) enqueues recovery of a failed orchestration. It rejects empty IDs and entity IDs. Missing instances, invalid states, unsupported services, and cancellation retain the existing errors.Is categories (api.ErrInstanceNotFound, api.ErrInvalidState, api.ErrFeatureNotSupported, and context errors). Cancellation stops enqueueing, not recovery that the service already accepted.

Recovery is a two-step worker/service operation. Before executing user code, the worker sends RewindOrchestrationAction.NewHistory: failed activity schedules and results, failed child results, and terminal completion records are removed. Successful history and failed child creation records are retained. The start record receives a new execution ID; a rewound child's parent execution ID is updated. DTS persists this history, recursively rewinds failed children, and wakes the leaf for ordinary replay. Successful activities are replayed from their results rather than executed again.

This follows the standalone Python SDK's _build_rewind_result, not the incomplete standalone .NET worker implementation. An accepted RPC alone does not prove recovery. Poll for a changed execution ID and successful final status/output; WaitForOrchestrationCompletion can initially return the old failed execution. The runnable sample demonstrates this and cleans up only its own instance.

Limitations: the Go worker conservatively rejects replacement history when anything other than orchestration lifecycle markers follows the first failed activity or child operation. This includes handled failures followed by more work, retry timers, and some concurrent executions. Removing those failures could prevent retained downstream work from replaying. The worker reports the unsupported history in its logs without emitting replacement history; the instance may retain its original failure metadata. Enqueue success does not imply that this worker-side validation succeeded. This guard is not a general proof of replay safety and does not infer whether a failure was handled. These history rejections describe a Go SDK limitation, not a lack of rewind support in DTS. They retain the api.ErrFeatureNotSupported error category. Rewind does not repair unchanged failing application code. Replacement history uses the normal large-payload handling and orchestration response-size bound; oversized responses fail explicitly rather than being silently truncated.

Worker routing and capabilities

Use client.WithTaskVersioning for None, Strict, or CurrentOrOlder acceptance and Reject or Fail mismatch handling. VersioningOptions.DefaultVersion is applied to sub-orchestrations. When Options.Versioning is set, its default also applies to top-level starts made by the DTS client.

Registrations use AddOrchestratorNVersion and AddActivityNVersion. Exact name/version matches win. Unversioned fallback is allowed only for a logical name with no versioned registrations. Activities inherit their parent orchestration version unless explicitly overridden, including an explicit unversioned "" override. ContinueAsNew can migrate with task.WithContinueAsNewVersion, including to task.UnversionedTaskVersion.

The current DTS service accepts numeric versions in Major[.Minor[.Patch]] form. The Go registry also supports case-insensitive opaque version strings, but DTS applications should use numeric versions such as "1.0" and "2.0".

Use client.WithAutoWorkItemFilters() to derive filters from the registry, or client.WithWorkItemFilters for an explicit override. Local enforcement is a fallback for services that ignore the filter request. CurrentOrOlder ranges cannot be represented by the protocol filter, so automatic filters advertise only registered versions at or below the worker version, including an explicit empty version for unversioned registrations. An unversioned-only handler also advertises the current worker version and a compatible configured default version, which it can resolve through fallback. Register every older version that the worker should receive; unversioned fallback does not advertise an open range. Wildcard handlers require explicit filters with this strategy. Service-side filters can leave a task pending indefinitely when no worker advertises it, whereas unfiltered delivery produces a deterministic task-not-found failure. Auto-generated filters reject task kinds with no registrations and validate that strict worker versions can resolve every advertised logical name. Capability advertisement is explicit: history streaming is enabled by default, while scheduled tasks use durabletaskscheduler.WithScheduledTasks(). Streamed history accumulation is bounded to 100,000 events and 64 MiB by default; use client.WithMaxStreamedHistoryEvents and client.WithMaxStreamedHistoryBytes to choose different bounded limits. These are per-work-item limits, so size worker concurrency with the aggregate memory budget in mind. Exceeding either limit delays and abandons the work item until the worker is reconfigured with a larger bounded limit.

Recurring scheduled tasks

Mixed-SDK constraint: this surface uses Go SDK-owned system entity and orchestrator names and state. Use it only in Go-only task hubs; interoperability with .NET workers sharing those system names and state is not defined.

Scheduled tasks use recurring UTC intervals, not cron expressions. Register the system entity and orchestrators before creating the worker:

registry := task.NewTaskRegistry()
registry.AddOrchestratorNVersion("Backup", "1.0", backup)
durabletaskscheduler.RegisterScheduledTasksWithDefaultVersion(registry, "1.0")

worker, _ := durabletaskscheduler.NewWorker(
    options,
    registry,
    logger,
    durabletaskscheduler.WithScheduledTasks(),
    client.WithAutoWorkItemFilters(),
)

Create and manage schedules through the DTS client:

schedules := schedulerClient.ScheduledTasks()
handle, _ := schedules.Create(ctx, durabletaskscheduler.ScheduleCreationOptions{
    ScheduleID:              "nightly-backup",
    OrchestrationName:       "Backup",
    TypedOrchestrationInput: backupRequest,
    Interval:                24 * time.Hour,
    StartAt:                 firstRun,
    StartImmediatelyIfLate:  true,
    Tags:                    map[string]string{"team": "storage"},
    ContextFields:           api.ContextFields{"tenant": "north"},
})

description, _ := handle.Describe(ctx)
_ = handle.Pause(ctx)
_ = handle.Update(ctx, durabletaskscheduler.ScheduleUpdateOptions{Interval: &newInterval})
_ = handle.Resume(ctx)
_ = handle.Delete(ctx)

Get returns nil, nil for a missing schedule. List is continuation-based; status and creation-time filters are applied after each entity page, so pages can be underfilled. Schedule operations return typed errors compatible with errors.Is and errors.As. A fixed target instance ID prevents overlapping runs. Fixed target instance IDs cannot be combined with a retry policy because each retry requires a distinct durable instance. Tags, context fields, and retry policies use an internal launch orchestrator because the entity-start wire action cannot carry them directly. Explicit work-item filters must include the Schedule entity and both system orchestrators; auto filters include them from the registry.

Data converters

Set Options.DataConverter to configure one api.DataConverter for the DTS client and worker. The default is api.JSONDataConverter. Conversion happens before large-payload externalization and after hydration. Converter errors are returned; the SDK never retries a payload with JSON. Raw input/output APIs and serialized metadata fields bypass conversion. Converter identity is not stored by the protocol, so deployments must retain backward decoding compatibility.

Large payloads

Large payload support uses an opaque, integrity-checked reference encoded in the existing string payload fields. Production DTS workloads should configure the same Azure Blob store for every management client and worker:

store, err := payload.NewAzureBlobStore(payload.AzureBlobStoreOptions{
    ConnectionString: os.Getenv("AzureWebJobsStorage"),
    Container:        "durabletask-payloads",
})
if err != nil {
    return err
}
options.LargePayloads = &api.LargePayloadOptions{
    Store:            store,
    Resolver:         store,
}

payload.NewAzureBlobStore emits the same self-describing blob:v2:<absolute-blob-url> token as the .NET SDK and can read legacy .NET blob:v1 tokens. payload.NewMemoryStore is only for tests and single-process experiments because references are lost on restart. payload.NewFileStore requires every client and worker to share the same durable filesystem path; container-local or other ephemeral storage is not safe.

Identity authentication uses AccountURL plus an azcore.TokenCredential. Cross-account identity reads require an explicit AllowedHosts entry. Azure Blob defaults match .NET: 256 KiB inclusive threshold, 10 MiB maximum, gzip enabled, eight exponential retries, and the durabletask-payloads container. Tokens are treated as untrusted: userinfo, SAS/query strings, fragments, unapproved hosts, malformed paths, oversized downloads, and invalid integrity metadata are rejected. Go-written blobs include size, SHA-256, and content-MD5 integrity data; .NET blobs without that metadata remain readable.

Workers advertise LARGE_PAYLOADS only when LargePayloads is configured. Resolver implementations must treat reference locations as untrusted and enforce their own scheme/account/path allow lists.

Worker lifecycle

Use Start for background execution or Run for blocking execution. Shutdown stops intake, allows in-flight execution and completion RPCs to drain, and cancels them only if the shutdown context expires.

Feature matrix

The V2 protobuf does not carry per-operation trace context or request time to an entity worker. DTS sends entity state on every work item; a null state means the entity does not exist. Entity-emitted actions cannot inherit per-operation trace context.