appsec

Appsec Go Design

This document describes the design of the internal/appsec package and everything under it. This package is responsible for securing the application by monitoring the operations that are executed by the application and applying actions in case a security threats is detected.

Most of the work is to forward information to the module github.com/DataDog/go-libddwaf which contains the WAF (Web Application Firewall) engine. The WAF does most of the decision making about events and actions. Our goal is to connect the different parts of the application and the WAF engine while keeping up to date the various sources of configuration that the WAF engine uses.

Instrumentation Gateway: Dyngo

Having the customer (or orchestrion) instrument their code is the hardest part of the job. That's why we want to provide the simplest API possible for them to use. This means loosing the flexibility or enabling and disabling multiple products and features at runtime. Flexibility that we still want to provide to the customer, that's why behind every API entrypoint present in dd-trace-go/contrib that support appsec is a call to the internal/appsec/dyngo package.

flowchart LR

UserCode[User Code] --> Instrumentation --> IG{Instrumentation<br> Gateway} --> Listener

Dyngo is a context-scoped event listener system that provide a way to listen dynamically to events that are happening in the customer code and to react to configuration changes and hot-swap event listeners at runtime.

flowchart LR

UserCode[contrib] --> appsec/emitter --> IG{dyngo} --> appsec/listener --> WAF
appsec/remoteconfig -->|config change| IG
appsec/config -->|config change| IG

Operation definition requirements

• Each operation must have a Start* and a Finish method covering calls to dyngo.
• The content of the arguments and results should not require any external package, at most the standard library.

Example operation:

package main

import (
	"context"

	"github.com/DataDog/dd-trace-go/v2/instrumentation/appsec/dyngo"
	"github.com/DataDog/dd-trace-go/v2/internal/log"
)

type (
	ExampleOperation struct {
		dyngo.Operation
	}

	ExampleOperationArgs struct {
		Type string
	}

	ExampleOperationResult struct {
		Code int
	}
)

func (ExampleOperationArgs) IsArgOf(*ExampleOperation)      {}
func (ExampleOperationResult) IsResultOf(*ExampleOperation) {}

func StartExampleOperation(ctx context.Context, args ExampleOperationArgs) *ExampleOperation {
	parent, ok := dyngo.FromContext(ctx)
	if !ok {
		log.Error("No parent operation found")
		return nil
	}
	op := &ExampleOperation{
		Operation: dyngo.NewOperation(parent),
	}
	return dyngo.StartOperation(op, args)
}

func (op *ExampleOperation) Finish(result ExampleOperationResult) {
	dyngo.FinishOperation(op, result)
}

[!CAUTION] Importing external packages in the operation definition will probably cause circular dependencies. This is because the operation definition can be used in the package is will instrument, and the package that will instrument it will probably import the operation definition.

Operation Stack

Current state of the possible operation stacks

flowchart TD

    subgraph Top Level Operation
        SES[trace.ServiceEntrySpanOperation]

        Context[waf.ContextOperation]

        HTTPH[httpsec.HandlerOperation]
        GRPCH[grpcsec.HandlerOperation]
        GQL[graphqlsec.RequestOperation]
    end

    subgraph HTTP
        RequestBody([httpsec.MonitorRequestBody])
        Roundtripper[httpsec.RoundTripOperation]
    end

    subgraph GRPC
        RequestMessage([grpcsec.MonitorRequestMessage])
        ResponseMessage([grpcsec.MonitorResponseMessage])
    end

    subgraph GraphQL
        Exec[graphqlsec.ExecutionOperation]
        Resolve[graphqlsec.ResolveOperation]
    end

    Code{User Code}

    SES --> Context
    Context --> HTTPH --> Code
    Context --> GRPCH --> Code
    Context --> GQL

    GQL --> Exec --> Resolve --> Code

    Code --> RequestBody

    Code --> RequestMessage
    Code --> ResponseMessage

    Code --> Span[trace.SpanOperation]

    Span --> Roundtripper
    Span --> OS[ossec.OpenOperation]
    Span --> SQL[sqlsec.SQLOperation]
    Span --> User[usersec.UserOperation]

[!IMPORTANT] Please note that this is how the operation SHOULD be stacked. If the user code does not have a Top Level Operation then nothing will be monitored. In this case an error log should be produced to explain thoroughly the issue to the user.

Features

Features represent an abstract feature added to the tracer by AppSec. They are the bridge between the configuration and its sources and the actual code that needs to be ran in case of enablement or disablement of a feature. Features are divided in two parts:

• The builder that should be a pure function that takes the configuration and returns a feature object.
• The listeners that are methods of the feature object that are called when an event from the Instrumentation Gateway is triggered.

From there, at each configuration change from any config source, the AppSec module will rebuild the feature objects, register the listeners to the Instrumentation Gateway, and hot-swap the root level operation with the new one, consequently making the whole AppSec code atomic.

Here is an example of how a system with only two features, GRPC and HTTP WAF Protection, would look like:

flowchart TD

    subgraph HTTP Feature
        HTTPListener
        HTTPBuilder
    end

    subgraph GRPC Feature
        GRPCBuilder
        GRPCListener
    end

    subgraph Configuration
        RemoteConfig
        EnvConfig
        ...
    end

    Configuration -->|config change| AppSec

    AppSec -->|rebuild| HTTPBuilder
    AppSec -->|rebuild| GRPCBuilder
    HTTPBuilder -->|register HTTP Listener| IG
    GRPCBuilder -->|register GRPC Listener| IG



    IG{Instrumentation<br> Gateway} -->|Start httpsec.HandlerOperation| HTTPListener
    IG{Instrumentation<br> Gateway} -->|Start grpcsec.HandlerOperation| GRPCListener

All currently available features are the following ones. Each feature is identified by a name (a slug, defined in the listener package and returned by the feature's String() method):

DD_APPSEC_DISABLE_FEATURES takes a comma-separated list of feature names (case-insensitive) that must not be started, e.g. DD_APPSEC_DISABLE_FEATURES=http,grpc. The waf and trace features are required by every other feature and cannot be disabled. Unknown names are ignored with a warning log. Because HTTP bodies, HTTP route parameters and gRPC messages reach the WAF context without going through the http and grpc listeners, disabling those features also removes their WAF addresses from the supported addresses, so the WAF never evaluates them.

Error Tracking eligibility

The AppSec package marks each log.Error and log.Warn site with an //errtrack:ignore directive when the logged failure is not eligible for Error Tracking reporting. The directive carries the reason, so an audit does not have to re-derive it from the code.

A site is ineligible only when its failure is outside the SDK's control or is reported through another channel:

• The caller used the AppSec API while AppSec is disabled, or without the request instrumentation that the API requires. These failures depend on the application, not on the SDK.
• The failure comes from a user-provided value, such as a blocked-response template path.
• The failure comes from the host or the native library, such as a libddwaf compatibility issue on the host.
• The failure is an expected limit on the request path: the global trace rate limit (DD_APPSEC_TRACE_RATE_LIMIT) or the maximum number of WAF events per request.
• Remote Config reports the failure back to the backend through an apply status. Only a site that always produces an apply status may use this reason.

A site without a directive stays actionable in the audit. Do not add a directive to hide an unclassified site; remove or fix the site instead.

Startup timing is not a reason to leave a site unclassified. A report made before telemetry.StartApp captures its stack trace eagerly at the call site and is replayed once telemetry starts, so the stack still points at the real call site. The failure mode to watch for is eviction from the 512-entry ring buffer shared by every global telemetry call, not a bad stack trace. Prefer a site that fires after StartApp when you have the choice; see the reporting policy in internal/README.md for the full criteria.

AppSec state checks

Enabled and RASPEnabled load the active AppSec instance and its started state atomically. Query hooks do not take the lifecycle mutex. Instance replacement remains serialized, and remote activation updates the instance's atomic state.

SQL monitoring in pgx

The native pgx integration only monitors SQL; it cannot block it. See contrib/jackc/pgx.v5/README.md for its behavior and limitations.

Block outcome telemetry

Tests that inspect WAF telemetry should install testutils.StartTelemetryRecorder before starting AppSec. This isolates the recorded metrics from submissions buffered by earlier tests while preserving exact request-count assertions.

The waf.requests metric has one data point for each WAF context. Two of its tags show the block outcome of the request:

The ContextMetrics type in emitter/waf/metrics.go records the outcome:

• runWAF calls SetBlockRequested when the WAF returns a WAF-scope block_request action. RASP-scope actions do not change these tags.
• SetBlockFailed records that the block was not enforced. runWAF calls it when the action cannot be built. Integrations call it when they cannot deliver the block response, for example because the response has already started, or because a block callback or the block response panicked.
• SetBlockApplied records that a block response was applied. The HTTP integration calls it when the block response was delivered.
• The HTTP integration reports a failure or an application only for the block that the WAF-scope block_request created. runWAF marks that block with actions.Config.ReportBlockOutcome, and BlockHTTP.ReportsBlockOutcome returns true for it. A redirect or a RASP-scope block uses the same BlockHTTP type, but its outcome does not change these tags. Thus, an applied redirect cannot hide a WAF block that failed.
• If a block panics before httpsec.BeforeHandle returns, the caller cannot run afterHandle. BeforeHandle then finishes the operation and the WAF context before the panic continues, so waf.requests is still submitted.
• Submit resolves the two tags at the end of the WAF context. A requested block counts as enforced unless a failure was reported. Thus, an integration that does not report its outcome keeps the previous behavior. An applied block has precedence: a failure that a later action reports does not change it.

RASP rules report their block outcome on rasp.rule.match with the block:success, block:failure, or block:irrelevant tag.

A monitor-only run (SubcontextOperation.RunMonitorOnly, used by the pgx SQL monitoring) removes the block_request and redirect_request actions. It reports a removed block_request as block:failure on rasp.rule.match. It does not count as a requested block on waf.requests, so it cannot hide a later block of the same request.