moderation

package
v1.24.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 23, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	ErrInvalidAttestation = errors.New("invalid attestation")
	ErrInvalidSignature   = errors.New("invalid attestation signature")
	ErrInvalidAttestor    = errors.New("attestor public key mismatch")
	ErrInvalidContentHash = errors.New("content hash mismatch")
	ErrExpiredAttestation = errors.New("attestation is expired")
)
View Source
var ErrUnsupportedLanguage = errors.New("unsupported language")

ErrUnsupportedLanguage is returned when the moderation service does not support the language of the provided text.

Functions

func HighestFlaggedCategory added in v1.19.0

func HighestFlaggedCategory(result *Result) moderationpb.FlaggedCategory

HighestFlaggedCategory maps a moderation Result's flagged categories onto the proto FlaggedCategory vocabulary, returning the highest-scoring one that maps to a well-defined category (falling back to OTHER). It is the shared mapping used both by this service's responses and by callers that surface a moderation verdict elsewhere (e.g. a blob's rejection metadata).

func NewIntegration

func NewIntegration(attestor *commonpb.PublicKey) ocp_integration.Moderation

func ValidateAttestation

func ValidateAttestation(attestation *moderationpb.ModerationAttestation, attestor *commonpb.PublicKey, content any) error

ValidateAttestation validates a moderation attestation against the expected attestor public key and content. Content must be a string (for text) or []byte (for images).

Types

type Client

type Client interface {
	// ClassifyText classifies the provided text for moderation. The result
	// indicates whether the text was flagged and includes per-category scores.
	ClassifyText(ctx context.Context, text string) (*Result, error)

	// ClassifyImage classifies the provided image data for moderation. The
	// result indicates whether the image was flagged and includes per-category
	// scores.
	ClassifyImage(ctx context.Context, data []byte) (*Result, error)

	// ClassifyCurrencyName checks whether a currency name infringes on
	// existing trademarks, impersonates known brands, or is otherwise
	// misleading. The result includes per-category scores for areas like
	// cryptocurrency, financial_service, impersonation, etc.
	ClassifyCurrencyName(ctx context.Context, name string) (*Result, error)

	// ClassifyDisplayName checks whether a user-chosen display name abuses the
	// name field to advertise, solicit, or expose others to harmful content.
	// The result includes per-category scores for areas like solicitation,
	// contact_info, hate, etc.
	//
	// It deliberately does not score impersonation: users are free to call
	// themselves whatever they like, including the name of a real person or
	// brand. It is also distinct from ClassifyText, which is tuned for prose
	// and has little to work with in a one- or two-word name.
	ClassifyDisplayName(ctx context.Context, name string) (*Result, error)
}

type Integration

type Integration struct {
	// contains filtered or unexported fields
}

func (*Integration) ValidateAttestation

func (i *Integration) ValidateAttestation(_ context.Context, _ *ocp_common.Account, rawAttestation []byte, content any) (bool, error)

type Result

type Result struct {
	// Is the piece of data flaggged as unsafe?
	Flagged bool

	// Categories that caused the data to be flagged
	FlaggedCategories []string

	// Various category scores applied to the piece of data
	CategoryScores map[string]float64
}

type Server

type Server struct {
	moderationpb.UnimplementedModerationServer
	// contains filtered or unexported fields
}

func NewServer

func NewServer(log *zap.Logger, authz auth.Authorizer, client Client, attestor model.KeyPair) *Server

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL