webprofiler

web-profiler

中文文档 | English

web-profiler is a lightweight request analysis middleware for net/http. It inspects incoming requests with bounded overhead, restores the request body for downstream handlers, and exposes structured results through context.Context.

It is designed as request-analysis infrastructure, not as a security decision engine.

🐲 Highlights

• Native net/http middleware API with easy integration into Gin, Chi, Echo, and other net/http-based frameworks
• One bounded body capture shared by all analyzers
• Structured request profile exposed through FromContext
• Per-request analysis duration with nanosecond precision
• Rich request metadata including observed bytes, header stats, and per-analyzer timings
• Multiple bounded sampling strategies: head, tail, and head_tail
• Optional compressed-body inspection, trusted-proxy CIDR checks, and alternate fingerprint hash algorithms
• Safe degradation with warnings instead of failing the request
• Built-in analyzers for entropy, fingerprint, complexity, and charset distribution

Installation

go get github.com/GoFurry/web-profiler

🚀 Quick Start

package main

import (
	"log"
	"net/http"

	webprofiler "github.com/GoFurry/web-profiler"
)

func main() {
	cfg := webprofiler.DefaultConfig()

	handler := webprofiler.Middleware(cfg)(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		profile, ok := webprofiler.FromContext(r.Context())
		if ok && profile != nil {
			if profile.Entropy != nil {
				log.Printf("entropy=%.4f", profile.Entropy.Value)
			}
			if profile.Fingerprint != nil {
				log.Printf("fingerprint=%s", profile.Fingerprint.Hash)
			}
		}

		// The request body is still readable here.
		w.WriteHeader(http.StatusOK)
	}))

	log.Fatal(http.ListenAndServe(":8080", handler))
}

You can also use the convenience helper:

handler := webprofiler.Wrap(mux, webprofiler.DefaultConfig())

A runnable native net/http example lives at example/main.go.

Using Profile Data In Handlers

FromContext gives you the collected Profile. You can inspect metadata, module outputs, and warnings inside any downstream handler:

func inspectHandler(w http.ResponseWriter, r *http.Request) {
	profile, ok := webprofiler.FromContext(r.Context())
	if !ok || profile == nil {
		http.Error(w, "profile not found", http.StatusInternalServerError)
		return
	}

	body, err := io.ReadAll(r.Body)
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

	log.Printf("path=%s content_type=%s observed_bytes=%d", profile.Meta.Path, profile.Meta.ContentType, profile.Meta.ObservedBytes)
	log.Printf("headers=%d header_bytes=%d", profile.Meta.HeaderCount, profile.Meta.HeaderBytes)
	log.Printf("analysis_duration=%s", profile.Meta.AnalysisDuration)
	log.Printf("body=%s", string(body))

	if profile.Entropy != nil {
		log.Printf("entropy=%.4f sample_bytes=%d", profile.Entropy.Value, profile.Entropy.SampledBytes)
	}
	if profile.Fingerprint != nil {
		log.Printf("fingerprint=%s fields=%v", profile.Fingerprint.Hash, profile.Fingerprint.Fields)
	}
	if profile.Complexity != nil {
		log.Printf("complexity_score=%d depth=%d fields=%d scalars=%d", profile.Complexity.Score, profile.Complexity.Depth, profile.Complexity.FieldCount, profile.Complexity.ScalarCount)
	}
	if profile.Charset != nil {
		log.Printf("non_ascii_ratio=%.2f scripts=%v suspicious=%v", profile.Charset.NonASCIIRatio, profile.Charset.UnicodeScriptCounts, profile.Charset.SuspiciousFlags)
	}
	if len(profile.Warnings) > 0 {
		log.Printf("warnings=%v", profile.Warnings)
	}

	w.WriteHeader(http.StatusOK)
}

Gin Example

Gin uses net/http under the hood, so you can wrap its engine directly:

package main

import (
	"log"
	"net/http"

	"github.com/gin-gonic/gin"
	webprofiler "github.com/GoFurry/web-profiler"
)

func main() {
	cfg := webprofiler.DefaultConfig()
	engine := gin.New()

	engine.POST("/inspect", func(c *gin.Context) {
		profile, ok := webprofiler.FromContext(c.Request.Context())
		if !ok || profile == nil {
			c.JSON(http.StatusInternalServerError, gin.H{"error": "profile not found"})
			return
		}

		c.JSON(http.StatusOK, gin.H{
			"path":         profile.Meta.Path,
			"content_type": profile.Meta.ContentType,
			"observed":     profile.Meta.ObservedBytes,
			"headers":      profile.Meta.HeaderCount,
			"analysis_ns":  profile.Meta.AnalysisDuration.Nanoseconds(),
			"entropy":      profile.Entropy,
			"fingerprint":  profile.Fingerprint,
			"complexity":   profile.Complexity,
			"charset":      profile.Charset,
			"warnings":     profile.Warnings,
		})
	})

	handler := webprofiler.Middleware(cfg)(engine)
	log.Fatal(http.ListenAndServe(":8080", handler))
}

Public API

func Middleware(cfg Config) func(http.Handler) http.Handler
func Wrap(next http.Handler, cfg Config) http.Handler
func DefaultConfig() Config
func FromContext(ctx context.Context) (*Profile, bool)

What Gets Collected

Meta

• Method, path, normalized content type, and request content length
• Observed body bytes, sample size, truncation state, and header count/bytes
• End-to-end analysis duration plus per-analyzer durations
• Explicit skip and downgrade warnings when body-based analysis does not run

Entropy

• Shannon entropy over the sampled request body bytes
• Normalized entropy, unique-byte count, repetition ratio, and approximate compressibility
• Sample size and observed body size
• Sampling strategy metadata

Fingerprint

• Normalized request headers
• Optional client IP and TLS metadata
• Source flags plus weak/strong hashes with versioning
• Optional hash-only mode when you do not want raw normalized fields in results
• Trusted-proxy CIDR policy and alternate hash algorithms such as sha1, sha256, sha512, and fnv1a64

Complexity

• JSON depth, field counts, scalar/null/string stats, and key-length summaries
• Object/array counts, max array length, and max object width
• URL-encoded form statistics plus key/value length summaries
• Optional multipart file metadata such as file counts, extensions, and content types
• Interpretable score factors

Charset

• ASCII, digit, whitespace, symbol, control, and non-ASCII ratios
• Emoji ratio and invisible-character density
• Unicode script distribution counts when enabled
• Optional confusable/homoglyph detection and format-specific metrics for JSON, XML, and form payloads
• Optional suspicious flags such as invalid UTF-8, zero-width characters, and mixed scripts

🧭 Configuration

DefaultConfig() returns a ready-to-use setup with bounded defaults:

• BodyConfig limits read size, sample size, methods, and content types
• FingerprintConfig controls headers, proxy trust, TLS metadata, and hash versioning
• ComplexityConfig controls JSON depth, field limits, and supported content types
• CharsetConfig controls text analysis size and suspicious-pattern detection

Typical customization:

cfg := webprofiler.DefaultConfig()
cfg.Body.MaxReadBytes = 64 << 10
cfg.Body.SampleBytes = 8 << 10
cfg.Body.SampleStrategy = webprofiler.SampleStrategyHeadTail
cfg.Body.EnableCompressedAnalysis = true
cfg.Fingerprint.IncludeIP = true
cfg.Fingerprint.TrustProxy = true
cfg.Fingerprint.TrustedProxyCIDRs = []string{"10.0.0.0/8", "192.168.0.0/16"}
cfg.Fingerprint.HashAlgorithm = "sha512"
cfg.Fingerprint.ExposeFields = false
cfg.Complexity.MaxJSONDepth = 16
cfg.Complexity.EnableMultipartMeta = true
cfg.Charset.EnableConfusableDetection = true
cfg.Charset.EnableFormatSpecificMetrics = true

Performance Notes

• MetaInfo.AnalysisDuration records middleware analysis time as time.Duration
• MetaInfo now also exposes per-analyzer timings so you can see where request profiling time is spent
• The example exposes both analysis_duration and analysis_duration_ns so you can read it directly or aggregate it precisely
• The SHA-256 fingerprint step hashes a very small normalized string built from a few headers and optional TLS/IP fields, so in most cases it is not the main cost
• Alternate fingerprint hash algorithms are available, but sha256 remains the best default tradeoff for compatibility and stability
• Compressed-body inspection is optional because it adds decompression work; enable it when encoded request payloads are common in your traffic
• If you want the cheapest possible fingerprint output, set Fingerprint.ExposeFields = false to keep only hashes and source metadata
• In practice, request-body capture, JSON parsing, and charset scanning are usually more expensive than the final SHA-256 call
• If you run at very high QPS, benchmark with your own traffic and disable EnableFingerprint, IncludeIP, or IncludeTLS if you want an even cheaper profile

Example Response Fields

The native example at example/main.go returns a JSON payload like the one you posted. The following table maps each field to its meaning:

🌟 Design Boundaries

This middleware:

• analyzes requests, not responses
• does not persist or export results
• does not block traffic on analyzer failures
• does not guarantee deep parsing for every content type
• does not produce business risk decisions

Result Model

Each request produces a Profile:

type Profile struct {
	Meta        MetaInfo
	Entropy     *EntropyResult
	Fingerprint *FingerprintResult
	Complexity  *ComplexityResult
	Charset     *CharsetResult
	Warnings    []Warning
}

Analyzer results are optional pointers so handlers can distinguish between disabled, skipped, and populated modules.

For a deeper breakdown of the internals and extension strategy, see Architecture.md.

Testing

go test ./...

The test suite covers middleware behavior, body replay, config normalization, analyzer outputs, and warning paths.

🐺 License

This project is open-sourced under the MIT License, which permits commercial use, modification, and distribution without requiring the original author's copyright notice to be retained.