Affected by 
Documentation
¶
Index ¶
- Constants
- Variables
- func AppendWithExactOverlap(acc, next string, overlap int) (string, bool)
- func AppendWithOverlap(acc, next string, positionOverlap int) string
- func BuildContentSignature(content string) string
- func BuildImageInfoMarkdownWithURL(url string, img *types.ImageInfo) string
- func BuildImageInfoXML(img *types.ImageInfo) string
- func BuildImageInfoXMLWithURL(url string, img *types.ImageInfo) string
- func CaptureImageEvidence(r *types.SearchResult)
- func ClampFloat(v, minV, maxV float64) float64
- func ClearImageInfoTextMatchingBody(imageInfoJSON, recognized, chunkType string) string
- func CollectImageInfoByChunkIDs(ctx context.Context, chunkRepo interfaces.ChunkRepository, tenantID uint64, ...) map[string]string
- func CollectImageInfoByChunkIDsOnly(ctx context.Context, chunkRepo interfaces.ChunkRepository, chunkIDs []string) map[string]string
- func ContainsChinese(text string) bool
- func ContainsChunkContent(container, contained string) bool
- func ContentOverlapRatio(a, b string) float64
- func ContextImages(ctx context.Context, results []*types.SearchResult, ...) (images []string, positions []int)
- func ConvertWebSearchResults(webResults []*types.WebSearchResult, opts ...ConvertWebResultOption) []*types.SearchResult
- func EnrichContentCaptionAndOCR(content string, imageInfoJSON string) string
- func EnrichContentCaptionOnly(content string, imageInfoJSON string) string
- func EnrichContentWithImageInfo(content string, imageInfoJSON string) string
- func EnrichContentWithImageInfoForChat(content string, imageInfoJSON string) string
- func EnrichSearchResultsImageInfo(ctx context.Context, chunkRepo interfaces.ChunkRepository, tenantID uint64, ...)
- func EnrichSearchResultsImageInfoOnly(ctx context.Context, chunkRepo interfaces.ChunkRepository, ...)
- func ExtractSnippet(content string, query string) string
- func FilterImageInfoByContentURLs(content string, imageInfoJSON string) string
- func FilterImageInfoByMatchRange(parentContent string, parentStartAt, matchStart, matchEnd int, ...) string
- func ImageURLsFromInfo(imageInfoJSON string) map[string]bool
- func ImageURLsInContent(content string) map[string]bool
- func InheritImageEvidence(kept, dropped *types.SearchResult)
- func IsContentContained(normalizedShort, normalizedLong string) bool
- func IsImageEvidence(r *types.SearchResult) bool
- func IsKeptOutsideTopK(r *types.SearchResult) bool
- func Jaccard(a, b map[string]struct{}) float64
- func JoinChunkContent(acc, next, separator string) string
- func MergeImageInfoJSON(perChunk map[string]string) string
- func MergeTextChunks(chunks []*types.Chunk, gapSep string) string
- func NormalizeContent(s string) string
- func NormalizeKeywordScores[T any](results []T, isKeyword func(T) bool, getScore func(T) float64, ...)
- func PruneMarkdownImagesByImageInfo(content, imageInfoJSON string) string
- func PruneMarkdownImagesOutsideRange(content string, contentStartAt, matchStart, matchEnd int) string
- func SliceContentByDocumentRange(content string, contentStartAt, rangeStart, rangeEnd int) string
- func TokenOverlapRatio(tokA, tokB map[string]struct{}) float64
- func TokenizeSimple(text string) map[string]struct{}
- func TopKKeepingKept(results []*types.SearchResult, k int) []*types.SearchResult
- type ConvertWebResultOption
- type KeywordScoreCallbacks
Constants ¶
const HTMLImageSrcURLGroup = 2
HTMLImageSrcURLGroup is the submatch index of the src value in HTMLImageSrcRegex.
const MaxContextImages = 3
MaxContextImages caps the retrieved images shown to a vision chat model in one answer: each costs as much context as a long passage.
Variables ¶
var HTMLImageSrcRegex = regexp.MustCompile(`(?i)<img\b([^>]*?)\ssrc\s*=\s*['"]([^'"]+)['"]([^>]*)>`)
HTMLImageSrcRegex matches an HTML <img> tag with a quoted src attribute. Documents that rely on surrounding markup for layout embed their screenshots this way, so image-derived text has to be matched back to these tags as well as to Markdown image links.
Submatches: 1 = attributes before src, 2 = the src value, 3 = attributes after. Callers index the value through HTMLImageSrcURLGroup rather than assuming a position.
src must be preceded by whitespace so that data-src and other hyphenated attribute names are not mistaken for it — matching those would capture a lazy-loading placeholder and leave the real src unvisited. An unquoted src and a srcset-only tag are deliberately out of scope; both are rare in authored documents and matching them reliably needs an HTML parser.
Both this package and the document parser share this one definition. Two copies would be free to drift, and an image the parser stores but the enricher cannot match back is exactly the kind of gap this exists to close.
var MarkdownImageRegex = regexp.MustCompile(`!\[([^\]]*)\]\(([^)]+)\)`)
MarkdownImageRegex matches Markdown image links: 
var VisionImageLimits = imageprep.Limits{ MaxBytes: 5_000_000, MIMETypes: []string{"image/png", "image/jpeg", "image/webp", "image/gif"}, }
VisionImageLimits are what every vision chat API this build talks to takes: PNG, JPEG, WebP and GIF, and Anthropic's 5 MB per image, the smallest of the documented limits (OpenAI and Gemini allow 20 MB).
Functions ¶
func AppendWithExactOverlap ¶
AppendWithExactOverlap 在调用方已确认位置坐标可信时，按坐标给出的精确重叠量 拼接 acc 与 next：校验 acc 的末 overlap 个字符与 next 的前 overlap 个字符逐字 符相等，相等则精确裁剪，overlap 为 0 时直接拼接。
与 AppendWithOverlap 的区别在于「不猜」：后者为兼容补写表头、HTML 实体等长度 偏差，会在窗口内搜索最长后缀匹配，重复周期性文本（表格、日志）可能被误判成重 叠而裁掉真实内容。坐标可信时重叠量是已知的，不需要搜索。
校验不通过返回 ok=false，由调用方决定是否回退到 AppendWithOverlap。
func AppendWithOverlap ¶
AppendWithOverlap 把 next 追加到 acc 之后，并去除二者之间的重叠部分。
positionOverlap 是由 StartAt/EndAt 估算的重叠量（lastEnd - curStart），仅用于 界定搜索窗口大小；真正的重叠按文本匹配，能兼容补写表头与 HTML 实体长度偏差。 若找不到文本重叠，则原样拼接（不裁剪），宁可保留也不破坏内容。
positionOverlap <= 0 时两段位置上严格相邻或不相交，没有可去重的重叠； 此时进入文本匹配会因 headSlack 下限 320 在 next 开头窗口内误命中 acc 后缀的真实内容重复（如同一句话在文档多次出现），把 next 开头 整段误判为补写表头删掉，造成不可逆的内容丢失。直接拼接，补写表头 重复交给调用方后处理。
func BuildContentSignature ¶
BuildContentSignature creates a normalized MD5 signature for content to detect duplicates. It normalizes the content by lowercasing, trimming whitespace, and collapsing multiple spaces.
func BuildImageInfoMarkdownWithURL ¶
BuildImageInfoMarkdownWithURL formats one image as answer-ready Markdown for LLM-facing chat/tool context. The URL is intentionally preserved verbatim; resource and provider URLs are opaque handles resolved by the frontend.
func BuildImageInfoXML ¶
BuildImageInfoXML returns XML-tagged caption / ocr for one image.
func BuildImageInfoXMLWithURL ¶
BuildImageInfoXMLWithURL wraps image info in an <image> element carrying the URL.
func CaptureImageEvidence ¶
func CaptureImageEvidence(r *types.SearchResult)
CaptureImageEvidence freezes the original image address while ImageInfo still belongs to the image hit. Call it before any merge rewrites ImageInfo.
func ClampFloat ¶
ClampFloat clamps a float value to the specified range [minV, maxV].
func ClearImageInfoTextMatchingBody ¶
ClearImageInfoTextMatchingBody removes the OCR or caption field that exactly matches recognized from image_info. Merge re-attaches that body onto Content for image_ocr / image_caption hits; chat enrichment would otherwise inject the same text again from ImageInfo.
func CollectImageInfoByChunkIDs ¶
func CollectImageInfoByChunkIDs( ctx context.Context, chunkRepo interfaces.ChunkRepository, tenantID uint64, chunkIDs []string, ) map[string]string
CollectImageInfoByChunkIDs collects merged image_info JSON for each given chunk ID by querying child chunks (image_ocr / image_caption). It supports two-level resolution:
- If chunkIDs are text chunks, their direct children are image chunks → one query.
- If chunkIDs are parent_text chunks, their children are text chunks whose children are image chunks → two queries.
Disabled children are skipped at both levels. Removing an image from a chunk disables its image_ocr / image_caption children (syncEditedChunkImages), and disabling a text chunk disables them too; honoring that flag here is what keeps a deleted image out of retrieval, summaries and model context.
Returns a map of input chunkID → merged image_info JSON string.
func CollectImageInfoByChunkIDsOnly ¶
func CollectImageInfoByChunkIDsOnly( ctx context.Context, chunkRepo interfaces.ChunkRepository, chunkIDs []string, ) map[string]string
CollectImageInfoByChunkIDsOnly is the shared-KB variant of CollectImageInfoByChunkIDs: the chunk IDs can belong to an org-shared KB owned by another workspace, so the child lookup must not filter by the caller's tenant (#3342).
func ContainsChinese ¶
ContainsChinese reports whether text contains any CJK unified ideographs.
func ContainsChunkContent ¶
ContainsChunkContent reports whether the complete current body is safely represented by another body. Very short substrings are not treated as containment because common words and punctuation would create false drops.
func ContentOverlapRatio ¶
ContentOverlapRatio estimates how much of a's content overlaps with b by comparing their token sets (Jaccard-like but using overlap coefficient: |intersection| / |smaller set|). Both inputs should be raw content strings. Returns a value in [0, 1] where 1 means the smaller set is fully contained in the larger set.
func ContextImages ¶
func ContextImages( ctx context.Context, results []*types.SearchResult, read func(context.Context, *types.SearchResult) ([]byte, error), maxImages int, ) (images []string, positions []int)
ContextImages reads the images of the image-evidence results for a vision chat model: at most maxImages distinct images, in result order, prepared to VisionImageLimits, as data URIs. positions are the indices into results the images belong to. An image that cannot be read or prepared is skipped; the model still has its caption.
func ConvertWebSearchResults ¶
func ConvertWebSearchResults( webResults []*types.WebSearchResult, opts ...ConvertWebResultOption, ) []*types.SearchResult
ConvertWebSearchResults converts []*types.WebSearchResult into []*types.SearchResult.
func EnrichContentCaptionAndOCR ¶
EnrichContentCaptionAndOCR is like EnrichContentCaptionOnly but ALSO embeds OCR text alongside captions. URL and <image_original> wrapper blocks are deliberately omitted (unlike EnrichContentWithImageInfo) — the summary LLM only needs the human-readable text, not opaque export hashes. Used as a fallback for image-dominated documents where caption alone carries too little signal.
func EnrichContentCaptionOnly ¶
EnrichContentCaptionOnly is like EnrichContentWithImageInfo but only includes image captions (no OCR text). Original content (including Markdown image links) is preserved. Useful for summary generation where OCR would add too much noise.
func EnrichContentWithImageInfo ¶
EnrichContentWithImageInfo embeds image info as XML tags into text content. Inline Markdown image links get wrapped in <image> with <image_caption> / <image_ocr>; images not found in content are appended as <image> blocks.
func EnrichContentWithImageInfoForChat ¶
EnrichContentWithImageInfoForChat enriches matching Markdown images with caption / OCR text while keeping the image itself as Markdown. Chat context is deliberately answer-ready: if a model copies a relevant image from its context, the copied content should still render instead of leaking the internal <image> XML protocol into the answer.
Only images with a matching image_info entry are enriched. This avoids adding every parent thumbnail when only a few pages were retrieved, and skips orphan image_info extras.
func EnrichSearchResultsImageInfo ¶
func EnrichSearchResultsImageInfo( ctx context.Context, chunkRepo interfaces.ChunkRepository, tenantID uint64, results []*types.SearchResult, )
EnrichSearchResultsImageInfo fills in ImageInfo for SearchResults that have none by batch-querying child image chunks. The same children give scanned PDF pages their recognized text as locator quotes, so a citation can be matched to the page it came from.
func EnrichSearchResultsImageInfoOnly ¶
func EnrichSearchResultsImageInfoOnly( ctx context.Context, chunkRepo interfaces.ChunkRepository, results []*types.SearchResult, )
EnrichSearchResultsImageInfoOnly is the shared-KB variant of EnrichSearchResultsImageInfo for results that may come from an org-shared KB owned by another workspace. The result IDs must already be authorized (they come from retrieval over KBs the caller may read).
func ExtractSnippet ¶
ExtractSnippet matches Agent wiki_search: ~60 runes of context around the first case-insensitive regex hit, match itself capped at 100. Invalid patterns return an empty string.
func FilterImageInfoByContentURLs ¶
FilterImageInfoByContentURLs keeps only image_info entries whose URL or OriginalURL is referenced by content, as either a Markdown image link or an HTML <img> tag. Returns empty string when nothing matches or imageInfoJSON is invalid.
func FilterImageInfoByMatchRange ¶
func FilterImageInfoByMatchRange( parentContent string, parentStartAt, matchStart, matchEnd int, imageInfoJSON string, ) string
FilterImageInfoByMatchRange keeps image_info entries whose image reference falls within the document rune range [matchStart, matchEnd). Used when parent content is expanded for context but multimodal enrichment should only cover the retrieved child window.
func ImageURLsFromInfo ¶
ImageURLsFromInfo returns every current and original URL represented by an image_info payload.
func ImageURLsInContent ¶
ImageURLsInContent returns the set of image URLs referenced by content, covering both Markdown image links and HTML <img> tags with a quoted src.
func InheritImageEvidence ¶
func InheritImageEvidence(kept, dropped *types.SearchResult)
InheritImageEvidence marks kept as resting on an image vector match when dropped, a copy de-duplication removed in its favour, did. An image hit and its caption hit carry the same text once merged, so either can be the one kept, and the image must not go with the dropped one.
func IsContentContained ¶
IsContentContained reports whether the normalized form of short is a substring of the normalized form of long. Both inputs must already be normalized via NormalizeContent.
func IsImageEvidence ¶
func IsImageEvidence(r *types.SearchResult) bool
IsImageEvidence reports whether a result rests on an image matched by its own vector: an image_vector hit, or the copy that stands in for one.
func IsKeptOutsideTopK ¶
func IsKeptOutsideTopK(r *types.SearchResult) bool
IsKeptOutsideTopK reports a result kept for a reason other than its score, which a top-k cut must not drop. See types.MetadataKeptBy.
func Jaccard ¶
Jaccard calculates Jaccard similarity between two token sets. Returns a value between 0 and 1, where 1 means identical sets.
func JoinChunkContent ¶
JoinChunkContent joins two current chunk bodies without relying on parser offsets. Exact containment is collapsed, a real suffix/prefix overlap is removed, and otherwise both bodies are retained with separator between them. The conservative fallback intentionally prefers small duplication over silently dropping edited content.
func MergeImageInfoJSON ¶
MergeImageInfoJSON combines per-chunk image_info JSON strings (from CollectImageInfoByChunkIDs) into a single JSON array, deduplicating by URL.
func MergeTextChunks ¶
MergeTextChunks 按 StartAt（并列时按 ChunkIndex）排序后，用 AppendWithOverlap 把多个 chunk 的内容重建为完整文本。gapSep 用于位置不相邻（有间隙）的两段之间 的分隔符（如 "\n"），传空串则直接拼接。
调用方负责先做类型过滤（例如只保留文本 chunk）；本函数不感知 ChunkType。
func NormalizeContent ¶
NormalizeContent returns a lowercased, whitespace-collapsed version of s suitable for containment and overlap checks.
func NormalizeKeywordScores ¶
func NormalizeKeywordScores[T any]( results []T, isKeyword func(T) bool, getScore func(T) float64, setScore func(T, float64), callbacks KeywordScoreCallbacks, )
NormalizeKeywordScores normalizes keyword match scores in-place using robust percentile bounds.
func PruneMarkdownImagesByImageInfo ¶
PruneMarkdownImagesByImageInfo keeps only Markdown images represented by the chunk-scoped image metadata. This is stable after text edits because it matches durable image URLs instead of parser character offsets.
func PruneMarkdownImagesOutsideRange ¶
func PruneMarkdownImagesOutsideRange( content string, contentStartAt, matchStart, matchEnd int, ) string
PruneMarkdownImagesOutsideRange removes Markdown image lines whose document-level rune offsets fall outside [matchStart, matchEnd). Non-image text is preserved so parent-child expansion still provides full textual context while dropping irrelevant page thumbnails.
func SliceContentByDocumentRange ¶
SliceContentByDocumentRange extracts the substring of content whose document-level rune offsets fall within [rangeStart, rangeEnd). contentStartAt is the document offset where content begins.
func TokenOverlapRatio ¶
TokenOverlapRatio is ContentOverlapRatio over token sets already built with TokenizeSimple, for callers comparing one text against many.
func TokenizeSimple ¶
TokenizeSimple tokenizes text into a set of unique tokens. For text containing Chinese characters, it uses jieba segmentation for accurate word boundaries. For pure non-Chinese text, it falls back to whitespace-based splitting. Returns a map where keys are lowercase tokens with rune length > 1.
func TopKKeepingKept ¶
func TopKKeepingKept(results []*types.SearchResult, k int) []*types.SearchResult
TopKKeepingKept returns the first k of the ranked results, followed by the results kept outside top-k, each group in its input order.
Types ¶
type ConvertWebResultOption ¶
type ConvertWebResultOption func(*convertWebResultOptions)
ConvertWebResultOption configures ConvertWebSearchResults behavior.
func WithSeqFunc ¶
func WithSeqFunc(f func(idx int) int) ConvertWebResultOption
WithSeqFunc overrides the default sequence assignment for converted results.
Source Files