Documentation
¶
Index ¶
- Constants
- func SimhashCompareItems(Item1, Item2 *Item) uint8
- type CsvItem
- type IntelligenceXAPI
- func (api *IntelligenceXAPI) DownloadZip(ctx context.Context, searchID uuid.UUID, Limit int, OutputFile string) (err error)
- func (api *IntelligenceXAPI) FilePreview(ctx context.Context, item *Item) (text string, err error)
- func (api *IntelligenceXAPI) FileRead(ctx context.Context, item *Item, Limit int64) (data []byte, err error)
- func (api *IntelligenceXAPI) Init(URL string, Key string)
- func (api *IntelligenceXAPI) Search(ctx context.Context, Selector string, Sort, Limit int, ...) (records []SearchResult, selectorInvalid bool, err error)
- func (api *IntelligenceXAPI) SearchGetResults(ctx context.Context, searchID uuid.UUID, Limit int) (records []SearchResult, status int, err error)
- func (api *IntelligenceXAPI) SearchGetResultsAll(ctx context.Context, searchID uuid.UUID, Limit int, Timeout time.Duration) (records []SearchResult, err error)
- func (api *IntelligenceXAPI) SearchStart(ctx context.Context, Term string) (searchID uuid.UUID, selectorInvalid bool, err error)
- func (api *IntelligenceXAPI) SearchStartAdvanced(ctx context.Context, Input IntelligentSearchRequest) (searchID uuid.UUID, selectorInvalid bool, err error)
- func (api *IntelligenceXAPI) SearchTerminate(ctx context.Context, searchID uuid.UUID) (err error)
- func (api *IntelligenceXAPI) SearchWithDates(ctx context.Context, Selector string, DateFrom, DateTo time.Time, ...) (searchID *uuid.UUID, records []SearchResult, selectorInvalid bool, err error)
- func (api *IntelligenceXAPI) SetAPIKey(URL string, Key string)
- type IntelligentSearchRequest
- type IntelligentSearchResponse
- type IntelligentSearchResult
- type Item
- type PanelSearchResultTag
- type Relationship
- type SearchResult
- type Tag
- type WriteCounter
Constants ¶
const ( SortNone = 0 // No sorting SortXScoreAsc = 1 // X-Score ascending = Least relevant first SortXScoreDesc = 2 // X-Score descending = Most relevant first SortDateAsc = 3 // Date ascending = Oldest first SortDateDesc = 4 // Date descending = Newest first )
Sort orders
const DefaultTimeoutGetResults = 20 * time.Second
DefaultTimeoutGetResults is the suggested timeout after which the search will be terminated.
const DefaultWaitSortTime = 400 * time.Millisecond
DefaultWaitSortTime is the suggested time to give the API to process and sort all the results, before the client queries them.
const TagLanguage = 0
TagLanguage is ISO 639-1 defined
Variables ¶
This section is empty.
Functions ¶
func SimhashCompareItems ¶
SimhashCompareItems compares 2 items for data equalness and returns the hamming distance. The closer to 0 the more equal they are. Never compare Simhashes directly because with different content types and even on the same type with different encoding they use different algorithms.
Types ¶
type IntelligenceXAPI ¶
type IntelligenceXAPI struct {
URL string // The API URL. Always ending with slash.
Key uuid.UUID // The API key assigned by Intelligence X. Contact the company to receive one.
// additional input. Set before calling Init
ProxyURL string // Proxy to use
BindToIP string // Bind to a specific IPv4 or IPv6
// one client for the session
Client http.Client
RetryAttempts int // in case of underlying transport failure
UserAgent string
HTTPMaxResponseSize int64
HTTPTimeout time.Duration
WriteCounter *WriteCounter
}
IntelligenceXAPI holds all information for communicating with the Intelligence X API. Call Init() first.
func (*IntelligenceXAPI) DownloadZip ¶ added in v0.1.8
func (api *IntelligenceXAPI) DownloadZip(ctx context.Context, searchID uuid.UUID, Limit int, OutputFile string) (err error)
DownloadZip reads the data of an item.
func (*IntelligenceXAPI) FilePreview ¶
FilePreview loads the preview of an item. Previews are always capped at 1000 characters.
func (*IntelligenceXAPI) FileRead ¶
func (api *IntelligenceXAPI) FileRead(ctx context.Context, item *Item, Limit int64) (data []byte, err error)
FileRead reads the data of an item.
func (*IntelligenceXAPI) Init ¶
func (api *IntelligenceXAPI) Init(URL string, Key string)
Init initializes the IX API. URL and Key may be empty to use defaults.
func (*IntelligenceXAPI) Search ¶
func (api *IntelligenceXAPI) Search(ctx context.Context, Selector string, Sort, Limit int, WaitSort, TimeoutGetResults time.Duration) (records []SearchResult, selectorInvalid bool, err error)
Search starts a search and queries all results. It takes a selector as input. WaitSort should be a few hundred ms, giving the API time to sort the results before querying them. Limit is the max count of results to query per bucket. The total number of results returned might be higher. TimeoutGetResults is the max amount of time for querying all results. This should be at least a few seconds but a timeout of 10-30 seconds makes sense. If the input selector is invalid (not a strong selector), the function returns selectorInvalid set to true with no error reported.
func (*IntelligenceXAPI) SearchGetResults ¶
func (api *IntelligenceXAPI) SearchGetResults(ctx context.Context, searchID uuid.UUID, Limit int) (records []SearchResult, status int, err error)
SearchGetResults returns results Status: 0 = Success with results (continue), 1 = No more results available (this response might still have results), 2 = Search ID not found, 3 = No results yet available keep trying, 4 = Error
func (*IntelligenceXAPI) SearchGetResultsAll ¶
func (api *IntelligenceXAPI) SearchGetResultsAll(ctx context.Context, searchID uuid.UUID, Limit int, Timeout time.Duration) (records []SearchResult, err error)
SearchGetResultsAll returns all results up to Limit and up to the given Timeout. It will automatically terminate the search before returning. Unless the underlying API requests report and error, no error will be returned. Deadline exceeded is treated as no error.
func (*IntelligenceXAPI) SearchStart ¶
func (api *IntelligenceXAPI) SearchStart(ctx context.Context, Term string) (searchID uuid.UUID, selectorInvalid bool, err error)
SearchStart starts a search
func (*IntelligenceXAPI) SearchStartAdvanced ¶
func (api *IntelligenceXAPI) SearchStartAdvanced(ctx context.Context, Input IntelligentSearchRequest) (searchID uuid.UUID, selectorInvalid bool, err error)
SearchStartAdvanced starts a search and allows the caller to set any advanced filter
func (*IntelligenceXAPI) SearchTerminate ¶
SearchTerminate terminates a search
func (*IntelligenceXAPI) SearchWithDates ¶
func (api *IntelligenceXAPI) SearchWithDates(ctx context.Context, Selector string, DateFrom, DateTo time.Time, Sort, Limit int, WaitSort, TimeoutGetResults time.Duration) (searchID *uuid.UUID, records []SearchResult, selectorInvalid bool, err error)
SearchWithDates starts a search and queries all results. It takes a selector and dates as input. Sorting is newest first. WaitSort should be a few hundred ms, giving the API time to sort the results before querying them. Limit is the max count of results to query per bucket. The total number of results returned might be higher. TimeoutGetResults is the max amount of time for querying all results. This should be at least a few seconds but a timeout of 10-30 seconds makes sense.
func (*IntelligenceXAPI) SetAPIKey ¶
func (api *IntelligenceXAPI) SetAPIKey(URL string, Key string)
SetAPIKey sets the API URL and Key. URL and Key may be empty to use defaults.
type IntelligentSearchRequest ¶
type IntelligentSearchRequest struct {
Term string `json:"term"` // Search term submitted by the user, e.g. "Document 1.docx" or "email@example.com"
Buckets []string `json:"buckets"` // Bucket identifiers
Timeout time.Duration `json:"timeout"` // Timeout in seconds. May be limited by API config. 0 means default.
MaxResults int `json:"maxresults"` // Total number of max results per bucket. May be limited by API config. 0 means default.
DateFrom string `json:"datefrom"` // Date from, both from/to are required if set, format "2006-01-02 15:04"
DateTo string `json:"dateto"` // Date to, both from/to are required if set, format "2006-01-02 15:04"
Sort int `json:"sort"` // Sort order: 0 = no sorting, 1 = X-Score ASC, 2 = X-Score DESC, 3 = Date ASC, 4 = Date DESC
Media int `json:"media"` // Media: 0 = not defined, otherwise MediaX as defined in ixservice
TerminateID []uuid.UUID `json:"terminate"` // Optional: Previous search IDs to terminate (normal search or Phonebook). This is if the user makes a new search from the same tab. Same as first calling /intelligent/search/terminate.
}
IntelligentSearchRequest is the information from the human for the search.
type IntelligentSearchResponse ¶
type IntelligentSearchResponse struct {
ID uuid.UUID `json:"id"` // id of the search job. This is used to get the results.
SoftSelectorWarning bool `json:"softselectorwarning"` // Warning of soft selectors, typically garbage in which results into garbage out
Status int `json:"status"` // Status of the search: 0 = Success (ID valid), 1 = Invalid Term, 2 = Error Max Concurrent Searches
}
IntelligentSearchResponse is the result to the initial search request
type IntelligentSearchResult ¶
type IntelligentSearchResult struct {
Records []SearchResult `json:"records"` // The result records
Status int `json:"status"` // Status: 0 = Success with results, 1 = No more results available, 2 = Search ID not found, 3 = No results yet available keep trying
}
IntelligentSearchResult contains the result items
type Item ¶
type Item struct {
ID uint `json:"id" gorm:"primarykey"`
SystemID string `gorm:"column:system_id;index:,unique;" json:"systemid"` // System identifier uniquely identifying the item
StorageID string `json:"storageid"` // Storage identifier, empty if not stored/available, otherwise a 64-byte blake2b hash hex-encoded
InStore bool `json:"instore"` // Whether the data of the item is in store and the storage id is valid. Also used to indicate update when false but storage id is set.
Size int64 `json:"size"` // Size in bytes of the item data
AccessLevel int `json:"accesslevel"` // Native access level of the item (0 = Public..)
Type int `json:"type"` // Low-level content type (0 = Binary..)
Media int `json:"media"` // High-level media type (User, Paste, Tweet, Forum Post..)
Added time.Time `json:"added"` // When the item was added to the system
Date time.Time `json:"date"` // Full time stamp item when it was discovered or created
Name string `json:"name"` // Name or title
Description string `json:"description"` // Full description, text only
XScore int `json:"xscore"` // X-Score, ranking its relevancy. 0-100, default 50
Simhash uint64 `json:"simhash"` // Simhash, depending on content type. Use hamming distance to compare equality of items data.
Bucket string `json:"bucket"` // Bucket
Filename string `json:"filename"` // FileName
Downloaded bool `json:"downloaded"` // If file already downloaded
// Tags are meta-data tags helping in classification of the items data. They reveal for example the language or a topic. Different to key-values they have hard-coded classes that
// allow anyone to take action on them.
Tags []Tag `json:"tags"`
// Relations lists all related items.
Relations []Relationship `json:"relations"`
}
Item represents any items meta-data. It origins from Indexed and is sent as search results. All fields except the identifier are optional and may be zero. It is perfectly valid that a service only knows partial information (like a name or storage id) of a given item.
func (*Item) BeforeCreate ¶ added in v0.1.23
func (*Item) GetExtension ¶
type PanelSearchResultTag ¶
type PanelSearchResultTag struct {
ID uint `json:"id" gorm:"primarykey"`
SearchResultID uint `json:"search_result_id"`
Class int16 `json:"class"` // Class of tag
ClassH string `json:"classh"` // Class of tag, human friendly
Value string `json:"value"` // The value
ValueH string `json:"valueh"` // Value, human friendly
}
PanelSearchResultTag represents a tag in human form.
func (PanelSearchResultTag) TableName ¶
func (PanelSearchResultTag) TableName() string
type Relationship ¶
type Relationship struct {
ID uint `json:"id" gorm:"primarykey"`
ItemID uint `json:"item_id";gorm:"uniqueIndex:idx_relation_target_r"`
Target string `json:"target";gorm:"uniqueIndex:idx_relation_target_r"` // Target item systemid
Relation int `json:"relation";gorm:"uniqueIndex:idx_relation_target_r"` // The relationship, see RelationX
}
Relationship defines a relation between 2 items.
func (Relationship) TableName ¶
func (Relationship) TableName() string
type SearchResult ¶
type SearchResult struct {
Item
AccessLevelH string `json:"accesslevelh"` // Human friendly access level info
MediaH string `json:"mediah"` // Human friendly media type info
SimhashH string `json:"simhashh"` // Human friendly simhash
TypeH string `json:"typeh"` // Human friendly content type info
TagsH []PanelSearchResultTag `json:"tagsh"` // Human friendly tags
RandomID string `json:"randomid"` // Random ID
BucketH string `json:"bucketh"` // Human friendly bucket name
Group string `json:"group"` // File Group
IndexFile string `json:"indexfile"` // Index file ID
}
SearchResult represents a single result record. The entire record IS the de-facto result. Every field is optional and may be empty.
func (*SearchResult) GetCsv ¶
func (sr *SearchResult) GetCsv() *CsvItem
func (SearchResult) TableName ¶
func (SearchResult) TableName() string
type Tag ¶
type Tag struct {
ID uint `json:"id" gorm:"primarykey"`
ItemID uint `json:"item_id";gorm:"uniqueIndex:idx_tag_class_v"`
Class int16 `json:"class";gorm:"uniqueIndex:idx_tag_class_v"` // Class of tag
Value string `json:"value";gorm:"uniqueIndex:idx_tag_class_v"` // The value
}
Tag classifies the items data
type WriteCounter ¶ added in v0.1.8
type WriteCounter struct {
Total uint64
}
WriteCounter counts the number of bytes written to it. It implements to the io.Writer interface and we can pass this into io.TeeReader() which will report progress on each write cycle.