Documentation
¶
Overview ¶
Documentation for the normalized metadata-representation that Indico-Proxy uses
The goal of this definition is to support a broad range of metadata that a client can produce, and then be able to map that data back to any of the supported backends.
All metadata-fields should be defined in close cooperation with backend-providers, as well as clients.
Definitions ¶
The following definitions are used throughout this document.
Backend: Any server that can receive files, and preferably, some metadata as well. Client: Any software that can produce files and metadata and provide it to Proxy. External ID: A Unique identifier on the backend of a file. Parent ID: A unique identifier on the backend of a collection of files, like a folder, or an album Case ID: A unique identifier for a case on of a case that the file belong to.
Clients and Backends do not need to know anything about each other, but the client needs to know about Proxy.
Client-integrations ¶
Indico provides a DotNET-SDK. If you want to create a custom-client, we recommend starting by using a compatible library from https://tus.io and then extending it with the custom-layer that Proxy adheres to. Proxy uses the TUS-protocol for all uploads.
Below, a basic flow for creating and uploading a new file is provided.
Creating files ¶
A file must be created, with all its metadata prior to upload. The metadata will be validated internally, and with the backend. Any validation-failures will be returned to the client.
Example of error returned:
{
"type": "https://httpstatuses.com/422",
"title": "Unprocessable Entity",
"status": 422,
"detail": "The following fields are missing: parentid",
"instance": ""
}
The size of each chunk can be set by the client, but we recommend at least 6 MB. Each chunk can optionally be verified by providing the checksum of the current chunk.
POST-ing metadata as JSON to `/create`.
A unique upload-url should have been returned in the previous step. Uploads can now begin, with chunk-uploads preferred for large uploads. This upload-url should be stored on the client at least until the upload is completed. The upload-url can also be used to update metadata, if the backend supports it.
Uploading file ¶
Info received from backend ¶
A backend will provide its External ID, Parent ID and Case ID (if supported) of the file, but the time of which it reports this information so may change from one backend to another. The client should accommodate this. It may return the data in the following events.
1. The client creates a new upload.
2. The client completes an upload
3. As a last resort, information can be available at a later time, and the client needs to accommodate this, if it wants to receive this information. The information will be available when the client completes an upload and the file is also sent to the backend. The client then needs to ask Proxy if the information is available.
Information can be retrieved by using a HEAD-request to the upload-url, or to get the information as JSON, A GET-request can be sent to the upload-url prepended by `info`.
Updating metadata ¶
The client may optionally, if the backend supports it, update the metadata. Proxy requires that the client sends in ALL the metadata.
Updates are done in the same way as creating files, as described above, but the url is `update`
Index ¶
- Constants
- Variables
- func AssignMetaLogger(logger logrus.FieldLogger)
- func Debug(enable bool)
- type Annotation
- type Bookmark
- type CheckSum
- type ClientMessage
- type Creator
- type EventKind
- type FormFields
- type GenderType
- type InternalInfoStr
- type Location
- type Mapper
- type MetaChecksum
- type Metadata
- func (m *Metadata) AppendClientMessage(cMsg ClientMessage)
- func (m *Metadata) Apply(info *tusd.FileInfo) tusd.FileInfo
- func (m *Metadata) GetClientId() string
- func (m *Metadata) GetClientMessages() (cMsgs []ClientMessage)
- func (m *Metadata) GetConnectorConfig() (map[string]string, error)
- func (m *Metadata) GetConnectorWritten() int64
- func (m *Metadata) GetErrorMessage() string
- func (m *Metadata) GetExtConfirmed() bool
- func (m *Metadata) GetExtId() string
- func (m *Metadata) GetExtParentId() string
- func (m *Metadata) GetExtUploaded() bool
- func (m *Metadata) GetFilename() string
- func (m *Metadata) GetNoMetadataMapping() bool
- func (m *Metadata) GetPostponedStatus() postponedStatus
- func (m *Metadata) GetRaw(k string) string
- func (m *Metadata) GetRawMetadata() string
- func (m *Metadata) GetReceiverChecksum() (CheckSum, error)
- func (m *Metadata) GetReqId() string
- func (m *Metadata) GetServiceQueueId() string
- func (m *Metadata) GetTemporaryChecksum() string
- func (m *Metadata) GetUploadMetadata() UploadMetadata
- func (m *Metadata) ReplaceUploadMetadata(um UploadMetadata)
- func (m *Metadata) SaveToInfo(info *tusd.FileInfo)
- func (m *Metadata) SetAsActiveDirectoryUserSid(sid string) *Metadata
- func (m *Metadata) SetAsUserId(id string) *Metadata
- func (m *Metadata) SetAsUserName(u string) *Metadata
- func (m *Metadata) SetClientId(cid string) *Metadata
- func (m *Metadata) SetConnectorConfig(cfg map[string]string)
- func (m *Metadata) SetConnectorWritten(written int64)
- func (m *Metadata) SetErrorMessage(msg string) *Metadata
- func (m *Metadata) SetExtConfirmed() *Metadata
- func (m *Metadata) SetExtId(d string)
- func (m *Metadata) SetExtParentId(d string)
- func (m *Metadata) SetExtUploaded() *Metadata
- func (m *Metadata) SetNoMetadataMapping(boolean bool)
- func (m *Metadata) SetPostponedStatus(status postponedStatus) error
- func (m *Metadata) SetRaw(k string, value string)
- func (m *Metadata) SetReceiverChecksum(value, kind, code, notes string) (CheckSum, error)
- func (m *Metadata) SetReqId(reqid string)
- func (m *Metadata) SetServiceQueueId(qID string)
- func (m *Metadata) SetTemporaryChecksum(reqid string)
- type Parent
- type Person
- type RecordingEvent
- type RecordingEvents
- type StringMap
- type Upl
- type UploadMetadata
- type Utterance
- type UtteranceType
- type ValidationRule
Constants ¶
const ( // The ID of the user the file belongs to. UserId = "userid" // Used by authenticators ClientId = "client-id" ConnectorWritten = "connector-written" ReqId = "req-id" TempChecksum = "_tmp_checksum_" PostponeStatus = "postpone-status" AsUserName = "as-username" AsUserId = "as-user-id" AsUserActiveDirectorySid = "as-user-sid" // The name of the container the file belongs to. ParentName = "parentname" SSN = "__ssn" NoMetadataMapping = "__no_metadata_mapping" ClientMediaId = "client-media-id" // The mime type of the file. FileType = "filetype" // A svcQueue-id in an installations svcQueue-system. Only valid for some systems, like Indico Gateway. ServiceQueueId = "serviceQueueId" // Deprecated ConnectorConfig, if needed for the svcQueue-handling ConnectorConfig = "connectorConfig" // The name given to the file by the user. DisplayName = "displayname" // The Checksum of the file. Checksum = "checksum" Filename = "filename" ErrorMessage = "errormsg" ExtId = "extid" ExtParentId = "extParentid" // Indicates whether the upload is verfied as completed. ExtUploaded = "extUploaded" ExtConfirmed = "extConfirmed" // A checkSum calculated by the receiver ReceiverChecksum = "receiverChecksum" // The data available to all, as submitted by the Client MUploadMetadata = "UploadMetadata" ClientMessages = "ClientMessages" CaseNumberIgnored InternalInfoStr = "CaseNumberIgnored" )
const ( PostponedInit postponedStatus = "init" PostponedUploadComplete postponedStatus = "uploadComplete" PostponedMetaComplete postponedStatus = "metaComplete" PostponedAllComplete postponedStatus = "allComplete" )
const ( RecordingEventRecording EventKind = "Recording" RecordingEventIncomingCall EventKind = "IncomingCall" RecordingEventLowPower EventKind = "LowPower" RecordingEventLowMemory EventKind = "LowMemory" RecordingEventFatal EventKind = "Fatal" RecordingEventUnknown EventKind = "Unknown" RecordingEventUserEnded EventKind = "UserEnded" RecordingEventStarted EventKind = "Started" RecordingEventPaused EventKind = "Paused" Saying UtteranceType = "Saying" Event UtteranceType = "Event" )
Variables ¶
var (
ErrRecordingEventInvalidOrder error = errors.New("the order of the events are logically correct")
)
Functions ¶
func AssignMetaLogger ¶
func AssignMetaLogger(logger logrus.FieldLogger)
Types ¶
type Annotation ¶
type Annotation struct {
CreationDate time.Time `json:"createdAt"`
ID string `json:"id"`
Title string `json:"title"`
Type string `json:"type"`
// Data will be a computer-friendly structure. The structure itself is not decided yet.
Data StringMap `json:"data"`
// Position on the image/video.
X1 int `json:"x1"`
X2 int `json:"x2"`
Y1 int `json:"y1"`
Y2 int `json:"y2"`
// Position, in milliseconds
StartPosition int `json:"startPosition"`
// EndPosition, in milliseconds
EndPosition int `json:"endPosition"`
}
Annotations can be small figures, highlights, descriptors on an image/vidoe/audio. They can be added by the user, or automatically through OCR or Machine Learning. For audio, it can also be a way to censor parts of the audio via for instance beeps.
type Bookmark ¶
type Bookmark struct {
CreationDate time.Time `json:"creationTime"`
ID string `json:"id"`
Title string `json:"title"`
// Position, in milliseconds
StartPosition int `json:"startPosition"`
// EndPosition, in milliseconds
EndPosition int `json:"endPosition"`
}
Bookmark object belonging to a certain recording.
type ClientMessage ¶
type ClientMessage struct {
Kind InternalInfoStr
Message string
}
type FormFields ¶
type FormFields struct {
// The metadata-key.
Key string `json:"key"`
// The id of the field, where available.
FieldId string `json:"fieldId"`
// The key used to get the translated VisualName, where available.
TranslationKey string `json:"translationKey"`
// The visual name, as reported by the client.
VisualName string `json:"visualName"`
// The value of the field, e.g. the user-input
Value string `json:"value"`
// Marks whether or not field is required or not.
Required bool `json:"required"`
// The kind of data in the Value-field.
DataType string `json:"dataType"`
ValidationRule ValidationRule `json:"validationRule"`
}
FormFields er unmapped metadata. In Indico Gateway, these represent an organizational-form.
type GenderType ¶
type GenderType string
const ( GenderFemale GenderType = "Female" GenderMale GenderType = "Male" GenderOther GenderType = "Other" GenderUnspecified GenderType = "" )
func ToGender ¶
func ToGender(s string) GenderType
type InternalInfoStr ¶
type InternalInfoStr string
type Location ¶
type Location struct {
// The text for the current location, like the current address, city, etc.
Text string `json:"text"`
// Geo-location
Latitude float64 `json:"latitude"`
// Geo-location
Longitude float64 `json:"longitude"`
Address string `json:"address"`
Address2 string `json:"address2"`
ZipCode string `json:"zipCode"`
PostArea string `json:"postArea"`
Country string `json:"country"`
Accuracy float64 `json:"accuracy"`
Altitude float64 `json:"altitude"`
}
type MetaChecksum ¶
type Metadata ¶
func (*Metadata) AppendClientMessage ¶
func (m *Metadata) AppendClientMessage(cMsg ClientMessage)
Appends ClientMessages, but ensures uniqueness.
func (*Metadata) GetClientId ¶
func (*Metadata) GetClientMessages ¶
func (m *Metadata) GetClientMessages() (cMsgs []ClientMessage)
func (*Metadata) GetConnectorConfig ¶
Deprecated
func (*Metadata) GetConnectorWritten ¶
func (*Metadata) GetErrorMessage ¶
func (*Metadata) GetExtConfirmed ¶ added in v0.1.5
func (*Metadata) GetExtParentId ¶
func (*Metadata) GetExtUploaded ¶
func (*Metadata) GetFilename ¶
func (*Metadata) GetNoMetadataMapping ¶ added in v0.3.0
func (*Metadata) GetPostponedStatus ¶ added in v0.4.1
func (m *Metadata) GetPostponedStatus() postponedStatus
func (*Metadata) GetRawMetadata ¶
func (*Metadata) GetReceiverChecksum ¶ added in v0.2.0
func (*Metadata) GetServiceQueueId ¶
func (*Metadata) GetTemporaryChecksum ¶ added in v0.4.1
func (*Metadata) GetUploadMetadata ¶
func (m *Metadata) GetUploadMetadata() UploadMetadata
Returns uploadMetadata, which contains all the information about the file that a connector should need.
func (*Metadata) ReplaceUploadMetadata ¶
func (m *Metadata) ReplaceUploadMetadata(um UploadMetadata)
func (*Metadata) SaveToInfo ¶
func (*Metadata) SetAsActiveDirectoryUserSid ¶
func (*Metadata) SetAsUserId ¶
func (*Metadata) SetAsUserName ¶
func (*Metadata) SetClientId ¶
func (*Metadata) SetConnectorConfig ¶
Deprecated
func (*Metadata) SetConnectorWritten ¶
func (*Metadata) SetErrorMessage ¶
func (*Metadata) SetExtConfirmed ¶ added in v0.1.5
func (*Metadata) SetExtParentId ¶
func (*Metadata) SetExtUploaded ¶
func (*Metadata) SetNoMetadataMapping ¶ added in v0.3.0
func (*Metadata) SetPostponedStatus ¶ added in v0.4.1
func (*Metadata) SetReceiverChecksum ¶ added in v0.2.0
func (*Metadata) SetServiceQueueId ¶
func (*Metadata) SetTemporaryChecksum ¶ added in v0.4.1
type Person ¶
type Person struct {
FirstName string `json:"firstName"`
LastName string `json:"lastName"`
Id string `json:"id"`
// Date of birth
Dob *time.Time `json:"dob"`
Gender GenderType `json:"gender"`
Nationality string `json:"nationality"`
Workplace string `json:"workplace"`
// TBD
Status string `json:"status"`
WorkPhone string `json:"workPhone"`
Phone string `json:"phone"`
Mobile string `json:"mobile"`
// TBD
Present bool `json:"isPresent"`
Location
}
type RecordingEvent ¶ added in v0.5.0
type RecordingEvent struct {
Id string
// Optional
Title string
// Optional
Description string
// Required
Kind EventKind
// Required
Start time.Time
// Optional
StartFrame int64
// Optional
End *time.Time
// Optional
EndFrame *int64
// Optional
EndKind *EventKind
// Optional
EndId *string
}
RecordingEvent uses
type RecordingEvents ¶ added in v0.5.0
type RecordingEvents []RecordingEvent
func (RecordingEvents) WithEndings ¶ added in v0.5.0
func (re RecordingEvents) WithEndings() (RecordingEvents, error)
Returns a new compact RecordingEvents by combining RecordingEvents which belong together in Start/End If the events are already using the End-fields, this has no effect
func (RecordingEvents) WithEndingsIgnoreErr ¶ added in v0.5.0
func (re RecordingEvents) WithEndingsIgnoreErr() RecordingEvents
type StringMap ¶
func (StringMap) MarshalXML ¶
StringMap marshals into XML.
type Upl ¶
type Upl struct {
UploadMetadata
}
type UploadMetadata ¶
type UploadMetadata struct {
// A unique identifier for the user in the backend-system
UserId string `json:"userId"`
// Active-Directory-user, if available
AdSid string `json:"adSid"`
// Active-Directory-user, if available, in the format 'user@domainname@
AdLogin string `json:"adLogin"`
// The parent of the current media.
Parent Parent `json:"parent"`
// The time of which the media was created in the backend-database
CreatedAt *time.Time `json:"createdAt"`
// The time of which the media was last updated in the backend-database
UpdatedAt *time.Time `json:"updatedAt"`
// Is the item marked as archived, meaning it is marked to be deleted (soft-deleted). If null, the item is
// not scheduled for deletion, if a date is set, the item is marked for deletion at that time.
ArchiveAt *time.Time `json:"archiveAt"`
// The date of which the item was marked as completed, E.g. the case was closed.
CompletedAt *time.Time `json:"completedAt"`
// The time of which the media was created by the user, on the client.
CapturedAt *time.Time `json:"capturedAt"`
// The fileType, as in MimeType. Example: 'image/jpeg' or 'video/mp4'
FileType string `json:"fileType"`
// A short description of the current file, submitted by the user
FileSize int64 `json:"fileSize"`
DisplayName string `json:"displayName"`
// A longer description of the current file, submitted by the uer.
Description string `json:"description"`
// Any checksums already calculated by the client.
Checksum []MetaChecksum `json:"checksum"`
// The filename,
FileName string `json:"fileName"`
// Tags
Tags []string `json:"tags"`
// The backend-database ID of the current file. Provide if updating metadata.
ExtId string `json:"extId"`
// A case-number returned by the user
CaseNumber string `json:"caseNumber"`
// Duration of the media, if audio or video. Int64. Should be in milliseconds when sending.
Duration int64 `json:"duration"`
// The creator of the current file, as in the current user, interviewer etc.
Creator Creator `json:"creator"`
// The location of the captured media
Location Location `json:"location"`
// Any subjects in the captured media.
Subject []Person `json:"subject"`
// TBD
EquipmentId string `json:"equipmentId"`
// TBD
InterviewType string `json:"interviewType"`
Bookmarks []Bookmark `json:"bookmarks"`
RecordingEvents RecordingEvents `json:"recordingEvents"`
Annotations []Annotation `json:"annotations"`
// TBD
Notes string `json:"notes"`
// A unique identifier of the file on the client.
ClientMediaId string `json:"clientMediaId"`
// ID of any backend-provided Group-id
GroupId string `json:"groupId"`
// Name of any backend-group. Providing it will c create a groupName, if supported by the backend.
GroupName string `json:"groupName"`
// Any custom-field. Should only be used for customer-specific fields that do not fit in any other field. Before use, please request Indico to add your required fields.
FormFields []FormFields `json:"formFields"`
// Transcribed audio/video-details
Transcription []Utterance `json:"transcription"`
}
UploadMetadata is the internal representation for Metadata on uploads.
All time.time-objects should be ISO-8601-compliant, and include the clients time-offset. The required metadata depends on the client, and can be retrieved by either visiting Proxy in the browser or doing a OPTIONS-request to its root.
func CreateSampleData ¶
func CreateSampleData() UploadMetadata
func GetUploadMetadata ¶
func GetUploadMetadata(info tusd.FileInfo) UploadMetadata
func (UploadMetadata) ConvertToMetaData ¶
func (um UploadMetadata) ConvertToMetaData() Metadata
func (UploadMetadata) PrintFieldsDebug ¶ added in v0.2.3
func (um UploadMetadata) PrintFieldsDebug() map[string]interface{}
func (UploadMetadata) PrintFieldsSafely ¶ added in v0.2.3
func (um UploadMetadata) PrintFieldsSafely() map[string]interface{}
func (UploadMetadata) ValidateRequiredFields ¶
func (um UploadMetadata) ValidateRequiredFields(r []string) (missing []string)
Can be used to quickly validate that certain fields are not empty
type Utterance ¶
type Utterance struct {
Type UtteranceType `json:"type"`
Person Person `json:"person"`
EventKind string `json:"eventKind"`
Source string `json:"source"`
Text string `json:"text"`
// Position, in milliseconds
StartPosition int `json:"startPosition"`
// EndPosition, in milliseconds
EndPosition int `json:"endPosition"`
}
type UtteranceType ¶
type UtteranceType string