metadata

package
v0.5.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 13, 2021 License: MIT Imports: 10 Imported by: 0

Documentation

Overview

Documentation for the normalized metadata-representation that Indico-Proxy uses

The goal of this definition is to support a broad range of metadata that a client can produce, and then be able to map that data back to any of the supported backends.

All metadata-fields should be defined in close cooperation with backend-providers, as well as clients.

Definitions

The following definitions are used throughout this document.

Backend: Any server that can receive files, and preferably, some metadata as well.
Client: Any software that can produce files and metadata and provide it to Proxy.
External ID: A Unique identifier on the backend of a file.
Parent ID: A unique identifier on the backend of a collection of files, like a folder, or an album
Case ID: A unique identifier for a case on of a case that the file belong to.

Clients and Backends do not need to know anything about each other, but the client needs to know about Proxy.

Client-integrations

Indico provides a DotNET-SDK. If you want to create a custom-client, we recommend starting by using a compatible library from https://tus.io and then extending it with the custom-layer that Proxy adheres to. Proxy uses the TUS-protocol for all uploads.

Below, a basic flow for creating and uploading a new file is provided.

Creating files

A file must be created, with all its metadata prior to upload. The metadata will be validated internally, and with the backend. Any validation-failures will be returned to the client.

Example of error returned:

{
  "type": "https://httpstatuses.com/422",
  "title": "Unprocessable Entity",
  "status": 422,
  "detail": "The following fields are missing: parentid",
  "instance": ""
}

The size of each chunk can be set by the client, but we recommend at least 6 MB. Each chunk can optionally be verified by providing the checksum of the current chunk.

POST-ing metadata as JSON to `/create`.

A unique upload-url should have been returned in the previous step. Uploads can now begin, with chunk-uploads preferred for large uploads. This upload-url should be stored on the client at least until the upload is completed. The upload-url can also be used to update metadata, if the backend supports it.

Uploading file

Info received from backend

A backend will provide its External ID, Parent ID and Case ID (if supported) of the file, but the time of which it reports this information so may change from one backend to another. The client should accommodate this. It may return the data in the following events.

1. The client creates a new upload.

2. The client completes an upload

3. As a last resort, information can be available at a later time, and the client needs to accommodate this, if it wants to receive this information. The information will be available when the client completes an upload and the file is also sent to the backend. The client then needs to ask Proxy if the information is available.

Information can be retrieved by using a HEAD-request to the upload-url, or to get the information as JSON, A GET-request can be sent to the upload-url prepended by `info`.

Updating metadata

The client may optionally, if the backend supports it, update the metadata. Proxy requires that the client sends in ALL the metadata.

Updates are done in the same way as creating files, as described above, but the url is `update`

Index

Constants

View Source
const (
	// The ID of the user the file belongs to.
	UserId = "userid"
	// Used by authenticators
	ClientId                 = "client-id"
	ConnectorWritten         = "connector-written"
	ReqId                    = "req-id"
	TempChecksum             = "_tmp_checksum_"
	PostponeStatus           = "postpone-status"
	AsUserName               = "as-username"
	AsUserId                 = "as-user-id"
	AsUserActiveDirectorySid = "as-user-sid"

	// The name of the container the file belongs to.
	ParentName        = "parentname"
	SSN               = "__ssn"
	NoMetadataMapping = "__no_metadata_mapping"

	ClientMediaId = "client-media-id"
	// The mime type of the file.
	FileType = "filetype"

	// A svcQueue-id in an installations svcQueue-system. Only valid for some systems, like Indico Gateway.
	ServiceQueueId = "serviceQueueId"

	// Deprecated ConnectorConfig, if needed for the svcQueue-handling
	ConnectorConfig = "connectorConfig"

	// The name given to the file by the user.
	DisplayName = "displayname"

	// The Checksum of the file.
	Checksum = "checksum"

	Filename     = "filename"
	ErrorMessage = "errormsg"

	ExtId       = "extid"
	ExtParentId = "extParentid"
	// Indicates whether the upload is verfied as completed.
	ExtUploaded  = "extUploaded"
	ExtConfirmed = "extConfirmed"
	// A checkSum calculated by the receiver
	ReceiverChecksum = "receiverChecksum"

	// The data available to all, as submitted by the Client
	MUploadMetadata                   = "UploadMetadata"
	ClientMessages                    = "ClientMessages"
	CaseNumberIgnored InternalInfoStr = "CaseNumberIgnored"
)
View Source
const (
	PostponedInit           postponedStatus = "init"
	PostponedUploadComplete postponedStatus = "uploadComplete"
	PostponedMetaComplete   postponedStatus = "metaComplete"
	PostponedAllComplete    postponedStatus = "allComplete"
)
View Source
const (
	RecordingEventRecording    EventKind     = "Recording"
	RecordingEventIncomingCall EventKind     = "IncomingCall"
	RecordingEventLowPower     EventKind     = "LowPower"
	RecordingEventLowMemory    EventKind     = "LowMemory"
	RecordingEventFatal        EventKind     = "Fatal"
	RecordingEventUnknown      EventKind     = "Unknown"
	RecordingEventUserEnded    EventKind     = "UserEnded"
	RecordingEventStarted      EventKind     = "Started"
	RecordingEventPaused       EventKind     = "Paused"
	Saying                     UtteranceType = "Saying"
	Event                      UtteranceType = "Event"
)

Variables

View Source
var (
	ErrRecordingEventInvalidOrder error = errors.New("the order of the events are logically correct")
)

Functions

func AssignMetaLogger

func AssignMetaLogger(logger logrus.FieldLogger)

func Debug

func Debug(enable bool)

Types

type Annotation

type Annotation struct {
	CreationDate time.Time `json:"createdAt"`
	ID           string    `json:"id"`
	Title        string    `json:"title"`
	Type         string    `json:"type"`
	// Data will be a computer-friendly structure. The structure itself is not decided yet.
	Data StringMap `json:"data"`
	// Position on the image/video.
	X1 int `json:"x1"`
	X2 int `json:"x2"`
	Y1 int `json:"y1"`
	Y2 int `json:"y2"`
	// Position, in milliseconds
	StartPosition int `json:"startPosition"`
	// EndPosition, in milliseconds
	EndPosition int `json:"endPosition"`
}

Annotations can be small figures, highlights, descriptors on an image/vidoe/audio. They can be added by the user, or automatically through OCR or Machine Learning. For audio, it can also be a way to censor parts of the audio via for instance beeps.

type Bookmark

type Bookmark struct {
	CreationDate time.Time `json:"creationTime"`
	ID           string    `json:"id"`
	Title        string    `json:"title"`
	// Position, in milliseconds
	StartPosition int `json:"startPosition"`
	// EndPosition, in milliseconds
	EndPosition int `json:"endPosition"`
}

Bookmark object belonging to a certain recording.

type CheckSum added in v0.2.0

type CheckSum struct {
	Value, Kind, Code, Notes string
}

type ClientMessage

type ClientMessage struct {
	Kind    InternalInfoStr
	Message string
}

type Creator

type Creator struct {
	// Can be an identifier, like an officer's badge-id, or a userId in the system.
	SysId      string `json:"sysId"`
	Attributes map[string]string
	Person
}

type EventKind added in v0.5.0

type EventKind string

type FormFields

type FormFields struct {
	// The metadata-key.
	Key string `json:"key"`
	// The id of the field, where available.
	FieldId string `json:"fieldId"`
	// The key used to get the translated VisualName, where available.
	TranslationKey string `json:"translationKey"`
	// The visual name, as reported by the client.
	VisualName string `json:"visualName"`
	// The value of the field, e.g. the user-input
	Value string `json:"value"`
	// Marks whether or not field is required or not.
	Required bool `json:"required"`
	// The kind of data in the Value-field.
	DataType       string         `json:"dataType"`
	ValidationRule ValidationRule `json:"validationRule"`
}

FormFields er unmapped metadata. In Indico Gateway, these represent an organizational-form.

type GenderType

type GenderType string
const (
	GenderFemale      GenderType = "Female"
	GenderMale        GenderType = "Male"
	GenderOther       GenderType = "Other"
	GenderUnspecified GenderType = ""
)

func ToGender

func ToGender(s string) GenderType

type InternalInfoStr

type InternalInfoStr string

type Location

type Location struct {
	// The text for the current location, like the current address, city, etc.
	Text string `json:"text"`
	// Geo-location
	Latitude float64 `json:"latitude"`
	// Geo-location
	Longitude float64 `json:"longitude"`
	Address   string  `json:"address"`
	Address2  string  `json:"address2"`
	ZipCode   string  `json:"zipCode"`
	PostArea  string  `json:"postArea"`
	Country   string  `json:"country"`
	Accuracy  float64 `json:"accuracy"`
	Altitude  float64 `json:"altitude"`
}

type Mapper

type Mapper func(data Metadata) Metadata

type MetaChecksum

type MetaChecksum struct {
	// The raw value of any checksum
	Value string `json:"value"`
	// SHA256, MD5, Blake3, CRC. Please advice with Indico before use.
	ChecksumType string `json:"checksumType"`
}

type Metadata

type Metadata map[string]string

func (*Metadata) AppendClientMessage

func (m *Metadata) AppendClientMessage(cMsg ClientMessage)

Appends ClientMessages, but ensures uniqueness.

func (*Metadata) Apply

func (m *Metadata) Apply(info *tusd.FileInfo) tusd.FileInfo

func (*Metadata) GetClientId

func (m *Metadata) GetClientId() string

func (*Metadata) GetClientMessages

func (m *Metadata) GetClientMessages() (cMsgs []ClientMessage)

func (*Metadata) GetConnectorConfig

func (m *Metadata) GetConnectorConfig() (map[string]string, error)

Deprecated

func (*Metadata) GetConnectorWritten

func (m *Metadata) GetConnectorWritten() int64

func (*Metadata) GetErrorMessage

func (m *Metadata) GetErrorMessage() string

func (*Metadata) GetExtConfirmed added in v0.1.5

func (m *Metadata) GetExtConfirmed() bool

func (*Metadata) GetExtId

func (m *Metadata) GetExtId() string

func (*Metadata) GetExtParentId

func (m *Metadata) GetExtParentId() string

func (*Metadata) GetExtUploaded

func (m *Metadata) GetExtUploaded() bool

func (*Metadata) GetFilename

func (m *Metadata) GetFilename() string

func (*Metadata) GetNoMetadataMapping added in v0.3.0

func (m *Metadata) GetNoMetadataMapping() bool

func (*Metadata) GetPostponedStatus added in v0.4.1

func (m *Metadata) GetPostponedStatus() postponedStatus

func (*Metadata) GetRaw

func (m *Metadata) GetRaw(k string) string

func (*Metadata) GetRawMetadata

func (m *Metadata) GetRawMetadata() string

func (*Metadata) GetReceiverChecksum added in v0.2.0

func (m *Metadata) GetReceiverChecksum() (CheckSum, error)

func (*Metadata) GetReqId

func (m *Metadata) GetReqId() string

func (*Metadata) GetServiceQueueId

func (m *Metadata) GetServiceQueueId() string

func (*Metadata) GetTemporaryChecksum added in v0.4.1

func (m *Metadata) GetTemporaryChecksum() string

func (*Metadata) GetUploadMetadata

func (m *Metadata) GetUploadMetadata() UploadMetadata

Returns uploadMetadata, which contains all the information about the file that a connector should need.

func (*Metadata) ReplaceUploadMetadata

func (m *Metadata) ReplaceUploadMetadata(um UploadMetadata)

func (*Metadata) SaveToInfo

func (m *Metadata) SaveToInfo(info *tusd.FileInfo)

func (*Metadata) SetAsActiveDirectoryUserSid

func (m *Metadata) SetAsActiveDirectoryUserSid(sid string) *Metadata

func (*Metadata) SetAsUserId

func (m *Metadata) SetAsUserId(id string) *Metadata

func (*Metadata) SetAsUserName

func (m *Metadata) SetAsUserName(u string) *Metadata

func (*Metadata) SetClientId

func (m *Metadata) SetClientId(cid string) *Metadata

func (*Metadata) SetConnectorConfig

func (m *Metadata) SetConnectorConfig(cfg map[string]string)

Deprecated

func (*Metadata) SetConnectorWritten

func (m *Metadata) SetConnectorWritten(written int64)

func (*Metadata) SetErrorMessage

func (m *Metadata) SetErrorMessage(msg string) *Metadata

func (*Metadata) SetExtConfirmed added in v0.1.5

func (m *Metadata) SetExtConfirmed() *Metadata

func (*Metadata) SetExtId

func (m *Metadata) SetExtId(d string)

func (*Metadata) SetExtParentId

func (m *Metadata) SetExtParentId(d string)

func (*Metadata) SetExtUploaded

func (m *Metadata) SetExtUploaded() *Metadata

func (*Metadata) SetNoMetadataMapping added in v0.3.0

func (m *Metadata) SetNoMetadataMapping(boolean bool)

func (*Metadata) SetPostponedStatus added in v0.4.1

func (m *Metadata) SetPostponedStatus(status postponedStatus) error

func (*Metadata) SetRaw added in v0.3.0

func (m *Metadata) SetRaw(k string, value string)

func (*Metadata) SetReceiverChecksum added in v0.2.0

func (m *Metadata) SetReceiverChecksum(value, kind, code, notes string) (CheckSum, error)

func (*Metadata) SetReqId

func (m *Metadata) SetReqId(reqid string)

func (*Metadata) SetServiceQueueId

func (m *Metadata) SetServiceQueueId(qID string)

func (*Metadata) SetTemporaryChecksum added in v0.4.1

func (m *Metadata) SetTemporaryChecksum(reqid string)

type Parent

type Parent struct {
	Id          string     `json:"id"`
	BatchId     string     `json:"batchId"`
	Name        string     `json:"name"`
	Description string     `json:"description"`
	CreatedAt   *time.Time `json:"createdAt"`
	UpdatedAt   *time.Time `json:"updatedAt"`
}

type Person

type Person struct {
	FirstName string `json:"firstName"`
	LastName  string `json:"lastName"`
	Id        string `json:"id"`
	// Date of birth
	Dob         *time.Time `json:"dob"`
	Gender      GenderType `json:"gender"`
	Nationality string     `json:"nationality"`
	Workplace   string     `json:"workplace"`
	// TBD
	Status    string `json:"status"`
	WorkPhone string `json:"workPhone"`
	Phone     string `json:"phone"`
	Mobile    string `json:"mobile"`
	// TBD
	Present bool `json:"isPresent"`
	Location
}

type RecordingEvent added in v0.5.0

type RecordingEvent struct {
	Id string
	// Optional
	Title string
	// Optional
	Description string
	// Required
	Kind EventKind
	// Required
	Start time.Time
	// Optional
	StartFrame int64
	// Optional
	End *time.Time
	// Optional
	EndFrame *int64
	// Optional
	EndKind *EventKind
	// Optional
	EndId *string
}

RecordingEvent uses

type RecordingEvents added in v0.5.0

type RecordingEvents []RecordingEvent

func (RecordingEvents) WithEndings added in v0.5.0

func (re RecordingEvents) WithEndings() (RecordingEvents, error)

Returns a new compact RecordingEvents by combining RecordingEvents which belong together in Start/End If the events are already using the End-fields, this has no effect

func (RecordingEvents) WithEndingsIgnoreErr added in v0.5.0

func (re RecordingEvents) WithEndingsIgnoreErr() RecordingEvents

type StringMap

type StringMap map[string]string

func (StringMap) MarshalXML

func (s StringMap) MarshalXML(e *xml.Encoder, start xml.StartElement) error

StringMap marshals into XML.

type Upl

type Upl struct {
	UploadMetadata
}

type UploadMetadata

type UploadMetadata struct {
	// A unique identifier for the user in the backend-system
	UserId string `json:"userId"`
	// Active-Directory-user, if available
	AdSid string `json:"adSid"`
	// Active-Directory-user, if available, in the format 'user@domainname@
	AdLogin string `json:"adLogin"`
	// The parent of the current media.
	Parent Parent `json:"parent"`
	// The time of which the media was created in the backend-database
	CreatedAt *time.Time `json:"createdAt"`
	// The time of which the media was last updated in the backend-database
	UpdatedAt *time.Time `json:"updatedAt"`
	// Is the item marked as archived, meaning it is marked to be deleted (soft-deleted). If null, the item is
	// not scheduled for deletion, if a date is set, the item is marked for deletion at that time.
	ArchiveAt *time.Time `json:"archiveAt"`
	// The date of which the item was marked as completed, E.g. the case was closed.
	CompletedAt *time.Time `json:"completedAt"`
	// The time of which the media was created by the user, on the client.
	CapturedAt *time.Time `json:"capturedAt"`
	// The fileType, as in MimeType. Example: 'image/jpeg' or 'video/mp4'
	FileType string `json:"fileType"`
	// A short description of the current file, submitted by the user
	FileSize    int64  `json:"fileSize"`
	DisplayName string `json:"displayName"`
	// A longer description of the current file, submitted by the uer.
	Description string `json:"description"`
	// Any checksums already calculated by the client.
	Checksum []MetaChecksum `json:"checksum"`
	// The filename,
	FileName string `json:"fileName"`
	// Tags
	Tags []string `json:"tags"`
	// The backend-database ID of the current file. Provide if updating metadata.
	ExtId string `json:"extId"`
	// A case-number returned by the user
	CaseNumber string `json:"caseNumber"`
	// Duration of the media, if audio or video. Int64. Should be in milliseconds when sending.
	Duration int64 `json:"duration"`
	// The creator of the current file, as in the current user, interviewer etc.
	Creator Creator `json:"creator"`
	// The location of the captured media
	Location Location `json:"location"`
	// Any subjects in the captured media.
	Subject []Person `json:"subject"`
	// TBD
	EquipmentId string `json:"equipmentId"`
	// TBD
	InterviewType   string          `json:"interviewType"`
	Bookmarks       []Bookmark      `json:"bookmarks"`
	RecordingEvents RecordingEvents `json:"recordingEvents"`
	Annotations     []Annotation    `json:"annotations"`
	// TBD
	Notes string `json:"notes"`
	// A unique identifier of the file on the client.
	ClientMediaId string `json:"clientMediaId"`
	// ID of any backend-provided Group-id
	GroupId string `json:"groupId"`
	// Name of any backend-group. Providing it will c create a groupName, if supported by the backend.
	GroupName string `json:"groupName"`
	// Any custom-field. Should only be used for customer-specific fields that do not fit in any other field. Before use, please request Indico to add your required fields.
	FormFields []FormFields `json:"formFields"`
	// Transcribed audio/video-details
	Transcription []Utterance `json:"transcription"`
}

UploadMetadata is the internal representation for Metadata on uploads.

All time.time-objects should be ISO-8601-compliant, and include the clients time-offset. The required metadata depends on the client, and can be retrieved by either visiting Proxy in the browser or doing a OPTIONS-request to its root.

func CreateSampleData

func CreateSampleData() UploadMetadata

func GetUploadMetadata

func GetUploadMetadata(info tusd.FileInfo) UploadMetadata

func (UploadMetadata) ConvertToMetaData

func (um UploadMetadata) ConvertToMetaData() Metadata

func (UploadMetadata) PrintFieldsDebug added in v0.2.3

func (um UploadMetadata) PrintFieldsDebug() map[string]interface{}

func (UploadMetadata) PrintFieldsSafely added in v0.2.3

func (um UploadMetadata) PrintFieldsSafely() map[string]interface{}

func (UploadMetadata) ValidateRequiredFields

func (um UploadMetadata) ValidateRequiredFields(r []string) (missing []string)

Can be used to quickly validate that certain fields are not empty

type Utterance

type Utterance struct {
	Type      UtteranceType `json:"type"`
	Person    Person        `json:"person"`
	EventKind string        `json:"eventKind"`
	Source    string        `json:"source"`
	Text      string        `json:"text"`
	// Position, in milliseconds
	StartPosition int `json:"startPosition"`
	// EndPosition, in milliseconds
	EndPosition int `json:"endPosition"`
}

type UtteranceType

type UtteranceType string

type ValidationRule

type ValidationRule struct {
	Min int `json:"min"`
	Max int `json:"max"`
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL