Documentation
¶
Index ¶
- Constants
- Variables
- func IsAvailable() bool
- func SetGlobalDB(db *NDB)
- type ExpirationTracker
- type KindStat
- type NDB
- func (db *NDB) BeginQuery() (*Txn, error)
- func (db *NDB) BootstrapExpirations() error
- func (db *NDB) CheckDuplicateEvent(evt nostr.Event) (bool, error)
- func (db *NDB) Close()
- func (db *NDB) CountFiltered(filters []nostr.Filter) (int, bool, error)
- func (db *NDB) DeleteNoteByID(id [32]byte) error
- func (db *NDB) EventStats() (total uint64, byKind []KindStat, err error)
- func (db *NDB) GetAllAuthors() []string
- func (db *NDB) MapUsage() (used, total uint64, ok bool)
- func (db *NDB) MapUsageFraction() float64
- func (db *NDB) ProcessDeletion(ctx context.Context, evt nostr.Event) error
- func (db *NDB) ProcessEvent(json string) error
- func (db *NDB) ProcessEvents(ldjson string) error
- func (db *NDB) PurgeOldEvents(cfg *cfgType.EventPurgeConfig, whitelistedPubkeys []string) int
- func (db *NDB) Query(filters []nostr.Filter, limit int) ([]nostr.Event, error)
- func (db *NDB) RunExpirationSweeper(ctx context.Context)
- func (db *NDB) ScheduleEventPurging(ctx context.Context, cfg *cfgType.ServerConfig, ...)
- func (db *NDB) StartMapUsageMonitor(ctx context.Context, interval time.Duration)
- func (db *NDB) Stat() (*C.struct_ndb_stat, error)
- func (db *NDB) StoreEvent(ctx context.Context, evt nostr.Event) error
- func (db *NDB) TextSearch(query string, base nostr.Filter, limit int) ([]nostr.Event, error)
- func (db *NDB) TrackArrivals(ctx context.Context, path string, late time.Duration) error
- func (db *NDB) WriteErrorCount() uint64
- type Options
- type Txn
Constants ¶
const ( MapUsageWarnFraction = 0.80 MapUsageErrorFraction = 0.95 MapUsageRejectFraction = 0.97 )
Map-usage thresholds. WARN/ERROR drive the periodic gauge; RejectFraction is where the write path starts refusing new events so the map never actually fills (a full LMDB map makes the writer thread fail silently while events are still acked). The gap below 1.0 leaves room for the purge's own delete transactions, which need free pages too.
const ( // FlagSkipNoteVerify makes the ingester skip signature verification. // Safe for imports from a trusted source (e.g. a previous grain/mongo // export) where events were already validated at original ingest time. FlagSkipNoteVerify = 1 << 1 )
NDB open flags. These map 1:1 onto nostrdb.h NDB_FLAG_* bits.
const MaxFulltextKinds = 64
MaxFulltextKinds mirrors NDB_MAX_FULLTEXT_KINDS in nostrdb.h.
Variables ¶
var DefaultFulltextKinds = []int{0, 1, 30023}
DefaultFulltextKinds is the kind set grain indexes for NIP-50 search when config.yml doesn't say otherwise: profile metadata, text notes and long-form articles.
Functions ¶
func IsAvailable ¶
func IsAvailable() bool
IsAvailable returns true if the database is initialized and ready.
Types ¶
type ExpirationTracker ¶ added in v0.6.0
type ExpirationTracker struct {
// contains filtered or unexported fields
}
ExpirationTracker is a thread-safe min-heap of pending event expirations. Created and owned by NDB; do not instantiate directly.
func (*ExpirationTracker) Len ¶ added in v0.6.0
func (t *ExpirationTracker) Len() int
Len returns the current number of tracked expirations. For tests/observability.
func (*ExpirationTracker) Track ¶ added in v0.6.0
func (t *ExpirationTracker) Track(expireAt int64, id [32]byte)
Track records a pending expiration. If the new entry beats the current heap top, the sweeper is woken so it doesn't oversleep.
type KindStat ¶ added in v0.8.0
type KindStat struct {
Kind int `json:"kind"` // the bucket's kind number; -1 only for "other"
Name string `json:"name"`
Count uint64 `json:"count"`
Bytes uint64 `json:"bytes"` // key_size + value_size for this bucket
}
KindStat is one bucket of the stored-event kind distribution: how many events of that kind and how much storage (keys + values) they occupy.
type NDB ¶
type NDB struct {
// contains filtered or unexported fields
}
NDB wraps a nostrdb instance. It is safe for concurrent use.
func GetDB ¶
func GetDB() *NDB
GetDB returns the global nostrdb instance. Returns nil if the database hasn't been initialized.
func Open ¶
Open initializes a new nostrdb database at the given directory path. mapSizeMB sets the maximum database size in megabytes (LMDB map size). ingestThreads controls how many threads nostrdb uses to process incoming events.
func OpenWithFlags ¶
OpenWithFlags is like Open but forwards an ndb_config_set_flags bitmask to nostrdb. Use FlagSkipNoteVerify for trusted-source bulk imports.
func OpenWithOptions ¶ added in v0.8.0
OpenWithOptions opens (or creates) the database at dbDir.
FulltextKinds only governs notes written from now on: nostrdb neither backfills nor prunes the text index when the set changes, so widening it on a populated relay only makes new events of the added kinds searchable.
func (*NDB) BeginQuery ¶
BeginQuery starts a read transaction for querying the database. The caller MUST call EndQuery when done.
func (*NDB) BootstrapExpirations ¶ added in v0.6.0
BootstrapExpirations scans the DB once at startup, deletes any events whose expiration has already passed, and populates the in-memory heap with the rest. Pages backwards through created_at to cover the whole history; each page is capped by the nostrdb query limit.
func (*NDB) CheckDuplicateEvent ¶
CheckDuplicateEvent checks if an event with the given ID already exists. Uses a lightweight pointer check — no deserialization needed.
func (*NDB) Close ¶
func (db *NDB) Close()
Close shuts down the nostrdb instance and frees resources.
func (*NDB) CountFiltered ¶ added in v0.6.0
CountFiltered returns the number of events matching `filters`. The nostrdb query API is capped at maxQueryResults per call, so we page backwards through created_at using the cursor pattern from the NIP-40 bootstrap. The bool return is true when the result is approximate:
- hardCap reached, or
- multiple filters supplied (we sum per-filter counts; a multi-filter union that overlaps in event ids would double-count, and dedupe would require buffering all matched ids — defeating the purpose of COUNT vs REQ).
Single-filter requests with sane time bounds get an exact count.
One known undercount edge case: if a full page (maxQueryResults events) shares the same created_at, the next page's cursor advances by one second and may skip same-second siblings beyond the page. Same trade-off as PurgeOldEvents and the expiration bootstrap.
func (*NDB) DeleteNoteByID ¶
DeleteNoteByID enqueues a real delete of an event from nostrdb by its raw 32-byte ID. The delete is applied by the nostrdb writer thread in FIFO order with ingests — a delete of an in-flight ingest of the same ID is committed atomically in the same batch and cannot race.
This is grain's one and only physical-delete primitive. All three deletion audiences (NIP-09 author deletes, operator retention / PurgeOldEvents, replaceable/addressable supersede, and admin --delete CLI) call through here. Authorization is the caller's responsibility: this function performs no checks beyond "is the DB open and is the writer queue accepting work".
Returns an error only if the DB is closed or the writer inbox is full. "Not found" is not an error at this layer — it's logged at C level and the call is a no-op.
func (*NDB) EventStats ¶ added in v0.8.0
EventStats returns the stored-event total and its distribution over nostrdb's common-kind buckets (Profile, Text, Contacts, …) plus an "other" bucket for everything else; zero-count buckets are omitted.
NOT cheap: ndb_stat walks every entry of every LMDB table (~340ms on a 300k-event relay). Never call it per request — the dashboard endpoint serves a cached snapshot (server/api/relayStats.go).
func (*NDB) GetAllAuthors ¶
GetAllAuthors returns all unique pubkeys that have events in the database.
func (*NDB) MapUsage ¶ added in v0.8.0
MapUsage returns the LMDB map's used and total (ceiling) bytes. ok is false when the stats can't be read (e.g. the DB is closed).
func (*NDB) MapUsageFraction ¶ added in v0.8.0
MapUsageFraction returns used/total in [0,1], or 0 if unavailable. Cheap enough (two txn-less LMDB header reads) to call on the write path.
func (*NDB) ProcessDeletion ¶
ProcessDeletion handles NIP-09 kind 5 deletion events. It walks the event's `e` and `a` tags, enforces NIP-09's same-pubkey authorization rule at each tag, physically removes matching events via the nostrdb writer thread (ndb_request_delete_note), then stores the kind-5 record itself so clients can still see the deletion marker per spec.
Tag processing is best-effort: a failure on one tag is logged and the next tag is still processed. The kind-5 is always ingested at the end so that a client querying for it still sees it, even if none of its targets existed.
func (*NDB) ProcessEvent ¶
ProcessEvent ingests a raw JSON Nostr event string into the database. nostrdb parses the JSON, validates, indexes, and stores the event internally. The JSON should be a relay message like: ["EVENT", <subscription_id>, <event>] or just the event object itself for direct ingestion.
func (*NDB) ProcessEvents ¶
ProcessEvents ingests multiple newline-delimited JSON events.
func (*NDB) PurgeOldEvents ¶
func (db *NDB) PurgeOldEvents(cfg *cfgType.EventPurgeConfig, whitelistedPubkeys []string) int
PurgeOldEvents removes events older than the configured retention window. An event's age runs from its created_at or, under the received retention clock, from when it arrived if it arrived late (see TrackArrivals), so an event received already old still gets the whole window.
Whitelisted pubkeys (configured members) are excluded when ExcludeWhitelisted is set — the "non-member cleanup" knob that keeps member content forever while aging out drive-by events.
It pages per kind through nostrdb's kind index, which is ordered by created_at, so it actually drains the backlog rather than re-scanning the same storage-order head each run. Deletes are capped at purgeRunBudget per run.
func (*NDB) Query ¶
Query executes NIP-01 filters against the database and returns matching events. This opens and closes a read transaction internally.
func (*NDB) RunExpirationSweeper ¶ added in v0.6.0
RunExpirationSweeper deletes events as their expiration passes. Blocks until ctx is cancelled. Safe to start before BootstrapExpirations finishes — Track and the bootstrap-time deletes both feed the same heap, and the sweeper just sleeps until the first deadline arrives.
func (*NDB) ScheduleEventPurging ¶
func (db *NDB) ScheduleEventPurging(ctx context.Context, cfg *cfgType.ServerConfig, getWhitelistedPubkeys func() []string)
ScheduleEventPurging runs periodic event purging at the configured interval. It returns when ctx is cancelled so the loop doesn't outlive the server instance (and its now-closed DB handle) on a config-reload restart (#93).
func (*NDB) StartMapUsageMonitor ¶ added in v0.8.0
StartMapUsageMonitor logs the LMDB map usage at startup and every interval, escalating WARN at 80% and ERROR at 95% so a filling database is visible in grain's own logs (the writer's MDB_MAP_FULL errors only reach stderr). Bounded to ctx like the other background loops.
func (*NDB) Stat ¶
func (db *NDB) Stat() (*C.struct_ndb_stat, error)
Stat returns database statistics.
func (*NDB) StoreEvent ¶
StoreEvent processes and stores a Nostr event in the database. nostrdb handles event ingestion internally including parsing and indexing. For replaceable and addressable events, we must handle the replacement semantics ourselves since nostrdb does not enforce NIP-01 replacement rules.
func (*NDB) TextSearch ¶ added in v0.6.0
TextSearch is the no-transaction convenience wrapper, mirroring Query.
func (*NDB) TrackArrivals ¶ added in v0.8.0
TrackArrivals opens the late-arrival ledger at path and starts recording: from now on an event stored more than late after its created_at has its arrival time kept, so the received retention clock can age it from then. The ledger is flushed every arrivalFlushInterval until ctx ends, and on Close.
func (*NDB) WriteErrorCount ¶ added in v0.8.0
WriteErrorCount returns nostrdb's running total of writer failures (map full, bad txn, etc.). The writer thread only logs these to stderr and events are acked OK before the write commits, so polling this is how grain notices silent write loss. It's a process-global count, valid regardless of db state.
type Options ¶ added in v0.8.0
type Options struct {
MapSizeMB int // LMDB map ceiling in MB
IngestThreads int // 0 = nostrdb's default (one per core)
Flags int // ndb_config_set_flags bitmask (FlagSkipNoteVerify, ...)
FulltextKinds []int // kinds tokenized for NIP-50 search; nil = DefaultFulltextKinds
}
Options carries everything Open needs beyond the directory.
type Txn ¶
type Txn struct {
// contains filtered or unexported fields
}
Txn represents a read transaction against nostrdb. Transactions must be short-lived to avoid blocking LMDB space reclamation.
func (*Txn) EndQuery ¶
func (txn *Txn) EndQuery()
EndQuery ends a read transaction. Must be called after BeginQuery.
func (*Txn) GetNoteByID ¶
GetNoteByID looks up a single event by its hex ID.
func (*Txn) TextSearch ¶ added in v0.6.0
TextSearch runs a NIP-50 fulltext query within an existing transaction. `base` carries the non-search constraints (kinds, authors, tags, since, until); its Search field is ignored — the query string is passed as a separate argument to ndb_text_search_with.
Result ordering is descending by created_at (newest-first), matching the rest of grain's read paths. Only kinds in database.fulltext_kinds (default: 0, 1, 30023) carry a text index — searches that filter to other kinds return nothing even if matching content exists in the DB.
Source Files