[Feature] Store BanyanDB's own logs in BanyanDB (self-stored logs)

Search before asking

I had searched in the issues and found no similar feature requirement.

Description

1. Feature introduction

BanyanDB self-stores its metrics, but not its logs.

	Metrics	Logs
Destinations	2 — Prometheus + self-storage	1 — console only
Abstraction	`meter.Provider` (pluggable)	a raw `io.Writer`
Fan-out	✅ `factory` → every provider	❌
Lifecycle service	✅ `metricService`	❌
Stored in BanyanDB	✅ `_monitoring` group	❌
Survives pod restart	✅	only if something else collected it

The entire log output story is one line, fixed at logger.Init() time:

// pkg/logger/setting.go
if development { w = zerolog.ConsoleWriter{Out: os.Stderr, …} } else { w = os.Stderr }
ctx := zerolog.New(w).Level(lvl).With().Timestamp()

Proposal: give logs the same two-destination model metrics already have — console and self-storage — reusing the native-metrics machinery wherever it already solves the problem.

Non-goals: replacing console output; ingesting non-BanyanDB logs; changing the module tree, level filtering, or existing flags.

2. Deployment architecture

flowchart TB
    C["clients (OAP, bydbctl)"] --> LB["gRPC load balancer :17912"]

    subgraph LT["Liaison tier — stateless, NO disk"]
        L0["liaison-0<br/>:17912 client · :18912 peer<br/>+ FODC agent"]
        L1["liaison-1<br/>+ FODC agent"]
    end

    subgraph DT["Data tier — OWNS THE DISK"]
        DH["data-hot-0 :17912<br/>+ FODC agent<br/>+ lifecycle sidecar<br/>+ backup sidecar<br/>+ restore init container<br/>PVC: measure/stream/trace/property"]
        DW["data-warm-0"]
        DC["data-cold-0"]
    end

    FP["fodc-proxy · 1 per cluster<br/>:17913 /metrics · /cluster/topology"]

    LB --> L0 & L1
    L0 & L1 -- "write / query :17912" --> DH
    DH -- "hot→warm→cold" --> DW --> DC
    L0 & L1 -. "gRPC register" .-> FP
    DH & DW & DC -. "gRPC register" .-> FP
    FP --> P["Prometheus → Grafana"]

Pairing rules

Component	Cardinality	Paired with	Form
`liaison`	N (≥2)	—	behind a gRPC LB, peers via `:18912`
`data`	M (≥2), hot/warm/cold	—	discovered by liaisons
FODC agent	1 : 1 per node	one liaison or one data node	sidecar
FODC proxy	1 per cluster	all agents	standalone pod
`lifecycle`	1 per data node	co-located data node	sidecar, `127.0.0.1:17912`
`backup`	1 per data node	co-located data node	sidecar, shares PVC
`restore`	1 per data node	nothing	init container
`migration`	1 per cluster	all PVCs, zero live nodes	standalone pod, data tier at `replicas=0`

liaison and data are cluster-scoped peers. Everything else is a per-node companion glued to one node by 127.0.0.1 or a shared volume.

3. Role → function → log destination

Self-storage needs a process owning the storage engine — the component holding queryable groups on disk. Only data and standalone do. Everything else must reach one, and the hop count differs.

Role	Function	Storage engine	Log destination
`standalone`	all-in-one	✅	in-process — `queue.Local()`, own shard
`data`	stores & serves shards	✅	in-process — `queue.Local()`, own shard
`liaison`	routes writes/queries	❌	two tiers — liaison wqueue → part-sync → data node (ref)
`lifecycle`	hot→warm→cold migration	❌	one hop — `pub` → co-located `127.0.0.1:17912`
`backup`	snapshots → S3/GCS/Azure	❌	one hop — `pub` → `--grpc-addr`; needs a schema-bootstrap decision
`restore`	remote backup → local dirs	❌	console only — init container, runs before its data node starts
`migration`	re-grid measure/stream data	❌	console only — runs with the data tier at `replicas=0`

Two corrections to a common mental model:

A liaison is not diskless. It has a write queue at --measure-data-path / --stream-data-path (banyand/measure/wqueue.go) where it buffers parts before syncing them out. It lacks the storage engine, so it can hold a log batch in flight but can never be where logs are read back from.
lifecycle and backup never go through a liaison. pub.NewWithoutMetadata(nil) defaults to ROLE_DATA, so they publish straight to their co-located data node — a strictly shorter path than the liaison's.

restore and migration are not gaps to fill later. A tool that runs while the database is down cannot log into that database — and for migration, a self-storing sink would violate its own precondition that nothing else writes to the target paths.

4. Proposed approach

4.1 The seam — one event, two writers

zerolog always encodes an event to JSON internally, and ConsoleWriter is itself an io.Writer that re-formats that JSON. So MultiLevelWriter hands the sink the fully-encoded line — module, level, timestamp, message, all structured fields — one source, one format, one set of parameters.

flowchart LR
    A["call site<br/>logger.GetLogger(measure).Info()<br/>.Str(group, g).Msg(flushed part)"] --> B["zerolog encodes ONE JSON event"]
    B --> CW["console writer<br/>stderr · ALWAYS ON"]
    B --> SW["switchableWriter<br/>atomic.Pointer"]
    SW --> S2["logSink — ONE ring buffer<br/>alive from Init()"]
    SW --> S3["io.Discard — after GracefulStop"]

❌ zerolog.Hook rejected — hooks see level + message but not the accumulated structured fields.

The writer contract

Three constraints come from zerolog's own implementation (v1.34.0), not from BanyanDB:

zerolog internal	Constraint on the sink
`Event.write()` calls `putEvent(e)`, returning `e.buf` to a `sync.Pool`	`p` is reused. Buffering it without `append([]byte(nil), p...)` yields a slice the next log line overwrites.
`multiLevelWriter.Write` maps `_n != len(p)` → `io.ErrShortWrite`	Always return `(len(p), nil)`. A dropped line must still report a full write; drops surface via the counter, never the return value.
`MultiLevelWriter` type-switches on `zerolog.LevelWriter` before wrapping in `LevelWriterAdapter`	Implement `WriteLevel(l zerolog.Level, p []byte)` and zerolog hands over the level as an enum — so `--logging-native-level` filtering is an int compare with no JSON parsing on the hot path, and the `level` entity tag needs no extraction.

type switchableWriter struct{ target atomic.Pointer[zerolog.LevelWriter] }

func (s *switchableWriter) WriteLevel(l zerolog.Level, p []byte) (int, error) {
    if w := s.target.Load(); w != nil {
        (*w).WriteLevel(l, p)   // errors deliberately ignored
    }
    return len(p), nil          // always
}

atomic.Pointer rather than a mutex: WriteLevel runs on every log line from every goroutine, while the pointer is swapped once per process (at GracefulStop). A lock-free load is the right trade at that ratio.

4.2 Lifecycle — deferred activation

Same deferral pendingMeasures / native.InitSchema already use for metric schemas — but applied to the consumer, not the buffer.

Init()        install switchableWriter → logSink   (buffer live, NO consumer, no I/O)
PreRun()      register drop counters                            (no metadata)
Serve()       1. create group + stream schema (idempotent)
              2. START the consumer goroutine (§4.3)
GracefulStop  closer.CloseNotify() → consumer drains once and exits (bounded)
              swap → io.Discard · close publisher

One buffer, two phases. The buffer is allocated at Init() and never replaced; "activation" only starts draining it. There is no second buffer and no hand-off, so a goroutine that loaded the writer just before activation cannot strand its line in an abandoned buffer. Producers see one unchanging target for the whole process lifetime.

4.3 Write workflow

The consumer is a dedicated goroutine, following accesslog.startConsumer — not a timestamp.Scheduler job as FlushMetrics uses. A scheduler job is a periodic callback: it fires on the tick and can do nothing between ticks, so it supports a time trigger and nothing else. The size trigger needs something that observes every push, which only a goroutine selecting on the buffer can do.

for {
    select {
    case <-s.closer.CloseNotify():
        s.flush(batch)                      // final drain, bounded
        return
    case <-flushTicker.C:                   // TIME trigger  (--logging-native-flush-interval, 5s)
        if len(batch) > 0 { s.flush(batch); batch = batch[:0] }
    case entry := <-s.buffered():
        batch = append(batch, entry)
        if len(batch) >= s.flushSize {      // SIZE trigger  (--logging-native-flush-size, 1024)
            s.flush(batch); batch = batch[:0]
        }
    }
}

flowchart LR
    R["buffer<br/>bounded · drop on full"] --> T{"consumer goroutine<br/>flush trigger"}
    T -- "interval 5s" --> B["build InternalWriteRequest"]
    T -- "size ≥ 1024" --> B
    T -- "CloseNotify" --> B
    B --> N{"nodeSelector"}
    N -- "nil (data/standalone)" --> LOC["queue.Local() → own shard"]
    N -- "set (liaison/lifecycle)" --> LO["Locate()"]
    LO -- ok --> PUB["pub → TopicStreamWrite → data node"]
    LO -- "fail" --> DROP["count no_node, drop<br/>(do NOT publish empty nodeID)"]

4.4 Schema

Derived field by field from the native-metrics schema in pkg/meter/native/provider.go. Each row names the proto field, so the diff against the existing implementation is explicit.

Running example — this line, emitted on node data-hot-0:

{"level":"warn","module":"MEASURE","group":"sw_metric","time":"2026-09-11T10:23:45.123Z","message":"flush took longer than expected"}

Group — common.v1.Group

Schema field	Metrics (`_monitoring`)	Logs (`_monitoring_log`)	Example value
`metadata.name`	`_monitoring`	`_monitoring_log`	`"_monitoring_log"`
`catalog`	`CATALOG_MEASURE`	`CATALOG_STREAM`	`Catalog_CATALOG_STREAM` (= 1)
`resource_opts.shard_num`	`1`	`1` (configurable)	`1`
`resource_opts.segment_interval`	`{UNIT_DAY, 1}`	`{UNIT_DAY, 1}`	`&IntervalRule{Unit: UNIT_DAY, Num: 1}`
`resource_opts.ttl`	`{UNIT_DAY, 1}`	`{UNIT_DAY, 7}` (configurable)	`&IntervalRule{Unit: UNIT_DAY, Num: 7}`

Resource — metrics use database.v1.Measure, logs use database.v1.Stream

Not a rename. Measure and Stream are two distinct proto messages that coexist; nothing is renamed or modified. Logs simply instantiate a different existing type. "Resource" is BanyanDB's own umbrella term for what a group holds — docs/concept/data-model.md: "A group's catalog fixes which one kind of resource it holds (MEASURE, STREAM, TRACE, or PROPERTY)." There is no Resource type in code.

Schema field	Metrics (`Measure`)	Logs (`Stream`)	Example value
`metadata.name`	one Measure per metric name	one Stream, `log`	metrics: `"total_written"` · logs: `"log"`
`tag_families[].name`	single `default`	`searchable` + `data`	`"searchable"`, `"data"`
`tag_families[].tags[]`	node/scope/metric labels	7 searchable + 1 binary	`{Name:"level", Type:TAG_TYPE_STRING}`, `{Name:"body", Type:TAG_TYPE_DATA_BINARY}`
`fields[]` (`FieldSpec`)	`value` FLOAT/GORILLA/ZSTD	field does not exist	— (no such field on `Stream`)
`entity.tag_names`	all 4 node tags + every label	`[node_id, module, level]`	`[]string{"node_id","module","level"}`

Write payload

Schema field	Metrics	Logs	Example value (for the line above)
request / value type	`measure.v1.InternalWriteRequest` / `DataPointValue`	`stream.v1.InternalWriteRequest` / `ElementValue`	—
`…element_id`	no counterpart	`<node_id>-<process_epoch>-<seq>`	`"data-hot-0-1757585021-42"`
`…timestamp`	`time.Now().Truncate(time.Second)` at flush	the event's own time	`2026-09-11T10:23:45.123Z`
`…tag_families[0]` (`searchable`)	`{Tags: labelValues}`	7 tag values	`["data","data-hot-0","MEASURE","warn","10.1.2.3:17912","10.1.2.3:17913","flush took longer than expected"]`
`…tag_families[1]` (`data`)	—	the raw line	`[]byte("{\"level\":\"warn\",\"module\":\"MEASURE\",…}")`
`…fields`	one `FieldValue_Float`	absent	—
`entity_values`	label values	must match `entity.tag_names` order	`["data-hot-0","MEASURE","warn"]`
topic	`data.TopicMeasureWrite`	`data.TopicStreamWrite`	`data.TopicStreamWrite`

tag_families:
  searchable: node_type, node_id, module, level, grpc_address, http_address, message
  data:       body  TAG_TYPE_DATA_BINARY   ← the complete original JSON line

grpc_address / http_address — searchable, not in the entity. Not cardinality (both follow from node_id) but address churn: a restarted pod keeps its node_id and gets a new IP, which would open a new series and fragment that node's history.
body holds the whole original line — no field lost to schema drift, byte-identical to the console.
element_id = <node_id>-<process_epoch>-<seq>. timestamp = the event's time, not flush time, so lines buffered before activation land correctly on the time axis.

4.5 Two things that differ from metrics

	Metrics	Logs
Semantics	sampled state — a missed flush costs nothing, next flush carries the current value	events — a dropped line is gone. Needs bounded ring + explicit drop-oldest + drop counters
Recursion	`gauge.Set()` doesn't log	the write path logs. `pub`, `sub`, `cluster-node-registry-*` all call `logger.GetLogger(...)` → a naive sink feeds itself

Recursion guard, three layers: module denylist for write-path modules (primary) → sink never logs through pkg/logger, only rate-limited stderr → bounded ring caps amplification (backstop).

⚠️ The denylist is load-bearing for the liaison specifically: its two-tier path traverses the write queue and part-sync, both of which log.

Invariant across every failure mode: console output is never degraded by the sink. Self-storage is best-effort; the console is not.

4.6 Implementation phases

See this comment — delivery order, and why each phase adds exactly one new failure domain.

5. Parameters and configuration

Derived from the two existing surfaces — prefix from console logging, vocabulary from metrics:

pkg/logger/setting.go RegisterFlags       observability/services/service.go FlagSet
  --logging-env      prod                   --observability-listener-addr         :2121
  --logging-level    info                   --observability-modes          [prometheus]
  --logging-modules  nil                    --observability-metrics-interval        15s
  --logging-levels   nil                    --observability-native-flush-interval    5s
        └──── "logging-" prefix ──┐   ┌──── "modes" + "-native-" infix ────┘
                                  ▼   ▼
                           --logging-native-*

Flag	Default	Meaning	From (where the idea comes from)
`--logging-modes`	`console`	`console`, `native`, or both	metric — the `modes` idea, from `--observability-modes` (`[prometheus]`)
`--logging-native-level`	`warn`	minimum level reaching storage	console log — `--logging-level`, but as an independent threshold. Metrics have no level concept
`--logging-native-flush-interval`	`5s`	time trigger	metric — `--observability-native-flush-interval`, same default
`--logging-native-flush-size`	`1024`	size trigger, in entries	access log — `accesslog.DefaultBatchSize` (100), a constant today
`--logging-native-buffer-size`	`8192`	ring capacity, in entries	access log — the `validRequests` channel capacity (100 sampled / 1000 not), a constant today
`--logging-native-group-ttl`	`7d`	`_monitoring_log` TTL	metric — `provider.go` hardcodes `ResourceOpts.Ttl = {UNIT_DAY, 1}`
`--logging-native-shard-num`	`1`	`_monitoring_log` shards	metric — `provider.go` hardcodes `ResourceOpts.ShardNum = 1`. Load-bearing for logs, given the single-node funnel

Deliberately not copied from metrics: --observability-listener-addr (logs are push-only — no pull endpoint to scrape) and --observability-metrics-interval (that is the Prometheus collection tick; logs have no collection phase).

Unchanged: the four existing --logging-env / -level / -modules / -levels flags. They apply upstream of both writers; altering them is an explicit non-goal. Each new flag gets the standard BYDB_* env binding, as logger.RegisterFlags already provides. restore / migration accept --logging-modes but reject native at Validate().

6. Reading logs back

No new query surface — _monitoring_log is an ordinary stream group:

bydbctl stream query -f - <<EOF
name: "log"
groups: ["_monitoring_log"]
timeRange: { begin: 2026-09-10T00:00:00Z, end: 2026-09-11T00:00:00Z }
criteria:
  condition: { name: "level", op: "BINARY_OP_EQ", value: { str: { value: "error" } } }
projection:
  tagFamilies:
    - { name: "searchable", tags: ["node_id", "module", "level", "message"] }
    - { name: "data", tags: ["body"] }
EOF

7. Failure modes

Each row is one stage of the write path, in path order. Derived by walking the pipeline and asking, at each stage, "what can fail here, and what happens when it does?"

producer ──► BUFFER ──► consumer ──► schema ──► Locate ──► publish ──► data node
   │          │          │           │          │           │           │
   │          │          │           │          │           │           └─ slow
   │          │          │           │          │           └─ error / timeout
   │          │          │           │          └─ no node available
   │          │          │           └─ CreateGroup / CreateStream fails
   │          │          └─ not started yet (pre-Serve)
   │          └─ full
   └─ never blocks  ← the invariant, not a failure

Every stage that can fail gets exactly one row, one behaviour, one counter label. The review test is therefore not "are these cases interesting" but "does every arrow in that diagram have a row" — a stage with no row is a stage whose failure is unhandled.

Stage	Situation	Behaviour
producer	always	never blocks the caller; console unaffected — the invariant, not a failure
buffer	consumer goroutine not yet started (pre-Serve)	accumulates; drained once `Serve()` starts the consumer
buffer	fills before the consumer starts	drop, count `not_ready`; console still has everything
buffer	full during a burst	drop, count `buffer_full`
schema	`CreateGroup` / `CreateStream` fails	stderr once; the ring keeps buffering (eventually dropping as `not_ready`); retry on the flush tick
schema	group dropped at runtime (`OnDelete`)	writes begin failing downstream → surfaces as `publish_error`. After N consecutive publish errors, re-run the idempotent `InitSchema` rather than failing forever
schema	exists but incompatible (hand-edited `_monitoring_log`)	`CreateStream` returns `AlreadyExists`, so the sink proceeds and writes fail on tag mismatch. Count `schema_mismatch`, log expected-vs-actual once to stderr. Never auto-migrate
Locate	no data node (liaison partition / migration window)	count `no_node`, drop — don't publish an empty nodeID
publish	error / timeout	count `publish_error`, drop that batch, continue
publish	data node disk full	also `publish_error` — indistinguishable from a network fault at this layer. Diagnose via the existing `banyandb_system_disk` metric
data node	slow	drops (whichever bound fills first); never backpressure into the caller
shutdown	`GracefulStop` with a full buffer	`closer.CloseNotify()` makes the consumer goroutine drain once and exit; that drain is time-bounded, and whatever cannot ship is counted `shutdown_drop` so the loss is not silent
sink itself	any internal error	direct rate-limited stderr, never through `pkg/logger`

Drops are exported as banyandb_logging_dropped_total{reason} through the existing metrics factory — loss is visible in the system that is reliable. Labels: not_ready, buffer_full, schema_mismatch, no_node, publish_error, shutdown_drop.

8. References

See this comment — existing log implementation, the native-metrics template, per-role wiring, and the liaison's two-tier write path.

Related issues

None found.

Are you willing to submit a pull request to implement this on your own?

Yes I am willing to submit a pull request on my own!

Code of Conduct

I agree to follow this project's Code of Conduct

Reference — the liaison's two-tier write path

Background for the liaison row in §3. A liaison write reaches disk through two independent node selections, not one. This is existing behaviour for user data and for native metrics.

flowchart TB
    S["liaison metricSvc / logSvc<br/>Locate(group, name, shardID, replicaID)"]
    T1["① a LIAISON node :18912<br/>intermediate buffer, NOT the store<br/>per-(group, shardID) write queue on its own disk"]
    T2["② a DATA node :17912<br/>via part-sync — THE ACTUAL STORE"]
    S -- "selector fed by tire1Client<br/>allowedRoles = [ROLE_LIAISON]" --> T1
    T1 -- "NewClusterNodeRegistry(TopicMeasurePartSync, tire2Client, …)<br/>allowedRoles = [ROLE_DATA]" --> T2

Streams mirror this with TopicStreamWrite / TopicStreamPartSync (banyand/stream/svc_liaison.go:207). Each selector's candidate pool is whatever its pub admits — pub.OnAddOrUpdate gates on allowedRoles (banyand/queue/pub/client.go:133) before a node reaches sel.AddNode.

How a node is chosen

roundRobinSelector.Pick is deterministic, not per-call rotation:

// pkg/node/round_robin.go:190
func (r *roundRobinSelector) Pick(group, _ string, shardID, replicaID uint32) (string, error) {
    //                                     ↑ resource name IGNORED
    i := sort.Search(len(r.lookupTable), …)              // entry for (group, shardID)
    return r.nodes[(i + int(replicaID)) % len(r.nodes)], nil
}

lookupTable — one entry per (group, shardID), sorted by group then shard, across all groups.
r.nodes — the sorted list of admitted node names.
The resource name is discarded, so every measure in _monitoring resolves identically.

Consequences for a single-shard group

Exactly one (group, shardID) key exists, so tier ① always picks the same liaison and tier ② always picks the same data node — and every liaison in the cluster computes the same answer.


Funnel	N liaisons' full log volume → one liaison's wqueue → one data node. On the tier that exists to scale horizontally.
Self-loop	There is no self-exclusion in the publisher — `ConnManager.OnAddOrUpdate` dials every role-matching node including itself, and `selfNode` only stamps `SenderNode` on the request. So the designated liaison's own logs publish over gRPC back to itself at `:18912` → sub server → writeListener → wqueue, which logs.
Drift	If that liaison restarts, `r.nodes` changes, `i % len(nodes)` moves, and a different peer becomes the collector.

All of _monitoring lives on one data node today. Tolerable for a few hundred datapoints every 5s; a different proposition for every WARN line in the cluster.

Implication for the design

FlushMetrics hardcodes Locate(group, name, uint32(0), uint32(0)). The log sink must not inherit that — it needs a varying shardID (e.g. hash(node_id) % shard_num) so --logging-native-shard-num actually spreads load. Tier ② already part-syncs with the real per-queue shardID, so the machinery exists.

docs/operation/observability/providers.md describes this as the liaison using "nodeSelector to select a data node" — that collapses the two tiers and predates the write queue. Worth correcting separately.

References — existing implementations

Existing log implementation

File	Role
`pkg/logger/setting.go`	root logger, the single hardcoded `io.Writer`, `RegisterFlags`
`pkg/logger/logger.go`	module tree (`Named`), per-module levels, zap bridge
`docs/operation/observability/logging.md`	current logging docs

Existing self-stored metrics (the template)

File	Role
`banyand/observability/services/service.go`	`metricService` — flags, PreRun, Serve, scheduler
`banyand/observability/services/factory.go`	fans one metric out to every enabled provider
`banyand/observability/services/meter_native.go`	`nativeProviderFactory`, deferred schema init
`pkg/meter/native/provider.go`	`_monitoring` group + measure schema, `registerOrDefer`
`pkg/meter/native/collection.go`	`FlushMetrics` — batch publish, node routing
`docs/operation/observability/providers.md`	the two metrics providers

Per-role wiring

pkg/cmdsetup/standalone.go · data.go · liaison.go · banyand/backup/lifecycle/lifecycle.go

Implementation phases

Delivery order for §4. The organising principle: each phase adds exactly one new failure domain, so when something breaks it is never ambiguous which layer is at fault.

Phase	Scope	Adds	Why this order
1	seam + sink + buffer + schema; `standalone` + `data`	—	`queue.Local()` is in-process: no gRPC, no node selection, no `Locate`, no discovery. Proves the `LevelWriter` contract, `switchableWriter`, buffer + drop accounting, schema deferral and the `ElementValue` payload with the fewest moving parts. Recursion risk is at its weakest here
2a	`lifecycle`	network, 1 hop	The shortest possible remote path — `pub` + pick-first to a co-located data node. Introduces `Locate` failure → degrade-to-console and the `no_node` / `not_ready` drop paths. `lifecycle` already has this exact wiring for metrics, so it is a copy, not an invention
2b	`liaison`	two-tier path, write queue, recursion	The hard one: two independent node selections, the wqueue and part-sync (both of which log), denylist completeness, and the varying-`shardID` requirement. Deliberately last among online roles
3	`backup`	bootstrap ordering	Blocked on a decision, not on code — add a metadata client, or depend on the group already existing (created by whichever data node booted first). The latter is cheaper but makes backup's logging depend on boot order. Separated so it cannot gate 1 or 2
—	`restore`, `migration`	—	Out of scope by design (§3). A tool that runs while the database is down cannot log into it. Document, don't implement

Two properties worth keeping

Phase 1 is independently shippable. Standalone and single-data-node deployments get durable logs with none of the hard parts done — a real chunk of the user base, satisfied first.
Phase 2 was originally one step. It was split after the two-tier write path and the liaison self-loop surfaced: bundling the easiest remote role with the hardest stopped making sense once the difference was clear.

What could reorder this

If the module denylist proves trivially complete in practice, 2a and 2b could merge.
If the wqueue-vs-direct routing question is settled in favour of direct-to-data, 2b loses most of its risk and could move ahead of 2a.

Quick question on 4.2 GracefulStop:

Should swap -> io.Discard happen before draining the consumer?

If the consumer drains and exits first, any peer services logging during teardown will strand lines in the buffer with no consumer left to flush them. Swapping to io.Discard first cuts ingestion so the final drain cleanly catches everything before the cutoff.

Also happy to help implement or test this (e.g. Phase 1) if you'd like an extra pair of hands!

@Sruhvx-jpg

@Sruhvx-jpg Thanks for catching this—your concern is valid. Native log ingestion should stop before the final drain so shutdown logs are not stranded in the buffer.

@tristaZero is assigned to this issue. You’re very welcome to help test once the implementation is ready!

Native logging: configuration and routing

Proposed configuration and routing for native logging.

1. Independent configuration

Use separate flag/env namespaces, levels, and module overrides:

Destination	Flag prefix	Environment prefix	Level	Format
Normal console (`stderr`)	`--logging-`	`BYDB_LOGGING_`	`error` in production	Existing presentation settings
Native stream	`--logging-native-`	`BYDB_LOGGING_NATIVE_`	Default `info`	Always JSON

For example:

# Flags
--logging-env=prod --logging-level=error \
--logging-native-enabled=true --logging-native-level=info

# Equivalent environment variables
BYDB_LOGGING_ENV=prod
BYDB_LOGGING_LEVEL=error
BYDB_LOGGING_NATIVE_ENABLED=true
BYDB_LOGGING_NATIVE_LEVEL=info

--logging-native-enabled controls collection of this process's logs and defaults to
false. Console remains enabled. Keep existing normal-log flag names. Receiving and
forwarding other processes' native logs is a separate capability.
Native module overrides, buffer limits, flushing, TTL, and shard count each get matching
native flag/env pairs. An explicit flag overrides its corresponding environment value;
neither destination inherits the other's configuration. Native byte/event-size flag
names and defaults remain to be selected.

Admit locally generated events needed by either enabled sink, then filter per sink,
including module overrides and startup logging. With console=error, native=info,
and no overrides or recursion exclusions: info/warn are eligible only for native;
error for both; debug for neither. A dropped native info event has no console copy.
Native failure must not lower the console threshold.

Receivers validate incoming native requests but do not apply their own local logging
levels/module filters or echo received events to their console. Source identity and
element IDs remain unchanged through forwarding.

2. Stream write-request ring and routing

Buffer stream WriteRequest objects in a bounded, in-memory ring: one request per log
event, with the original JSON in the element's body tag. Do not buffer write-queue parts
or reuse the disk-backed queue's storage format. The consumer batches requests and adds
InternalWriteRequest routing fields where the internal write path requires them.

Native pending data stays in memory through liaison forwarding: no queue files, WAL,
or disk fallback. Destination stream persistence is expected.

Adaptive memory limit: use the memory protector's AvailableBytes() to calculate
bufferBudget = min(configuredMaxBytes, fraction * max(0, availableBytes - safetyReserve)).
Recalculate as memory availability changes; the fraction and reserve need defined
defaults and native flag/env settings. If availability is unknown (-1), use the
configured cap, not an unlimited budget. Account for queued and in-flight requests
with synchronized byte reservations; the protector's sampled value is not a reservation.
When over budget, evict oldest queued requests or reject new ones; never reclaim
in-flight memory or wait for capacity. Do not call the protector's blocking
AcquireResource() on the logging path. Count memory-pressure drops.

Producer	Route
Standalone / data	In-process write to its own stream storage.
Liaison	Existing shard-based routing, with a native write-request ring, then dispatch to data.
Backup / lifecycle / FODC agent beside data	Colocated data node's existing gRPC endpoint, normally `127.0.0.1:17912`.
FODC proxy	Liaison client endpoint, then the native forwarding path.
FODC agent beside liaison	Proposed: colocated liaison client endpoint; there is no local data node.
Restore / migration	Console only; the data tier is unavailable.

Use configured endpoints, not a new logging port. Reuse shard/node selection, not the
ordinary disk-backed queue or part-sync pipeline. The request-forwarding path still needs to be specified;
existing routing logic alone does not provide a memory-only transport.
Shard selection is deterministic, not per-batch rotation: use actual stream shard IDs.
Multiple shards permit distribution; a single shard does not. Whether a liaison's own
logs need peer forwarding remains a shard-ownership decision.

Phase 1 covers standalone/data. Add lifecycle and data-side agents with one-hop publishing,
then liaison-side agents/proxies with liaison integration. Backup requires schema
bootstrap; restore and migration remain excluded.

@Sruhvx-jpg

Thanks for all your feedback @Sruhvx-jpg @hanahmily — after going through all these inputs, I'd like to take most of them.

Two items may need our discussion:

--logging-modes=console,native vs --observability-modes=prometheus,native — keep the two schemes the same, or go with an independent one for logging?
Should we include FODC? It sits outside BanyanDB itself and looks like an optional observability module.

Details on both below, after what I'm adopting.

Accepted

#	Point	Note
①	Console `error` / native `info` — native more verbose than console	Our defaults had native ⊂ console, so the feature stored less than it printed. This was the biggest flaw in the proposal.
③	Independent namespaces, per-sink module overrides, no cross-inheritance	Adopting as specified.
④	Adaptive memory budget via the protector	Clearly better than our fixed `8192 entries`, which bounded count not bytes — an 8 KB stack trace and a 200 B info line cost the same under our scheme.
⑥	Deterministic shard selection, real stream shard IDs	Confirms what we found: `roundRobinSelector.Pick` ignores the resource name, so with `ShardNum: 1` every liaison resolves to the same node. All of `_monitoring` lives on one data node today.
—	@Sruhvx-jpg's `GracefulStop` ordering	Correct — ours drained before cutting ingestion, leaving a window the length of one network publish where lines were admitted and then silently stranded. Swapping first makes the cutoff explicit.

Two consequences of ① worth recording, because they invalidate parts of our write-up:

The root level must be the more verbose of the two sinks. zerolog filters once at the root, upstream of every writer, so with root=error and native=info the native sink never sees an info event at all.
"Console always has everything" is no longer true. Under native ⊃ console, a dropped native info has no console copy — a total loss. Our failure-mode section treated console as a safety net; that assumption is gone, which makes the drop counters the only evidence the data existed.

Discussion 1 — align with the metrics config pattern?

Metrics pattern today: a validated list, both destinations optional, at least one required.

--observability-modes=prometheus,native      {prometheus} | {native} | {both}

What we proposed: the same shape for logs.

--logging-modes=console,native               {console} | {native} | {both}

Same list, same three states, same validator — one mental model across both observability surfaces.

Your proposal uses --logging-native-enabled (bool) instead, which assumes console can never be disabled — so logs would have two states where metrics has three.

What do you think: keep the two schemes aligned, or go independent for logging?

Discussion 2 — should FODC be in scope?

Two reasons to leave rows 4 and 5 out:

1. It contradicts §1. Both rows need a liaison to accept and forward another process's logs — which §1 defers: "receiving and forwarding other processes' native logs is a separate capability."

2. FODC is optional, and the dependency is one-way.

fodc/  ──97 imports──►  banyand/ , pkg/
fodc/  ◄──0 imports───  banyand/ , pkg/

A cluster runs fine without FODC. Building liaison-side receive-and-forward into banyand/ to serve it would invert a dependency the codebase currently keeps clean.

Proposal — FODC becomes a consumer of this feature once forwarding exists, tracked separately:

Milestone	Needs new transport?
1. `standalone` / `data` — in-process	❌
2. `lifecycle`, `backup`, FODC agent beside data — one hop	❌
3. `liaison` — ring + tier-2 routing	ring only
4. (separate issue) forwarding → FODC proxy / agent-beside-liaison	✅

Two smaller confirmations

Scope of ⑤. Agreed on memory-only, no WAL, no disk fallback. On "existing routing logic alone does not provide a memory-only transport" — the gap may be smaller than that reads. Data nodes already accept TopicStreamWrite; publishing straight to a data node already happens (banyand/backup/lifecycle/steps.go:415); the liaison already holds tire2Client; shard selection already exists. The genuinely new piece looks like the byte-bounded ring with protector integration. If that matches your intent, the liaison row is cheaper than it appears.
"Existing shard-based routing … then dispatch to data." The liaison's existing TopicStreamWrite routing points at tier-1 (StreamLiaisonNodeRegistry over tire1Client, pkg/cmdsetup/liaison.go:102), so reusing it literally would dispatch to a peer liaison, not to data. I read the intent as reusing the selector mechanism with a new registry over tire2Client — 1 hop rather than 2. Please confirm.

The remaining details — buffering encoded bytes vs WriteRequest protos, whether to keep a bounded window for shutdown logs, where the per-sink module filter sits, and defaults for the TBD buffer flags — I'll work through in the design doc and bring back rather than take up space here.

@Sruhvx-jpg — help testing Phase 1 would be very welcome once there's something to run.

Thanks for incorporating the feedback. On the two discussion points:

Keep --logging-native-enabled. Native self-storage is an additional capability, independent of normal logging. Zerolog supports file and other writer outputs—not just console—although BanyanDB currently uses stderr. Preserving the normal logging path matters more than matching the metrics configuration shape; native-only logging is not a requirement here.
Keep FODC in scope, with phased delivery. “Receiving and forwarding other processes’ native logs is a separate capability” distinguishes forwarding from collecting the current process’s logs; it does not defer forwarding to another issue. A generic liaison-side receiver/forwarder does not invert the dependency: FODC consumes that capability, and BanyanDB need not import FODC packages.

The proposed implementation order is fine, but the final forwarding phase should remain part of this feature. Please carry the remaining implementation details into the design doc under the previously stated routing and memory-only requirements.

Both open points are settled: --logging-native-enabled stays a bool defaulting to false, and FODC stays in scope with forwarding as the final phase of this feature. The design is complete; what follows is the whole solution.

1. Feature introduction and scope

BanyanDB already self-stores its own telemetry, but only half of it. Metrics have a native path into the database; logs do not.

	Metrics today	Logs today
Destination	`_monitoring` group, one Measure per metric	one `io.Writer`, fixed at `logger.Init` time
Turned on by	`--observability-modes=native`	always on, no choice of destination
Queryable, retained	yes, with a TTL	no, stderr is the log driver's problem

During an incident the metrics are in the database and the logs are not. When a pod recycles, they are gone.

This feature adds a second, optional destination for logs: native self-storage into a new _monitoring_log stream group, behind --logging-native-enabled (bool, default false). Normal logging keeps its current behaviour — one zerolog writer, stderr today, though zerolog also supports file writers — and is never disabled.

	Roles	Rationale
In scope	standalone, data, liaison	reach stream storage directly
In scope	lifecycle, backup	colocated producers, using the receiver on their host's data node
In scope	FODC — the on-demand diagnostics collector, agents plus a proxy — final phase	a generic receiver/forwarder serves it, so BanyanDB imports nothing from FODC
Out of scope	restore, migration	they run when the data tier is unavailable

The sections that follow go roles, approach, configuration, failure modes, phases.

2. Role → function → log destination

One binary, several roles. One property decides the destination: whether the process owns a storage engine or must reach one.

Role	Function	Log destination
`standalone`	full server	own storage, in-process (`queue.Local()`) — as native metrics already do
`data`	shards, writes, queries	own storage, in-process (`TopicStreamWrite`)
`liaison`	front door: validate, route, batch	one hop to a data node over its tier-2 client, the pool spelled `tire2Client` in the code
`backup`, `lifecycle`, FODC agent beside data	snapshot, migration, diagnostics	colocated data node's cluster server, `127.0.0.1:17912`
FODC proxy, FODC agent beside a liaison	diagnostics, no colocated data node	liaison gRPC port (`--grpc-port`, default 17912)
`restore`, `migration`	offline tools	normal logging only

A liaison has local disk: it runs a two-tier write queue whose normal StreamService/Write route ends in part files — more involved than the name suggests. What it lacks is the storage engine, so it can hold a batch in flight but can never be where logs are read back from.

Three shapes: in-process; the liaison selecting a data node by shard; and every other producer publishing into the generic receiver, subscribed on data nodes' cluster server and on the liaison.

Restore and migration use normal logging only by construction, not as a gap to close later: a tool that runs while the database is down cannot log into it. The native flags are registered only by the server, backup, lifecycle and FODC binaries.

3. Proposed approach

3.1 The writer seam

One event, two level-gated writers under zerolog's MultiLevelWriter. The zerolog level becomes an admission floor — the more verbose of the enabled sinks — and each sink then applies its own threshold, letting native run at info while normal logging stays at error.

flowchart LR
  E["log event"] --> F{"admission floor: min over enabled sinks"}
  F -->|below| X["dropped"]
  F -->|at or above| M["MultiLevelWriter"]
  M --> C["gate: --logging-level"] --> S["stderr or file"]
  M --> N["gate: --logging-native-level"] --> R["in-memory ring"]
  R --> K["consumer"] --> P["shard and node selection"] --> D["stream log"]

Worked example, normal logging at error and native at info, so the floor is info.

event	admitted	normal sink	native sink
debug	no	–	–
info / warn	yes	dropped by its own gate	stored
error	yes	printed	stored

Per-module thresholds resolve once when a logger is built, with normal level, native level and exclusion cached on it, never by parsing JSON per line. Matching is by prefix, since an override truncates the module name it stamps: --logging-modules=liaison-grpc stamps LIAISON-GRPC.STREAM-T1 as LIAISON-GRPC.

3.2 Buffer and transport

property	value
unit	one stream `WriteRequest` per event, drawn from and returned to a pool
payload	values carried in tags. The `data:fields` tag holds only keys with no tag of their own — no duplicated body
bounds	`--logging-native-max-bytes` `32mb`, `--logging-native-max-event-bytes` `64kb` (proposed defaults, section 4)
triggers	`--logging-native-flush-interval` `1s` and `--logging-native-flush-size` `100`
durability	none — no queue files, no WAL, no disk fallback

The consumer is a dedicated goroutine selecting over the closer, the flush ticker and the ring, so both triggers fire.

When native is enabled the ring is live from logger initialisation; service start only begins draining it, so no startup line is stranded in a buffer nobody drains.

3.3 Routing and shard selection

Shard IDs are deterministic and real, computed as the normal write path computes them — entity locator from the fetched schema, then hash modulo shard count — not the hardcoded shard 0 of native metrics. The default is --logging-native-shard-num=2; at 1 every event keys to one shard and the whole cluster funnels onto a single data node. The count can be raised later through the group schema, though lowering it silently hides shards already written.

Decided: keep the selection, replace the transport. Reusing the liaison's existing shard-based routing and never touching disk cannot both hold. That route terminates in the liaison's disk-backed write queue and writes part files. The liaison therefore keeps the shard and node selection and publishes directly to the selected data node over its tier-2 client, bypassing the write queue. The new native log topic serves the receiver's foreign producers, not the liaison's own route.

The receiver is deliberately thin.

Rule	Behaviour
Validation	`InvalidArgument` on a nil or non-millisecond timestamp, empty `element_id` or `node_id`, oversized body, out-of-enum level, oversized batch
Back-pressure	`ResourceExhausted` when its own ring is over budget; never blocks
Filtering	no local level, no module override — what arrives is what is stored
Echo	received events never enter its own logging path
Identity	`element_id` and identity tags pass through unchanged; identity must live in the element

--logging-native-enabled governs only a process's own collection: a node with it false still accepts, routes and persists forwarded writes.

3.4 Schema

object	value
group	`_monitoring_log`, catalog `STREAM`, TTL `--logging-native-ttl-days` (7), `Replicas` `0` — matching `_monitoring`
stream	`log`, one shared by every role
entity	`node_id`, `level`
searchable tags	`node_id`, `node_type`, `module`, `level`, `grpc_address`, `http_address`, `message`, `log_id`
data tag	`fields`, binary — the leftover keys only, omitted when there are none
index rules	none in v1

level is an entity tag; entity tags accept only equality and set membership, so selecting several levels uses IN rather than a negation. module stays an ordinary tag — module strings splice in group, measure and task names, so entity membership would give unbounded series cardinality. Nothing stores the original line verbatim: known keys become tags, and only the remainder lands in fields.

4. Parameters and configuration

Two independent namespaces; neither inherits from the other.

namespace	flags	environment	notes
normal logging	`--logging-*`	`BYDB_LOGGING_*`	unchanged; typically `error` in production
native stream	`--logging-native-*`	`BYDB_LOGGING_NATIVE_*`	default level `info`, always JSON

Every dashed flag binds to BYDB_<UPPER_SNAKE>, and the env value applies only when the flag was not set on the command line, so an explicit flag wins.

flag	default	meaning
`--logging-native-enabled`	`false`	collect this process's own logs natively
`--logging-native-level`	`info`	root level of the native sink
`--logging-native-modules` / `--logging-native-levels`	`nil`	per-module overrides, prefix-matched, length-checked
`--logging-native-exclude-modules`	built-in prefix set	modules never sent natively, breaking the write-path feedback loop; replaces, not appends
`--logging-native-flush-interval`	`1s`	longest a buffered event waits
`--logging-native-flush-size`	`100`	events that trigger a batch
`--logging-native-write-timeout`	`5s`	per-batch publish timeout
`--logging-native-drain-timeout`	`5s`	bound on the final drain at shutdown
`--logging-native-max-bytes` *	`32mb`	configured ring cap
`--logging-native-max-event-bytes` *	`64kb`	oversize events dropped whole
`--logging-native-memory-fraction` *	`0.02`	fraction of available memory in the adaptive budget
`--logging-native-memory-reserve` *	`64mb`	reserve subtracted before the fraction
`--logging-native-shard-num`	`2`	shards of `_monitoring_log`; raisable later through the group schema
`--logging-native-ttl-days`	`7`	retention
`--logging-native-receiver-enabled`	`false`	accept forwarded writes, wherever hosted; independent of collection

* Names and defaults for the four starred flags are proposals, not settled. Level semantics are in section 3.1; the four existing --logging-* flags keep their names and defaults.

With --logging-native-enabled=false — the default — behaviour is exactly as today: the ring is never allocated, no schema and no counters are created, and the cost is one atomic pointer load per event.

5. Failure modes

event -> producer -> buffer(ring) -> consumer -> schema -> selection -> publish -> destination
                                                                                       |
                      GracefulStop: stop admission, then bounded drain ----------------+

Every stage that can fail has a row, carrying one behaviour and one reason from the closed nine-value reason set on native_dropped_total.

Stage	Situation	Behaviour	Counter reason
producer	event over `--logging-native-max-event-bytes`	dropped whole, never truncated — a truncated JSON body is unparseable	`oversize_event`
buffer	ring over budget	drop the newest event, the writer still returning `(len(p), nil)`; queued events are never evicted, so a burst keeps its head	`buffer_full` / `memory_pressure`
consumer	stalled in a slow publish	the ring absorbs it; batches capped at 25% of budget, one in flight, so a stall cannot pin admission	– gauge `native_buffer_bytes{state=in_flight}`
schema	create fails for anything but `AlreadyExists`, or the group is dropped at runtime	retry next flush, batch in hand dropped	`schema_unavailable`
schema	existing `log` stream is incompatible	refuse, one error through normal logging, sink stays disabled	`schema_incompatible`
selection	`Locate` errors or returns an empty node ID	count and return; never publish to an empty node	`locate_failed`
local publish	listener unhealthy	the bus skips it and drops the payload while returning an error, so any error means full batch loss	`publish_failed`
remote publish	error, breaker open, admission timeout, auth rejection	batch dropped, never re-queued; classified from the per-node error map from `Close`	`publish_failed`
destination	local topic not yet subscribed	retain, N retries, then drop	`destination_unready`
read path	a data node has not yet caught up on the schema	its query processor returns stream-not-exist and the distributed planner discards the whole response, blanking the query. Pre-existing for any group created while the cluster is running, user groups included — not introduced here. Mitigation is a one-site change in the planner, out of scope for this feature	–
shutdown	buffer still full at the deadline	remainder dropped once `--logging-native-drain-timeout` expires; teardown never blocked	`shutdown_deadline`

No row blocks, retries forever, or falls back to disk. Loss is bounded and always counted.

5.1 Memory budget

budget = min(max-bytes, memory-fraction * max(0, availableBytes - memory-reserve))
availableBytes < 0  ->  budget = max-bytes          # unknown is not unlimited

All three terms are native flags whose defaults are proposals. The budget is recomputed every 5s, counting queued and in-flight bytes alike. The sink reads AvailableBytes() only — never the blocking AcquireResource on the logging path, never waiting, never reclaiming bytes in flight; over budget it drops and counts. Phase 1 runs on the cap alone, the adaptive term arriving in phase 2.

Process	Adaptive term	Why
standalone, data	binds	protector registered as a run unit
everything else	cap only	availability reports `-1`, or no protector is registered. We leave the liaison as it is today rather than registering one, since that would arm load shedding that has never run

That gap is pre-existing. The liaison's protector is deliberately left unregistered, because registering it would also arm liaison load shedding that has never run in production. The sink needs no change if it is registered later — the same formula simply begins to adapt — so the cap is sized to be safe on its own.

Normal logging is never degraded by the sink. Native is the more verbose sink, so at normal error and native info a dropped native info event has no normal copy anywhere — the drop counters are the only record it existed.

With --observability-modes=native alone those counters travel the same transport as the logs, so the same outage loses both. The sink therefore also emits a periodic drop summary straight to stderr, and we recommend keeping prometheus enabled alongside.

6. Implementation phases

Each phase adds exactly one new failure domain, so a breakage is never ambiguous about which layer is at fault.

Phase	Scope	What it de-risks
1	Writer seam in `pkg/logger`, both gates, ring, consumer, `_monitoring_log` group and `log` stream; standalone and data in-process	Capture and schema, with no network hop or node selection
2	Liaison route — tier-2 registry and its own selector over `tire2Client` — adaptive budget, metrics	Cross-node routing, and the adaptive budget where a protector is registered
3	Generic native receiver and forwarder on liaison and data nodes. Also: a batch message for the new topic — no `repeated WriteRequest` exists in `write.proto` today — plus method-policy rows and their contract test, without which the liaison refuses to boot	Foreign-event admission: validation, identity passthrough, back-pressure
4	Colocated producers (backup, lifecycle, FODC agent beside data) and the FODC proxy, onto the phase-3 receiver. `backup` and `lifecycle` reuse their existing connection settings; the FODC binaries have none to reuse — every FODC dial is currently insecure and the proxy has no outbound BanyanDB client, so both need new address and TLS flags	Producer integration alone; the receiver is proven

The receiver precedes those producers by design, so they never need a metadata client or their own copy of the shard rule. Background: implementation phases.

Phase 1 is independently shippable: standalone and single-data-node deployments get queryable, retained logs without cross-node routing or the adaptive budget.

Decisions taken

All five open questions are answered and folded in above.

	Decision
Liaison transport	direct tier-2 publish, reusing shard and node selection, bypassing the disk-backed write queue
Shard count	`2`, changeable later through the group schema
Ring representation	pooled `WriteRequest` objects. Requests and their nested allocations are reused; they return to the pool only after a publish completes or the request is dropped, with retained references cleared and oversized buffers discarded. Byte limits still account for queued and in-flight requests
Overflow policy	drop newest; admission never waits for capacity
Entity	`[node_id, level]`, `module` as an ordinary tag. Selecting several levels uses `IN`

Inherited rather than decided, following what the project already does: a data node stores its own logs locally, as native metrics do; Replicas 0, matching _monitoring; the liaison's memory protector is left unregistered, so the cap is the only term there; and the schema-propagation window is accepted, as it is pre-existing for any group created while the cluster is running.

Two consequences of the ring decision are raised separately below this comment: where the remaining structured fields live, and where the parse happens.

The design doc is updated to match and will land with the implementation PR.

Thanks for working through the design. On the five open questions:

Liaison transport: Direct tier-2 publishing is fine. Reuse shard/node selection and bypass the disk-backed write queue.
Shard count: Start with 2. Shard count can be changed through the group schema. I verified the existing TestGroupCRUD, which updates a stream group from 2 to 4 shards and reads back the updated value; the test passes.
Ring representation: Keep WriteRequest objects, using a pool to reuse requests and their nested allocations. The stream schema defines the keys—level, timestamp, content, etc.—and requests carry the corresponding values. Do not retain the original JSON as an additional body. The proposed 5–7× comparison therefore does not describe the intended representation and should be removed unless supported by measurements of this design.

Pooling reduces allocation churn and GC pressure; byte limits must still account for queued and in-flight requests. Return requests to the pool only after publishing completes or they are dropped, clear retained references, and avoid retaining oversized buffers.
Overflow policy: Drop newest. Admission must never wait for buffer capacity.
Entity: Use [node_id, level]. Keep module as an ordinary tag. We accept the current entity-filter restriction to EQ/IN; queries selecting multiple levels should use IN.

Please incorporate these decisions into the design doc and make the schema, configuration tables, and memory-accounting description consistent with them.

All five decisions are folded into the design doc and the solution comment above: direct tier-2 publish, shard count 2, pooled WriteRequest objects, drop-newest, and entity [node_id, level] with module as an ordinary tag. The 5–7× figure is removed — it described a representation with a retained body, so it does not apply here and we have not measured this one.

Two consequences of the ring decision are worth settling before implementation.

1. Where the remaining structured fields live

Removing the duplicated body is clear. What it surfaces is that our log lines are not a fixed key set: 172 distinct structured field names appear across banyand/ and pkg/ today — group, listenAddr, shard, sender_node, elapsed_ms, collector_count and so on — and which ones appear depends entirely on the call site.

{"level":"warn","module":"MEASURE","time":"…","message":"flush took longer than expected",
 "group":"sw_metric","elapsed_ms":8200}
  └── tags ────────────────────────────────────────────────┘ └── no tag exists ──────┘

Proposal: a single data:fields tag carrying only the keys that have no tag of their own, omitted entirely when there are none — which is the common case for a plain .Msg(...). That is not the original JSON and not an additional copy of anything: level, module and message appear once, in their tags, and never inside fields.

The alternative is to drop those keys, which costs the part of a line that usually identifies what actually happened. Happy either way — just flagging that "no additional body" and "keep the field data" are separable, and the second one is not settled by the first.

2. Where the parse happens

Holding WriteRequest objects in the ring means each one is built before it goes in, on the goroutine that called the logger — and building it needs the field values, which exist only inside the encoded JSON that zerolog hands the writer. So the parse lands on the hot path.

	Ring holds	Parse runs on
Earlier proposal	raw line + header	the consumer goroutine, at flush
As decided	`WriteRequest` objects	the calling goroutine, at push

level still arrives free as an enum and module free from a per-module writer, so it is one small parse of roughly a 200-byte object — single-digit microseconds — but it happens on every log call on every code path, including the write path.

We are fine to proceed with it; the cost is real but bounded, and pooling offsets the allocation half. Raising it only because the earlier text said gating never parses, and that will no longer be true at push. If the latency matters more than the representation, the raw-line variant is still available and we can benchmark rather than argue it.

docs: add the self-stored logs design document (#1367)

[Feature] Store BanyanDB's own logs in BanyanDB (self-stored logs) #14076

Description

Search before asking

Description

1. Feature introduction

2. Deployment architecture

3. Role → function → log destination

4. Proposed approach

4.1 The seam — one event, two writers

The writer contract

4.2 Lifecycle — deferred activation

4.3 Write workflow

4.4 Schema

4.5 Two things that differ from metrics

4.6 Implementation phases

5. Parameters and configuration

6. Reading logs back

7. Failure modes

8. References

Related issues

Are you willing to submit a pull request to implement this on your own?

Code of Conduct

Activity

tristaZero commented on Sep 11, 2026

Reference — the liaison's two-tier write path

How a node is chosen

Consequences for a single-shard group

Implication for the design

References — existing implementations

tristaZero commented on Sep 11, 2026

Implementation phases

Two properties worth keeping

What could reorder this

Sruhvx-jpg commented on Sep 12, 2026

hanahmily commented on Sep 13, 2026

hanahmily commented on Sep 13, 2026

Native logging: configuration and routing

1. Independent configuration

2. Stream write-request ring and routing

tristaZero commented on Sep 14, 2026

Accepted

Discussion 1 — align with the metrics config pattern?

Discussion 2 — should FODC be in scope?

Two smaller confirmations

hanahmily commented on Sep 16, 2026

tristaZero commented on Sep 16, 2026

1. Feature introduction and scope

2. Role → function → log destination

3. Proposed approach

3.1 The writer seam

3.2 Buffer and transport

3.3 Routing and shard selection

3.4 Schema

4. Parameters and configuration

5. Failure modes

5.1 Memory budget

6. Implementation phases

Decisions taken

hanahmily commented on Sep 19, 2026

tristaZero commented on Sep 20, 2026

1. Where the remaining structured fields live

2. Where the parse happens

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions