Skip to content

configuration.md

Latest commit

 

History

History
882 lines (680 loc) · 31.7 KB

configuration.md

File metadata and controls

882 lines (680 loc) · 31.7 KB
Raw

HugeGraph PD Configuration Guide

This document provides comprehensive configuration guidance for HugeGraph PD, including parameter descriptions, deployment scenarios, and production tuning recommendations.

Table of Contents

Configuration File Overview

Configuration Files

PD uses the following configuration files (located in conf/ directory):

File Purpose
application.yml Main PD configuration (gRPC, Raft, storage, etc.)
log4j2.xml Logging configuration (log levels, appenders, rotation)
verify-license.json License verification configuration (optional)

Configuration Hierarchy

application.yml
├── spring              # Spring Boot framework settings
├── management          # Actuator endpoints and metrics
├── logging             # Log configuration file location
├── license             # License verification (optional)
├── grpc                # gRPC server settings
├── server              # REST API server settings
├── pd                  # PD-specific settings
├── raft                # Raft consensus settings
├── store               # Store node management settings
└── partition           # Partition management settings

Core Configuration Parameters

gRPC Settings

Controls the gRPC server for inter-service communication.

grpc:
  host: 127.0.0.1    # gRPC bind address
  port: 8686         # gRPC server port
Parameter Type Default Description
grpc.host String 127.0.0.1 IMPORTANT: Must be set to actual IP address (not 127.0.0.1) for distributed deployments. Store and Server nodes connect to this address. In Docker bridge networking, set this to the container hostname (e.g., pd0) via HG_PD_GRPC_HOST env var.
grpc.port Integer 8686 gRPC server port. Ensure this port is accessible from Store and Server nodes.

Production Notes:

  • Set grpc.host to the node's actual IP address (e.g., 192.168.1.10)
  • Avoid using 0.0.0.0 as it may cause service discovery issues
  • Ensure firewall allows incoming connections on grpc.port

REST API Settings

Controls the REST API server for management and monitoring.

server:
  port: 8620    # REST API port
Parameter Type Default Description
server.port Integer 8620 REST API port for health checks, metrics, and management operations.

Endpoints:

  • Health check: http://<host>:8620/actuator/health
  • Metrics: http://<host>:8620/actuator/metrics
  • Prometheus: http://<host>:8620/actuator/prometheus

REST Authentication Settings

Every REST request except the probes below must carry HTTP Basic auth: one of the internal service names (hg, store, hubble, vermeer) as the user, and the shared secret as the password. A missing or wrong credential gets HTTP 401. Unauthenticated paths: /v1/health, /v1/ready, /actuator/** and /v1/prom/targets/*.

auth:
  secret-key: <a value you generate, e.g. `openssl rand -hex 24`>
Parameter Type Default Description
auth.secret-key String none (required) Password checked against the Basic credential. There is no default: a secret shipped in the source tree would be published to everyone. While it is empty PD refuses every authenticated REST request and logs an error naming this parameter, and PD refuses to start at all if it is set to the value that earlier revisions carried as a placeholder.

Every REST client needs the same value: the Server's bin/wait-storage.sh reads it from PD_AUTH_PASSWORD, Hubble from operations.pd.password, and the Docker image takes HG_PD_AUTH_SECRET_KEY.

curl -u hg:<secret> http://<host>:8620/v1/stores

-u puts the secret in curl's process arguments, where any local account can read it while the call runs, and PD REST is plain HTTP. On a shared host, or across a network you do not control, keep the secret out of argv by reading it from a file mode 0600:

printf 'user = "hg:%s"\n' "${PD_SECRET}" > pd.curlrc && chmod 600 pd.curlrc
curl -K pd.curlrc http://<host>:8620/v1/stores

Raft Consensus Settings

Controls Raft consensus for PD cluster coordination.

raft:
  address: 127.0.0.1:8610                  # This node's Raft address
  peers-list: 127.0.0.1:8610               # All PD nodes in the cluster
Parameter Type Default Description
raft.address String 127.0.0.1:8610 Raft service address for this PD node. Format: <ip>:<port>. Must be unique across all PD nodes.
raft.peers-list String 127.0.0.1:8610 Comma-separated list of all PD nodes' Raft addresses. Used for cluster formation and leader election.
raft.rpc-timeout Integer 10000 Timeout of a Raft RPC between PD nodes, in milliseconds. It is also the deadline of the ReadIndex wait PD runs before it reads an id counter: getId and getCId fail when the leader cannot confirm and apply a ReadIndex in this time, for example while a new leader is still committing its first entry.
raft.rpc-connect-timeout Integer 1000 Timeout for opening a connection to a peer, in milliseconds. A candidate opens its connections one peer at a time while it holds the Raft node lock, so an election waits this long for each peer that accepts the connection but does not answer (a stopped process, a lost host). It is the timeout of the Ping that opens a connection; requests on a connection that is already open use raft.rpc-timeout.
raft.rpc-install-snapshot-timeout Integer 0 Timeout for sending a snapshot to a follower that fell behind the log, in milliseconds. It bounds the transfer of the whole metadata store. 0 uses raft.rpc-timeout, the value this timeout had before it became a separate option; raise it if large snapshots time out.

Critical Rules:

  1. raft.address must be unique for each PD node
  2. raft.peers-list must be identical on all PD nodes
  3. raft.peers-list must contain all PD nodes (including this node)
  4. Use actual IP addresses, not 127.0.0.1, for multi-node clusters
  5. Cluster size should be odd (3, 5, 7) for optimal Raft quorum

Example (3-node cluster):

# Node 1
raft:
  address: 192.168.1.10:8610
  peers-list: 192.168.1.10:8610,192.168.1.11:8610,192.168.1.12:8610

# Node 2
raft:
  address: 192.168.1.11:8610
  peers-list: 192.168.1.10:8610,192.168.1.11:8610,192.168.1.12:8610

# Node 3
raft:
  address: 192.168.1.12:8610
  peers-list: 192.168.1.10:8610,192.168.1.11:8610,192.168.1.12:8610

Docker Bridge Network Deployment

When deploying PD in Docker with bridge networking (e.g., docker/docker-compose-3pd-3store-3server.yml), container hostnames are used instead of IP addresses. Configuration is injected via HG_PD_* environment variables:

# pd0 — set via HG_PD_RAFT_ADDRESS and HG_PD_RAFT_PEERS_LIST env vars
raft:
  address: pd0:8610
  peers-list: pd0:8610,pd1:8610,pd2:8610

# pd1
raft:
  address: pd1:8610
  peers-list: pd0:8610,pd1:8610,pd2:8610

# pd2
raft:
  address: pd2:8610
  peers-list: pd0:8610,pd1:8610,pd2:8610

The grpc.host must also use the container hostname (e.g., pd0) set via HG_PD_GRPC_HOST. Do not use 127.0.0.1 or 0.0.0.0 in bridge networking mode.

See docker/README.md for the full environment variable reference.

PD Core Settings

Controls PD-specific behavior.

pd:
  data-path: ./pd_data              # Metadata storage path
  patrol-interval: 1800             # Partition rebalancing interval (seconds)
  initial-store-count: 1            # Bootstrap gate while no shard groups exist
  min-store-count: 0                # First-bootstrap seed for the persisted placement floor
  initial-store-list: 127.0.0.1:8500  # Auto-activated stores
Parameter Type Default Description
pd.data-path String ./pd_data Directory for RocksDB metadata storage and Raft logs. Ensure sufficient disk space and fast I/O (SSD recommended).
pd.patrol-interval Integer 1800 Interval (in seconds) for partition health patrol and automatic rebalancing. Lower values = more frequent checks.
pd.initial-store-count Integer 1 Required live Store count while no shard groups exist. It does not disable existing voting majorities after bootstrap.
pd.min-store-count Integer 0 First-bootstrap seed for persisted PDConfig.min_store_count, an extra eligible-Store floor for new placement and drain preflight. Existing persisted configuration takes precedence. Zero means no extra floor beyond the desired replica count.
pd.initial-store-list String 127.0.0.1:8500 Comma-separated list of Store gRPC addresses to auto-activate on startup. Useful for bootstrapping.

Production Recommendations:

  • pd.data-path: Use dedicated SSD with at least 50GB free space
    • PD opens its metadata RocksDB at <pd.data-path>/rocksdb. If its LOCK is still held, usually by a previous PD process that is still stopping, PD retries for up to 30 seconds and then fails startup. Run PD in the foreground (-d false) under a supervisor so that a failed start exits and is restarted.
    • Never delete the LOCK files in <pd.data-path>/rocksdb/ or <pd.data-path>/pd_raft/log/ to get past this: while the previous process is alive, two processes then write the same database. Stop or kill the old process instead.
  • pd.patrol-interval:
    • Development: 300 (5 minutes) for fast testing
    • Production: 1800 (30 minutes) to reduce overhead
    • Large clusters: 3600 (1 hour)
  • pd.initial-store-count: Set to expected initial store count (e.g., 3 for 3 stores)

Store bootstrap, placement and runtime quorum

The PD distribution and Java fallback default to one initial Store and one shard replica (partition.default-shard-count: 1). Helm explicitly selects three shard replicas when store.replicas >= 3, otherwise one. pd.initial-store-count controls the initial number of Stores, not the replica count; set both independently for other multi-Store deployments.

Bootstrap admission is separate from the metadata health hint used for existing data. Before any group exists, readiness requires enough eligible Up Stores for the initial count, desired replica count and persisted placement floor. Pending Stores need explicit management admission to Up; an old active lease does not make them placement capacity. An existing group is healthy only when a strict majority of its known Leader/Follower voter IDs have fresh Store leases. Exiting sources can still contribute a live vote while serving; they cannot receive new replicas. Learner and pending None targets do not count as voters. Zero-voter groups and duplicate IDs fail the health check. The result remains the conjunction of every group's view, rather than a per-request-group availability gate.

The known-voter check keeps an ordinary migration with two retained live voters and one pending None target from stopping writes. PD currently writes target membership before Raft commits it, so this metadata hint is not proof of the actual committed membership, successful writes or a finished operation. JRaft still enforces actual quorum. Final node removal requires checking actual membership and full replica recovery separately.

Ordinary Up Store heartbeats already recompute this health hint before returning. An Exiting source now also rechecks a preceding Cluster_Warn against known voting quorum, rather than forwarding Warn as a write veto. Lost majorities remain NotReady; explicit Fault and maintenance Offline responses remain unchanged. Partition warnings still describe observed replica state and are not erased from group metadata. This does not unify every writer of clusterStats or resolve the independent registration/lazy-read windows and maintenance-state aggregation issues.

New allocations and reconfiguration require at least the configured replica count and the persisted eligible-Store floor. They fail rather than reducing replicas to the live Store count or silently converting two copies to one. Drain preflight checks the current quorum with the still-serving source included, then requires enough eligible destinations excluding that source; this permits a healthy one-copy source to begin copying before removal. A single-copy group can temporarily report NotReady when PD replaces its sole known voter with a pending target; this policy does not guarantee uninterrupted traffic during that transition. Final retirement has its own empty-engine/member gate. Normal node removal must use Exiting. A direct REST Tombstone update is a force-removal entry point and does not guarantee lossless removal. When PD still references that nonterminal Store, insufficient remaining eligible capacity is rejected before the terminal state is persisted. Group-free cleanup and metadata-only updates to an already terminal Store do not start a new migration.

Older PD metadata omits min_store_count, which reads as zero and means automatic placement minimum. It must not inherit an old bootstrap count. Explicit positive floors are seeded and reloaded through ConfigService; they do not stop an existing healthy quorum. The current proto3 partial-update merge cannot reset a positive floor to zero; resending zero retains the old value. Resetting this floor requires a future field-presence-aware API.

Without a new persisted initialization marker, an entirely group-free cluster is treated as bootstrap again, including after all groups are deliberately deleted. Configuration loading also retains its existing exception handling; the metadata health failure check does not claim to make configuration loading fail closed.

Store Management Settings

Controls how PD monitors and manages Store nodes.

store:
  max-down-time: 172800              # Store permanent failure threshold (seconds)
  monitor_data_enabled: true         # Enable metrics collection
  monitor_data_interval: 1 minute    # Metrics collection interval
  monitor_data_retention: 1 day      # Metrics retention period
Parameter Type Default Description
store.max-down-time Integer 172800 Time (in seconds) after which a Store is considered permanently offline and its partitions are reallocated. Default: 48 hours.
store.monitor_data_enabled Boolean true Enable collection of Store metrics (CPU, memory, disk, partition count).
store.monitor_data_interval Duration 1 minute Interval for collecting Store metrics. Format: <value> <unit> (second, minute, hour).
store.monitor_data_retention Duration 1 day Retention period for historical metrics. Format: <value> <unit> (day, month, year).

Production Recommendations:

  • store.max-down-time:
    • Development: 300 (5 minutes) for fast failover testing
    • Production: 86400 (24 hours) to avoid false positives during maintenance
    • Conservative: 172800 (48 hours) for network instability
  • store.monitor_data_interval:
    • High-frequency monitoring: 10 seconds
    • Standard: 1 minute
    • Low overhead: 5 minutes
  • store.monitor_data_retention:
    • Short-term: 1 day
    • Standard: 7 days
    • Long-term: 30 days (requires more disk space)

Partition Settings

Controls partition allocation and replication.

partition:
  default-shard-count: 1              # Replicas per partition
  store-max-shard-count: 12           # Max partitions per store
Parameter Type Default Description
partition.default-shard-count Integer 1 Number of replicas per partition. Typically 3 in production for high availability.
partition.store-max-shard-count Integer 12 Maximum number of partition replicas a single Store can hold. Used for initial partition allocation.

Initial Partition Count Calculation:

initial_partitions = (store_count * store_max_shard_count) / default_shard_count

Example:

  • 3 stores, store-max-shard-count=12, default-shard-count=3
  • Initial partitions: (3 * 12) / 3 = 12 partitions
  • Each store hosts: 12 * 3 / 3 = 12 shards (4 partitions as leader + 8 as follower)

Production Recommendations:

  • partition.default-shard-count:
    • Development/Testing: 1 (no replication)
    • Production: 3 (standard HA configuration)
    • Critical systems: 5 (maximum fault tolerance)
  • partition.store-max-shard-count:
    • Small deployment: 10-20
    • Medium deployment: 50-100
    • Large deployment: 200-500
    • Limit based on Store disk capacity and expected data volume

Management and Metrics

Controls Spring Boot Actuator endpoints for monitoring.

management:
  metrics:
    export:
      prometheus:
        enabled: true    # Enable Prometheus metrics export
  endpoints:
    web:
      exposure:
        include: "health,metrics,prometheus"   # Allowlist; see note below
Parameter Type Default Description
management.metrics.export.prometheus.enabled Boolean true Enable Prometheus-compatible metrics at /actuator/prometheus.
management.endpoints.web.exposure.include String "health,metrics,prometheus" Actuator endpoints to expose. Actuator is served by its own handler mapping, which the REST authentication interceptor is not attached to, so every endpoint listed here is reachable without a credential on port 8620. This allowlist is what bounds which endpoints exist there, so prefer it over "*". The interceptor's /actuator/** exclusion records the same intent but is not what makes these paths anonymous. In the PD Docker image the entrypoint emits this key in SPRING_APPLICATION_JSON, which outranks a mounted conf/application.yml, so editing it there has no effect; set HG_PD_ACTUATOR_EXPOSURE on the container instead. That variable defaults to the same allowlist and refuses a value containing *, because /actuator/env returns the SPRING_APPLICATION_JSON entry verbatim, PD's REST secret included.

Deployment Scenarios

Single-Node Deployment (Development/Testing)

Minimal configuration for local development.

grpc:
  host: 127.0.0.1
  port: 8686

server:
  port: 8620

raft:
  address: 127.0.0.1:8610
  peers-list: 127.0.0.1:8610

pd:
  data-path: ./pd_data
  patrol-interval: 300              # Fast rebalancing for testing
  initial-store-count: 1
  initial-store-list: 127.0.0.1:8500

store:
  max-down-time: 300                # Fast failover for testing
  monitor_data_enabled: true
  monitor_data_interval: 10 seconds
  monitor_data_retention: 1 day

partition:
  default-shard-count: 1            # No replication
  store-max-shard-count: 10

Characteristics:

  • Single PD node (no HA)
  • No replication (default-shard-count=1)
  • Fast rebalancing for quick testing
  • Suitable for development, not for production

3-Node Cluster Deployment (Production Standard)

Recommended configuration for production deployments.

Node 1: 192.168.1.10

grpc:
  host: 192.168.1.10
  port: 8686

server:
  port: 8620

raft:
  address: 192.168.1.10:8610
  peers-list: 192.168.1.10:8610,192.168.1.11:8610,192.168.1.12:8610

pd:
  data-path: /data/pd/metadata
  patrol-interval: 1800
  initial-store-count: 3
  initial-store-list: 192.168.1.20:8500,192.168.1.21:8500,192.168.1.22:8500

store:
  max-down-time: 86400              # 24 hours
  monitor_data_enabled: true
  monitor_data_interval: 1 minute
  monitor_data_retention: 7 days

partition:
  default-shard-count: 3            # Triple replication
  store-max-shard-count: 50

Node 2: 192.168.1.11

grpc:
  host: 192.168.1.11
  port: 8686

server:
  port: 8620

raft:
  address: 192.168.1.11:8610
  peers-list: 192.168.1.10:8610,192.168.1.11:8610,192.168.1.12:8610

pd:
  data-path: /data/pd/metadata
  patrol-interval: 1800
  initial-store-count: 3
  initial-store-list: 192.168.1.20:8500,192.168.1.21:8500,192.168.1.22:8500

store:
  max-down-time: 86400
  monitor_data_enabled: true
  monitor_data_interval: 1 minute
  monitor_data_retention: 7 days

partition:
  default-shard-count: 3
  store-max-shard-count: 50

Node 3: 192.168.1.12

grpc:
  host: 192.168.1.12
  port: 8686

server:
  port: 8620

raft:
  address: 192.168.1.12:8610
  peers-list: 192.168.1.10:8610,192.168.1.11:8610,192.168.1.12:8610

pd:
  data-path: /data/pd/metadata
  patrol-interval: 1800
  initial-store-count: 3
  initial-store-list: 192.168.1.20:8500,192.168.1.21:8500,192.168.1.22:8500

store:
  max-down-time: 86400
  monitor_data_enabled: true
  monitor_data_interval: 1 minute
  monitor_data_retention: 7 days

partition:
  default-shard-count: 3
  store-max-shard-count: 50

Characteristics:

  • 3 PD nodes for high availability
  • Tolerates 1 PD node failure
  • Triple replication (default-shard-count=3)
  • 3 Store nodes specified in initial-store-list
  • Standard monitoring and metrics collection

Network Requirements:

  • Low latency (<5ms) between PD nodes for Raft
  • Open ports: 8620 (REST), 8686 (gRPC), 8610 (Raft)

5-Node Cluster Deployment (High Availability)

Configuration for mission-critical deployments requiring maximum fault tolerance.

# Node 1: 192.168.1.10
grpc:
  host: 192.168.1.10
  port: 8686

raft:
  address: 192.168.1.10:8610
  peers-list: 192.168.1.10:8610,192.168.1.11:8610,192.168.1.12:8610,192.168.1.13:8610,192.168.1.14:8610

pd:
  data-path: /data/pd/metadata
  patrol-interval: 3600             # Lower frequency for large clusters
  initial-store-count: 5
  initial-store-list: 192.168.1.20:8500,192.168.1.21:8500,192.168.1.22:8500,192.168.1.23:8500,192.168.1.24:8500

store:
  max-down-time: 172800             # 48 hours (conservative)
  monitor_data_enabled: true
  monitor_data_interval: 1 minute
  monitor_data_retention: 30 days   # Long-term retention

partition:
  default-shard-count: 3            # Or 5 for extreme HA
  store-max-shard-count: 100

Characteristics:

  • 5 PD nodes for maximum HA
  • Tolerates 2 PD node failures
  • 5 Store nodes for data distribution
  • Lower patrol frequency to reduce overhead
  • Long-term metrics retention (30 days)

Production Tuning

JVM Tuning

JVM options are specified via the startup script (bin/start-hugegraph-pd.sh).

Memory Configuration

# Option 1: Via startup script flag
bin/start-hugegraph-pd.sh -j "-Xmx8g -Xms8g"

# Option 2: Edit start-hugegraph-pd.sh directly
JAVA_OPTIONS="-Xmx8g -Xms8g -XX:+UseG1GC"

Recommendations by Cluster Size:

Cluster Size Partitions Heap Size Notes
Small (1-3 stores, <100 partitions) <100 -Xmx2g -Xms2g Development/testing
Medium (3-10 stores, 100-1000 partitions) 100-1000 -Xmx4g -Xms4g Standard production
Large (10-50 stores, 1000-10000 partitions) 1000-10000 -Xmx8g -Xms8g Large production
X-Large (50+ stores, 10000+ partitions) 10000+ -Xmx16g -Xms16g Enterprise scale

Key Principles:

  • Set -Xms equal to -Xmx to avoid heap resizing
  • Reserve at least 2GB for OS and off-heap memory
  • Monitor GC pause times and adjust accordingly

Garbage Collection

G1GC (Default, Recommended):

bin/start-hugegraph-pd.sh -g g1 -j "-Xmx8g -Xms8g \
  -XX:MaxGCPauseMillis=200 \
  -XX:G1HeapRegionSize=16m \
  -XX:InitiatingHeapOccupancyPercent=45"
  • MaxGCPauseMillis: Target GC pause time (200ms recommended)
  • G1HeapRegionSize: Region size (16m for 8GB heap)
  • InitiatingHeapOccupancyPercent: When to trigger concurrent GC (45% recommended)

ZGC (Low-Latency, Java 11+):

bin/start-hugegraph-pd.sh -g ZGC -j "-Xmx8g -Xms8g \
  -XX:ZCollectionInterval=30"
  • Ultra-low pause times (<10ms)
  • Recommended for latency-sensitive deployments
  • Requires Java 11+ (Java 15+ for production)

GC Logging

-Xlog:gc*:file=logs/gc.log:time,uptime,level,tags:filecount=10,filesize=100M

Raft Tuning

Raft parameters are typically sufficient with defaults, but can be tuned for specific scenarios.

Election Timeout

Increase election timeout for high-latency networks.

Default: 1000ms (1 second)

Tuning (requires code changes in RaftEngine.java):

// In hg-pd-core/.../raft/RaftEngine.java
nodeOptions.setElectionTimeoutMs(3000);  // 3 seconds

When to Increase:

  • Network latency >10ms between PD nodes
  • Frequent false leader elections
  • Cross-datacenter deployments

Snapshot Interval

Control how often Raft snapshots are created.

Default: 3600 seconds (1 hour)

Tuning (in RaftEngine.java):

nodeOptions.setSnapshotIntervalSecs(7200);  // 2 hours

Recommendations:

  • Frequent snapshots (1800s): Faster recovery, more I/O overhead
  • Infrequent snapshots (7200s): Less I/O, slower recovery

Disk I/O Optimization

RocksDB Configuration

PD uses RocksDB for metadata storage. Optimize for your workload.

SSD Optimization (default, recommended):

  • RocksDB uses default settings optimized for SSD
  • No configuration changes needed

HDD Optimization (not recommended): If using HDD (not recommended for production):

// In MetadataRocksDBStore.java, customize RocksDB options
Options options = new Options()
    .setCompactionStyle(CompactionStyle.LEVEL)
    .setWriteBufferSize(64 * 1024 * 1024)  // 64MB
    .setMaxWriteBufferNumber(3)
    .setLevelCompactionDynamicLevelBytes(true);

Key Metrics to Monitor:

  • Disk I/O utilization
  • RocksDB write stalls
  • Compaction backlog

Network Tuning

gRPC Connection Pooling

For high-throughput scenarios, tune gRPC connection pool size.

Client-Side (in PDClient):

PDConfig config = PDConfig.builder()
    .pdServers("192.168.1.10:8686,192.168.1.11:8686,192.168.1.12:8686")
    .maxChannels(5)  // Number of gRPC channels per PD node
    .build();

Recommendations:

  • Low traffic: maxChannels=1
  • Medium traffic: maxChannels=3-5
  • High traffic: maxChannels=10+

TCP Tuning (Linux)

Optimize OS-level TCP settings for low latency.

# Increase TCP buffer sizes
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"
sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"

# Reduce TIME_WAIT connections
sysctl -w net.ipv4.tcp_tw_reuse=1
sysctl -w net.ipv4.tcp_fin_timeout=30

Monitoring and Alerting

Key Metrics to Monitor

Metric Threshold Action
PD Leader Changes >2 per hour Investigate network stability, increase election timeout
Raft Log Lag >1000 entries Check follower disk I/O, network latency
Store Heartbeat Failures >5% Check Store node health, network connectivity
Partition Imbalance >20% deviation Reduce patrol-interval, check rebalancing logic
GC Pause Time >500ms Tune GC settings, increase heap size
Disk Usage (pd.data-path) >80% Clean up old snapshots, expand disk, increase monitor_data_retention

Prometheus Scrape Configuration

scrape_configs:
  - job_name: 'hugegraph-pd'
    static_configs:
      - targets:
          - '192.168.1.10:8620'
          - '192.168.1.11:8620'
          - '192.168.1.12:8620'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 15s

Grafana Dashboards

Key panels to create:

  • PD Cluster Status: Leader, follower count, Raft state
  • Store Health: Online/offline stores, heartbeat success rate
  • Partition Distribution: Partitions per store, leader distribution
  • Performance: QPS, latency (p50, p95, p99)
  • System Resources: CPU, memory, disk I/O, network

Logging Configuration

log4j2.xml

Located at conf/log4j2.xml.

Log Levels

<Loggers>
    <!-- PD application logs -->
    <Logger name="org.apache.hugegraph.pd" level="INFO"/>

    <!-- Raft consensus logs (verbose, set to WARN in production) -->
    <Logger name="com.alipay.sofa.jraft" level="WARN"/>

    <!-- RocksDB logs -->
    <Logger name="org.rocksdb" level="WARN"/>

    <!-- gRPC logs -->
    <Logger name="io.grpc" level="WARN"/>

    <!-- Root logger -->
    <Root level="INFO">
        <AppenderRef ref="RollingFile"/>
        <AppenderRef ref="Console"/>
    </Root>
</Loggers>

Recommendations:

  • Development: Set PD logger to DEBUG for detailed tracing
  • Production: Use INFO (default) or WARN for lower overhead
  • Troubleshooting: Temporarily set specific package to DEBUG

Log Rotation

<RollingFile name="RollingFile" fileName="logs/hugegraph-pd.log"
             filePattern="logs/hugegraph-pd-%d{yyyy-MM-dd}-%i.log.gz">
    <PatternLayout>
        <Pattern>%d{ISO8601} [%t] %-5level %logger{36} - %msg%n</Pattern>
    </PatternLayout>
    <Policies>
        <TimeBasedTriggeringPolicy interval="1" modulate="true"/>
        <SizeBasedTriggeringPolicy size="100 MB"/>
    </Policies>
    <DefaultRolloverStrategy max="30"/>
</RollingFile>

Configuration:

  • Size: Rotate when log file reaches 100MB
  • Time: Rotate daily
  • Retention: Keep last 30 log files

Monitoring and Metrics

Health Check

curl http://localhost:8620/actuator/health

Response (healthy):

{
  "status": "UP"
}

Metrics Endpoint

curl http://localhost:8620/actuator/metrics

Available Metrics:

  • pd.raft.state: Raft state (0=Follower, 1=Candidate, 2=Leader)
  • pd.store.count: Number of stores by state
  • pd.partition.count: Total partitions
  • jvm.memory.used: JVM memory usage
  • jvm.gc.pause: GC pause times

Prometheus Metrics

curl http://localhost:8620/actuator/prometheus

Sample Output:

# HELP pd_raft_state Raft state
# TYPE pd_raft_state gauge
pd_raft_state 2.0

# HELP pd_store_count Store count by state
# TYPE pd_store_count gauge
pd_store_count{state="Up"} 3.0
pd_store_count{state="Offline"} 0.0

# HELP pd_partition_count Total partitions
# TYPE pd_partition_count gauge
pd_partition_count 36.0

Configuration Validation

Pre-Deployment Checklist

  • grpc.host set to actual IP address or container hostname (not 127.0.0.1). For Docker bridge networking use container hostname via HG_PD_GRPC_HOST env var.
  • raft.address unique for each PD node
  • raft.peers-list identical on all PD nodes
  • raft.peers-list contains all PD node addresses
  • pd.data-path has sufficient disk space (>50GB)
  • pd.initial-store-count matches expected store count
  • partition.default-shard-count = 3 (for production HA)
  • Ports accessible from Store/Server nodes (8620, 8686, 8610)
  • NTP synchronized across all nodes

Configuration Validation Tool

# Check Raft configuration
grep -A2 "^raft:" conf/application.yml

# Verify peers list on all nodes
for node in 192.168.1.{10,11,12}; do
    echo "Node $node:"
    ssh $node "grep peers-list /path/to/conf/application.yml"
done

# Check port accessibility
nc -zv 192.168.1.10 8620 8686 8610

Summary

Key configuration guidelines:

  • Single-node: Use defaults with 127.0.0.1 addresses
  • 3-node cluster: Standard production setup with triple replication
  • 5-node cluster: Maximum HA with increased fault tolerance
  • JVM tuning: Allocate 4-8GB heap for typical production deployments
  • Monitoring: Enable Prometheus metrics and create Grafana dashboards

For architecture details, see Architecture Documentation.

For API usage, see API Reference.

For development, see Development Guide.