[radkomih]

Description

This pull request adds pure-Go key management for consensus nodes. Operators no longer need openssl or fixed sample secrets.

This pull request changes the following:

• Adds pkg/consensus/keys, which generates the gossip key (RSA-3072) and the gRPC TLS key, each with a self-signed certificate. The admin key comes from an external wallet.
• Adds the consensus keys commands: generate, import, export, verify, init and inspect.
• If no secret flags are given, consensus node install uses the secrets of the live capsule or the labelled secrets of the node. Preflight checks the secret entries.
• The UAT harness creates node keys with consensus keys init, not with sample secrets.
• Adds docs/commands/consensus.md.

Related Issues

• Closes feat(consensus/keys): 'consensus keys inspect' to view certs and public keys without openssl #1278
• Refs Solo Provisioner: Consensus Node Key & Secret Management #1164, feat(consensus/keys): pure-Go key-generation core (gossip/gRPC/admin) #1165, feat(consensus/keys): 'consensus node keys generate' command #1166, feat(consensus/keys): typed Secret I/O — 'keys import' + 'keys export' #1167, feat(consensus/keys): 'consensus node keys verify' command #1168, feat(consensus): wire operator AdminPublicKey ref + neutral index-free secret keys #1169, feat(consensus/keys): 'consensus node keys init' one-shot (generate+import) #1195

# In the VM:

cd /mnt/solo-weaver
task uat:rebuild                       # binary must include the account ID fix
export NS=hiero-network-4 PKG=./test/data/build-v0.74.2 K=/opt/solo/cn-keys/hiero-network-4

# If the VM has no cluster, run these commands:

task uat:registry:login:ghcr
sudo solo-provisioner kube cluster install --profile local --node-type consensus

# Then create the pull secrets and install the operator:

task uat:secrets NS=$NS SAMPLE_KEYS=false   # expect: "Skipping sample gossip/gRPC secrets"
sudo solo-provisioner kube operator install

# NOTE: all nodes in this plan run on one VM. The gossip certificate CN defaults to
# /etc/machine-id, so every `keys generate` / `keys init` below passes a distinct
# --machine-id test-node<N>. Without it, all nodes get the same CN and the platform
# crashes at startup with "Duplicate key CN=..." (step 6 never reaches Active).

# 1. Offline key flow (no cluster)

# 1.1 Generate
sudo solo-provisioner consensus keys generate --experimental --keys-dir $K/node0 --machine-id test-node0 \
--grpc-cn node0 --grpc-org "Example Org" --grpc-dns node0.example.org --grpc-ip 10.1.2.3 --grpc-validity-days 365
# Expect: Generated key material in … and a gRPC cert hash (SHA-384) line. Write down the hash.

# 1.2 Permissions
sudo sh -c "stat -c '%a %n' $K/node0/*"
# Expect: hedera.key and s-private.pem are 600.

# 1.3 Idempotent, and --force does not replace keys
sudo sh -c "sha384sum $K/node0/*" > /tmp/k1.txt
sudo solo-provisioner consensus keys generate --experimental --keys-dir $K/node0
sudo solo-provisioner consensus keys generate --experimental --keys-dir $K/node0 --force
sudo sh -c "sha384sum $K/node0/*" | diff /tmp/k1.txt - && echo "unchanged"
# Expect: both runs print nothing to generate, and the last command prints unchanged.

# 1.4 Admin public key (stands in for a wallet key)
openssl rand -hex 32 | sudo tee $K/node0/admin.pub

# 1.5 Inspect
sudo solo-provisioner consensus keys inspect --experimental --keys-dir $K/node0
sudo solo-provisioner consensus keys inspect --experimental --file $K/node0/hedera.crt
# Expect: three sections. The gossip section shows CN=test-node0. The gRPC section shows CN=node0, O=Example Org, the SANs localhost, node0.example.org, 127.0.0.1, ::1, 10.1.2.3, and a Not after date about 365 days from today. Its SHA-384 value is the same as the hash from step 1.1.

# 1.6 A hex file that is not *.pub is not printed
sudo cp $K/node0/admin.pub /tmp/x.hex
sudo solo-provisioner consensus keys inspect --experimental --file /tmp/x.hex; echo "exit=$?"
# Expect: the exit is not 0, and the output does not show the hex.

# 1.7 Verify
sudo solo-provisioner consensus keys verify --experimental --keys-dir $K/node0
# Expect: [OK  ] for gossip, grpc and admin, then All checks passed.

# 2. Cluster key flow (node 0)

# 2.1 Import, then import again
sudo solo-provisioner consensus keys import --experimental --namespace $NS --node-id 0 --keys-dir $K/node0
sudo solo-provisioner consensus keys import --experimental --namespace $NS --node-id 0 --keys-dir $K/node0
# Expect: the first run prints three secrets: node0-gossip-keys-<date>-<fp8>, node0-grpc-tls-keys-<date>-<fp8> and node0-admin-pubkey-<fp8>. The second run prints (already up to date).

# 2.2 Labels, date and fingerprint
kubectl -n $NS get secret -l solo.hiero.org/node=node0 -L solo.hiero.org/key-type,solo.hiero.org/cert-fingerprint
for f in s-public.pem hedera.crt; do
  date -u -d "$(sudo openssl x509 -in $K/node0/$f -noout -startdate | cut -d= -f2)" +%Y%m%d
  sudo sha384sum $K/node0/$f | cut -c1-8
done
sudo cat $K/node0/admin.pub | xxd -r -p | sha384sum | cut -c1-8
# Expect: each date and each 8-character hash is the same as the <date> and <fp8> in the matching secret name.

# 2.3 Verify and export from the cluster
sudo solo-provisioner consensus keys verify --experimental --from-cluster --namespace $NS --node-id 0
sudo solo-provisioner consensus keys export --experimental --namespace $NS --node-id 0 --keys-dir /tmp/exp
sudo sh -c 'stat -c "%a %n" /tmp/exp/*'
for f in admin.pub hedera.crt hedera.key s-private.pem s-public.pem; do sudo cmp $K/node0/$f /tmp/exp/$f && echo "same: $f"; done
sudo solo-provisioner consensus keys export --experimental --namespace $NS --node-id 0 --keys-dir /tmp/exp; echo "exit=$?"
# Expect:
# - The verify prints All checks passed.
# - The export folder has exactly five files, and the two private keys are 600.
# - The loop prints five same: lines.
# - The second export ends with exit=0.

# 3. Install with labelled secrets

# 3.1 Keys for node 1 (no admin.pub, so no admin secret)
sudo solo-provisioner consensus keys init --experimental --namespace $NS --node-id 1 --keys-dir $K/node1 --machine-id test-node1

# 3.2 Node 0: no secret flags, no --account-id
sudo solo-provisioner consensus node install --experimental --non-interactive \
  --namespace $NS --node-id 0 --profile local --deployment-package-dir $PKG

# 3.3 Node 1: interactive, to test the account ID prompt
sudo solo-provisioner consensus node install --experimental \
  --namespace $NS --node-id 1 --profile local --deployment-package-dir $PKG
# Expect: the wizard asks only for "Node Account ID". The field is empty and shows the hint 0.0.(3 + node-id). If you type 1 and press Enter, the field shows an error that contains invalid account ID "1". Clear the field and press Enter, and the install continues.

# 3.4 Bad account ID from the flag
sudo solo-provisioner consensus node install --experimental --non-interactive \
  --namespace $NS --node-id 1 --account-id 1 --profile local --deployment-package-dir $PKG; echo "exit=$?"
# Expect: the exit is not 0, and the error contains invalid account ID "1".

# 3.5 Check the capsules
for i in 0 1; do
  kubectl -n $NS get consensuscapsule $NS-consensus-$i \
    -o jsonpath='{.spec.accountId}{"\n"}{range .spec.secrets.secretResources.*}{.secretName}{"\n"}{end}' | sort -u
done
# Expect:
# - Node 0 has 0.0.3, and its capsule uses its gossip, grpc-tls and admin secrets.
# - Node 1 has 0.0.4, and its capsule uses its gossip and grpc-tls secrets only.

# 4. Rotation and the choice of secrets

# 4.1 Rotate node 0
kubectl -n $NS get secret -l solo.hiero.org/node=node0 -L solo.hiero.org/key-type,solo.hiero.org/cert-fingerprint --no-headers | sort > /tmp/before.txt
sudo solo-provisioner consensus keys init --experimental --namespace $NS --node-id 0 --keys-dir $K/node0 --machine-id test-node0 --overwrite
kubectl -n $NS get secret -l solo.hiero.org/node=node0 -L solo.hiero.org/key-type,solo.hiero.org/cert-fingerprint --no-headers | sort > /tmp/after.txt
diff /tmp/before.txt /tmp/after.txt
kubectl -n $NS get secret -l solo.hiero.org/node=node0 -o jsonpath='{range .items[*]}{.metadata.labels.solo\.hiero\.org/key-type}{"\n"}{end}' | sort | uniq -c
# Expect:
# - The diff shows only lines that start with >: one new gossip secret and one new grpc secret.
# - The counts are 1 admin, 2 gossip and 2 grpc.

# 4.2 Reinstall node 0: keeps the live secrets
sudo solo-provisioner consensus node install --experimental --non-interactive --namespace $NS --node-id 0 --profile local --deployment-package-dir $PKG
kubectl -n $NS get consensuscapsule $NS-consensus-0 -o jsonpath='{range .spec.secrets.secretResources.*}{.secretName}{"\n"}{end}' | sort -u
# Expect: the install completes, and the capsule shows the old names from /tmp/before.txt.

# 4.3 Node 2: two generations stop a first install
sudo solo-provisioner consensus keys init --experimental --namespace $NS --node-id 2 --keys-dir $K/node2 --machine-id test-node2
sudo solo-provisioner consensus keys init --experimental --namespace $NS --node-id 2 --keys-dir $K/node2 --machine-id test-node2 --overwrite
sudo solo-provisioner consensus node install --experimental --non-interactive --namespace $NS --node-id 2 --profile local --deployment-package-dir $PKG; echo "exit=$?"
# Expect: the exit is not 0, and the error is 2 gossip key secrets found for node2 (…); choose one with --fingerprint or --secret-suffix.

G=$(kubectl -n $NS get secret -l solo.hiero.org/node=node2,solo.hiero.org/key-type=gossip -o name | head -1 | cut -d/ -f2)
T=$(kubectl -n $NS get secret -l solo.hiero.org/node=node2,solo.hiero.org/key-type=grpc -o name | head -1 | cut -d/ -f2)
sudo solo-provisioner consensus node install --experimental --non-interactive \
  --namespace $NS --node-id 2 --profile local --deployment-package-dir $PKG \
  --signing-secret $G --grpc-tls-secret $T
kubectl -n $NS get consensuscapsule $NS-consensus-2 \
-o jsonpath='{.spec.accountId}{"\n"}{range .spec.secrets.secretResources.*}{.secretName}{"\n"}{end}'
# Expect: the install completes with 0.0.5, and the capsule uses $G and $T.

# 4.4 Export and verify select a generation (node 0)
FP=$(head -1 /tmp/before.txt >/dev/null; grep gossip /tmp/before.txt | awk '{print $1}' | rev | cut -d- -f1 | rev)
sudo solo-provisioner consensus keys export --experimental --namespace $NS --node-id 0 --keys-dir /tmp/exp-n0; echo "exit=$?"
sudo solo-provisioner consensus keys verify --experimental --from-cluster --namespace $NS --node-id 0; echo "exit=$?"
sudo solo-provisioner consensus keys export --experimental --namespace $NS --node-id 0 --keys-dir /tmp/exp-n0 --gossip --fingerprint $FP; echo "exit=$?"
sudo solo-provisioner consensus keys verify --experimental --from-cluster --namespace $NS --node-id 0 --gossip --fingerprint $FP; echo "exit=$?"
sudo solo-provisioner consensus keys verify --experimental --from-cluster --namespace $NS --node-id 0 --secret-suffix $(date -u +%Y%m%d); echo "exit=$?"
# Expect:
# - The first two commands fail and list the two generations.
# - The two commands with --fingerprint $FP end with exit=0.
# - The --secret-suffix <today> command still lists two candidates, because both generations have the same date.
#   The admin line also fails, because the admin secret name has no date.

# 5. Negative checks

# 5.1 Secret without s-private.pem: preflight rejects it
GT=$(grep grpc /tmp/before.txt | awk '{print $1}')
kubectl -n $NS create secret generic node3-bad-gossip --from-file=s-public.pem=<(sudo cat $K/node0/s-public.pem)
sudo solo-provisioner consensus node install --experimental --non-interactive \
  --namespace $NS --node-id 3 --profile local --deployment-package-dir $PKG \
  --signing-secret node3-bad-gossip --grpc-tls-secret $GT; echo "exit=$?"
kubectl -n $NS get consensuscapsule $NS-consensus-3
kubectl -n $NS delete secret node3-bad-gossip
# Expect: the error is Secret "node3-bad-gossip" (from --signing-secret) … is missing entries: s-private.pem, and the capsule is NotFound.

# 5.2 Node ID with no keys
sudo solo-provisioner consensus node install --experimental --non-interactive \
  --namespace $NS --node-id 4 --profile local --deployment-package-dir $PKG; echo "exit=$?"
# Expect: no gossip key secret found for node4 in namespace "hiero-network-4"; …

# 5.3 Sample secrets (no labels): same error
task uat:secrets NS=$NS NODE=node4 SAMPLE_KEYS=true
sudo solo-provisioner consensus node install --experimental --non-interactive \
  --namespace $NS --node-id 4 --profile local --deployment-package-dir $PKG; echo "exit=$?"
kubectl -n $NS delete secret node4-gossip-keys node4-grpc-tls-keys
# Expect: the task prints the "no solo.hiero.org labels" hint, and the install fails as in 5.2.

# 5.4 Bad generate values
for args in "--grpc-dns node2" "--grpc-ip not-an-ip" "--grpc-validity-days 99999" "extra-arg"; do
  sudo solo-provisioner consensus keys generate --experimental --keys-dir /tmp/keys-bad --tls $args; echo "[$args] exit=$?"
done

# 5.5 Empty folder
mkdir -p /tmp/empty-keys
sudo solo-provisioner consensus keys inspect --experimental --keys-dir /tmp/empty-keys; echo "exit=$?"
# Expect: all five exits are not 0.

# 6. Genesis

# 6.1 Each live gossip certificate has its own CN
for i in 0 1 2; do
  S=$(kubectl -n $NS get consensuscapsule $NS-consensus-$i \
    -o jsonpath='{range .spec.secrets.secretResources.*}{.secretName}{"\n"}{end}' | grep gossip | head -1)
  echo -n "node$i $S: "
  kubectl -n $NS get secret $S -o jsonpath='{.data.s-public\.pem}' | base64 -d | openssl x509 -noout -subject
done
# Expect: the CNs are test-node0, test-node1 and test-node2. If two CNs are the same, stop:
# the nodes will crash with "Duplicate key CN=..." (re-create the keys with a distinct --machine-id).

# 6.2 Generate the genesis and wait for Active
sudo solo-provisioner consensus network genesis --experimental --namespace $NS; echo "exit=$?"
kubectl -n $NS get consensuscapsules
# Expect: the genesis ends with exit=0, and the three capsules (nodes 0, 1 and 2) show PLATFORM ACTIVE.

# 7. Daemon harness (optional)

# To test the harness, use one more namespace and run the cluster, secrets and operator steps for it. The full order is in docs/dev/daemon/README.md. Then run:

task -d docs/dev/daemon keys NS=hiero-network-5 NODE_ID=0
task -d docs/dev/daemon consensus NS=hiero-network-5 NODE_ID=0

# Expect: the keys task prints All checks passed. and lists the three labelled secrets. The consensus task installs the node with no secret flags and runs genesis.

[swirlds-automation]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[radkomih]

Manual UAT tests pass, but I have found some issues that I need to address when testing via the go-sdk

[leninmehedy]

Thanks. Very close! I've added some comments.

[leninmehedy]

solo-provisioner should have a command to export the keys in DER format. Therefore, user can just use a single command to export in a format they want.

[leninmehedy]

I forgot to mention, but we need to set a default --keys-dir value. Let's set to /opt/solo/weaver/consensus/keys/

[leninmehedy]

This is incorrect. NodeUpdate (a type of DAB transaction) is only effective after a freeze upgrade, and we don't have the support for it yet in solo-operator. So as of now we cannot update admin key after genesis. DAB transaction support is an epic being worked in solo-operator.

Therefore, we should enforce the existence of data/config/node-admin-keys.json in the deployment package, and if that is missing, we must confirm with user to proceed with the default genesis key (only for local dev/testing)

[leninmehedy]

Let's remove the sample secrets. For testing people should generate using the command and backup as required.