Skip to main content

TMKMS Setup Guide

TL;DR: Run TMKMS on a separate Linux host. Import the validator key, enable remote signing in zigchaind, confirm signing in the logs, then remove the key from the validator node.

Overview​

TMKMS (Tendermint Key Management System) runs on a dedicated host for ZIGChain. It signs consensus messages for zigchaind over the network. The default port is 26659.

This guide covers TMKMS installation, configuration, and troubleshooting. It explains how to securely manage the private validator key of a ZIGChain validator. Topics include install steps, key import, tmkms.toml, systemd, validator config.toml changes, and troubleshooting.

Complete Set Up a ZIGChain Validator before remote signing. For node install and sync, see Set Up a ZIGChain Node.

Key Concepts​

  • TMKMS: Service that holds the validator private key and signs consensus messages for a remote validator process.
  • Remote signing: Validator runs without a local priv_validator_key.json; TMKMS signs over TCP (default port 26659).
  • softsign: TMKMS feature that stores the key on disk without HSM encryption. It still isolates the key from the validator host.
  • secret_connection_key: Key pair TMKMS uses to authenticate the priv-validator connection to zigchaind.
  • priv_validator_laddr: config.toml setting that tells zigchaind to listen for an external signer.
  • state_file: Path where TMKMS maintains sign state. This helps prevent double-signing.

Prerequisites and Warnings​

warning

Test this procedure on non-production environments before deploying to production. We disclaim any responsibility for damage caused by applying this procedure on production servers.

Installation​

This section covers installing TMKMS on a server. Adapt these steps to your deployment environment, whether that's cloud infrastructure, on-premises servers, or containerized deployments.

System Requirements​

Set up a server with the following characteristics:

  • A Linux distribution (Debian 12 or similar is recommended)
  • Sufficient storage for logs and configuration files
  • Network connectivity to your validator node
  • Appropriate security measures for your environment (firewalls, access controls, etc.)

The TMKMS server should be isolated from the validator node. Configure it according to your security requirements.

Install Rust, GCC, and Dependencies​

Install the Rust toolchain (compiler + Cargo):

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env

Verify that gcc is installed:

gcc --version

Expected output example:

gcc (Debian 12.2.0-14) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.

Install required packages using your system's package manager. For Debian/Ubuntu, this installs gcc via build-essential if it is not already present:

sudo apt update
sudo apt install git build-essential curl jq --yes

Install libusb (required for hardware key support, if needed):

sudo apt install libusb-1.0-0-dev

Set the RUSTFLAGS environment variable for x86_64 optimizations. Follow the Tendermint KMS installation instructions:

export RUSTFLAGS=-Ctarget-feature=+aes,+ssse3

Install TMKMS​

Use Cargo to build and install TMKMS with the softsign feature. Although softsign stores the validator key unencrypted, it isolates the key from the validator node and may improve security in some deployment models.

cd $HOME
git clone https://github.com/iqlusioninc/tmkms.git
cd tmkms/
cargo install tmkms --features=softsign
tmkms init config
tmkms softsign keygen ./config/secrets/secret_connection_key

Configure Logging, Log Rotation, and systemd Service​

Create a dedicated directory for TMKMS logs. Replace USERNAME with the user that will run TMKMS:

USER=USERNAME
sudo mkdir -p /var/log/tmkms
sudo touch /var/log/tmkms/main_tmkms.log
sudo touch /var/log/tmkms/err_tmkms.log
sudo chown -R $USER:$USER /var/log/tmkms

Create a systemd service for TMKMS. Adjust paths and usernames to match your environment:

sudo vi /etc/systemd/system/tmkms.service

Example systemd service configuration. Replace USERNAME and paths as needed:

[Unit]
Description=TMKMS Custom Service
After=network-online.target

[Service]
ExecStart=/home/USERNAME/.cargo/bin/tmkms start -c /home/USERNAME/tmkms/config/tmkms.toml
StandardOutput=append:/var/log/tmkms/main_tmkms.log
StandardError=append:/var/log/tmkms/err_tmkms.log
Restart=always
User=USERNAME

[Install]
WantedBy=multi-user.target

Install and configure log rotation. On Debian/Ubuntu:

sudo apt update
sudo apt install logrotate

Create a log rotation configuration:

sudo vi /etc/logrotate.d/tmkms

Example log rotation configuration:

/var/log/tmkms/*_tmkms.log {
  daily
  rotate 7
  compress
  missingok
  notifempty
  copytruncate
}

Set Up TMKMS for a Validator​

Import the Validator Key into TMKMS​

Copy the validator key from the validator node to the TMKMS server. Ensure secure access is configured between the two systems. Confirm firewall rules allow the necessary traffic. Adjust the following command to match your environment:

scp USER@VALIDATOR_IP:~/.zigchain/config/priv_validator_key.json ~/tmkms/config/secrets/

Example:

scp [email protected]:~/.zigchain/config/priv_validator_key.json ~/tmkms/config/secrets/

Then, import the key into TMKMS and delete the original key file:

tmkms softsign import ~/tmkms/config/secrets/priv_validator_key.json ~/tmkms/config/secrets/priv_validator_key
rm ~/tmkms/config/secrets/priv_validator_key.json

The resulting file $HOME/tmkms/config/secrets/priv_validator_key is what TMKMS uses for signing.

Configure and Start TMKMS​

Before configuring TMKMS, gather the following information:

  • User account running zigchaind on the validator
  • Validator IP address or hostname
  • Validator port for priv_validator (default: 26659)
  • chain_id: Run grep chain_id ~/.zigchain/config/genesis.json on the validator
  • account_key_prefix: Defined in the chain's code (for example, zigpub)
  • consensus_key_prefix: Defined in the chain's code (for example, zigvalconspub)

Edit the TMKMS config file:

vi ~/tmkms/config/tmkms.toml

Replace the default values with your setup. Example configuration below. Adjust paths, chain_id, and addresses to match your environment:

# Tendermint KMS configuration file

## Chain Configuration

[[chain]]
id = "your-chain-id"
key_format = { type = "bech32", account_key_prefix = "zigpub", consensus_key_prefix = "zigvalconspub" }
state_file = "/home/USERNAME/tmkms/config/state/priv_validator_state.json"

## Signing Provider Configuration

[[providers.softsign]]
chain_ids = ["your-chain-id"]
key_type = "consensus"
path = "/home/USERNAME/tmkms/config/secrets/priv_validator_key"

## Validator Configuration

[[validator]]
chain_id = "your-chain-id"
addr = "tcp://VALIDATOR_IP:26659"
secret_key = "/home/USERNAME/tmkms/config/secrets/secret_connection_key"
protocol_version = "v0.34"
reconnect = true

Start the TMKMS service:

sudo systemctl daemon-reload
sudo systemctl start tmkms.service
systemctl status tmkms.service
tail -f /var/log/tmkms/main_tmkms.log

Leave the tail -f running to monitor the validator connection.

Reconfigure and Restart the Validator​

On the validator node, edit the config.toml file to:

  • Enable the priv_validator listener on port 26659.
  • Comment out the priv_validator_key_file line.
  • Comment out the priv_validator_state_file line.
vi ~/.zigchain/config/config.toml

The corresponding section should look like this:

# Path to the JSON file containing the private key to use as a validator in the consensus protocol
#priv_validator_key_file = "config/priv_validator_key.json"

# Path to the JSON file containing the last sign state of a validator
#priv_validator_state_file = "data/priv_validator_state.json"

# TCP or UNIX socket address for CometBFT to listen on for
# connections from an external PrivValidator process
priv_validator_laddr = "tcp://0.0.0.0:26659"
warning

Restarting zigchaind with these changes will modify validator behavior. Test this procedure in a non-production environment first.

Restart zigchaind and check the logs:

sudo systemctl restart zigchaind.service
tail -f /var/log/zigchaind/main.log

Verify Operation​

If everything is working correctly, the TMKMS logs show successful connection and signing operations:

INFO tmkms::connection::tcp: KMS node ID: xxxxxxxxxxxxxxxxxxxxxxxxxxx
INFO tmkms::session: [chain-id@tcp://validator-ip:26659] connected to validator successfully
INFO tmkms::session: [chain-id@tcp://validator-ip:26659] signed Prevote:... at h/r/s .../0/1 (0 ms)
INFO tmkms::session: [chain-id@tcp://validator-ip:26659] signed Precommit:... at h/r/s .../0/2 (0 ms)
INFO tmkms::session: [chain-id@tcp://validator-ip:26659] signed Proposal:... at h/r/s .../0/0 (0 ms)

Note: Initial connection refused errors are normal. TMKMS may start before the validator is ready.

Once the operation is confirmed, you can remove the validator key from the validator server for security. Do this only if the key is kept in an alternate safe place apart from the TMKMS node. To wipe the file securely, use shred -u:

shred -u ~/.zigchain/config/priv_validator_key.json

Enable the Service and Test a Reboot​

Enable the service to run at boot:

sudo systemctl enable tmkms.service
systemctl status tmkms.service

Reboot the server to verify the service starts automatically:

sudo shutdown -r now

After boot, check the service status and the logs:

systemctl status tmkms.service
tail -f /var/log/tmkms/main_tmkms.log

Configuration Best Practices​

Security Considerations​

  1. Key Isolation: The primary security benefit of TMKMS is isolating the validator key from the validator node. Even with softsign (unencrypted storage), this separation protects against validator node compromises.

  2. Network Security: Ensure that:

    • Secure access is properly configured between validator and TMKMS servers
    • Firewall rules and network security policies allow communication on port 26659
    • TMKMS servers are not unnecessarily exposed to public networks
    • Network traffic between validator and TMKMS is restricted to necessary ports only

  3. Key Management:

    • Always keep a secure backup of the validator key. Store it separate from both the validator and TMKMS nodes
    • Use secure deletion (shred -u) when removing keys from the validator node
    • Delete the key from the validator node only after confirming TMKMS is operating correctly

  4. Infrastructure Security: Apply appropriate security measures for your deployment environment. Examples include disk encryption, access controls, and network segmentation.

Service Configuration​

  1. Logging: The configured log rotation ensures logs are:

    • Rotated daily
    • Kept for 7 days
    • Compressed to save space
    • Truncated rather than moved (using copytruncate) to avoid service interruption

  2. Service Reliability: The systemd service is configured with:

    • Restart=always to automatically restart on failure
    • Proper dependency on network-online.target
    • Separate log files for standard output and errors

  3. Resource Optimization: The RUSTFLAGS environment variable enables hardware-accelerated cryptographic operations. It uses +aes and +ssse3 for better performance.

Configuration File Best Practices​

  1. Path Consistency: Ensure all file paths in tmkms.toml use absolute paths. They must match the actual user's home directory.

  2. Chain Configuration:

    • Verify chain_id matches exactly (case-sensitive)
    • Confirm account_key_prefix and consensus_key_prefix match the chain's configuration
    • Use the correct protocol_version (typically v0.34 for CometBFT)

  3. Validator Connection:

    • Use the correct validator IP address and port (default 26659)
    • Enable reconnect = true for automatic reconnection on network issues
    • Verify the secret_connection_key path is correct

  4. State File Management: Point state_file to a location where TMKMS can maintain validator state. This is critical for preventing double-signing.

Troubleshooting​

Connection Issues​

If the communication fails, check that:

  1. Firewall Rules: Ensure firewall rules and network security policies are configured correctly. The validator must allow inbound connections on port 26659 from the TMKMS server.

  2. Network Connectivity: Verify network connectivity between the TMKMS server and the validator. Replace VALIDATOR_IP with your validator's address:

telnet VALIDATOR_IP 26659

or use nc:

nc -zv VALIDATOR_IP 26659
  1. Service Status: Check that both services are running:
systemctl status tmkms.service
systemctl status zigchaind.service
  1. Configuration Verification: Verify the validator's config.toml has:
    • priv_validator_laddr = "tcp://0.0.0.0:26659" uncommented
    • priv_validator_key_file commented out
    • priv_validator_state_file commented out

Log Analysis​

Monitor the TMKMS logs for common issues:

  1. Connection Refused: Initial connection refused errors are normal. TMKMS may start before the validator is ready. Persistent errors indicate network or configuration issues.

  2. Unverified Validator Peer ID Warning: The warning unverified validator peer ID! is informational. It does not prevent operation. For production, consider implementing peer ID verification.

  3. Signing Errors: If signing operations fail, check the following:

    • The validator key file exists and has correct permissions
    • The key format matches the chain configuration
    • The chain_id in the config matches the actual chain

Handling Chain Migrations​

If zigchaind goes through an init on the validator, follow these steps. One example is when testnet moved from zig-test-1 to zig-test-2:

  1. Stop the TMKMS service:
sudo systemctl stop tmkms.service
  1. Save the config file:
mv ~/tmkms/config/tmkms.toml /tmp
  1. Remove the config directory:
cd ~/tmkms
rm -rf config/
  1. Reinitialize the config:
cd ~/tmkms
tmkms init config
tmkms softsign keygen ./config/secrets/secret_connection_key
  1. Import the validator key again (adjust the command to match your environment):
scp USER@VALIDATOR_IP:~/.zigchain/config/priv_validator_key.json ~/tmkms/config/secrets/
tmkms softsign import ~/tmkms/config/secrets/priv_validator_key.json ~/tmkms/config/secrets/priv_validator_key
rm ~/tmkms/config/secrets/priv_validator_key.json
  1. Restore and update the config file:
mv /tmp/tmkms.toml ~/tmkms/config/
vi ~/tmkms/config/tmkms.toml

Update any parameters that may have changed (chain_id, prefixes, etc.).

  1. Restart both services:
sudo systemctl start tmkms.service
# On the validator node:
sudo systemctl restart zigchaind.service

Service Not Starting After Reboot​

If the TMKMS service does not start after reboot:

  1. Check if the service is enabled:
systemctl is-enabled tmkms.service
  1. If not enabled, enable it:
sudo systemctl enable tmkms.service
  1. Check service logs for errors:
journalctl -u tmkms.service -n 50
  1. Verify file permissions on the key files and the config directory:
ls -la ~/tmkms/config/secrets/

Performance Issues​

If signing operations are slow:

  1. Verify RUSTFLAGS is set correctly in the environment where TMKMS runs
  2. Check system resources (CPU, memory)
  3. Review network latency between TMKMS and validator servers
  4. Ensure the server has adequate resources (CPU, memory) for the workload

Common Questions​

What should I do when zigchaind is re-inited on the validator?​

When zigchaind goes through an init on the validator—for example, when testnet migrated from zig-test-1 to zig-test-2—stop TMKMS and save your tmkms.toml. Re-initialize TMKMS, import the validator key again, and update chain_id and prefix settings before you restart TMKMS and zigchaind. See Handling Chain Migrations for the full command sequence.