HashiCorp Vault is a powerful secrets management solution. IBM Cloud Object Storage (COS) allows you to leverage persistent, scalable S3-compatible storage for Vault data, which is valuable for high availability and disaster recovery scenarios. This guide covers installing Vault, configuring systemd, and enabling S3 storage with IBM COS on RHEL.

Prerequisites

• RHEL server (tested on RHEL 8/9)
• Vault OSS binary (latest stable version)
• IBM COS bucket and credentials
• Sudo/admin access

Step 1: Install HashiCorp Vault

Add the HashiCorp RPM repository and install Vault via YUM:

sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo yum -y install vault
vault --version

Verify installation by running vault --version.¹²

Step 2: Create Vault System User

For security, run Vault under a dedicated user:

sudo useradd --system --home /etc/vault.d --shell /bin/false vault

Assign permissions for configuration file:

sudo chown vault:vault /etc/vault.d/vault.hcl
sudo chmod 640 /etc/vault.d/vault.hcl

Step 3: Prepare IBM COS Bucket

• Ensure your COS bucket exists in IBM Cloud.
• Retrieve your S3 API credentials (access_key, secret_key).
• Note your region and endpoint (e.g., https://s3.us-south.cloud-object-storage.appdomain.cloud).³

It’s recommended to enable Object Versioning for easier recovery.⁴

Step 4: Vault Configuration (vault.hcl)

Place the following config in /etc/vault.d/vault.hcl:

ui = true

storage "s3" {
  bucket = "wxd-local-dev"                       # Your IBM COS bucket
  region = "us-south"                            # COS region
  endpoint = "https://s3.us-south.cloud-object-storage.appdomain.cloud"
  access_key = "<access_key>"
  secret_key = "<secret_key>"
  disable_ssl = false
  s3_force_path_style = true
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1                               # Set to 0 and provide certs for TLS
  tls_cert_file = "/opt/vault/tls/tls.crt"      # Your self-signed or public cert
  tls_key_file  = "/opt/vault/tls/tls.key"
}

• Set s3_force_path_style = true for IBM COS compatibility.⁵
• Use TLS for production, even if you disable temporarily for testing.

Step 5: Setup Vault as a Systemd Service

Create and edit a service file /etc/systemd/system/vault.service:

sudo vi /etc/systemd/system/vault.service

Add the following:

[Unit]
Description=HashiCorp Vault - A tool for managing secrets
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault.hcl

[Service]
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
LimitNOFILE=65536
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target

Reload and enable service:

sudo systemctl daemon-reload
sudo systemctl enable vault
sudo systemctl start vault
sudo systemctl status vault

Step 6: Initialize and Unseal Vault

Set Vault address, then initialize:

export VAULT_ADDR='http://<YOUR_SERVER_IP>:8200'
vault operator init
vault operator unseal

• Save unseal keys and the initial root token securely.
• For test setups with TLS disabled, set export VAULT_SKIP_VERIFY=true.

Step 7: Basic Vault Operations

Login with your initial root token:

vault login
vault kv put secret/hello foo=world
vault kv get secret/hello
vault token lookup
vault secrets list
vault secrets enable -path=secret kv

Notes and Best Practices

• Use dedicated Vault user/group for minimal privileges.
• Secure Vault config and credentials (600 permission for secrets).
• Always prefer enabling TLS in production.
• Enable versioning on COS bucket if possible.⁴
• Use systemd for auto-restart and recoverability.

Troubleshooting

• Ensure network connectivity to IBM COS endpoints.
• Double check endpoint and region values; COS is S3-compatible, but requires explicit path-style addressing and the endpoint.
• Confirm Vault UI is accessible at http://<server_ip>:8200/ui (if ui = true).⁶⁷

This workflow gives you an enterprise-grade secrets backend on RHEL with robust IBM COS S3 storage integration, streamlining secret recovery and failover. Adjust paths, credentials, and TLS configuration for your production use case.

References