Vault 를 통한 ssh login 인증 구성 방법

안녕하세요 오늘은 BESPIN GLOBAL SRE실 이동열님이 작성해주신 ‘Vault 를 통한 ssh login 인증 구성 방법’ 대해 소개해드리도록 하겠습니다.

목차

1. Vault 를 통한 ssh login 인증 구성
2. Vault 설치
3. Authn Setup
4. 사용자 Token 사용, 만료, 폐기, 재발급 workflow 및 검증

1. Vault 를 통한 ssh login 인증 구성

1-1. Pre-requisites

상황에 따라 (1) Vault/K8s 관리자(Node 관리자와 같거나 다를 수 있음) (2) Node 접속자로 구분하여 적용합니다.

• Management Client 환경
  • Vault client(Vault/K8s 관리자 & Node 접속자)
  • kubectl(대상이 되는 cluster에 맞춰 context 설정, Vault/K8s 관리자 & Node 접속자)
  • git client(Vault/K8s 관리자 & Node 접속자)
  • ansible >= 2.9.x(Vault/K8s 관리자)
  • Helm >= v3.x(Vault/K8s 관리자)

• Vault 서버 설치 영역(Kubernetes 1.19+)
  • MetalLB(bitnami/metallb), IP 영역(10.2111.55.0/24): 10.211.55.150-10.211.55.200

$ cat << EOF > config-inline.yaml
configInline:
 address-pools:
 - name: default
 protocol: layer2
 addresses:
 - 10.211.55.150-10.211.55.200
EOF
$ kubectl create namespace metallb-system
$ helm install metallb -n metallb-system bitnami/metallb -f configinline.yaml

• Local-path provisioner

kubectl apply -f \
 https://raw.githubusercontent.com/rancher/local-pathprovisioner/v0.0.22/deploy/local-path-storage.yaml

• Ssh 대상 nodes(Linux hosts)
  • 여기서는 k8s node들을 대상으로 하며, 이미 vagrant 계정을 통해 ansible 접속이 가능하도록 다음과 같이 key 교환이 설정된 상태가 준비되어 있는 상황이 예시입니다.
    • nodes: kube-1, kube-2, kube-3
    • account: vagrant
      • 암호 없이 sudo 실행 가능해야 함 -> 필요 시 visudo 에서 NOPASSWD 설정

# This keypair is generated by command below:
ssh-keygen -t rsa -b 2048 -f my_sshkey
# Copy public key file into each servers
for i in `seq 1 3`; do \
 ssh-keygen -R kube-$i;
 ssh-copy-id -i my_sshkey.pub vagrant@kube-$i;
done
# Use default password for vagrant :) for each 3 nodes(Hit 'yes' &
'vagrant')

2. Vault 설치

2-1. Repo clone, 작업 디렉토리 준비, 환경 변수 설정

git clone {This repo}
cd {This repo home directory}/vault-setup

• 00-env.sh 스크립트 작성(편집)

#!/usr/bin/env bash
mkdir ./tmp
export NAMESPACE=vault
export VAULT_HELM_RELEASE_NAME=vault
export SECRET_NAME=vault-server-tls
export K8S_CLUSTER_NAME=cluster.local
export VAULT_INTERNAL=${VAULT_HELM_RELEASE_NAME}-internal
export CSR_NAME=${VAULT_HELM_RELEASE_NAME}-csr
export TMPDIR=./tmp

2-2. Self-signed key 생성 및 signing

Kubernetes 내부의 CSR(Certificate Signing Request)로 signing
최종적으로 vault-server-tls secret에 vault.key, vault.crt, vault.ca 키 값이 저장

• 01-gen-cert.sh 스크립트 작성

#!/usr/bin/env bash
source 00-env.sh
mkdir tmp
kubectl create namespace ${NAMESPACE}
kubectl delete certificatesigningrequests.certificates.k8s.io vault-csr --
ignore-not-found
kubectl delete secret -n ${NAMESPACE} ${SECRET_NAME} -n ${NAMESPACE} --
ignore-not-found
# Generate key for the certificate.
openssl genrsa -out ${TMPDIR}/vault.key 2048
cat << EOF > ${TMPDIR}/csr.conf
[req]
default_bits = 2048
prompt = no
encrypt_key = yes
default_md = sha256
distinguished_name = kubelet_serving
req_extensions = v3_req
[kubelet_serving]
O = system:nodes
CN = system:node:*.${VAULT_HELM_RELEASE_NAME}.svc.${K8S_CLUSTER_NAME}
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment,
dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = *.${VAULT_INTERNAL}
DNS.2 =
*.${VAULT_INTERNAL}.${VAULT_HELM_RELEASE_NAME}.svc.${K8S_CLUSTER_NAME}
DNS.3 = *.${VAULT_HELM_RELEASE_NAME}
IP.1 = 127.0.0.1
EOF
# Create a Certificate Signing Request (CSR) with config.
openssl req -new \
 -key ${TMPDIR}/vault.key \
 -out ${TMPDIR}/server.csr \
 -config ${TMPDIR}/csr.conf
cat << EOF > ${TMPDIR}/csr.yaml
# apiVersion: certificates.k8s.io/v1 # v1beta1 if <= k8s v1.19.x
apiVersion: certificates.k8s.io/v1
01-gen-cert.sh 실행
kind: CertificateSigningRequest
metadata:
 name: ${CSR_NAME}
spec:
 signerName: kubernetes.io/kubelet-serving
 expirationSeconds: 86400 # one day
 request: $(cat ${TMPDIR}/server.csr | base64 | tr -d '\n')
 usages:
 - digital signature
 - key encipherment
 - server auth
EOF
# Send the CSR to Kubernetes.
kubectl create -f ${TMPDIR}/csr.yaml
# Approve the CSR in Kubernetes.
kubectl certificate approve ${CSR_NAME}
# Retrieve the approved certificate.
serverCert=$(kubectl get csr ${CSR_NAME} -o 
jsonpath='{.status.certificate}')
# Save the approved certificate to cert file
echo "${serverCert}" | openssl base64 -d -A -out ${TMPDIR}/vault.crt
# Retrieve Kubernetes CA data, save it to CA file.
kubectl config view \
 --raw \
 --minify \
 --flatten \
 -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64
-d > ${TMPDIR}/vault.ca
# Save the Key, Cert and Kubernetes CA into Kubernetes secret.
kubectl create secret generic ${SECRET_NAME} \
 --namespace ${NAMESPACE} \
 --from-file=vault.key=${TMPDIR}/vault.key \
 --from-file=vault.crt=${TMPDIR}/vault.crt \
 --from-file=vault.ca=${TMPDIR}/vault.ca

• 01-gen-cert.sh 실행

chmod +x 01-gen-cert.sh
./01-gen-cert.sh
----------
mkdir: tmp: File exists
Error from server (AlreadyExists): namespaces "vault" already exists
certificatesigningrequest.certificates.k8s.io "vault-csr" deleted
secret "vault-server-tls" deleted
Generating RSA private key, 2048 bit long modulus
.........................................+++
......................................+++
e is 65537 (0x10001)
certificatesigningrequest.certificates.k8s.io/vault-csr created
certificatesigningrequest.certificates.k8s.io/vault-csr approved
secret/vault-server-tls created
kubectl get csr
----------
NAME AGE SIGNERNAME REQUESTOR 
REQUESTEDDURATION CONDITION
vault-csr 4s kubernetes.io/kubelet-serving kubernetes-admin 24h 
Approved,Issued

2-3. Vault server 설치

• 아래의 3가지 옵션 중 한 가지를 선택(SSL/TLS enabled)
  • Option 1: Vault standalone(single vault pod, PV datastorage)
  • Option 2: Vault standalone(single vault pod, Consul backend)
  • Option 3: Vault HA(3 node statefulset vault, Consul backend)

2-3-1. Vault standalone(single vault pod, PV datastorage)

• Install by Helm

source 00-env.sh
export CA_BUNDLE=$(cat ${TMPDIR}/vault.crt | base64)
helm install $VAULT_HELM_RELEASE_NAME -n $NAMESPACE hashicorp/vault \
 --set injector.certs.secretName=${SECRET_NAME?} \
 --set injector.certs.caBundle=${CA_BUNDLE?} \
 --values tls-values/tls-standalone-pv-values.yaml

• Init & Unseal
  • keyshare=1, threshold=1 로 최소화된 키공유 요건

kubectl exec -n $NAMESPACE $VAULT_HELM_RELEASE_NAME-0 -- vault
operator init \
 -key-shares=1 -key-threshold=1 -format=json > vault-creds.json
# Save this file in secure place
cat vault-creds.json
----------
{
 "unseal_keys_b64": [
 "mFAIYOT7GG9Qgwv0lPOP2TKQ5RJQdCe3ntZD6ueD83E="
 ],
 "unseal_keys_hex": [
 "98500860e4fb186f50830bf494f38fd93290e512507427b79ed643eae783f371"
 ],
 "unseal_shares": 1,
 "unseal_threshold": 1,
 "recovery_keys_b64": [],
 "recovery_keys_hex": [],
 "recovery_keys_shares": 5,
 "recovery_keys_threshold": 3,
 "root_token": "hvs.OZ0bmOD4RPwVMS6bYGGX2GnW"
}
export VAULT_UNSEAL_KEY=$(cat vault-creds.json | jq -r 
".unseal_keys_b64[]")
export VAULT_TOKEN=$(cat vault-creds.json | jq -r ".root_token")
kubectl exec -n $NAMESPACE $VAULT_HELM_RELEASE_NAME-0 -- vault
operator unseal $VAULT_UNSEAL_KEY

• Uninstall

helm uninstall $VAULT_HELM_RELEASE_NAME -n $NAMESPACE
kubectl delete pvc -n $NAMESPACE data-$VAULT_HELM_RELEASE_NAME-0

2-3-2. Vault standalone(single vault pod, Consul backend)

• Install by Helm

source 00-env.sh
# Consul
kubectl delete secret -n $NAMESPACE gossip-encryption-key --ignore-notfound
kubectl create secret -n $NAMESPACE generic gossip-encryption-key \
 --from-literal=key=$(consul keygen)
helm install consul hashicorp/consul -n $NAMESPACE \
 --set global.datacenter=dc1 \
 --set global.name=consul \
 --set global.gossipEncryption.secretName=gossip-encryption-key \
 --set global.gossipEncryption.secretKey=key \
--set server.storageClass=local-path \
 --set server.storage=200Mi \
 --set ui.enabled=true
# Vault
export CA_BUNDLE=$(cat ${TMPDIR}/vault.crt | base64)
helm install $VAULT_HELM_RELEASE_NAME -n $NAMESPACE hashicorp/vault \
 --set injector.certs.secretName=${SECRET_NAME?} \
 --set injector.certs.caBundle=${CA_BUNDLE?} \
 --values tls-values/tls-standalone-consul-values.yaml

• Init & Unseal
  • keyshare=1, threshold=1 로 최소화된 키공유 요건

kubectl exec -n $NAMESPACE $VAULT_HELM_RELEASE_NAME-0 -- vault
operator init \
 -key-shares=1 -key-threshold=1 -format=json > vault-creds.json
 
# Save this file in secure place
cat vault-creds.json
----------
{
 "unseal_keys_b64": [
 "mFAIYOT7GG9Qgwv0lPOP2TKQ5RJQdCe3ntZD6ueD83E="
 ],
 "unseal_keys_hex": [
 "98500860e4fb186f50830bf494f38fd93290e512507427b79ed643eae783f371"
 ],
 "unseal_shares": 1,
 "unseal_threshold": 1,
 "recovery_keys_b64": [],
 "recovery_keys_hex": [],
 "recovery_keys_shares": 5,
 "recovery_keys_threshold": 3,
 "root_token": "hvs.OZ0bmOD4RPwVMS6bYGGX2GnW"
}
export VAULT_UNSEAL_KEY=$(cat vault-creds.json | jq -r 
".unseal_keys_b64[]")
export VAULT_TOKEN=$(cat vault-creds.json | jq -r ".root_token")
kubectl exec -n $NAMESPACE $VAULT_HELM_RELEASE_NAME-0 -- vault
operator unseal $VAULT_UNSEAL_KEY

• Uninstall

helm uninstall $VAULT_HELM_RELEASE_NAME -n $NAMESPACE
helm uninstall -n $NAMESPACE consul
for i in `seq 0 2`; \
 do kubectl delete pvc -n $NAMESPACE data-$VAULT_HELM_RELEASE_NAMEconsul-server-$i; \
done

2-3-3. Vault HA(3 node statefulset vault, Consul backend)

• Install by Helm

source 00-env.sh
# Consul
kubectl delete secret -n $NAMESPACE gossip-encryption-key --ignore-notfound
kubectl create secret -n $NAMESPACE generic gossip-encryption-key \
 --from-literal=key=$(consul keygen)
helm install consul hashicorp/consul -n $NAMESPACE \
 --set global.datacenter=dc1 \
 --set global.name=consul \
 --set global.gossipEncryption.secretName=gossip-encryption-key \
 --set global.gossipEncryption.secretKey=key \
 --set server.storageClass=local-path \
 --set server.storage=200Mi \
 --set ui.enabled=true
# Vault
export CA_BUNDLE=$(cat ${TMPDIR}/vault.crt | base64)
helm install $VAULT_HELM_RELEASE_NAME -n $NAMESPACE hashicorp/vault \
 --set injector.certs.secretName=${SECRET_NAME?} \
 --set injector.certs.caBundle=${CA_BUNDLE?} \
 --values tls-values/tls-ha-consul-vaules.yaml

• Init & Unseal
  • keyshare=1, threshold=1 로 최소화된 키공유 요건

kubectl get pods -n $NAMESPACE --
selector=app.kubernetes.io/name=vault,component=server
----------
NAME READY STATUS RESTARTS AGE
vault-0 0/1 Running 0 2m44s
vault-1 0/1 Running 0 2m44s
vault-2 0/1 Running 0 2m44s
kubectl exec -n $NAMESPACE $VAULT_HELM_RELEASE_NAME-0 -- vault
operator init -key-shares=1 -key-threshold=1 -format=json > vaultcreds.json
# Save this file in secure place
cat vault-creds.json
----------
{
 "unseal_keys_b64": [
 "mFAIYOT7GG9Qgwv0lPOP2TKQ5RJQdCe3ntZD6ueD83E="
 ],
 "unseal_keys_hex": [
 "98500860e4fb186f50830bf494f38fd93290e512507427b79ed643eae783f371"
 ],
 "unseal_shares": 1,
 "unseal_threshold": 1,
 "recovery_keys_b64": [],
 "recovery_keys_hex": [],
 "recovery_keys_shares": 5,
 "recovery_keys_threshold": 3,
 "root_token": "hvs.OZ0bmOD4RPwVMS6bYGGX2GnW"
}
export VAULT_UNSEAL_KEY=$(cat vault-creds.json | jq -r 
".unseal_keys_b64[]")
export VAULT_TOKEN=$(cat vault-creds.json | jq -r ".root_token")
for i in `seq 0 2`; \
 do kubectl exec -n $NAMESPACE $VAULT_HELM_RELEASE_NAME-$i -- vault
operator unseal $VAULT_UNSEAL_KEY; \
done 
kubectl get pods -n $NAMESPACE --
selector=app.kubernetes.io/name=vault,component=server
----------
NAME READY STATUS RESTARTS AGE
vault-0 1/1 Running 0 8m34s
vault-1 1/1 Running 0 8m34s
vault-2 1/1 Running 0 8m34s

• Uninstall

helm uninstall $VAULT_HELM_RELEASE_NAME -n $NAMESPACE
helm uninstall -n $NAMESPACE consul
for i in `seq 0 2`; \
 do kubectl delete pvc -n $NAMESPACE data-$VAULT_HELM_RELEASE_NAME-
consul-server-$i; \
done

2-3-4. Vault 상태 확인 및 접속

• Vault-UI, CLI 접속 확인 및 브라우저/curl API 접속

export UI_IPPORT=$(kubectl get service -n $NAMESPACE
$VAULT_HELM_RELEASE_NAME-ui -o 
jsonpath='{.status.loadBalancer.ingress[0].ip}{":"}{.spec.ports[0].port}')
export IPPORT=$(kubectl get service -n $NAMESPACE $VAULT_HELM_RELEASE_NAME
-o jsonpath='{.status.loadBalancer.ingress[0].ip}{":"}
{.spec.ports[0].port}')
export VAULT_UI_ADDR=https://${UI_IPPORT}
export VAULT_ADDR=https://${IPPORT}
export VAULT_SKIP_VERIFY=1
curl --insecure --silent --header "X-Vault-Token: ${VAULT_TOKEN}" --
request GET ${VAULT_UI_ADDR}/v1/sys/seal-status | jq
vault status

• CLI 명령 기본 확인

vault status
vault secrets enable -path=mysecret kv
vault secrets list
vault write mysecret/hello name=lee
vault kv list mysecret/
vault kv get mysecret/hello
vault kv delete mysecret/hello

3. Authn Setup

Vault 에서 발급된 한시적 Token에 의한 Ssh 접속 설정의 전체 작업 흐름은 아래 그림에 설명되어 있습니다.

cd {This repo home directory}/ssh-authn-setup/admin

3-1. (Vault 관리자) Ssh keypair, ansible 설정

Ansible 에서 사용할 ssh keypair 준비(이미 존재한다면 ./ssh-keys에 복사)
각 ansible host의 ansible 계정(vagrant)에 public key 복사
Ansible 의 호스트 정보(inventory) 업데이트

• Ssh keypair 준비, ansible inventory 구성
  • 사전에 준비된 keypair가 있으면 아래 inventory/hosts.yml 에 등록 사용, 여기서는 my_sshkey 사용
  • Ansible inventory 설정

• ansible.cfg

...
[defaults]
remote_user=vagrant
...

• 00-update-inventory.sh

#!/usr/bin/env bash
PRIVATEKEY_PATH=${PWD}/ssh-keys/my_sshkey
# MacOS Catalina, Big Sur, +
sed -i '' 's| ansible_ssh_private_key_file: .*$| 
ansible_ssh_private_key_file: '"$PRIVATEKEY_PATH"'|g'
inventory/hosts.yml
# Linux
#sed -i 's| ansible_ssh_private_key_file: .*$| 
ansible_ssh_private_key_file: '"$PRIVATEKEY_PATH"'|g'
inventory/hosts.yml

• Ansible ping & update hosts.yaml

./00-update-inventory.sh
ansible -v -b -i inventory/hosts.yml all --become --become-user=root -m 
ping
----------
kube-3 | SUCCESS => {
 "ansible_facts": {
 "discovered_interpreter_python": "/usr/bin/python3"
 },
 "changed": false,
 "ping": "pong"
}
kube-2 | SUCCESS => {
 "ansible_facts": {
 "discovered_interpreter_python": "/usr/bin/python3"
 },
 "changed": false,
 "ping": "pong"
}
kube-1 | SUCCESS => {
 "ansible_facts": {
 "discovered_interpreter_python": "/usr/bin/python3"
 },
 "changed": false,
"ping": "pong"
}

3-2. Vault 구성 및 토큰 발급
3-2-1. (Vault 관리자) Vault client 실행을 위한 환경 변수 설정

• vault-env.sh

#!/usr/bin/env bash
export NAMESPACE=vault
export VAULT_HELM_RELEASE_NAME=vault
export VAULT_ROOT_TOKEN=$(cat ../../vault-setup/vault-creds.json | jq -r 
".root_token")
export VAULT_SERVICE_ADDRESS=https://$(kubectl get service -n $NAMESPACE
$VAULT_HELM_RELEASE_NAME -o jsonpath='{.status.loadBalancer.ingress[0].ip}
{":"}{.spec.ports[0].port}')
export VAULT_TLS_ENABLED=1

3-2-2. (Vault 관리자) Vault 구성 및 토큰 발급

• Signing 용 CA 생성, Token 발급용 role, policy 등록
• Node 에 ssh 접속할 account: bespin
• Token TTL: 각각 720h, 10m

• 01-gen_ca_set_policy.sh 편집

#!/usr/bin/env bash
...
vault write ssh-signer/roles/ssh-user - << EOF
{
 "allow_user_certificates": true,
 "allowed_users": "*",
 "default_extensions": [
 {
 "permit-pty": "",
 "permit-agent-forwarding": ""
 }
 ],
 "key_type": "ca",
 "default_user": "bespin", <== Ssh account
 "ttl": "720h0m0s", <== Token TTL
 "allow_user_key_ids": "false",
 "key_id_format": "{{token_display_name}}"
}
EOF
...
vault write ssh-signer/roles/ssh-user-10m - << EOF
{
 "allow_user_certificates": true,
 "allowed_users": "*",
 "default_extensions": [
 {
 "permit-pty": "",
 "permit-agent-forwarding": ""
 }
 ],
 "key_type": "ca",
 "default_user": "bespin", <== Ssh account
 "ttl": "0h10m0s", <== Token TTL
 "allow_user_key_ids": "false",
 "key_id_format": "{{token_display_name}}"
}
EOF
...
curl -s -k --header "X-Vault-Token: ${VAULT_TOKEN}" --request LIST 
${VAULT_ADDR}/v1/ssh-signer/roles | jq .

• 01-gen_ca_set_policy.sh 실행

./01-gen_ca_set_policy.sh
----------
...
ssh secret engine enabled
Success! Enabled the ssh secrets engine at: ssh-signer/
Generating CA key for signing
...
Registering policy for reading CA public key
Success! Uploaded policy: ssh-pubkey-read
Now registering policy for signing with CA and update capability
The signed cert for client ssh with "ssh-user" will last only 30 days
Success! Data written to: ssh-signer/roles/ssh-user
Success! Uploaded policy: require-ssh-sign
Now registering another policy for signing with CA and update capability
The signed cert for client ssh with "ssh-user-10m" will last only 10
minutes
Success! Data written to: ssh-signer/roles/ssh-user-10m
Success! Uploaded policy: require-ssh-sign
...
Done! - gerarating CA for signing, writing policies related with pubkeyread, ssh-sign-request

• renew_token.sh 실행, 3가지 종류의 토큰 발급
• Output 아래에 3개의 token 저장 json 파일 생성

./renew_token.sh
----------
Renewing token for ssh-pubkey-read, ttl: 3 days
Renewing token for require-ssh-sign, ttl: 3 days
Generating additional token for require-ssh-sign, ttl: 10 minutes
Done! - check the files in output

3-2-3. (Vault 관리자 or Node 관리자) Node에 embed할 Trusted CA pubkey 다운로드

• Vault 관리자일 경우 VAULT_TOKEN = ROOT_TOKEN string 사용
• Node 관리자일 경우 output/token-ssh-pubkey-read.json 의 client_token string 사용

./02-read_ca_and_get_trusted_pubkey.sh hvs.CAESIFqAsW-vault-token-stringXXX
----------
...
Generating trusted CA public key for ssh server
Done!
ls output
----------
token-require-ssh-sign-10m.json token-ssh-pubkey-read.json
token-require-ssh-sign.json trusted-user-ca-keys.pem

3-2-4. (Vault 관리자 or Node 관리자) Node에 Trusted CA pubkeyembed 및 sshd 설정

• 03-push_pubkey_into_node.sh 실행

./03-push_pubkey_into_node.sh
----------
...
PLAY [all]
******************************************************************************************
TASK [Gathering Facts]
**************************************************************************
****
Sunday 05 June 2022 03:56:13 +0900 (0:00:00.093) 0:00:00.093
***********
ok: [kube-2]
ok: [kube-3]
ok: [kube-1]
...
PLAY RECAP
**************************************************************************
****************
kube-1 : ok=2 changed=0 unreachable=0 
failed=0 skipped=0 rescued=0 ignored=0
kube-2 : ok=2 changed=0 unreachable=0 
failed=0 skipped=0 rescued=0 ignored=0
kube-3 : ok=2 changed=0 unreachable=0 
failed=0 skipped=0 rescued=0 ignored=0
Sunday 05 June 2022 03:56:25 +0900 (0:00:00.919) 0:00:01.543
***********
==========================================================================
=====
Ensure sshd is running ---------------------------------------------------
-------------------- 0.92s
Replace lines in sshd_config file ----------------------------------------
-------------------- 0.50s

3-2-5. (Node 사용자)

• Vault client, kubectl cliet 필요(kubectl 사용 불가일 경우 vault-env.sh 에 VAULT_SERVICE_ADDRESS 수동 입력)
  • 관리자(Vault 관리자 or Node 관리자)는 사용자에게 status.sh, token*.json, vault-env.sh, update_local_cert.sh 파일을 전달
  • 관리자에게서 제공 받은 Token과 설정 스크립트를 사용
  • Vault 내에 local pubkey 를 signing 하여 저장

cd {This repo home directory}/ssh-authn-setup/user

• vault-env.sh 편집

#!/usr/bin/env bash
export NAMESPACE=vault
export VAULT_HELM_RELEASE_NAME=vault
export VAULT_USER_TOKEN_30D=$(cat token-require-ssh-sign.json | jq -r 
".auth.client_token")
export VAULT_USER_TOKEN_10M=$(cat token-require-ssh-sign-10m.json | jq -r 
".auth.client_token")
export VAULT_SERVICE_ADDRESS=https://$(kubectl get service -n $NAMESPACE
$VAULT_HELM_RELEASE_NAME -o jsonpath='{.status.loadBalancer.ingress[0].ip}
{":"}{.spec.ports[0].port}')
export VAULT_TLS_ENABLED=1

• update_local_cert.sh 실행

다음과 같이 Vault 내에 signed pub key가 저장되어 있으면, 허용된 accoun로 각 node에 ssh 로그인 가능

./update_local_cert.sh
----------
Usage: ./update_local_cert.sh {10m | 30d}
./update_local_cert.sh 30d
----------
Key Value
--- -----
serial_number e5ea8207b7f70abd
signed_key ssh-rsa-cert-v01@openssh.com
AAAAHHNzaC1yc2EtY2VydC12MDFAb3BlbnNzaC5 ...
...
/Users/bryanlee/.ssh/id_rsa-cert.pub:
 Type: ssh-rsa-cert-v01@openssh.com user certificate
 Public key: RSA-CERT
SHA256:MOl6JwzUK0y0XDllJnCDksfBB0Q1iLbmBfLRcrgy/7g
 Signing CA: RSA SHA256:LnVm03/Ola4RzbUvhjvNA+547rT7G874qkF+QUvEYws
(using rsa-sha2-256)
 Key ID: "token"
 Serial: 1508968394777390882
 Valid: from 2022-06-07T19:30:44 to 2022-07-07T19:31:14
 Principals:
 bespin
 Critical Options: (none)
 Extensions:
 permit-agent-forwarding
 permit-pty

• ssh 로그인 수행

ssh bespin@kube-1
----------
Last login: Sat Jun 4 14:07:01 2022 from 10.211.55.2
bespin@kube-1:~$ exit
ssh bespin@kube-2
----------
Last login: Sat Jun 4 14:07:24 2022 from 10.211.55.2
bespin@kube-2:~$ exit
ssh bespin@kube-3
----------
Last login: Sat Jun 4 14:07:30 2022 from 10.211.55.2
bespin@kube-3:~$ exit

4. 사용자 Token 사용, 만료, 폐기, 재발급 workflow 및 검증

4-1. 사용자 Token 사용, 만료 과정

• 10분 후 자동 만료 되는 token 사용, ssh 로그인 및 정상 만료(로그인 실패) 확인

cd {This repo home directory}/ssh-authn-setup/admin
ls -l output
----------
total 32
-rw-r--r--@ 1 bryanlee staff 622 6 7 14:16 token-require-ssh-sign10m.json
-rw-r--r--@ 1 bryanlee staff 625 6 7 14:16 token-require-ssh-sign.json
...

cd ../user
cd {This repo home directory}/ssh-authn-setup/user
cp ../admin/output/token-require-ssh-sign*.json .

• 10분용 token으로 signing 된 local 인증서 갱신 및 만료 검증

./update_local_cert.sh
----------
Usage: ./update_local_cert.sh {10m | 30d}
./update_local_cert.sh 10m
----------
Key Value
--- -----
serial_number b88d267beaa848a9
signed_key ssh-rsa-cert-v01@openssh.com
AAAAHHNzaC1yc2EtY2VydC12MDFAb3BlbnNzaC5j...cBOktkHqb/DgQ==
/Users/bryanlee/.ssh/id_rsa-cert.pub:
 Type: ssh-rsa-cert-v01@openssh.com user certificate
 Public key: RSA-CERT
SHA256:MOl6JwzUK0y0XDllJnCDksfBB0Q1iLbmBfLRcrgy/7g
 Signing CA: RSA SHA256:LnVm03/Ola4RzbUvhjvNA+547rT7G874qkF+QUvEYws
(using rsa-sha2-256)
 Key ID: "token"
 Serial: 16819506517038396566
 Valid: from 2022-06-07T17:26:47 to 2022-06-07T17:37:17
 Principals:
 bespin
 Critical Options: (none)
 Extensions:
 permit-agent-forwarding
 permit-pty
ssh bespin@kube-3
----------
Last login: Tue Jun 7 03:27:28 2022 from 10.211.55.2
bespin@kube-3:~$ exit
# Retry after 10 minutes
ssh bespin@kube-3
----------
bespin@kube-3: Permission denied (publickey).

4-2. 사용자 Token 폐기 및 로그인 차단 과정

30일 후 자동 만료 되는 token 사용, ssh 정상 로그인 및 차단, token 폐기 절차

• 30일용 token으로 signing 된 local 인증서 갱신 및 사용

./update_local_cert.sh 30d
----------
Key Value
--- -----
serial_number 34503584873b0124
signed_key ssh-rsa-cert-v01@openssh.com
AAAAHHNzaC1yc2EtY2VydC12MDFAb3BlbnNzaC5j...1Jl5iVKXyiCnQ==
/Users/bryanlee/.ssh/id_rsa-cert.pub:
 Type: ssh-rsa-cert-v01@openssh.com user certificate
 Public key: RSA-CERT
SHA256:MOl6JwzUK0y0XDllJnCDksfBB0Q1iLbmBfLRcrgy/7g
 Signing CA: RSA SHA256:LnVm03/Ola4RzbUvhjvNA+547rT7G874qkF+QUvEYws
(using rsa-sha2-256)
 Key ID: "token"
 Serial: 5178507730652081649
 Valid: from 2022-06-07T18:06:12 to 2022-07-07T18:06:42
 Principals:
 bespin
 Critical Options: (none)
 Extensions:
 permit-agent-forwarding
 permit-pty
ssh bespin@kube-2
----------
Last login: Mon Jun 6 23:59:47 2022 from 10.211.55.2
bespin@kube-2:~$ exit

• Ssh 로그인 차단
• Ssh node의 CA pubkey 삭제: ssh host의 /etc/ssh/trusted-user-ca-keys.pem 파일을 삭제하면 모든 사용자가 ssh 로그인을 못하게 됨
• 각 노드의 해당 account 를 통한 ssh 로그인 원천 차단 시에만 사용

cd {This repo home directory}/ssh-authn-setup/admin
./purge_pubkey.sh
----------
Using /Users/bryanlee/Dropbox/macbook-share/tmp/k8s-vault-ssh-loginconfiguration/ssh-authn-setup/admin/ansible.cfg as config file
kube-3 | CHANGED => {
 "ansible_facts": {
 "discovered_interpreter_python": "/usr/bin/python3"
 },
 "changed": true,
 "path": "/etc/ssh/trusted-user-ca-keys.pem",
 "state": "absent"
}
kube-2 | CHANGED => {
 "ansible_facts": {
 "discovered_interpreter_python": "/usr/bin/python3"
 },
 "changed": true,
 "path": "/etc/ssh/trusted-user-ca-keys.pem",
 "state": "absent"
}
kube-1 | CHANGED => {
 "ansible_facts": {
 "discovered_interpreter_python": "/usr/bin/python3"
 },
 "changed": true,
 "path": "/etc/ssh/trusted-user-ca-keys.pem",
 "state": "absent"
}
# Try ssh login(will fail)
ssh bespin@kube-3
----------
bespin@kube-3: Permission denied (publickey).

4-2-1. Vault token 폐기 방법(기존 token으로 신규 signing 방지)

cd {This repo home directory}/ssh-authn-setup/admin

• output 아래의 json 에 포함 된 기존 token 들 폐기(revoke)
• 단, 기존 token을 폐기하더라도 기존 signing 된 터미널에서의 로그인은 30일 기간 내에서는 여전히 가능
• 따라서 해당 token을 사용하여 발급 후 3일 내의 새로운 signing 만 불가하게되므로 로그인 자체의 원천 차단은 불가능

source vault-env.sh
export VAULT_ADDR=${VAULT_SERVICE_ADDRESS}
export VAULT_TOKEN=${VAULT_ROOT_TOKEN}
export VAULT_SKIP_VERIFY=${VAULT_TLS_ENABLED}
# revoke token - 30d
vault token revoke hvs.CAESIHa2RFi8W...Sa3Q
----------
Success! Revoked token (if it existed)
# revoke token - 10m
vault token revoke hvs.CAESIFD4p-Xf4...zU0U
----------
Success! Revoked token (if it existed)
# Check if revoked(will fail)
vault token lookup hvs.CAESIFD4p-Xf4...zU0U
----------
Error looking up token: Error making API request.
URL: POST https://10.211.55.150:443/v1/auth/token/lookup
Code: 403. Errors:
* bad token

4-2-2. Token 체계 res

• 갱신 지급되는 token 을 통해서만 signing & 로그인 가능하도록 토큰 체계 재설정
• 기존 token으로 signing된 터미널에서 로그인 원천 차단
• 3-2-2 ~ 3-2-5 과정 전체를 재수행하여 renewed token을 모든 사용자에게 전달
  • 사용자는 지급 받은 새로운 token으로 local keypair signing 재수행

여기까지 ‘’Vault 를 통한 ssh login 인증 구성 방법’에 대해 소개해드렸습니다. 유익한 정보가 되셨길 바랍니다. 감사합니다. 

Written by 이 동열/ SRE실

BESPIN GLOBAL