Author Archives: mcontino

Device Trust with step-ca, Google Cloud CAS, and SCEP: a Practical, Cloud-Ready Device Trust Build Out

Posted on by

I’m often asked how to stand up a device trust layer that scales from homelab to enterprise without the additional complexity of Hardware security modules. In this post, I’ll show you how to wire up step-ca (Smallstep’s open-source CA), Google Cloud Certificate Authority Service (CAS) as a managed signing backend, and SCEP for mass device enrollment. The result is a modern, automatable PKI that issues device certificates for clientAuth, mTLS, Wi-Fi, VPN, Access Gateways, and beyond.

Why this stack?

  • step-ca: A lightweight CA with batteries included—ACME, OIDC, SCEP, SSH CA, templates, audit logs, and more. It’s ideal as your “front-door” RA/CA service and policy brain.
  • Google Cloud CAS: A managed, audited CA that signs certificates on your behalf. Offload availability and compliance to Google while keeping your issuance logic under your control.
  • SCEP: A well-supported protocol for bulk device enrollment—especially useful for legacy or embedded systems, printers, network gear, and certain MDM agents.

This combo gives you:

  • Cloud scalability + low ops: Managed CA with step-ca’s simple deployment.
  • Security: Keep the signing CAs inside Google CAS; expose only step-ca to your fleet.
  • Automation: Scriptable bootstraps, ephemeral certs, and policy controls.

Architecture at a glance

Flow summary

  1. Devices talk SCEP to step-ca.
  2. step-ca acts as a Registration Authority, relaying/signing requests via CAS.
  3. SCEP returns a signed device certificate.
  4. Devices use certs for clientAuth challenges from the IdP/SSO provider or an access gateway before reaching protected apps

Prerequisites

  • A Google Cloud project with CAS enabled and an active CA in an appropriate CA Pool.
  • A Linux host/VM/container to run step-ca (front-door RA/CA and SCEP endpoint).
  • DNS for your step-ca endpoint (public or private, as needed).
  • Firewall rules allowing inbound 443 to step-ca.
  • gcloud CLI and step CLI.

Installation docs for step-ca: https://smallstep.com/docs/step-ca/installation/

Step 1 — Create a service account for step-ca → CAS access

Create a service account that step-ca will use to interact with CAS:

gcloud iam service-accounts create step-cas-sa \
    --description "Step-CA Service Account" \
    --display-name "Step-CA Service Account"

Grant this SA appropriate CAS roles (for certificate issuance). In many deployments that’s at least roles/privateca.certificateRequester on the CA Pool. (Use the principle of least privilege.)

Tip: You can utilize a certificate template within GCP CAS to further restrict the usage and options available to the service account when an certificate is requested.

Step 2 — Install step-ca and initialize with Cloud CAS as the RA

Create a working directory and initialize step-ca’s config to use Cloud CAS as the Registration Authority:

mkdir /etc/step-ca
export STEPPATH=/etc/step-ca
step ca init --name="CasPoC" --deployment-type standalone --remote-management --provisioner="admin@example.com" --ra=CloudCAS --issuer=projects/<project>/locations/<us-central1>/caPools/<Ca Pool>/certificateAuthorities/<ca ID> --dns="<FQDN>" --address=":443"

What this does:

  • Creates /etc/step-ca/config/ca.json and related directories.
  • Sets CloudCAS as the RA, pointing at your CA (–issuer=projects/…/certificateAuthorities/<ca ID>).
  • Binds HTTPS on :443 (we’ll allow a non-root user to bind shortly).
  • Registers a default OIDC or local provisioner (here, admin@example.com) for management.

Tip: Authenticate gcloud as the service account, or run step-ca compute with that service account identity, so Cloud CAS requests succeed.

Step 3 — Create a restricted system user and grant low-port bind

Run step-ca as a non-root user:

useradd step
passwd -l step
chown -R step:step /etc/step-ca

Allow binding to port 443 without root:

setcap CAP_NET_BIND_SERVICE=+eip /usr/bin/step-ca

This capability approach is safer than running as root.

Step 4 — (Optional) Prepare an intermediate CA template

 (CSR → CAS-signed)

We’ll generate an intermediate key and CSR locally, then have CAS sign it to establish our operational intermediate for issuance:

cat <<EOF >  /etc/step-ca/templates/rsa_intermediate_ca.tpl
{
  "subject": {{ toJson .Subject }},
  "issuer": {{ toJson .Subject }},
  "keyUsage": ["certSign", "crlSign"],
  "basicConstraints": {
    "isCA": true,
    "maxPathLen": 0
  }
  {{- if typeIs "*rsa.PublicKey" .Insecure.CR.PublicKey }}
    , "signatureAlgorithm": "SHA256-RSAPSS"
  {{- end }}
}
EOF

Generate the CSR and key:

step certificate create "SCEP Intermediate CA" \
    /etc/step-ca/certs/intermediate_ca.csr \
    /etc/step-ca/secrets/intermediate_ca_key \
    --template /etc/step-ca/templates/rsa_intermediate_ca.tpl \
    --kty RSA \
    --size 3072 --csr

Ask Cloud CAS to sign the CSR:

gcloud privateca certificates create CERT_ID \
    --issuer-pool <Pool> \
    --issuer-location <Location> \
    --csr /etc/step-ca/certs/intermediate_ca.csr \
    --cert-output-file /etc/step-ca/certs/intermediate_ca.crt \
    --validity "P1Y"

Update your ca.json so step-ca uses the newly minted intermediate:

        "root": "/etc/step-ca/certs/root_ca.crt",
        "federatedRoots": null,
        "crt": "/etc/step-ca/certs/intermediate_ca.crt",
        "key": "/etc/step-ca/secrets/intermediate_ca_key",

Notes
• Store /etc/step-ca/secrets on encrypted disk or a secrets-managed volume.
• Rotate the intermediate on a schedule (e.g., annually) and keep a CRL/OCSP strategy in place.

Step 5 — Dry run the CA

Before opening the floodgates, do a dry run:

sudo -u step step-ca /etc/step-ca/config/ca.json

If it boots cleanly, you should see the HTTP listener and provisioners registered in logs.

Step 6 — Enable SCEP  for device enrollment

Add a SCEP provisioner to your existing CA service. We’ll set challenge credentials and certificate lifetimes:

step ca provisioner add poc_devicetrust \
  --type SCEP --challenge "<redacted>" \
   --x509-min-dur=24h \
   --x509-max-dur=8760h \
   --x509-default-dur=1080h \
  --encryption-algorithm-identifier 2 --admin-name step

A few practical notes:

  • Challenge: Keep it secret (vault, KMS, or MDM payloads). Consider migrating to SCEP with client-side RA or EST for stronger auth if your device ecosystem supports it.
  • Durations: Default shown is 45 days (1080h). Short-lived certs reduce revocation surface.
  • Algorithm Identifier 2: This sets the SCEP Encryption Algorithm Identifier (commonly RSA/3DES/AES variations). Keep this consistent with your device agents.

Step 7 — (Workaround) Clean up ca.json authorities section if needed

Some versions/paths require removing the cloudcas entry from the authorities section. If you see startup errors or odd RA behavior, adjust and restart:

sudo vi /etc/step-ca/config/ca.json

Then relaunch:

sudo -u step step-ca /etc/step-ca/config/ca.json

Note: SCEP can operate with just the local Intermediate CA being used to sign device cert requests. We don’t need direct access to the Root CA or subordinate CAs, so they can be isolated within CAS.

Step 8 — Bootstrap devices and request a cert via SCEP

Once your root CA is trusted on the device (via MDM, config management, or manual import), request a certificate with a SCEP client. Example:

scepclient -private-key client.key -server-url=https://<Domain/IP>/scep/poc_devicetrust -challenge=<Redacted> -dnsname "Lab-PC.local" -cn "Lab-PC" -country "US" -organization "Lab" -ou "Device Trust"

This will:

  • Pull the CA chain from GetCACert
  • Submit a PKCSReq containing your CSR and challenge
  • Receive a signed certificate in CertRep

SCEP Enrollment Sequence

SCEP Enrollment Sequence

Hardening, Operations, and Best Practices

1) TLS and network posture

  • Put step-ca behind a reverse proxy or L7 load balancer for WAF/DoS controls.
  • Consider utilizing a URL Map and/or HTTP targets to restrict acces only to your provisioners
  • Use mTLS for internal admin APIs and restrict management endpoints by IP/VPN.

2) Identity for step-ca

  • Run as the dedicated step user (as above) and consider systemd hardening (ProtectSystem, NoNewPrivileges, PrivateTmp, AmbientCapabilities=CAP_NET_BIND_SERVICE).
  • Keep /etc/step-ca on a read-only or append-only partition where possible; separate secrets onto encrypted volumes.

3) Secrets management

  • Store SCEP challenge in a secret manager and template it into device configs at enrollment time.
  • If using multiple SCEP realms (e.g., printers vs. laptops), separate provisioners with distinct challenges and policies.

4) Short-lived certs + automation

  • Favor short lifetimes (7–45 days) and auto-renew via SCEP or ACME where supported.
  • For services (ingress gateways, sidecars), consider ACME provisioners in step-ca instead of SCEP.

5) Revocation and status

  • Enable OCSP and/or regularly published CRLs. Some gear only understands CRLs; others can do OCSP.
  • Document how to revoke by CN/serial and how MDM/CM tooling redistributes CRL/OCSP endpoints.

6) Names and OIDs

  • Standardize Subject and SANs. For devices, prefer DNS SANs and URNs (e.g., urn:device:asset:1234) over stuffing identifiers in CN.
  • Use policy OIDs or Extended Key Usages (EKUs) that match your relying parties (ClientAuth, ServerAuth, Wi-Fi EAP-TLS, IPsec, etc.).

7) Auditing

  • Step-ca logs each issuance; forward logs to a SIEM with context (device inventory ID, enrollment workflow ID).
  • Cloud CAS has control plane logs—monitor for volume spikes or unusual issuers.

Validating the build (quick checks)

  • Health: curl -ik https://<FQDN>/health (if you expose a health endpoint or use LB health checks).
  • SCEP reachability: curl -I https://<FQDN>/scep/poc_devicetrust should not 404.
  • CAS connectivity: Attempt a test cert request; if it fails, check service account auth and CAS IAM.
  • Chain trust: On a device, ensure the root (and intermediate, if needed) are in the appropriate trust store. Many SCEP clients install the chain automatically after GetCACert.

Troubleshooting tips

  • Challenge mismatch: 401/failed enrollment—verify SCEP challenge and transport (hidden in MDM payloads).
  • CAS quota or IAM: CAS may rate-limit or block by IAM; review Google Cloud audit logs.
  • Template oddities: If you need custom subject or SAN logic per device group, use step templates and multiple SCEP provisioners.

Where to go from here

  • Add ACME for servers and gateways while keeping SCEP for legacy endpoints.
  • Introduce device attestation checks before issuance (e.g., callbacks/webhooks from step-ca to your SCEP/ACME services).
  • Build up resilience by added device posture checks with existing VPN, EDR/XDR. and other security agents
  • Automate intermediate rotation with change windows and controlled CRL/OCSP updates.

Certificate-Based Device Trust: The Cornerstone of Modern Zero Trust Architecture

Posted on by

In today’s digital-first landscape, traditional network security models are rapidly becoming obsolete. The once-reliable perimeter-based security, built on a clear boundary between trusted internal networks and untrusted external ones, is insufficient in the face of today’s distributed workforces, hybrid cloud environments, and rapidly evolving cyber threats. Enter zero trust—a paradigm shift that demands continuous verification, rigorous authentication, and tightly controlled access. While user authentication has rightly received significant attention, an equally crucial aspect— device trust—is often overlooked. Here’s why focusing on device trust is essential to strengthening your organization’s security posture.

The Shift from Network Perimeters to Zero Trust

Zero trust isn’t just a buzzword; it’s a fundamental redesign of security frameworks. It assumes that threats exist both inside and outside traditional boundaries. As a result, every access request—be it from users or devices—is considered potentially risky and must be explicitly verified before granting access. Central to this approach is the principle of “never trust, always verify.”

But while user authentication using multi-factor authentication (MFA) has become standard, verifying devices has lagged behind. Organizations need a more robust, cryptographically strong method to ensure devices interacting with sensitive data are trustworthy. This is where certificate-based device trust becomes indispensable.

Why Certificate-Based Device Trust?

Certificate-based device trust provides a reliable and cryptographically secure way to authenticate and authorize devices. Unlike simple identity checks, device certificates involve digital identities issued by trusted Certificate Authorities (CAs), which verify the device’s authenticity and ensure integrity. Devices without proper certification are automatically denied access, significantly reducing risk.

When properly implemented, certificate-based device trust offers:

  • Better Compliance: Provides clear audit trails required by compliance regulations like GDPR, HIPAA, and ISO 27001.
  • Enhanced Security: Cryptographic certificates are nearly impossible to forge.
  • Reduced Risk of Insider Threats: Only trusted, verified devices are permitted access to sensitive resources.

Securing Remote Workers with Device Certificates

Consider a scenario where a global company employs thousands of remote workers. Each employee must access sensitive data stored in cloud applications like Salesforce, Slack, or internal portals hosted in AWS. While MFA ensures user identity verification, it doesn’t validate whether the device itself is secure or managed.

In this scenario, certificate-based device trust ensures that only devices enrolled and authorized by corporate IT can access company resources. When an employee tries to connect, the device presents its digital certificate, signed by the company’s trusted CA. If the certificate is valid, unexpired, and cryptographically verified, the device is permitted limited, role-specific access to necessary applications.

This granular approach ensures that even if credentials are compromised, attackers can’t gain unauthorized access without possessing the correct device certificates.

Implementing Certificate-Based Device Trust Effectively

For effective deployment, follow these steps:

  1. Set up an Internal Public Key Infrastructure (PKI):Create or deploy a private CA responsible for issuing and managing digital certificates.
  2. Automate Certificate Enrollment and Renewal: Leverage automation tools such as Terraform, step-ca, SCEP, or ACME to streamline certificate issuance, distribution, and renewal processes.
  3. Employ Device Management Solutions: Integrate posture checking on Mobile Device Management (MDM), Endpoint Detection and Response (EDR), or Unified Endpoint Management (UEM) tools to enforce compliance and validate certificates continuously.
  4. Regular Auditing and Monitoring: Implement continuous monitoring solutions like Wazuh and Splunk to log certificate usage, detect suspicious activity, and automate responses to anomalies.

Challenges and Solutions in Adopting Device Trust

Despite clear advantages, some organizations hesitate due to perceived complexity. Common objections include:

  • Complexity of PKI Management: Adopting cloud-managed PKI services (AWS Certificate Manager, Azure Key Vault) simplifies management significantly.
  • Operational Overhead: Automation mitigates operational burdens by handling certificate management tasks efficiently.
  • Initial Implementation Costs: While upfront investments may be required, long-term security enhancements and risk reduction offer substantial returns.

Trust the Device, Not Just the User

In a world where cybersecurity threats evolve daily, organizations can’t afford to rely solely on user-centric authentication. Certificate-based device trust is no longer optional—it’s an essential component of modern security frameworks. By adopting cryptographically verified device certificates, organizations can confidently establish a zero trust model, ensuring secure interactions, safeguarding data integrity, and meeting compliance demands.

Prioritizing certificate-based device trust doesn’t just boost security; it ensures your organization’s resilience in an increasingly digital future.

Kubernetes Harvester to Gather Credentials with Limited Access

Posted on by

Project URL: https://github.com/sleventyeleven/Kubernetes-Harvester

Kubernetes Harvester Example Run

What is Kubernetes Harvester?

Harvester is a new python based project that attempts to leverage access in order to gather potentially sensitive information. Its designed to either leverage the access of users credentials or the default access granted to a pod via automountServiceAccountToken, which I wrote about recently. The harvester.py script currently primarily targets pod container environment variables, container manifest environment variables, and config map entries utilized as environment variables, to look for potential credentials.

Why Create A Credential Harvester

The default admission controls in many of the Kubernetes implementation apply a read/view policy to newly created users. However custom policies, admissions, and operators have become more common place. What’s more troublesome is the read permissions given to the automountServiceAccountToken by default. Without adjusting or disabling service tokens, compromised containers could effectively read all pod specs in all namespaces. With access to all pod specs, an attacker could potentially gather credentials or other sensitive information. Harvester is a tool that attempts to help automate the review process.

How Kubernetes Harvester Works

The harvester.py script utilizes the automountServiceAccountToken mounted within a given container or the standard user credentials within the Kube config file (~/.kube/config). Then the Kube API server is queried to look for sensitive information within the pod spec of each pod in the following steps.

  1. Use access to request pod specs for all namespaces within the cluster.
  2. Parse all pod specs to map and dedupe container container information
  3. Review each containers environment variables for sensitive values
  4. Review each config map entry, mapped to container environment variables for sensitive values
  5. Attempt to pull each container image and review the manifest environment variables for sensitive values
  6. Attempt to request authentication tokens from the internal metadata API for each of the major cloud provider

Other Resources:

  • Introduction to Kubernetes – A Free introduction course diving into Kubernetes as a tool for containerized infrastructure. Its a a great place to begin if your just getting started with Kubernetes.
  • The Linux Foundations Official Course – This is the most robust general knowledge based course I’ve seen. If you want to learn Kubernetes and how to do almost anything with it, get the CKA + CKAD combo package.