Technical Architecture

Designing a Secure, Scalable Public API on Kubernetes (EKS) with a Service Mesh and Infrastructure-as-Code

Target Audience: Platform engineers, DevOps, SREs, and architects building public-facing APIs on AWS.

This guide outlines a pragmatic, production-ready pattern for exposing an internal backend running on Amazon EKS through a service mesh (Istio) and a hardened external proxy (HAProxy) managed via Terraform. We will cover architecture, traffic flow, security, observability, cost considerations, and deployment snippets.

Goals

Expose a REST/gRPC API securely to public clients.
Keep the application deployment and platform concerns separate.
Use a service mesh for fine-grained traffic control and mTLS inside the cluster.

Use an external, configurable reverse proxy (HAProxy) to terminate public traffic, provide WAF-like protections, and centralize TLS.
Manage infra consistently with Terraform and GitOps for app config.

Architecture Flow

1. Clients (public internet) → 2. Cloud Load Balancer / CDN (optional) → 3. External HAProxy (EC2/ASG) → 4. Ingress Gateway (Istio) on EKS → 5. Istio Mesh + App Pods → 6. Backends (DBs, caches, third-party APIs)

Why this split?

HAProxy provides a mature, low-latency L7 proxy with advanced ACLs, traffic shaping, header rewriting, and integration with existing datacenter tooling. Istio handles service-to-service security, retries, circuit breaking, and observability inside the cluster.

Management Split

Terraform manages HAProxy infrastructure and cloud resources; GitOps (ArgoCD/Flux) manages Kubernetes manifests and Istio configuration.

Components & Responsibilities

Amazon EKS

Host application workloads and the Istio control/data plane.
Use node groups for separation (system, app, infra) and enable cluster auto scaler.
Use IAM Roles for Service Accounts (IRSA) for least privilege access to AWS APIs.

Istio (Service Mesh)

Sidecar proxies (Envoy) for observability and traffic control.
Define Gateway and VirtualService for external ingress rules.

HAProxy (External Proxy)

TLS termination and certificate management (ACM or ACME).
Edge filtering (rate limits, IP ACLs, geo-blocking), header sanitation, and bot mitigation.
TCP/HTTP health checks and dynamic reconfiguration (Runtime API or consul-template).

Terraform

Provision EC2 ASG or fleet for HAProxy (or NLB + target groups).
Provision VPCs, subnets, security groups, ALB/NLB, Route53, and IAM roles.
Store state securely with S3 backend + DynamoDB locking.

Detailed Traffic Journey

Client connects to the public DNS (optionally via CloudFront).
TLS is terminated at HAProxy (or CloudFront if used for WAF/CDN).
HAProxy performs ACL checks and can route to different EKS clusters / regions.
HAProxy forwards to the Istio Gateway (secured channel, optionally mTLS) exposed via NodePorts or internal LoadBalancer.
Istio Gateway routes the request to the correct VirtualService which applies policies (retries, timeouts, mirroring).
Sidecar proxies enforce mTLS and observability; service receives traffic.

Security Patterns

Edge & Network Control

Edge TLS: Terminate on HAProxy; use Let's Encrypt or ACM + CloudFront for managed certs.
Network controls: Use security groups and NACLs to limit traffic between HAProxy fleet and EKS nodes.

Cluster & Application Security

mTLS inside cluster: Enforce mutual TLS using Istio policies.
WAF: Use CloudFront + AWS WAF or integrate WAF-like rules inside HAProxy.
Secrets: Use AWS Secrets Manager or HashiCorp Vault; mount to pods via CSI drivers.

Rate limiting & quotas: Implement at the edge (HAProxy) and again at mesh-level for defense-in-depth.

Deployment Examples (Snippets)

1. Minimal Istio Gateway + VirtualService (Kubernetes Manifest)

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: public-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts: ["api.example.com"]
    - port:
        number: 443
        name: https
        protocol: HTTPS
      tls:
        mode: SIMPLE
        credentialName: istio-public-cert
      hosts: ["api.example.com"]
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: api
spec:
  hosts: ["api.example.com"]
  gateways: ["public-gateway"]
  http:
    - match:
        - uri:
            prefix: "/v1"
      route:
        - destination:
            host: api-service
            port:
              number: 80

2. Terraform: Simple HAProxy ASG (Pseudo-code)

resource "aws_launch_template" "haproxy" { 
  /* AMI with HAProxy baked */ 
}

resource "aws_autoscaling_group" "haproxy_asg" { 
  /* subnets, LT, min/max, target group */ 
}

resource "aws_lb_target_group" "haproxy_tg" { 
  /* health checks */ 
}

resource "aws_lb" "public" { 
  /* NLB or ALB in front */ 
}

Observability

Metrics

Prometheus scraping Envoy/HAProxy metrics.

Tracing

Jaeger or Tempo integrated via Istio.

Logs

Centralized logging to CloudWatch/ELK via Fluentd/Fluent Bit.

Operational Concerns

Blue/Green or Canary: Use Istio traffic-splitting for progressive rollouts.
Scaling: EKS Horizontal Pod Autoscaler + Cluster Autoscaler; HAProxy ASG sized for peak load with warm-up strategy.

Disaster Recovery: Multi-AZ at minimum; for DR, replicate cluster/data to another region and use DNS failover.
Testing: Synthetic canaries, load testing, and chaos experiments (simulate pod/az failures).

Cost Considerations & Tradeoffs

Approach	Pros	Cons
HAProxy on EC2	Full control, advanced features	EC2 cost + management overhead
Managed Alternatives	Reduced operational burden	ALB/WAF may lack HAProxy flexibility
Istio Mesh	Deep observability & security	Sidecar overhead (budget CPU/RAM)

Best Practices Checklist

Enforce least privilege (IRSA + fine-grained IAM).
Use TLS everywhere (edge + mesh).
Centralize cert management and automated rotation.

Use GitOps for Kubernetes manifests; Terraform for cloud infra.
Implement progressive delivery (canaries) with Istio.
Monitor latency and error budgets; set SLOs.

Conclusion

This pattern gives teams the ability to combine a hardened, featureful proxy with the rich in-cluster capabilities of a service mesh. It balances operational control and platform consistency while enabling advanced traffic management, security, and progressive deployments. Adapt the exact placement of TLS, WAF, and proxying to match organizational risk and operational maturity.

Back to Blogs

Secure Public APIs on Kubernetes