Automating Shard Provisioning with CircleCI Pipelines

At a Glance

Detail	Value
Industry	SaaS: Unified IT Operations Platform
Environment	AWS multi-account, ECS Fargate, RDS PostgreSQL, EC2 services
Timeline	6 weeks design and implementation
Key Results	90% reduction in shard provisioning time, zero manual errors, auditable deployments

The Challenge

The customer, a SaaS company building a unified IT operations platform, operated a multi-tenant architecture where each shard served a subset of tenants. As platform usage grew, telemetry signals (CPU, memory, connection pool pressure, request latency) indicated when a shard was approaching capacity and a new one needed to be provisioned.

Provisioning a shard involved spinning up a set of interconnected resources:

• EC2 instances running services that had yet to be containerised.
• ECS Fargate services running the platform’s microservices.
• RDS PostgreSQL instances with custom parameter groups, initialised schemas, and seed data.

These expansions were driven by capacity planning, not new customer onboarding.

AWS CDK handled the infrastructure provisioning. EC2 instances, RDS PostgreSQL, ECS Fargate services, and SSM Parameter Store entries were all defined as TypeScript stacks. After CDK deployed the infrastructure, Ansible playbooks took over for post-provisioning configuration: ensuring the EC2 instances had the correct package state, service configuration, and firewall rules, then running Liquibase-based database migrations against the newly provisioned RDS instance.

But the entire flow was executed manually by a senior engineer sitting at a terminal. cdk deploy invoked by hand, then ansible-playbook run in sequence. Each step required the engineer to wait, verify, and proceed.

Each new shard required:

1. Run cdk deploy for the shared infrastructure stack (EC2, RDS, SSM parameters, networking), wait for CloudFormation to complete.
2. Run the Ansible playbook to harden and configure the EC2 instances. Package installation, service enablement, firewall rules.
3. Run the Ansible Liquibase playbook against the new RDS endpoint. Schema migrations, seed data, stored procedures.
4. Manually verify ECS service connectivity to the new RDS instance and SSM parameters.
5. Run a final smoke test to confirm end-to-end health.

A single shard took 2 to 3 hours of an engineer’s focused time. With customer acquisition accelerating, the team was facing a choice: hire more infrastructure engineers, or automate the process end-to-end.

The Solution: CircleCI Pipeline

We designed a CircleCI pipeline that orchestrated CDK deployments and Ansible playbooks into a single fully automated, gated workflow. From infrastructure provisioning to configuration to validation.

Pipeline Architecture

The pipeline was structured as a sequential workflow with five stages:

Stage 0: Ticket Creation. Telemetry signals from the platform’s observability stack were wired to automatically create a ticket in the operations system when a shard approached capacity thresholds. The ticket contained a summary of the triggering metrics, the target account, and a single “Approve and Schedule” button.

Stage 1: CDK Infrastructure Deploy. When an operator clicked the button, CircleCI kicked off the pipeline. It ran cdk deploy to provision the core infrastructure: EC2 instances, RDS PostgreSQL, SSM Parameter Store entries, and supporting networking. All defined as TypeScript stacks. CloudFormation outputs (RDS endpoint, EC2 instance IDs) were captured as pipeline artifacts for downstream stages.

Stage 2: Ansible EC2 Configuration. With infrastructure in place, a dedicated job ran the Ansible playbook that configured the EC2 instances. Ensuring correct package state, service configuration files, firewall rules, and telemetry agent setup. Idempotency was verified by running Ansible in check mode after the apply.

Stage 3: Ansible RDS Migration. The pipeline dynamically retrieved the RDS endpoint from the captured CloudFormation outputs and executed the Ansible playbook that ran Liquibase-based schema migrations against the new database. Migrations were wrapped in a transaction, and the pipeline halted on any migration failure. This prevented a half-migrated shard from reaching production.

Stage 4: End-to-End Validation. A validation job verified the EC2 services were serving health checks, tested connectivity between ECS tasks and the RDS instance, confirmed the ECS service could retrieve parameters from SSM, and ran a synthetic transaction through the entire shard. Only on all-green status was the shard marked as active.

Gating and Error Handling

Each stage included automated rollback. If Ansible EC2 configuration failed, the partially configured shard was left for investigation but never exposed to traffic.

Observability and Notifications

The pipeline posted status updates to a dedicated Slack channel at each stage boundary. On success, the channel received the new shard’s connection details and a link to the CloudFormation stack. On failure, it included the failing stage, a link to the CircleCI job, and the engineer to page.

The Results

• 90% reduction in active engineer time per shard (from 2 to 3 hours down to roughly 15 minutes of review).
• 30-minute end-to-end provisioning time (down from 2 to 3 hours).
• Zero provisioning errors in the first 60 days post-deployment.
• Fully auditable: every pipeline run produced a complete log of every CDK deployment, Ansible execution, and migration step.
• Self-service: telemetry-driven ticketing meant operators approved expansions rather than manually detecting and scheduling them.

Key Takeaways

1. Automate the process, not just the tasks. The CDK stacks and Ansible playbooks existed and worked. But running them by hand in sequence introduced delay and error. Wrapping existing tools in a CI/CD pipeline with gating and validation was higher leverage than rewriting either one.
2. Validation is part of provisioning. A pipeline that provisions infrastructure without verifying it works is just faster manual failure. Stage-gate validation at every step. Verify SSM parameters, test RDS connectivity, confirm ECS task health. Then mark a shard active.
3. Use the right tool for each layer. CDK excelled at declaring cloud infrastructure as TypeScript. Ansible excelled at post-provisioning configuration and database migrations. The pipeline didn’t force a choice. It orchestrated both where each was strongest.
4. Wire telemetry to action. The pipeline’s real power was that it closed the loop between observability and operations. Telemetry detected the need, created the ticket, and presented a single button to execute a multi-tool workflow. No dashboards to watch, no manual triage, no context-switching between CDK and Ansible.