Remember that feeling when you needed a new Kafka topic, a dedicated database instance, or a new S3 bucket for a critical service? You'd open a ticket, wait for days (sometimes weeks!) for the operations team to provision it, and then often go back and forth with clarifications. This "ticket-driven development" for infrastructure is a common pain point, a productivity killer, and frankly, a source of frustration for everyone involved.
As developers, we thrive on autonomy and fast feedback loops. Waiting around for infrastructure is the antithesis of that. As organizations scale, this bottleneck only tightens, leading to slower innovation, inconsistent environments, and burnt-out operations teams.
This is where Internal Developer Platforms (IDPs) come into play. An IDP isn't just a collection of tools; it's a paradigm shift that empowers developers to safely and independently provision and manage their infrastructure and services within well-defined guardrails. It's about shifting from a "you build it, we run it (eventually)" mentality to "you build it, you run it, we give you the tools to do it right, right now."
In this article, we're going to dive deep into building a practical, hands-on IDP using two powerful open-source tools: Backstage as our developer portal and Crossplane as our infrastructure orchestration engine. Together, they create a seamless, self-service experience that will transform your development workflow.
The Bottleneck of Infrastructure Provisioning
Let's face it: infrastructure management is complex. Even with Infrastructure as Code (IaC) tools like Terraform or CloudFormation, developers often still hit a wall. Why? Because while IaC defines infrastructure, it doesn't inherently provide a self-service interface that's easy for every developer to use without deep knowledge of cloud providers or IaC specifics.
- Slow Feedback Loops: A developer needs a resource, files a ticket, waits for ops to write/run IaC, and then gets access. This cycle can take hours or days, breaking flow and forcing context switching.
- Context Switching Overhead: Developers spend valuable time understanding cloud provider nuances or IaC syntax, diverting focus from application logic.
- Inconsistent Environments: Manual provisioning or slight variations in IaC execution can lead to subtle differences between development, staging, and production environments, causing "works on my machine" syndromes.
- Ops Team Burnout: The operations team becomes a constant bottleneck, handling repetitive provisioning requests instead of focusing on strategic initiatives, reliability, and platform improvements.
- Security & Compliance Risks: Without centralized control, developers might inadvertently provision non-compliant resources, leading to security vulnerabilities or regulatory issues.
Traditional IaC is a fantastic tool for the platform or ops team, enabling them to codify desired states. However, it still leaves a "last mile" problem: how do we empower every developer to *consume* that codified infrastructure safely and efficiently without becoming an infrastructure expert? This is precisely the gap Backstage and Crossplane fill, acting as the bridge between developer intent and infrastructure reality.
The Synergy: Backstage for Developer Experience, Crossplane for Orchestration
At the heart of our IDP are two projects that, when combined, create a powerful, self-service ecosystem:
Backstage: Your Unified Developer Portal
Developed at Spotify and now a CNCF project, Backstage is an open-source platform for building developer portals. Think of it as your company's internal "App Store" for everything developers need. It provides:
- Service Catalog: A centralized, searchable inventory of all your software, services, APIs, and libraries. It helps you understand who owns what, its dependencies, and its operational status.
- Software Templates: A powerful feature that allows developers to spin up new projects, services, or infrastructure components from predefined templates with sensible defaults. This is where the magic of self-service begins.
- Documentation as Code: Integrate your tech documentation directly into the portal, making it easy to find and keep up-to-date.
- Plugins: An extensible ecosystem that allows you to integrate virtually any tool your developers use, from CI/CD dashboards to monitoring tools, cost management, and more.
Backstage streamlines the developer experience, making it easier to discover, use, and contribute to software within your organization. It's the UI that brings everything together.
Crossplane: The Kubernetes-Native Control Plane for Everything
Crossplane, also a CNCF project, extends Kubernetes to manage and provision external infrastructure and services. It transforms your Kubernetes cluster into a universal control plane, allowing you to manage cloud services (like AWS RDS, Azure Cosmos DB, Google Cloud SQL), SaaS products (like DataDog, GitHub repositories), and even on-premises resources using the familiar Kubernetes API.
The core concepts of Crossplane are:
- Providers: Extend Kubernetes to interact with specific external APIs (e.g.,
provider-aws,provider-gcp,provider-azure). - Managed Resources (MRs): Kubernetes Custom Resources (CRs) that represent external infrastructure resources (e.g.,
RDSInstance,S3Bucket). - Composite Resources (XRs) & Compositions: This is where Crossplane truly shines. Compositions allow you to abstract complex infrastructure blueprints into simpler, opinionated APIs that developers can consume. For example, an
XPostgreSQLInstancecould compose an AWS RDS instance, a VPC security group, and a secret in Kubernetes – all managed as a single logical unit.
Crossplane brings the power of the Kubernetes control plane to infrastructure provisioning. It's the engine that safely and declaratively creates and manages your cloud resources.
Together, Backstage provides the engaging frontend for developers, while Crossplane acts as the robust, opinionated backend, translating developer requests into actual, compliant infrastructure. It's a match made in platform engineering heaven!
Step-by-Step Guide: Building Your IDP with Backstage & Crossplane
Let's get our hands dirty and start building! We'll set up Backstage, deploy Crossplane, and then connect them to create a self-service infrastructure template.
Phase 1: Setting up Backstage
First, we need our developer portal. Backstage requires Node.js (v18 or higher recommended), Yarn, and Docker (for local development/testing).
- Create your Backstage app:
npx @backstage/create-app # Follow the prompts, e.g., 'my-idp-backstage' for the app name. cd my-idp-backstage yarn install yarn devThis will start Backstage locally, usually on
http://localhost:3000. You'll see a basic catalog and some demo components. - Explore the Service Catalog:
The Service Catalog is central to Backstage. Your existing services, libraries, and even infrastructure components can be registered here. For example, to define a simple service, you'd create a
catalog-info.yamlfile:# packages/my-microservice/catalog-info.yaml apiVersion: backstage.io/v1alpha1 kind: Component metadata: name: my-first-microservice description: A sample microservice for user management annotations: github.com/project-slug: my-org/my-first-microservice backstage.io/techdocs-ref: url:https://github.com/my-org/my-first-microservice spec: type: service lifecycle: experimental owner: team-alpha system: user-management consumesApis: [user-api] providesApis: [order-api] # We'll link this to our provisioned database later!You'd then add this file to your Backstage catalog configuration (e.g., in
app-config.yamlundercatalog.providers.githuborcatalog.locations).
Phase 2: Deploying Crossplane on Kubernetes
Crossplane runs on Kubernetes. You'll need access to a Kubernetes cluster (minikube, Kind, GKE, EKS, AKS, etc.) and Helm.
- Install Crossplane:
helm repo add crossplane-stable https://charts.crossplane.io/stable helm repo update helm install crossplane crossplane-stable/crossplane --namespace crossplane-system --create-namespace - Install a Provider (e.g., AWS):
kubectl apply -f https://raw.githubusercontent.com/crossplane/crossplane/release-1.14/docs/snippets/aws/provider-config/provider.yamlThis installs the AWS provider. You'll also need to configure AWS credentials, typically via a Kubernetes Secret and a
ProviderConfigresource. Refer to Crossplane's official docs for your specific cloud provider. - Define a Composite Resource Definition (XRD):
This is where we define our opinionated, self-service API. Let's create an
XPostgreSQLInstancethat developers can request.# crossplane/xrd-postgresql.yaml apiVersion: apiextensions.crossplane.io/v1 kind: CompositeResourceDefinition metadata: name: xpostgresqlinstances.platform.example.org spec: group: platform.example.org names: kind: XPostgreSQLInstance plural: xpostgresqlinstances claimNames: kind: PostgreSQLInstance plural: postgresqlinstances versions: - name: v1alpha1 served: true referenceable: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: id: type: string description: Identifier for the instance. parameters: type: object properties: storageGB: type: integer description: The amount of storage to allocate in GB. default: 20 engineVersion: type: string description: PostgreSQL engine version. default: "13.9" # Add more parameters developers might care about required: - storageGB - engineVersion required: - idApply this to your cluster:
kubectl apply -f crossplane/xrd-postgresql.yaml. - Create a Composition:
This composition tells Crossplane *how* to provision an AWS RDS PostgreSQL instance when a developer requests an
XPostgreSQLInstance. This is where your ops team defines the actual infrastructure blueprint, including security, networking, and tagging.In my last project, we struggled with developers accidentally deploying non-compliant S3 buckets. Crossplane's compositions became our guardrails. We defined strict policies (encryption, public access blocks) directly in the composition, ensuring that every S3 bucket provisioned through our IDP was compliant by default. It was a game-changer for enforcing best practices without hindering developer velocity.
# crossplane/composition-aws-rds.yaml apiVersion: apiextensions.crossplane.io/v1 kind: Composition metadata: name: xpostgresqlinstances.aws.platform.example.org labels: provider: aws engine: postgresql spec: compositeTypeRef: apiVersion: platform.example.org/v1alpha1 kind: XPostgreSQLInstance resources: - name: rds-instance base: apiVersion: rds.aws.upbound.io/v1beta1 kind: DBInstance spec: forProvider: engine: postgres engineVersion: "13.9" class: db.t3.medium # Enforce specific instance types allocatedStorage: 20 skipFinalSnapshot: true publiclyAccessible: false # NEVER allow public access by default! # ... other AWS specific configurations (VPC, SubnetGroup, SecurityGroup IDs, etc.) tags: - key: managedBy value: Crossplane - key: owner value: platform-team patches: - fromFieldPath: spec.id toFieldPath: metadata.annotations[crossplane.io/external-name] # Use the ID as the AWS instance name type: FromCompositeFieldPath - fromFieldPath: spec.parameters.storageGB toFieldPath: spec.forProvider.allocatedStorage type: FromCompositeFieldPath - fromFieldPath: spec.parameters.engineVersion toFieldPath: spec.forProvider.engineVersion type: FromCompositeFieldPath # Expose connection details back to the composite resource or a Kubernetes Secret - fromFieldPath: status.atProvider.endpoint toFieldPath: status.connectionDetails.host type: FromManagedResourceStatus - fromFieldPath: status.atProvider.port toFieldPath: status.connectionDetails.port type: FromManagedResourceStatusApply this to your cluster:
kubectl apply -f crossplane/composition-aws-rds.yaml.
Phase 3: Integrating Backstage Templates with Crossplane Compositions
This is the core of our self-service IDP. We'll create a Backstage Software Template that allows developers to request our PostgreSQLInstance.
- Create a Backstage Software Template:
Define a template in Backstage that asks for the necessary parameters (e.g., service name, storage size). This template will then generate a Kubernetes manifest for our
PostgreSQLInstanceclaim.# packages/backend/templates/new-service-with-db/template.yaml apiVersion: backstage.io/v1alpha1 kind: Template metadata: name: new-service-with-db title: New Service with PostgreSQL Database description: Scaffolds a new microservice with a dedicated PostgreSQL instance. spec: owner: platform-team type: service parameters: - title: Service Information properties: serviceName: title: Service Name type: string description: Unique name for your new microservice (e.g., 'user-auth-service'). ui:autofocus: true ui:options: rows: 5 owner: title: Owner type: string description: The team or individual who will own this service. ui:field: OwnerPicker ui:options: allowedKinds: [ "Group", "User" ] - title: Database Configuration properties: dbStorageGB: title: Database Storage (GB) type: number minimum: 10 maximum: 100 default: 20 description: Amount of storage for your PostgreSQL database. steps: - id: template name: Generate Files action: fetch:template input: url: ./content # Folder containing your microservice boilerplate values: serviceName: ${{ parameters.serviceName }} owner: ${{ parameters.owner }} dbStorageGB: ${{ parameters.dbStorageGB }} - id: publish name: Publish to GitHub action: publish:github input: repoUrl: github.com?owner=${{ parameters.owner | parseRepoUrl | pick 'owner' }}&repo=${{ parameters.serviceName }} # token: ${{ secrets.GITHUB_TOKEN }} # If using authenticated repos - id: register name: Register in Catalog action: catalog:register input: repoContentsUrl: ${{ steps.publish.output.repoContentsUrl }} catalogInfoPath: '/catalog-info.yaml' - id: create-db-claim name: Create PostgreSQL Instance Claim action: 'kubernetes:apply' # Backstage Kubernetes plugin action input: manifest: | apiVersion: platform.example.org/v1alpha1 kind: PostgreSQLInstance metadata: name: ${{ parameters.serviceName }}-db labels: app.kubernetes.io/part-of: ${{ parameters.serviceName }} backstage.io/managed-by-template: new-service-with-db spec: id: ${{ parameters.serviceName }}-db # Our custom ID for the XR parameters: storageGB: ${{ parameters.dbStorageGB }} engineVersion: "13.9" # Can be exposed as a parameter or fixedPlace this
template.yamland a `content` folder (containing a basic microservice structure and its `catalog-info.yaml`) into your Backstage backend's templates directory (e.g.,packages/backend/templates/new-service-with-db). You'll also need to configure the Kubernetes plugin for Backstage and ensure it has permissions to apply resources to your cluster. - Developer Workflow:
Now, a developer logs into Backstage, navigates to "Create New Software," selects "New Service with PostgreSQL Database," fills in the simple form fields (Service Name, Storage GB), and clicks "Create."
- Backstage generates the microservice boilerplate.
- It publishes the code to GitHub.
- It registers the new service in the Backstage Service Catalog.
- Crucially, it applies the
PostgreSQLInstanceclaim manifest to Kubernetes.
Crossplane, watching the Kubernetes API, sees the new
PostgreSQLInstanceclaim. Because we defined aCompositionearlier, Crossplane knows exactly how to fulfill this claim by provisioning an AWS RDS PostgreSQL instance with the specified storage and all the underlying AWS resources (like VPC, security groups) and compliance guardrails defined by the platform team.
Phase 4: Enhancing the Developer Experience
The journey doesn't end with provisioning. Backstage can show the status of the provisioned resources and provide actionable insights:
- Resource Visibility: You can create Backstage plugins or integrate existing ones to display the status of the Crossplane-managed resources directly on the service's overview page in the catalog. Developers can see if their database is healthy, its connection details, and even links to cloud provider consoles (if allowed).
- Cost Management: Integrate cost management tools to show the running cost of the provisioned database, fostering cost awareness.
- Monitoring & Logs: Link directly to monitoring dashboards (Grafana, Datadog) and log aggregators (Elasticsearch, Loki) relevant to their database instance.
This creates a true single pane of glass, reducing context switching and empowering developers with all the information they need to operate their services effectively.
Outcome and Takeaways: The Power of Self-Service
Implementing an IDP with Backstage and Crossplane transforms your organization's approach to infrastructure:
- Accelerated Development Cycles: Developers get the infrastructure they need in minutes, not days or weeks. This translates directly to faster feature delivery and quicker experimentation.
- Reduced Ops Toil: The operations team moves from handling repetitive tickets to defining powerful, reusable infrastructure compositions and maintaining the platform itself. Their work becomes more strategic and less reactive.
- Improved Consistency & Compliance: All infrastructure is provisioned through standardized, opinionated compositions defined by the platform team. This ensures consistency across environments and embeds security and compliance best practices by default.
- Empowered Developers: Developers gain a sense of ownership and autonomy, with clear boundaries and guardrails. They understand the infrastructure supporting their applications and can respond quickly to operational needs.
- Shift-Left Infrastructure Concerns: By providing self-service capabilities, infrastructure considerations become part of the development workflow earlier, leading to better architectural decisions.
This setup fundamentally shifts infrastructure from a ticket-based request system to an API-driven, self-service model. Developers interact with familiar concepts in Backstage, which under the hood, orchestrates complex infrastructure using Crossplane's powerful Kubernetes-native capabilities.
Conclusion
Building an Internal Developer Platform with Backstage and Crossplane is an investment, but the returns in developer productivity, operational efficiency, and consistent infrastructure are immense. It's about empowering your engineers, reducing friction, and fostering a culture of ownership and innovation.
You've seen how to stitch together a powerful developer portal with an extensible infrastructure control plane. This is just the beginning. You can expand your IDP with more sophisticated software templates, integrate more providers into Crossplane, and build custom Backstage plugins to expose even more insights to your developers. Start small, iterate, and watch your team's velocity soar. It's time to move from manual ops to self-service development – your developers will thank you!
