Sharing Resources with Pulumi

I’ve been trying out Pulumi lately to automate the deployment of cloud infrastructure in Azure. Pulumi is a great way to specify your deployments in a declarative manner, while also providing the ability to leverage the familiar constructs you are used to with common languages such as TypeScript, Python, Go and C#.

My project uses the new free tier of CosmosDb as its data store. To keep within the free tier limits of CosmosDb, I use a single, shared throughput database for both my development and production data, which will be separated into different collections.

Pulumi allows the creation of Stacks, which are great for using the same Pulumi deployment code across different environments, perfect for my dev and prod scenario. However, the Pulumi programming model assumes that the resources in each stack are separate, and it tracks the state of each individually. This means that I can’t have just a single Pulumi project to achieve this setup, which is what I originally tried.

To solve this, I structured my deployments to have one Pulumi project that represents the shared infrastructure I need, and a second project to deploy the infrastructure specific to each environment.

Creating a project for the shared infrastructure

The shared project will deploy the Azure resources I want to use across all of my release environments. These are:

Azure Resource Group
CosmosDb Account
CosmosDb Database

I use the TypeScript offering from Pulumi, so to create a project I executed the following:

pulumi new azure-typescript

I created a single stack through the CLI walkthrough, which I named shared.

I then created the Pulumi program:

import * as azure from '@pulumi/azure';

const resourceGroup = new azure.core.ResourceGroup("resource-group",
    {
        name: "myProject",
    }
);

const commonArgs = {
    resourceGroupName: resourceGroup.name,
    location: resourceGroup.location,
};

const cosmosDbAccount = new azure.cosmosdb.Account("cosmosdb-account",
    {
        name: "myProject",
        ...commonArgs,
        offerType: "Standard",
        consistencyPolicy: {
            consistencyLevel: "Session",
        },
        geoLocations: [{ location: resourceGroup.location, failoverPriority: 0 }],
    }
);

const cosmosDb = new azure.cosmosdb.SqlDatabase("cosmosdb-database",
    {
        name: "myProject-database",
        resourceGroupName: resourceGroup.name,
        accountName: cosmosDbAccount.name,
        throughput: 400,
    }
);

export const resourceGroupId = resourceGroup.id;
export const cosmosDbAccountId = cosmosDbAccount.id;
export const cosmosDbId = cosmosDb.id;

The key here is to export the id of each shared resource, so that they can be imported in the next Pulumi program.

Creating a project for the dev and prod environments

I then created a second project, which deploys the infrastructure specific to each environment, in this case a CosmosDb Collection for each. I used the same command as above to create a Pulumi project in a different folder, but this time I created a stack for dev and a stack for prod.

pulumi stack init prod

I then created a program within this project to deploy a CosmosDb collection.

import * as azure from '@pulumi/azure';
import * as pulumi from '@pulumi/pulumi';

const sharedInfra = new pulumi.StackReference("pulumiUserName/pulumiSharedProjectName/shared");
const resourceGroupId = sharedInfra.requireOutput("resourceGroupId");
const cosmosDbAccountId = sharedInfra.requireOutput("cosmosDbAccountId");
const cosmosDbId = sharedInfra.requireOutput("cosmosDbId");

const resourceGroup = azure.core.ResourceGroup.get("resource-group", resourceGroupId);
const cosmosDbAccount = azure.cosmosdb.Account.get("cosmosdb-account", cosmosDbAccountId);
const cosmosDb = azure.cosmosdb.SqlDatabase.get("cosmosdb-database", cosmosDbId);

const commonArgs = {
    resourceGroupName: resourceGroup.name,
    location: resourceGroup.location,
};

const cosmosDbContainer = new azure.cosmosdb.SqlContainer("cosmosdb-container",
    {
        name: `Container-${pulumi.getStack()}`,
        ...commonArgs,
        partitionKeyPath: "/Id",
        accountName: cosmosDbAccount.name,
        databaseName: cosmosDb.name
    }
);

This program pulls down a reference to the shared stack we created earlier, and then imports the previously exported resource ids. This allows us to create a reference to these resources, and then create the CosmosDb container specific to this stack.

By using pulumi.getStack() we can append the stack name to the container name, allowing us to have separate container for dev and prod while reusing the same Pulumi program.

Conclusion

And there we have it! We now have a Pulumi project we can use to automate the deployment of all of our shared infrastructure, and a second project we can deploy afterwards to create the resources we need for each specific environment!

In my next post, I’ll show how I execute these projects from Azure DevOps pipelines to completely automate the infrastructure deployments.

Sharing Resources with Pulumi.

Creating a project for the shared infrastructure

Creating a project for the dev and prod environments

Conclusion