Auto cancellation
Automatically cancel complex Terraform applies when alarms fire
Overview
One common issue when managing changes to Kubernetes infrastructure is the very long-running nature of the operations. Cluster upgrades can take hours for large node counts, plenty can go wrong, and engineers waste immense numbers of hours babysitting infrastructure automation to make sure it doesn't.
Plural helps solve this by polling the monitors you likely already have set up in tools like Datadog or New Relic to ensure infrastructure health, and automatically cancelling your IaC when they fire. Because we closely manage the commands themselves, we'll gracefully shut them down, ensuring things like annoying state locks are cleaned up and no resources are left dangling. We're trying to automate one of the most boring but labor-intensive parts of your DevOps workflow.
Setting this up is simple: you'll need to create an ObservabilityProvider resource and then set a list of observableMetrics on your stack.
Create an ObservabilityProvider
To do this in one swoop for Datadog, create resources like:
apiVersion: v1
kind: Secret
metadata:
name: datadog
namespace: stacks
stringData:
apiKey: YOUR_API_KEY
appKey: YOUR_APP_KEY
---
apiVersion: deployments.plural.sh/v1alpha1
kind: ObservabilityProvider
metadata:
name: datadog
spec:
type: DATADOG
name: datadog
credentials:
datadog:
name: datadog
namespace: stacksInfo:
You can also create this in the UI to avoid creating the secret, and reference it by name without the credentials block.
Add observableMetrics to your Stack
An example setup for this is here: