Provisioning a secured Grafana instance in Google Cloud thanks to Terraform
Provisioning Grafana in the Cloud can be a bit exhausting: setup the database, the admin credential, the API key, the public access... Let me show you, it’s not so hard!
--
Before playing with the API of a Grafana instance to create dynamic stuff like dashboards, you’ll have to provision it properly:
- first, creating the Grafana database,
- then, changing the admin password for a more secured one (
adminby default), - then, running and exposing the Grafana instance,
- finally, generating an API token from the admin credential, for a full access to the API.
The Google Cloud toolbox is perfect to accomplish theses tasks:
- Cloud SQL for the Grafana database,
- Cloud Run for running the Grafana instance and for the CLI commands like changing the admin password.
Fortunately, theses steps can be described as code thanks to Terraform. This article is going to detail each step.
Creating the Grafana database
Grafana allows different configurations for its data storage. Among those compatible with Cloud SQL, MySQL is a good candidate.
provider "google" {
project = var.project
region = var.region
}
variable "project" {
default = "my-gcloud-project"
}
variable "region" {
default = "europe-west6"
}
variable "sql_instance" {
default = "my-sql-instance"
}
resource "random_password" "sql_user" {
length = 16
}
resource "google_sql_user" "default" {
instance = var.sql_instance
name = "grafana"
password = random_password.sql_user.result
host = "%"
}
resource "google_sql_database" "default" {
instance = var.sql_instance
name = "grafana"
collation = "utf8_general_ci"
charset = "utf8"
}️️️To ️️️️simplify code snippets, I’m using the random_password resource directly. It would be quite better in a security matter to use the Secret Manager like done in this community tuto.
Changing the admin password
On a fresh instance, the default admin password is admin . It can be easily changed thanks to the Grafana CLI. The Jobs of Cloud Run can be used to run the command.
variable "grafana_version" {
default = "latest"
}
locals {
sql_instance_connection_name = "${var.project}:${var.region}:${var.sql_instance}"
}
resource "random_password" "admin_password" {
length = 16
special = false
}
resource "google_cloud_run_v2_job" "reset_admin_password" {
name = "grafana-reset-admin-password"
location = var.region
launch_stage = "BETA"
template {
task_count = 1
template {
containers {
image = "docker.io/grafana/grafana-oss:${var.grafana_version}"
command = ["grafana-cli"]
args = [
"admin",
"reset-admin-password",
random_password.admin_password.result,
]
env {
name = "GF_DATABASE_HOST"
value = "/cloudsql/${local.sql_instance_connection_name}"
}
env {
name = "GF_DATABASE_TYPE"
value = "mysql"
}
env {
name = "GF_DATABASE_NAME"
value = google_sql_database.default.name
}
env {
name = "GF_DATABASE_USER"
value = google_sql_user.default.name
}
env {
name = "GF_DATABASE_PASSWORD"
value = google_sql_user.default.password
}
env {
name = "GF_LOG_MODE"
value = "console"
}
env {
name = "GF_LOG_CONSOLE_FORMAT"
value = "json"
}
env {
name = "GF_LOG_LEVEL"
value = "warn"
}
volume_mounts {
name = "cloudsql"
mount_path = "/cloudsql"
}
resources {
limits = {
"cpu" = "1000m"
"memory" = "512Mi"
}
}
}
volumes {
name = "cloudsql"
cloud_sql_instance {
instances = [local.sql_instance_connection_name]
}
}
max_retries = 0
}
}
}As described in the doc, it’s possible to configure Grafana with environment variables thanks to this pattern GF_<SectionName>_<KeyName> . We can see them in action with GF_DATABASE_* configs thanks to the previous chapter about the database resources creation. Moreover, for better logs structure in Cloud Logging, I also set some GF_LOG_* env vars to print them as json in stderr from the warning level.
Be aware that the job is only created at this stage and not started yet. To execute it, the gcloud module resolves this situation, performing the remote execution in the cloud:
module "execute_reset_admin_password" {
source = "terraform-google-modules/gcloud/google"
additional_components = ["beta"]
create_cmd_body = "beta run jobs execute ${google_cloud_run_v2_job.reset_admin_password.name} --region ${var.region}"
}Run the Grafana instance
Now that the database and the admin credential are properly setup, we can expose the Grafana instance through a Cloud Run Service connected to the database.
provider "google-beta" {
project = var.project
region = var.region
}
resource "google_cloud_run_service" "default" {
provider = google-beta
location = var.region
name = "grafana"
autogenerate_revision_name = true
template {
metadata {
annotations = {
"autoscaling.knative.dev/maxScale" = "1"
"run.googleapis.com/cloudsql-instances" = local.sql_instance_connection_name
}
}
spec {
container_concurrency = 80
containers {
image = "docker.io/grafana/grafana-oss:${var.grafana_version}"
ports {
container_port = 3000
}
env {
name = "GF_DATABASE_HOST"
value = "/cloudsql/${local.sql_instance_connection_name}"
}
env {
name = "GF_DATABASE_TYPE"
value = "mysql"
}
env {
name = "GF_DATABASE_NAME"
value = google_sql_database.default.name
}
env {
name = "GF_DATABASE_USER"
value = google_sql_user.default.name
}
env {
name = "GF_DATABASE_PASSWORD"
value = google_sql_user.default.password
}
env {
name = "GF_LOG_MODE"
value = "console"
}
env {
name = "GF_LOG_CONSOLE_FORMAT"
value = "json"
}
env {
name = "GF_LOG_LEVEL"
value = "warn"
}
resources {
limits = {
"cpu" = "1000m"
"memory" = "256Mi"
}
requests = {}
}
liveness_probe {
http_get {
path = "/api/health"
}
}
}
}
}
traffic {
latest_revision = true
percent = 100
}
depends_on = [module.execute_reset_admin_password]
}Note that the google-beta provider is used here because the liveness probe isn’t yet released in stable. That calls the Grafana health check before serving traffic.
This Cloud Run Service depends on module.execute_reset_admin_password in order to expose it only after enforcing the admin password.
By default, the service isn’t public. It gives access to the authorizes users through Cloud IAM. To make it public, the next custom IAM policy is required:
data "google_iam_policy" "noauth" {
binding {
role = "roles/run.invoker"
members = ["allUsers"]
}
}
resource "google_cloud_run_service_iam_policy" "default" {
service = google_cloud_run_service.default.name
location = var.region
policy_data = data.google_iam_policy.noauth.policy_data
}Generating a Grafana API token
To create the token that gives full access to the API, a special provider using the admin credential as basic HTTP auth is defined. Its unique use case will be to create the API key. It’s why this provider is aliased in order to not clash with the default one.
provider "grafana" {
alias = "basic_auth_admin"
url = google_cloud_run_service.default.status[0].url
auth = "admin:${random_password.admin_password.result}"
}Then, we can create the grafana_api_key using the special provider.
resource "grafana_api_key" "default" {
provider = grafana.basic_auth_admin
name = "admin"
role = "Admin"
depends_on = [
google_cloud_run_service.default,
time_sleep.wait_after_iam_policy,
]
}
resource "time_sleep" "wait_after_iam_policy" {
depends_on = [google_cloud_run_service_iam_policy.default]
create_duration = "1m"
}Before contacting the API, an 1 minute sleep is started in order to let the public access propagation reaches the Google front load balancer. Without it, a 401 response is sometimes get.
Let’s play with the Grafana API!
Once the API key is succesfully created and stored in the Terraform state, we can configure the default Grafana provider with it.
provider "grafana" {
url = google_cloud_run_service.default.status[0].url
auth = grafana_api_key.default.key
}Then… let’s create our very first dashboard 📈 🎉
resource "grafana_dashboard" "my_dashbaord" {
config_json = jsonencode({title: "my dashboard"})
}Useful outputs
Thanks to the next outputs, you can more easily login to your Grafana instance.
output "grafana_url" {
value = google_cloud_run_service.default.status[0].url
}
output "admin_password" {
value = random_password.admin_password.result
sensitive = true
}To print them:
$ terraform output grafana_url
"https://grafana-wd7qskso4a-oa.a.run.app"
$ terraform output admin_password
"..."