The Ultimate Guide: A Real-World Migration from Grafana 8 to 9 with PostgreSQL

Upgrading Grafana is a standard maintenance task, but moving from version 8 to version 9 is a significant leap. This isn't just a minor version bump; it's a fundamental change in Grafana's most critical feature: alerting. The deprecation of legacy dashboard alerts in favor of the new, centralized Unified Alerting system requires a careful, deliberate migration plan.

This guide provides a complete, battle-tested walkthrough based on a real-world migration. We won't just show you the ideal path; we'll show you how to do it safely by cloning your production environment first. Most importantly, we'll detail a sophisticated method to bypass all the common schema and data type errors by using Grafana 9 to create a "perfect" target schema before migrating any data.

The Core Challenge: A Two-Part Migration

The Grafana 8 to 9 upgrade is unique because it involves two distinct migrations:

1. Database Backend Migration: Moving your configuration store from SQLite to a production-grade PostgreSQL database.
2. Alerting System Migration: Converting all legacy panel-based alerts to the new Unified Alerting system, which is mandatory for Grafana 9.

Attempting to do both at once by simply changing the config and image tag is a recipe for failure. We will tackle this with a robust, controlled process.

The Game Plan: A Four-Phase Approach

Our strategy is designed to eliminate risk and ensure a perfect migration by tackling schema and data separately.

1. Phase 1: Clone Production. We'll create an isolated, high-fidelity clone of your Grafana 8 (SQLite) instance in a new namespace. This is our risk-free sandbox.
2. Phase 2: Generate the Perfect Target Schema. We'll cleverly use a temporary Grafana 9 instance to create a brand new, empty PostgreSQL database with the exact schema it needs, avoiding all legacy data type issues.
3. Phase 3: The pgloader Data-Only Migration. With a perfect schema in place, we will use the powerful pgloader tool to migrate only the data from our cloned SQLite database.
4. Phase 4: The Final Launch. We will deploy the final Grafana 9 instance, which will start up cleanly against the fully migrated and schema-perfect database.

Phase 1: Clone Your Production Grafana (The Safe Way)

Rule #1: Never test in production. Our first step is to replicate the production environment, including its SQLite grafana.db file, in a new, isolated namespace.

(For detailed steps on this cloning process, including backing up manifests and using a utility pod to copy the grafana.db file from one PVC to another, please refer to our previous guide on Cloning a Grafana Instance on Kubernetes.)

At the end of this phase, you should have a fully functional clone of your Grafana 8 instance, running on SQLite in a dedicated namespace (e.g., grafana-upgrade-test).

Phase 2: Generate the Perfect Grafana 9 Schema

This is the most critical and innovative step. Instead of fighting with schema mismatches, we will have Grafana 9 create the perfect schema for us from scratch.

1. Deploy a Fresh PostgreSQL Instance

In your grafana-upgrade-test namespace, deploy a new PostgreSQL instance. At this point, it will contain an empty, default grafana database.

export NEW_NAMESPACE="grafana-upgrade-test"

# Deploy the Postgres Secret, StatefulSet, and Service
kubectl apply -f postgres-secret.yaml -n $NEW_NAMESPACE
kubectl apply -f statefulset.apps.postgres-ss.yaml -n $NEW_NAMESPACE
kubectl apply -f service.postgres-svc.yaml -n $NEW_NAMESPACE

# Wait for the pod to be ready
kubectl wait --for=condition=ready pod/postgres-ss-0 -n $NEW_NAMESPACE --timeout=300s

# Login to new postgres pod, delete and recreate the grafana DB
kubectl exec -n $NEW_NAMESPACE postgres-ss-0 -- bash -c 'dropdb -U postgres grafana && createdb -U postgres grafana'

2. Deploy a Temporary Grafana 9 Instance

We will now briefly start a Grafana 9 pod. Its only purpose is to connect to the empty database and run its initial migrations, thereby building the perfect schema.

1. Prepare the Manifests:

   • Take your Grafana deployment.yaml and change the image tag to grafana/grafana:9.5.21.
   • Remove the PVC volume mount from the deployment. This instance is temporary and stateless.
   • Ensure the database environment variables (GF_DATABASE_...) point to your new PostgreSQL service.

2. Deploy and Let It Build:

   # Apply the ConfigMap (feature toggles are not needed yet)
   kubectl apply -f grafana-config.yaml -n $NEW_NAMESPACE
   
   # Deploy the modified Grafana 9 instance
   kubectl apply -f deployment-grafana-v9-stateless.yaml -n $NEW_NAMESPACE
   

   The pod will start, connect to Postgres, run all its v9 migrations, and then be in a Running state.

3. Dump the Perfect Schema and Tear Down: Once the pod is running, the schema is ready. We can now capture it and delete the temporary Grafana instance.

   PG_POD="postgres-ss-0"
   
   # Dump the schema ONLY to a file
   kubectl exec -n $NEW_NAMESPACE $PG_POD -- \
     dropdb -U postgres grafana && createdb -U postgres grafana \  
     pg_dump --schema-only -U postgres grafana > perfect-v9-schema.sql
     
   # Delete the temporary Grafana 9 deployment
   kubectl delete deployment grafana -n $NEW_NAMESPACE
   

You now possess a SQL file containing the exact table structures, column types (BOOLEAN, not BIGINT), and constraints that Grafana 9 expects.

Phase 3: The pgloader Data-Only Migration

Now we will combine our perfect schema with our original data.

1. Wipe and Prepare the Database

First, we wipe the database and restore only the perfect schema we just created.

# Copy the schema file into the Postgres pod
kubectl cp perfect-v9-schema.sql $NEW_NAMESPACE/$PG_POD:/tmp/perfect-schema.sql

# Exec in and restore the clean schema
kubectl exec -n $NEW_NAMESPACE $PG_POD -- bash -c ' \
  dropdb -U postgres grafana && \
  createdb -U postgres grafana && \
  psql -U postgres -d grafana -f /tmp/perfect-schema.sql \
'

2. Migrate Data with pgloader (Inside the Pod)

Our earlier attempts showed that running pgloader over the network via port-forward can be unstable due to Kubernetes liveness probes or network fragility. The most robust method is to run it directly inside the Postgres pod.

1. Copy Files to the Pod:

   # Copy the source SQLite database
   kubectl cp grafana.db $NEW_NAMESPACE/$PG_POD:/tmp/grafana.db
   
   # Create and copy the pgloader command file
   # data-only.load:
   LOAD DATABASE
    FROM sqlite://./grafana.db
    INTO postgresql://postgres:GRAFANA2026@localhost:5432/grafana
   
   WITH data only, reset sequences
   SET work_mem to '16MB', maintenance_work_mem to '512 MB';
   
   # LOAD DATABASE FROM sqlite:///tmp/grafana.db INTO postgresql:///grafana WITH data only...
   kubectl cp data-only.load $NEW_NAMESPACE/$PG_POD:/tmp/data-only.load
   

2. Exec and Run pgloader:

   # Exec into the pod for an interactive session
   kubectl exec -n $NEW_NAMESPACE -it $PG_POD -- bash
   
   # Inside the pod, run pgloader
   root@postgres-ss-0:/# pgloader /tmp/data-only.load
   

   This process will now succeed. It will load the data from the SQLite file into the perfectly prepared tables. The warnings about mismatched data types are expected and a sign of success, indicating that pgloader is correctly casting BIGINTs to BOOLEANs.

Phase 4: The Final Launch and Verification

The database is now fully migrated and has the correct schema and data. We are ready for the final launch.

1. Deploy the Final Grafana 9 Instance: Use your Grafana 9 deployment manifest (the stateless one from Phase 2). No feature toggles are needed in the ConfigMap, as Unified Alerting is the default in Grafana 9.

   kubectl apply -f grafana-config.yaml -n $NEW_NAMESPACE
   kubectl apply -f deployment-grafana-v9-stateless.yaml -n $NEW_NAMESPACE
   kubectl apply -f service.grafana.yaml -n $NEW_NAMESPACE
   

2. Monitor the Logs:

   kubectl logs -f -l app=grafana -n $NEW_NAMESPACE
   

   This time, Grafana will start, connect to the database, see that the schema is already at the latest version, and run its internal "legacy alert migration" on the data. Because the schema is perfect, this migration will succeed without error.

3. Verify Everything:

   • Log in to the new Grafana 9 UI.
   • Go to the Alerting page and confirm all your rules have been migrated.
   • Check your key dashboards and data sources to ensure everything works as expected.

By using Grafana 9 to define our target and pgloader to handle the complex data transfer, we bypassed a chain of frustrating schema errors and achieved a clean, reliable migration.