Airbyte Deployment

Overview

This example shows how to deploy Airbyte on Qovery using Terraform. Airbyte is deployed from its official Helm chart, with a PostgreSQL database and a proxy that protects the Airbyte web app with basic authentication.

This page is based on qovery-airbyte, an external example repository. The proxy application of this page builds from that repository.

What You’ll Deploy

PostgreSQL Database: Managed database for Airbyte metadata
Airbyte (Helm): Airbyte deployed from the official Helm chart
Proxy Application: Reverse proxy with basic authentication in front of the Airbyte web app
Deployment Stages: Ordered deployment (Database → App → Proxy)

Prerequisites

Before deploying, gather these values:

A Qovery API token and the IDs of your organization, project and cluster. See Find resource IDs.
A cluster on AWS. The database uses db.t3.small, an AWS RDS instance type. On another cloud provider, set an instance type that the provider offers.
Basic authentication credentials for the proxy: one htpasswd entry with a SHA-1 password, username:{SHA}.... The Gateway API expects this format, and NGINX accepts it too, so the credentials work whichever of the two routes the traffic.

File Structure

qovery-airbyte/
├── main.tf
├── variables.tf
└── airbyte-values.yaml

variables.tf

variable "qovery_access_token" {
  description = "Qovery API token"
  type        = string
  sensitive   = true
}

variable "qovery_organization_id" {
  description = "Qovery Organization ID"
  type        = string
}

variable "qovery_project_id" {
  description = "Qovery Project ID"
  type        = string
}

variable "qovery_cluster_id" {
  description = "Qovery Cluster ID"
  type        = string
}

variable "airbyte_helm_version" {
  description = "Airbyte Helm chart version"
  type        = string
  default     = "1.7.1"
}

variable "airbyte_service_name" {
  description = "Airbyte service name"
  type        = string
  default     = "Airbyte"
}

variable "qovery_airbyte_web_app_proxy_basic_auth" {
  description = "Basic Auth for Airbyte web app proxy: one htpasswd entry, username:{SHA}..."
  type        = string
  sensitive   = true
}

main.tf

terraform {
  required_providers {
    qovery = {
      source  = "qovery/qovery"
      version = "~> 1.0"
    }
  }
}

provider "qovery" {
  token = var.qovery_access_token
}

# Create environment for Airbyte
resource "qovery_environment" "airbyte" {
  project_id = var.qovery_project_id
  cluster_id = var.qovery_cluster_id
  name       = "airbyte"
  mode       = "PRODUCTION"
}

# Deployment stages: the database, then Airbyte, then the proxy
resource "qovery_deployment_stage" "database" {
  environment_id = qovery_environment.airbyte.id
  name           = "DATABASE"
}

resource "qovery_deployment_stage" "app" {
  environment_id = qovery_environment.airbyte.id
  name           = "APP"
  is_after       = qovery_deployment_stage.database.id
}

resource "qovery_deployment_stage" "proxy" {
  environment_id = qovery_environment.airbyte.id
  name           = "PROXY"
  is_after       = qovery_deployment_stage.app.id
}

# Create PostgreSQL database for Airbyte
resource "qovery_database" "airbyte_db" {
  environment_id      = qovery_environment.airbyte.id
  deployment_stage_id = qovery_deployment_stage.database.id
  name                = "airbyte-db"
  type                = "POSTGRESQL"
  version             = "17"
  mode                = "MANAGED"
  instance_type       = "db.t3.small"
  storage             = 20
  accessibility       = "PRIVATE"
}

locals {
  # Prefix of the built-in variables of the database, for example QOVERY_POSTGRESQL_ZA1B2C3D4
  airbyte_db_variables = "QOVERY_POSTGRESQL_Z${upper(split("-", qovery_database.airbyte_db.id)[0])}"
}

# Add Airbyte Helm repository
resource "qovery_helm_repository" "airbyte" {
  organization_id       = var.qovery_organization_id
  name                  = "airbyte"
  kind                  = "HTTPS"
  url                   = "https://airbytehq.github.io/helm-charts"
  skip_tls_verification = false
}

# Deploy Airbyte using Helm
resource "qovery_helm" "airbyte" {
  environment_id      = qovery_environment.airbyte.id
  deployment_stage_id = qovery_deployment_stage.app.id
  name                = var.airbyte_service_name
  description         = "Airbyte deployed from its official Helm chart"

  source = {
    helm_repository = {
      helm_repository_id = qovery_helm_repository.airbyte.id
      chart_name         = "airbyte"
      chart_version      = var.airbyte_helm_version
    }
  }

  # Allow Airbyte to create cluster-wide resources
  allow_cluster_wide_resources = true

  # Override default values with airbyte-values.yaml, keyed by file name
  values_override = {
    file = {
      raw = {
        "airbyte-values.yaml" = {
          content = file("${path.module}/airbyte-values.yaml")
        }
      }
    }
  }

  # Database connection details, referenced as qovery.env.<NAME> in airbyte-values.yaml
  environment_variable_aliases = [
    {
      key   = "DATABASE_HOST"
      value = "${local.airbyte_db_variables}_HOST_INTERNAL"
    },
    {
      key   = "DATABASE_PORT"
      value = "${local.airbyte_db_variables}_PORT"
    },
    {
      key   = "DATABASE_NAME"
      value = "${local.airbyte_db_variables}_DEFAULT_DATABASE_NAME"
    },
    {
      key   = "DATABASE_USER"
      value = "${local.airbyte_db_variables}_LOGIN"
    }
  ]

  secret_aliases = [
    {
      key   = "DATABASE_PASSWORD"
      value = "${local.airbyte_db_variables}_PASSWORD"
    }
  ]
}

# Deploy proxy application for Airbyte web app
resource "qovery_application" "airbyte_webapp_proxy" {
  environment_id      = qovery_environment.airbyte.id
  deployment_stage_id = qovery_deployment_stage.proxy.id
  name                = "airbyte-webapp-proxy"

  # Caddy reverse proxy from the external example repository
  git_repository = {
    url    = "https://github.com/evoxmusic/qovery-airbyte.git"
    branch = "main"
  }

  build_mode      = "DOCKER"
  dockerfile_path = "Dockerfile.webappproxy"

  cpu    = 100
  memory = 128

  min_running_instances = 1
  max_running_instances = 1

  ports = [{
    internal_port       = 80
    external_port       = 443
    protocol            = "HTTP"
    publicly_accessible = true
    name                = "http"
  }]

  environment_variables = [
    {
      # Qovery replaces {{...}} with the internal host name of the Airbyte Helm release.
      key   = "AIRBYTE_WEBAPP_INTERNAL_HOST"
      value = "{{QOVERY_HELM_Z${upper(split("-", qovery_helm.airbyte.id)[0])}_HOST_INTERNAL}}-airbyte-webapp-svc"
    },
    {
      key   = "AIRBYTE_WEBAPP_INTERNAL_PORT"
      value = "80"
    },
    {
      key   = "AIRBYTE_WEBAPP_BASIC_AUTH"
      value = var.qovery_airbyte_web_app_proxy_basic_auth
    }
  ]

  # Protect the public endpoint with the credentials of AIRBYTE_WEBAPP_BASIC_AUTH,
  # whether NGINX (ingress) or the Gateway API routes the traffic.
  advanced_settings_json = jsonencode({
    "network.ingress.basic_auth_env_var"     = "AIRBYTE_WEBAPP_BASIC_AUTH"
    "network.gateway_api.basic_auth_env_var" = "AIRBYTE_WEBAPP_BASIC_AUTH"
  })

  healthchecks = {
    readiness_probe = {
      type = {
        http = {
          path   = "/"
          port   = 80
          scheme = "HTTP"
        }
      }
      initial_delay_seconds = 30
      period_seconds        = 10
      timeout_seconds       = 5
      success_threshold     = 1
      failure_threshold     = 3
    }

    liveness_probe = {
      type = {
        http = {
          path   = "/"
          port   = 80
          scheme = "HTTP"
        }
      }
      initial_delay_seconds = 30
      period_seconds        = 10
      timeout_seconds       = 5
      success_threshold     = 1
      failure_threshold     = 3
    }
  }
}

# Creating the services does not deploy them: this resource deploys the environment,
# stage by stage.
resource "qovery_deployment" "airbyte" {
  environment_id = qovery_environment.airbyte.id
  desired_state  = "RUNNING"

  depends_on = [
    qovery_database.airbyte_db,
    qovery_helm.airbyte,
    qovery_application.airbyte_webapp_proxy,
  ]
}

output "airbyte_url" {
  value = "https://${qovery_application.airbyte_webapp_proxy.external_host}"
}

airbyte-values.yaml

Create a airbyte-values.yaml file to customize Airbyte configuration. Qovery replaces each qovery.env.<NAME> with the value of the <NAME> variable of the Helm service when it deploys the chart, as described in Environment Variables in Values.

global:
  serviceAccountName: airbyte-admin
  deploymentMode: oss

  database:
    type: external
    host: qovery.env.DATABASE_HOST
    port: qovery.env.DATABASE_PORT
    database: qovery.env.DATABASE_NAME
    user: qovery.env.DATABASE_USER
    password: qovery.env.DATABASE_PASSWORD

# Use the Qovery database instead of the PostgreSQL of the chart
postgresql:
  enabled: false

# The managed database requires TLS connections
temporal:
  extraEnv:
    - name: POSTGRES_TLS_ENABLED
      value: "true"
    - name: POSTGRES_TLS_DISABLE_HOST_VERIFICATION
      value: "true"
    - name: SQL_TLS_ENABLED
      value: "true"
    - name: SQL_TLS_DISABLE_HOST_VERIFICATION
      value: "true"

webapp:
  enabled: true
  replicaCount: 1
  service:
    type: ClusterIP
    port: 80

server:
  enabled: true
  replicaCount: 1

worker:
  enabled: true
  replicaCount: 1

airbyte-bootloader:
  enabled: true

Deployment Steps

Clone or Create Files

# Option 1: Clone the external example repository
git clone https://github.com/evoxmusic/qovery-airbyte
cd qovery-airbyte

# Option 2: Create the files manually
# Create main.tf, variables.tf, and airbyte-values.yaml

The files of the repository can differ from the ones on this page.

Set Environment Variables

export TF_VAR_qovery_access_token="your-api-token"
export TF_VAR_qovery_organization_id="your-org-id"
export TF_VAR_qovery_project_id="your-project-id"
export TF_VAR_qovery_cluster_id="your-cluster-id"

Set Basic Authentication

# Generate an htpasswd entry with a SHA-1 password (username:{SHA}...)
export TF_VAR_qovery_airbyte_web_app_proxy_basic_auth=$(printf '%s\n' 'yourpassword' | htpasswd -ni -s admin)

Initialize Terraform

terraform init

Plan and Apply

# Review changes
terraform plan

# Create the resources and deploy the environment
terraform apply

terraform apply waits for the deployment of the three stages.

Access Airbyte

Open the URL of the proxy and log in with your basic authentication credentials:

terraform output airbyte_url

You can also forward a local port to the proxy. When prompted, select the airbyte-webapp-proxy service:

qovery port-forward --port 8080:80

# Open http://localhost:8080

Port forwarding connects to the proxy directly, without the basic authentication of the public endpoint.

Cleanup

terraform destroy

Key Takeaways

This example demonstrates several advanced Terraform patterns:

Multi-Service Deployment

Deploying multiple interconnected services:

Managed PostgreSQL database
Helm chart application
Proxy application for authentication

Deployment Stages

Ensuring proper deployment order with qovery_deployment_stage resources and the deployment_stage_id of each service:

DATABASE: PostgreSQL deploys first
APP: Airbyte Helm chart deploys after database
PROXY: Authentication proxy deploys last

Dynamic Configuration

Using Qovery’s database connection details in Helm values:

Aliases of the database’s built-in variables on the Helm service
Variables referenced as qovery.env.<NAME> in airbyte-values.yaml
The internal host of the Helm release injected in the proxy with {{...}} interpolation

External Helm Charts

Integrating third-party Helm repositories:

Adding Airbyte’s official Helm repository
Deploying specific chart versions
Overriding values with custom configuration

Security

Basic authentication on the public endpoint of the proxy
Private database accessibility
Database password passed as a secret alias, never written in the configuration

Health Checks

Liveness and readiness probes on the proxy:

Initial delay to allow services to start
Regular health checks
Automatic restarts on failure

Resource Management

Setting CPU and memory for each service:

Minimal resources for proxy (100 mCPU, 128 MB)
Managed database with an instance type and defined storage
Cluster-wide resource permissions for Airbyte

Troubleshooting

Deployment Takes Too Long

Airbyte deployment typically takes several minutes:

Database must be ready first, and a managed database takes several minutes to create
Helm chart pulls multiple images
Airbyte bootloader initializes the database

If the Helm deployment times out, increase timeout_sec on qovery_helm (600 seconds by default).Check deployment status:

qovery status

Cannot Access Web UI

If you can’t access the Airbyte UI:

Verify all services are deployed
Check proxy application is running
Ensure basic auth credentials are correct
Try port-forwarding locally

Database Connection Issues

If Airbyte can’t connect to the database:

Verify database is in RUNNING state
Check the aliases of the Helm service point to the variables of the database
Review airbyte-values.yaml configuration
Check Airbyte logs for connection errors

Next Steps

Example Repository

External example repository this page is based on

Airbyte Documentation

Learn more about Airbyte

Helm Configuration

Deploy more Helm charts

Advanced Patterns

Learn advanced Terraform techniques

Getting Started

Examples

Airbyte Deployment

Overview

What You’ll Deploy

Prerequisites

File Structure

variables.tf

main.tf

airbyte-values.yaml

Deployment Steps

Cleanup

Key Takeaways

Troubleshooting

Next Steps

Example Repository

Airbyte Documentation

Helm Configuration

Advanced Patterns

​Overview

​What You’ll Deploy

​Prerequisites

​File Structure

​variables.tf

​main.tf

​airbyte-values.yaml

​Deployment Steps

​Cleanup

​Key Takeaways

​Troubleshooting

​Next Steps

Example Repository

Airbyte Documentation

Helm Configuration

Advanced Patterns

Overview

What You’ll Deploy

Prerequisites

File Structure

variables.tf

main.tf

airbyte-values.yaml

Deployment Steps

Cleanup

Key Takeaways

Troubleshooting

Next Steps