... or at least that's what they say.

There are multiple non-trivial steps required to enable it, and under the hood Azure is also performing some implicit configuration. It is easy to make a mistake in the configuration, which breaks connectivity. There are also some limitations, which could only be learned the hard way.

So, what is required to configure a Private Endpoint? And what each step actually imply?

Step 1: Create a Private Endpoint

This automatically creates a Network Interface Card (NIC) and connects it to the desired VNet. The NIC is assigned with a private IP from the range of the VNet.

💥 Important side effect: when the first Private Endpoint is created for a resource, then Azure automatically amends records in the public DNS.

🔗 DNS resolution before creating a Private Endpoint:

1. Resource hostname → CNAME record for the Service Endpoint (hostname of the actual hosting server, e.g., blob.ams23prdstr16a.store.core.windows.net).
2. Service Endpoint -> A record for the public IP of the hosting server.

🔗 DNS resolution after creating a Private Endpoint:

1. Resource hostname → CNAME record for privatelink.{resource hostname}.
2. privatelink.{resource hostname} -> CNAME record for the Service Endpoint.
3. Service Endpoint -> A record for the public IP.

⚠️ At this point, the target resource is already reachable via the private IP, but the hostname of the resource does not resolve to it yet.

Step 2: Create a Private DNS Zone

The name of the Private DNS Zone must match the type of the target resource — check out the table in the documentation to find out the correct one.

There can be only one Private DNS Zone of each type in the same Resource Group — as the name of the Zone is a part of the Resource ID.

Step 3: Link the Private DNS Zone to the desired VNet

One Private DNS Zone can be linked to multiple VNets.

One VNet can be linked only to one Private DNS Zone of each type.

Step 4: Register the Private DNS Zone under the Private Endpoint

... or, in Azure Portal terms: add the Private DNS Zone into a Private DNS Zone Group under the Private Endpoint.

💥 Important side effect: Azure automatically creates an A record in the Private DNS Zone for the private IP of the NIC of the Private Endpoint.

⚠️ It is impossible to add multiple Private DNS Zones of the same type into the same Private DNS Zone Group under a Private Endpoint (docs).

⚠️ It is impossible to add multiple Private DNS Zone Groups under a Private Endpoint (docs).

⚠️ It is possible to add only up to 5 Private DNS Zones into a Private DNS Zone Group under a Private Endpoint (docs).

Step 4 (alternative): Add A record manually

When any of the limitations above are hit, then the only way to enable the correct DNS resolution is to manually manipulate the Private DNS Zone.

It is possible to find out which private IP was assigned to the NIC of the Private Endpoint. Then, it is required to manually add an A record for this IP — thus, at last enabling the correct DNS resolution from inside the VNet.

🔗 DNS resolution after step 4 (see also an image in the docs):

1. (public DNS) Resource hostname → CNAME record for privatelink.{resource hostname}.
2. (private DNS)privatelink.{resource hostname} -> A record for the private IP.

The connectivity using the Private Endpoints is tricky, because it requires not only to connect resources to VNets, but also to establish the correct DNS resolution. For some of the steps, Azure is implicitly performing some configurations in the public global DNS — which is not trivial and is not really expected to happen. Then, there are other side effects and limitations of configuring the Private DNS Zone. At the end, it is still possible to link everything together and achieve connectivity which does not ever leave the backbone Azure networks.

• Create
• Destroy
• Drift Mitigation

Usually, when we want to cover some infrastructure with Terraform, we try to find an existing provider with a resource we need. For most usual cases (for major clouds, for most-used resources) we do find necessary implementations, but sometimes that happens, that we need either a not-so-popular resource, or something highly specialized, that no-one else bothered before us to it.

The normal way would be then to implement a custom Terraform provider. But what if we don't introduce Go into our codebase (as that is the only language the providers could be written with)? What if we don't want to set up the full-blown tool chain for development, including CI/CD? What if we also don't want to solve the trouble of publishing the resource into some registry (public registry, or a self-hosted one)?

As long as there are SDKs or tools that can create the needed resources for us, we can stay with the already known experiences and technologies, while using Terraform only for gluing everything together, so that we could manage the lifecycle of all resources (including their interdependencies) in one place.

Important: the new to-be-implemented custom resource should follow the declarative approach, and it should have characteristics of a resource. I.e., it should have clear lifecycle states, it should be possible to provision the resource, to destroy it, to read its current state (for drift detection). The resource may be virtual, or it may represent only a part of another resource. But as long as it complies with the Declarative approach (when we describe some desired state of resources – as opposed to the Imperative approach, when we write down the steps needed to achieve the desired state), it could be a good candidate for a custom resource.

So, to create a custom resource, we need to cover with some custom code the usual lifecycle states of a Terraform resource: provisioning (create), deprovisioning (destroy) and drift mitigation (refresh).

When using Terraform with Azure, we employ the recommended approach:

• At first, we try to find a definition for a needed resource in the AzureRM provider (implemented and maintained by Hashicorp).
• Then, if the needed resource is not available, we try to find it in AzAPI (a thin wrapper over Azure APIs, which is maintained by Microsoft).
• And we fall back to the solution of custom resources only when neither AzureRM not AzAPI support what we need.

Let's take as an example the task of managing the AppSettings of an Azure App Service separately from the App Service itself. For example, the need for it may arise in the following cases:

• When the App Service is created in one Terraform project, but its configuration is extended from another Terraform project (and there are some reasons to keep this separation).
• When there is a need to break a cyclic dependency: two App Services depend on each other, because they require some info from one another to complete their configuration.

Example: App Service 1 wants to call App Service 2, and we want to implement authentication using their SystemAssigned managed identities (which are created automatically during provisioning of the App Services). App Service 1 needs to know the principal_id of App Service 2, so it could request a token for it. App Service 2 in its turn needs to know principal_ids of all callers (in this example, App Service 1), so it could validate them.

The needed resource is not supported by AzureRM (there is only a closed feature-request). In AzAPI there is a page on the resource of AppSettings, but it seems to be auto-generated, and it also lacks examples of usage. Anyway, we still need an example to illustrate the approach, so we will proceed with creating a custom resource for it 😉

We will use this configuration of App Services (link to the repo with complete code, see at the end of the article).

# apps.tf
# Declarations of Resource Group and AppService Plan are skipped.

resource "azurerm_linux_web_app" "appService1" {
  name                = "app-service1"
  location            = azurerm_resource_group.resourceGroup.location
  resource_group_name = azurerm_resource_group.resourceGroup.name
  service_plan_id     = azurerm_service_plan.appServicePlan.id

  site_config {}

  identity {
    type = "SystemAssigned"
  }

  app_settings = {
    EXAMPLE_SETTING_1 = 42
  }
}

resource "azurerm_linux_web_app" "appService2" {
  name                = "app-service2"
  location            = azurerm_resource_group.resourceGroup.location
  resource_group_name = azurerm_resource_group.resourceGroup.name
  service_plan_id     = azurerm_service_plan.appServicePlan.id

  site_config {}

  identity {
    type = "SystemAssigned"
  }

  app_settings = {
    EXAMPLE_SETTING_2 = 43
  }
}

Create

For reusability of the created custom resource, we will introduce a module. First, we will create a script that performs the creation of necessary App Settings. We will use PowerShell which invokes Az CLI, but as said before, any tool or programming language can be used. It will be invoked by Terraform in the same way it can be invoked from terminal, so we only need to make sure that the necessary tooling is available on the host machine.

# additional-app-settings/assets/create.ps1

[CmdletBinding()]
param (
  [Parameter(Mandatory)] [string] ${subscription-id},
  [Parameter(Mandatory)] [string] ${resource-group-name},
  [Parameter(Mandatory)] [string] ${app-service-name},
  [Parameter(Mandatory)] [hashtable] ${app-settings}
)

$settings = (${app-settings}.Keys | ForEach-Object { "$($_)=$(${app-settings}[$_])" }) -join " "

az webapp config appsettings set `
  --subscription ${subscription-id} `
  --resource-group ${resource-group-name} `
  --name ${app-service-name} `
  --settings $settings

To include the script into Terraform lifecycle, we will use the fake built-in resource terraform_data with a custom provisioner.

# additional-app-settings/main.tf
# See link to the repo below for configuration of the module (providers and inputs).

locals {
  # The property id is marked as 'known after apply' during initial creation.
  # This avoids deadlocking the implemented custom refresh mechanism.
  # We parse the id to retrieve name and resource group name.
  appService        = provider::azurerm::parse_resource_id(var.appService.id)
  resourceGroupName = local.appService.resource_group_name
  appServiceName    = local.appService.resource_name
}

resource "terraform_data" "appSettings" {
  triggers_replace = {
    subscriptionId    = var.subscriptionId
    resourceGroupName = local.resourceGroupName
    appServiceName    = local.appServiceName
    appSettings       = var.appSettings
  }

  input = {
    subscriptionId    = var.subscriptionId
    resourceGroupName = local.resourceGroupName
    appServiceName    = local.appServiceName
    appSettings       = jsonencode(var.appSettings)
  }

  provisioner "local-exec" {
    when        = create
    interpreter = ["pwsh", "-Command"]
    command     = <<-EOT
      ${path.module}/assets/create.ps1 `
        -subscription-id $env:subscriptionId `
        -resource-group-name $env:resourceGroupName `
        -app-service-name $env:appServiceName `
        -app-settings ($env:appSettings | ConvertFrom-Json -AsHashtable)
    EOT
    environment = self.input
    quiet       = true # Silences printing of the invoked command. All other output is not silenced.
  }
}

Note the following:

• All inputs of the module are added into triggers_replace, which will make sure that any changes of the parameters are noticed and reconciled (although this is achieved via recreation).
• The properties interpreter and command together fulfil the task of invocation of custom code. If your script is written in Bash, you may use ["/bin/bash", "-c"].
• Parameters to the invoked script are passed via properties input and environment. This will become relevant for the destroy-time provisioner (explained below).

Let's instantiate the created module and check out it works.

# app-settings.tf

module "appService1AppSettings" {
  source = "./additional-app-settings"

  subscriptionId = data.azurerm_client_config.current.subscription_id
  appService     = azurerm_linux_web_app.appService1
  appSettings = {
    CALLEE = azurerm_linux_web_app.appService2.identity[0].principal_id
  }
}

module "appService2AppSettings" {
  source = "./additional-app-settings"

  subscriptionId = data.azurerm_client_config.current.subscription_id
  appService     = azurerm_linux_web_app.appService2
  appSettings = {
    CALLER = azurerm_linux_web_app.appService1.identity[0].principal_id
  }
}

When we try to invoke terraform apply at this point, we will see that it works – the necessary app settings are created successfully. But if we try to run it once again, we will see that Terraform detected them as a drift and wants to remove them:

  # azurerm_linux_web_app.appService1 will be updated in-place
  ~ resource "azurerm_linux_web_app" "appService1" {
      ~ app_settings                                   = {
          - "CALLEE"           = "808d076e-0d68-45e6-80aa-d7e194ddaed6" -> null
            # (1 unchanged element hidden)
        }
        # (28 unchanged attributes hidden)

        # (2 unchanged blocks hidden)
    }

  # azurerm_linux_web_app.appService2 will be updated in-place
  ~ resource "azurerm_linux_web_app" "appService2" {
      ~ app_settings                                   = {
          - "CALLER"           = "384fe864-f61e-4335-bb1b-65198b89e872" -> null
            # (1 unchanged element hidden)
        }
        # (28 unchanged attributes hidden)

        # (2 unchanged blocks hidden)
    }    

To mitigate that, we need to add the following section to the declarations of App Services:

# apps.tf

resource "azurerm_linux_web_app" "appService1" {
  ...
  
  lifecycle {
    ignore_changes = [app_settings["CALLEE"]]
  }
}

resource "azurerm_linux_web_app" "appService2" {
  ...
  
  lifecycle {
    ignore_changes = [app_settings["CALLER"]]
  }
}

Destroy

Deprovisioning phase will be covered by another script which is to be invoked by a destroy-time provisioner.

# additional-app-settings/assets/destroy.ps1

[CmdletBinding()]
param (
  [Parameter(Mandatory)] [string] ${subscription-id},
  [Parameter(Mandatory)] [string] ${resource-group-name},
  [Parameter(Mandatory)] [string] ${app-service-name},
  [Parameter(Mandatory)] [hashtable] ${app-settings}
)

$settings = ${app-settings}.Keys -join " "

az webapp config appsettings delete `
  --subscription ${subscription-id} `
  --resource-group ${resource-group-name} `
  --name ${app-service-name} `
  --setting-names $settings  

# additional-app-settings/main.tf

resource "terraform_data" "appSettings" {
  ...

  input = {
    subscriptionId    = var.subscriptionId
    resourceGroupName = local.resourceGroupName
    appServiceName    = local.appServiceName
    appSettings       = jsonencode(var.appSettings)
  }

  ...

  provisioner "local-exec" {
    when        = destroy
    interpreter = ["pwsh", "-Command"]
    command     = <<-EOT
      ${path.module}/assets/destroy.ps1 `
        -subscription-id $env:subscriptionId `
        -resource-group-name $env:resourceGroupName `
        -app-service-name $env:appServiceName `
        -app-settings ($env:appSettings | ConvertFrom-Json -AsHashtable)
    EOT
    environment = self.input
    quiet       = true
    on_failure  = continue
  }
}

The destroy-time provisioner imposes some differences as compared to the create-time provisioner:

• The destroy-time provisioner cannot reference any local variables, input parameters or other resources. Instead, they always use the captured state of the existing (just-to-be-destroyed) resource. Thus, we capture all the necessary values in the available property input, and then access them in the provisioner block via the special self object.
• We use environment to inject the values into the script. With this, we follow the recommendation against the code injection attack.
• The property environment expects the type map(string). When we need to pass some complex object (in our case – a map of key-values pairs of AppSettings), we need to serialize it before passing and deserialize it in the script (thus the invocations of jsonencode() and ConvertFrom-Json).
• And we also don't want to be too strict in case of possible failures during the destruction of the resource. Supporting all cases which could go wrong is tricky (maybe the App Service itself has been deleted - we don't want to cause a complete deadlock of Terraform), thus, we relax the requirement of successfulness with on_failure = continue.

We can now test if the destroy-time provisioner works:

terraform apply -replace module.appService1AppSettings.terraform_data.appSettings

Drift Mitigation

One of the powerful Terraform features is drift mitigation. Every resource of every normal provider implements a special Read method, which is invoked during the refresh phase.

Unfortunately, provisioners can be invoked only be of one of two types: create or destroy. There is no special provisioner type to hook into the refresh phase of terraform apply. For the drift mitigation, we will have to employ something else.

The implemented resource already reacts to changes of the input parameters (via triggers_replace). We need to add another 'synchronization pulse' to mark the resource for recreation based on external changes. Luckily, there is a provider pseudo-dynamic/value with a resource that implements exactly the capability we need.

At the first step, we need to read the current AppSettings of the App Service. We can achieve it with a data-resource.

# additional-app-settings/refresh.tf

data "azurerm_linux_web_app" "appService" {
  resource_group_name = local.resourceGroupName
  name                = local.appServiceName
}

Then, we need to find out, if the current AppSettings are in desired state (if all the necessary AppSettings are present and if the values are the same as we expect them to be).

# additional-app-settings/refresh.tf

locals {
  currentAppSettings = data.azurerm_linux_web_app.appService.app_settings

  areAppSettingsInDesiredState = alltrue([
    for desiredKey, desiredValue in var.appSettings :
    contains(keys(local.currentAppSettings), desiredKey) ?
    local.currentAppSettings[desiredKey] == desiredValue :
    false # We can't use the logical operator '&&' here due to a bug in short-circuiting.
  ])
}

Afterward, we will configure the reaction, when the AppSettings no longer appear to be in the desired state. We use the resource value_replaced_when for that.

# additional-app-settings/refresh.tf

resource "value_replaced_when" "driftDetected" {
  condition = !local.areAppSettingsInDesiredState
}

This resource is special: it expects only one boolean value as condition, and it produces a new random value every time when during an invocation of terraform apply the condition is false. Otherwise, it locks the previously produced value and does not change it. This fancy behavior then plays well with the property triggers_replace, as it causes recreation of the resource every time, when anything inside it changes.

# additional-app-settings/main.tf

resource "terraform_data" "appSettings" {
  triggers_replace = {
    ...
    driftDetectionTrigger = value_replaced_when.driftDetected.value
  }

  ...
}

With this, we made our custom resource to detect and react on any drift: be it someone accidentally changing an AppSetting, or even maliciously removing it.

Now, there is just one feature missing. When Terraform detects some drift during the refresh phase, it reports it in the log, so we could verify and explicitly approve it. We could achieve it with another fancy resource that prints custom warnings to the console logs.

# additional-app-settings/main.tf

data "validation_warnings" "appSettingsAreNotInDesiredState" {
  dynamic "warning" {
    for_each = var.appSettings
    iterator = each
    content {
      condition = !contains(keys(local.currentAppSettings), each.key)
      summary   = "AppSetting ${each.key} is not present, so it will be added"
    }
  }

  dynamic "warning" {
    for_each = var.appSettings
    iterator = each
    content {
      condition = (
        contains(keys(local.currentAppSettings), each.key) ?
        local.currentAppSettings[each.key] != each.value :
        false
      )
      summary = "AppSetting ${each.key} does not have desired value, so it will be updated"
    }
  }
}

The approach expressed above could be used as a pattern. It bridges the gap between Terraform and other tooling that is not available via some custom provider. It can be used instead of implementing some custom provider, which allows to stay in the technology stack already adopted by the team.

The approach plays nicely in the cases, which the concepts provisioning, deprovisioning and drift mitigation could be applied to. There are some things that one needs to know when using it, but in general some new case can be implemented with the pattern only once, and as long there is no need to change it drastically, it will continue to live (it is even resilient to external impact – which is covered by the drift mitigation).

The complete executable code could be found in this repo: https://github.com/egorshulga/terraform-custom-resource.

Upd. 2023: free domain zone .ga was taken over by the Gabonese government. It seems, all access to previously registered domains is lost.

Upd. 2024: free registrar Freenom stopped operations. All top-level domains ceased to exist.

Oracle Cloud offers really good terms in the Always Free tier. As of January 2022 it includes 4 CPUs and 24 GB of memory for ARM-based VMs.

There are 2 options: we can use free resources while staying on the Always Free tier, or we can upgrade to the Pay-as-You-Go subscription. There are some differences in available resources limits, that is why there will be differences in cluster architecture. This post describes approach for provisioning cluster in the Always Free tier.

N.B.: one won’t get charged in the Always Free tier, even after trial is over. One may get chargedafter upgrading to Pay-as-You-Go subscription.

Oracle Cloud has a resources of a managed K8s cluster, but unfortunately it is not available for Always Free tenancies (the limit is set to 0, all limits in this article are valid as of January 2022).

That means that for the Always Free tier we need to stick to the completely manual process of compute resources provisioning and K8s cluster deployment. So this post presents a way of provisioning a manually managed K8s cluster on Oracle Cloud ARM VMs. It describes architecture considerations, and also workarounds for issues that appeared on the road.

TL;DR: reproducible Terraform scripts and steps to get started could be found here.

Compute resources provisioning

We will be consuming all available compute resources in our cluster. We need a designated node for K8s control-plane (this will be a leader node), and multiple worker nodes. Each node will have 1 OCPU, which means that we can provision 1 leader node and 3 worker nodes. In our cluster leader node will have 3 GB of RAM, and each worker node will have 7 GB of RAM (making a total of 24 GB memory used).

N.B.: K8s issues a warning when a node has less then 2 CPUs. Our cluster is not a production one, and the goal is to maximize the number of nodes in the cluster. That is why we silence the error at K8s deployment.

The most often issue that Always Free tenancies face when provisioning VMs, is the Out of host capacity error.

The thing is that free compute resources resources are limited, and this error means that we got in the case when it has run out. Oracle says, that it is constantly adding capacity to its data centers, so we should just try another attempt in a couple of days.

Sometimes it helps to switch to another availability domain, if the region has it. Always Free tenancies can only provision resources in their home region only, that is why it should be selected carefully. A list of regions with appropriate availability domains could be found here.

At some point I also noticed, that if there are two accounts in the same region, but one of them has a Pay-as-You-Go subscription, and another does not, then the first one gets some priority in provisioning, so it was possible to provision ARM VMs, while the other one could not (it took another week to become available for the second account).

Network architecture

We need to provision a Virtual Cloud Network (VCN) to allow instances to connect to the internet, as well as become accessible from it. VCNs have subnets, which could be public or private. To open incoming and outgoing connectivity for resources in public subnets, a) VCN must have an Internet gateway, and b) each resource must have a public IP assigned. To open outgoing connectivity for resources in private subnets, VCN must have a NAT gateway. Incoming connectivity is initially unavailable.

So, desired network architecture will be as follows: VCN with 2 subnets, public and private, and compute resources are assigned to the private subnet. The trouble with this approach is that it works for Pay-as-You-Go tenancies only. As of January 2022 Oracle does not allow provisioning of NAT gateways in the Always Free tier, which leads to unavailable outgoing connectivity for nodes in private subnets.

To overcome this limitation, our VCN will have just a single public subnet, and in order to open outgoing connectivity, each compute resource will be assigned with a public IP. Luckily, Oracle does not limit availability of ephemeral public IPs.

Now each node becomes independently accessible from the internet (e.g. we can SSH to all of them). But we want to have as single efficient entry point for apps deployed to the cluster (as pods). We will achieve it by using a load balancer.

Load balancing

Oracle Cloud provides 2 types of load balancer. The first one works on the OSI level 7, which basically makes it a reverse proxy. E.g., it can handle SSL termination. But when we are creating a load balancer of this type, we need to select its shape. Load balancer shapes specify available bandwidth, and Always Free tenancies are eligible for a single 10 Mbps load balancer.

Another load balancer type is called Network Load Balancer (NLB). It works on OSI levels 3 and 4, and it can balance requests by IP-port pairs only. But this type does not have any specification for bandwidth limit, that is why we’ll use it in our cluster. We will put the NLB into the public subnet, and we will assign a reserved public IP, so it will become available from the Internet.

To enable load balancing, we also need to specify the following:

• Listeners, which represent ports that are available from the Internet
• Backend sets, which represent sets of resources the requests are balanced to.
• For each backend set we need to add appropriate backends, which are target links to compute resources.

In our case we’ll make the NLB listen to the following TCP ports:

• 80 — for HTTP traffic forwarded to worker nodes.
• 443 — for HTTPS traffic to worker nodes.
• 6443 — kubectl traffic to the leader node (for remote K8s management and apps deployment).

We also configure appropriate VCN ingress rules, to allow traffic to reach appropriate nodes.

K8s deployment

We use kubeadm to make a completely silent installation of K8s components. First we need to deploy a control plane. To support automatic joining of worker nodes, a) each node has private in-cluster DNS name, and b) we generate a discovery token (kubeadm token generate), which is copied from the leader node to all worker nodes. After that we invoke kubeadm init. After control plane is up, we can set up worker nodes with kubeadm join. We need to allow TCP port 10250 in the inner-cluster communication, because that’s a management port for kubelet (K8s agent running on each node).

K8s requires an overlay network plugin for the pods communications, and we’ll use Flannel for it. It works on a designated port on each node in the cluster (UDP 8472), that’s why we need to open this port in the VCN rules.

We will also deploy some useful infrastructure in the cluster. First, we need an ingress controller (we’ll use one based on nginx), which will be used for exposing web apps using a route-based approach. We’ll deploy a NodePort Service to listen on ports 30080 and 30443 for HTTP and HTTPS respectively (BTW, these are the ports that are registered as targets in NLB). With that said, we have complete network architecture in our cluster.

Using this ingress controller, we’ll deploy a dashboard. Once it is available, we can open it in browser: https://{cluster-public-ip}/dashboard.

We’ll also deploy a cert-manager, which helps with issuance of Let’sEncrypt HTTPS certificates. After its deployment is complete, we will deploy ClusterIssuer for Let’sEncrypt. There is a small peculiarity, as it takes some time for the cert-manager to become available, and until that attempts to create a ClusterIssuer will fail with a cryptic error, and we can’t know about cert-manager readiness via some K8s API call. That’s why we retry creation of ClusterIssuer until it succeeds (usually it takes a minute or so).

It works in conjunction with ingress-controller. To enable or, Ingress resource must be set up with appropriate public DNS name as a host.

Bonus: free public domain name

We can register a free domain name at the Freenom registrar. It is reserved for a year (after it elapses, we should manually prolong it, we can do it for free as well).

Once we have it, we can use configure the domain to target to the reserved public IP of the NLB. Go to Services - My Domains - Manage Domain - Manage Freenom DNS. We can add multiple 3rd level domains, and target it to the same public IP.

On the image above you can see an example. The intention is to make cluster-specific apps available under cluster subdomain, while regular apps are to become available under the domain itself. As now we have a public domain, we can issue a proper LetsEncrypt HTTPS certificate for the app. These rules are to be setup using Ingress resources in the cluster. That’s how it could be done for the dashboard.

N.B.: as it is free, we are left with almost no warranty. I registered a domain in the zone .ga, and to my surprise I found out that it was not available in some locations (precisely, it could not be resolved from the US West coast, from New Zealand, from Singapore). I wrote to .ga zone support, and after a couple of days the issue got resolved (I did not get any response though).

Domain in another zone did not have such issues.

So that’s how we can provision compute and network resources in the Oracle Cloud to deploy a K8s cluster with public IP and load balancing, while staying in the Always Free tier.