Ludovic Alarcon

Ludovic Alarcon .

Kubernetes and Cloud technologies enthusiast, DevOps believer - Golang and Dotnet Developer

Oracle Cloud + Kubeadm


Oracle cloud offers a pretty good free offering with ARM based compute.

Ampere A1 Compute instances (Arm processor): All tenancies get the first 3,000 OCPU hours and 18,000 GB hours per month for free for VM instances using the VM.Standard.A1.Flex shape, which has an Arm processor. For Always Free tenancies, this is equivalent to 4 OCPUs and 24 GB of memory.

I got inspired by the post on how to setup a always free k3s cluster on oracle cloud write by zwindler.

As the machines are quite good (especially for a free tier), I wanted to deploy a full kubernetes cluster and not a lightweight one.

I decide to go with Kubeadm to setup the cluster with Containerd for container runtime and Flannel for the network.
The cluster will be composed of a control plane and two workers defined as follow:

Actually at first I wanted to use Cilium but I had network issues with workers node.
Then I fallback to Calico but got also issues with network.
So I finally used Flannel but I don’t give up on using Cilium :)

Create resources on Oracle Cloud

This article is not a deep dive into Oracle Cloud, so I will quickly show what to create.

First we will need to create a Virtual Cloud Networks (VCN).

A Virtual Cloud Network is a virtual private network that you set up in Oracle data centers. It closely resembles a traditional network, with firewall rules and specific types of communication gateways that you can choose to use.

Go to Networking -> Virtual Cloud Networks -> Start VCN Wizard
Add a name and keep the default values

Now we will create our instances Go to Compute -> Instances -> Create Instance

The last thing we need to do before starting the setup of our instance, is to open the connectivity on the firewall.
Basically by default only ssh on port 22 is opened.
As this is for testing purpose and not a production cluster, we will just open all port for TCP and UDP protocol for internal IPs (
For production I highly recommand to open individually the ports required
We will open the port 80 on every host ( to be able to expose http traffic later.

Go to Networking -> Virtual Cloud Networks -> your VCN -> Subnets -> public-subnet

Then go to Security Lists -> Default Security List

Adn finally to Ingress Rules -> Add Ingress Rules

Setup our instances

Common configuration

This is the common configuration that should be apply to all nodes (controlplane, worker-01 and worker-02).
First we will ssh on the machine.



I was really surprise, the default configuration of iptables made by Oracle is pretty strict and basically drop everything.
Their default security rules are quite good. So first, we will need to edit iptables rules file to open some ports.

sudo vi /etc/iptables/rules.v4

Let’s add our rules after the line for SSH
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT

-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 6443 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 2379 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 2380 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 10250 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 10251 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 10252 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --match multiport --dports 30000:32767 -j ACCEPT

And delete this two lines

-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited

Now, we need to run the following command to take into account our iptables change

sudo iptables-restore < /etc/iptables/rules.v4


Now we open all the ports we need, we will setup and install Containerd.

Swap should be disabled in order for kubelet to work properly.
Since Kubernetes 1.8, a kubelet flag fail-on-swap has been set to true by default, that means swap is not supported by default.
More information on the Why, there is a great article

# disable swap
sudo sed -i "/ swap / s/^/#/" /etc/fstab
sudo swapoff -a

We also need to let IPtables see bridged traffic

sudo modprobe overlay
sudo modprobe br_netfilter

# Configure IPTables to see bridged traffic
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf

cat <<EOF | sudo tee /etc/sysctl.d/k8s-cri-containerd.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1

# Apply sysctl without reboot
sudo sysctl --system

First we will install Containerd as a service

# Install containerd as a service
curl -LO
sudo tar Cxzvf /usr/local containerd-1.7.2-linux-arm64.tar.gz
curl -LO
sudo cp containerd.service /lib/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now containerd
# Install runc
curl -LO
sudo install -m 755 runc.arm64 /usr/local/sbin/runc
curl -LO
sudo mkdir -p /opt/cni/bin
sudo tar Cxzvf /opt/cni/bin cni-plugins-linux-arm64-v1.3.0.tgz

Now we will generate the default configuration for containerd and enable Cgroup

# Create containerd configuration
sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml

# Enable Cgroup
sudo sed -i 's/            SystemdCgroup = false/            SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd

Kubeadm, Kubelet and Kubectl

Then, we will install Kubeadm, Kubelet and Kubectl with version 1.27.3.

# Adding source for kubectl, kubeadm and kubelet
curl -s | sudo apt-key add -
echo "deb kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update

# Install tools
sudo apt-get install -y kubelet=$KUBERNETES_VERSION \
                        kubeadm=$KUBERNETES_VERSION \
sudo apt-mark hold kubelet=$KUBERNETES_VERSION kubeadm=$KUBERNETES_VERSION kubectl=$KUBERNETES_VERSION

Control plane

On the control plane, we will run the kubeadm init command.
We will use the following CIDR for the pod network needed by Flannel CNI that will be installed later on.

CONTROL_PLANE_IP="10.0.0.XX" # Private IP of control plane
POD_CIDR="" # Flannel default values
sudo kubeadm init --pod-network-cidr=$POD_CIDR --apiserver-advertise-address=$CONTROL_PLANE_IP --apiserver-cert-extra-sans=$CONTROL_PLANE_IP --node-name=$(hostname -s)

Note the kubeadm join command line given at the end of the kubeadm init output, we will use it right after.

Worker node

We will run the previously retrieve kubeadm join command on every worker node.

kubeadm join ...

Control plane next step

First let’s retrieve the kubeconfig file to be able to interact with our fresh next cluster.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

If we execute the get node command we can see that all our nodes are in a NotReady state.
No worries, it’s perfectly normal, we first need to install a CNI.

kubectl get no

It’s time to install Flannel

kubectl apply -f

When all pods are running in the kube-flannel namespace, all your nodes will have a Ready state.

kubectl get no
NAME           STATUS   ROLES           AGE     VERSION
controlplane   Ready    control-plane   1h      v1.27.3
worker-01      Ready    worker          1h      v1.27.3
worker-02      Ready    worker          1h      v1.27.3

To ensure that everything is working properly, we will run a dnsutils pod and try the dns resolution.

kubectl apply -f

Once the pod is running

kubectl exec -it dnsutils -- nslookup kubernetes.default

Ok, so now, we will install metrics-server.

curl -OL
sed -i "/--metric-resolution/a\        - --kubelet-insecure-tls" components.yaml
kubectl apply -f components.yaml
rm components.yaml

We can now use the following command to ensure metrics-server is running

kubectl top no
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
controlplane   70m          3%     1874Mi          15%
worker-01      28m          2%     1612Mi          27%
worker-02      21m          2%     1463Mi          25%

We now have a basic kubernetes setup in Oracle Cloud ARM VMs but we cannot expose http traffic.
That’s not really great…

Expose traffic


According to their documentation

MetalLB is a load-balancer implementation for bare metal Kubernetes clusters, using standard routing protocols.

MetalLB supports both BGP and Layer2 configuration. We will setup a basic Layer2 configuration with IPv4.

kubectl apply -f

Now we need to configure our IPAddressPool for MetalLB.

kind: IPAddressPool
  name: default-pool
  namespace: metallb-system
kind: L2Advertisement
  name: default
  namespace: metallb-system
  - default-pool

Nginx Ingress Controller

We will use Nginx as Ingress Controller.

kubectl apply -f

Thanks to metalLB, we should now have an external-ip of for the loadbalancer service of nginx.

kubectl get svc -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer    80:30325/TCP,443:32653/TCP   2h
ingress-nginx-controller-admission   ClusterIP     <none>        443/TCP                      2h

We will now deploy a sample application to ensure everything is working properly.

apiVersion: apps/v1
kind: Deployment
  name: whoami-deployment
      app: whoami
  replicas: 2
        app: whoami
      - name: whoami
        image: containous/whoami
        - containerPort: 80
apiVersion: v1
kind: Service
  creationTimestamp: null
    app: whoami
  name: whoami
  - port: 80
    protocol: TCP
    targetPort: 80
    app: whoami
kind: Ingress
  name: whoami
  ingressClassName: nginx
  - http:
      - path: /whoami
        pathType: Prefix
            name: whoami
              number: 80

And we should be able to curl it from our controlplane or workers node


Great ! But the external-ip is a private ip… it’s not very useful…
We will then install nginx as reverse proxy to be able to expose traffic from public internet!
It’s possible to do NAT routing with iptables but I prefer have a reverse proxy.

sudo apt install nginx
sudo vi /etc/nginx/sites-enabled/default

Replace the location / block by the following

location / {

And then reload nginx

sudo systemctl reload nginx

We are now able to query our whoami from public internet using the public ip of our VMs.

Oracle Load Balancer

The free tier also have a Load Balancer, let’s setup one to target all our nodes reverse-proxy.
I’m using the whoami app as health check prob for the LB.
Go to Networking -> Load Balancers -> Create a Load Balancers

After few minutes, your LB will be ready.


We now have a Kubernetes cluster ready on the free tier of Oracle Cloud.
It’s a nice playground for testing everything Kubernetes related.

Related articles:

I hope this was useful!