Integrating Istio and Kubermatics for nested cluster and RBAC

In the last blog we discussed how Istio can be used for multitenancy and RBAC for Kubernetes workloads and microservices. In this blog, we will discuss Istio and Kubermatics platform integration and its use-cases. 

We discussed with a lot of prospects and clients and we came across one problem to help an automobile giant to develop vehicle-2-vehicle communication using nested clustering. 

They wanted more control on their workloads and wanted closer isolation. 

Intro to Kubermatic platform 

Kubermatic Kubernetes Platform (KKP) is a Kubernetes management platform that helps address the operational and security challenges of enterprise customers seeking to run Kubernetes at scale. KKP automates deployment and operations of hundreds or thousands of Kubernetes clusters across hybrid-cloud, multi-cloud and edge environments while enabling DevOps teams with a self-service developer and operations portal.

Why Kubermatic platform is getting famous in enterprises

Well we all are exposed to multi-cloud and multi-cluster applications in Kubernetes. But there is a trade-off between security and manageability. You see, as the manageability and isolation of workspace increases with multiple clusters, security (authentication, authorization, SDLC policies, etc) becomes a big concern. To avoid that Kubermatic allows DevOps and architects to create nested clusters. 

As you can see in the below image, there are 3-levels to a nested cluster, where level-1 represents the Master cluster, level-2 represents two Seed clusters and level-3 represents two user-clusters.

Now such kind of complex setup can be achieved using Kubermatic Kubernetes Platform (KKP). The KKP allows to create nested clusters, where master controls the seed and there can be n-number of user clusters created under a seed cluster for computation. 

Three things to note:

1. User clusters can be used for deploying applications such as infotainment, TPMS, messaging apps, etc.
2. Visibility and governance can be done from the central plane (besides seed level). 

KKP also provides a range of integrations with cloud providers such as AWS, GCP, Azure, Red Hat Openstack, VMware vSphere, Digital Ocean, Alibaba Cloud, etc. to set up nested Kubernetes clusters.  

Source: https://docs.kubermatic.com/kubermatic/v2.23/architecture/

However there can be a few challenges of setting up a nested cluster (well, not a limitation to KKP). 

Limitation of Nested cluster and Multicluster approach

In the below diagram there are two architectures which are compared with each other:

1. N-Nested clusters: This is achieved using the Kubermatic platform. Enterprises can have multi-level requirements for supporting their governance and business process. 
2. Multicloud/Multiclusters with Istio: The ALMOST same-level of observability and granularity can be achieved from multicluster and multicloud set up using Istio. 

There are limitations of each of these approaches. First, the N-level nested cluster will be difficult to create and manage using the Kubermatic platform. Setting up unified observability across platform, apps, database and network can be cumbersome and costly. The most risky part is the achieving high-available or high performance feature- what-if the master cluster or any seed cluster, mother of some user clusters, goes down? 

Similarly, multicluster setup using Istio can be used to achieve the same level of isolation and multi tenancy, but those isolations are all logical. The risk in this case is that only experts can work with Istio to set up multicluster for large enterprises. Since there will be a side-car against each pod, the second risk is there will be additional resources consumed by the data plane (Envoy proxy) of Istio. If the DevOps team knows how to optimize Istio then it is well and great otherwise there will be performance issue in open source Istio for large scale systems. 

Integrating Istio and Kubermatic for RBAC for multicloud application

At IMESH, we have done a lot of research and POC in this area and came up with a conclusion to use an hybrid approach:

1. Let Kubermatic handle the nested cluster with only 3-level ( Master, Seed and users). There will be hard isolation of workspace using nesting that can be used for various use-cases like vehicle-to-vehicle communication or to operate multi-chain businesses easily. 
2. Istio can be used for observability from an enterprise perspective. And all the workload-to-workload authorization and authentication are done using Istio security policies. 

The advantage of this approach is there will be more flexibility to achieve RBAC and unified observability at workspace and global level. There is one disadvantage of this approach is the Istio optimization can be tricky if it is used for mTLS for large-scale systems. 

Next Step

To avoid all the limitations and use Istio in different business use-cases by integrating with various platforms like Kubermatic, talk to one of our Istio experts. 

IMESH helps large enterprises to simplify their network, achieve zero trust security and implement unified observability with open source Istio.