News

Microsoft Uncovers Large-Scale Cryptocurrency Mining Attack Against Kubernetes Clusters

• By David Ramel
• 07/07/2020

Kubeflow, an open source machine learning (ML) platform for working with Kubernetes, was the target of a recent cryptocurrency mining attack in which a malicious container was deployed on tens of Kubernetes clusters within two hours.

The large-scale attack was discovered by Microsoft's Azure Security Center (ASC) in June. The attackers exploited user misconfigurations in Kubeflow to abuse exposed Kubernetes dashboards for deploying cryptocurrency miners, explained Yossi Weizman, security research software engineer with the ASC, in a blog post. Kubeflow exposes its UI functionality via a dashboard that is deployed in the cluster. The dashboard is exposed by Istio ingress gateway, which is, by default, accessible only internally.

Istio is an open source independent service mesh that provides services needed to run a distributed microservice architecture. Some users modified the setting of the Istio Service to Load-Balancer, which exposes the Service (istio-ingressgateway in the namespace istio-system) to the Internet.

"We believe that some users chose to do it for convenience," Weizman said. "Without this action, accessing the dashboard requires tunneling through the Kubernetes API server and isn't direct. By exposing the Service to the Internet, users can access to the dashboard directly. However, this operation enables insecure access to the Kubeflow dashboard, which allows anyone to perform operations in Kubeflow, including deploying new containers in the cluster."

Since it was open-sourced at Kubecon USA in 2017, the Kubeflow Project has become a popular framework for running ML tasks in Kubernetes. The project evolved from an effort to open source the way Google ran its TensorFlow ML library internally, based on a pipeline called TensorFlow Extended.

The ASC earlier published a Kubernetes Threat Matrix detailing the major techniques that are relevant to container orchestration security, with a focus on Kubernetes. Weizman said the attack affected tens of Kubernetes clusters, but there was no information provided about how much mining was conducted or if attackers managed to do other nefarious deeds via the exposed dashboards.

Although the ASC has detected similar campaigns against Kubernetes implementations that leverage exposed services to the Internet as an access vector, this is the first Kubeflow-specific attack. Weizman described a similar exploit of Kubernetes in April.

User misconfigurations are a prime enabler of such attacks on Azure and other clouds. Notoriously, the Amazon Web Services (AWS) cloud suffered a long string of such attacks beginning a few years ago.

Weizman provided guidance on how organizations can check to see if their clusters are impacted and provided advice going forward, warning that they should do things like:

1. Authentication and access control to the application.
2. Monitor the public-facing endpoints of the cluster. Make sure that sensitive interfaces are not exposed to the Internet in an unsecure method. You can restrict public load balancers in the cluster by using Azure Policy, which now has integration with Gatekeeper.
3. Regularly monitor the runtime environment. This includes monitoring the running containers, their images and the processes that they run.
4. Allow deployments of only trusted images and scan your images for vulnerabilities. The allowed images in the cluster can be restricted by using Azure Policy.

Crypto mining (or cryptocurrency mining or bitcoin mining) is a way to generate digital currency wealth by leveraging powerful computing power. Although it's not illegal, it requires tremendous computing effort for usually minimal gains. The whole process is explained here.