Production Story: Cloud Armour — Saviour

Lot of our application is on GCP Cloud. We have seen lot of attack on our application. Each time we see new pattern. As a SRE personal we are ready to mitigate all such.

Attacker had plan to use DDOS attack & get us down. We were clueless of the plan.

Day started with a call on the service that GLB error rates are high. We observed 5xx, our user authentication service going down.

GLB Error

We observed backend istio is giving low success rate for same application. The request rates also increased manyfold.

Istio Success Rate

It was hitting application badly, We took help from developer to understand problem & identified a legitimate user call process is kept on loop by attacker. Our application cannot handle the load & was restarting. We can observe that with tomcat threads.

By noon, we were able to block user with application logic. That had helped application threads to stabalise. however attacker is yet able to come to our application layer & hog/waste lot of resources. We needed to defend it at first layer from below diagram.

Architecture Diagram

We observe that attacker has simple strategy of running jmeter with huge load on the application url. GLB is capable of handling DDOS attack, however it do not provide layer 7 security. Hence we reached out to Cloud Armor. Below diagram shows that Cloud Armor works on network layer with backend services defination of GLB.

Evening we found second phase of attackers action that leads us to setup cloud armor with below simple steps. We blocked traffic from attacker’s IP

#Create a Policy
gcloud compute --project=<project-id> security-policies \
         create block-user-attack \
         --description=block-user-attack
#Block IP or IP ranges upto 10
gcloud compute --project=<project-id> security-policies rules \
         create 100 --action=deny\(403\) \
         --security-policy=block-user-attack \
         --src-ip-ranges=<attacker ip>/32
#Default allow all traffic
gcloud compute --project=<project-id> security-policies rules \
         create 2147483647 --action=allow \
         --security-policy=block-user-attack \
         --description="Default rule,higher priority overrides it" \
         --src-ip-ranges=\*
#Assign Policy to Backend Services
gcloud compute --project=<project-id> backend-services \
          update um-istio-beckendservice \
          --security-policy=block-user-attack

We were relaxed as attacker’s traffic was blocked at GLB level & we can see istio failure rates are under control in below image.

We say bye bye to attacker

This is our production story. For our SRE team, Cloud Armor came as super hero.

2020 being interesting year, I have started my story with Thanos & Ending with Armor.

Happy New year everyone.

Reference:

Configuring Google Cloud Armor security policies