Building Cloud-Native Security for Apps and APIs with NGINX

Stepan Ilyin, Co-founder, Wallarm twitter, linkedin
abstract slides video

NGINX is a good place to implement protections for several reasons: traffic in many environments is already passed through NGINX reverse-proxies, the Kubernetes Ingress Controller is based on NGINX, and it can be easily deployed as a side-car for a microservice.

Chapter 1. APIs and microservices security with NGINX

WAF for APIs/Microservices in Kubernetes

In NGINX, you can enable ModSecurity with:

location / {
  ModSecurityEnabled on;
  ModSecurityConfig modsecurity.conf;
  proxy_pass http://localhost:8011;
  proxy_read_timeout 180s; 
}

For an the ingress controller, enable the flag in the configmap:

apiVersion: v1
kind: ConfigMap
metadata:  
  name: nginx-configuration-external  n
  amespace: ingress-nginx
data:  
  enable-modsecurity: "true"  
  enable-owasp-modsecurity-crs: "true"

Enabling ModSecurity in Kubernetes:

metadata:  
  annotations:    
    nginx.ingress.kubernetes.io/configuration-snippet: |
      modsecurity_rules '
        SecRuleEngine On
        SecAuditLog /var/log/modsec/audit.log 
        SecAuditLogParts ABCIJDEFHZ
        SecAuditEngine RelevantOnly
      ';

Building a security dashboard to gain visibility of malicious traffic

By enabling ModSecurity we’re now blocking some malicious requests, but it’d be nice to have some insight into what malicious traffic we’re seeing. Here’s how to build a dashboard with open source tools.

Fluentd can be run as a sidecar in the ingress-nginx pod, and sharing a volume mount between ingress-nginx and Fluentd allows Fluentd to access the ModSecurity logs.

Dashboards are nice, but you don’t want to have to always be watching them. Instead, we’d like automatic alerts to ensure we don’t miss anything.

Yelp open sourced ElastAlert, a simple framework for alerting on anomalies, spikes, or other interesting patterns of data in Elasticsearch.

Alerts can be sent to a number of external systems, such as e-mail, Jira, OpsGenie / VictorOps / PagerDuty, chat apps (HipChat / Slack / Telegram / Gitter), and Twilio.

Mirroring traffic for async analysis with 3rd party tools

When you’re integrating a new service into your production traffic flow, it’s crucial to make sure you don’t break things. This is often a concern with tools like WAFs, whose false positives may block legitimate user traffic.

NGINX has a mirror mode in which it sends every request to an additional backend (e.g. a WAF), which enables you to determine the effects of the new tool you’re considering while ensuring that nothing breaks.

location / {
  mirror /mirror;
  proxy_pass http://backend;
}  

location /mirror {
  internal;
  proxy_pass http://mirror_backend$request_uri;  
}

WAFs for NGINX

ModSecurity

ModSecurity is an open source WAF that can be efficient in detecting and blocking common web app security attacks (e.g. OWASP Top 10). It supports virtual patching and is quite customizable.

Challenges:

• Default rulesets tend to result in a huge number of false positives.

• As with any signature-based tool, it requires tuning, which can be challenging.

• Signatures can’t block every attack, a creative attacker can bypass them.

Best practice:

• Monitoring mode: Use the public ruleset

• Blocking mode: craft rules from scratch sspecifically for your apps and API

To learn more, see the ModSecurity Handbook. Christian Folini has a nice blog post on how to tune your WAF installation to reduce false positives

Naxsi

Naxsi doesn’t rely on complex signatures but instead looks for “dangerous” characters and expressions. It uses a small set of simple scoring rules containing 99% of known patterns involved in web vulnerabilities. For example, <, |, and drop are generally not supposed to be part of a URI.

Each time any of the characters or expressions in an incoming HTTP request match one of the rules, Naxsi increases the “score” of the request (SQLi, XSS, …). If a request’s score is above a threshold, the request is blocked.

Sometimes these patterns may match legitimate queries, so you need to configure whitelists to avoid false positives.

• Pros: Naxsi is resistant to many WAF-bypass techniques

• Cons: You need to use LearningMode with every significant code deployment.

Repsheet (behaviour based security)

Repsheet is a reputation engine designed to help manage threats against web apps. It uses a behavior-based approach to detect malicious and fraudulent activity, using different resources to score incoming requests: HTTP header info, GeoIP, ModSecurity alerts, external feeds, and any custom rules you’ve set up.

Protecting against Bots and DDoS

Use the test_cookie module

Testcookie-nginx acts as a filter between bots and the backend using L7 DDoS attacks, helping you to filter out junk requests. This works because HTTP-flooding bots are often dumb and lack browser features such as HTTP cookies and supporting redirects.

Testcookie-nginx works by doing a series of checks to determine if the requesting client supports cookies, JavaScript, and Flash, and if it can perform HTTP redirects.

• Pro: Blocks many (dumb) bots and prevents automatic scraping

• Con: Cuts out the Google bot and does not protect against sophisticated bots, for example, those that use a full browser stack.

Analyze web server logs for anomalies

You can use AI to detect bots using a neural network (e.g. PyBrain), and determine which requests during a DDoS are legitimate. A Dropbox engineer wrote a PoC of this approach here. The access.log file before the DDoS attack is useful for NN/ML training because it should list mostly your legitimate clients.

Block traffic from data centers, cloud / hosting providers and Tor

You could use GeoIP to ban all traffic from a country or area using [ngx_http_geoip_module](https://nginx.org/en/docs/http/ngx_http_geoip_module.html). This tends to be a bad practice, as GeoIP data may not be accurate.

You could block the IP ranges of popular cloud providers, as requests from their ranges may be more likely to be bots. Each platform lists their IP ranges: GCP, AWS, Azure.

You could also block Tor exit nodes, proxies / anonymizers / etc. using a commercial feed like MaxMind, or specific malicious IPs using services like Project HoneyPot.

Disable resource-intensive application endpoints

L7 DDoS attacks often look for resource-intensive endpoints, such as /search, that can be queried in a way to make the server do a lot of computation and/or use up memory.

NGINX supports a custom HTTP code 444 that allows you to close the connection and return nothing in the response.

location /search {
  return 444;
}

Limit buffer, timeouts, connections in NGINX

If you’re under a DDoS, there are several NGINX directives you can tweak:

• Limiting buffers: client_header_buffer_size, large_client_header_buffers, client_body_buffer_size, client_max_body_size

• Timeouts: reset_timeout_connection, client_header_timeout, client_body_timeout, keepalive_timeout, send_timeout