Using Proxy Protocol With Nginx

linux 2014-03-20

We all know and dearly love Nginx. One very common way to use it is to put it in front of some other application server such as Tomcat, Node, or Tornado as a reverse proxy. About a year ago, Nginx got the ability to proxy WebSockets connections to a backend server that supports them.

This is great, but there can be an issue if your application nodes are behind a load balancer such as an AWS ELB. An ELB can work in one of two modes. It can load balance HTTP/HTTPS traffic specifically, or it can load balance straight TCP traffic without regard to the protocol. If you use an HTTP (Layer 7) load balancer, the ELB injects an X-Forwarded-For header that has the original client IP address. However, this kind of load balancer does not support WebSockets. On the other hand, if you use a TCP load balancer, your application can use WebSockets, but it won’t get an HTTP header with the client IP, because the load balancer doesn’t know anything about HTTP headers in this case. So your application won’t know the IP of the client actually connecting to it, which you obviously might want.

The folks at the HAProxy Project developed the Proxy Protocol to get around this limitation. Amazon added this feature to ELBs in the summer of 2013. And, as of the 1.5.12 release, Nginx has support for it as well!

There’s not much in the way of documentation or examples for this feature yet, so I thought I would walk you through how to configure and use it.

Set up a node and an ELB

I’ll assume that you’re using a reasonably new release of Ubuntu Linux for your node (12.04 or newer). Spin up the instance, making sure you can SSH into it and that it accepts traffic on port 80, and then install Nginx from my PPA:

sudo add-apt-repository -y install ppa:chris-lea/nginx-devel
sudo apt-get update
sudo apt-get -y install nginx-full

Next, set up an ELB from the AWS web console. Make sure that it is a TCP load balancer that accepts traffic on port 80 and forwards it to port 80. Specify that the health check is a ping to port 80 on the end node, and then put your instance in the ELB’s pool.

Enable Proxy Protocol on the ELB

This is a somewhat annoying process because you can’t currently do it from the web console. (Please join me in asking Amazon to get this feature added.) Fortunately, there is a very detailed tutorial in the AWS ELB Developer Guide that explains how to install and use the command line ELB tools needed. They do a better job of explaining the procedure than I can, so I’ll assume you’ve gone through that HOWTO and successfully enabled proxy protocol support on your ELB before going on to the next step.

Configure Nginx for Proxy Protocol

When you use proxy protocol support in Nginx, it defines a $proxy_protocol_addr variable that you can use for things like logging. Open up the file:

/etc/nginx/nginx.conf

and define a new log format by adding this line before the inclusion of the virtual hots info:

log_format elb_log '$proxy_protocol_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent "$http_referer" ' '"$http_user_agent"';

Next, open up

/etc/nginx/sites-enabled/default

You need to modify the listen directive, as well as set things up with the realip module. Here’s a bare bones (but functional) configuration:

server {
    listen 80 proxy_protocol;
    root /usr/share/nginx/www;
    index index.html index.htm;
 
    # Make site accessible from http://localhost/
    server_name localhost;
    set_real_ip_from 172.31.0.0/20;
    real_ip_header proxy_protocol;
 
    access_log /var/log/nginx/elb-access.log elb_log;
 
    location / {
         try_files $uri $uri/ /index.html;
     }
}

There are four things to take note of here.

We’ve added proxy_protocol to the listen directive.
We’ve set set_real_ip_from to the CIDR range of addresses that our ELB could be using.
We’ve added proxy_protocol to the real_ip_header directive.
We’re using the elb_log format for the access_log which we previously defined.

And that’s it! If everything is working correctly, and you visit the URL that AWS provides for the ELB, then you should see something roughly like this in /var/log/nginx/elb-access.log:

96.251.49.3 - - [20/Mar/2014:03:50:47 +0000] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1897.2 Safari/537.36"

Here, 96.251.49.3 is the IP address of the client device that connected to the ELB. So your app now has knowledge of this information to make use of in whatever way is needed.

Please note that technically, you don’t have to set set_real_ip_from and real_ip_header in order to use the $proxy_protocol_addr variable, but it’s still best to define those so Nginx has the maximum amount of information to work with.

Please leave questions or comments below.

19 Comments

Gordon Rankin 11 years ago Reply

Thanks for the guide… I’m trying to get SPDY to work through ELB. I have successfully installed the proxy_protocol policy on my ELB but am unsure how I go about setting up the NGINX part. It works fine without the ELB.

Since SPDY requires an SSL connection I ave set up my ELB to use SSL / 443 rather than TCP / 80 as in your guide.

And normally in my nginx config I have the line:

listen 443 ssl spdy

Which again works fine when access directly… How would I change this to listen to the proxy_protocol and allow SPDY to operate behind the ELB.

Thanks in advance for any help.
Max Mautner 10 years ago Reply

It sounds like you left out the last part–which is “what do you do in your application code to keep websockets alive (where you previously couldn’t) now that you have clients’ IP addresses?”
- chris lea Post Author 10 years ago Reply
  
  I’m just here to talk about Nginx. What you choose to do in your applications is all up to you. 🙂
  - mmautner 10 years ago Reply
    
    Appreciated! This blog post is the best I’ve found about getting websockets working from behind ELBs. Tremendously helpful 🙂
Ankur Agarwal 10 years ago Reply

I am facing a 400 Bad Request error in the nginx after enabling the Proxy protocol in AWS Elb. I have enabled tcp for handing websockets connections. Any suggestions ?.
- chris lea Post Author 10 years ago Reply
  
  That’s not really enough info to make much of a guess. But in cases like this, I usually start by running strace on the Nginx processes to see what’s going on. This post may be helpful to you:
  
  https://chrislea.com/2013/04/22/using-strace-on-multiple-processes-in-linux/
  - Ankur Agarwal 10 years ago Reply
    
    Thanks, here is what I get in the trace. Still not sure whats going on.
    
    [pid 26033] epoll_wait(11, {}, 512, 500) = 0
    [pid 26033] gettimeofday({1408691676, 848974}, NULL) = 0
    [pid 26033] epoll_wait(11, {{EPOLLIN|EPOLLOUT, {u32=30330400, u64=30330400}}}, 512, 500) = 1
    [pid 26033] gettimeofday({1408691677, 147443}, NULL) = 0
    [pid 26033] recvfrom(8, “GET /splash.html HTTP/1.1rnhost:”…, 1024, 0, NULL, NULL) = 122
    [pid 26033] getsockopt(9, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
    [pid 26033] writev(9, [{“GET /splash.html HTTP/1.1rnHost:”…, 154}], 1) = 154
    [pid 26033] recvfrom(8, 0x7fff879b90bf, 1, 2, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
    [pid 26033] epoll_wait(11, {{EPOLLIN|EPOLLOUT, {u32=30330608, u64=30330608}}}, 512, 500) = 1
    [pid 26033] gettimeofday({1408691677, 151519}, NULL) = 0
    [pid 26033] recvfrom(9, “HTTP/1.1 200 OKrnX-Powered-By: E”…, 4096, 0, NULL, NULL) = 4096
    [pid 26033] readv(9, [{“”security-container”>ntt<div cla"…, 4096}], 1) = 1868
    [pid 26033] readv(9, 0x7fff879b8ee0, 1) = -1 EAGAIN (Resource temporarily unavailable)
    [pid 26033] writev(8, [{"HTTP/1.1 200 OKrnServer: nginxrn"…, 333}, {"nntt<div cla"…, 1868}], 3) = 6002
    [pid 26033] write(6, "172.31.3.100 – – [22/Aug/2014:07"…, 115) = 115
    [pid 26033] recvfrom(8, 0x1cd0570, 1024, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
    [pid 26033] epoll_wait(11, {}, 512, 500) = 0
    [pid 26033] gettimeofday({1408691677, 654861}, NULL) = 0
    [pid 26033] epoll_wait(11, {}, 512, 500) = 0
    [pid 26033] gettimeofday({1408691678, 156123}, NULL) = 0
    [pid 26033] epoll_wait(11, {}, 512, 500) = 0
    [pid 26033] gettimeofday({1408691678, 657386}, NULL) = 0
    [pid 26033] epoll_wait(11, {}, 512, 500) = 0
    [pid 26033] gettimeofday({1408691679, 158632}, NULL) = 0
    [pid 26033] epoll_wait(11, {}, 512, 500) = 0
    [pid 26033] gettimeofday({1408691679, 659922}, NULL) = 0
    [pid 26033] epoll_wait(11,
    [pid 26032] {{EPOLLIN, {u32=30329984, u64=30329984}}}, 512, -1) = 1
    [pid 26032] gettimeofday({1408691679, 769891}, NULL) = 0
    [pid 26032] accept4(7, {sa_family=AF_INET, sin_port=htons(58934), sin_addr=inet_addr(“172.31.3.100”)}, [16], SOCK_NONBLOCK) = 5
    [pid 26032] epoll_ctl(9, EPOLL_CTL_ADD, 5, {EPOLLIN|EPOLLRDHUP|EPOLLET, {u32=30332480, u64=30332480}}) = 0
    [pid 26032] epoll_wait(9, {{EPOLLIN, {u32=30332480, u64=30332480}}}, 512, 60000) = 1
    [pid 26032] gettimeofday({1408691679, 771214}, NULL) = 0
    [pid 26032] recvfrom(5, “PROXY TCP4 103.5.132.66 10.0.19.”…, 1024, 0, NULL, NULL) = 566
    [pid 26032] writev(5, [{“HTTP/1.1 400 Bad RequestrnServer”…, 145}, {“rn400 Bad Req”…, 120}, {“nginxrn</bo"…, 46}], 3) = 311
    [pid 26032] shutdown(5, SHUT_WR) = 0
    [pid 26032] recvfrom(5, 0x7fff879b8000, 4096, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
    [pid 26032] epoll_wait(9, {{EPOLLIN, {u32=30329984, u64=30329984}}}, 512, 5000) = 1
    [pid 26032] gettimeofday({1408691679, 858024}, NULL) = 0
    [pid 26032] accept4(7, {sa_family=AF_INET, sin_port=htons(58935), sin_addr=inet_addr("172.31.3.100")}, [16], SOCK_NONBLOCK) = 11
    [pid 26032] epoll_ctl(9, EPOLL_CTL_ADD, 11, {EPOLLIN|EPOLLRDHUP|EPOLLET, {u32=30331441, u64=30331441}}) = 0
    [pid 26032] epoll_wait(9, {{EPOLLIN, {u32=30331441, u64=30331441}}}, 512, 4913) = 1
    [pid 26032] gettimeofday({1408691679, 859352}, NULL) = 0
    [pid 26032] recvfrom(11, "PROXY TCP4 103.5.132.66 10.0.19."…, 1024, 0, NULL, NULL) = 45
    [pid 26032] writev(11, [{"HTTP/1.1 400 Bad RequestrnServer"…, 145}, {"rn400 Bad Req”…, 120}, {“nginxrn</bo"…, 46}], 3) = 311
    [pid 26032] shutdown(11, SHUT_WR) = 0
    [pid 26032] recvfrom(11, 0x7fff879b8000, 4096, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
    [pid 26032] epoll_wait(9, {{EPOLLIN|EPOLLHUP|EPOLLRDHUP, {u32=30332480, u64=30332480}}}, 512, 4912) = 1
    [pid 26032] gettimeofday({1408691680, 72435}, NULL) = 0
    [pid 26032] recvfrom(5, "", 4096, 0, NULL, NULL) = 0
    [pid 26032] write(6, "172.31.3.100 – – [22/Aug/2014:07"…, 112) = 112
    [pid 26032] close(5) = 0
    [pid 26032] epoll_wait(9,
    [pid 26033] {}, 512, 500) = 0
Gaurav Purandare 10 years ago Reply

What would be the tweak required to make proxy_protocol work on ports other than 80, say 8080:

listen 8080 proxy_protocol;

After trying this, it gives “No data received”. (Works fine if proxy_protocol is not mentioned.)

Any clues?
- Gaurav Purandare 10 years ago Reply
  
  Sorry wrote it too soon. The proxy protocol on ELB was enabled only for backend server port 80, had to enable it again for other port if needed, say 8080 etc.
Pingback: WebSockets on AWS’s ELB | raw engineering
javier 10 years ago Reply

Hi Chris,

I’m facing an issue. I get nginx working with properly headers, I’m plugging this nginx in front of the same instance which serves node.js application.
For achive that I use proxy pass, but I’m getting all the time
No ‘Access-Control-Allow-Origin’ header is present on the requested resource. Origin

and If I add Allow-Control-Allow-Origin header It gets twice (This is related to put a reverse proxy and elb with proxy protocol I guess).

Could you give me some hint, or post your entire server listening for port 81.

Thanks in advance.

Here is my cfg:

server {

listen 80 proxy_protocol;
# root /usr/share/nginx/html;
# index index.html index.htm;

# Make site accessible from http://localhost/
server_name xxxxx.com;
set_real_ip_from 172.31.0.0/16;
real_ip_header proxy_protocol;
access_log /var/log/nginx/elb-access.log elb_log;

location / {

if ($request_method = ‘OPTIONS’) {
add_header ‘Access-Control-Max-Age’ 1728000;
add_header ‘Access-Control-Allow-Methods’ ‘GET, POST, OPTIONS’;
add_header ‘Access-Control-Allow-Headers’ ‘Authorization,Content-Type,Accept,Origin,User-Agent,DNT,Cache-Control,X-Mx-ReqToken,Keep-Alive,X-Requested-With,If-Modified-Since’;

add_header ‘Content-Length’ 0;
add_header ‘Content-Type’ ‘text/plain charset=UTF-8′;
return 204;
}

proxy_pass http://127.0.0.1:3000/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection “upgrade”;
# proxy_set_header Host $host;
# proxy_cache_bypass $http_upgrade;

}
}
Josh 10 years ago Reply

I’ve followed all the directions, but I’m seeing this now in my nginx logs:

while reading PROXY protocol, client: 10.0.18.131, server: 0.0.0.0:80
2015/01/02 10:40:24 [error] 9899#0: *20 broken header: “GET /contact HTTP/1.1
host: 10.0.28.87
User-Agent: ELB-HealthChecker/1.0
Accept: */*
Connection: keep-alive

From what I can tell, the policy is live on port 80 on my ELB…
- Josh 10 years ago Reply
  
  This seems to be occurring because ELB HTTP healthchecks don’t use the proxy protocol. This is a problem, as a ping healthcheck isn’t good enough for us and may succeed even if a node should be taken out of rotation for some other reason (network connectivity problem with internal resources, etc.)
  
  See the AWS forum here: https://forums.aws.amazon.com/thread.jspa?messageID=592631&#592631
  - Josh 9 years ago Reply
    
    good news is, I was just informed by my AWS account rep that ELB is now officially supported as a HIPAA compliant AWS solution, so this no longer applies.
Pingback: Получаем IP-адреса HTTPS-клиентов с HAProxy (frontend) на Nginx (backend) в режимах HTTP и TCP-балансировки | ITFM
Scott 10 years ago Reply

Hi Chris,

Thanks for the super helpful notes! To anyone else coming here, this workaround is also needed if for some reason you have to terminate SSL not at the ELB but the EC2 instance (e.g. for proper HIPPA compliance on EC2).

In this case the ELB must pass raw TCP on 443 to instance port 443 and the AWS CLI command for setting the EnableProxyProtocol policy should be something like:

$ aws elb set-load-balancer-policies-for-backend-server
–load-balancer-name [BALANCERNAME]
–instance-port 443
–policy-names EnableProxyProtocol [OTHER POLICIES]

Again, thanks for these notes!
Pingback: Получаем IP-адреса HTTPS-клиентов с HAProxy (frontend) на Nginx (backend) в режимах HTTP и TCP-балансировки | FNIT.RU
Pingback: WebSockets on AWS’s ELB | Blog | builtio
Gold Collar Soutions 8 years ago Reply

Hi Chris..

I am getting a strange scenario on using this nginx proxy. When I directly hit a node.js app running in AWS under port 1212 from browser, I am getting the response. But when I do the same proxy (from public DNS) it works on Browser, but not on any Mobile App using fetch() API.

It is throwing recvfrom failed: ECONNRESET (Connection Reset by peer). What way nginx treats differently and make it fail?

my nginx conf is like this

server {
listen 80;
server_name localhost;
location /backend/ {
proxy_pass http://localhost:1212/;
proxy_redirect off;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
proxy_set_header X-NginX-Proxy true;
proxy_set_header Connection “”;
proxy_http_version 1.1;
}

}