Proxy Systems

I just ran into an error trying to use SquirrelMail behind a Varnish proxy because apparently SquirrelMail was setting too many duplicate headers:

   14 RxProtocol   b HTTP/1.0
   14 RxStatus     b 302
   14 RxResponse   b Found
   14 RxHeader     b Date: Tue, 01 Jun 2010 17:12:19 GMT
   14 RxHeader     b Server: Apache
   14 RxHeader     b X-Powered-By: PHP/5.3.2-1
   14 RxHeader     b Expires: Thu, 19 Nov 1981 08:52:00 GMT
   14 RxHeader     b Cache-Control: public
   14 RxHeader     b Pragma: no-cache
   14 RxHeader     b Set-Cookie: SQMSESSID=mp24q63lol8gjjvsdtus6qfdi7; path=/a/webmail/; HttpOnly
   14 RxHeader     b Set-Cookie: SQMSESSID=mp24q63lol8gjjvsdtus6qfdi7; path=/a/webmail/; HttpOnly
   14 RxHeader     b Set-Cookie: SQMSESSID=mp24q63lol8gjjvsdtus6qfdi7; path=/a/webmail/; HttpOnly
   14 RxHeader     b Set-Cookie: SQMSESSID=mp24q63lol8gjjvsdtus6qfdi7; path=/a/webmail/; HttpOnly
   14 RxHeader     b Set-Cookie: SQMSESSID=mp24q63lol8gjjvsdtus6qfdi7; path=/a/webmail/; HttpOnly
   14 RxHeader     b Set-Cookie: squirrelmail_language=en_US; expires=Thu, 01-Jul-2010 17:12:19 GMT; path=/a/webmail/; HttpOnly
   14 RxHeader     b Set-Cookie: SQMSESSID=bhtr9skc1e2jsdv33pfsl9daj5; path=/a/webmail/
   14 RxHeader     b Set-Cookie: SQMSESSID=bhtr9skc1e2jsdv33pfsl9daj5; path=/a/webmail/; HttpOnly
   14 RxHeader     b Set-Cookie: SQMSESSID=bhtr9skc1e2jsdv33pfsl9daj5; path=/a/webmail/; HttpOnly
   14 RxHeader     b Set-Cookie: SQMSESSID=bhtr9skc1e2jsdv33pfsl9daj5; path=/a/webmail/; HttpOnly
   14 RxHeader     b Set-Cookie: SQMSESSID=bhtr9skc1e2jsdv33pfsl9daj5; path=/a/webmail/; HttpOnly
   14 RxHeader     b Set-Cookie: SQMSESSID=bhtr9skc1e2jsdv33pfsl9daj5; path=/a/webmail/; HttpOnly
   14 RxHeader     b Set-Cookie: key=YdO4%2FQ%3D%3D; path=/a/webmail/; HttpOnly
   14 RxHeader     b Set-Cookie: SQMSESSID=bhtr9skc1e2jsdv33pfsl9daj5; path=/a/webmail/; HttpOnly
   14 RxHeader     b Set-Cookie: SQMSESSID=bhtr9skc1e2jsdv33pfsl9daj5; path=/a/webmail/; HttpOnly
   14 RxHeader     b Set-Cookie: SQMSESSID=bhtr9skc1e2jsdv33pfsl9daj5; path=/a/webmail/; HttpOnly
   14 RxHeader     b Set-Cookie: SQMSESSID=bhtr9skc1e2jsdv33pfsl9daj5; path=/a/webmail/; HttpOnly
   14 RxHeader     b Set-Cookie: SQMSESSID=bhtr9skc1e2jsdv33pfsl9daj5; path=/a/webmail/; HttpOnly
   14 RxHeader     b Set-Cookie: SQMSESSID=bhtr9skc1e2jsdv33pfsl9daj5; path=/a/webmail/; HttpOnly
   14 RxHeader     b Set-Cookie: SQMSESSID=bhtr9skc1e2jsdv33pfsl9daj5; path=/a/webmail/; HttpOnly
   14 RxHeader     b Set-Cookie: SQMSESSID=bhtr9skc1e2jsdv33pfsl9daj5; path=/a/webmail/; HttpOnly
   14 LostHeader   b Set-Cookie: SQMSESSID=bhtr9skc1e2jsdv33pfsl9daj5; path=/a/webmail/; HttpOnly
   14 HttpGarbage  b HTTP/1.0
   12 FetchError   c http format error
   14 BackendClose b pro2glight
   12 VCL_call     c error

I was able to confirm this thanks to this post by Robin Clarke about switching ISPs without any downtime.

I use NGINX in front of Varnish so I simply proxied requests to squirrelmail around Varnish, to yet another proxy server, lighttpd, which does more host-specific routing.

Also noteworthy - I hadn't used lighttpd that much at all before now, but I've certainly dabbled with it. I don't think its proxy capabilities are as extensive as NGINX, but it does have mod_magnet with built in Lua scripting, which is really awesome. I've even wrote up a Lua script to load a JSON hash of hosts to redirect - I'm definitely planning on extending that. I sort of wish they used more standard interfaces though, something along the lines of Rack or wsapi with app, env, and so on - instead they use the lighty object as the interface.

Lighttpd also has dynamically loaded modules and compression caching, something which NGINX does not have (though NGINX has gzip precompression, but its not a standard module on Debian - my OS of choice).

I finally managed to setup grace on Varnish so that it works when a backend server goes down AND when the backend does not have polling setup.

I think it does require polling to be setup on a fake server though. I got the idea from Alex Mizrahi's email to the Linpro Varnish mailing list:

serving stale content during grace period works fine if backend polling can detect that server is down. however in some cases polling thinks that server is fine, but it returns 50x error for some requests (e.g. when database server is overloaded), and it might be better if stale content would be served. (by the way, that's how grace works in nginx web server's cache.)

as i understand, varnish does not support this case now. nevertheless, i've tried to hack something with vcl, using example with restarts as inspiration. i've made a nonexistant backend for this purposes, with polling enabled:

backend nonexistant {
        .host = "localhost";
        .port = "31337";
        .probe = {
           .interval = 10s;
           .timeout = 0.3 s;
           .window = 2;
           .threshold = 1;
        }
}

then it vcl_fetch, when it sees status 500, it calls restart:

sub vcl_fetch {
        if (obj.status == 500) {
                restart;
        }
}

and vcl_recv sets backend to nonexistant in case of restart:

sub vcl_recv {
        if (req.restarts == 0) {
                set req.grace = 2m;
                set req.backend = apache;
        } else {
                set req.grace = 2m;
                set req.backend = nonexistant;
        }
}

Except on my server, when Varnish would try to fetch, the missing backend would send it right to error, and it would never get the chance to restart, resulting in a 503 response (yes, I tried changing the 500 in the example to 503).

I tried putting the restart block in my VCL error code, but no luck. Varnish reported an error. I did some digging on the idea of putting restart into the error vcl and learned that the functionality was added this past year. I upgraded my varnish version to 2.0.5 and it now works. Rock on!

This is what I use in the sub vcl_error block:

    if(req.restarts < 1) {
        restart;
    }

Hopefully I won't get too many errors, but regardless of what causes the error, Varnish will try one more time before it gives up.

Here are a few Squid version 3 configuration notes I've take lately.

Starting Squid HTTP Proxy 3.0: squid32009/12/15 11:01:38| WARNING: '127.0.0.1' is a subnetwork of '127.0.0.1'
2009/12/15 11:01:38| WARNING: because of this '127.0.0.1' is ignored to keep splay tree searching predictable
2009/12/15 11:01:38| WARNING: You should probably remove '127.0.0.1' from the ACL named 'localhost'

To avoid this log warning, I've removed the localhost settings from the configuration. If I find any problems, I'll adjust it. UPDATE: Yes, this setting is required for accessing squid from the localhost. I changed 127.0.0.1 to 127.0.0.2 and its fine now. Its not the best fix, but it quiets squid3 a bit.

I also edited the negative ttl for http responses of "not found", and similar:

negative_ttl 10 seconds

It defaults to five minutes! That seems illogical to me, but perhaps that's what most systems use.

I recently read up about Unicorn, a web server written in Ruby. The README file states the following:

Unicorn is a HTTP server for Rack applications designed to only serve fast clients on low-latency, high-bandwidth connections and take advantage of features in Unix/Unix-like kernels. Slow clients should only be served by placing a reverse proxy capable of fully buffering both the the request and response in between Unicorn and slow clients.

This is a very clear and concise explanation of on example of how and when HTTP proxies might be used.

Everything I've read about Unicorn has been well put, and since there is more to read - I'm on it! :-)