Explorations of Varnish
For the last couple days, I’ve been working on a Sinatra-based web service for resizing images. You can hit the service with an image URL and a desired size, and it uses the high-speed image library VIPS to convert it to the appropriate size. To cache images, the Populr team decided to use Varnish. Varnish makes it painless to cache HTTP responses by proxying requests to your web service and returning cached data when it’s available.
EngineYard provides a Chef script for configuring Varnish that automatically configures it based on the size of your instance. However, when I activated the Chef script and started a deployment, the instance wouldn’t spin up. The /var/log/syslog on the machine showed that Varnish was failing to spawn a child process:
1 2 3 4
Jun 28 12:21:28 varnishd: Pushing vcls failed: CLI communication error (hdr) Jun 28 12:21:28 varnishd: Child (12521) died signal=16 Jun 28 12:21:28 varnishd: Child (-1) said Jun 28 12:21:28 varnishd: Child (-1) said Child starts
It turns out, Varnish is built with 64bit architectures in mind and the default settings assume a 64bit stack. On his blog, Kristian Lyngstol noted:
Varnish works on 32-bit, but was designed for 64bit. It’s all about virtual memory: Things like stack size suddenly matter on 32bit. If you must use Varnish on 32-bit, you’re somewhat on your own. However, try to fit it within 2GB. I wouldn’t recommend a cache larger than 1GB, and no more than a few hundred threads… (Why are you on 32bit again?)
Antonio Carpentieri wrote a great blog post on trying to start Varnish on a 32-bit Amazon instance, and found that the problem is the sess_workspace configuration parameter. It’s default value 262144 (256k) is too large to fit on a 32-bit stack. He suggests setting it to only 19264 (16k) and also changing the thread_pool_stack to 64k to prevent problems with Varnish trying to start child processes.
When using the Varnish Chef recipe, these changes need to be made in varnishd.monitrc.erb, which contains the commands monit uses to start and stop Varnish. The configuration parameters are conveniently passed as arguments to the start command, so it’s easy to edit them.
1 2 3 4
check process varnish_80 with pidfile /var/run/varnish.80.pid start program = "/usr/sbin/varnishd -a :<%= @varnish_port %> -T 127.0.0.1:6082 -s <%= @cache %> -f /etc/varnish/app.vcl -P /var/run/varnish.80.pid -u nobody -g nobody -p obj_workspace=4096 -p sess_workspace=262144 -p listen_depth=2048 -p overflow_max=<%= @overflow_max %> -p ping_interval=2 -p log_hashstring=off -h classic,5000009 -p thread_pool_max=<%= @thread_pool_max %> -p lru_interval=60 -p esi_syntax=0x00000003 -p sess_timeout=10 -p thread_pools=<%= @thread_pools %> -p thread_pool_min=100 -p shm_workspace=32768 -p thread_pool_add_delay=1" stop program = "/usr/bin/pkill -KILL varnish"
Rather than pursue these changes and handicap Varnish on a 32-bit server, we decided to create a custom build of the VIPS static library for EngineYard’s 64-bit high CPU instances. This turned out to be a bit of a hassle, but EngineYard’s default Varnish Chef recipe ran without a hitch on the 64 bit machine.