A proposal to do storage to volume assignment in Traffic Server

Sat Feb 23 03:12:10 UTC 2013
Justin Laue <justin at fp-x.com>
Jan van Doorn <jvd at knutsel.com>


We have two very specific use cases for ATS that we want to combine on a single server, but that seem to require contradicting settings. Both are adaptive bitrate fragmented video delivery using protocols like Apple HLS or Adobe HDS, but they difffer a lot in the requirements:
  1. Video On Demand (VOD) This traffic is fairly long-tail, and is helped by using longer TTL in the cache. We want to use spinning disks for this traffic.
  2. Live television This traffic is really short-tail, and is not helped by long TTL in cache, and in some cases is even prohibited from being active for a long time. We want to use RAM for this traffic.
We think there's a more general use case for admins to want to direct certain traffic to certain groups of devices in the storage.config file.

The patch

Introduces the volume assignment in storage.config, on an optional basis. No change is needed in the config if you don't want to use this feature. Raw file here, the patch is against master @ Feb 22. What it does:

Performance of the patch


In the next side-by-side sections, left is without the patch, and right is with the patch.

Maximum Bits/sec served until long TTS:
Maximum Trans/Sec until long TTS:
Unix Load at peak:
Cache HIT Ratio:

Configuration

storage.configstorage.config
# This is the only volume - 20TB in 24 900GB drives
/dev/sdb
/dev/sdc
/dev/sdd
/dev/sde
/dev/sdf
/dev/sdg
/dev/sdh
/dev/sdi
/dev/sdj
/dev/sdk
/dev/sdl
/dev/sdm
/dev/sdn
/dev/sdo
/dev/sdp
/dev/sdq
/dev/sdr
/dev/sds
/dev/sdt
/dev/sdu
/dev/sdv
/dev/sdw
/dev/sdx
/dev/sdy
# This is the disk volume - 20TB in 24 900GB drives
/dev/sdb volume=1
/dev/sdc volume=1
/dev/sdd volume=1
/dev/sde volume=1
/dev/sdf volume=1
/dev/sdg volume=1
/dev/sdh volume=1
/dev/sdi volume=1
/dev/sdj volume=1
/dev/sdk volume=1
/dev/sdl volume=1
/dev/sdm volume=1
/dev/sdn volume=1
/dev/sdo volume=1
/dev/sdp volume=1
/dev/sdq volume=1
/dev/sdr volume=1
/dev/sds volume=1
/dev/sdt volume=1
/dev/sdu volume=1
/dev/sdv volume=1
/dev/sdw volume=1
/dev/sdx volume=1
/dev/sdy volume=1

# This is the RAM volume - 64GB in 8 8GB 'drives'
/dev/ram0 volume=2
/dev/ram1 volume=2
/dev/ram2 volume=2
/dev/ram3 volume=2
/dev/ram4 volume=2
/dev/ram5 volume=2
/dev/ram6 volume=2
/dev/ram7 volume=2


volume.config


volume.config
volume=1 scheme=http size=50%
volume=2 scheme=http size=50%
# the size is irrelevant
volume=1 scheme=http size=1%
volume=2 scheme=http size=1%


hosting.config


hosting.config
hostname=ipcdn-live-01-19.cdnlab.com volume=2
hostname=ipcdn-live-02-19.cdnlab.com volume=2
hostname=* volume=1
hostname=ipcdn-live-01-19.cdnlab.com volume=2
hostname=ipcdn-live-02-19.cdnlab.com volume=2
hostname=* volume=1


records.config


records.config
Tried with both CONFIG proxy.config.cache.ram_cache.size INT -1 
and CONFIG proxy.config.cache.ram_cache.size INT 88080384, didn't seem to matter.
CONFIG proxy.config.cache.ram_cache.size INT -1


grub.config


grub.config
Standard grub.conf - no changes needed.
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.32-220.el6.x86_64)
	root (hd0,0)
		kernel /vmlinuz-2.6.32-220.el6.x86_64 ro root=UUID=b83d1c1f-d680-4d1a-8f1b-0765ba5d8b4f \
		rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb \
		crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM ramdisk_size=8388608
			initrd /initramfs-2.6.32-220.el6.x86_64.img

Testing

lll.pl is a script that will simulate a load of many clients to a cache, it is very light-weight, since it actually doesn't need a manifest to get "real" timecodes, but, it makes the timecodes up, and depends on mod_hotair to generate a fragment for any timecode it comes up with.

mod_hotair is an apache httpd module that gnerates HLS, HDS or Smooth fake fragments that are valid fragments for this transports, and have representative sizes for the requested bitrates, but carry no actual video. This way the module can easily "fill" a 10GE pipe on a medium server.

The test results above are from a test where half the requsted bitrates are from a 4.3Mbps simulated video and half of the fragments from a 3.0Mbps simulated video, Smooth transport (so seperate audio fragments @ 96kBps) and a fragment duration of 2 seconds. The test will simulate a load of 128 linear channels and a variable cache-hit ratio on the VOD load, displayed on the x-axis in the graphs. The Linear channels have a "DVR window" of 150 active fragments, or 5 minutes. The number of simultaneous clients is ramped up until 0.5% of the responses has a TTS of > 1sec (half the fragment play duration).

The server under test is a Dell Rx720d with 32 cores, 96G RAM, and 24 900GB cache disk drives, hooked up to the router with dual 10GE in an ethernet bonding configuration.