Ad Serving in Erlang

Bob IppolitoMochi Media, Inc.

Erlang Factory Bay Area 2009 - Palo Alto

May 1, 2009

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 1 / 23

Author: Bob Ippolito

Date: May 2009

Venue: Erlang Factory Bay Area 2009

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 2 / 23

What’s MochiAds?

MochiAds:

Monetization platform for Flash game ecosystem

Advertising solution for game developers

Revenue share and distribution for publishers

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 2 / 23

Road to Erlang

MochiBot:

(Originally) Python w/ Twisted

Fast, but not fast enough (CPU bound)

Wanted easy multi-node distribution

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 3 / 23

Previous Experience

Python:

Non-blocking sockets are tedious

Threads are too heavy

C/C++:

Too low-level

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 4 / 23

Why Erlang?

Performance

Concurrency

Distribution

Fault Tolerance

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 5 / 23

(Bad) Benchmarks

ab -c 50 -n 10000 localhost

Apache: 1x

Twisted: 1.12x

mochiweb: 2.5x

nginx: 3.9x

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 6 / 23

Erlang at Mochi Media

MochiAds, MochiBot

High-performance HTTP servers

Ad targeting

Real-time analytics

Social gaming

Lots of internal use

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 7 / 23

MochiAds Service

Front-end

Data warehouse

Ad server

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 8 / 23

What’s Not Erlang

Front-end:

Python and PostgreSQL

Data warehouse:

Python and Vertica

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 9 / 23

Ad Serving Platform

Juniper Routers

Cisco Switches

OpenBSD load balancer

Nginx HTTP load balancer (Linux)

Erlang/OTP R12B-3 (Linux)

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 10 / 23

Ad Server Stack

Erlang/OTP R12B-3

mochiweb (http)

egeoip (geolocation)

eswf (SWF file format)

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 11 / 23

Ad Server

Gather information about client

Choose an ad

Log impression data

Log click data, redirection

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 12 / 23

Gathering Client Info

Mostly client-side Flash code

SharedObject (like cookies)

Feeds targeting decisions for ad choice

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 13 / 23

Choose an ad

Fold over in-memory data structure

Filter out ads that don’t match targeting info

Weight the rest

Choose a random number; 0 <= N < sum(Weights)

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 14 / 23

Log impression data

Validation

Stream to hourly disk log

Increment counters in RAM db

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 15 / 23

Log click data

Validation

Stream to hourly disk log

Increment counters in RAM db

Redirect (HTTP 302) to destination URL

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 16 / 23

Cheap Tricks

Ad request data in URL

Long-term state in SharedObject

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 17 / 23

Short-term Feedback Loop

Client state

RAM db counters from previous serves that day

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 18 / 23

Analytics Feedback Loop

ETL processes adjust per hour

Campaign weights, budget adjustments, etc.

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 19 / 23

Lessons Learned (part 1)

Network partitioning sucks

Network partitioning sucks

Network partitioning sucks

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 20 / 23

Lessons Learned (part 2)

pg2 is broken

mnesia is too slow (for us)

Inter-node distribution protocol can be flaky

Erlang open source not always robust

Lists are not a good data type for strings

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 21 / 23

Favorite Erlang Features

Module reloading

Pattern matching

Binaries

Lightweight processes

Concise but not cryptic

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 22 / 23

More Erlang at Mochi Media

MochiBot

MochiCrypt

MochiScore

IRC bot

SVN deployment

Monitoring system

Node discovery

Bob Ippolito (Erlang Factory Bay Area 2009) Ad Serving in Erlang May 1, 2009 23 / 23