How do you design and implement a custom, complex web-based email system that must be robust, scalable across dozens of machines, and support two million users at the start? The small size of your entire staff is a handicap. Everything must be tested to the rigorous standards of telecom giant in Asia which overlooks very, very few details.
I'll describe one way to do it, using Erlang at its heart for:
* Communicating with the Web browser client
* Speaking SMTP, POP, and IMAP to the rest of the email world
* Interfacing with a proprietary, external text index/search and user authentication services
* Gluing C++ components for speed-critical and legacy feature support
* Using a dynamically scalable key-value DB to store everything.
All email, user profile, and vCard data is stored in a
distributed key-value database that supports dynamic online resizing as the service grows while keeping email stored safely and consistently.
We make heavy use of protocol contracts (using UBF) to enforce over-the-wire sanity of all intra-system communication and with the browser. It's tested extensively with EUnit, QuickCheck, and Python scripts. I'll give details of what has worked well and what we should have done differently through each stage of design, implementation,
testing, and deployment.