Whatsapp Backend Scalability

This is a very old talk, talking about even older things that are no longer in use.

Although, to answer your question, there is no problem with (semi-automated) expansion done through re-partitioning (re-sharding):

  1. Start with “double writes” when all writes are going to both “old” and “new” partition
  2. Run a crawler that reads all data from all “old” partitions and writes into “new” partitions
  3. Switch reads to the “new” partitioning schema and stop double-writing.
  4. Remove old partitioning.

This method is usually applied to split a single shard in a consistent hash ring, but nothing prevents you from using it at scale to re-shard everything at once.

2 Likes