When to Scale?

You know that you need to make significant changes to be able to scale sometime in the future.
Your service is serving thousands of users and the number is growing. You also have plans for wonderful features that will only bring more and more users to your service. Your business development are also telling you of a great partner (Microsoft/Google/Twitter...), who is going to sign a partnership with you and multiply your user base 10 times at least. The future is bright, if only you could scale your service in time.

Theoretical measurement of module cost/performance

Don't Scale too early

Why couldn't you build the system to scale in the first place?
This is usually the delicate balance between "Building it right" and "Building the right 'it'" (if I may borrow the terms from Pretotyping Manifesto). You could either spend your time in laying the grounds for future possible scale and risking over architecture, or you could focus on testing your ideas in a "quick-and-dirty" way. If you are successful, I can bet that you use the latter, therefore, you need to scale now.
If you scaled too early, the risk is that you spent too much time ("Initial Cost" in the diagram above) and now you have to wait longer and be much more successful to get to the "Break Even" point. I see many start up companies who are coming to raise VC funding "to finish the development", while VC prefer to put their money "to boost the marketing and sales".
Even if you could build the system to scale from the beginning, you shouldn't!

Don't Scale too late

Why couldn't you scale the service at the right time?
If you take a look at the diagram above, you can understand that taking the right time to work on scale is not simple. At the beginning of the "Change Decision Range", your system is performing well and scale in almost linear. The more efforts you put in the system (Software Development, Performance Tuning, Hardware Addition...), the better results you receive. This is certainly not the time to spend on rewriting your system for scale, or is it?
You do start to see signs of problems; some features are taking too much time to implement, some errors are causing the system to fail, some upgrades are not going smoothly as they used to be. But everything is manageable more or less, and rather quickly these problems are resolved. These are system smells, if I can borrow the concept of code smell from Martin Fowler.

The following facts are causing you to make the scale decision too late:

• The slow gradual degradation of your architecture is building your confidence that you are still on the right track, as you are getting better in solving these problems
• The boost in the business is putting more pressure on "keeping the lights on"
• The focus on "keeping the lights on" is preventing you from working on deep rewrites
• The slow gradual trend change from linear growth to plateau and to linear decline is difficult to detect
• The more you invest in the current technology, the harder it gets to decide to switch to a new technology. This is true for managers ("How can I come and say that I was wrong before") and for developers ("I'm a professional Java developer, why should I learn Scala?"). 

You could scale the system at the right time, but it is very difficult!

Scale Right

How can you know when it is the best time to scale?
First, measure!
Work on measuring your effort spending (development, optimization, hardware...) vs. your benefit received (request number, response time, accuracy...).
Second, visualize!
Put your measurement in front of your eyes, and the eyes of your peers. A diagram similar to the one above should be reviewed every week.
Third, realize!
You can't fight reality, although it is very easy to ignore it. Realize that the linear growing curve will turn flat and then down. Know that you are going to invest in a new technology, that will return its investment in the "Break Even" point only after considerable time period.
Fourth, decide!
When you see the signs that you hit that point in the curve, start working on making the needed technology changes and rewriting of your great performing system.