A2A

The most important thing is your understanding of what you are building 
(purpose, shape, scale...). The architecture for a product which will 
change a lot during its developement should be very different from an 
architecture tailored to well-defined specifications. The second most 
important thing is the team that will be building it. I will try to 
illustrate with two examples.

Case 1: you are the technical co-founder of a young company. You do not 
know your market very well, and frankly you are not sure which features 
your customers want or don't want, but you need to start working on a 
product. You need something you can show early investors and prospects, 
and use to validate assumptions about your business.

In those circumstances, what you should optimize for is short-term 
speed of iteration. If you want to change something or add a feature, 
you should be able to do it fast.

You should look for a solution where you won't have to care about 
hosting or administration much, something you can just put on Heroku or 
that VPS of yours in a few minutes, and which lets you easily deploy 
several times per day. You should go with a monolithic architecture 
(still properly modularized though), and whatever language you know 
best, provided it is fit for rapid web development (has enough Open 
Source libraries, etc). You can either choose SQL if you have 
significant experience with it or a NoSQL database for the flexibility 
(PostgreSQL being a good middle ground).

Case 2: the product from case 1 has been successful, it now has a 
million users. The technical team has grown to 40 people. You are 
having issues:

- There are performance problems and it looks like your architecture 
will crumble at the next order of magnitude in growth.

- You have moved off your PaaS for cost reasons. You have several 
servers and an "operations team" now; they complain about how hard 
their job is.

- Your developers all work on the same codebase and find it 
problematic. Nobody is clearly responsible for anything. There are 
parts of the codebase noone even knows about.

- That database schema is becoming a monster which nobody fully 
understands anymore.

You decide to got for a rewrite. This is a case where I would say SOA 
makes a lot of sense: break the monolith into separate blocks with 
well-defined interfaces. They should be independently testable, and you 
should organize your team into teams around those services.

The services can be in different programming languages if they have 
different requirements. There can be several simpler databases instead 
of a large one. Speed of iteration is still a goal (it almost always 
is) but no longer the only one.

You should aim for:

- independently scalable parts;
- easier operations and debugging in production;
- stability, redundancy, reliable backups...

Finally, I would like to add three things:

- Like Lon Hosford said, always keep said security (and auditability 
when you reach a certain scale) in mind.

- I do not necessarily advocate going from step 1 to step 2 by means 
of a rewrite. You can (and should) try to evolve from case 1 to case 2 
gradually. But sometimes people don't do that, they let the system from 
case 1 grow too much until it is too late, and then it is better to 
start from a clean sheet.

- If you have not seen this video yet, you should watch it: 
http://www.youtube.com/watch?v=WpkDN78P884