The big difference is that a big team is capable of achieving many things that a small team is not. Either through sheer amount of work they are capable of doing, or through the fact that they represent more skills than small team can have.
The benefit of adding people to a team is rarely that it improves your efficiency: All things being equal, a small team will outperform a big team at a task they are both capable of achieving.
The main thing I think about communication is how expensive it is, especially synchronous communication.
And this isn't "meetings are expensive". Literally all communication is expensive. It takes up a lot of time, and unless you're very careful about your team topology that time grows quadratically in the number of people in the team.
I think a lot about communication, especially as I'm currently working in a team of size one.