For me it depends on how "isolated" the installation will be.
If it has fast and easy access to the Internet, then I'll just stick with the default Raspian install and add other items as necessary.
If Internet access will be slow, iffy, or infrequent, then I'll load up everything and the kitchen sink "just in case".
I haven't run into any storage space, RAM, or performance issues that force me to economize, though.