Half of the decision is to go, the other half is who will go. The key here is to know your group. Do you do lots of activities that separate ages/genders or is everything open to everyone? If the group is always whole, a little separation could go a long way. If you already have groups within the church, maybe just one of them wants to go.
When deciding to go on a retreat, also part of it is what you want to accomplish. This could be a message or theme you want to embed in members. If you have the topic before picking the group, then figure out which one could benefit most from it. For example, talking about marriage to the entire group is wasted on the many who are widowed or not looking at marriage for many years. You could target this topic at a couples retreat. The main idea here is that everyone is at their own stage in life and at their own path in their walk with Christ.
Inviting everyone on a retreat has many benefits, too. While there are loads of topics that are more specific to certain people, there are still plenty of things that the Bible teaches for everyone. These retreats also help those in the congregation really get to know others they may not connect with much on Sunday mornings. The tricky part in planning this kind of retreat is the activities. Not everyone is into the same thing. Scheduling only highly active recreation can mean nothing to do for older or less active people. Make sure there are choices for all. However, once everyone comes together to worship, it can be a truly amazing experience.