I think you are going about this incorrectly. Do not prioritize on commonality as your user count remains small. Instead prioritize on severity then carefully examine each defect as they come in. You are either willing to work the defect or you aren't. There is no harm in marking an issue as out of scope or won't fix. Just be clear and direct with your users that you have plans for this application and those plans only move in one direction.
This is where it can be helpful to have some folks on your team whose primary focus is user satisfaction (and/or sales). They can help guide issue prioritization in a way that code-oriented team members may not.
Sometimes it can be frustrating to ignore a bunch of easy-to-fix issues, but often those issues have the lowest impact on the perceived quality of the software.
