In a recent project I was asked to create a user story classifier to identify whether stories were "new development" or "maintenance of existing features". I tried both approaches, embeddings + cosine distance vs. directly asking a language model to classify the user story. The embeddings approach was, despite being fueled by the most powerful SOTA embedding model available, surprisingly worse than simply asking GPT 4.1 to give me the correct label.

OP here. It depends what you use it for. You do want the tags if you intend to generate data. Let's say you prompt an LLM to go tweet on your behalf for a week, having the ability to:

- Fetch a list of my unique tags to get a sense of my topics of interests

- Have the AI dig into those specific niches to see what people have been discussing lately

- Craft a few random tweets that are topic-relevant and present them to me to curate

Is very powerful workflow that is hard to deliver on without the class labels.