> Returning regular pointers which point to memory which is not managed by you and that you by default know could disappear at any time is not the C++ way of doing anything. Thing is, a class has a public interface for a reason, and if you as an outsider want to start playing pointer games you know from the start what you are in for.
Maybe not "regular pointers", but returning iterators is very common (all throughout the STL, for one thing.) Those iterators are part of the public interface, and they provide no real guarantees over bare pointers.
I don't have a horse in this race, though. I think the "nicest" solution in abstract is for `tokenize` to be a token-generator* that takes a character generator argument. If you want to store the tokens yourself you can copy them, if you're happy to stream-process them then you can do that, and if you want a reference into the original string storage you can go fly a kite.
*: Where a "token" is probably another character generator.
Maybe not "regular pointers", but returning iterators is very common (all throughout the STL, for one thing.) Those iterators are part of the public interface, and they provide no real guarantees over bare pointers.
I don't have a horse in this race, though. I think the "nicest" solution in abstract is for `tokenize` to be a token-generator* that takes a character generator argument. If you want to store the tokens yourself you can copy them, if you're happy to stream-process them then you can do that, and if you want a reference into the original string storage you can go fly a kite.
*: Where a "token" is probably another character generator.