I would say returning regular pointers is more C++-esque than your handmade range implementation. Pointers are, after all, a kind of iterator, which is an essential C++ concept. (std::string::iterator is a handful.)
(Furthermore, there is nothing "C" or wrong with regular pointers. Their only "flaw" is that they can't manage an object, but they're still the semantical way to refer to one.)
Returning regular pointers which point to memory which is not managed by you and that you by default know could disappear at any time is not the C++ way of doing anything. Thing is, a class has a public interface for a reason, and if you as an outsider want to start playing pointer games you know from the start what you are in for.
I never implied that pointers are bad, just that this particular usage of pointers is bad, and as such should not be used as an example of why we would want something else over C++. If I ever saw a function with a signature like the one proposed I'd be extremely suspicious of what was going on.
Note that this would happen in the same way if the function returned pairs of iterators. It is a choice that depends on the documented and intended usage of a function. If you feel your tokenizer will be used on temporaries (I fail to see how that would ever be useful though) you could overload the function to also take universal references or change the return parameters so that nobody can get hurt. Otherwise you return whatever delimiters you are comfortable with, with the assumption that they will not be used in some way. All code can be broken if you actively try to, so it makes no sense to protect against everything.
> Returning regular pointers which point to memory which is not managed by you and that you by default know could disappear at any time is not the C++ way of doing anything. Thing is, a class has a public interface for a reason, and if you as an outsider want to start playing pointer games you know from the start what you are in for.
Maybe not "regular pointers", but returning iterators is very common (all throughout the STL, for one thing.) Those iterators are part of the public interface, and they provide no real guarantees over bare pointers.
I don't have a horse in this race, though. I think the "nicest" solution in abstract is for `tokenize` to be a token-generator* that takes a character generator argument. If you want to store the tokens yourself you can copy them, if you're happy to stream-process them then you can do that, and if you want a reference into the original string storage you can go fly a kite.
*: Where a "token" is probably another character generator.
I guess technically 'if you try to' means also abusing unsafe, so in theory he's still sorta right. Every time I see someone ask a question about transmute on IRC I shiver a little.
But yes, Rust is way, way, way better in this regard.
The difference is safe-by-default languages force you to turn off the safeties before they will let you blow your own foot off. C++ doesn't believe in safeties since (at the time the language was designed) it introduces an unacceptable delay between you wanting to pull the trigger and your target and/or feet blowing up.
> I never implied that pointers are bad, just that this particular usage of pointers is bad, and as such should not be used as an example of why we would want something else over C++. If I ever saw a function with a signature like the one proposed I'd be extremely suspicious of what was going on.
You would. I wouldn't because I don't know any better. How many others are there out there that don't know any better? If you code by yourself, fine, but if you work with a team you are gonna need a higher level of training, so you could teach people like me to catch the problem. Are you always going to be disciplined enough to do code reviews that are in depth enough to capture fishiness in the type signature?
With Rust, if it compiles and passes unit tests, the code review is already 90% done.
Compilation and unit testing will not catch suboptimal expression of logic/thought (which in my experience count for way more than 10% of code reviews).
>Their only "flaw" is that they can't manage an object, but they're still the semantical way to refer to one.)
Yes, that's their only flaw. A flaw which has caused countless bugs and will continue to do so in the future. This is why C++ provides so many utilities designed to keep you from using them in 99% of cases. But yeah, that's their only flaw.
(Furthermore, there is nothing "C" or wrong with regular pointers. Their only "flaw" is that they can't manage an object, but they're still the semantical way to refer to one.)