Non-native speaker as well. I recently noticed the "b" is supposed to be silent, but then every time I say "debt" without the "b", phonetically I'm saying the same as "that". They sound the same to me, at least.
This is probably related to the fact that "bad" and "bed" sound exactly the same to me.
Related, yes, but the "th" in "that" should sound different (tongue touching edges of both top and bottom teeth, extending just past) from the "d" in "debt" (tongue touching roof of mouth just behind front teeth)
The vowels are tougher, though. I didn't appreciate how many different vowels there are in English until I married a woman whose native language doesn't have most of them (Hebrew doesn't have the "a" in "bad", the "i" in "grin", nor a bunch of weird regional American "o" sounds)
Native speaker here, from the Ozarks (US, on the Missouri/Arkansas border).
I can't hear the distinction between "pin" and "pen" unless I'm really paying attention. Even then I have to say it a dozen or so time before I can make myself say them distinctively. My birth accent is to pronounce them both mostly like "pin".
This is probably related to the fact that "bad" and "bed" sound exactly the same to me.