Not a tax accountant, but while just failing to file is a misdemeanor, if you actually owed money and it can be shown that you willfully attempted to evade paying taxes owed, that's a felony (I.R.C. § 7201).

Now, chances are they don't find out, or if they do you can play innocent and pay a fine. Personally, if I were in this situation I would either file, renounce, or never set foot on US soil.