I think smalltalks just didn't bring you to the right topic, if you were to reach topic you both fancy, you would definitely remember each other. If anything, having a smalltalk is nicer than staying in awkward silence.