That's likely the fault of the GVFS and other virtual-fs-on-top-of-kernel-VFS layers. Gtk seems to be slower than others (such as KIO/KDE), but all of them are pretty slow. E.g. the KDE file picker takes about a second or so to finish listing the contents of my home directory (though it is usable while it is doing so, unlike the Gtk dialog), where the Gtk takes quite a bit longer (perhaps on the order of 3-5 seconds). Meanwhile ls is essentially instant, as you'd expect for just a few thousand files or so.
KIO is pretty fast I think. Dolphin is insanely fast at least - if you list a large directory almost everything that can't be loaded instantly is loaded asynchronously and streamed in to the UI after the UI becomes responsive. I think the KDE file picker is using other code altogether, I'm not sure why it's so much slower. GIO is horribly slow and makes everything that uses it next to useless, eg [1]
Interesting explanation at the end of the bug report:
The slowness is only caused by the GIO's G_FILE_ATTRIBUTE_STANDARD_CONTENT_TYPE and g_file_info_get_content_type. Wherever they are used, there will be a slowdown because the contents of files, that lack extensions, should be read for finding their mimetypes. Almost all the time is taken by that; the other operations are done pretty fast — particularly, the Qt GUI takes no time in comparison.
Yes, actually reading that it makes it sound like the file manager itself is just badly architechted - of course you shouldn't load mimetypes synchronously before listing a folder in the UI, that seems obvious.
So maybe I'm not right about GIO as such. But Nautilus is also excruciatingly slow despite being more fully asynchronous.