Is the Ethernet choice in the top of the selections available, as a priority,
above all others? If wi-fi is higher on the list, or if Ethernet is not active
such a situation may be a reason why that is slow to connect...
You may need to look into Network settings to see if this should be changed.
Also see what Console logs (if any) may suggest about this matter.
Even a re-start into Safe Mode, repair disk permissions, and a regular re-start
may help in some circumstances where the matter has little to do with Ethernet.
There likely are specific logs that may point to underlying causes.
But I'm not sure how to retrieve them or of another cures outside
the settings in Network and other devices that may be acting up
and should be disconnected when you are troubleshooting.
Good luck in this matter! 🙂