You may need to contact support on this one so they can troubleshoot it with you.
After contacting support, this is what we discovered:
The import procedure have no issues managing over 3,000 LDAP entries, however the code that checks for deleted users has a bug in it and fails out, throwing the client side timeout error. This has been reported to the development team for investigation.
Assuming everything else is properly configured, the only problem this might cause is holding licenses for deleted users.
If everything else is not properly configured.. all sorts of things can go sideways.