Don't confuse available bandwidth (30 Mbps or whatever) with the network latency.
For a responsive web site, latency is the key measure, so long as your bandwidth is appropriate for the size of the pages (and images) you're tossing around. This is where tools such as traceroute help — I've seen a half-second delay in some ISP routers and occasionally more, and there's zilch I can do about that beyond reporting it, but that delay inherently factors into the web page responsiveness that the users and the crawlers encounter.
To use an automotive analogy for network bandwidth and latency, folks usually buy horsepower, but love torque.
To use an analogy more typical of broadband in parts of Nueva Inglaterra en Los Estados Unitos de América, satellite broadband networking can provide good bandwidth, but the latency of the ~35,786 Km round-trip via geostationary orbit contributes mightily to the technology's unsuitability for networking tasks requiring lower latency. (Which can point to another factor when creating and operating your web site: if your audience is in low-bandwidth areas or is heavily using mobile devices, serving big pages and big images is a bad idea. But I digress.)
I've read a fair amount about performance optimizations and search engine optimization and the rest over the years (and Google and others have some reasonable resources here), and decided that I'd ignore most of what's published (particularly anything around the Get Links Fast genre), and get to work creating quality, unique, searchable, current content. Using content-appropriate keywords here can help folks more easily find your content, too. This is how you get readers and get good links, and it's the core factor in any search ranking.
Don't forget to test with Bing and the other search engines, as more than a few of us are finding that Google has an enviably massive corpus, but increasingly poor and increasingly gamed search ranking.
For lower-level performance details (and particularly around page responsiveness quests), use the available browser-based tools to measure the performance of your site, such as the Safari developer menu and its page performance profiling or other similar tools in other browsers.
For a fast site, fewer and smaller and better-compressed images are a win.
Zipped pages can be a win, if you have more CPU than network.
Work to profile and to locate and to remove the slower bits. Over the years, I've found various surprises while profiling pages, whether it was a glacially slow disk serving content, or "under-performing" JavaScript or a maladriot CMS add-on module, generic logging activity that was hampered by slow (and technically entirely unnecessary) DNS back-translations within the logging configuration, or an image somewhere on the web page that was unnecessarily huge.
Pre-generated web page caches are usually a win, as that can avoid the majority of the overhead and the database activity involved with a CMS page generation. If your CMS doesn't have a decent static-page caching capability, either find or create an extension that works, or migrate to the CMS that works faster — remember that the goal of using a CMS is to make your work easier, but your overarching goal here is to quickly and efficiently serve your content to your readers. Pick your tools appropriately.
Tools to monitor and flag web outages and network disconnections, having viable and tested and restorable backups (and preferably offsite, but take what you can get), and somebody that's monitoring for the inevitable "that's weird" moments that can point to performance or stability or backup failures or database corruption issues, and for site hacks — WordPress and most other CMS packages tend to be subject to these, and this ignoring the current pingback mess — and to the necessity of keeping WordPress and MySQL and the rest current.
CDNs? Yeah. Definitely look at those once you've got the rest of the chain under control, or when your own big-box server is bogging under the load, or when you've got a special case such as serving large files or video; when your network pipe becomes an issue. If you're approaching this stage, you probably have somebody specifically targeting this performance and tuning and re-hosting work, too — you'll probably also be partially migrating to a hosting service provider or CDN here and this potentially in addition to your own geographically distributed data centers.
In short: don't get ahead of yourself with scaling issues you don't (yet) have, and learn where the issues that you do have are lurking, and then the usual tradeoffs involving how much time and effort and cash it'll cost to remove those issues.
