From the course: HTML and CSS: Linking
Join today to access over 22,600 courses taught by industry experts or purchase this course individually.
Controlling which links search engines follow
From the course: HTML and CSS: Linking
Controlling which links search engines follow
- [Instructor] By default, any search engine can visit your site, index your pages and include your site in search results. There's nothing special that you need to do to make this happen. It will eventually happen if you wait for a while and you can read the latest instructions for Google and other search engines to speed up this process. However, occasionally you may want to prevent search engines from visiting all or part of your website. There is a way to tell search engine robots called, spiders, from visiting your website and indexing your pages. This method uses something called the robots.txt file on your web server and it assumes that the spider will pay attention to this file in order for this to work. There is no way to ensure that your pages will not show up in search results, if the page is open to the public, not behind a login or some other exclusionary device, but this method will work with most of the major search engines. It's always helpful to look at the latest…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.