From the course: Technical SEO
Join today to access over 22,600 courses taught by industry experts or purchase this course individually.
The role of robots.txt files
- [Instructor] Robots.txt files are simple text files that live in what's called the root of your site. You can check if your website has one, by going to your domain.com/robots.txt. Many CMSs such as WordPress will automatically create one. But that doesn't mean it's set up quite the way that it should be. Robots.txt files Tell search engine crawlers, not just Google, but all search engine crawlers, what pages or files they can or cannot access on your website. These are useful to keep robots out of areas where they wouldn't find anything useful, such as the WP admin directory on WordPress. Your robots.txt file contains disallow, allow or both types of instructions. The basic format of a robots.txt file is like this, User-agent, name of the user agent, and then disallow the thing you don't want crawled. You only really need these two lines to make up the file. It just needs to include what search engine crawlers you want to…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.