A robot is a program used by search engines to collect data from the Internet. When a crawler visits a website, it selects all content from the website and stores it in a database. It also stores all the external and internal links of the website and will visit them at a later time, this is how it moves from one website to another.
