The following script will scrape a URL for comments and links in the source code. It will then also perform limited spidering and search linked URLs for comments:

import requests
import re

from bs4 import BeautifulSoup
import sys

if len(sys.argv) !=2:
    print "usage: %s targeturl" % (sys.argv[0])
    sys.exit(0)

urls = []

tarurl = sys.argv[1]
url = requests.get(tarurl)
comments = re.findall('<!--(.*)-->',url.text)
print "Comments on page: "+tarurl
for comment in comments:
    print comment

soup = BeautifulSoup(url.text)
for line in soup.find_all('a'):
    newline = line.get('href')
    try:
        if newline[:4] == "http":
            if tarurl in newline:
                urls.append(str(newline))
        elif newline[:1] == "/":
            combline = tarurl+newline
            urls.append(str(combline))
    except:
        pass
        print "failed"
for uurl in urls:
    print "Comments on page: "+uurl
    url = requests.get(uurl)
    comments = re.findall('<!--(.*)-->',url.text)
    for comment in comments:
        print comment

After the initial import of the necessary modules and setting up of variables, the script first gets the source code of the target URL.

You may have noticed that for Beautifulsoup, we have the following line:

from bs4 import BeautifulSoup

This is so that when we use BeautifulSoup, we just have to type BeautifulSoup instead of bs4.BeautifulSoup.

It then searches for all instances of HTML comments and prints them out:

url = requests.get(tarurl)
comments = re.findall('<!--(.*)-->',url.text)
print "Comments on page: "+tarurl
for comment in comments:
    print comment

The script will then use Beautifulsoup in order to scrape the source code for any instances of absolute (starting with http) and relative (starting with /) links:

if newline[:4] == "http":
            if tarurl in newline:
                urls.append(str(newline))
        elif newline[:1] == "/":
            combline = tarurl+newline
            urls.append(str(combline))

Once the script has collated a list of URLs linked to from the page, it will then search each page for HTML comments.

This recipe shows a basic example of comment scraping and spidering. It is possible to add more intelligence to this recipe to suit your needs. For instance, you may want to account for relative links that use start with . or .. to denote the current and parent directories.

You can also add more control to the spidering part. You could extract the domain from the supplied target URL and create a filter that does not scrape links for domains external to the target. This is especially useful for professional engagements where you need to adhere to a scope of targets.