mirror of
https://git.freebsd.org/ports.git
synced 2025-06-02 11:26:29 -04:00
fb16dfecae
Commit b7f05445c0 has added WWW entries to port Makefiles based on
WWW: lines in pkg-descr files.
This commit removes the WWW: lines of moved-over URLs from these
pkg-descr files.
Approved by: portmgr (tcberner)
23 lines
1 KiB
Text
23 lines
1 KiB
Text
Despite the name, the crawler gets info for more services than just DNS:
|
|
- DNS:
|
|
- all A/AAAA records (for the 2nd level domain and www.subdomain), annotated
|
|
with GeoIP
|
|
- TXT records (with SPF and DMARC parsed for easier filtering)
|
|
- TLSA (for the 2nd level domain and www.subdomain)
|
|
- MX
|
|
- DNSSEC validation
|
|
- nameservers:
|
|
- each server IP annotated with GeoIP
|
|
- HOSTNAME.BIND, VERSION.BIND, AUTHORS.BIND and fortune (also for all IPs)
|
|
- users can add custom additional RRs in the config file
|
|
- E-mail (for every server from MX):
|
|
- SMTP server banners (optional, ports are configurable)
|
|
- TLSA records
|
|
- Web:
|
|
- HTTP status & headers (inc. parsed cookies) for ports 80 & 443 on each IP
|
|
from A/AAAA records
|
|
- certificate info for HTTPS (optionally with an entire cert chain)
|
|
- webpage content (optional)
|
|
- everything of the above is saved for each step in the redirect history --
|
|
the crawler follows redirects until it gets a non-redirecting status or hits
|
|
a configurable limit
|