Skip to content

package

robots-parser

v2.4.0

NodeJS robots.txt parser with support for wildcard (*) matching.

node any0 depscjstypesopen source
license
MIT
deps
0
unpacked
18.9 kB
files
6
node
any
npm install robots-parser@2.4.0

README

GitHub-flavored package documentation.

markdown
downloads

Across all versions.

Download trend unavailable.

compatibility

Runtime metadata.

nodeany
cjstypes
keywords

Tagged topics.

robots.txtparseruser-agentscraperspiderbotrobots-exclusion-standard