websphinx
Class RobotExclusion

java.lang.Object
  |
  +--websphinx.RobotExclusion

public class RobotExclusion

extends java.lang.Object

Constructor Summary

RobotExclusion(java.lang.String userAgent)
Make a RobotExclusion object.

Method Summary

void	clear() Clear the cache of robots.txt entries.
boolean	disallowed(java.net.URL url) Check whether a URL is disallowed by robots.txt.
static void	main(java.lang.String[] argv)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RobotExclusion

public RobotExclusion(java.lang.String userAgent)

Make a RobotExclusion object.

Parameters:

userAgent - name of the robot using this object, as shown in the User-Agent header fields of its HTTP requests. Use null for anonymous robots.

Method Detail

disallowed

public boolean disallowed(java.net.URL url)

Check whether a URL is disallowed by robots.txt.

Parameters:

url - URL to test

Returns:

true if url's Web site denies robot access to the url

clear

public void clear()

Clear the cache of robots.txt entries.

main

public static void main(java.lang.String[] argv)
                 throws java.lang.Exception

java.lang.Exception