Add Custom Headers to HTTP Requests
The Web V1 connector retrieves data from a Web site using HTTP and starting from a specified URL.
You can optionally add custom headers to all http get
requests from the Web connector.
For example, you might want to add a header that includes Connection: keep-alive
to prevent the connector from timing out while crawling your Web site.
To add a custom header, use the configuration parameter f.addedHeaders
. To send multiple headers, use the following format:
Header1: Value1
Header2: Value2
-
Click Indexing > Datasources.
-
Click Add+, then Web.
-
Enter a datasource ID and a start link.
-
Click Link discovery.
-
Fill in the Headers to add to HTTP requests field.
-
Add each header in the format
HeaderName: HeaderValue
. -
To add multiple headers to all HTTP requests, put each header on a new line.
-
-
Fill in any remaining options to configure the datasource.
-
Click Save.