Web V2

​

Basic form-based Authentication

• ttl. The “time to live” for the session created after authentication. After the specified time, the crawler logs in again to keep the crawl activity from failing due to an expired session. This value is defined in seconds.
• action. The action to take to log in. That is, the URL for the login form.
• params. The parameters for the form, likely the username and password, but could be other required properties. In the example below, we pass two parameters, os_username and os_password, which are expected by the system we crawl.

[ {
        "credential" : {
            "type" : "form",
            "ttl" : 300000,
            "action" : "http://some.server.com/login.action?os_destination=%2Fpages%2Fviewpage.action%3Ftitle%3DAcme%2B5%2BDocumentation%26spaceKey%3DAcme5",
            "params" : {
                "os_username" : "username",
                "os_password" : "password"
            }
        }
  } ]

​

Complex Form-based Authentication

• ttl. The “time to live” for the session created after authentication. After the specified time, the crawler logs in again to keep the crawl activity from failing due to an expired session. This value is defined in seconds.
• loginUrl. The URL on which the first page that initializes the login chain is located
• params. A list of parameters to use for the form logins, likely the username and password, but could be other required properties. In the example below, we pass two parameters, os_username and os_password, which are expected by the system we crawl. Additionally we expect that once that login has happened, a new form is presented to the user which then posts back to where we came from. No data need to be entered in this form, which is why we include an empty { } in the params list.

[ {
        "credential" : {
            "type" : "smartForm",
            "ttl" : 300000,
            "loginUrl" : "http://some.example.com/login",
            "params" : [{
                "os_username" : "username",
                "os_password" : "password"
            }, {

            } ]
        }
  } ]

​

Basic and Digest Authentication

• host. The host of the site.
• port. The port, if any.
• userName. The username to use for authentication.
• password. The password for the userName.
• realm. The security realm for the site, if any.

[ {
        "credential" : {
            "type" : "basic",
            "ttl" : 300000,
            "userName" : "usr",
            "password" : "pswd",
            "host":"hostname.exampledomain.com”
            "port": 443
        }
  }
]

​

NTLM Authentication

• host. The host of the site.
• port. The port, if any.
• userName. The username to use for authentication.
• password. The password for the userName.
• realm. The security realm for the site, if any.
• domain. The domain.
• workstation. The workstation, as needed.

[ {"credential" :
   { "type" : "ntlm",
     "ttl" : 300000,
     "port" : 80,
     "host" : "someHost",
     "domain" : "someDomain",
     "userName" : "someUser",
     "password" : "XXXXXXXX"
   }
} ]

​

Example with PingFederate

1. Create new app in PingIdentity.
2. Edit the app configuration:
   1. Check Client Credentials Grant Type.
   2. Set Token Endpoint Authentication Method to Client Secret Post.
3. Enable the app by clicking on the toggle button to the right of the app name.
4. Go to the Configuration tab and click Get Access Token. At this point it will probably show an error about scope. Fix it by adding a scope, such as testScope.
5. Click the Get Access Token button again under the Configuration tab. It should generate and display your access token.
6. Go to the Configuration tab and click URLs. Copy the Token Endpoint URL for further use.
7. You can view a sample of Client Credentials Grant Type access token request in the PingIdentity documentation. In this example the access token request is:
   $ curl -X POST "https://auth.pingone.com/ENV_ID/as/token"
   -d "grant_type=client_credentials&client_id=CLIENT_ID&client_secret=CLIENT_SECRET&scope=testScope" --header "Content-Type:application/x-www-form-urlencoded"
   {
     "access_token" : "ACCESS_TOKEN",
     "token_type" : "Bearer",
     "expires_in" : 3600,
     "scope" : "testScope"
   }
   
8. In Fusion, go to CRAWL AUTHENTICATION PROPERTIES in the datasource configuration and fill in the ACCESS TOKEN section by adding the URL and parameters of the request. Replace CLIENT_ID with your own client ID.
   • ACCESS TOKEN REQUEST URL: https://auth.pingone.com/ENV_ID/as/token
   • PARAMETER NAME | PARAMETER VALUE: grant_type | client_credentials
   • PARAMETER NAME | PARAMETER VALUE: client_id | CLIENT_ID
   • PARAMETER NAME | PARAMETER VALUE: scope | testScope

​

Example with Microsoft Azure AD

1. In Microsoft Entra ID, go to App Registrations and register a new app.
2. Go to your app and click Certificates & secrets and add a client secret.
3. Copy the client secret Value for further use.
4. View the Microsoft documentation for a request to get access token. In this example, we use the client credentials grant type:
   $ curl -X POST "https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/token"
   -d "client_id=CLIENT_ID&scope=https%3A%2F%2Fgraph.microsoft.com%2F.default&client_secret=CLIENT_SECRET&grant_type=client_credentials"
   -H "Content-Type: application/x-www-form-urlencoded"
   {"token_type":"Bearer","expires_in":3599,"ext_expires_in":3599,"access_token":"ACCESS_TOKEN"}
   
5. Copy the endpoint URL and the parameters to the CRAWL AUTHENTICATION PROPERTIES > ACCESS TOKEN section in the datasource configuration in Fusion. Replace CLIENT_ID with your own client ID.
   • ACCESS TOKEN REQUEST URL: https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/token
   • PARAMETER NAME | PARAMETER VALUE: grant_type | client_credentials
   • PARAMETER NAME | PARAMETER VALUE: client_id | CLIENT_ID
   • PARAMETER NAME | PARAMETER VALUE: scope | https://graph.microsoft.com/.default