Crawl Web Application

Crawl Web Application mode crawls the web application and will list all the webpages crawled. This mode will not set up the K2agent.

Crawl Web Application

Crawl Web Application mode of K2HackBot works as a basic web application crawler. It will try to access all the HTML components present across the application.

This mode does not require K2agent installation.

Options

Note: The options marked with ** are MANDATORY.

Option	Description
--isAuthRequired	Set this to true when application has authenticated routes.
--applicationLoginUrl	Provide the URL of the login page when --isAuthRequired is set to true
**--applicationUrl	The URLs pointing to the application.
--applicationLoginIdentifier	Provide the application login identifier when --isAuthRequired is set to true. How to prepare the application login identifier?
--ignoreUrl	Provide a list of keywords/URLs that will be ignored while crawling.
--allowedDomain	Provide a list of domains that will be considered while crawling.
--config	Provide the config file path. How to setup config file?

Commands

Make sure that K2HackBot is setup correctly at this point before running crawl-web-application commands.

• Command to display the help page:

$ k2hackbot crawl-web-application --help

  
Options:
  --applicationLoginUrl TEXT      Provide Login Url for your application.
                                  
                                  Eg. k2hackbot crawl-web-application
                                  --applicationLoginUrl
                                  https://myapplication.com/login

  --isAuthRequired BOOLEAN        If your applications requires authentication
                                  using login.
                                  
                                  Eg. k2hackbot crawl-web-application
                                  --isAuthRequired true

  --applicationUrl TEXT           [REQUIRED] Provide Application URls to be
                                  crawled.
                                  
                                  Eg. k2hackbot crawl-web-application
                                  --applicationUrl http://myapplication.com

  --applicationLoginIdentifier TEXT
                                  If your Application needs a user to be
                                  authenticated using login, we need
                                  identifier to do that ourself. Provide
                                  identifier  for Application form fields in
                                  format as string:
                                  
                                  '{"username": {"identification":
                                  "user_field_id","value":
                                  "user_name"},"password": {"identification":
                                  "password_field_id","value":
                                  "password"},"submit": {"identification":
                                  "submit_button_id","value": "Nothing"}}'
                                  
                                  You can find Guide to Do so on following git
                                  repository https://github.com/k2io/K2ADS
                                  
                                  Eg. k2hackbot crawl-web-application
                                  --applicationLoginIdentifier '{"username":
                                  {"identification": "user_field_id","value":
                                  "user_name"},"password": {"identification":
                                  "password_field_id","value":
                                  "password"},"submit": {"identification":
                                  "submit_button_id","value": "Nothing"}}'

  --ignoreUrl TEXT                Provide Application URLs to be skipped for
                                  crawling.
                                  
                                  Eg. k2hackbot crawl-web-application
                                  --ignoreUrl
                                  ['http://myapplication.com/contact']

  --allowedDomain TEXT            Application Domains to be used for crawling
                                  
                                  Eg. k2hackbot crawl-web-application
                                  --allowedDomain
                                  ['http://myapplication.com/home']

  --force TEXT                    Enable Hackbot to take over and execute
                                  forcefully

  --config FILE                   Read configuration from FILE.
  --help                          Show this message and exit.

• Command to crawl applications without authentication:

k2hackbot crawl-web-application --applicationUrl 'https://example.com'

• Command to scan applications with authentication:

k2hackbot crawl-web-application --applicationUrl 'https://example.com' --isAuthRequired true --applicationLoginUrl 'http://my-application.com/login' --applicationLoginIdentifier "{\"username\": {\"identification\": \"name='user'\",\"value\": \"cody\"},\"password\": {\"identification\": \"name='password'\",\"value\": \"cody\"}, \"submit\": {\"identification\": \"type='submit'\",\"value\": \"Nothing\"}}" 

Config File Setup

K2HackBot support two ways to define options:

• CLI

• Config File

The Hierarchy for reading config is as follows:

CLI > Environment > Configuration file > Default

The config file can be created with any name but it MUST be in JSON format. Below is a sample config file for Crawl Web Application:

$ cat k2config.json 
{
	"applicationurl": ["https://example.com"],
	"isauthrequired": true,
	"applicationloginurl": "https://example.com/login",
	"applicationloginidentifier": {
		"username": {
			"identification": "name='user'",
			"value": "test"
		},
		"password": {
			"identification": "name='password'",
			"value": "PASSWORD"
		},
		"submit": {
			"identification": "type='submit'",
			"value": "Nothing"
		}
	}
}

Command to run crawl-web-application using the config file:

k2hackbot crawl-web-application --config k2config.json