v6 Scan Services¶
These services allow a user to create, view, and manage Scan processes.
v6 Scan List¶
Example GET /v6/scans/ API call
Request: GET http://.../v6/scans/
Response: 200 OK
{ "count": 3, "next": null, "previous": null, "results": [ { "id": 1, "name": "my-scan-process", "title": "My Scan Process", "description": "This is my Scan process for detecting my favorite files!", "file_count": 50, "job": { "id": 7, "job_type": { "id": 2, "name": "scale-scan", "version": "1.0.0", "title": "Scale Scan", "description": "Scans a workspace for existing source files to ingest", "is_active": true, "is_paused": false, "is_published": true, "icon_code": "f013", "unmet_resources": "chocolate,vanilla" }, "job_type_rev": { "id": 2 }, "status": "RUNNING" }, "dry_run_job": { "id": 6, "job_type": { "id": 2, "name": "scale-scan", "version": "1.0.0", "title": "Scale scan", "description": "Scans a workspace for existing source files to ingest", "is_active": true, "is_paused": false, "is_published": true, "icon_code": "f013", "unmet_resources": "chocolate,vanilla" }, "status": "RUNNING" }, "created": "2015-03-11T00:00:00Z", "last_modified": "2015-03-11T00:00:00Z" }, ... ] }
Scan List | |||
---|---|---|---|
Returns a list of all Scan processes. | |||
GET /v6/scans/ | |||
Query Parameters | |||
page | Integer | Optional | The page of the results to return. Defaults to 1. |
page_size | Integer | Optional | The size of the page to use for pagination of results. Defaults to 100, and can be anywhere from 1-1000. |
started | ISO-8601 Datetime | Optional | The start of the time range to query. Supports the ISO-8601 date/time format, (ex: 2015-01-01T00:00:00Z). Supports the ISO-8601 duration format, (ex: PT3H0M0S). |
ended | ISO-8601 Datetime | Optional | End of the time range to query, defaults to the current time. Supports the ISO-8601 date/time format, (ex: 2015-01-01T00:00:00Z). Supports the ISO-8601 duration format, (ex: PT3H0M0S). |
name | String | Optional | Return only Scan processes with a given name. Duplicate it to filter by multiple values. |
order | String | Optional | One or more fields to use when ordering the results. Duplicate it to multi-sort, (ex: order=name&order=description). Prefix fields with a dash to reverse the sort, (ex: order=-name). |
Successful Response | |||
Status | 200 OK | ||
Content Type | application/json | ||
JSON Fields | |||
count | Integer | The total number of results that match the query parameters. | |
next | URL | A URL to the next page of results. | |
previous | URL | A URL to the previous page of results. | |
results | Array | List of result JSON objects that match the query parameters. | |
.id | Integer | The unique identifier of the model. Can be passed to the details API. (See Scan Details) | |
.name | String | The identifying name of the Scan process used for queries. | |
.title | String | The human readable display name of the Scan process. | |
.description | String | A longer description of the Scan process. | |
.file_count | Integer | Count of files identified from last scan operation (either dry run or ingest). | |
.job | JSON Object | The job that is associated with the Scan process. (See Job Details) | |
.dry_run_job | JSON Object | The dry run job that is associated with the Scan process. (See Job Details) | |
.created | ISO-8601 Datetime | When the associated database model was initially created. | |
.last_modified | ISO-8601 Datetime | When the associated database model was last saved. |
v6 Create Scan¶
Example POST /v6/scans/ API call
Request: POST http://.../v6/scans/
{ "title": "My Scan Process", "description": "This is my Scan process for detecting my favorite files!", "configuration": { "workspace": "my-workspace", "scanner": { "type": "dir", }, "recursive": true, "files_to_ingest": [{ "filename_regex": ".*txt" }], "recipe": { "name": "my-recipe", "revision_num": 1 } } }
Response: 201 Created Headers: Location http://.../v6/scans/105/
{ "id": 1, "name": "my-scan-process", "title": "My Scan Process", "description": "This is my Scan process for detecting my favorite files!", "file_count": 50, "job": { "id": 7, "job_type": { "id": 2, "name": "scale-scan", "version": "1.0.0", "title": "Scale Scan", "description": "Scans a workspace for existing source files to ingest", "is_active": true, "is_paused": false, "is_published": true, "icon_code": "f013", "unmet_resources": "chocolate,vanilla" }, "status": "RUNNING" }, "dry_run_job": { "id": 6, "job_type": { "id": 2, "name": "scale-scan", "version": "1.0.0", "title": "Scale Scan", "description": "Scans a workspace for existing source files to ingest", "is_active": true, "is_paused": false, "is_published": true, "icon_code": "f013", "unmet_resources": "chocolate,vanilla" }, "status": "RUNNING" }, "created": "2015-03-11T00:00:00Z", "last_modified": "2015-03-11T00:00:00Z", "configuration": { "workspace": "my-workspace", "monitor": { "type": "dir" }, "recursive": true, "files_to_ingest": [{ "filename_regex": ".*txt" }], "recipe": { "name": "my-recipe", "revision_num": 1 } } }
Create Scan | |||
---|---|---|---|
Creates a new Scan. To start a dry run or actual scan job, use the /scans/{id}/process/ endpoint. | |||
POST /v6/scans/ | |||
Content Type | application/json | ||
JSON Fields | |||
title | String | Required | The human readable display name of the Scan process. |
description | String | Optional | A longer description of the Scan process. |
configuration | JSON Object | Required | JSON defining the Scan configuration. (See Scan Configuration JSON) |
Successful Response | |||
Status | 201 CREATED | ||
Location | URL pointing to the details for the newly created scan process | ||
Content Type | application/json | ||
JSON Fields | |||
JSON Object | All fields are the same as the Scan process details model. (See Scan Details) |
v6 Scan Details¶
Example GET /v6/scans/{id}/ API call
Request: GET http://.../v6/scans/{id}/
Response: 200 OK
{ "id": 1, "name": "my-scan-process", "title": "My Scan Process", "description": "This is my Scan process for detecting my favorite files!", "file_count": 50, "job": { "id": 7, "job_type": { "id": 2, "name": "scale-scan", "version": "1.0.0", "title": "Scale Scan", "description": "Scans a workspace for existing source files to ingest", "is_active": true, "is_paused": false, "is_published": true, "icon_code": "f013", "unmet_resources": "chocolate,vanilla" }, "status": "RUNNING" }, "dry_run_job": { "id": 6, "job_type": { "id": 2, "name": "scale-scan", "version": "1.0.0", "title": "Scale Scan", "description": "Scans a workspace for existing source files to ingest", "is_active": true, "is_paused": false, "is_published": true, "icon_code": "f013", "unmet_resources": "chocolate,vanilla" }, "status": "RUNNING" }, "created": "2015-03-11T00:00:00Z", "last_modified": "2015-03-11T00:00:00Z", "configuration": { "workspace": "my-workspace", "monitor": { "type": "dir" }, "recursive": true, "files_to_ingest": [{ "filename_regex": ".*txt" }], "recipe": { "name": "my-recipe", "revision_num": 1 } } }
Scan Details | ||
---|---|---|
Returns Scan process details | ||
|
||
Successful Response | ||
Status | 200 OK | |
Content Type | application/json | |
JSON Fields | ||
id | Integer | The unique identifier of the model. Can be passed to the details API. (See Scan Details) |
name | String | The identifying name of the Scan process used for queries. |
title | String | The human readable display name of the Scan process. |
description | String | A longer description of the Scan process. |
file_count | Integer | Count of files identified from last scan operation (either dry run or ingest). |
job | JSON Object | The job that is associated with the Scan process. (See Job Details) |
dry_run_job | JSON Object | The dry run job that is associated with the Scan process. (See Job Details) |
created | ISO-8601 Datetime | When the associated database model was initially created. |
last_modified | ISO-8601 Datetime | When the associated database model was last saved. |
configuration | JSON Object | JSON defining the Scan configuration. (See Scan Configuration JSON) |
v6 Validate Scan¶
Example POST /v6/scans/validation/ API call
Request: POST http://.../v6/scans/validation/
{
"title": "My Scan Process",
"description": "This is my Scan process for detecting my favorite files!",
"configuration": {
"workspace": "my-workspace",
"monitor": {
"type": "dir"
},
"recursive": true,
"files_to_ingest": [{
"filename_regex": ".*txt"
}],
"recipe": {
"name": "my-recipe",
"revision_num": 1
}
}
}
Response: 200 OK
{
"is_valid": true,
"errors": [],
"warnings": [{"name": "EXAMPLE_WARNING", "description": "This is an example warning."}],
}
Validate Scan | |||
---|---|---|---|
Validates a new Scan process configuration without actually saving it | |||
POST /v6/scans/validation/ | |||
Content Type | application/json | ||
JSON Fields | |||
title | String | Required | The human readable display name of the Scan process. |
description | String | Optional | A longer description of the Scan process. |
configuration | JSON Object | Required | JSON defining the Scan configuration. (See Scan Configuration JSON) |
Successful Response | |||
Status | 200 OK | ||
Content Type | application/json | ||
JSON Fields | |||
is_valid | Boolean | Indicates if the given fields were valid for creating a new scan. If this is true, then submitting the same fields to the /scans/ API will successfully create a new scan. | |
errors | Array | Lists any errors causing is_valid to be false. The errors are JSON objects with name and description string fields. | |
warnings | Array | A list of warnings discovered during validation. | |
.id | String | An identifier for the warning. | |
.details | String | A human-readable description of the problem. |
v6 Edit Scan¶
Example PATCH /v6/scans/{id}/ API call
Request: PATCH http://.../v6/scans/{id}/
{
"title": "My Scan Process",
"description": "This is my Scan process for detecting my favorite files!",
"configuration": {
"workspace": "my-workspace",
"monitor": {
"type": "dir"
},
"recursive": true,
"files_to_ingest": [{
"filename_regex": ".*txt"
}],
"recipe": {
"name": "my-recipe",
"revision_num": 1
}
}
}
Response: 204 NO CONTENT
Edit Scan | |||
---|---|---|---|
Edits an existing Scan process with associated configuration | |||
|
|||
Content Type | application/json | ||
JSON Fields | |||
title | String | Optional | The human readable display name of the Scan process. |
description | String | Optional | A longer description of the Scan process. |
configuration | JSON Object | Optional | JSON defining the Scan configuration. (See Scan Configuration JSON) |
Successful Response | |||
Status | 204 No Content |
v6 Process Scan¶
Example POST /v6/scans/{id}/process/ API call
Request: POST http://.../v6/scans/{id}/process/
{ "ingest": true }
Response: 200 OK
{ "id": 1, "name": "my-scan-process", "title": "My Scan Process", "description": "This is my Scan process for detecting my favorite files!", "file_count": 50, "job": { "id": 7, "job_type": { "id": 2, "name": "scale-scan", "version": "1.0.0", "title": "Scale Scan", "description": "Scans a workspace for existing source files to ingest", "is_active": true, "is_paused": false, "is_published": true, "icon_code": "f013", "unmet_resources": "chocolate,vanilla" }, "status": "RUNNING" }, "dry_run_job": { "id": 6, "job_type": { "id": 2, "name": "scale-scan", "version": "1.0.0", "title": "Scale Scan", "description": "Scans a workspace for existing source files to ingest", "is_active": true, "is_paused": false, "is_published": true, "icon_code": "f013", "unmet_resources": "chocolate,vanilla" }, "status": "RUNNING" }, "created": "2015-03-11T00:00:00Z", "last_modified": "2015-03-11T00:00:00Z", "configuration": { "workspace": "my-workspace", "monitor": { "type": "dir" }, "recursive": true, "files_to_ingest": [{ "filename_regex": ".*txt" }], "recipe": { "name": "my-recipe", "revision_num": 1 } } }
Process Scan | |||
---|---|---|---|
Launches an existing Scan with associated configuration | |||
|
|||
Content Type | application/json | ||
JSON Fields | |||
ingest | Boolean | Optional | Whether a dry run or ingest triggering scan should be run. Defaults to false when unset. |
Successful Response | |||
Status | 200 OK | ||
Content Type | application/json | ||
JSON Fields | |||
id | Integer | The unique identifier of the model. Can be passed to the details API. (See Scan Details) | |
name | String | The identifying name of the Scan process used for queries. | |
title | String | The human readable display name of the Scan process. | |
description | String | A longer description of the Scan process. | |
file_count | Integer | Count of files identified from last scan operation (either dry run or ingest). | |
job | JSON Object | The job that is associated with the Scan process. (See Job Details) | |
dry_run_job | JSON Object | The dry run job that is associated with the Scan process. (See Job Details) | |
created | ISO-8601 Datetime | When the associated database model was initially created. | |
last_modified | ISO-8601 Datetime | When the associated database model was last saved. | |
configuration | JSON Object | JSON defining the Scan configuration. (See Scan Configuration JSON) |
v6 Cancel Scan¶
This call will send a cancel job request for the job associated with the scan, as well as all queued or running ingest jobs spawned from the scan. recipes triggered by completed ingest jobs will not be effected. Returns the job ids of all canceled jobs.
Example POST /v6/scans/cancel/{id} API call
Request: POST http://.../v6/scans/cancel/{id}/
Response: 202 Accepted
{
"id": 1,
"canceled_jobs": [1,2,3]
}
Cancel Scan | ||
---|---|---|
Cancels a scan and associated ingest jobs | ||
|
||
Successful Response | ||
Status | 202 ACCEPTED | |
Content Type | application/json | |
JSON Fields | ||
id | Integer | The unique identifier of the scan. |
canceled_jobs | Array | List of the job ids associated with the scan that were canceled. |
Scan Configuration JSON¶
A scan configuration JSON describes a set of configuration settings that affect how a scanner executes.
Example interface:
{
"workspace" : "workspace_name",
"scanner" : {
"type" : "dir",
"transfer_suffix" : "_tmp"
},
"recursive" : true,
"files_to_ingest":[
{
"filename_regex" : ".*txt",
"data_types": [ "type1", "type2" ],
"new_workspace" : "workspace_name",
"new_file_path" : "wksp/path"
}
],
"recipe": {
"name": "my-recipe",
"revision_num": 1
}
}
Scan Configuration | |||
---|---|---|---|
workspace | String | Required | String that specifies the name of the workspace that is being scanned. The type of the workspace (its broker type) will determine which types of scanner can be used. |
scanner | JSON Object | Required | JSON object representing the type and configuration of the scanner that will scan workspace for files. |
.type | String | Required | The type of the scanner. Must be either ‘dir’ or ‘s3’ |
.transfer_suffix | String | Optional | Defines a suffix that is used on the file names to indicate that files are still transferring and have not yet finished being copied into the scanned directory |
recursive | Boolean | Optional | Indicates whether a scanner should be limited to the root of a workspace (false) or traverse the entire tree (true). If ommitted, the default is true |
files_to_ingest | Array | Required | List of JSON objects that define the rules for how to handle files that appear in the scanned workspace. The array must contain at least one item. |
.filename_regex | String | Required | Regular expression to check against the names of new files in the scanned workspace. When a new file appears in the workspace, the file’s name is checked against each expression in order of the files_to_ingest array. If an expression matches the new file name in the workspace, that file is ingested according to the other fields in the JSON object and all subsequent rules in the list are ignored (first rule matched is applied). |
.data_types | Array | Optional | Any file that matches the corresponding file name regular expression will have these data type strings “tagged” with the file. If not provided, data_types defaults to an empty array. |
.new_workspace | String | Optional | Specifies the name of a new workspace to which the file should be copied. This allows the ingest process to move files to a different workspace after they appear in the scanned workspace. |
.new_file_path | String | Optional | Specifies a new relative path for storing new files. If new_workspace is also specified, the file is moved to the new workspace at this new path location (instead of using the current path the new file originally came in on). If new_workspace is not specified, the file is moved to this new path location within the original scanned workspace. In either of these cases, three additional and dynamically named directories, for the current year, month, and day, will be appended to the new_file_path value automatically by the Scale system (i.e. workspace_path/YYYY/MM/DD). |
recipe | JSON Object | Required | Specifies the natural key of the recipe the Strike will start when a file is ingested. |
.name | String | Required | Specifies the name of the recipe the Scan will attempt to start when the file is ingested. |
.revision_num | Integer | Optional | Specifies the revision number of the recipe the Scan will attempt to start when the file is ingested. |