Capturing your algorithm outputs with a Scale Results Manifest

Generating a results manifest should either be done within your algorithm or within the algorithm’s wrapper script. Once an algorithm is complete a results manifest file is used to convey what products should be archived by Scale and passed onto other algorithms. It is HIGHLY recommended that the algorithm write out the manifest JSON file as prescribed in the Scale documentation.

Click here for results manifest specification: Results Manifest

Below are examples of wrapper scripts if you cannot modify the algorithm. These should only be used if you do not have access to the source code for your algorithm.

ONLY FILES IN THE RESULTS MANIFEST WILL BE SAVED BY SCALE

File paths listed in the results manifest must be absolute paths to the file and the files must exist within the output directory provided by Scale. The results manifest must be called “results_manifest.json” and must be present in the root of the output directory. All output files and the results manifest must have full read permission (as well as execute for any directories) granted for ALL users, allowing the Scale post task to read the outputs. The output names in the results manifest must match the output_data field in the job interface.

Simple Results Manifest Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
 {
    "version": "1.1",
    "output_data": [
       {
          "name" : "output_file_csv",
          "file": {
             "path" : "/tmp/job_exe_231/outputs/results.csv"
          }
       },
       {
          "name" : "output_file_tif",
          "file": {
             "path" : "/tmp/job_exe_231/outputs/myimage1.tif"
          }
       },
       {
          "name" : "output_file_tif2",
          "file": {
             "path" : "/tmp/job_exe_231/outputs/myimage2.tif"
          }
       }
    ]
 }

Example Code for generating results manifests

Shell Script

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# Gather the output files and put them  in a manifest
function join { local IFS="$1"; shift; echo "$*"; }


manifest_files=()

#Find the files in the output directory
image1_tif=$(find $OUTDIR -name "*image1.tif")
image2_tif=$(find $OUTDIR -name "*image2.tif")
csv_data=$(find $OUTDIR -name "*results.csv")

#If the file was found, add it to the manifest list
if [ -n "$image1_tif" ] ; then
  manifest_files+=("{\"name\":\"output_file_tif\", \"path\":\"$image1_tif\"}")
fi

if [ -n "$image2_tif" ] ; then
  manifest_files+=("{\"name\":\"output_file_tif2\", \"path\":\"$image2_tif2\"}")
fi

if [ -n "$csv_data" ] ; then
  manifest_files+=("{\"name\":\"output_file_csv\", \"path\":\"$csv_data\"}")
fi


manifest_files_text=$(join , "${manifest_files[@]}")

results_manifest_text={\"version\":\"1.0\",\"files\":[$manifest_files_text]}
echo "$results_manifest_text" > $OUTDIR/results_manifest.json

Python

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
import json
from glob import glob

def generateResultsManifest(outdir):

    try:
        outputCSV = glob(os.path.join(outdir, '*results.csv'))[0]
        outputImage1 = glob(os.path.join(outdir, '*image1.tif'))[0]
        outputImage2 = glob(os.path.join(outdir, '*image2.tif'))[0]
    except:
        #Error in finding results
        sys.exit(5)

    jsonDict={}
    jsonDict['version'] = '1.1'
    jsonDict['output_data'] = []

    tempDict = {}
    tempDict['name'] = 'output_file_tif'
    tempDict['file'] = {'path': outputImage1}
    jsonDict['output_data'].append(tempDict)

    tempDict = {}
    tempDict['name'] = 'output_file_tif2'
    tempDict['file'] = {'path': outputImage2}
    jsonDict['output_data'].append(tempDict)

    tempDict = {}
    tempDict['name'] = 'output_file_csv'
    tempDict['file'] = {'path': outputCSV}
    jsonDict['output_data'].append(tempDict)

    with open(os.path.join(outdir, 'results_manifest.json'), 'w') as fout:
        jsonString = json.dumps(jsonDict)
        fout.write(jsonString)

See the example algorithms for additional examples.