Capturing your algorithm outputs with a Scale Results Manifest¶
Generating a results manifest should either be done within your algorithm or within the algorithm’s wrapper script. Once an algorithm is complete a results manifest file is used to convey what products should be archived by Scale and passed onto other algorithms. It is HIGHLY recommended that the algorithm write out the manifest JSON file as prescribed in the Scale documentation.
Click here for results manifest specification: Results Manifest
Below are examples of wrapper scripts if you cannot modify the algorithm. These should only be used if you do not have access to the source code for your algorithm.
ONLY FILES IN THE RESULTS MANIFEST WILL BE SAVED BY SCALE
File paths listed in the results manifest must be absolute paths to the file and the files must exist within the output directory provided by Scale. The results manifest must be called “results_manifest.json” and must be present in the root of the output directory. All output files and the results manifest must have full read permission (as well as execute for any directories) granted for ALL users, allowing the Scale post task to read the outputs. The output names in the results manifest must match the output_data field in the job interface.
Simple Results Manifest Example¶
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | {
"version": "1.1",
"output_data": [
{
"name" : "output_file_csv",
"file": {
"path" : "/tmp/job_exe_231/outputs/results.csv"
}
},
{
"name" : "output_file_tif",
"file": {
"path" : "/tmp/job_exe_231/outputs/myimage1.tif"
}
},
{
"name" : "output_file_tif2",
"file": {
"path" : "/tmp/job_exe_231/outputs/myimage2.tif"
}
}
]
}
|
Example Code for generating results manifests¶
Shell Script¶
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | # Gather the output files and put them in a manifest
function join { local IFS="$1"; shift; echo "$*"; }
manifest_files=()
#Find the files in the output directory
image1_tif=$(find $OUTDIR -name "*image1.tif")
image2_tif=$(find $OUTDIR -name "*image2.tif")
csv_data=$(find $OUTDIR -name "*results.csv")
#If the file was found, add it to the manifest list
if [ -n "$image1_tif" ] ; then
manifest_files+=("{\"name\":\"output_file_tif\", \"path\":\"$image1_tif\"}")
fi
if [ -n "$image2_tif" ] ; then
manifest_files+=("{\"name\":\"output_file_tif2\", \"path\":\"$image2_tif2\"}")
fi
if [ -n "$csv_data" ] ; then
manifest_files+=("{\"name\":\"output_file_csv\", \"path\":\"$csv_data\"}")
fi
manifest_files_text=$(join , "${manifest_files[@]}")
results_manifest_text={\"version\":\"1.0\",\"files\":[$manifest_files_text]}
echo "$results_manifest_text" > $OUTDIR/results_manifest.json
|
Python¶
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | import json
from glob import glob
def generateResultsManifest(outdir):
try:
outputCSV = glob(os.path.join(outdir, '*results.csv'))[0]
outputImage1 = glob(os.path.join(outdir, '*image1.tif'))[0]
outputImage2 = glob(os.path.join(outdir, '*image2.tif'))[0]
except:
#Error in finding results
sys.exit(5)
jsonDict={}
jsonDict['version'] = '1.1'
jsonDict['output_data'] = []
tempDict = {}
tempDict['name'] = 'output_file_tif'
tempDict['file'] = {'path': outputImage1}
jsonDict['output_data'].append(tempDict)
tempDict = {}
tempDict['name'] = 'output_file_tif2'
tempDict['file'] = {'path': outputImage2}
jsonDict['output_data'].append(tempDict)
tempDict = {}
tempDict['name'] = 'output_file_csv'
tempDict['file'] = {'path': outputCSV}
jsonDict['output_data'].append(tempDict)
with open(os.path.join(outdir, 'results_manifest.json'), 'w') as fout:
jsonString = json.dumps(jsonDict)
fout.write(jsonString)
|
See the example algorithms for additional examples.