gen_dl_model_heatmapT_gen_dl_model_heatmapGenDlModelHeatmapGenDlModelHeatmap (Operator)

Name

gen_dl_model_heatmapT_gen_dl_model_heatmapGenDlModelHeatmapGenDlModelHeatmap — Infer the sample and generate a heatmap.

Signature

gen_dl_model_heatmap( : : DLModelHandle, DLSample, HeatmapMethod, TargetClasses, GenParam : DLResult)

Herror T_gen_dl_model_heatmap(const Htuple DLModelHandle, const Htuple DLSample, const Htuple HeatmapMethod, const Htuple TargetClasses, const Htuple GenParam, Htuple* DLResult)

void GenDlModelHeatmap(const HTuple& DLModelHandle, const HTuple& DLSample, const HTuple& HeatmapMethod, const HTuple& TargetClasses, const HTuple& GenParam, HTuple* DLResult)

HDictArray HDlModel::GenDlModelHeatmap(const HDictArray& DLSample, const HString& HeatmapMethod, const HTuple& TargetClasses, const HDict& GenParam) const

HDictArray HDlModel::GenDlModelHeatmap(const HDictArray& DLSample, const char* HeatmapMethod, const HTuple& TargetClasses, const HDict& GenParam) const

HDictArray HDlModel::GenDlModelHeatmap(const HDictArray& DLSample, const wchar_t* HeatmapMethod, const HTuple& TargetClasses, const HDict& GenParam) const   (Windows only)

static void HOperatorSet.GenDlModelHeatmap(HTuple DLModelHandle, HTuple DLSample, HTuple heatmapMethod, HTuple targetClasses, HTuple genParam, out HTuple DLResult)

HDict[] HDlModel.GenDlModelHeatmap(HDict[] DLSample, string heatmapMethod, HTuple targetClasses, HDict genParam)

Description

The operator gen_dl_model_heatmapgen_dl_model_heatmapGenDlModelHeatmapGenDlModelHeatmapGenDlModelHeatmap infers the sample given by DLSampleDLSampleDLSampleDLSampleDLSample and generates a heatmap for the class specified by TargetClassesTargetClassesTargetClassesTargetClassestargetClasses. DLSampleDLSampleDLSampleDLSampleDLSample can be a single dictionary or a tuple of dictionaries for different samples. To do so, the operator uses the deep learning model given by DLModelHandleDLModelHandleDLModelHandleDLModelHandleDLModelHandle. The parameter HeatmapMethodHeatmapMethodHeatmapMethodHeatmapMethodheatmapMethod determines, which method is used to calculate the heatmap. The operator returns DLResultDLResultDLResultDLResultDLResult with a result dictionary for every given sample. Note, DLResultDLResultDLResultDLResultDLResult will be newly created and already existing parameters with this name will be overwritten.

A heatmap can be useful to analyze which parts of an image have a strong influence for the inference into a certain class. Shape and area of these parts and therewith the heatmaps may vary widely for differing networks especially if their architectures differ. There are different methods how a heatmap is calculated. The following ones can be selected using HeatmapMethodHeatmapMethodHeatmapMethodHeatmapMethodheatmapMethod:

'grad_cam'"grad_cam""grad_cam""grad_cam""grad_cam":

Gradient-weighted Class Activation Mapping (Grad-CAM). See reference cited below for further information.

The input parameter TargetClassesTargetClassesTargetClassesTargetClassestargetClasses determines the target class, the class for which the heatmap is generated. As value the class ID has to be set. Alternatively, an empty tuple can be handed over. In this case, the heatmap is calculated for the class with the highest confidence value, thus the top inferred class.

The following entries can be set in the dictionary GenParamGenParamGenParamGenParamgenParam:

target_conv_layertarget_conv_layertarget_conv_layertarget_conv_layertargetConvLayer:

Specifies the convolution layer whose activation and gradient are used for the heatmap. The convolution layer can be specified using its NAME, as returned using 'summary'"summary""summary""summary""summary" in get_dl_model_paramget_dl_model_paramGetDlModelParamGetDlModelParamGetDlModelParam, e.g., 'conv10'"conv10""conv10""conv10""conv10". As default, the last (deepest) convolution layer of the network with a non-trivial activation (width and height not equal to 1) is used. As a general rule, the activation of the deepest convolution layer is most suitable for calculating the heatmap, therefore it is recommended to keep the default layer.

use_conv_onlyuse_conv_onlyuse_conv_onlyuse_conv_onlyuseConvOnly:

In case the convolution layer (selected with target_conv_layertarget_conv_layertarget_conv_layertarget_conv_layertargetConvLayer) is fused with a directly following ReLU activation layer, this parameter can be used to determine, from which of these layers the activation and gradient will be used for the heatmap. The following values are supported:

'true'"true""true""true""true": convolution layer

'false'"false""false""false""false": ReLU layer

As default, use_conv_onlyuse_conv_onlyuse_conv_onlyuse_conv_onlyuseConvOnly is set to 'true'"true""true""true""true".

scalingscalingscalingscalingscaling:

Sets the scaling method for the heatmap. The following values are supported:

'scale_after_relu'"scale_after_relu""scale_after_relu""scale_after_relu""scale_after_relu": Negative values of the heatmap are set to 0 and then all values are scaled within the range [0,1]. As a consequence, areas within the heatmap can attain very high values close to 1 although their contribution to the classification result might be small.

'scale_before_relu'"scale_before_relu""scale_before_relu""scale_before_relu""scale_before_relu": All values of the heatmap are divided by the maximum absolute value and then negative values are set to 0. This leads to values for the heatmap in [0,1]. However, areas of less activation (and hence with a small contribution to the classification result) may not tend to attain values near 1.

'none'"none""none""none""none": The heatmap values are not scaled.

As default, scalingscalingscalingscalingscaling is set to 'scale_after_relu'"scale_after_relu""scale_after_relu""scale_after_relu""scale_after_relu".

Every output dictionary in DLResultDLResultDLResultDLResultDLResult contains the inference results as obtained using apply_dl_modelapply_dl_modelApplyDlModelApplyDlModelApplyDlModel. Additionally it includes a nested dictionary under the key heatmap_methodheatmap_methodheatmap_methodheatmap_methodheatmapMethod, where 'method' is the name of the specified method as given in HeatmapMethodHeatmapMethodHeatmapMethodHeatmapMethodheatmapMethod. In this nested dictionary the heatmap is saved under the key heatmap_image_class_classIDheatmap_image_class_classIDheatmap_image_class_classIDheatmap_image_class_classIDheatmapImageClassClassID, where classID is the ID of the target class.

Attention

To run this operator, cuDNN and cuBLAS are required and 'runtime'"runtime""runtime""runtime""runtime" has to be set to 'gpu'"gpu""gpu""gpu""gpu", see get_dl_model_paramget_dl_model_paramGetDlModelParamGetDlModelParamGetDlModelParam. For further details, please refer to the “Installation Guide”, paragraph “Requirements for Deep Learning”.

The heatmap should be used as a tool for visualizing and better understanding classification results. It is not intended as a segmentation tool. Moreover, gen_dl_model_heatmapgen_dl_model_heatmapGenDlModelHeatmapGenDlModelHeatmapGenDlModelHeatmap currently only supports models with 'type'"type""type""type""type"='classification'"classification""classification""classification""classification".

Execution Information

Parameters

DLModelHandleDLModelHandleDLModelHandleDLModelHandleDLModelHandle (input_control)  dl_model HDlModel, HTupleHTupleHtuple (handle) (IntPtr) (HHandle) (handle)

Handle of a Deep learning model.

DLSampleDLSampleDLSampleDLSampleDLSample (input_control)  dict-array HDict, HTupleHTupleHtuple (handle) (IntPtr) (HHandle) (handle)

Dictionaries with the sample input data.

HeatmapMethodHeatmapMethodHeatmapMethodHeatmapMethodheatmapMethod (input_control)  string HTupleHTupleHtuple (string) (string) (HString) (char*)

Method to be used for the heatmap calculation.

Default value: 'grad_cam' "grad_cam" "grad_cam" "grad_cam" "grad_cam"

TargetClassesTargetClassesTargetClassesTargetClassestargetClasses (input_control)  tuple(-array) HTupleHTupleHtuple (integer) (int / long) (Hlong) (Hlong)

ID of the target class.

Default value: []

GenParamGenParamGenParamGenParamgenParam (input_control)  dict HDict, HTupleHTupleHtuple (handle) (IntPtr) (HHandle) (handle)

Dictionary for generic parameters.

Default value: []

DLResultDLResultDLResultDLResultDLResult (output_control)  dict(-array) HDict, HTupleHTupleHtuple (handle) (IntPtr) (HHandle) (handle)

Dicitonaries with the result data.

Possible Predecessors

apply_dl_modelapply_dl_modelApplyDlModelApplyDlModelApplyDlModel

References

R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra: "Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization," 2017 IEEE International Conference on Computer Vision (ICCV), Venice, 2017, pp. 618-626. doi: 10.1109/ICCV.2017.74

Module

Deep Learning Inference