gen_dl_model_heatmapT_gen_dl_model_heatmapGenDlModelHeatmapGenDlModelHeatmapgen_dl_model_heatmap (Operator)

Name

gen_dl_model_heatmapT_gen_dl_model_heatmapGenDlModelHeatmapGenDlModelHeatmapgen_dl_model_heatmap — Infer the sample and generate a heatmap.

Signature

gen_dl_model_heatmap( : : DLModelHandle, DLSample, HeatmapMethod, TargetClasses, GenParam : DLResult)

Herror T_gen_dl_model_heatmap(const Htuple DLModelHandle, const Htuple DLSample, const Htuple HeatmapMethod, const Htuple TargetClasses, const Htuple GenParam, Htuple* DLResult)

void GenDlModelHeatmap(const HTuple& DLModelHandle, const HTuple& DLSample, const HTuple& HeatmapMethod, const HTuple& TargetClasses, const HTuple& GenParam, HTuple* DLResult)

HDictArray HDlModel::GenDlModelHeatmap(const HDictArray& DLSample, const HString& HeatmapMethod, const HTuple& TargetClasses, const HDict& GenParam) const

HDictArray HDlModel::GenDlModelHeatmap(const HDictArray& DLSample, const char* HeatmapMethod, const HTuple& TargetClasses, const HDict& GenParam) const

HDictArray HDlModel::GenDlModelHeatmap(const HDictArray& DLSample, const wchar_t* HeatmapMethod, const HTuple& TargetClasses, const HDict& GenParam) const   (Windows only)

static void HOperatorSet.GenDlModelHeatmap(HTuple DLModelHandle, HTuple DLSample, HTuple heatmapMethod, HTuple targetClasses, HTuple genParam, out HTuple DLResult)

HDict[] HDlModel.GenDlModelHeatmap(HDict[] DLSample, string heatmapMethod, HTuple targetClasses, HDict genParam)

def gen_dl_model_heatmap(dlmodel_handle: HHandle, dlsample: Sequence[HHandle], heatmap_method: str, target_classes: MaybeSequence[int], gen_param: HHandle) -> Sequence[HHandle]

def gen_dl_model_heatmap_s(dlmodel_handle: HHandle, dlsample: Sequence[HHandle], heatmap_method: str, target_classes: MaybeSequence[int], gen_param: HHandle) -> HHandle

Description

The operator gen_dl_model_heatmapgen_dl_model_heatmapGenDlModelHeatmapGenDlModelHeatmapGenDlModelHeatmapgen_dl_model_heatmap infers the sample given by DLSampleDLSampleDLSampleDLSampleDLSampledlsample and generates a heatmap for the class specified by TargetClassesTargetClassesTargetClassesTargetClassestargetClassestarget_classes. DLSampleDLSampleDLSampleDLSampleDLSampledlsample can be a single dictionary or a tuple of dictionaries for different samples. To do so, the operator uses the deep learning model given by DLModelHandleDLModelHandleDLModelHandleDLModelHandleDLModelHandledlmodel_handle. The parameter HeatmapMethodHeatmapMethodHeatmapMethodHeatmapMethodheatmapMethodheatmap_method determines, which method is used to calculate the heatmap. The operator returns DLResultDLResultDLResultDLResultDLResultdlresult with a result dictionary for every given sample. Note, DLResultDLResultDLResultDLResultDLResultdlresult will be newly created and already existing parameters with this name will be overwritten.

A heatmap can be useful to analyze which parts of an image have a strong influence for the inference into a certain class. Shape and area of these parts and therewith the heatmaps may vary widely for differing networks especially if their architectures differ. There are different methods how a heatmap is calculated. The following ones can be selected using HeatmapMethodHeatmapMethodHeatmapMethodHeatmapMethodheatmapMethodheatmap_method:

'grad_cam'"grad_cam""grad_cam""grad_cam""grad_cam""grad_cam":

Gradient-weighted Class Activation Mapping (Grad-CAM). See reference cited below for further information.

The input parameter TargetClassesTargetClassesTargetClassesTargetClassestargetClassestarget_classes determines the target class, the class for which the heatmap is generated. As value the class ID has to be set. Alternatively, an empty tuple can be handed over. In this case, the heatmap is calculated for the class with the highest confidence value, thus the top inferred class. Currently, TargetClassesTargetClassesTargetClassesTargetClassestargetClassestarget_classes only supports a single value or an empty tuple.

The following entries can be set in the dictionary GenParamGenParamGenParamGenParamgenParamgen_param:

target_conv_layer:

Specifies the convolution layer whose activation and gradient are used for the heatmap. The convolution layer can be specified using its NAME, as returned using 'summary'"summary""summary""summary""summary""summary" in get_dl_model_paramget_dl_model_paramGetDlModelParamGetDlModelParamGetDlModelParamget_dl_model_param, e.g., 'conv10'"conv10""conv10""conv10""conv10""conv10". As default, the last (deepest) convolution layer of the network with a non-trivial activation (width and height not equal to 1) is used. As a general rule, the activation of the deepest convolution layer is most suitable for calculating the heatmap, therefore it is recommended to keep the default layer.

use_conv_only:

In case the convolution layer (selected with target_conv_layer) is fused with a directly following ReLU activation layer, this parameter can be used to determine, from which of these layers the activation and gradient will be used for the heatmap. The following values are supported:

'true'"true""true""true""true""true": convolution layer

'false'"false""false""false""false""false": ReLU layer

As default, use_conv_only is set to 'true'"true""true""true""true""true".

scaling:

Sets the scaling method for the heatmap. The following values are supported:

'scale_after_relu'"scale_after_relu""scale_after_relu""scale_after_relu""scale_after_relu""scale_after_relu": Negative values of the heatmap are set to 0 and then all values are scaled within the range [0,1]. As a consequence, areas within the heatmap can attain very high values close to 1 although their contribution to the classification result might be small.

'scale_before_relu'"scale_before_relu""scale_before_relu""scale_before_relu""scale_before_relu""scale_before_relu": All values of the heatmap are divided by the maximum absolute value and then negative values are set to 0. This leads to values for the heatmap in [0,1]. However, areas of less activation (and hence with a small contribution to the classification result) may not tend to attain values near 1.

'none'"none""none""none""none""none": The heatmap values are not scaled.

As default, scaling is set to 'scale_after_relu'"scale_after_relu""scale_after_relu""scale_after_relu""scale_after_relu""scale_after_relu".

Every output dictionary in DLResultDLResultDLResultDLResultDLResultdlresult contains the inference results as obtained using apply_dl_modelapply_dl_modelApplyDlModelApplyDlModelApplyDlModelapply_dl_model. Additionally it includes a nested dictionary under the key heatmap_method, where method is the name of the specified method as given in HeatmapMethodHeatmapMethodHeatmapMethodHeatmapMethodheatmapMethodheatmap_method. In this nested dictionary the heatmap is saved under the key heatmap_image_class_classID, where classID is the ID of the target class.

Attention

System requirements: To run this operator on GPU by setting 'runtime'"runtime""runtime""runtime""runtime""runtime" to 'gpu'"gpu""gpu""gpu""gpu""gpu" (see get_dl_model_paramget_dl_model_paramGetDlModelParamGetDlModelParamGetDlModelParamget_dl_model_param), cuDNN and cuBLAS are required. For further details, please refer to the “Installation Guide”, paragraph “Requirements for Deep Learning and Deep-Learning-Based Methods”. Alternatively, this operator can also be run on CPU by setting 'runtime'"runtime""runtime""runtime""runtime""runtime" to 'cpu'"cpu""cpu""cpu""cpu""cpu".

The heatmap should be used as a tool for visualizing and better understanding classification results. It is not intended as a segmentation tool. Moreover, gen_dl_model_heatmapgen_dl_model_heatmapGenDlModelHeatmapGenDlModelHeatmapGenDlModelHeatmapgen_dl_model_heatmap currently only supports models with 'type'"type""type""type""type""type"='classification'"classification""classification""classification""classification""classification".

Execution Information

Parameters

DLModelHandleDLModelHandleDLModelHandleDLModelHandleDLModelHandledlmodel_handle (input_control)  dl_model HDlModel, HTupleHHandleHTupleHtuple (handle) (IntPtr) (HHandle) (handle)

Handle of a Deep learning model.

DLSampleDLSampleDLSampleDLSampleDLSampledlsample (input_control)  dict-array HDict, HTupleSequence[HHandle]HTupleHtuple (handle) (IntPtr) (HHandle) (handle)

Dictionaries with the sample input data.

HeatmapMethodHeatmapMethodHeatmapMethodHeatmapMethodheatmapMethodheatmap_method (input_control)  string HTuplestrHTupleHtuple (string) (string) (HString) (char*)

Method to be used for the heatmap calculation.

Default value: 'grad_cam' "grad_cam" "grad_cam" "grad_cam" "grad_cam" "grad_cam"

TargetClassesTargetClassesTargetClassesTargetClassestargetClassestarget_classes (input_control)  tuple(-array) HTupleMaybeSequence[int]HTupleHtuple (integer) (int / long) (Hlong) (Hlong)

ID of the target class.

Default value: []

GenParamGenParamGenParamGenParamgenParamgen_param (input_control)  dict HDict, HTupleHHandleHTupleHtuple (handle) (IntPtr) (HHandle) (handle)

Dictionary for generic parameters.

Default value: []

DLResultDLResultDLResultDLResultDLResultdlresult (output_control)  dict(-array) HDict, HTupleSequence[HHandle]HTupleHtuple (handle) (IntPtr) (HHandle) (handle)

Dictionaries with the result data.

Possible Predecessors

apply_dl_modelapply_dl_modelApplyDlModelApplyDlModelApplyDlModelapply_dl_model

References

R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra: "Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization," 2017 IEEE International Conference on Computer Vision (ICCV), Venice, 2017, pp. 618-626. doi: 10.1109/ICCV.2017.74

Module

Deep Learning Inference