Skip to main content
`Ctrl`+`K`

# SGD

MIOpen: SGD
Functions
SGD

## Functions

miopenStatus_t

miopenFusedAdam (miopenHandle_t handle, const miopenTensorDescriptor_t paramDesc, void *param, const miopenTensorDescriptor_t gradDesc, const void *grad, const miopenTensorDescriptor_t expAvgDesc, void *expAvg, const miopenTensorDescriptor_t expAvgSqDesc, void *expAvgSq, const miopenTensorDescriptor_t maxExpAvgSqDesc, void *maxExpAvgSq, const miopenTensorDescriptor_t stateStepDesc, void *stateStep, const unsigned int state_step, const float lr, const float beta1, const float beta2, const float weight_decay, const float eps, const bool amsgrad, const bool maximize, const bool adamw, const miopenTensorDescriptor_t gradScaleDesc, const void *gradScale, const miopenTensorDescriptor_t foundInfDesc, const void *foundInf)

| Perform Fused Adam optimization for a single tensor (Adaptive Moment Estimation).

miopenStatus_t

miopenFusedAdamWithOutput (miopenHandle_t handle, const miopenTensorDescriptor_t paramInDesc, void *paramIn, const miopenTensorDescriptor_t paramOutDesc, void *paramOut, const miopenTensorDescriptor_t paramOutFloat16Desc, void *paramOutFloat16, const miopenTensorDescriptor_t gradInDesc, const void *gradIn, const miopenTensorDescriptor_t expAvgInDesc, void *expAvgIn, const miopenTensorDescriptor_t expAvgOutDesc, void *expAvgOut, const miopenTensorDescriptor_t expAvgSqInDesc, void *expAvgSqIn, const miopenTensorDescriptor_t expAvgSqOutDesc, void *expAvgSqOut, const miopenTensorDescriptor_t maxExpAvgSqInDesc, void *maxExpAvgSqIn, const miopenTensorDescriptor_t maxExpAvgSqOutDesc, void *maxExpAvgSqOut, const miopenTensorDescriptor_t stateStepInDesc, void *stateStepIn, const miopenTensorDescriptor_t stateStepOutDesc, void *stateStepOut, const unsigned int state_step, const float lr, const float beta1, const float beta2, const float weight_decay, const float eps, const bool amsgrad, const bool maximize, const bool adamw, const miopenTensorDescriptor_t gradScaleDesc, const void *gradScale, const miopenTensorDescriptor_t foundInfDesc, const void *foundInf)

| Execute single tensor Adam optimization and receive the result in a separate output tensor.

miopenStatus_t

miopenTransformersAdamW (miopenHandle_t handle, const miopenTensorDescriptor_t paramDesc, void *param, const miopenTensorDescriptor_t gradDesc, const void *grad, const miopenTensorDescriptor_t expAvgDesc, void *expAvg, const miopenTensorDescriptor_t expAvgSqDesc, void *expAvgSq, const miopenTensorDescriptor_t stateStepDesc, void *stateStep, const unsigned int state_step, const float lr, const float beta1, const float beta2, const float weight_decay, const float eps, const bool correct_bias, const miopenTensorDescriptor_t gradScaleDesc, const void *gradScale, const miopenTensorDescriptor_t foundInfDesc, const void *foundInf)

| Implements Adam algorithm with weight decay fix as introduced in Decoupled Weight Decay Regularization. This is the fused kernel version of AdamW included in the Hugging Face Transformers module.

miopenStatus_t

miopenTransformersAdamWWithOutput (miopenHandle_t handle, const miopenTensorDescriptor_t paramInDesc, void *paramIn, const miopenTensorDescriptor_t paramOutDesc, void *paramOut, const miopenTensorDescriptor_t paramOutFloat16Desc, void *paramOutFloat16, const miopenTensorDescriptor_t gradInDesc, const void *gradIn, const miopenTensorDescriptor_t expAvgInDesc, void *expAvgIn, const miopenTensorDescriptor_t expAvgOutDesc, void *expAvgOut, const miopenTensorDescriptor_t expAvgSqInDesc, void *expAvgSqIn, const miopenTensorDescriptor_t expAvgSqOutDesc, void *expAvgSqOut, const miopenTensorDescriptor_t stateStepInDesc, void *stateStepIn, const miopenTensorDescriptor_t stateStepOutDesc, void *stateStepOut, const unsigned int state_step, const float lr, const float beta1, const float beta2, const float weight_decay, const float eps, const float step_size, const bool correct_bias, const miopenTensorDescriptor_t gradScaleDesc, const void *gradScale, const miopenTensorDescriptor_t foundInfDesc, const void *foundInf)

| Execute single tensor Adam optimization and receive the result in a separate output tensor.

## Detailed Description

## Function Documentation

## ◆ miopenFusedAdam()

miopenStatus_t miopenFusedAdam | ( | miopenHandle_t | handle,
|  | const miopenTensorDescriptor_t | paramDesc,
|  | void * | param,
|  | const miopenTensorDescriptor_t | gradDesc,
|  | const void * | grad,
|  | const miopenTensorDescriptor_t | expAvgDesc,
|  | void * | expAvg,
|  | const miopenTensorDescriptor_t | expAvgSqDesc,
|  | void * | expAvgSq,
|  | const miopenTensorDescriptor_t | maxExpAvgSqDesc,
|  | void * | maxExpAvgSq,
|  | const miopenTensorDescriptor_t | stateStepDesc,
|  | void * | stateStep,
|  | const unsigned int | state_step,
|  | const float | lr,
|  | const float | beta1,
|  | const float | beta2,
|  | const float | weight_decay,
|  | const float | eps,
|  | const bool | amsgrad,
|  | const bool | maximize,
|  | const bool | adamw,
|  | const miopenTensorDescriptor_t | gradScaleDesc,
|  | const void * | gradScale,
|  | const miopenTensorDescriptor_t | foundInfDesc,
|  | const void * | foundInf )

Perform Fused Adam optimization for a single tensor (Adaptive Moment Estimation).

This function implements the Fused Adam optimization algorithm. Adam, short for Adaptive Moment Estimation, extends the RMSProp optimizer. It combines the advantages of AdaGrad and RMSProp by adaptively adjusting learning rates for each parameter using the first and second moments of gradients. Fused Adam optimization efficiently combines multiple operations into a single kernel, reducing memory access overhead and improving performance.

Additionally, Fused Adam can be utilized in both adam w and Automatic Mixed Precision (AMP), enabling accelerated model training and reduced memory consumption. AMP supports FP16 computation, optimizing model calculations using a mixture of FP32 and FP16 precision to enhance training speed. When utilizing AMP, FoundInf, ScaleGrad, and step tensors should be employed. In AMP mode, the execution of Adam is determined based on the FoundInf value. State Step accepts both int values and int tensors. If a Step tensor is employed, the step received as an int is disregarded, and if Adam is executed, the step tensor is incremented by 1.

// Execute Adam
miopenFusedAdam(handle,
paramDesc,
param,
gradDesc,
grad,
expAvgDesc,
expAvg,
expAvgSqDesc,
expAvgSq,
NULL, // Unused maxExpAvgSqDesc because amsgrad is false
NULL,
NULL, // Unused stateStep Tensor because use step integer argument
NULL,
step,
lr,
beta1,
beta2,
weight_decay,
eps,
false, // amsgrad
false, // maximize
false, // adamw
NULL, // Unused gradScale Tensor because not amp
NULL,
NULL, // Unused foundInf Tensor because not amp
NULL);
// Execute AdamW
miopenFusedAdam(handle,
paramDesc,
param,
gradDesc,
grad,
expAvgDesc,
expAvg,
expAvgSqDesc,
expAvgSq,
NULL, // Unused maxExpAvgSqDesc because amsgrad is false
NULL,
NULL, // Unused stateStep Tensor because use step integer argument
NULL,
step,
lr,
beta1,
beta2,
weight_decay,
eps,
false, // amsgrad
false, // maximize
true, // adamw
NULL, // Unused gradScale Tensor because not amp
NULL,
NULL, // Unused foundInf Tensor because not amp
NULL);
// Execute AMP Adam
miopenFusedAdam(handle,
paramDesc,
param,
gradDesc,
grad,
expAvgDesc,
expAvg,
expAvgSqDesc,
expAvgSq,
NULL, // Unused maxExpAvgSqDesc because amsgrad is false
NULL,
stateStepDesc,
stateStep,
-1, // Ignore step value because stateStep Tensor is used
lr,
beta1,
beta2,
weight_decay,
eps,
false, // amsgrad
false, // maximize
false, // adamw
gradScaleDesc,
gradScale,
foundInfDesc,
foundInf);
miopenFusedAdam
miopenStatus_t miopenFusedAdam(miopenHandle_t handle, const miopenTensorDescriptor_t paramDesc, void *param, const miopenTensorDescriptor_t gradDesc, const void *grad, const miopenTensorDescriptor_t expAvgDesc, void *expAvg, const miopenTensorDescriptor_t expAvgSqDesc, void *expAvgSq, const miopenTensorDescriptor_t maxExpAvgSqDesc, void *maxExpAvgSq, const miopenTensorDescriptor_t stateStepDesc, void *stateStep, const unsigned int state_step, const float lr, const float beta1, const float beta2, const float weight_decay, const float eps, const bool amsgrad, const bool maximize, const bool adamw, const miopenTensorDescriptor_t gradScaleDesc, const void *gradScale, const miopenTensorDescriptor_t foundInfDesc, const void *foundInf)
Perform Fused Adam optimization for a single tensor (Adaptive Moment Estimation).

Parameters

  handle | MIOpen handle (input)
  paramDesc | Tensor descriptor for the input parameter tensor (input)
  param | Input parameter tensor (input)
  gradDesc | Tensor descriptor for the input gradient tensor (input)
  grad | Input gradient tensor (input)
  expAvgDesc | Tensor descriptor for the input exponential moving average tensor (input)
  expAvg | Input exponential moving average tensor (input)
  expAvgSqDesc | Tensor descriptor for the input exponential moving average squared tensor (input)
  expAvgSq | Input exponential moving average squared tensor (input)
  maxExpAvgSqDesc | Tensor descriptor for the input maximum exponential moving average squared tensor. Used when amsgrad is true (input, optional)
  maxExpAvgSq | Input maximum exponential moving average squared tensor. Used when amsgrad is true (input, optional)
  stateStepDesc | Tensor descriptor for the input state step tensor (input)
  stateStep | Input state step tensor (input)
  state_step | Input state step. used when the step tensor is null (input)
  lr | Learning rate (input)
  beta1 | Coefficient used for computing the first moment running average of gradient (input)
  beta2 | Coefficient used for computing the second moment running average of gradient (input)
  weight_decay | Weight decay (input)
  eps | Term added to the denominator to improve numerical stability (input)
  amsgrad | Flag indicating whether to use the AMSGrad variant of Adam (input)
  maximize | Flag indicating whether to maximize the objective with respect to the parameters (input)
  adamw | If true, the operation becomes AdamW (input)
  gradScaleDesc | Tensor descriptor for the input grad scale tensor (input, optional)
  gradScale | Input grad scale tensor (input, optional)
  foundInfDesc | Tensor descriptor for the input found inf tensor (input, optional)
  foundInf | Tensor indicating the presence of inf or NaN in gradients. If true, skips operation and step update (input, optional)

Returns
  miopenStatus_t

## ◆ miopenFusedAdamWithOutput()

miopenStatus_t miopenFusedAdamWithOutput | ( | miopenHandle_t | handle,
|  | const miopenTensorDescriptor_t | paramInDesc,
|  | void * | paramIn,
|  | const miopenTensorDescriptor_t | paramOutDesc,
|  | void * | paramOut,
|  | const miopenTensorDescriptor_t | paramOutFloat16Desc,
|  | void * | paramOutFloat16,
|  | const miopenTensorDescriptor_t | gradInDesc,
|  | const void * | gradIn,
|  | const miopenTensorDescriptor_t | expAvgInDesc,
|  | void * | expAvgIn,
|  | const miopenTensorDescriptor_t | expAvgOutDesc,
|  | void * | expAvgOut,
|  | const miopenTensorDescriptor_t | expAvgSqInDesc,
|  | void * | expAvgSqIn,
|  | const miopenTensorDescriptor_t | expAvgSqOutDesc,
|  | void * | expAvgSqOut,
|  | const miopenTensorDescriptor_t | maxExpAvgSqInDesc,
|  | void * | maxExpAvgSqIn,
|  | const miopenTensorDescriptor_t | maxExpAvgSqOutDesc,
|  | void * | maxExpAvgSqOut,
|  | const miopenTensorDescriptor_t | stateStepInDesc,
|  | void * | stateStepIn,
|  | const miopenTensorDescriptor_t | stateStepOutDesc,
|  | void * | stateStepOut,
|  | const unsigned int | state_step,
|  | const float | lr,
|  | const float | beta1,
|  | const float | beta2,
|  | const float | weight_decay,
|  | const float | eps,
|  | const bool | amsgrad,
|  | const bool | maximize,
|  | const bool | adamw,
|  | const miopenTensorDescriptor_t | gradScaleDesc,
|  | const void * | gradScale,
|  | const miopenTensorDescriptor_t | foundInfDesc,
|  | const void * | foundInf )

Execute single tensor Adam optimization and receive the result in a separate output tensor.

This function is equivalent to miopenFusedAdam but receives the result in a separate output tensor.

See also
  miopenFusedAdam

// Execute Adam
miopenFusedAdamWithOutput(handle,
paramInDesc,
paramIn,
paramOutDesc,
paramOut,
NULL, // Unused paramOutFloat16 tensor because is not amp
NULL,
gradInDesc,
gradIn,
expAvgInDesc,
expAvgIn,
expAvgOutDesc,
expAvgOut,
expAvgInSqDesc,
expAvgSqIn,
expAvgSqOutDesc,
expAvgSqOut,
NULL, // Unused maxExpAvgSqIn tensor because amsgrad is false
NULL,
NULL, // Unused maxExpAvgSqOut tensor because amsgrad is false
NULL,
NULL, // Unused stateStepIn tensor because use step integer argument
NULL,
NULL, // Unused stateStepOut tensor because use step integer argument
NULL,
step,
lr,
beta1,
beta2,
weight_decay,
eps,
false, // amsgrad
false, // maximize
false, // adamw
NULL, // Unused gradScale Tensor because not amp
NULL,
NULL, // Unused foundInf Tensor because not amp
NULL);
// Execute Amp Adam
miopenFusedAdamWithOutput(handle,
paramInDesc,
paramIn,
paramOutDesc,
paramOut,
paramOutFloat16Desc, // paramOutFloat16 tensor is optional in amp
paramOutFloat16,
gradInDesc,
gradIn,
expAvgInDesc,
expAvgIn,
expAvgOutDesc,
expAvgOut,
expAvgInSqDesc,
expAvgSqIn,
expAvgSqIn,
expAvgSqOutDesc,
expAvgSqOut,
NULL, // Unused maxExpAvgSqIn tensor because amsgrad is false
NULL,
NULL, // Unused maxExpAvgSqOut tensor because amsgrad is false
NULL,
stateStepInDesc,
stateStepIn,
stateStepOutDesc,
stateStepOut
-1, // Ignore step value because stateStep Tensor is used
lr, beta1, beta2, weight_decay, eps,
false, // amsgrad
false, // maximize
false, // adamw
gradScaleDesc,
gradScale,
foundInfDesc,
foundInf);
miopenFusedAdamWithOutput
miopenStatus_t miopenFusedAdamWithOutput(miopenHandle_t handle, const miopenTensorDescriptor_t paramInDesc, void *paramIn, const miopenTensorDescriptor_t paramOutDesc, void *paramOut, const miopenTensorDescriptor_t paramOutFloat16Desc, void *paramOutFloat16, const miopenTensorDescriptor_t gradInDesc, const void *gradIn, const miopenTensorDescriptor_t expAvgInDesc, void *expAvgIn, const miopenTensorDescriptor_t expAvgOutDesc, void *expAvgOut, const miopenTensorDescriptor_t expAvgSqInDesc, void *expAvgSqIn, const miopenTensorDescriptor_t expAvgSqOutDesc, void *expAvgSqOut, const miopenTensorDescriptor_t maxExpAvgSqInDesc, void *maxExpAvgSqIn, const miopenTensorDescriptor_t maxExpAvgSqOutDesc, void *maxExpAvgSqOut, const miopenTensorDescriptor_t stateStepInDesc, void *stateStepIn, const miopenTensorDescriptor_t stateStepOutDesc, void *stateStepOut, const unsigned int state_step, const float lr, const float beta1, const float beta2, const float weight_decay, const float eps, const bool amsgrad, const bool maximize, const bool adamw, const miopenTensorDescriptor_t gradScaleDesc, const void *gradScale, const miopenTensorDescriptor_t foundInfDesc, const void *foundInf)
Execute single tensor Adam optimization and receive the result in a separate output tensor.

Parameters

  handle | MIOpen handle (input)
  paramInDesc | Tensor descriptor for the input parameter tensor (input)
  paramIn | Input parameter tensor (input)
  paramOutDesc | Tensor descriptor for the output parameter tensor (input)
  paramOut | Output parameter tensor (output)
  paramOutFloat16Desc | Tensor descriptor for the output parameter tensor float16 (input, optional)
  paramOutFloat16 | Output parameter tensor (output, optional)
  gradInDesc | Tensor descriptor for the input gradient tensor (input)
  gradIn | Input gradient tensor (input)
  expAvgInDesc | Tensor descriptor for the input exponential moving average tensor (input)
  expAvgIn | Input exponential moving average tensor (input)
  expAvgOutDesc | Tensor descriptor for the output exponential moving average tensor (input)
  expAvgOut | Output exponential moving average tensor (output)
  expAvgSqInDesc | Tensor descriptor for the input exponential moving average squared tensor (input)
  expAvgSqIn | Input exponential moving average squared tensor (input)
  expAvgSqOutDesc | Tensor descriptor for the output exponential moving average squared tensor (input)
  expAvgSqOut | Output exponential moving average squared tensor (output)
  maxExpAvgSqInDesc | Tensor descriptor for the input maximum exponential moving average squared tensor. Used when amsgrad is true (input, optional)
  maxExpAvgSqIn | Input maximum exponential moving average squared tensor. Used when amsgrad is true (input, optional)
  maxExpAvgSqOutDesc | Tensor descriptor for the output maximum exponential moving average squared tensor. Used when amsgrad is true (input, optional)
  maxExpAvgSqOut | Output maximum exponential moving average squared tensor. Used when amsgrad is true (output, optional)
  stateStepInDesc | Tensor descriptor for the input state step tensor (input, optional)
  stateStepIn | Input state step tensor (input, optional)
  stateStepOutDesc | Tensor descriptor for the output state step tensor (input, optional)
  stateStepOut | Output state step tensor that stores the updated step value. (output, optional)
  state_step | Input state step, It is used when the step tensor is null. (input)
  lr | Learning rate (input)
  beta1 | Coefficient used for computing the first moment running average of gradient (input)
  beta2 | Coefficient used for computing the second moment running average of gradient (input)
  weight_decay | Weight decay (input)
  eps | Term added to the denominator to improve numerical stability (input)
  amsgrad | Flag indicating whether to use the AMSGrad variant of Adam (input)
  maximize | Flag indicating whether to maximize the objective with respect to the parameters (input)
  adamw | If it is true, the operation becomes AdamW (input)
  gradScaleDesc | Tensor descriptor for the input grad scale tensor (input, optional)
  gradScale | Input grad scale tensor (input, optional)
  foundInfDesc | Tensor descriptor for the input found inf tensor (input, optional)
  foundInf | Tensor indicating presence of inf or nan in gradients. If true, skips operation and step update. (input, optional)

Returns
  miopenStatus_t

## ◆ miopenTransformersAdamW()

miopenStatus_t miopenTransformersAdamW | ( | miopenHandle_t | handle,
|  | const miopenTensorDescriptor_t | paramDesc,
|  | void * | param,
|  | const miopenTensorDescriptor_t | gradDesc,
|  | const void * | grad,
|  | const miopenTensorDescriptor_t | expAvgDesc,
|  | void * | expAvg,
|  | const miopenTensorDescriptor_t | expAvgSqDesc,
|  | void * | expAvgSq,
|  | const miopenTensorDescriptor_t | stateStepDesc,
|  | void * | stateStep,
|  | const unsigned int | state_step,
|  | const float | lr,
|  | const float | beta1,
|  | const float | beta2,
|  | const float | weight_decay,
|  | const float | eps,
|  | const bool | correct_bias,
|  | const miopenTensorDescriptor_t | gradScaleDesc,
|  | const void * | gradScale,
|  | const miopenTensorDescriptor_t | foundInfDesc,
|  | const void * | foundInf )

Implements Adam algorithm with weight decay fix as introduced in Decoupled Weight Decay Regularization. This is the fused kernel version of AdamW included in the Hugging Face Transformers module.

See also
  miopenFusedAdam

// Execute Adam
miopenTransformersAdamW(handle,
paramDesc,
param,
gradDesc,
grad,
expAvgDesc,
expAvg,
expAvgSqDesc,
expAvgSq,
NULL, // Unused stateStep Tensor because use step integer argument
NULL,
step,
lr,
beta1,
beta2,
weight_decay,
eps,
true, // correct_bias
NULL, // Unused gradScale Tensor because not amp
NULL,
NULL, // Unused foundInf Tensor because not amp
NULL);
// Execute AMP Adam
miopenTransformersAdamW(handle,
paramDesc,
param,
gradDesc,
grad,
expAvgDesc,
expAvg,
expAvgSqDesc,
expAvgSq,
stateStepDesc,
stateStep,
-1, // Ignore step value because stateStep Tensor is used
lr,
beta1,
beta2,
weight_decay,
eps,
true, // correct_bias
gradScaleDesc,
gradScale,
foundInfDesc,
foundInf);
miopenTransformersAdamW
miopenStatus_t miopenTransformersAdamW(miopenHandle_t handle, const miopenTensorDescriptor_t paramDesc, void *param, const miopenTensorDescriptor_t gradDesc, const void *grad, const miopenTensorDescriptor_t expAvgDesc, void *expAvg, const miopenTensorDescriptor_t expAvgSqDesc, void *expAvgSq, const miopenTensorDescriptor_t stateStepDesc, void *stateStep, const unsigned int state_step, const float lr, const float beta1, const float beta2, const float weight_decay, const float eps, const bool correct_bias, const miopenTensorDescriptor_t gradScaleDesc, const void *gradScale, const miopenTensorDescriptor_t foundInfDesc, const void *foundInf)
Implements Adam algorithm with weight decay fix as introduced in Decoupled Weight Decay Regularizatio...

Parameters

  handle | MIOpen handle (input)
  paramDesc | Tensor descriptor for the input parameter tensor (input)
  param | Input parameter tensor (input)
  gradDesc | Tensor descriptor for the input gradient tensor (input)
  grad | Input gradient tensor (input)
  expAvgDesc | Tensor descriptor for the input exponential moving average tensor (input)
  expAvg | Input exponential moving average tensor (input)
  expAvgSqDesc | Tensor descriptor for the input exponential moving average squared tensor (input)
  expAvgSq | Input exponential moving average squared tensor (input)
  stateStepDesc | Tensor descriptor for the input state step tensor (input)
  stateStep | Input state step tensor (input)
  state_step | Input state step. used when the step tensor is null (input)
  lr | Learning rate (input)
  beta1 | Coefficient used for computing the first moment running average of gradient (input)
  beta2 | Coefficient used for computing the second moment running average of gradient (input)
  weight_decay | Weight decay (input)
  eps | Term added to the denominator to improve numerical stability (input)
  correct_bias | Whether or not to correct bias in Adam (for instance, in Bert TF repository they use False).
  gradScaleDesc | Tensor descriptor for the input grad scale tensor (input, optional)
  gradScale | Input grad scale tensor (input, optional)
  foundInfDesc | Tensor descriptor for the input found inf tensor (input, optional)
  foundInf | Tensor indicating the presence of inf or NaN in gradients. If true, skips operation and step update (input, optional)

Returns
  miopenStatus_t

## ◆ miopenTransformersAdamWWithOutput()

miopenStatus_t miopenTransformersAdamWWithOutput | ( | miopenHandle_t | handle,
|  | const miopenTensorDescriptor_t | paramInDesc,
|  | void * | paramIn,
|  | const miopenTensorDescriptor_t | paramOutDesc,
|  | void * | paramOut,
|  | const miopenTensorDescriptor_t | paramOutFloat16Desc,
|  | void * | paramOutFloat16,
|  | const miopenTensorDescriptor_t | gradInDesc,
|  | const void * | gradIn,
|  | const miopenTensorDescriptor_t | expAvgInDesc,
|  | void * | expAvgIn,
|  | const miopenTensorDescriptor_t | expAvgOutDesc,
|  | void * | expAvgOut,
|  | const miopenTensorDescriptor_t | expAvgSqInDesc,
|  | void * | expAvgSqIn,
|  | const miopenTensorDescriptor_t | expAvgSqOutDesc,
|  | void * | expAvgSqOut,
|  | const miopenTensorDescriptor_t | stateStepInDesc,
|  | void * | stateStepIn,
|  | const miopenTensorDescriptor_t | stateStepOutDesc,
|  | void * | stateStepOut,
|  | const unsigned int | state_step,
|  | const float | lr,
|  | const float | beta1,
|  | const float | beta2,
|  | const float | weight_decay,
|  | const float | eps,
|  | const float | step_size,
|  | const bool | correct_bias,
|  | const miopenTensorDescriptor_t | gradScaleDesc,
|  | const void * | gradScale,
|  | const miopenTensorDescriptor_t | foundInfDesc,
|  | const void * | foundInf )

Execute single tensor Adam optimization and receive the result in a separate output tensor.

This function is equivalent to miopenTransformersAdam but receives the result in a separate output tensor.

See also
  miopenTransformersAdamW
  miopenFusedAdamWithOutput

// Execute Adam
miopenTransformersAdamWWithOutput(handle,
paramInDesc,
paramIn,
paramOutDesc,
paramOut,
NULL, // Unused paramOutFloat16 tensor because is not amp
NULL,
gradInDesc,
gradIn,
expAvgInDesc,
expAvgIn,
expAvgOutDesc,
expAvgOut,
expAvgInSqDesc,
expAvgSqIn,
expAvgSqOutDesc,
expAvgSqOut,
NULL, // Unused stateStepIn tensor because use step int
NULL,
NULL, // Unused stateStepOut tensor because use step int
NULL,
step,
lr,
beta1,
beta2,
weight_decay,
eps,
-1, // step_size
true, // correct_bias
NULL, // Unused gradScale Tensor because not amp
NULL,
NULL, // Unused foundInf Tensor because not amp
NULL);
// Execute Amp Adam
miopenTransformersAdamWWithOutput(handle,
paramInDesc,
paramIn,
paramOutDesc,
paramOut,
paramOutFloat16Desc, // optional in amp
paramOutFloat16,
gradInDesc,
gradIn,
expAvgInDesc,
expAvgIn,
expAvgOutDesc,
expAvgOut,
expAvgInSqDesc,
expAvgSqIn,
expAvgSqIn,
expAvgSqOutDesc,
expAvgSqOut,
stateStepInDesc,
stateStepIn,
stateStepOutDesc,
stateStepOut
-1, // Ignore step value because stateStep Tensor is used
lr,
beta1,
beta2,
weight_decay,
eps,
-1, // step_size
true, // correct_bias
NULL, // Unused gradScale Tensor because not amp
NULL,
NULL, // Unused foundInf Tensor because not amp
NULL);
miopenTransformersAdamWWithOutput
miopenStatus_t miopenTransformersAdamWWithOutput(miopenHandle_t handle, const miopenTensorDescriptor_t paramInDesc, void *paramIn, const miopenTensorDescriptor_t paramOutDesc, void *paramOut, const miopenTensorDescriptor_t paramOutFloat16Desc, void *paramOutFloat16, const miopenTensorDescriptor_t gradInDesc, const void *gradIn, const miopenTensorDescriptor_t expAvgInDesc, void *expAvgIn, const miopenTensorDescriptor_t expAvgOutDesc, void *expAvgOut, const miopenTensorDescriptor_t expAvgSqInDesc, void *expAvgSqIn, const miopenTensorDescriptor_t expAvgSqOutDesc, void *expAvgSqOut, const miopenTensorDescriptor_t stateStepInDesc, void *stateStepIn, const miopenTensorDescriptor_t stateStepOutDesc, void *stateStepOut, const unsigned int state_step, const float lr, const float beta1, const float beta2, const float weight_decay, const float eps, const float step_size, const bool correct_bias, const miopenTensorDescriptor_t gradScaleDesc, const void *gradScale, const miopenTensorDescriptor_t foundInfDesc, const void *foundInf)
Execute single tensor Adam optimization and receive the result in a separate output tensor.

Parameters

  handle | MIOpen handle (input)
  paramInDesc | Tensor descriptor for the input parameter tensor (input)
  paramIn | Input parameter tensor (input)
  paramOutDesc | Tensor descriptor for the output parameter tensor (input)
  paramOut | Output parameter tensor (output)
  paramOutFloat16Desc | Tensor descriptor for the output parameter tensor float16 (input, optional)
  paramOutFloat16 | Output parameter tensor (output, optional)
  gradInDesc | Tensor descriptor for the input gradient tensor (input)
  gradIn | Input gradient tensor (input)
  expAvgInDesc | Tensor descriptor for the input exponential moving average tensor (input)
  expAvgIn | Input exponential moving average tensor (input)
  expAvgOutDesc | Tensor descriptor for the output exponential moving average tensor (input)
  expAvgOut | Output exponential moving average tensor (output)
  expAvgSqInDesc | Tensor descriptor for the input exponential moving average squared tensor (input)
  expAvgSqIn | Input exponential moving average squared tensor (input)
  expAvgSqOutDesc | Tensor descriptor for the output exponential moving average squared tensor (input)
  expAvgSqOut | Output exponential moving average squared tensor (output)
  stateStepInDesc | Tensor descriptor for the input state step tensor (input, optional)
  stateStepIn | Input state step tensor (input, optional)
  stateStepOutDesc | Tensor descriptor for the output state step tensor (input, optional)
  stateStepOut | Output state step tensor that stores the updated step value. (output, optional)
  state_step | Input state step, It is used when the step tensor is null. (input)
  lr | Learning rate (input)
  beta1 | Coefficient used for computing the first moment running average of gradient (input)
  beta2 | Coefficient used for computing the second moment running average of gradient (input)
  weight_decay | Weight decay (input)
  eps | Term added to the denominator to improve numerical stability (input)
  step_size | Pre-calculated step_size, used for performance enhancement (input)
  correct_bias | Whether or not to correct bias in Adam (for instance, in Bert TF repository they use False) (input)
  gradScaleDesc | Tensor descriptor for the input grad scale tensor (input, optional)
  gradScale | Input grad scale tensor (input, optional)
  foundInfDesc | Tensor descriptor for the input found inf tensor (input, optional)
  foundInf | Tensor indicating presence of inf or nan in gradients. If true, skips operation and step update. (input, optional)

Returns
  miopenStatus_t