RReLU module

Usage

nn_rrelu(lower = 1/8, upper = 1/3, inplace = FALSE)

Arguments

lower

lower bound of the uniform distribution. Default: $\frac{1}{8}$

upper

upper bound of the uniform distribution. Default: $\frac{1}{3}$

inplace

can optionally do the operation in-place. Default: FALSE

Details

Empirical Evaluation of Rectified Activations in Convolutional Network.

The function is defined as:

$$\text{RReLU}(x)=\{\begin{array}{ll}x& \text{if }x\ge 0\\ ax& \text{ otherwise }\end{array}$$

where $a$ is randomly sampled from uniform distribution $\mathcal{U}(\text{lower},\text{upper})$. See: https://arxiv.org/pdf/1505.00853.pdf

Shape

• Input: $(N,\ast )$ where * means, any number of additional dimensions

• Output: $(N,\ast )$, same shape as the input

Examples

if (torch_is_installed()) {
m <- nn_rrelu(0.1, 0.3)
input <- torch_randn(2)
m(input)
}
#> torch_tensor
#> -0.1383
#> -0.1487
#> [ CPUFloatType{2} ]