Perl AI Deep Learning Tutorial

Initial value of He

Let's find the initial value of He. The initial value of He is random number that follows a normal distribution, with the mean specified as 0 and the standard deviation specified as "sqrt(2 / number of inputs)". ..

The number of inputs is the value of m in the conversion from m to n.

Initial value of weight when mainly using ReLU function as activation function It seems to be used for. By choosing a good initial value, the value to which the activation function is applied after the conversion from m to n in each layer will vary moderately.

```Get the initial value of #He
sub he_init_value {
my (\$inputs_length) = @_;

return randn (0, sqrt(1 / \$inputs_length));
}
```

Initialize an array of weights using the initial value of He

```use strict;
use warnings;

# Function to find random numbers that follow a normal distribution
# \$m is mean, \$sigma is standard deviation,
sub randn {
my (\$m, \$sigma) = @_;
my (\$r1, \$r2) = (rand(), rand());
while (\$r1 == 0) {\$r1 = rand();}
return(\$sigma * sqrt(-2 * log(\$r1)) * sin(2 * 3.14159265359 * \$r2)) + \$m;
}

# Create initial value of He
sub create_he_init_value {
my (\$inputs_length) = @_;

return randn (0, sqrt(2 / \$inputs_length));
}

#Create an array with the default value of He
sub array_create_he_init_value {
my (\$array_length, \$inputs_length) = @_;

my \$nums_out = [];
for (my \$i = 0; \$i <\$array_length; \$i ++) {
\$nums_out->[\$i] = create_he_init_value (\$inputs_length);
}

return \$nums_out;
}

# If the number of inputs is 728 and the number of outputs is 30, the length of the array of the matrix is ​​"728 * 30".
my \$inputs_length = 728;
my \$outputs_length = 30;
my \$weights_mat = {
rows_length => \$outputs_length,
columns_length => \$inputs_length,
};;
my \$weights_values_length = \$inputs_length * \$outputs_length;
\$weights_mat->{values} = array_create_he_init_value (\$weights_values_length, \$inputs_length);

use Data::Dumper;
print Dumper \$weights_mat;
```

Initial values ​​other than the initial value of He

When using the sigmoid function as the activation function, it is better to use Xavier default value.