Batch Gradient Method with Smoothing L1/2 Regularization and Momentum for Pi-sigma Networks

Kh. Sh. Mohamed, Y. Sh. Mohammed, A. A. Elzain, Kh. M. Makin, Elnoor. A. A. Noh

Abstract: : In this paper, we study the Batch gradient method with Smoothing L1/2 Regularization and Momentum for Pi-sigma Networks, assuming that the training samples are permuted stochastically in each cycle of iteration. The usual L1/2 Regularization term involves absolute value and is not differentiable at the origin, which typically causes oscillation of the gradient method of the error function during the training. However, using the Smoothing approximation techniques, the deficiency of the norm L1/2 Regularization term can be addressed. Corresponding convergence results of Smoothing L1/2 Regularization are proved, that is, the weak convergence result is proved under the uniformly boundedness assumption of the activation function and its derivatives.

Keywords: Pi-sigma Network, Batch Gradient Method, smoothing L1/2 Regularization, Momentum, boundedness, convergence