Application of spectral conjugate gradient methods for solving unconstrained optimization problems

Conjugate gradient (CG) methods are among the most efficient numerical methods for solving unconstrained optimization problems. This is due to their simplicty and less computational cost in solving large-scale nonlinear problems. In this paper, we proposed some spectral CG methods using the classical CG search direction. The proposed methods are applied to real-life problems in regression analysis. Their convergence proof was establised under exact line search. Numerical results has shown that the proposed methods are efficient and promising.


Introduction
The spectral CG methods are among the most efficient variant of CG methods designed to solve large-scale problems. The methods possess the global convergent properties in addition to the sufficient descent condition. Moreso, the spectral CG methods are less expensive and requires less storage location. Some outstanding features of the spectral CG method are their simplicity in algebraic processes and development of computer codes [1]. Spectral CG method is formulated by combining the CG search direction and a scalar spectral parameter to form a new search direction. Birgin and Martinez [2], introduced a spectral CG method using standard secant equation [3].
Consider the following minimization problem.
where : → is continuous and differentiable, g is the gradient of ( ) and the vector 0 ∈ is known as the initial point. The CG method are iterative scheme of the form +1 = + , = 0,1,2,3,4, … (2) where the vector is the current iterate, +1 is the new iteration point, and > 0 is the step-dimension obtained by the line search method defined as = min >0 ( + ) (3) also, is the classical search direction given as = { −g , = 0 −g + −1 , ≥ 1 (4) g = ∇ ( ), is the gradient and the parameter ∈ is the CG coefficient that characterizes different CG methods. Some known CG coefficients are the Polak-Ribiére-Polyak (PRP) and Wei-Yao-Liu (WYL) methods with formulas as follows.
In this paper, the spectral PRP and spectral WYL CG methods are presented without the secant equation. Their performance is verified using the least square and trend line methods in regression analysis. The regression analysis is an important tool for the analysis of statistical data utilized in the field of economics, engineering, sciences and many more [15]. The analysis is use for forecasting and to comprehend the relation between dependent and independent variables in real life applications. The dependent variable is denoted by and independent is denoted by for = 1,2,3 … … . , , > 0, and is an integer constant in the error term. The model is defined by and generalized as follows where 0 , 1 , 2 , … … . . , are the parameters for the regression analysis. The values of the parameters are estimate by using the nonlinear least square method defined by where is the estimated data of ℎ response and 1 , 2 , . . . … , are data evaluation of the response variables [16]. The formula for predicting data in regression analysis is derive from calculating the relative error. However, the error is obtained by comparing the approximate value and actual value as described below The least square determines the best approximation models by comparing the total least square errors. The error is defined as The strategy of fitting the best line through the data would minimize the sum of the residual error squares for the data available. This problem is similar to the minimization problem in unconstrained optimization [17]. Thus, we employ the spectral PRP and WYL CG parameter to obtain the solution of the given unconstrained optimization problem.

Derivation of spectral CG methods
Spectral CG method was introduce by [2] with direction defined as = − g + −1 , where −1 = −1 −1 and is a spectral scalar parameter. Motivated by the procedure of [5], we proposed the following search direction Recall that the orthogonality of gradients g g −1 = 0 and thus, and ∅ are the new spectral parameters computed by exact line search procedure.

The global convergence analysis of spectral CG methods
The Sufficient descent condition ensures that global convergence of iterative procedures or algorithm is achieved. Therefore, all CG methods must satisfy the following.
From the search direction (11) multiply both sides by g +1 and substitute parameter (13) gives It is known from the conjugacy conditions g +1 = 0.

Proof. With
, also by induction, since g 0 0 = −‖g 0 ‖ 2 , the condition (15) satisfied as = 0. Now we assume it is true for ≥ 0. Also, the inequality (15) hold true, from the search direction (12) multiply both sides of the equation by g +1 and substitute (14) gives Therefore, from the conjugacy conditions g +1 = 0. Hence for constant = 1 condition (15) hold for + 1. ∎ The following assumptions are needed for the convergence analysis of the CG method.
is continuously differentiable in a neighborhood of the level set Ω and 0 is a starting point.

Description of the real life application
In this section, the detailed description of the real-life problem considered are in Table 1. These problems were obtained from [19]. The approximate function for the nonlinear least square method is formed as follows The data set in Table 1 shows the rate of drug abuse among the youth with aged 18 to 25 in Kano city, Nigeria for the years 2009-2017. The statistical data was obtained yearly by the National Drug Law enforcement agency (NDLEA), Kano. From the Table  1, the -variable represent the year of the operation while the -variable represent the rate of drug abuse among the youth in the city. For the data fitting, only the data from 2009 to 2016 is been considered. The data for the year 2017 is reserved for the error analysis. Let the number of data be the number of years and the value be the rate of drug abuse in percentages. Then, the data from 2009 to 2016 are utilized to formulate the nonlinear quadratic model for the least square method and the corresponding test function of unconstrained optimization problem. From the above problem, observation reveals that the data and the value of have parabolic relations with the regression function defined by (26) and the regression parameters 0 , 1 and 2 .
Similar transformation of the above least squares problem using the data from Table 1  However, the data for 2017 is excluded from the unconstrained optimization function so that it could be used to compute the relative errors of the predicted data. Therefore, the proposed spectral PRP and WYL CG methods are applied to solve the test function using exact line search technique. Table 2 and Table 3 shows the test results for the spectral PRP, spectral HS, spectral WYL and MSCG methods for some selected initial point.

202
S.M. Ibrahim, U.A. Yakubu, M. Mamat / IJOCTA, Vol.10, No.2, pp.198-205 (2020)   To avoid computing the values of 0 , 1 , 2 using matrix inverse, we employ the Spectral PRP, Spectral WYL, SHS and MSCG using four initial points as presented in the Table 2 and Table 3. The iteration is terminated if the number of iterations exceed 10000 or if the method fails to solve a test problem and denoted the point of failure as 'NaN'. The approximation functions of the spectral CG methods is given in Table  4.

Trend line method
The rate of drug abuse in Kano city, Nigeria is estimated using the least square method and the proposed spectral CG methods. The tread line is plotted based on the original data from Table 1 using Microsoft Excel software. The equation for the trend line is in the form of nonlinear quadratic equation. Based on the actual data, the index of drug abuse denoted by is represented in the -axis. The -axis represent the year and denoted by .

Figure 1. Nonlinear Quadratic Trend Line for Rate of Drug Abuse in Kano City
The functions of trend line and least square methods are compared with approximation functions obtained from the Spectral CG methods presented in Table 4.

Numerical result
Algorithm 1.1 have been tested on some benchmark problems and its performance are compared with RSPRP method [10], Wei-Yao-Liu (WYL) method [23], and Polak-Ribierre-Polyak (PRP) method [24] respectively. The comparisons are based on CPU time and number of iterations. The stopping criteria used is ‖g ‖ < where = 10 −6 as suggested by Hillstrom [21]. The set of standard test functions are considered from [1] and utilised with four different initial values. The codes are written on MatlabR2015 subroutine programming and run on an Intel® Core™ i5-3317U (1.7GHz with 4 GB (RAM)). Table 5 and 6 presents the list of standard test problems with dimensions and initial points used to test the efficiency of the proposed spectral CG methods. The numerical performance of the proposed algorithms is presented in Figures 1.2 -1.5 based on a number of iterations and CPU time. The maximum value of the percentage of probability ( ) and the solver that reached the solution point foremost are regarded as the best performing CG methods for unconstrained optimization problems [25,27]. From the results obtained, the SPRP and SWYL CG methods are able to solve the standard benchmark problems as compared to the existing methods used in the analysis. Similarly, the data for 2017 are estimated using the nonlinear unconstrained optimization model in Table 4. and the relative error for each model using equation (10) is presented in Table 7. Table 6. Standard Test Problems functions for Figure 4 and  The efficiency of each method is measure by equation (9). All the computations are carried out using Microsoft Excel 2016 and MATLAB 2015a subroutine programme. The model with the smallest relative error is considered the best model that estimate the rate of drug abuse in Kano city for the year 2017.

Conclusion
This paper focuses on the application of the spectral CG methods for unconstrained optimization. The proposed methods are compared with the classical WYL, PRP, least square and Trend line methods. The sum of relative error for the proposed spectral CG methods are computed based on four categories of initial values and three set of real numbers for nonlinear quadratic model. From the Table 7, the average relative error for the predicted data against the actual data 1.83 are calculated. The relative error for the data generated from nonlinear quadratic models of spectral PRP and spectral WYL methods are smaller compared to the least square and trend line models, which is around 0.3821846916967. The smallest relative error signifies the success of the spectral CG methods.