New complex-valued activation functions

We present a new type of activation functions for a complex-valued neural network (CVNN). A proposed activation function is constructed such that it ﬁxes a given ellipse. We obtain an application to a complex-valued Hopﬁeld neural network (CVHNN) using a special form of the introduced complex functions as an activation function. Considering the interesting geometric properties of the plane curve ellipse such as focusing property, we emphasize that these properties may have possible applications in various neural networks.


Introduction
Recently, complex-valued neural networks (CVNN) have been used in various fields such as optoelectronics, imaging, signal processing, quantum neural devices and artificial neural information processing by many researchers (see [1][2][3][4][5][6][7][8] for more details). For example, Gandal et al. tried to evaluate and compare the relative performance of CVNN using different error functions [3]. Hirose studied what merits of CVNNs arise from [4]. Jalab and Ibrahim introduced a new type of complex-valued sigmoid function for a fully multi-layered CVNN [5]. Zimmerman et al. gave the differences between complex-valued and real-valued neural networks and studied the problems of CVNNs gradients computations by combining the global and local optimization algorithm [8]. Oladipo and Gbolagade investigated modified logistic sigmoid as relates to analytic univalent functions by means of subordination properties in terms of starlikeness, convexity and close-to-convexity [6]. In [7], it was proposed a wind prediction system for the wind power generation using ensemble of multiple complex extreme learning machines and used the elegant theory of conformal mapping to find better transformations in the complex domain for enhancing its prediction capability.
In a CVNN, one of the main problems is selecting of nodes activation function (see [9]). In this paper, we propose a new type of complex-valued functions as an activation function for a complexvalued Hopfield neural network (CVHNN). These functions fix a given ellipse on the complex plane. To construct an appropriate activation function, a suitable ellipse can be chosen according to the particular problem. The main advantage of this choice is the increase in the number of fixed points of a neural network with a geometrical meaning.
On the other hand, an ellipse has some interesting properties such as the focusing property (see [10]). We recall the focusing property of an ellipse. Let E be the ellipse of the normal form with semimajor axis a and semi-minor axis b: E : x 2 a 2 +

*Corresponding Author
The foci of the ellipse E of the form (1) are as follows: The radius of the ellipse E is r = 2a and so we rewrite the equation of the ellipse of the form (1) as E : |z − c 1 | + |z − c 2 | = r. It is a well-known fact from geometry that a light ray which leaves a focus c 1 of an ellipse will be reflected to other focus c 2 (see [10] and [11] for more details). Using this interesting property and the following proposition, Frantz proposed an application to the open problem about trapped reflections described in [12]. It was seen that the light ray gradually approaches a horizontal trajectory and never leaves the container (see [10] for more details).
Therefore, it is possible to get some applications of these kind properties of an ellipse in neural networks. It is known that the plane curve ellipse has appeared in many applications in real life problems (for example, see [13][14][15][16][17][18][19][20][21]). We expect that our study will help to generate some new researches and applications on complex-valued neural networks.

Complex Functions That Fix an Ellipse
In this section, we investigate a new type of complex-valued function which fixes an ellipse. We begin with the following definition.
Definition 1. Let E be any ellipse on the complex plane. If a complex function T satisfies the condition T (z) = z for each complex number z ∈ E, then the ellipse E is called the fixed ellipse of T .
Now we consider an ellipse E of the form (1). If we take x = z+z 2 and y = z−z 2i , then we can rewrite the equation of this ellipse as where α = 1 4a 2 − 1 4b 2 and β = 1 2a 2 + 1 2b 2 . Conversely, let us consider the following general equation The equation (3) defines an ellipse if the following conditions hold: (1) α, β, γ ∈ R and β > 0, (2) γ < 0, 2α + β > 0 and 2α − β < 0. Indeed, if we write z = x + iy then we have If we choose a and b such as then the equation (3) defines the ellipse x 2 a 2 + y 2 b 2 = 1. Now we present a complex function which fixes an ellipse of the form (3). For any complex number z on the ellipse E, we get Hence we obtain the following theorem.
Theorem 1. Let E be any ellipse with the equation (3). If we define the transformation T 1 as then T 1 fixes the ellipse E.
If we consider the following transformation T 2 defined as then it can be easily seen that T 2 also fixes the ellipse E. Clearly, the transformations T 1 • T 2 and T 2 • T 1 fix also the ellipse E. The transformations T 1 and T 2 are not always commutative, that is, For example, if we consider γ = −1, α = 1, β = 3 and z = 1 then we get Consequently, we can give the following corollary.
Corollary 1. For each ellipse E, there are at least three transformations T such that Then E is exactly the set of fixed points of each T . Taking α = 0, γ = −1, that is, a = b in the equation (4), then we get the transformation T 3 (z) = 1 βz and E becomes the circle with the center z 0 = 0 and the radius r = 1 √ β . Hence, the transformation T 3 fixes the circle E and it is known that the transformation T 3 is an anti-conformal map on the complex plane. Now we consider the following two families of ellipses: 1) Let α = c be fixed in the equation (5). Then we get β ∈ (2c, ∞) when c ≥ 0 and β ∈ (−2c, ∞) when c < 0. For example, if we choose α = 1 in the equation (5), we get β ∈ (2, ∞). In this case, the transformation T 1 fixes the following ellipses E β defined according to β: where a β = 1 β+2 and b β = 1 β−2 for each β ∈ (2, ∞). In the following figure, which has been drawn by Mathematica [22], it is seen how the ellipses E β change (see Figure 1). The ellipses E β are indicated with different colors: E 3 is the red ellipse, E 4 is the blue ellipse, E 5 is the orange ellipse, E 6 is the green ellipse and E 7 is the pink ellipse.

An Application to Complex-Valued Hopfield Neural Networks
Möbius transformations and some related (anticonformal) maps have been used as activation functions in complex-valued neural networks using some different point of views such as fixed points or fixed circles. It is known that Möbius transformations are the conformal mappings of the complex plane C. A Möbius transformation is a rational function of the form where a, b, c, d are complex numbers satisfying ad − bc = 0. A point z on the complex plane is said to be a fixed point of the Möbius transformation T (z) if T (z) = z. A Möbius transformation T (z) has at most two fixed points if it is not identity transformation (see [23], [24] and [25] for the basic properties of Möbius transformations).
In [26], it was identified the activation function of a neuron and a single-pole all-pass digital filter section as Möbius transformations and then, the existence of fixed points of a neural network were guaranteed by the underlying Möbius transformation. In [27],Özdemir et al. proposed new types of activation functions which fix a circle for a CVNN. The usage of these types of activation functions leads us to guarantee the existence of the fixed points of a CVHNN. In this section, we consider the special form of the transformation (5) defined as follows: where α ≥ 0 (notice that γ = −1 and β is chosen as 2α + 1). We propose this function as an activation function for a CVHNN.
If we take α = 0 in the equation (7), then we get the following activation function used in a CVHNN in [27]: Therefore, the transformation S α defined in (7) which fixes an ellipse of the form (3) with γ = −1, α ≥ 0, β = 2α + 1 > 0 becomes a transformation S 0 which fixes the unit circle. The transformation S 0 (z) = 1 z was used to guarantee the existence of the fixed points of a complex-valued Hopfield neural network (CVHNN). The transformation S α (z) is not injective while S 0 (z) is injective. Also this transformation S α (z) maps an ellipse of the form (3) onto itself, outside of the ellipse to its inside and inside of the ellipse to its outside. For example, in the following Figure 3, we see the image of the outside of the ellipse E : 5x 2 + y 2 = 1 under the transformation defined as At first, we give a brief summary about CVHNNs. In [28], Hopfield presented a recurrent neural network model referred to as the Hopfield neural network (HNN). HNN has been generalized to CVHNN and this generalized neural network has been studied by many authors using different aspects. For example, Kobayashi defined the concept of a hyperbolic neuron and constructed hyperbolic Hopfield neural network [18]. Also he described the symmetric complex-valued Hopfield neural networks using the complex-valued multistate neurons [29].
Following the studies given in [27], here we consider the class of system in C in order to interest CVHNN given by where T ∈ C n×n , U ∈ C n are matrices, z(t) ∈ C n is state vector, H(z) : C n → C n×n is a nonlinear function and F (z) = (S 1 (z 1 ), S 2 (z 2 ), . . . , S n (z n )) T : C n → C n is an activation function with for some fixed α ≥ 0 and all k ∈ {1, 2, . . . , n}. We note that the parameter α can be chosen appropriately according to the studied problem. We choose T ∈ R n×n and U = 0 in the equation (8) to obtain a relationship between the fixed points of the activation function F (z) and the fixed points of the network. Fixed points of the equation Suppose that H(z) is a nonsingular matrix then the fixed points are F (z) = T z, which correspond to the fixed points of the activation function.
In our approach, we increase the number of fixed points using the activation function defined in (9) by a point of geometric view. We use the Lyapunov stability to determine whether the fixed points are stable or not (see [27], [30] and [31] and for more details). The fixed points of the CVHNN are isolated since they are on an ellipse. Following the steps used in the proof of Theorem 2 on page 4701 in [27] and using the property S k (z k ) = S k (z k ), it can be easily obtained that z(t) = 0. So the following theorem gives the stability of the fixed points.
Theorem 2. Let the inner product be defined on C n as z 1 , z 2 = z * 2 z 1 where z 1 , z 2 ∈ C n and (.) * denotes the conjugate transpose. Assume that the matrix T ∈ R n×n is symmetric and the matrix Re [H(z)] is positive definite. Then the function z(t) = −H(z(t))(−T z(t) + F (z(t))).

Remarks and Conclusion
We note that a general activation function for a CVHNN can be obtained using the transformation T 1 given in (5). The fixed points of this activation function are on an ellipse with the form (3). This allows us to choose the appropriate activation function according to the considered problem on a neural network. This activation function can be helpful to construct several neural nets and lead to interesting applications. Proposed activation functions can be considered as the generalizations of ones used in [27].
Finally, we emphasize that the properties of the ellipse, which is fixed by the chosen activation function, are applicable to the neural networks. Therefore, our results have possible applications in real life problems.