Using Models of Collective Neural Networks for Classification of the Input Data Applying Simple Voting

This paper deals with the use of neural networks in binary classification problems based on the simple voting method. It specifies that the accuracy of the neural network classification depends both on the choice of the network architecture and on the partitioning of data into training and test sets. It is noted that the process of building a neural network model is probabilistic in nature. To eliminate this drawback and improve the accuracy of classification, the need to combine several models in the form of a collective of neural networks is actualized. To build such a model, it is proposed to use the 0.632-bootstrap method. To aggregate individual solutions formed at the output of each neural network, it is proposed to use a single-choice simple voting. The choice of the model structure in the form of a single-layer Perceptron is justified, and its mathematical model is presented. Using the evaluation data of the functional state of a drunk human as an example, the results of an experimental assessment of the bootstrap error and the accuracy of the neural network model are presented. It is concluded that it is possible to achieve a higher accuracy of classification based on the neural network model when aggregating the results of all bootstrap models using the simple voting method. The accuracy of the constructed model is compared with the accuracy of other classification models. The accuracy of the constructed model was 96.7%, which on average exceeded the accuracy of other classification models by 6.6%. Thus, the neural network collective model is an effective tool for classifying input data using the simple voting method.


Introduction
It is known that the classification accuracy of the neural network model depends directly on the chosen neural network (NN) architecture Salakhutdinov et al., 2004). However, the optimal structure can often be chosen only after its training Mikhailov and Staroverov, 2013). In turn, the stage of training should be preceded by the stage of formation of training and test samples (Azimov et al., 2015;Ismagilov et al., 2018). The effective implementation of this stage is possible on the basis of the 0.632-bootstrap method (Efron and Tibshirani, 1997;Paklin and Oreshkov, 2009), which implements the procedure of repeated random sampling of values from the initial data. However, the result of training a neural network for different bootstrap samples is actually random, since it is largely determined by the random nature of splitting the source data into training and test sets (Efron, 1979). As a result, in some cases, NN models built at different stages of bootstrapping will recognize the same data in different ways (Paklin and Oreshkov, 2009). Thus, to improve the accuracy of classification, it is important to combine several models in the form of a collective of neural networks (Bova and Dukkardt, 2012;Paklin and Oreshkov, 2009;Zhou et al., 2002). In this case, it is reasonable to consider the result of the classification the aggregation of individual decisions of each neural network based on a single-choice simple vote (Geron and Frid, 2007).
The mathematical representation of the aggregation of individual solutions by voting is presented in (Bova and Dukkardt, 2012;Vol'skii and Lezina, 1991;Zhou et al., 2002). However, the use of simple voting schemes by a majority vote in neural network models is not fully explored. In this paper, we develop a model of a collective of neural network (CoNN) and evaluate its effectiveness using the example of solving a binary classification problem.

Methods
In most cases, to solve the problems of binary classification, perceptron models of neural networks (Katasev,2010;Hecht-Nielsen,1987) are used based on the classical consequence of the Arnold -Kolmogorov -Hecht -Nielsen theorem (Hecht-Nielsen,1987), according to which the maximum number of neurons of the hidden perceptron layer is determined on the basis of the expression: where N h is the number of hidden neurons, and N in is the number of input neurons. Thus, for example, with 4 input neurons, the neural network built on the basis of expression (1) will additionally have 9 hidden neurons and 1 output neuron (Krug, 2002).
We shall consider a model of an artificial neuron, presented graphically in Figure 1.

Figure-1. A model of an artificial neuron
The mathematical model of such a neuron can be described by the following expression: where x i is the input signals of the neuron; w i  (Salakhutdinov et al., 2004) is the weight of synaptic connections; s is the result of the weighted sum of the input signals; y is the output of the neuron; n is the number of inputs of the neuron; f -activation function (Bukhtoiarov, 2012;Korneev et al., 2001).
In vector form, this model looks as follows: , where the parameter a determines the steepness of the sigmoid (often the value of the parameter a = 0.5 (Katasev,2010) is specified). It is obvious that the output value of the neuron is in the range (0,1).
We introduce the following designations: 1) X = {x 1 , x 2 , …, x i , …, x n } is the set of inputs of the neural network, where x i is the i-th neuron of the input layer, n is the number of input neurons; 2) H = {h 1 , h 2 , …, h j , …, h m } is the set of neurons of the hidden layer, where h jj-th hidden neuron, m is the number of neurons determined by expression (1); 3) y is the output of the neural network; 4) W 1a rectangular matrix of weight factors between the neurons of the input and hidden layers, while Subject to the introduced designations and, based on the mathematical model of an artificial neuron (2), we build a mathematical model of the original neural network. In vector form, this model is defined as follows: 1) Y = f (HW 2 )the vector of the output activity of the neural network, consisting of a single y-neuron.
2) H = f (XW 1 )the vector of the output values of the hidden neurons (the vector of input values for the yneuron). Thus, the model can be represented as follows: Since XW 1 is a product of vectors, it can be written in scalar form: At the same time, the output value of the j-th neuron of the hidden layer (j-th input value of the y-neuron) is . Then, in the scalar form, the model of the original neural network is as follows: , the formula for the output activity of the neural network is as follows: (3) Consequently, the output activity of the built neural network is determined by real numbers of the interval (0,1). To solve the problem of binary classification, it is necessary to match the calculated output of the network to one of two classes: "0" or "1" (Akhmetvaleev and Katasev, 2018;Paklin and Oreshkov, 2009).
The assessment of the accuracy of the classification of the traditional perceptron model of the neural network, formed with regard to expression (1) and based on the 0.632-bootstrap method, can be represented as follows.  Table 1 presents the results of an experimental assessment of the bootstrap error and accuracy of the neural network model for determining the functional state of human intoxication (Akhmetvaleev et al., 2018a). According to the results given, the adequacy of each individual model built at the appropriate stage of bootstrapping is not high. As one can see, the bootstrap error  i varies in the range from 0.02 to 0.09 of conventional units, while the final model error is 6.8%, which indicates a higher accuracy in the classification of the neural network model when aggregating the results of all bootstrap models. Therefore, during bootstrapping it is possible to form a collective of some neural networks of the same architecture. The result of the classification is the aggregation of decisions of all NN based on a voting, which diagram of application can be presented in the following figure. The figure shows a diagram of application of the CoNN model for classifying input data based on the simple voting method. Each NN, which is part of the model, "votes" for the choice of one or another class of decisions, after which the votes are aggregated and a general decision is made by a majority vote. The number N of NN models in the collective must be odd in order that only one of the classes be selected collectively (Vol'skii and Lezina, 1991).
Subject to the traditional aggregation of voting results (Bova and Dukkardt, 2012;Paklin and Oreshkov, 2009;Vol'skii and Lezina, 1991;Zhou et al., 2002), the general collective decision formed by the CoNN can be represented as a certain function F (aggregating rule), the input of which is the individual decisions of individual NN that make up the collective: where Qa collective decision, Q lan individual decision of the l-nt NN model, Nnumber of NN models in a collective; Xpresentation (a set of classes of decisions), consisting of two options; F is a simple voting function (Kupriianov, 2012).
The choice of a particular class at the output of individual NN models is formed by binarization of the calculated output of the corresponding neural network ("0" or "1"). Thus, the decision of the l-th NN model can be represented as follows: where Cutt_offan expert decision cutoff point. The aggregate {Q l } of individual elections from X of all N members of a collective is called the profile of individual elections of voters. The voting procedure, which according to the {Q l } profile produces a collective choice Q, is called a "choice-choice" procedure. Such procedures are applied in practice much more frequently than other collective decision procedures (Vol'skii and Lezina, 1991).
To implement the voting rule by a majority of votes, it is necessary to fix the real threshold k (1 ≤ k ≤ N) as follows: k = 0,5N, then the collective choice is such a variant X = {0, 1}, for which the individual choices of Q l from the profile {Q l } will be greater than or equal to the threshold k, in other words f (s l ) ≥ k.
Thus, the expression for the output of the CoNN model will be y = Q = F (Q 1 ,Q 2 ,…,Q l ,…,Q N ), where F is the voting rule (4), which means that the NN model's output is equal to the collective decision Q: . We shall further consider an example of a model of a collective of neural networks, where the result of the classification of NN's individual decisions is formed on the basis of a simple vote with a single choice.

Results and Discussion
The paper (Akhmetvaleev et al., 2018b) presents a model of a CNS for solving the problem of determining the functional state (FS) of a human intoxication by his pupillary response to a pulsed light exposure (Akhmetvaleev et al., 2018a). The following figure shows the structure of this model (Lowenstein, 1959).

Figure-3. The model of the collective of neural networks
The CoNN model consists of 7 reduced NNs with 10 hidden and 8 input neurons, the composition of which is determined by pupillometric parameters D 0 , D min , D k , D ps , A s , t l , t s , and t pr (Janisse, 1977;Velkhover and Ananin, 1991). Mathematically, this model can be represented as follows.
1. Output class: где X is the set of classes of decisions consisting of two options: 0 is a normal FS, and 1 is a deviation of the FS. 2. Individual decision of the l-th NN (class of human FS) at the output of the l-th NN: 3. The activation function of the output of the l-th NN corresponds to expression (3) and is defined as follows: , where w lj is the weight of the synaptic connection of the output neuron of the l-th NN with the corresponding j-th hidden neuron; x l is the input signal of the neuron; w lij is the weight of the synaptic connection of the i-th input neuron with the j-th hidden neuron in the l-th NN. For the resulting model, calculations of errors of the 1st and 2nd kind were made. In this case, the error of the first kind occurs when the model incorrectly classifies FS of intoxication as the norm. Accordingly, the error of the 2nd kind occurs when the normal classification of the FS is incorrectly classified.
the accuracy of classification of a CoNN model and its errors of the 1st and 2nd kind with the corresponding values of accuracy and classification errors obtained on the basis of additional models based on the same initial data: models of single-layer perceptron, logistic regression, decision tree and the Kohonen network. As follows from the table, when solving the problem of determining the FS of a human intoxication based on the analysis of pupillometry data, the classification accuracy based on the proposed model of the collective of neural networks is 96.7%, which on average exceeds the accuracy of other classification models used by 6.6%.

Summary
Thus, the combination of several models in the form of a collection of neural networks can improve the adequacy of the model and the accuracy of classification, compared with single models. At the same time, the classification of input data by simple voting is an important tool when using in the models of the collective of neural networks, which makes it possible to effectively solve classification problems in various subject areas.

Conclusions
The conducted studies have shown the high efficiency of the simple voting method in classifying input data based on neural network models. First of all, the voting on the basis of a majority vote is relevant in the problems of binary classification. The CoNN model presented in the paper for determining the functional state of a human intoxication by his pupillary response to a pulsed light effect confirms the effectiveness of the proposed mathematical tool.
In order to develop a scientific direction on the classification of input data in collectives of neural networks, it is advisable to improve the mathematical apparatus, as well as its implementation and practical use in various subject areas.