Perception machine model for data requirements: The training data set requires a certain ultra -plane that can completely divide the authentic example points and negative example points of the data set to both sides of the ultra -plane, that is, the training data set is linearly divided. Because only when the training data set is linearly divided, the perception machine learning algorithm is converging; if the linearity of the training data set is inseparable, the perception machine learning algorithm does not converge, and iteration results will fluctuate. When training data hub is inseparable, a linear support vector machine can be used.

The learning process of the perception machine model: Seek the sensor model according to the training dataset, that is, find the model parameter$w$and harmony$b$。

Predictive process of perception machine model: The output category corresponding to the new input instance is calculated through the sensor model obtained by learning.

Category of Perception Machine Model：

• is used to solve the supervision and learning model of the second classification problem
• Non -probability model: Model takes function forms (rather than the conditional probability distribution form of the probability model)
• linear model: the model function is a linear function
• Parameterization model: The dimension of the model parameter is fixed
• Differential model: Learning decision -making functions directly$f(x)$

The main advantage of the perception machine model: The algorithm is simple and easy to implement.

The main disadvantages of the perception machine model: Requires training data sets to be divided.

Fanwat, corresponding to Minkowski Distance. Assuming the N dimension vector$x=(x_1,x_2,\cdots ,x_n{)}^T$, its LP model is recorded as$||x|{|}_p$, defined as

twenty three# . The number of samples has the following definition:$||x|{|}_p=(|x_1{|}^p+|x_2{|}^p+\cdots +|x_n{|}^p{)}^{\frac{1}{p}}$

• Positive settings:$||x||\ge 0$, and there are$||x||=0\iff x=0$；
• positive and secondary:$||cx||=|c|||x||$；
• Adgasability (Triangle Revitalization):$||x+y||\le ||x||+||y||$。

l0 model

Assuming N -dimensional vector$x=(x_1,x_2,\cdots ,x_n{)}^T$, its L0 model is recorded as$||x|{|}_0$, defined as the number of 0 elements in the vector.

l1 model number

Assuming the n -dimensional vector$x=(x_1,x_2,\cdots ,x_n{)}^T$, its L1 model is remembered$||x|{|}_1$, defined as$||x|{|}_1=|x_1|+|x_2|+\cdots +|x_n|$. The L1 model of the vector is the sum of the absolute values of each element in the vector, corresponding to Manhattan Distance.

L2 model

Assuming the n -dimensional vector$x=(x_1,x_2,\cdots ,x_n{)}^T$, its L2 model is remembered$||x|{|}_2$, defined as$||x|{|}_2=(|x_1{|}^2+|x_2{|}^2+\cdots +|x_n{|}^2{)}^{\frac{1}{2}}$. The L2 model of the vector is the square root of each element in the vector, corresponding to the European -style distance (Manhattan Distance).

Infinite model

Assuming the N -dimensional vector$x=(x_1,x_2,\cdots ,x_n{)}^T$, its infinite model is recorded as$||x|{|}_{\infty }$, defined as$||x|{|}_{\infty }=max(|x_1|,|x_2|,\cdots ,|x_n|)$. The infinite number of vectors is the maximum value of the absolute value of each element in the vector, corresponding to the Chebyshev Distance.

Known$S$is N-1-dimensional super plane in N-dimensional European-style space$w\cdot x+b=0$, of which$w$and$x$are all n -dimensional vectors;$x_0=(x_0^{(1)},x_0^{(2)},\cdots ,x_0^{(n)})$. Verification: Click$P$to ultra -flat plane$S$distance$d=\frac{1}{||w|{|}_2}|w\cdot x_0+b|$, among them$||w|{|}_2$$w$2-model.

The proof is as follows:

from super plane

The definition of$S$can be known$w$is ultra -flat$S$The method vector,$b$is ultra -flat$S$The interception.

Set point$x_0$6 6

The projection on$S$is$x_1=(x_1^{(1)},x_1^{(2)},\cdots ,x_1^{(n)})$, there are

$$w\cdot x_1+b=0\text{\hspace{1em}(1)}$$

$P$to ultra -flat plane$S$distance$d$is the vector$\overrightarrow{x_0{x}_1}$The length.

because of$\overrightarrow{x_0{x}_1}$and ultra -flat plane$S$French vector$w$Parallel, so$\overrightarrow{x_0{x}_1}$and French vector angle of the string value$cos\theta =0$, hence

$$\begin{array}{rl}w\cdot \overrightarrow{x_0{x}_1}& =|w||\overrightarrow{x_0{x}_1}|cos\theta \\ & =|w||\overrightarrow{x_0{x}_1}|\\ & =[(w^{(1)}{)}^2+(w^{(2)}{)}^2+\cdots +(w^{(n)}{)}^2{]}^{\frac{1}{2}}d\\ & =||w|{|}_{2}d\end{array}\text{\hspace{1em}(2)}$$

There are again (the distribution law of the application vector point)

$$\begin{array}{rl}w\cdot \overrightarrow{x_0{x}_1}& =w^{(1)}(x_1^{(1)}-x_0^{(1)})+w^{(2)}(x_1^{(2)}-x_0^{(2)})+\cdots +w^{(n)}(x_1^{(n)}-x_0^{(n)})\\ & =(w^{(1)}{x}_1^{(1)}+w^{(2)}{x}_1^{(2)}+\cdots +w^{(n)}{x}_1^{(n)})-(w^{(1)}{x}_0^{(1)}+w^{(2)}{x}_0^{(2)}+\cdots +w^{(n)}{x}_0^{(n)})\\ & =w\cdot x_1-w\cdot x_0\end{array}\text{\hspace{1em}(3)}$$

from the formula (1), there is$w\cdot x_1=-b$, so the style (3) can be written

$$\begin{array}{rl}w\cdot \overrightarrow{x_0{x}_1}& =w\cdot x_1-w\cdot x_0\\ & =-b-w\cdot x_0\end{array}\text{\hspace{1em}(4)}$$

from the formula (2) and the formula (4), get

$$\begin{array}{r}||w|{|}_{2}d=|-b-w\cdot x_0|\\ d=\frac{1}{||w|{|}_2}|w\cdot x_0+b|\end{array}$$

[Supplementary instructions] In the iteration of Example 2.1, the rule of choosing error classification points is to select the smallest error classification points of the index.

For a parameter combination$(w_0,b_0)$, because of the mistake of classification points$M$is fixed, so gradient$\nabla L(w_0,b_0)$is also fixed, the gradient is$w$component$\nabla wL(w_0,b_0)$is the loss function pair$w$partial guidance, gradient in$b$component${\nabla }_{b}L(w_0,b_0)$is the loss function pair$b$partial guidance, so there is a gradient

$${\nabla }_{w}L(w_0,b_0)=L_w^{\prime }(w_0,b_0)=-\sum _{x_i\in M}{y}_i{x}_i$$

$${\nabla }_{b}L(w_0,b_0)=L_b^{\prime }(w_0,b_0)=-\sum _{x_i\in M}{y}_i$$

Because all sample points are required to complete the gradient, the time cost is high, so here we use the random gradient drop method of fast time speed. In each iteration process, it is not a way to make it

All the gradients of all error classification points in$M$are decreased, but a random classification point is randomly selected to reduce its gradient. For a single error classification point$(x_i,y_i)$, gradient

$${\nabla }_{w}L(w_0,b_0)=L_w^{\prime }(w_0,b_0)=-y_i{x}_i$$

$${\nabla }_{b}L(w_0,b_0)=L_b^{\prime }(w_0,b_0)=-y_i$$

Based on this update$w$and$b$：

$$w\leftarrow w+\eta (-{\nabla }_{w}L(w_0,b_0))=w+\eta y_i{x}_i$$

$$b\leftarrow b+\eta (-{\nabla }_{b}L(w_0,b_0))=b+\eta y_i$$

Among them,$\eta$is a step -long, usually the range of the value is$(0,1]$, also known as learning rate.

【source code address】code.perceptron.original_form_of_perceptron

# https://github.com/ChangxingJiang/Data-Mining-HandBook/blob/master/code/perceptron/_original_form.py

def original_form_of_perceptron(x, y, eta):
    "" Performing machine learning algorithm original form 

     : Param X: Input variable 
     : Param Y: Output variable 
     : Param Eta: Learning Rate 
     : RETURN: W and B of the Perception Machine model 
     "" "
    n_samples = len(x)  # Sample point quantity
    n_features = len(x[0])  # Feature vector dimension
    w0, b0 = [0] * n_features, 0  # Select the initial value W0, B0

    while True:  # constantly iterate until there is no error classification point
        for i in range(n_samples):
            xi, yi = x[i], y[i]
            if yi * (sum(w0[j] * xi[j] for j in range(n_features)) + b0) <= 0:
                w1 = [w0[j] + eta * yi * xi[j] for j in range(n_features)]
                b1 = b0 + eta * yi
                w0, b0 = w1, b1
                break
        else:
            return w0, b0

【source code address] Test

>>> from code.perceptron import original_form_of_perceptron
>>> dataset = [[(3, 3), (4, 3), (1, 1)], [1, 1, -1]]
>>> original_form_of_perceptron(dataset[0], dataset[1], eta=1)
([1, 1], -3)

[Supplementary instructions] Expansion of the weight vector: The vector of the weight vector of the expansion right is about to be biased into the weight vector$\hat{w}=(w^T,b{)}^T$. (The name of the “Expansion of Rights” is directly used below)

[Supplementary instructions] The number of mistakes classification: that is, there are the number of iterations of the wrong classification instance.

$||{\hat{w}}_{opt}||=1$？

May wish to set the over -the -plans to expand the weight vector to

$${\hat{w}}_{opt}^{\prime }=({w^{\prime }}_{opt}^T,b^{\prime }{)}^T=(w_{opt}^{\prime (1)},w_{opt}^{\prime (2)},\cdots ,w_{opt}^{\prime (n)},b^{\prime }{)}^T$$

$||{\hat{w}}_{opt}^{\prime }||!=1$, so exist

$${\hat{w}}_{opt}=(\frac{w_{opt}^{\prime (1)}}{||{\hat{w}}_{opt}^{\prime }||},\frac{w_{opt}^{\prime (2)}}{||{\hat{w}}_{opt}^{\prime }||},\cdots ,\frac{w_{opt}^{\prime (n)}}{||{\hat{w}}_{opt}^{\prime }||},\frac{b^{\prime }}{||{\hat{w}}_{opt}^{\prime }||}{)}^T$$

At this time$||{\hat{w}}_{opt}||=1$. Send a certificate.

For example:$x^{(1)}+x^{(2)}-3$The weight vector of the expansion right is$\widehat{w^{\prime }}=(1,1,-3{)}^T$，$||\widehat{w^{\prime }}||=\sqrt{11}$, so there is$\hat{w}=(\frac{1}{\sqrt{11}},\frac{1}{\sqrt{11}},\frac{-3}{\sqrt{11}}{)}^T$, make

The$||\hat{w}||=1$。

Conclusion is the higher operational efficiency of the puppet form of the perceptual learning algorithm under certain conditions. Let’s discuss below.

First of all, consider the original form and time complexity of the perception of the learning algorithm. The complexity of the time of each iteration of the original form is$O(S\times N)$, the total time complexity is$O(S\times N\times K)$; The complexity of each iteration of each form is$O(S^2)$, also need to calculate the time complexity of the Gram matrix$O(S^2\times N)$, the total time complexity is$O(S^2\times N+S^2\times K)$; Among them$S$is the number of sample points,$N$is the dimension of the sample feature vector,$K$is the number of iterations.

Because the original form and the iterative steps of the puppet form correspond to each other, in general, the original form is more suitable for training data with less dimensions and high quantities, and it is more suitable for training data with high dimensions and less quantities.

【source address】code.perceptron.count_gram

# https://github.com/ChangxingJiang/Data-Mining-HandBook/blob/master/code/perceptron/_gram.py

def count_gram(x):
    "" Calculating GRAM matrix 

     : Param X: Input variable 
     : Return: Enter the gram matrix of the variable 
     "" "
    n_samples = len(x)  # Sample point quantity
    n_features = len(x[0])  # Feature vector dimension
    gram = [[0] * n_samples for _ in range(n_samples)]  # initialized GRAM matrix

    # Calculate Gram matrix
    for i in range(n_samples):
        for j in range(i, n_samples):
            gram[i][j] = gram[j][i] = sum(x[i][k] * x[j][k] for k in range(n_features))

    return gram

【source address】code.perceptron.dual_form_perceptron

# https://github.com/ChangxingJiang/Data-Mining-HandBook/blob/master/code/perceptron/_dual_form.py

from . import count_gram  # code.perceptron.count_gram

def dual_form_perceptron(x, y, eta):
    "" "Performing machine learning algorithm format form 
     : Param X: Input variable 
     : Param Y: Output variable 
     : Param Eta: Learning Rate 
     : Return: A (alpha) and B of the perception machine model 
     "" "
    n_samples = len(x)  # Sample point quantity
    a0, b0 = [0] * n_samples, 0  # Select the initial value A0 (alpha), B0
    gram = count_gram(x)  # Calculate Gram matrix

    while True:  # continuously iterate until there is no error classification point
        for i in range(n_samples):
            yi = y[i]

            val = 0
            for j in range(n_samples):
                xj, yj = x[j], y[j]
                val += a0[j] * yj * gram[i][j]

            if (yi * (val + b0)) <= 0:
                a0[i] += eta
                b0 += eta * yi
                break
        else:
            return a0, b0

【Source code address] Test

>>> from code.perceptron import dual_form_perceptron
>>> dataset = [[(3, 3), (4, 3), (1, 1)], [1, 1, -1]]  # 1 1 1
>>> dual_form_perceptron(dataset[0], dataset[1], eta=1)
([2, 0, 5], -3)