The Cross-Layer Connectivity for Deep Learning From Net95 to ResNet and DenseNet

The Cross-Layer Connectivity for Deep Learning From Net95 to ResNet and DenseNet Yaqing Yang [email protected] Department of Data Science & Artificial Intelligence Hong Kong Polytechnic University & Hong Kong, China Abstract—This brief survey traces a connectivity-first lineage in deep learning: from the early Net95 network with non-adjacent layer connections to the residual reformulation in ResNet and the dense concatenation strategy in DenseNet. We summarize each work’s core idea, empirical footprint, and how their wiring patterns improve gradient flow, information preservation, and feature reuse. We also note archival context for Net95 to facilitate further study. Index Terms—Net95, non-adjacent connection, residual learning, dense connectivity, shortcut connections, feature reuse Cross-layer connectivity underpins trainable deep networks: Net95 (1995) added non-adjacent feed-forward links, improving convergence and car-shape recognition [1]1;ResNet (2015/2016) learned residuals with identity shortcuts, enabling very deep models and dealing with vanishing gradient problem [2]DenseNet (2017) concatenated all prior features per layer, similar to Net95, strengthening gradients and reuse with high parameter efficiency [3]. I. FROM NET95 TO RESNSET AND DENSENET A. Net95: Non-Adjacent Layer Connections Net95 augments a multilayer perceptron with direct connections between neurons among non-adjacent layers. The each output of layer K in Feed-Forward Network of Net95 is computed with YKi =fPK−1 L=1 PNL j=1 WKLij ·YLj +θKi. The corresponding BP derivation aggregates multi-hop contributions from upper layers, effectively shortening optimization paths and widening gradient routes. In the work, the authors observed improved convergence and accuracy over a comparable adjacent-only BP baseline [1]. Conceptually, Net95 anticipates the later shortcut philosophy: provide short, information-preserving routes across depth. B. ResNet: Residual Learning with Identity Shortcuts ResNet reformulates the learning target from H(x)to a residual F(x) = H(x)−x, yielding y=F(x) + xwith identity skip connections. Identity paths preserve the input representation and allow gradients to propagate unimpeded, making near-identity solutions easy to represent and optimize. This resolves degradation in deep plain nets and enables very deep models (e.g., 152 layers) with state-of-the-art ImageNet 1Archival note. Since many early IEEE conference papers are hard to access online, we provide a scanned and lightly re-typeset L A T EX version of Net95 [1] solely for scholarly reference, by the agreement of the author [email protected]. results [2]. In practice, residual blocks generalize robustly across recognition, detection, and segmentation. C. DenseNet: Dense Concatenation and Feature Reuse DenseNet increases connectivity further: each layer consumes all previous feature maps and contributes a small “growth rate” of new maps, forming L(L+ 1)/2explicit forward paths. Unlike ResNet’s addition, DenseNet concatenates features so that later layers directly reuse earlier representations, yielding strong data/parameter efficiency and implicit deep supervision across the network [3]. II. LINKS AND CONTRASTS Optimization motif. All three designs reduce effective path length for signals and gradients: Net95 via explicit nonadjacent weights, ResNet via identity addition, DenseNet via cumulative concatenation. Information handling. ResNet preserves inputs through addition while learning refinements; DenseNet reuses features via concatenation; Net95 mixes multi-layer signals in an early feed-forward setting. Cost and scaling. ResNet scales depth with modest overhead; DenseNet trades wider inputs for reuse and parameter economy; Net95 demonstrates the pre-CNN intuition that cross-layer routes aid convergence and recognition. In terms of topological structure, the three should be essentially consistent. III. CONCLUSION Non-adjacent connections (Net95), residual additions (ResNet), and dense concatenations (DenseNet) constitute a coherent evolution of cross-layer communication. Their crosslayer principle remains foundational and continues to inspire hybrid architectures. REFERENCES [1] Can Yang, Dehua Li, and Yanwen Zhu. “A Car Shape Classification Based on Neural Networks with Non-Adjacent Layers Connection.” In Proc. IEEE Int. Conf. Neural Netw. Signal Process. (ICNNSP95), pp. 697–701, Dec. 1995. [2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. “Deep Residual Learning for Image Recognition.” In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 770–778, 2016. [3] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. “Densely Connected Convolutional Networks.” In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 2261–2269, 2017. Net95∗:A Car Shape Classification Based on Neural Networks with Non-adjacent Layer Connection Can Yang, Dehua Li, Yanwen Zhu Inst. of Pattern Recognition & Artificial Intelligence Huazhong Univ. of Sci. & Tech., Wuhan, Hubei 430074, P.R. China Abstract—Car shape recognition is a research subject of wide application. Traditional pattern recognition techniques will generally fail due to the complexity and variability of background and the vague effect of car movement. In this paper, a new classificator based on neural network, a feed-forward net with two hidden layers and with non-adjacent layer connection, is designed. The net is trained with the input of the effectual features extracted from samples and the output of expected patterns. Based upon BP algorithm, error back propagation learning algorithm for the classificator is given. Experimental result shows the correctness of theoretical analysis and the good convergence of the network and is satisfactory. Index Terms—Pattern recognition, Neural network, Feature extraction I. INTRODUCTION In the application of pattern recognition, two-dimensional image recognition is of most interest. Car shape recognition, a typical example of 2-D image recognition, suffers from both the universal problems existing in the field of pattern recognition and the vague effect of car movement. Meanwhile, the difficulty of feature extraction is increased because of the complex and unpredictable background. Traditional pattern recognition techniques are not suited for this kind of applications. Fortunately, in recent years, much progress has been made in the car industry and modernization of communications, which stimulates the rapid development of car shape recognition techniques. For an image whose edge feature is useful or whose feature can be its area, we can extract the edge feature of the corresponding binary image to reduce information which may be too much for 2-D images. In our experiment, three types of car models amounting to 103 samples are taken. The classification is performed based on edge feature extraction * This revised version was proofread and edited by Yang Yaqing, Hong Kong Polytechnic University, Aug. 2025. In this resubmission, the 1995 work is referred to as Net95, and minor revisions to the original are highlighted in blue. The citation for the original Net95 is: C. Yang, D. Li, and Y. Zhu, A car shape classification based on neural networks with non-adjacent layers connection, in Proc. IEEE Int. Conf. Neural Netw. Signal Process. ICNNSP95, Dec 1995, pp. 697701. from the binary image of the car’s silhouette using autocorrelation function. Also, a discussion about the network’s structural effect on recognition result is presented. II. PREPROCESSING AND FEATURE EXTRACTION A. Image preprocessing Car samples used in our experiment are 103 black-white photos taken from three different types of real cars and from different angles. We use Threshold Curve Method[1] for binary processing of image, Adaptive Smoothing Technique[2] for elimination of noise, Roblinson Operator[3] for edge detection and Doutch Technique[4] for edge thinning. The details are not discussed here. Interested readers can refer to [1], [2], [3] and [4]. B. Feature extraction[5] The basic principle for feature extraction of car patterns requires three properties: i. translation invariance; ii. scale invariance; iii. rotation invariance. In general, two types of features can be extracted for pattern recognition: i. structural feature; ii. mathematical model feature. We choose the second type of feature and the autocorrelation function values as the input of the network. Fig. 1. Suppose that Iis a binary image in XOY plane, the centre of gravity of car silhouette is taken as the origin of XOY coordinate system (shown in Fig. 1). f(x, y)is a binary function defined in XOY plane: f(x, y) = (1,for (x, y)∈I, 0,for (x, y)∈I. (1) Obviously, the area of Ican be A=Z+∞ −∞ Z+∞ −∞ f(x, y)dx dy. We can define g(u, v) = Z+∞ −∞ Z+∞ −∞ f(x, y)f(x+u, x +v)dx dy. (2) Fig. 2. Where g(u.v) is the auto-correlation function of f(x, y). Geometrically, g(u, v) represents the area of overlapped region by 1(u, v) and 1(0.0), with 1(0,0) being the original image and I(u, v) the translated one of the original from (0, 0) to (u, v) (shown in Fig.2). Thus, we obtain the standard auto-correlation function, G0(u, v) = g(u, v) A=Z+∞ −∞ Z+∞ −∞ f(x, y)f(x+u, x +v)dx dy Z+∞ −∞ Z+∞ −∞ f(x, y)dx dy .(3) As can be easily seen, G0(u, v)is both translation invariant and scale invariant, but not rotation invariant. For this reason, we can discretize G0(u, v)under the polar coordinate system (r, θ). The circular property with respect to dof the discrete values eliminates the sensitiveness of image feature produced by image rotation. Let u=a√Acos θ, v=a√Asin θ. (4) Substituted with (4), (3) can be rewritten as G(a, θ) = G0 a√Acos θ, a√Asin θ =Z+∞ −∞ Z+∞ −∞ f(x, y)fx+a√Acos θ, x +a√Asin θdx dy Z+∞ −∞ Z+∞ −∞ f(x, y)dx dy . (5) From (5), we can easily verify the following relation: G(a, θ +π) = G(a, θ).(6) That is to say, G(a, θ)is periodic with πwith respect to θ. Consequently, our consideration can be limited to θ∈[0, π). Let a=ai, i = 1,2, . . . , M, (7) θ=θj=(j−1)π N, j = 1,2, . . . , N, (8) where Mis quantitation granule of polar radius and θthe rotation angle. Substituting (5) with (7) and (8), we obtain discrete values of G(a, θ)which can be regarded as the feature of image I. The feature vector of Iis given by Fc=G(a1, θ1), G(a2, θ2), . . . , G(aM, θN)T.(9) Based on the above, 103 sample patterns are taken and a 12 (M= 3,N= 4) dimension feature vector is constructed for each. Then, 48 sample patterns are randomly picked out as training set from three types of samples and the rest makes up the testing set. The training set and the testing one are used for the network’s train and test respectively. III. NEURAL NETWORK CLASSIFICATOR Neural network classificator is a non-linear dynamic system which can be applied to solve the problems with complex environment information, indefinite background knowledge and reasoning rules. It tolerates the great loss and distortion of samples. The neural network classifier will classify an unknown pattern as the most recently retained one by training with the samples and retaining the results. 1. The structure of neural network Fig. 3 shows the topological structure of the network designed in our work. Table I gives the detailed information of it. 2. Algorithm of neural network General BP network, with adjacent layer all connection or local connection, can be equivalent to a biological circuit model of series connection (shown in Fig.4). But in our design, the network with non-adjacent layer connection can be equivalent to a biological circuit model of both series and 2 Fig. 3. Topological structure of network TABLE I DETAILED INFORMATION OF NEURAL NETWORK layer order layer name number of nodes connection activation function learning rule initial scope of weight 1 2 3 4 1 input 12 / / / / / / / 2 hidden 1 16 all / / / sigmoid delta [-0.1, 0.1] 3 hidden 2 6 all all / / sigmoid delta [-0.1, 0.1] 4 output 3 all all all / sigmoid delta [-0.1, 0.1] Fig. 4. Circuit model of general BP net parallel connection (shown in Fig.5, →represents neural fibre and neural node group). Because we introduce such model as shown in Fig. 5, the sum function and the formula for computing correction parameter of BP algorithm will change correspondingly. Let XKi represent the input of the ith node in the Kth layer and YKj the output of the same node in the same layer. Let WKLij denote the connecting weight between the ith node in the Kth layer and the jth node in the Lth layer, θKi the threshold of the ith node in the Kth layer and NLthe number of nodes in the Lth layer. Thus, we have YKi =         XKi, K = 1, f  K−1 X L=1 NL X j=1 WKLji YLj +θKi , K > 1,(10) Fig. 5. Circuit model of net in our design where f(∗) = 1 1 + e−∗ . Where f(∗) = 1 1+e−∗ . The deduction of correspondent formula for computing error back propagation and connectioing weight is given below. Let EPbe error function in output layer when the network is fed with pattern pair (XP, YP), it be given by, EP=1 2 NP X j=1 ¯ YT j −YT j2.(11) Where ¯ YT j is the expected output of the jth output layer node, YT j is the real output of it. NPis the number of output layer nodes. PEPis the total error accumulated by all sample 3 patterns in the training set. Let net YKj = K−1 X L=1 NL X i=1 WKLji YLi.(12) be the input sum function of the jth node in the Kth layer. The real output of it is designated by (10) and f(·) is differentiable and non-decreased progressively. For node jin layer K, the general error is defined as eKj =−∂EP ∂net YKj =−∂EP ∂YKj ·∂YKj ∂net YKj =−∂EP ∂YKj ·f0 net YKj .(13) If K=P, for output layer node, from (11) we have ∂EP ∂YP j =∂ ∂YP j   1 2 NP X j=1 ¯ YP j −YP j 2 =−¯ YP j −YP j ,(14) and therefore eP j =¯ YP j −YP jf0net YP j .(15) If K < P , for hidden layer node, we have ∂EP ∂YKj = P X L=K+1 NL X i=1 ∂EP ∂net YLi ·∂net YLi ∂YKj = P X L=K+1 NL X i=1 eLi WLKij. (16) Substituted with (16), (13) can be rewritten as eKj =f0net YKj  P X L=K+1 NL X i=1 eLi WLKij.(17) (17) can be regarded as error propagated from all nodes of former (P →K) layers to all nodes of Kth layer. To reduce the error function EP, we adjust the connecting weight using Gradient Descent Method based upon the original connecting weight WKLij: ∆WKLij =−a∂EP ∂WKLij =−a∂EP ∂net YKi ·∂net YKi ∂WKLij =a eKi YLj , (18) where ais learning rate 0<a<1,eKi is expressed by (17) and YLj is the output of the jth node in the Kth layer. From above deduction, it can be seen that the sum function is significantly increased and f(·) is rapidly trends to be stable by this kind of connection. Furthermore, the learning process is accelerated. Although the increase of multiply and add operation times, the computational cost of activation function f(·) is greatly reduced and the model designed in our work is more similar to biological neural network structure. Experiment reveals the correctness of our analysis. Compared with general BP network, the network designed in our experiment have a good convergence and satisfactory recognition ratio is obtained. IV. EXPERIMENT AND DISCUSSION The neural network shown in Fig. 3 is a highly reliable structure which has been designed through many experiments over and over again. The classificator is trained 500,000 times with 48 training samples and tested with 55 testing samples. The experimental results are shown in Table II. TABLE II EXPERIMENT RESULTS set type number of samples wrong recognition ratio refusive recognition ratio right recognition ratio training set 48 0 0 100% test set 55 0 5.55% 94.45% In order to reduce wrong-recognition ratio to the greatest extent, an appropriate judge function is chosen to increase the refusive-recognition ratio. If 1.85% wrong-recognition ratio is allowed, the right recognition ratio will increase to 98.65%. Note: our formula for computing recognition ratio is given by R=Number of correctly recognized testing samples Total Number of testing samples .(19) To verify the advantages of network designed in our work, a comparison experiment is carried out between general BP feed-forward propagation net and ours. The number of layers and nodes in each corresponds to what is shown in Fig.3, but there is no non-adjacent layer connection. Table III shows the results. As can be seen from the experimental results, when certain training time is reached, dependability of the network on samples in testing set is increased, which results in the increase of recognition ratio of samples in training set and the decrease of recognition ratio of samples in testing set. Fig.6.a and Fig.6.b gives the mean-square-error curve in the former 16,000 trains for the two types of network. V. CONCLUSION In this paper, a new way cf weight connection is adopted and a new algorithm for error back propagation is deduced based upon general BP algorithm. Our network has an excellent performance and a rapid convergence speed which is noticeably superior to general BP network. Satisfactory results is obtained in our application to car shape recognition. Though our basic idea about learning algorithm inherits that about traditional BP algorithm and local minimization problem is not satisfactorily solved, our research work, as an application technology, is of 4 TABLE III EXPERIMENTAL RESULTS FOR GENERAL BP NET set type number of samples training times wrong recognition ratio refusive recognition ratio right recognition ratio training set 48 50,000 3/48 = 6.3% 1/48 = 2.1% 91.6% 100,000 2/48 = 4.2% 1/48 = 2.1% 93.7% 150,000 2/48 = 4.2% 1/48 = 2.1% 93.7% test set 55 50,000 3/55 = 5.6% 5/55 = 9.1% 85.3% 100,000 3/55 = 5.6% 5/55 = 9.1% 85.3% 150,000 2/55 = 3.6% 7/55 = 12.7% 83.7% (a) Net95 in our experiment (b) General BP net Fig. 6. Curves for mean square error great importance to theoretical study on neural network and development of pattern recognition techniques. REFERENCE [1] S.D.Yanowitz and A.M.Bruckstein, A New Method for Image Segmentation, Computer Vision, Graphics and Image Processing, 1989 (46) : 82-95 [2] M.Nagao and T.Matsuyam, A Structural Analysis of Complex Aerial Photographs, Plenum Press, 1980 [3] G.Robinson, Eige Detection by Compassgradient mask, CG1 P6, 1977:492—501 [4] Deutch, E.S. Thinning algorithms on rect-anglular, hexagonal, and trianglar arrays, Communications of ACM, 15, 9, 1972 [5] Licheng Jiao, Application and Implementation of Neural Network, 1993: 04-20 [6] Rumelhart DE, MeClelland JL. Parallel Distributed Processing,18(2): MIT Press, Cambridge, MA, 1986 5