Ronny Ramlau, Lothar Reichel (Hg.)


ETNA - Electronic Transactions on Numerical Analysis






ISBN 978-3-7001-8258-0
Online Edition

  Research Article
Open access


Electronic Transactions on Numerical Analysis (ETNA) is an electronic journal for the publication of significant new developments in numerical analysis and scientific computing. Papers of the highest quality that deal with the analysis of algorithms for the solution of continuous models and numerical linear algebra are appropriate for ETNA, as are papers of similar quality that discuss implementation and performance of such algorithms. New algorithms for current or new computer architectures are appropriate provided that they are numerically sound. However, the focus of the publication should be on the algorithm rather than on the architecture. The journal is published by the Kent State University Library in conjunction with the Institute of Computational Mathematics at Kent State University, and in cooperation with the Johann Radon Institute for Computational and Applied Mathematics of the Austrian Academy of Sciences (RICAM). Reviews of all ETNA papers appear in Mathematical Reviews and Zentralblatt für Mathematik. Reference information for ETNA papers also appears in the expanded Science Citation Index. ETNA is registered with the Library of Congress and has ISSN 1068-9613.

Verlag der Österreichischen Akademie der Wissenschaften
Austrian Academy of Sciences Press
A-1011 Wien, Dr. Ignaz Seipel-Platz 2
Tel. +43-1-515 81/DW 3420, Fax +43-1-515 81/DW 3400
https://verlag.oeaw.ac.at, e-mail: verlag@oeaw.ac.at

Bestellung/Order


ETNA - Electronic Transactions on Numerical Analysis



ISBN 978-3-7001-8258-0
Online Edition



Send or fax to your local bookseller or to:

Verlag der Österreichischen Akademie der Wissenschaften
Austrian Academy of Sciences Press
A-1011 Wien, Dr. Ignaz Seipel-Platz 2,
Tel. +43-1-515 81/DW 3420, Fax +43-1-515 81/DW 3400
https://verlag.oeaw.ac.at, e-mail: bestellung.verlag@oeaw.ac.at
UID-Nr.: ATU 16251605, FN 71839x Handelsgericht Wien, DVR: 0096385

Bitte senden Sie mir
Please send me
 
Exemplar(e) der genannten Publikation
copy(ies) of the publication overleaf


NAME


ADRESSE / ADDRESS


ORT / CITY


LAND / COUNTRY


ZAHLUNGSMETHODE / METHOD OF PAYMENT
    Visa     Euro / Master     American Express


NUMMER

Ablaufdatum / Expiry date:  

    I will send a cheque           Vorausrechnung / Send me a proforma invoice
 
DATUM, UNTERSCHRIFT / DATE, SIGNATURE

BANK AUSTRIA CREDITANSTALT, WIEN (IBAN AT04 1100 0006 2280 0100, BIC BKAUATWW), DEUTSCHE BANK MÜNCHEN (IBAN DE16 7007 0024 0238 8270 00, BIC DEUTDEDBMUC)
X
BibTEX-Export:

X
EndNote/Zotero-Export:

X
RIS-Export:

X 
Researchgate-Export (COinS)

Permanent QR-Code

doi:10.1553/etna_vol56s157



doi:10.1553/etna_vol56s157



Thema: natural
Ronny Ramlau, Lothar Reichel (Hg.)


ETNA - Electronic Transactions on Numerical Analysis






ISBN 978-3-7001-8258-0
Online Edition

  Research Article
Open access


Linyan Gu, Wei Zhang, Jia Liu, Xiao-Chuan Cai
S.  157 - 186
doi:10.1553/etna_vol56s157

Open access

Verlag der Österreichischen Akademie der Wissenschaften


doi:10.1553/etna_vol56s157
Abstract:
Deep convolutional neural network (DCNN) has led to significant breakthroughs in deep learning. However, larger models and larger datasets result in longer training times slowing down the development progress of deep learning. In this paper, following the idea of domain decomposition methods, we propose and study a new method to parallelize the training of DCNNs by decomposing and composing DCNNs. First, a global network is decomposed into several sub-networks by partitioning the width of the network (i.e., along the channel dimension) while keeping the depth constant. All the sub-networks are individually trained, in parallel without any interprocessor communication, with the corresponding decomposed samples from the input data. Then, following the idea of nonlinear preconditioning, we propose a sub-network transfer learning strategy in which the weights of the trained sub-networks are recomposed to initialize the global network, which is then trained to further adapt the parameters. Some theoretical analyses are provided to show the effectiveness of the sub-network transfer learning strategy. More precisely speaking, we prove that (1) the initialized global network can extract the feature maps learned by the sub-networks; (2) the initialization of the global network can provide an upper bound and a lower bound for the cost function and the classification accuracy with the corresponding values of the trained sub-networks. Some experiments are provided to evaluate the proposed methods. The results show that the sub-network transfer learning strategy can indeed provide good initialization and accelerate the training of the global network. Additionally, after further training, the transfer learning strategy shows almost no loss of accuracy and sometimes the accuracy is higher than if the network is initialized randomly.

Keywords:  deep convolutional neural networks, decomposition and composition, parallel training, transfer learning, domain decomposition
  2022/03/01 10:50:44
Object Identifier:  0xc1aa5572 0x003d44fb
.

Electronic Transactions on Numerical Analysis (ETNA) is an electronic journal for the publication of significant new developments in numerical analysis and scientific computing. Papers of the highest quality that deal with the analysis of algorithms for the solution of continuous models and numerical linear algebra are appropriate for ETNA, as are papers of similar quality that discuss implementation and performance of such algorithms. New algorithms for current or new computer architectures are appropriate provided that they are numerically sound. However, the focus of the publication should be on the algorithm rather than on the architecture. The journal is published by the Kent State University Library in conjunction with the Institute of Computational Mathematics at Kent State University, and in cooperation with the Johann Radon Institute for Computational and Applied Mathematics of the Austrian Academy of Sciences (RICAM). Reviews of all ETNA papers appear in Mathematical Reviews and Zentralblatt für Mathematik. Reference information for ETNA papers also appears in the expanded Science Citation Index. ETNA is registered with the Library of Congress and has ISSN 1068-9613.



Verlag der Österreichischen Akademie der Wissenschaften
Austrian Academy of Sciences Press
A-1011 Wien, Dr. Ignaz Seipel-Platz 2
Tel. +43-1-515 81/DW 3420, Fax +43-1-515 81/DW 3400
https://verlag.oeaw.ac.at, e-mail: verlag@oeaw.ac.at