In this talk I will explain Fourier analytical insights into RNNs and DNNs.

  RNN Abstract: Glipin (2023) performed a comparative study of 24 representative machine learning models using a variety of data. CNNs and RNNs required very long training times, on the other hand, reservoir computing (RC) exhibited competitive performance with two orders of magnitude less learning time than those methods. However, RC needs to employ random variables for the recurrent and input weight matrices, so, it seems that new ideas are needed for further improvement of the RC model. In this talk, I will show that it is possible to explicitly construct ``pattern memory matrices of time series" and replace the random matrices to such pattern memories (arXiv:2310.00290).

  DNN Abstract: Imaizumi-Fukumizu (2019) and Suzuki (2019) showed that DNNs are advantageous (compared to conventional methods) for estimating discontinuous functions and functions with non-uniform smoothness. These would be suggestive results. For example, in image recognition, we essentially determine whether each given image is ``match" or ``not match", and this ``determination" can be translated into discontinuous surfaces (related to non-separability) in the feature space. On the other hand, Cybenko (1989) and other conventional universal approximation theorems are implicitly based on ``separability of function spaces". In this talk, I will specifically point out the importance of ``non-separability" in DNNs (arXiv:2304.08172).