進階搜尋


   電子論文尚未授權公開,紙本請查館藏目錄
(※如查詢不到或館藏狀況顯示「閉架不公開」,表示該本論文不在書庫,無法取用。)
系統識別號 U0026-1907202021260400
論文名稱(中文) 新型非揮發性記憶體於晶片中之類神經網路訓練應用探討
論文名稱(英文) Exploration of Emerging Non-Volatile Memory for On-chip Training of Artificial Neural Network
校院名稱 成功大學
系所名稱(中) 微電子工程研究所
系所名稱(英) Institute of Microelectronics
學年度 108
學期 2
出版年 109
研究生(中文) 蔡承憲
研究生(英文) Cheng-Hsien Tsai
學號 Q16074207
學位類別 碩士
語文別 英文
論文頁數 72頁
口試委員 指導教授-盧達生
口試委員-林英超
口試委員-陳貞夙
口試委員-張順志
中文關鍵字 非揮發性記憶體  記憶體中運算  晶片上類神經網路訓練 
英文關鍵字 Non-volatile memory  Computing-in-memory  on-chip learning 
學科別分類
中文摘要 近年來隨著物聯網與人工智慧高速發展,需要運算大量的資料。在傳統的馮紐曼電腦架構中,因為資料計算時,需要搬運資料往返於中央處理器與記憶體,而消耗大量能量與時間。各界開始尋求高速且低功耗的記憶體中運算架構。由非揮發性記憶體組成的記憶體陣列,可將計算與儲存同時進行達成「記憶體內運算」,並具備平行化處理大量矩陣運算的能力,成為近年來熱門的研究項目。本篇論文旨在探討非揮發性記憶體元件之特性如有限權重位元、變異性、非線性突觸特性對神經網路訓在晶片上訓練之準確度與功耗的影響。並改良固有的權重更新方式,優化訓練準確度。
本篇論文透過以Python及Tensorflow 為基礎寫成的非揮發性記憶體於類神經網路之模擬驗證平台,利用突觸行為模型擬合真實元件特性。並根據元件特性加入常態分佈或對數常態分布之元件間變異性與寫入周期間變異性。並探討在不同輸入方式、電路架構與權重更新演算法下,元件之非理想效應如何影響類神經網路之預測準確率。最後探討三種非揮發性記憶體,包含「電阻式記憶體」、「相變化記憶體」、「鐵電電晶體」當作類神經網路中之突觸(或稱權重)準確度與功耗。
英文摘要 In recent years, with the rapid development of internet of things (IOT) and artificial intelligence (AI), a large amount of data needs to be processed. In the traditional von-Neumann computer architecture, data moves back and forth between separated processing and memory units costing time and energy. This phenomenon is known as von-Neumann bottleneck. To eliminate the von-Neumann bottleneck, a new computational approach named computing-in-memory (CIM) is adopted using memory array composed of emerging non-volatile memory (eNVM). This thesis aims to explore the impacts of nonideal properties of eNVM, such as limited weight bits, variability and nonlinear synaptic behavior on accuracy and power consumption of on-chip training of neural network. And improve the inherent weight update method to optimize training accuracy.
This thesis uses the non-volatile memory based neuromorphic simulation platform, which is written in Python and Tensorflow. Using the synaptic behavior model to fit the characteristics of real devices and add the device-to-device (D2D) and cycle-to-cycle (C2C) variation. Besides, this thesis also discusses the accuracy and energy consumption under different synapse architecture, input scheme and weight update method. Finally, real devices including RRAM, PCM and FeFET is simulated to benchmark different technology.
論文目次 摘要 i
Abstract ii
Acknowledgment iii
Contents iv
List of Table vii
List of Figure viii

Chapter 1. Introduction 1
1.1 Research background and motivation 1
1.2 Research Objective 3
Chapter 2. Literature Review 4
2.1 Artificial Intelligence 4
2.2 Artificial Neural Networks 5
2.2.1 Multilayer Perceptron (MLP) 5
2.2.2 Convolutional Neural Network (CNN) 9
2.2.3 Spiking Neural Network (SNN) 10
2.3 Emerging nonvolatile memory 11
2.3.1 Resistive Random-Access Memory (RRAM) 11
2.3.2 Phase Change Memory (PCM) 15
2.3.3 Ferroelectric field-effect transistor (FeFET) 17
2.4 NVM-based array architecture 19
2.4.1 NVM-based crossbar array 19
2.4.2 NVM-based synapse circuit architecture 21
2.5 NVM-based program scheme 23
Chapter 3. Methodology 24
3.1 Synapse device modeling - 24
3.2 Simulation platform 26
3.2.1 Framework 26
3.2.2 Forward Propagation 27
3.2.3 Backpropagation 28
3.2.4 Weight update method 30
3.2.4.1 1D1S linear weight update method 30
3.2.4.2 High precision digital-assisted weight update method 34
3.2.4.3 2D1S Weight update method 36
3.2.4.4 Input scheme 37
3.3 Nonideal characteristics 39
1. Bit Precisio 39
3.4 Energy consumption 43
3.4.1 Forward energy consumption 44
3.4.2 Backward energy consumption 45
3.4.3 Update energy consumption 45
Chapter 4. Result and Discussion 47
4.1 Bit level limitation and nonlinearity 47
4.2 Variation 50
4.2.1 Device-to-device variation 50
1. D2D nonlinearity variation 50
2. D2D conductance variation 50
4.2.2 Cycle-to-cycle variation 54
4.3 Evaluation of impact of reliability 57
4.4 Real device simulation 59
Chapter 5. Conclusion 63
Answer to Thesis Defense Questions 64
Reference 67
參考文獻 [1] Bing Chen et al., "Efficient in-memory computing architecture based on crossbar array," in IEEE International Electron Devices Meeting (IEDM), Washington, DC, USA, 2015.
[2] "Application of AI," Javapoint, [Online]. Available: https://www.javatpoint.com/application-of-ai.
[3] "McCulloch-Pitts Model — Mankind’s First Mathematical Model Of A Biological Neuron," BREAKING NEWS, 24 Jul 2018. [Online]. Available: https://mc.ai/mcculloch-pitts-model-mankinds-first-mathematical-model-of-a-biological-neuron/.
[4] Vivienne Sze et al., "Efficient Processing of Deep Neural Networks: A Tutorial and Survey," Proceedings of the IEEE, pp. 2295 - 2329, 20 Nov 2017.
[5] S. Saha, "A comprehensive guide to convolutional neural networks," [Online]. Available: https://towardsdatascience.com/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53.
[6] Markram Henry et al., "A History of Spike-Timing-Dependent Plasticity," Frontiers in Synaptic Neuroscience, p. 4, 29 Aug 2011.
[7] Qingzhou Wan et al., "Emerging Artificial Synaptic Devices for Neuromorphic Computing," Advanced Materials Technologies, p. 1900037, 6 Mar 2019.
[8] Choi S. et al., "SiGe epitaxial memory for neuromorphic computing with reproducible high performance based on engineered dislocations," Nature Mater, vol. 17, pp. 335-340, 2018.
[9] H.-S. Philip Wong et al., "Metal–Oxide RRAM," Proceedings of the IEEE, vol. 100, no. 6, pp. 1951-1970, 2012.
[10] Abu Sebastian et al., "Tutorial: Brain-inspired computing using phase-change memory devices," Journal of Applied Physics, vol. 124, no. 11, p. 111101, 2018.
[11] Manan Suri et al., "Phase change memory as synapse for ultra-dense neuromorphic systems: Application to complex visual pattern extraction," in IEEE International Electron Devices Meeting (IEDM), Washington, DC, USA, 2011.
[12] M. Jerry, "Ferroelectric FET analog synapse for acceleration of deep neural network training," in IEEE International Electron Devices Meeting (IEDM), San Francisco, CA, USA, 2017.
[13] Cong Xu et al., "Overcoming the challenges of crossbar resistive memory architectures," in IEEE 21st International Symposium on High Performance Computer Architecture (HPCA), Burlingame, CA, USA, 2015.
[14] Gong, N. et al., "Signal and noise extraction from analog memory elements for neuromorphic computing," Nature Communications, vol. 9, no. 2102, 2018.
[15] Truong Ngoc et al., "New Memristor-Based Crossbar Array Architecture with 50-% Area Reduction and 48-% Power Saving for Matrix-Vector Multiplication of Analog Neuromorphic Computing," JSTS:Journal of Semiconductor Technology and Science, vol. 14, no. 3, Jun 2014.
[16] Zihan Xu et al., "Parallel Programming of Resistive Cross-point Array for Synaptic Plasticity," Procedia Computer Science, vol. 41, pp. 126-133, 2014.
[17] Sung Hyun Jo et al., "Nanoscale Memristor Device as Synapse in Neuromorphic Systems," Nano Letters, vol. 10, no. 4, pp. 1297-1301, 2010.
[18] S. R. Nandakumar et al., "Mixed-precision architecture based on computational," in IEEE International Symposium on Circuits and Systems (ISCAS), Florence, Italy, 2018.
[19] M. Courbariaux et al., "BinaryConnect: Training deep neural," Neural Information Processing Systems (NIPS), pp. 3123-3131, 2015.
[20] Itay Hubara et al., "Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations," Journal of Machine Learning Research, vol. 18, Sep 2016.
[21] Irem Boybat, "Stochastic weight updates in phase-change memory-based synapses and their influence on artificial neural networks," in Conference on Ph.D. Research in Microelectronics and Electronics (PRIME), Giardini Naxos, Italy, 2017.
[22] W. Hung, "A deep learning simulation platform for non-volatile memory-based analog neuromorphic circuits," National Cheng Kung University (NCKU), 2019.
[23] Yao, P. et al., "Fully hardware-implemented memristor convolutional neural network," Nature, vol. 577, pp. 641-646, 2020.
[24] Pai-Yu Chen et al., "NeuroSim: A Circuit-Level Macro Model for Benchmarking Neuro-Inspired Architectures in Online Learning," IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 37, no. 12, pp. 3067-3080, 2018.
[25] R. Gonzalez et al., "Energy dissipation in general purpose microprocessors," IEEE Journal of Solid-State Circuits, vol. 331, no. 9, pp. 1277-1284, 1996.
[26] Jiale Liang et al., "Effect of Wordline/Bitline Scaling on the Performance, Energy Consumption, and Reliability of Cross-Point Memory Array," ACM Journal on Emerging Technologies in Computing Systems, vol. 9, no. 1, 2013.
[27] T. Sakurai et al., "Simple formulas for two- and three-dimensional capacitances," IEEE Transactions on Electron Devices , vol. 30, no. 2, pp. 183-185, 1983.
[28] Hoang-Hiep Le et al., "Ultralow Power Neuromorphic Accelerator for Deep Learning Using Ni/HfO2/TiN Resistive Random Access Memory," in 4th IEEE Electron Devices Technology & Manufacturing Conference (EDTM), Penang, Malaysia, Malaysia, 2020.
[29] Irem Boybat et al., "Neuromorphic computing with multi-memristive," Nature communications, vol. 9, no. 2514, 2018.
[30] Darsen D. Lu et al., "A computationally efficient compact model for ferroelectric FETs for the simulation of online training of neural networks," Semiconductor Science and Technology, 2020.
[31] Salman Khan et al., A Guide to Convolutional Neural Networks for Computer Vision, Morgan & Claypool, 2018.
[32] Deepak Kadetotad et al., "Parallel Architecture With Resistive Crosspoint Array for Dictionary Learning Acceleration," IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 5, no. 2, pp. 194-204, 2015.
[33] Nathan Serafino et al., "Review of nanoscale memristor devices as synapses in neuromorphic systems," International Midwest Symposium on Circuits and Systems (MWSCAS), 4-7 Aug 2013.
[34] Yi Wu et al., "Resistive switching random access memory — Materials, device, interconnects, and scaling considerations," in IEEE International Integrated Reliability Workshop Final Report, South Lake Tahoe, CA, USA, 2012.
[35] Pai-Yu Chen et al., "Reliability perspective of resistive synaptic devices on the neuromorphic system performance," in IEEE International Reliability Physics Symposium (IRPS), Burlingame, CA, USA, 2018.
論文全文使用權限
  • 同意授權校內瀏覽/列印電子全文服務,於2025-07-31起公開。
  • 同意授權校外瀏覽/列印電子全文服務,於2025-07-31起公開。


  • 如您有疑問,請聯絡圖書館
    聯絡電話:(06)2757575#65773
    聯絡E-mail:etds@email.ncku.edu.tw