It covers technological trends, industry challenges, and advances made by key players, while forecasting market trends in the semiconductor packaging sector.
Challenges of large language models (LLM):
The emergence of large linguistic models (LLMs) marks a milestone in AI, revolutionizing natural language processing (NLP) and related fields. Models like OpenAI's GPT series demonstrate high accuracy in understanding, generating, and translating human language. While they have diverse applications, they also present challenges.
Large-scale modeling (LLM) systems have experienced exponential growth in model size, increasing from 340 million parameters in BERT-L in 2019 to 1.76 trillion parameters in the upcoming GPT-4. This expansion translates into high computational complexity, as GPT-4 requires over 10¹⁰ petaFLOPS. Compared to other neural networks, LLMs demand significantly more computational resources; for example, training GPT-3 requires 3,841 GPU hours compared to just 11 hours for a ResNet-60 (a convolutional neural network used for image classification). LLMs have low operational intensity, meaning they rely heavily on matrix-vector operations. This necessitates moving more data to perform the same number of arithmetic operations. Therefore, memory bandwidth is critical to their performance.
Ways to improve system bandwidth:
In recent years, a gap has emerged between the rapid increase in processor computing density and the slower increase in memory bandwidth. This "memory wall" problem causes processors to frequently wait for data, leading to underutilization and posing a significant challenge to future performance improvements.
NVIDIA and AMD have introduced NVLink and Infinity Fabric technologies, respectively, to improve interconnectivity between the CPU and GPU, which is crucial for improving bandwidth between logic components.
The incorporation of next-generation high-bandwidth memory (HBM) architectures, consisting of multiple vertically stacked DRAM dies connected by silicon vias (TSVs), offers a substantial increase in bandwidth between logic and memory. While previously limited to GPUs, HBM is now being integrated into CPUs, bridging the bandwidth gap between them and traditional DRAM. AMD's Instinct MI300 exemplifies this trend by integrating advanced HBM with both the CPU and GPU in a single package. 2.5D semiconductor packaging technologies, such as TSMC's Chip on Wafer on Substrate (CoWoS), play a crucial role in increasing the number of I/O (input/output) points while reducing the interconnect length between logic and memory components, thus improving performance and reducing latency. However, new HPC workloads, particularly those related to AI training, demand even greater memory bandwidth (more than 45x) due to frequent memory accesses. Increasing I/O speed can improve HBM bandwidth, but is limited by higher power consumption.
To address this problem, new processor designs have focused on increasing the capacity of on-chip SRAM, as it is located close to the processing chips, enabling higher bandwidth with denser interconnects and lower latency. Early implementations used 3D hybrid junction packaging technology to bond an SRAM die onto a logic die with a 9 μm pitch, tripling the SRAM capacity. To further improve bandwidth, the hybrid junction pitch was reduced from 9 μm to 0.4 μm, increasing the bandwidth density more than 300-fold.
Co-encapsulated optics is another approach that has gained significant momentum in recent years. Optical communication offers several advantages over traditional electrical signal transmission. It has lower transmission losses, which reduces signal degradation over distance. Furthermore, optical communication is less susceptible to crosstalk, which occurs when signals interfere with each other, causing data errors. In addition, optical signals can achieve higher bandwidth compared to their electrical counterparts, making them ideal for data-intensive applications.
In summary, for the next generation of AI computing systems, high bandwidth between logic and logic, and between logic and memory, are crucial development trends. Advanced semiconductor packaging technologies, such as 2.5D and 3D hybrid junctions, along with emerging solutions like optical communication, are fundamental to enabling this growth and overcoming computational challenges.
Author: Dr. Yu-Han Chang, Senior Technology Analyst at IDTechEx
