dReDBox: The edge of an interwoven design method.
We swear by interwoven design to help you get in control of your electronics. Experts on schematics design, analysis and PCB layout continuously collaborate to actively engineer out all issues.
Does our interwoven design method pay off? We used it to successfully design dReDBox. dReDBox is an example of what we could do for you, to give you a competitive edge.
dReDBox is an Horizon2020 EU project that aspires to use datacenter resources more efficiently to save floor space and energy. It is based on a new hardware concept that uses pooled disaggregated compute, memory and accelerator resources instead of fixed server configurations.
dReDBox contains hardware building blocks called dBRICK. Sixteen dBRICKs are stacked on the dTRAY, a connecting board with a high speed electrical-network that supports extreme low latency transactions from one dBRICK memory to another. dTRAY has three outgoing networks: an optical network, PCIe and Ethernet.
To make dReDBox easy to use for programmers, the platform supports Virtual Machine (VM) which configures resources on-the-fly to perfectly match user requirements.
dReDBox challenges for Sintecs
A disaggregate, high-performance computing platform is a complex, high end system. We built it from scratch, with a completely new architecture, a different physical distribution of processors and high speed memory, and high speed board-to-board interconnects. We faced three important challenges when designing the dReDBox hardware.
# 1. Give the memory speed you need
The dBRICKs rely on a stable DDR4 memory for fast, reliable operation. For a design like this, the timing margins are so tight that the exact physical configuration has a significant impact on maximum memory speed. Timing analysis showed that it would be fundamentally impossible to use the memory at its maximum speed of 2400 MT/s if we kept to the PCB layout guidelines of the Zynq UltraScale+ MPSoC. To get out of this deadlock, we asked Xilinx for more in-depth information. They collaborated and provided us with timing details that allowed us to optimize the PCB layout.
Hardware verification on the first run of PCBs confirmed that we can indeed use the DDR4 at its maximum speed, in a broad temperature range, without any timing violations nor EMI problems.
#2. Keep it cool
To avoid thermal issues, we performed thermal analysis and verification early in the design phase. We analysed the individual dBRICKs and the complete dReDBox system, which dissipates an estimated 750 W. We identified hot spots, looked for zones of flow stagnation and identified overstressed components. The results of the thermal analysis serve as an input for mechanical and electrical design, reliability predictions and stress analyses.
Thermal modelling allowed us to optimize for airflow and temperature distribution, size and number of the fans. It gave insight in the impact of the configuration changes and failures of fans.
# 3 Keep it free from power dips and noise
To assure power integrity and absence of noise on the delivered power, we ran a signal and power integrity co-simulation. We determined the optimal decoupling, experimented with PCB stack-up, and examined many aspects of PCB technology such as used materials, dimensions, tracks, and vias. We selected a solution which meets the power requirements yet avoids using exotic materials and processes that drive up costs.
To mitigate the influence of fast switching high-speed interfaces on the dBRICKs and to avoid problems in its fifteen power rails, we adapted the layout of the planes and the decoupling capacitors so that they quickly distribute the large currents resulting from fast signal switching.
No budget nor time for respins
A PCB design iteration for a high-end system like this is around fifty thousand euros per round. The only way we could finish the dReDBox design in time and within budget, is by making the design right the first time. And we managed that!
How did we get a complex design like dReDBox up and running on the first round of PCBs? Whenever we made a design choice, we immediately analyzed it and acted on the analysis results, over and over as we were designing. We iterated, analyzed and optimized the hardware schematics and PCB layout in over 50 cycles. Our interwoven design method pays off.