3.6. IceNet

IceNet is a library of Chisel designs related to networking. The main component of IceNet is IceNIC, a network interface controller that is used primarily in FireSim for multi-node networked simulation. A diagram of IceNet’s microarchitecture is shown below.

../_images/nic-design.png

There are four basic parts of the NIC: the Controller, which takes requests from and sends responses to the CPU; the Send Path, which reads data from memory and sends it out to the network; the Receive Path, which receives data from the network and writes it to memory; and, optionally, the Pause Handler, which generates Ethernet pause frames for the purpose of flow control.

3.6.1. Controller

The controller exposes a set of MMIO registers to the CPU. The device driver writes to registers to request that packets be sent or to provide memory locations to write received data to. Upon the completion of a send request or packet receive, the controller sends an interrupt to the CPU, which clears the completion by reading from another register.

3.6.2. Send Path

The send path begins at the reader, which takes requests from the controller and reads the data from memory.

Since TileLink responses can come back out-of-order, we use a reservation queue to reorder responses so that the packet data can be sent out in the proper order.

The packet data then goes to an arbiter, which can arbitrate access to the outbound network interface between the NIC and one or more “tap in” interfaces, which come from other hardware modules that may want to send Ethernet packets. By default, there are no tap in interfaces, so the arbiter simply passes the output of the reservation buffer through.

3.6.3. Receive Path

The receive path begins with the packet buffer, which buffers data coming in from the network. If there is insufficient space in the buffer, it will drop data at packet granularity to ensure that the NIC does not deliver incomplete packets.

From the packet buffer, the data can optionally go to a network tap, which examines the Ethernet header and select packets to be redirected from the NIC to external modules through one or more “tap out” interfaces. By default, there are no tap out interfaces, so the data will instead go directly to the writer, which writes the data to memory and then sends a completion to the controller.

3.6.4. Pause Handler

IceNIC can be configured to have pause handler, which sits between the send and receive paths and the Ethernet interface. This module tracks the occupancy of the receive packet buffer. If it sees the buffer filling up, it will send an Ethernet pause frame out to the network to block further packets from being sent. If the NIC receives an Ethernet pause frame, the pause handler will block sending from the NIC.

3.6.5. Linux Driver

The default Linux configuration provided by firesim-software contains an IceNet driver. If you launch a FireSim image that has IceNIC on it, the driver will automatically detect the device, and you will be able to use the full Linux networking stack in userspace.

3.6.6. Configuration

To add IceNIC to your design, add HasPeripheryIceNIC to your lazy module and HasPeripheryIceNICModuleImp to the module implementation. If you are confused about the distinction between lazy module and module implementation, refer to Cake Pattern.

Then add the WithIceNIC config mixin to your configuration. This will define NICKey, which IceNIC uses to determine its parameters. The mixin takes two arguments. The inBufFlits argument is the number of 64-bit flits that the input packet buffer can hold and the usePauser argument determines whether or not the NIC will have a pause handler.