10.2. Running a Design on VCU118

10.2.1. Basic VCU118 Design

The default Xilinx VCU118 harness is setup to have UART, a SPI SDCard, and DDR backing memory. This allows it to run RISC-V Linux from an SDCard while piping the terminal over UART to the host machine (the machine connected to the VCU118). To extend this design, you can create your own Chipyard configuration and add the WithVCU118Tweaks located in fpga/src/main/scala/vcu118/Configs.scala. Adding this config fragment will enable and connect the UART, SPI SDCard, and DDR backing memory to your Chipyard design/config.

class WithVCU118Tweaks extends Config(
  // clocking
  new chipyard.harness.WithAllClocksFromHarnessClockInstantiator ++
  new chipyard.clocking.WithPassthroughClockGenerator ++
  new chipyard.config.WithMemoryBusFrequency(100) ++
  new chipyard.config.WithSystemBusFrequency(100) ++
  new chipyard.config.WithControlBusFrequency(100) ++
  new chipyard.config.WithPeripheryBusFrequency(100) ++
  new chipyard.config.WithControlBusFrequency(100) ++
  new WithFPGAFrequency(100) ++ // default 100MHz freq
  // harness binders
  new WithUART ++
  new WithSPISDCard ++
  new WithDDRMem ++
  // other configuration
  new WithDefaultPeripherals ++
  new chipyard.config.WithTLBackingMemory ++ // use TL backing memory
  new WithSystemModifications ++ // setup busses, use sdboot bootrom, setup ext. mem. size
  new chipyard.config.WithNoDebug ++ // remove debug module
  new freechips.rocketchip.subsystem.WithoutTLMonitors ++
  new freechips.rocketchip.subsystem.WithNMemoryChannels(1)
)

class RocketVCU118Config extends Config(
  new WithVCU118Tweaks ++
  new chipyard.RocketConfig
)

10.2.2. Brief Implementation Description + More Complicated Designs

The basis for a VCU118 design revolves around creating a special test harness to connect the external IOs to your Chipyard design. This is done with the VCU118TestHarness in the basic default VCU118 FPGA target. The VCU118TestHarness (located in fpga/src/main/scala/vcu118/TestHarness.scala) uses Overlays that connect to the VCU118 external IOs. Generally, the Overlays take an IO from the ChipTop (labeled as topDesign in the file) when “placed” and connect it to the external IO and generate necessary Vivado collateral. For example, the following shows a UART Overlay being “placed” into the design with a IO input called io_uart_bb.

  // 1st UART goes to the VCU118 dedicated UART

  val io_uart_bb = BundleBridgeSource(() => (new UARTPortIO(dp(PeripheryUARTKey).head)))
  dp(UARTOverlayKey).head.place(UARTDesignInput(io_uart_bb))

Here the UARTOverlayKey is referenced and used to “place” the necessary connections (and collateral) to connect to the UART. The UARTDesignInput is used to pass in the UART IO from the ChipTop/topDesign to the Overlay. Note that the BundleBridgeSource can be viewed as a glorified wire (that is defined in the LazyModule scope). This pattern is similar for all other Overlays in the test harness. They must be “placed” and given a set of inputs (IOs, parameters). The main exception to this pattern is the Overlay used to generate the clock(s) for the FPGA.

  // place all clocks in the shell
  require(dp(ClockInputOverlayKey).size >= 1)
  val sysClkNode = dp(ClockInputOverlayKey)(0).place(ClockInputDesignInput()).overlayOutput.node

  /*** Connect/Generate clocks ***/

  // connect to the PLL that will generate multiple clocks
  val harnessSysPLL = dp(PLLFactoryKey)()
  harnessSysPLL := sysClkNode

  // create and connect to the dutClock
  val dutFreqMHz = (dp(SystemBusKey).dtsFrequency.get / (1000 * 1000)).toInt
  val dutClock = ClockSinkNode(freqMHz = dutFreqMHz)
  println(s"VCU118 FPGA Base Clock Freq: ${dutFreqMHz} MHz")
  val dutWrangler = LazyModule(new ResetWrangler)
  val dutGroup = ClockGroup()
  dutClock := dutWrangler.node := dutGroup := harnessSysPLL

Without going into too much detail, the clocks overlay is placed in the harness and a PLL node (harnessSysPLL) generates the necessary clocks specified by ClockSinkNodes. For ease of use, you can change the FPGAFrequencyKey to change the default clock frequency of the FPGA design.

After the harness is created, the BundleBridgeSource’s must be connected to the ChipTop IOs. This is done with harness binders and io binders (see fpga/src/main/scala/vcu118/HarnessBinders.scala and fpga/src/main/scala/vcu118/IOBinders.scala). For more information on harness binders and io binders, refer to IOBinders and HarnessBinders.

10.2.3. (Legacy) Introduction to the Legacy Bringup Design

Warning

The bringup VCU118 design described here is designed for old versions of Chipyard SoCs, pre-1.9.1. The key difference is that these designs rely on a clock generated on-chip to synchronize the slow serialized-TileLink interface. After Chipyard 1.9.1, the FPGA host is expected to pass the clock to the chip, instead of the other way around. A new bringup solution will be developed for post-1.9.1 Chipyard designs.

An example of a more complicated design used for Chipyard test chips can be viewed in fpga/src/main/scala/vcu118/bringup/. This example extends the default test harness and creates new Overlays to connect to a DUT (connected to the FMC port). Extensions include another UART (connected over FMC), I2C (connected over FMC), miscellaneous GPIOS (can be connected to anything), and a TSI Host Widget. The TSI Host Widget is used to interact with the DUT from the prototype over a SerDes link (sometimes called the Low BandWidth InterFace - LBWIF) and provide access to a channel of the FPGA’s DRAM.

Note

Remember that since whenever a new test harness is created (or the config changes, or the config packages changes, or…), you need to modify the make invocation. For example, make SUB_PROJECT=vcu118 CONFIG=MyNewVCU118Config CONFIG_PACKAGE=this.is.my.scala.package bitstream. See Generating a Bitstream for information on the various make variables.

10.2.4. Running Linux on VCU118 Designs

As mentioned above, the default VCU118 harness is setup with a UART and a SPI SDCard. These are utilized to both interact with the DUT (with the UART) and load in Linux (with the SDCard). The following steps describe how to build and run buildroot Linux on the prototype platform.

10.2.4.1. Building Linux with FireMarshal

Since the prototype currently does not have a block device setup for it, we build Linux with the rootfs built into the binary (otherwise known as “initramfs” or “nodisk” version of Linux). To make building this type of Linux binary easy, we will use the FireMarshal platform (see FireMarshal for more information).

  1. Setup FireMarshal (see FireMarshal on the initial setup).

  2. By default, FireMarshal is setup to work with FireSim. Instead, we want to target the prototype platform. This is done by switching the FireMarshal “board” from “firechip” to “prototype” using marshal-config.yaml:

# this assumes you do not have a `marshal-config.yaml` file already setup
echo "board-dir : 'boards/prototype'" > $PATH_TO_FIREMARSHAL/marshal-config.yaml

Note

Refer to the FireMarshal docs on more ways to set the board differently through environment variables and more.

  1. Next, build the workload (a.k.a buildroot Linux) in FireMarshal with the nodisk option flag. For the rest of these steps, we will assume you are using the base br-base.json workload. This workload has basic support for GPIO and SPI drivers (in addition to the default UART driver) but you can build off it in different workloads (refer to FireMarshal docs on workload inheritance).

./marshal -v -d build br-base.json # here the -d indicates --nodisk or initramfs

Note

Using the “board” FireMarshal functionality allows any child workload depending on the br-base.json workload specification to target a “prototype” platform rather than FireChip platform. Thus, you can re-use existing workloads that depend on br-base.json on the prototype platform by just changing the “board”!

  1. The last step to generate the proper binary is to flatten it. This is done by using FireMarshal’s install feature which will produce a *-flat binary in the $PATH_TO_FIREMARSHAL/images directory (in our case br-base-bin-nodisk-flat) from the previously built Linux binary (br-base-bin-nodisk).

./marshal -v -d install -t prototype br-base.json

10.2.4.2. Setting up the SDCard

These instructions assume that you have a spare uSDCard that can be loaded with Linux and other files using two partitions. The 1st partition will be used to store the Linux binary (created with FireMarshal or other means) while the 2nd partition will store a file system that can be accessed from the DUT. Additionally, these instructions assume you are using Linux with sudo privileges and gdisk, but you can follow a similar set of steps on Mac (using gpt or another similar program).

  1. Wipe the GPT on the card using gdisk. Use the z command from the expert menu (opened with ‘x’, closed with ‘m’) to zap everything. For rest of these instructions, we assume the SDCard path is /dev/sdc (replace this with the path to your SDCard).

sudo gdisk /dev/sdc
  1. Create the new GPT with o. Click yes on all the prompts.

  2. The VCU118 bootrom assumes that the Linux binary to load into memory will be located on sector 34 of the SDCard. Change the default partition alignment to 1 so you can write to sector 34. Do this with the l command from the expert menu (opened with ‘x’, closed with ‘m’).

  3. Create a 512MiB partition to store the Linux binary (this can be smaller but it must be larger than the size of the Linux binary). Use n, partion number 1 and select sector 34, with size +1048576 (corresponding to 512MiB). For the type, search for the apfs type and use the hex number given.

  4. Create a second partition to store any other files with the rest of the SDCard. Use n and use the defaults for partition number, starting sector and overall size (expand the 2nd partition to the rest of the SDCard space). For the type, search for the hfs and use the hex number given.

  5. Write the changes using w.

  6. Setup the filesystem on the 2nd partition. Note that the /dev/sdc2 points to the 2nd partition. Use the following command:

sudo mkfs.hfs -v "PrototypeData" /dev/sdc2

10.2.4.3. Transfer and Run Linux from the SDCard

After you have a Linux boot binary and the SDCard is setup properly (1st partition at sector 34), you can transfer the binary to the 1st SDCard partition. In this example, we generated a br-base-bin-nodisk-flat from FireMarshal and we will load it using dd. Note that sdc1 points to the 1st partition (remember to change the sdc to your own SDCard path).

sudo dd if=$PATH_TO_FIREMARSHAL/br-base-bin-nodisk-flat of=/dev/sdc1

If you want to add files to the 2nd partition, you can also do this now.

After loading the SDCard with Linux and potentially other files, you can program the FPGA and plug in the SDCard. To interact with Linux via the UART console, you can connect to the serial port (in this case called ttyUSB1) using something like screen:

screen -S FPGA_UART_CONSOLE /dev/ttyUSB1 115200

Once connected, you should see the binary being loaded as well as Linux output (in some cases you might need to reset the DUT). Sign in as ‘root’ with password ‘fpga’.