10.2. Running a Design on VCU118
10.2.1. Basic VCU118 Design
The default Xilinx VCU118 harness is setup to have UART, a SPI SDCard, and DDR backing memory.
This allows it to run RISC-V Linux from an SDCard while piping the terminal over UART to the host machine (the machine connected to the VCU118).
To extend this design, you can create your own Chipyard configuration and add the WithVCU118Tweaks
located in fpga/src/main/scala/vcu118/Configs.scala
.
Adding this config fragment will enable and connect the UART, SPI SDCard, and DDR backing memory to your Chipyard design/config.
class WithVCU118Tweaks extends Config(
// clocking
new chipyard.harness.WithAllClocksFromHarnessClockInstantiator ++
new chipyard.clocking.WithPassthroughClockGenerator ++
new chipyard.config.WithMemoryBusFrequency(100) ++
new chipyard.config.WithSystemBusFrequency(100) ++
new chipyard.config.WithControlBusFrequency(100) ++
new chipyard.config.WithPeripheryBusFrequency(100) ++
new chipyard.config.WithControlBusFrequency(100) ++
new WithFPGAFrequency(100) ++ // default 100MHz freq
// harness binders
new WithUART ++
new WithSPISDCard ++
new WithDDRMem ++
new WithJTAG ++
// other configuration
new WithDefaultPeripherals ++
new chipyard.config.WithTLBackingMemory ++ // use TL backing memory
new WithSystemModifications ++ // setup busses, use sdboot bootrom, setup ext. mem. size
new freechips.rocketchip.subsystem.WithoutTLMonitors ++
new freechips.rocketchip.subsystem.WithNMemoryChannels(1)
)
class RocketVCU118Config extends Config(
new WithVCU118Tweaks ++
new chipyard.RocketConfig
)
10.2.2. Brief Implementation Description + More Complicated Designs
The basis for a VCU118 design revolves around creating a special test harness to connect the external IOs to your Chipyard design.
This is done with the VCU118TestHarness
in the basic default VCU118 FPGA target.
The VCU118TestHarness
(located in fpga/src/main/scala/vcu118/TestHarness.scala
) uses Overlays
that connect to the VCU118 external IOs.
Generally, the Overlays
take an IO from the ChipTop
(labeled as topDesign
in the file) when “placed” and connect it to the external IO and generate necessary Vivado collateral.
For example, the following shows a UART Overlay
being “placed” into the design with a IO input called io_uart_bb
.
// 1st UART goes to the VCU118 dedicated UART
val io_uart_bb = BundleBridgeSource(() => (new UARTPortIO(dp(PeripheryUARTKey).head)))
dp(UARTOverlayKey).head.place(UARTDesignInput(io_uart_bb))
Here the UARTOverlayKey
is referenced and used to “place” the necessary connections (and collateral) to connect to the UART.
The UARTDesignInput
is used to pass in the UART IO from the ChipTop
/topDesign
to the Overlay
.
Note that the BundleBridgeSource
can be viewed as a glorified wire (that is defined in the LazyModule
scope).
This pattern is similar for all other Overlays
in the test harness.
They must be “placed” and given a set of inputs (IOs, parameters).
The main exception to this pattern is the Overlay
used to generate the clock(s) for the FPGA.
// place all clocks in the shell
require(dp(ClockInputOverlayKey).size >= 1)
val sysClkNode = dp(ClockInputOverlayKey)(0).place(ClockInputDesignInput()).overlayOutput.node
/*** Connect/Generate clocks ***/
// connect to the PLL that will generate multiple clocks
val harnessSysPLL = dp(PLLFactoryKey)()
harnessSysPLL := sysClkNode
// create and connect to the dutClock
val dutFreqMHz = (dp(SystemBusKey).dtsFrequency.get / (1000 * 1000)).toInt
val dutClock = ClockSinkNode(freqMHz = dutFreqMHz)
println(s"VCU118 FPGA Base Clock Freq: ${dutFreqMHz} MHz")
val dutWrangler = LazyModule(new ResetWrangler)
val dutGroup = ClockGroup()
dutClock := dutWrangler.node := dutGroup := harnessSysPLL
Without going into too much detail, the clocks overlay is placed in the harness and a PLL node (harnessSysPLL
) generates the necessary clocks specified by ClockSinkNodes
.
For ease of use, you can change the FPGAFrequencyKey
to change the default clock frequency of the FPGA design.
After the harness is created, the BundleBridgeSource
’s must be connected to the ChipTop
IOs.
This is done with harness binders and io binders (see fpga/src/main/scala/vcu118/HarnessBinders.scala
and fpga/src/main/scala/vcu118/IOBinders.scala
).
For more information on harness binders and io binders, refer to IOBinders and HarnessBinders.
Note
Remember that since whenever a new test harness is created (or the config changes, or the config packages changes, or…), you need to modify the make invocation.
For example, make SUB_PROJECT=vcu118 CONFIG=MyNewVCU118Config CONFIG_PACKAGE=this.is.my.scala.package bitstream
.
See Generating a Bitstream for information on the various make variables.
10.2.3. Running Linux on VCU118 Designs
As mentioned above, the default VCU118 harness is setup with a UART and a SPI SDCard. These are utilized to both interact with the DUT (with the UART) and load in Linux (with the SDCard). The following steps describe how to build and run buildroot Linux on the prototype platform.
10.2.3.1. Building Linux with FireMarshal
Since the prototype currently does not have a block device setup for it, we build Linux with the rootfs built into the binary (otherwise known as “initramfs” or “nodisk” version of Linux). To make building this type of Linux binary easy, we will use the FireMarshal platform (see FireMarshal for more information).
Setup FireMarshal (see FireMarshal on the initial setup).
By default, FireMarshal is setup to work with FireSim. Instead, we want to target the prototype platform. This is done by switching the FireMarshal “board” from “firechip” to “prototype” using
marshal-config.yaml
:
# this assumes you do not have a `marshal-config.yaml` file already setup
echo "board-dir : 'boards/prototype'" > $PATH_TO_FIREMARSHAL/marshal-config.yaml
Note
Refer to the FireMarshal docs on more ways to set the board differently through environment variables and more.
Next, build the workload (a.k.a buildroot Linux) in FireMarshal with the
nodisk
option flag. For the rest of these steps, we will assume you are using the basebr-base.json
workload. This workload has basic support for GPIO and SPI drivers (in addition to the default UART driver) but you can build off it in different workloads (refer to FireMarshal docs on workload inheritance).
./marshal -v -d build br-base.json # here the -d indicates --nodisk or initramfs
Note
Using the “board” FireMarshal functionality allows any child workload depending on the br-base.json
workload specification to target a “prototype” platform rather than FireChip platform.
Thus, you can re-use existing workloads that depend on br-base.json
on the prototype platform by just changing the “board”!
The last step to generate the proper binary is to flatten it. This is done by using FireMarshal’s
install
feature which will produce a*-flat
binary in the$PATH_TO_FIREMARSHAL/images
directory (in our casebr-base-bin-nodisk-flat
) from the previously built Linux binary (br-base-bin-nodisk
).
./marshal -v -d install -t prototype br-base.json
10.2.3.2. Setting up the SDCard
These instructions assume that you have a spare uSDCard that can be loaded with Linux and other files using two partitions.
The 1st partition will be used to store the Linux binary (created with FireMarshal or other means) while the 2nd partition will store a file system that can be accessed from the DUT.
Additionally, these instructions assume you are using Linux with sudo
privileges and gdisk
, but you can follow a similar set of steps on Mac (using gpt
or another similar program).
Wipe the GPT on the card using
gdisk
. Use the z command from the expert menu (opened with ‘x’, closed with ‘m’) to zap everything. For rest of these instructions, we assume the SDCard path is/dev/sdc
(replace this with the path to your SDCard).
sudo gdisk /dev/sdc
Create the new GPT with o. Click yes on all the prompts.
The VCU118 bootrom assumes that the Linux binary to load into memory will be located on sector 34 of the SDCard. Change the default partition alignment to 1 so you can write to sector 34. Do this with the l command from the expert menu (opened with ‘x’, closed with ‘m’).
Create a 512MiB partition to store the Linux binary (this can be smaller but it must be larger than the size of the Linux binary). Use n, partion number 1 and select sector 34, with size +1048576 (corresponding to 512MiB). For the type, search for the apfs type and use the hex number given.
Create a second partition to store any other files with the rest of the SDCard. Use n and use the defaults for partition number, starting sector and overall size (expand the 2nd partition to the rest of the SDCard space). For the type, search for the hfs and use the hex number given.
Write the changes using w.
Setup the filesystem on the 2nd partition. Note that the
/dev/sdc2
points to the 2nd partition. Use the following command:
sudo mkfs.hfs -v "PrototypeData" /dev/sdc2
10.2.3.3. Transfer and Run Linux from the SDCard
After you have a Linux boot binary and the SDCard is setup properly (1st partition at sector 34), you can transfer the binary to the 1st SDCard partition.
In this example, we generated a br-base-bin-nodisk-flat
from FireMarshal and we will load it using dd
.
Note that sdc1
points to the 1st partition (remember to change the sdc
to your own SDCard path).
sudo dd if=$PATH_TO_FIREMARSHAL/br-base-bin-nodisk-flat of=/dev/sdc1
If you want to add files to the 2nd partition, you can also do this now.
After loading the SDCard with Linux and potentially other files, you can program the FPGA and plug in the SDCard.
To interact with Linux via the UART console, you can connect to the serial port (in this case called ttyUSB1
) using something like screen
:
screen -S FPGA_UART_CONSOLE /dev/ttyUSB1 115200
Once connected, you should see the binary being loaded as well as Linux output (in some cases you might need to reset the DUT). Sign in as ‘root’ with password ‘fpga’.