PCI Express advanced topics: Part 2

PCIe Enumeration 

Before we go to the details of enumeration, lets use the following nomenclature. 

PCIe port : physical group of transmitter(s)/receiver(s) on same chip

PCIe lane : one pair of differential transmitter and receiver

PCIe link : collection of lane(s). 

Upstream device : is the root complex

Downstream device : end point that implements atleast one function

Upstream port : resides on the downstream device facing the upstream device

Downstream port : resides on the upstream device facing the downstream device

Figure -1 denotes these nomenclature.

Figure -1

Now, let's take an example of single lane (lane 0) - one root complex connected to one PCIe device that implements one function for illustrating the enumeration process.

Enumeration is the process where the root complex driver discovers the PCI(e) bus topology by traversing through the hierarchy from root complex (which is at the top of the hierarchy). 

Following are the per-requisites that needs to be satisfied

a) Root complex is on the Bus 0 and configuration space access of the root complex is accessible using embedded CPU (via API or known mechanism)
b) The EP device that implements a function must have a Vendor ID that is not all '0xF's
c) link training has completed and link has been established on both the sides of the link and the Data link layer is in DL_ACTIVE state.

Steps below outlines the simplified enumeration procedure.

1) RC driver steps through the hierarchy to find all the connected downstream devices by looping the bus number. For E.g. read the vendor ID of the downstream device for bus 1 and device 0, function 0 

using the API to access the configuration space with the address set to 

CFG_ADDR = base_addr | 1<<20.

If there is valid function implemented on this downstream device, it must return a valid vendor ID and it is first step to discover this device

2) If the device is found, now the device needs to be assigned the memory window so that it gets allocated the memory space on the system memory map. This is done by scanning the BAR's and check for the BAR size and do the allocation accordingly. Downstream device BAR size can be found by writing all one's to the respective BAR register and checking for the bits that are zero's. Root complex driver needs to only set here the base address of the requested memory size.

e.g. EP device BAR0 after writing all '1's returns with  0xFF0000000 implies that the memory size requested by this EP device is 16MB and it is 32 bit addressable. Therefore the RC driver can assign any based address in the 4GB space to allocate  the requested 16MB memory 

e.g. BAR0 = 0x1C000000 as the base address

3) After the base address is assigned EP device needs to be configured to enable as bus master so that it can start sending memory TLP requests. 

set the command register to enable Bus master, memory enable and IO enable.

CMD_REGISTER | =  1<<2 | 1<<1 | 1<<0;

the above steps are basic functions required to for any PCIe based applications. End point now has to manage this allocated memory from the system as a memory resource for the their application specific implementation.