Sorry, you need to enable JavaScript to visit this website.

PCIe root complex Error with PCIe to PCI adapter (7z045)

Unsolved
7 posts / 0 new
Hugo.KIEFFER's picture
Hugo.KIEFFER
Junior(0)
PCIe root complex Error with PCIe to PCI adapter (7z045)

Hi,

I try to use a PCIe to PCI adapter on Zynq Minit-itx board.

Link to PCIe to PCI adapter: http://www.startech.com/Cards-Adapters/Slot-Extension/PCI-Express-to-PCI...

I use the PCIe root complex design provided at http://zedboard.org/content/zynq-mini-itx-7z045-pcie-design-vivado-20144
I use version 2014.4 for petalinux tools, Vivado and SDK.

After generating bitstream, FSBL, u-boot and kernel I test PCIe with Ethernet device.

PCIe root complex work, I connect an ethernet PCIe device to test it. Below, a log of PCI initialization when the PCIe ethernet is connected:
xaxi_pcie_init_port: LINK IS UP
AXI PCIe Root Port Probe Successful
xaxi_pcie_set_bridge_resource:pci_space: 0x02000000 pci_addr:0x0000000060000000 size: 0x0000000010000000
xaxi_pcie_set_bridge_resource:Setting resource in Memory Space
PCI host bridge /amba_pl/pciex@50000000 (primary) ranges:
MEM 0x0000000060000000..0x000000006fffffff -> 0x0000000060000000
PCI: PHB MEM resource 0 = 0000000060000000-000000006fffffff [200]
PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fffffff]
pci_bus 0000:00: root bus resource [io 0x1000-0xffff]
pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]
PCI: bus0: Fast back to back transfers disabled
PCI: bus1: Fast back to back transfers disabled
pci 0000:00:00.0: BAR 0: no space for [mem size 0x40000000]
pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x40000000]
pci 0000:00:00.0: BAR 14: assigned [mem 0x60000000-0x600fffff]
pci 0000:00:00.0: BAR 13: assigned [io 0x1000-0x1fff]
pci 0000:01:00.0: BAR 4: assigned [mem 0x60000000-0x60003fff 64bit pref]
pci 0000:01:00.0: BAR 2: assigned [mem 0x60004000-0x60004fff 64bit]
pci 0000:01:00.0: BAR 0: assigned [io 0x1000-0x10ff]
pci 0000:00:00.0: PCI bridge to [bus 01]
pci 0000:00:00.0: bridge window [io 0x1000-0x1fff]
pci 0000:00:00.0: bridge window [mem 0x60000000-0x600fffff]

The kernel boot and I can view the device in a lspci.

When I connect the PCIe to PCI adapter the kernel lock during PCI initialization. Below, a log of PCI initialization when PCIe to PCI adapter is connected:
xaxi_pcie_init_port: LINK IS UP
AXI PCIe Root Port Probe Successful
xaxi_pcie_set_bridge_resource:pci_space: 0x02000000 pci_addr:0x0000000060000000 size: 0x0000000010000000
xaxi_pcie_set_bridge_resource:Setting resource in Memory Space
PCI host bridge /amba_pl/pciex@50000000 (primary) ranges:
MEM 0x0000000060000000..0x000000006fffffff -> 0x0000000060000000
PCI: PHB MEM resource 0 = 0000000060000000-000000006fffffff [200]
PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fffffff]
pci_bus 0000:00: root bus resource [io 0x1000-0xffff]
pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]
pci_bus 0000:00: scanning bus
pci 0000:00:00.0: [10ee:0706] type 01 class 0x060400
pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x3fffffff]
pci 0000:00:00.0: calling pci_fixup_ide_bases+0x0/0x3c
pci_bus 0000:00: fixups for bus
PCI: bus0: Fast back to back transfers disabled
pci 0000:00:00.0: scanning [bus 01-07] behind bridge, pass 0
pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 1
pci_bus 0000:01: scanning bus
pci 0000:01:00.0: [12d8:e111] type 01 class 0x060400
pci 0000:01:00.0: calling pci_fixup_ide_bases+0x0/0x3c
pci_bus 0000:01: fixups for bus
PCI: bus1: Fast back to back transfers disabled
pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 0
pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 1
pci_bus 0000:02: scanning bus

After this line the kernel is locked, no other information is printed and no boot.

The PCIe to PCI adapter is OS independant (No additional drivers or software required).
I test it on a standard PC (with BIOS), it worked.

I don't understand what is wrong in this configuration.
Do I need to update my device-tree with information about the PCIe to PCI adapter ?
Or hardware compatibility issue between PCIe to PCI adapter and PCIe root complex Xilinx design ?

Best regards,
Hugo

npoureh's picture
npoureh
Moderator(3)
Just to be sure, are you

Just to be sure, are you using the Avnet reference design or are you creating your own HW and SW projects? If you are using your own, would you please try the Avnet ref design.

Hugo.KIEFFER's picture
Hugo.KIEFFER
Junior(0)
Yes I use the Avnet design

Yes I use the Avnet design (http://zedboard.org/content/zynq-mini-itx-7z045-pcie-design-vivado-20144).

First I try with ready_to_test example (zynq_mini_itx_7z045_pcie_design_v2014_4.zip/ready_to_test/sd_image_nic) and I have the same behavior a PCIe ethernet card is detected but if I connect PCIe to PCI adapter the kernel lock during PCI initialization.

After this, I try to regenrate Bitsrteam FSBL and u-boot from the hardware design. The only modification I made on the design is to make PTP_ETHERNET port external (I need Ethernet connexion). I ran synhtesis, implementation, bitstream generation and hardware export. With the .hdf I create a petalinux project and I generate FSBL u-boot and kernel with petalinux tools. I have encoutered some issues with the auto generated device-tree, the Ethernet and PCIe sections was wrong so I decompile the device-tree blob provided in ready_to_test example to get PCIe and Ethernet sections. I copy those information into the petalinux device-tree source before building (petalinux-build).

With this build I observe the same behavior as the ready_to_test example (a PCIe ethernet card is detected but when I connect adapter the kernel lock during PCI init).

npoureh's picture
npoureh
Moderator(3)
Is MSI enabled on the PCIe-to

Is MSI enabled on the PCIe-to-PCI adapter?

Hugo.KIEFFER's picture
Hugo.KIEFFER
Junior(0)
Yes, I think.

Yes, I think.

Below, a lspci -vvv of adapter when it is connected in standard desktop computer:
02:00.0 PCI bridge: Pericom Semiconductor Device e111 (rev 02) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=02, secondary=03, subordinate=03, sec-latency=0
I/O behind bridge: 00003000-00003fff
Memory behind bridge: e1000000-e10fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [80] PCI-X bridge device
Secondary Status: 64bit- 133MHz- SCD- USC- SCO- SRD- Freq=conv
Status: Dev=02:00.0 64bit- 133MHz- SCD- USC- SCO- SRD-
Upstream: Capacity=16 CommitmentLimit=16
Downstream: Capacity=16 CommitmentLimit=16
Capabilities: [a8] Subsystem: Device 0000:0000
Capabilities: [b0] Express (v1) PCI/PCI-X Bridge, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE- FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr+ FatalErr- UnsuppReq+ AuxPwr+ TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
Capabilities: [d8] Vital Product Data
Unknown small resource type 00, will not decode more.
Capabilities: [f0] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff

Thank for your help.

Best regards,
Hugo

npoureh's picture
npoureh
Moderator(3)
Sorry it took so long to get

Sorry it took so long to get to the root cause of the issue you are having with the PCIe Root Complex design, but here is a brief description of what is causing this behavior.

The root cause of this problem is related to the PCIe wrapper (fabric logic wrapped around the PCIe Hard IP in Zynq PL to implement the PCIe Root Complex IP). The AXI Memory Mapped bridge in the PCIe wrapper can only handle memory transactions to the PCIe Devices. It can not handle any sort of IO transactions, which would be required to communicate with the PCIe-to-PCI bridge (a legacy device).

In order to support the PCIe-PCI adapter, the PCIe wrapper must be modified to access the adapter via IO transactions. Unfortunately, Xilinx has no plans on implementing this feature and if needed it must be implemented by the user. If you are willing to implement this feature yourself, I can put you in contact with someone at Xilinx who can provide some limited assistance.

Hugo.KIEFFER's picture
Hugo.KIEFFER
Junior(0)
Thank for your feedback.

Thank for your feedback.

I have a working configuration since last evening. I meant to post here this morning.

If I remove the Multi-core scheduler from kernel configuration the adapter works. (I will post an issue at linux-xlnx github page).

To sumerize my kernel configuration:
CONFIG_SCHED_MC must be unset (Kernel Features => Multi-core scheduler support)
CONFIG_XILINX_AXIPCIE=y (System Type => Xilinx Zynq ARM Cortex A9 Platform => Xilinx AXI PCIe host bridge support)
CONFIG_PCI=y (Bus support => PCI support)
CONFIG_PCI_MSI=y (Buss support => Message Signaled Interrupts (MSI and MSI-X))
CONFIG_PCI_REALLOC_ENABLE_AUTO=y (Buss support => Enable PCI resource re-allocation detection)
CONFIG_PCIEPORTBUS=y (Buss support => PCI Express Port Bus support)
CONFIG_PCIEAER=y (Buss support => Root Port Advanced Error Reporting support)

I'm not sure that MSI, REALLOC, PCIEPORTBUS and PCIEAER are necessary, I need to test this.

I change two configurations in Vivado hardware design, I set MSI Vectors Requested to 1 and COMP TIMEOUT to 50µs, I'm not sure that is necessary.