Sorry, you need to enable JavaScript to visit this website.

PCIe problems when using USB 3.0 cards

Unsolved
12 posts / 0 new
Cone83's picture
Cone83
Junior(0)
PCIe problems when using USB 3.0 cards

Hi,

I'm trying to get USB 3.0 running through the PCIe slot. I have been using two different USB 3.0 cards, one with a NEC and one with a VLI chip, but have experienced (different) problems with each. My hardware design very closely matches the PCIe reference design, and I'm running a small Linux system based on the Xilinx kernel v2015.4.01 (Linux 4.0.0).

The PCI bridge is correctly detected at boot, as is the USB PCIe card. USB devices that are plugged-in are detected and some actually seem to work without problems. I tested a USB 3.0 flash drive which seemed to work fine with both cards, and gets a read speed of 55 MB/s.

Problems seem to occur when I use devices with higher data rates. I have two different USB3 cameras from different manufacture. Both work well when connected to the USB2 ports, or to a USB3 port on a Linux PC. When connected to one of the PCIe cards I experience the following problems

NEC Card
========

For the first camera, reading and writing the camera settings work but when starting the image transfer I see the following errors:

usb 2-2: usbfs: process 9843 did not claim interface 2 before use
usb 2-1: usbfs: process 9843 did not claim interface 2 before use
xhci_hcd 0000:01:00.0: WARN Successful completion on short TX: needs XHCI_TRUST_TX_LENGTH quirk?

The other camera works when configured to small image sizes, but the frame rate drastically drops when transferring large images (like 1 frame per second)

VLI Card
========

The first camera doesn't work at all (not even reading / writing settings). No error is logged, only read-errors from the user space application.

The second camera again works well with small image sizes, but when configured to larger sizes, I receive USB disconnects and re-connects:

usb 2-2: USB disconnect, device number 5
usb 2-2: new SuperSpeed USB device number 8 using xhci_hcd
usb 2-2: New USB device found, idVendor=1409, idProduct=3240
usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 2-2: Product: USB 3.0 Camera
usb 2-2: Manufacturer: Camera Manufacturer

======

So, I suppose there is a compatibility problem with the PCI-bridge and the USB3 cards when transferring high-bandwidth data. I receive some PCI errors during boot which might be related. Here are all PCI-related kernel messages:

xilinx-pcie 50000000.axi-pcie: PCIe Link is UP
PCI host bridge /amba/axi-pcie@50000000 ranges:
No bus range found for /amba/axi-pcie@50000000, using [bus 00-ff]
MEM 0x60000000..0x6fffffff -> 0x60000000
xilinx-pcie 50000000.axi-pcie: PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [bus 00-ff]
pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fffffff]
PCI: bus0: Fast back to back transfers disabled
pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
PCI: bus1: Fast back to back transfers disabled
pci 0000:00:00.0: BAR 0: no space for [mem size 0x40000000]
pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x40000000]
pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
pci 0000:01:00.0: BAR 0: assigned [mem 0x60000000-0x60001fff 64bit]
pci 0000:00:00.0: PCI bridge to [bus 01]
pci 0000:00:00.0: bridge window [mem 0x60000000-0x600fffff]
pci 0000:00:00.0: enabling device (0140 -> 0142)
pci 0000:01:00.0: enabling device (0140 -> 0142)

I couldn't really fix the memory assignment error. However, the messages seem to match the logs posted by other users. For completion, the USB related messages are as follows:

xhci_hcd 0000:01:00.0: xHCI Host Controller
xhci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 1
xhci_hcd 0000:01:00.0: hcc params 0x014051cf hci version 0x100 quirks 0x00000090
usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: xHCI Host Controller
usb usb1: Manufacturer: Linux 4.0.0-xilinx xhci-hcd
usb usb1: SerialNumber: 0000:01:00.0
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
xhci_hcd 0000:01:00.0: xHCI Host Controller
xhci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 2
usb usb2: New USB device found, idVendor=1d6b, idProduct=0003
usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb2: Product: xHCI Host Controller
usb usb2: Manufacturer: Linux 4.0.0-xilinx xhci-hcd
usb usb2: SerialNumber: 0000:01:00.0
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci-pci: EHCI PCI platform driver
usbcore: registered new interface driver usb-storage

Also, the relevant section of the device tree is as follows:

pci_express: axi-pcie@50000000 {
#address-cells = <3>;
#size-cells = <2>;
#interrupt-cells = <1>;
compatible = "xlnx,axi-pcie-host-1.00.a";
reg = < 0x50000000 0x04000000 >;
device_type = "pci";
interrupts = < 0 33 4 >;
interrupt-map-mask = <0 0 0 7>;
interrupt-map = <0 0 0 1 &pcie_intc 1>,
<0 0 0 2 &pcie_intc 2>,
<0 0 0 3 &pcie_intc 3>,
<0 0 0 4 &pcie_intc 4>;
ranges = < 0x02000000 0 0x60000000 0x60000000 0 0x10000000 >;

pcie_intc: interrupt-controller {
interrupt-controller;
#address-cells = <0>;
#interrupt-cells = <1>;
};
};

I know that the interrupt number is a bit odd, but that is correct (I have a few more things connected to IRQ_F2P). The interrupt is called, as I can see in /proc/interrupts.

I haven't yet tried any other PCIe cards, as I don't have any lying around. I'm really out of ideas on this problem by now. Any suggestions would be greatly welcome.

Thanks!

TroutChaser's picture
TroutChaser
Moderator(18)
Have you configured the PCIe

Have you configured the PCIe core for Gen 1 or Gen 2?
 
If Gen 1 then you might want to try configuring for Gen 2.
 
-Gary

Cone83's picture
Cone83
Junior(0)
Hi Gary!

Hi Gary!

I suppose with Gen2 you mean setting a link speed of 5 GT/s in the IP block? I have actually tried both settings, but didn't notice a difference either way.

Cone83's picture
Cone83
Junior(0)
I just got lspci running on

I just got lspci running on the linux system (only had the busybox version before which isn't of much help). So, in case this might shed any light on this issue, here's the output when using the NEC card:

/ # /usr/sbin/lspci -vv
00:00.0 Class 0604: Device 10ee:7022
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000-000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [60] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 1
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [128 v1] Vendor Specific Information: ID=0001 Rev=0 Len=038 <?>
Capabilities: [200 v1] Vendor Specific Information: ID=0002 Rev=0 Len=038 <?>

01:00.0 Class 0c03: Device 1912:0015 (rev 02) (prog-if 30)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 46
Region 0: Memory at 60000000 (64-bit, non-prefetchable) [size=8K]
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [90] MSI-X: Enable+ Count=8 Masked-
Vector table: BAR=0 offset=00001000
PBA: BAR=0 offset=00001080
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 unlimited
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [150 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Kernel driver in use: xhci_hcd

Kennysvk's picture
Kennysvk
Junior(0)
Hello,

Hello,
I am facing the same issue. Did you find some solution?
My project involves two USB3.0 cameras. In order to handle the bandwidth I have a USB3.0 expansion card for the PCIe slot with 4 dedicated 5Gbps channels, the card is a x4 gen2 PCIe card. The card uses 4 Renesas chips.
My Vivado design is similar to the reference design, I work on a Ubuntu 14.04 using the Vivado version 2016.1.
The kernel version I use is the one from xilinx with the branch 2016.1.01 (version 4.4). The card gets enumerated correctly and each of the 4 ports work.
When using a USB2.0 camera from the same vendor as the USB3.0 camera (same userspace drivers), I get correctly transferred data. But when using the USB3.0 camera with the higher speed, I get a transfer error. The communication seems to work, as I am able to control and setup the camera. Connecting the camera to a USB2.0 port works without any issues.
I have also a "smaller" USB3.0 expansion card, which is a x1 gen2 PCIe with 2 ports but just one Renesas chip. This results in the same behavior, as with the "larger" card.
I have also tested other branches until the latest master branch, still the version is 4.4, but the problem was the same.
One thing I found out, that with the branch 2015.2.03, version 3.19, the "small" expansion card is working and I get correct data from the USB3.0 camera. The main issue with this kernel version is, that I cannot use the "larger" card, because the kernel somehow cannot handle the multiple Renesas chips on this card and during the boot, it hangs with the error message "xilinx-pcie 50000000.axi-pcie: Slave unsupported request".
So basically in my case the transfer error on the USB3.0 occurs from the kernel version 4.0 and up. And interestingly the USB3.0 card with multiple Renesas chips is working only on kernel version 4.0 and up. Which leaves me with no working solution at this time.
Any help or suggestion will be highly appreciated.
Thanks!

Cone83's picture
Cone83
Junior(0)
Hi!

Hi!
 
Unfortunately I haven't made any progress on this issue. My project has been on hold for a while as my Mini-ITX is currently in repair due to another problem. I'm still very keen on solving the USB 3.0 issue once I get it back.
 
I think that I did also try Kernel 3.19 before without any luck. Which is the brand and model of the x1 card that you did get working? I will give this a try once I get my Mini-ITX board back.
 
I have also purchased the very same GigE network card that was used in the PCIe demo. This was actually quite hard to get as apparently that card was never sold in Europe and seems to be out of production. Anyway, the card worked without any problems on the same Kernel / Bitfile, but a GigE network card produces much less bandwidth than a USB 3.0 card. So I still think that this is some sort of a bandwidth problem. I did try to slow things down by configuring the IP to use only one lane, but that also didn't have any effect.
 
Regards,
C.

Kennysvk's picture
Kennysvk
Junior(0)
Hello,

Hello,
the only card that I got to work on the 3.19 kernel was from Transcend, PDU3. Link: http://www.transcend-info.com/Products/No-283
Unfortunately this card, since its only x1 will not be able to function correctly, when both ports need to run with 5Gbps, as the available bandwidth on a x1 gen2 card is 5Gbps. So theoretically the bandwidth on both ports will be 2.5Gbps, when running simultaneously.
The same card works also on the 4.4 kernel, but using the same settings with the camera, I do not get any frames out of it. The desing is the same, only thing that changes, is the kernel version and the drivers for the pcie bridge.
The other card I tried was from startech.com, PEXUSB3S44V. Link: https://www.startech.com/Cards-Adapters/USB-3.0/Cards/PCI-Express-USB-3-...
This card only works with the kernel 4.0 and 4.4. With the 3.19 kernel it cannot boot up. Maybe the drivers for the pcie cannot handle the 4 chips on this card. On the other card there is only one chip. The model of the chip is on both cards the same, Renesas µPD720202, with the latest firmware 2026.
I even tried to port the drivers for the pcie bridge from the 4.4 kernel into the 3.19 kernel (since the x4 card works on 4.4), but I could not do it. There are too many changes in all of the files needed by the driver between the two kernel versions. And I am not sure, that it would even work that way.
I personally also think, that the core itself, or the corresponding drivers have some sort of bug, that does not allow the full bandwidth of 5Gbps per lane to be used.
I will do some more tests next week.
Regards,
Andrej

Cone83's picture
Cone83
Junior(0)
Thanks for the information. I

Thanks for the information. I'm quite frustrated with this USB 3 issue. I very much hope that you gain some insight on this.
 
Regards

Cone83's picture
Cone83
Junior(0)
Hi Andrej,

Hi Andrej,
 
have you made any progress on this? I was recently talking to another engineer about this issue, and he said this might be because the PS is not fast enough to collect the data in time. I don't quite see how we could work around this.

Kennysvk's picture
Kennysvk
Junior(0)
Hello,

Hello,
sadly I was not able to get it working. I managed to get some frames over both of the USB3.0 cards (PCIe x1 and x4), even on the Kernel version 4.4, by lowering the pixelclock on the camera. But with a low pixelclock, the available fps are also low. I reached only about 8 to 10 fps using one camera and the resolution of 2048x768. Unfortunately even with the lower pixelclock I get failed USB transfers, meaning not every frame captured by the camera gets transferred to the system and the system drops some of the frames. So even the mentioned 8 - 10 fps are even less I guess. With higher pixelclock the frames get dropped right away.
It really can be the slow PS, that is causing this issue. I got a look at the top command in Linux, while transferring data from the camera and the usage of the CPU was almost maxed out by the daemon for the camera driver and the application for the camera. So I think, there is not really something that can be done to solve this.
What image resolution did you try out? The camera I used can be set to a maximal resolution of 2048x2048, but with this setting I am able to run only with the lowest possible pixelclock and the resulting about 4 fps.

Cone83's picture
Cone83
Junior(0)
I wasn't even able to get 640

I wasn't even able to get 640 x 480 running, but I am accessing two cameras at the same time. This is very frustrating. Seems like the Mini-ITX was a big waste of money. Particularly as I haven't yet seen an actually functioning unit despite having already waited for 8 month, but that's another frustrating story...

Kennysvk's picture
Kennysvk
Junior(0)
In my project I also need to

In my project I also need to have two cameras accessed at the same time, but I wanted to try only one fist and even that is a issue.
That is strange, that even that low resolution was not running. I tried also two other cameras with a resolution of 752 x 480 and they were running pretty fast connected to the USB3.0 card. But they were only USB2.0 cameras and had much lower pixelclock, than the USB3.0 cameras.
Sorry to hear that, I´m also sad that it seems, that there is no way to get it working ...