Starting from scratch is not always the most prudent way to approach a new standard implementation, especially when there is a fully compliant version of the standard waiting inside a low-priced starter kit (less than $350) accompanied by a fully functional reference design, design software and documentation. Given the continued growth in popularity of the PCI Express (PCIe) serial IO standard, and the mounting need for a low-cost endpoint solution that can hasten next-generation products to market, prudence would suggest taking a closer look at the remarkable value Xilinx Spartan-3 FPGAs can deliver to an incredibly broad range of PCI Express applications and form factors.

PCI Evolution

PCI Express is a general-purpose serial I/O technology suitable for numerous market segments, including desktop, mobile, server, storage and embedded communications. System architects and designers use the PCI Express serial link as a peripheral device interconnect, a chip-to-chip interconnect, and a bridge to other interconnects like 1394b, USB2.0, InifiniBand™ and Ethernet. It can also be used in graphics chipsets for increased graphics bandwidth.

The relatively rapid adoption of PCI Express comes as no surprise. The demands of emerging and future computing models are already exceeding the capabilities of the traditional PCI bus. With CPU speeds that exceed 10 GHz, faster memory speeds, higher-speed graphics, 1 Gigabit and 10 Gigabit LAN, and other mounting pressures, the need for much greater internal system bandwidth is paramount.

Unfortunately, PCI-based shared, parallel-bus-signaling technology is approaching its practical performance limits, making it increasingly difficult to scale-up bandwidth just by increasing the number of signal lines. Besides creating difficult to manage clock-to-data skew and complex (read costly) PCB layout rules, increasing the number of signal lines also increases power dissipation.

The PCI Express architecture will scale to frequencies up to 80 Gigabits per second, support multiple widths, or as defined in serial architecture, multiple lanes (1, 2, 4, 8, 12, 16 and 32), and scale to the limits of copper. It is a low-pin-count interface offering maximum bandwidth per pin, lowering cost and enabling small form factors. Happily, the PCI Express architecture also follows the established PCI software model, allowing backward-compatible support for add-in modules and plug-n-play capabilities.

Additionally, the PCI Express specification has from the outset targeted the requirements of multiple market segments, including desktop, mobile, server, communications, embedded and other emerging and future applications. In short, PCI Express allows for very high available bandwidth per pin with the ability to cost-effectively scale towards the 12 GHz limits of copper signaling technology.

Why Turn an FPGA into PCIe Endpoint?

As is the case with adopting any evolving interconnect standard, one of the first orders of business is to develop solutions that can bridge legacy interface standards (even proprietary interfaces) to the new one. This process is essential to prepare for the anticipated proliferation of boards and devices sporting the new standard.

Whether for a desk-top PC add-in card or an enterprise server IO module, adapting existing systems to accommodate new PCI Express end applications requires the flexibility, scalability and quick time to market inherently available in FPGAs to bridge numerous standards on multiple form factors. Neither of the alternatives—waiting for an ASSP to come along that fits the particular requirement or spinning a custom ASIC to do the job—offer much in the way of good business practice, especially during the new standard’s initial deployment phase.

ASSPs and ASICs are by nature far too limited in their scope to accommodate all of today’s requirements, let alone what may be on the horizon. Ultimately, it may well cost a great deal more in time, development costs, Bill of Materials (BOM) costs and power expenditures to force fit a pre-defined, first-generation ASIC or ASSP into an existing system architecture than it would to create a customized, but fully adaptable PCI Express solution using an FPGA. Moreover, if an FPGA already exits on the board, it may well be possible to integrate the PCI Express endpoint solution with little more than a size upgrade.

The greater value of this approach becomes immediately apparent at the arrival of desirable extensions to the standard or when the need arises to add more “lanes” of bandwidth—a process that will continue for quite some time with the PCI Express standard. With perhaps nothing more than a fresh download of a new configuration, your old system can remain where it is, or your new product line can quickly fill the new market demand. This sort of future-proofing is the essence of the FPGA’s value proposition.

PCI Express Architecture

The PCI Express topology is essentially based on a single host processor or root complex which controls the global memory address space of the entire system (see Figure 1). The root complex enumerates through the hierarchical tree-topology and allocates global memory for all end-point devices it detects in the system.

Figure 1 - PCI Express Topology

Endpoints are the source and sink for PCIe traffic, providing either the request for or completion of all PCIe transactions. The various endpoints interact with each other through memory-mapped transactions on the root complex

The switching function manages the endpoints in a PCI Express implementation as a logical assembly of multiple virtual devices (many more than one would connect directly to the root complex). The switch also bridges endpoints for legacy and new standards (i.e., PCI to PCI, PCIe to PCI, etc.) channelizing traffic between input and output ports while aggregating multiple PCIe functions in the system.

The bridge function enables the connection of standard but dissimilar protocols, e.g., a PCI Express-to-PCI Bridge provides a connection between a PCI Express fabric and a PCI/PCI-X hierarchy.

Similar to the seven-layer Open Systems Interconnection (OSI) model, PCI Express specifies three layers—the physical (PHY), data link, and transaction layers—all defining separate functions. The physical layer provides the signaling and electrical interface, the data link layer creates a reliable lossless medium through which it provides data transport services, and the transaction layer interacts with software to ensure adherence to the protocol standard while passing PCI Express traffic through the channel.

This layered architecture permits upgrades and changes to an individual layer without impacting the others. This is especially valuable if after the PHY specification has solidified, one must make modifications to one or more of the other layers to accommodate new extensions or new requirements. Also included in the specification are advanced features for hardware error recovery and system power management. (For more information about PCI Express, visit www.pcisig.com.)

Spartan-3 PCIe Starter Kit and Reference Designs

The Spartan-3 PCI Express Starter Kit (see Figure 2) is a complete development board solution that gives designers instant access to the capabilities of the Spartan-3 family and the Xilinx PCI Express Endpoint Core. Consisting of a x1 add-in card with a one-million gate Spartan-3 3S1000 FPGA, the Xilinx 1-lane PCI Express PIPE (PHY Interface for PCI Express) Endpoint IP LogiCORE (evaluation version), the Philips PX1011A-EL1 PHY, and DDR memory and Flash memories, this programmable PCI Express Endpoint silicon solution has been successfully adopted in the industry. The entire kit, which also comes with evaluation versions of the Xilinx Integrated Software Environment (ISE™) Foundation and Embedded Developer Kit (EDK) design software suites is available now for only $349. (Visit www.xilinx.com/s3pcie for more information about the Starter Kit.)

Figure -2 Spartan-3 PCI Express Starter Kit

Priced at less than $12.00 US in high volumes, this two-chip solution (see Figure 3) is the industry’s lowest cost programmable PCI Express solution to be added to the PCI-SIG PCI Express Integrators List and to demonstrate full compliance to the PCI Express 1.1 specification.

Figure 3 - Layered architecture enables modularity, reuse and scalability across chip-to-chip interfaces, fabrics and market segments.

This robust solution includes the Philips PCI Express PHY, delivering SERDES (serializer/de-serializer) and Physical Coding Sub-Layer (PCS) capabilities and excellent bit rate performance. The Philips PHY consumes little power and features a small form factor that makes it ideal for ExpressCard applications. The Xilinx Spartan-3 FPGA features an industry-leading combination of block and distributed RAM, up to 84 I/Os, MicroBlaze ™ 32-bit RISC soft processors and embedded XtremeDSP™ functionality supporting multiply and accumulates (MAC) functions (dedicated 18x18 multipliers deliver up to 330 billion MACs per second).

PCI Express LogiCORE IP

Endpoint Core Functional Description

The PCI Express PIPE Endpoint core is organized into four main modules based on the three discrete logical layers defined by the PCI Express Base Specification v1.1. The logic modules that manage all of the system-level functions are the Physical Layer Module (PLM), Data Link Layer Module (LLM), Transaction Layer Module (TLM), and the Configuration Management Module (CMM).

Figure 4 – PCI Express PIPE Endpoint top-level functional blocks and interfaces

Each of the modules is further partitioned into the Receive and the Transmit sections. The Receive processes the inbound information while the Transmit processes the outbound information. The main modules interface with each other and with the User Application using a set of four interfaces as shown in Figure 4. These interfaces are the System interface (SYS), PIPE, Configuration Interface (CFG) and the Transaction Interface (TRN).

The core uses packets to exchange information between the various modules. Packets are formed in the Transaction and Data Link Layers to carry information from the transmitting component to the receiving component. Necessary information is added to the packet being transmitted to handle the packet at those layers. Each layer of the receiving element processes the incoming packet, strips the relevant information and forwards the packet to the next layer. As a result, the received packets get transformed from their Physical Layer representation to their Data Link Layer representation and the Transaction Layer representation.

The main modules comprising the PCI Express Endpoint core and their interfaces are described in the following sections.

Four modules are responsible for handling the functionality related to each of the layers defined by the PCI Express Base Specification v1.1. The functions of these modules include generation and processing of Transaction Layer Packets (TLPs), flow control management, initialization and power management functions, data protection, error checking and retry functions, physical link interface initialization, maintenance and status tracking, serialization, de-serialization and other circuitry for interface operation. These modules and their functionality are described below.

Transaction Layer Module

The upper layer of the PCI Express core architecture is the Transaction Layer. The primary function of the Transaction Layer is to accept, buffer and disseminate TLPs. Packets are formed in the Transaction and Data Link Layers to carry the information from the transmitting component to the receiving component. TLPs are used to communicate transactions, such as read and write, as well as certain types of events. To maximize the efficiency of communication between devices, the Transaction Layer implements a pipelined, full split-transaction protocol; manages credit-based flow control of TLP which eliminates wasted Link bandwidth due to retries; and offers optional support for data poisoning.

Data Link Layer Module

The Data Link Layer acts as an intermediate stage between the Transaction Layer and the Physical Layer. Its primary responsibility is to provide a reliable mechanism for the exchange of TLPs between the two Components on a Link. Services provided by the Data Link Layer include data exchange (TLPs), error detection and recovery, initialization services and the generation and consumption of Data Link Layer Packets (DLLPs). DLLPs are the mechanism used to transfer information between Data Link Layers of two directly connected Components on the Link. They are used for conveying information such as Flow Control and TLP acknowledgments.

Physical Layer Module

The Physical Layer exchanges information with the Data Link Layer in an implementation-specific format. This layer is responsible for converting information received from the Data Link Layer into an appropriate format and transmitting it across the PXPIPE interface at a frequency and width compatible with the external PHY.

Configuration Management Module

This module supports generation and reception of System Management Messages by communicating with the other modules and the user application. The Configuration Management module implements the Configuration Space registers which support legacy PCI2.3 configuration space and the new PCI Express capabilities. It supports both Power Management functions, viz. legacy Programmed Power Management (PPM) and the hardware-autonomous Active State Power Management (ASPM). The Configuration Management Module also supports error reporting and status functionality, and as well, provides interrupts through MSI and INTx interrupt emulation.

Applications and Form Factors

The Spartan-3 based PCI Express solution enables applications requiring PCIe connectivity in low-cost form factors. A simple application of a data acquisition system built around the Spartan-3 PCIe solution in an add-in card form factor can be scaled up to higher link widths if demand for bandwidth increases. Also, if power is the key constraint, one can scale the same add-in card form factor solution to a lower power form factor on a Mini-card or, if feasible, on an ExpressCard. By simply re-targeting the application to another Spartan3 device this portability ensures software preservation and will save the development and debug costs.

AMC form-factor cards provide expansion capabilities for Line Cards in ATCA chassis and are the norm in most server and telecom applications. Spartan-3 PCIe solutions can easily accomplish the protocol bridging (e.g., 10/100 or GbE connectivity to a PCIe Line card) using the Spartan3 PCIe solution.

Also, for single board computer systems, where the BOM cost and the aspect ratios are the primary drivers for simple connectivity and point solutions, Spartan3-based PCI Express solutions can provide the low-cost traffic aggregation and bandwidth.

Table 1 – Applications for Spartan 3 PCI Express Solutions

ExpressCard Application

The new PCMCIA ExpressCard standard supports PCI Express at 2.5 GB/s and USB2.0 at 480 Mb/s. Although the higher throughput requirements of some applications would mandate PCI Express over USB, many applications will find PCI Express more appealing, mainly because:

Manufacturers offering add-on cards for desktop and notebook form factors (such as mini PCI Express cards and ExpressCard modules) only have to support one set of software drivers in PCI Express (which is also backward compatible to PCI).

Having the same PCI Express interface for both internal and external add-on cards results in lower inventory costs for the manufacturers and faster time to market for product development.

Using FPGAs to implement an ExpressCard module has several advantages over alternative solutions such as a PCI Express-to-PCI bridge or an ASIC with PCI Express interface. The bridge solution inevitably increases the cost of the total solution and increases system latency, while the throughput is still limited by the inherent PCI interface. ASICs are fine once you get past the initial NRE costs and development time, unless some system requirement changes or a new extension on the standard appears. So a programmable solution using standard FPGA serves as a middle ground to enable emerging applications with uncompromised throughput, while allowing a short time to market.

Figure 5 – ExpressCard Form Factors

Summary

Change creates challenges. If change is inevitable, flexibility is essential. In the world of interconnect standards, change is the fruit of technological progress toward mutually beneficial ends. In that same world, the flexibility afforded by FPGAs turns change into a market advantage.

Note: PXPIPE is the Philips PHY Specification, an extended version of the PIPE specification.

by Navneet Rao, Connectivity System Architect, Platform

November 16, 2006

[back to top]

Comments on this article? Send them to comments@fpgajournal.com