Skip to content

Technical Overview

Understand the architecture of OFP, the purpose of the project, main features and components

PURPOSE

The intent of this project is to enable accelerated routing/forwarding for IPv4 and IPv6, tunneling and termination for a variety of protocols. Unsupported functionality is provided by the host OS networking stack or slowpath. However, in the future we expect to have OFP working independently of a slowpath in a bare metal environment.

OpenFastPath functionality is provided as a library to Fast Path applications that use ODP run to completion execution model and framework. DPDK is in this initial release supported through the ODP-DPDK layer. Native DPDK support is being evaluated.

On UDP and TCP level, the stack uses an optimized callback based zero-copy socket API which enables the usage of the complete packet, including metadata, in user space. This is done without the copy operation typically used by the traditional BSD sockets. Termination of protocols with BSD socket interface for legacy applications is also supported.

UDP, TCP, ICMP code was ported from libuinet which is a well known user space FreeBSD port. The remaining software is ported from reliable open source projects or code developed in house by the OFP team.

ARCHITECTURE

A Linux system built with OpenFastPath support consists of a number main building blocks.

  • Network interface in a separate NIC or embedded in an SoC
  • ODP implemented in hardware, firmware or software
  • OFP library functions
  • User application code
  • Linux host system

The green boxes represents OFP, light blue user application code, black hardware, dark blue ODP and finally orange represents Linux host system:

ofp_system_view
OFP System View

One core (typically #0) is required for Linux system calls, mainly for CLI and route copy and for communication with Linux kernel using TUN/TAP interface. Additional Linux cores might be needed for slowpath if there are a lot of slowpath traffic. Other cores are allocated by ODP for fast path processing.

ODP by default takes control of all Ethernet which means that packet sent by Linux slow path are sent though the TAP interface to Ethernet interfaces.

User Conf Code is a management thread that is running on the Linux core. Usually it is the main thread of the application. It is used to start and configure ODP and OFP and shares same memory as the fastpath cores.

The block called user/default dispatcher implements the dispatcher functionality that reads packets through the ODP API’s. It’s been placed outside OFP to give the user control over which API to use to get packets form ODP. Depending on the underlying ODP implementation and HW, different methods can be selected.

ofp_multicore_system_view
OFP Multicore System View

OFP is a multithreaded multicore application so there is one instance of OFP that run across all data plane cores. However there is a separate independent dispatcher threads to allow different dispatchers on each core

On the cores allocated to fastpath processing, ODP starts only one thread where the dispatcher, OFP and the user application code runs. This is the case when using the hook or callback APIs. If legacy BSD socket APIs are used, those need to run on a separate core or cores not to interfere with the ODP run to completion thread.

Setup is done when the application starts. All the memory for the port configuration table, routing trees, AVL trees etc. is allocated. This ensures that any pointer to a data structure will never be invalid although data may not be correct. Memory is shared between processor cores.

State is kept in memory zones shared between packet processing cores and master core. Routing tables and port configurations are synchronized with slowpath through Netlink API and system commands.

MAIN FEATURES AND COMPONENTS

Main features are:

  • Fast path protocols processing:
    • Layer 4: UDP termination, TCP termination, ICMP protocol
    • Layer 3
      • ARP/NDP
      • IPv4 and IPv6 forwarding and routing
      • IPv4 fragmentation and reassembly
      • VRF for IPv4
      • IGMP and multicast
      • Basic IPsec support
    • Layer 2: Ethernet, VLAN
    • VxLan and GRE Tunneling
  • Routes and MACs are in sync with Linux through Netlink
  • Integration with Linux Slow path IP stack through TAP interface
    • Support for packet callbacks (local, forward)
  • Command line interface
    • Packet dumping and other debugging
    • Statistics, ARP, routes, and interface printing
    • Configuration of routes and interfaces with VRF support
  • Configuration file support.
ofp_system_components
OFP System Components

OFP is composed of the following components:

  • Command line interface (CLI): Internal Command Line Interface to configure and debug OFP.
  • Packet processing: Handling of received packet and protocol implementation
  • Init: ODP environment, and internal memory initialization.
  • Netlink: Copies routes and interfaces from Linux.
  • TUN/TAP: Communication with Linux kernel, slowpath.
  • Port configuration: Ethernet interface functionality.
  • Route, ARP: Route and ARP functions.
  • Application API: Function calls for configuration, packet processing, logging, etc…
  • AVL and Radix/Mtrie trees: Functionalities to be used by other modules.
  • Socket APIs: Optimized zero-copy sockets and BSD style socket interface
  • Basic epoll implementation

INGRESS AND EGRESS PACKET PROCESSING

The packet processing is handled through a series of self-contained processing functions which means that traffic can be inserted at various places in the packet processing flow. Through the use of ODP Classification API OFP performance can be improved by use of hardware to shorten the packet processing paths.

More hardware supported packet processing functions like packet validation, checksum and cryptographic transformations as well as optimized memory/buffers operations abstracted underneath ODP API will further improve overall packet processing rate.

Packet processing component receives data packets from Ethernet interfaces and Linux kernel. Ethernet packets are processed the following way:

ingress_packet_processing
Ingress Packet Processing

The packet processing component also provides API for sending packets.

egress_packet_processing
Egress Packet Processing

OPENFASTPATH APIs

New zero-copy APIs optimized for single thread run-to-completion environments

  • UDP
    • Send: Optimized send function with ODP packet container (odp_packet_t)
    • Receive: A function callback can be registered to read on a socket. Receives ODP packet container and socket handle
  • TCP
    • Accept event: A function callback can be registered for TCP accept event. Receives socket handle.
    • Receive: A function callback can be registered to read on socket. Receives ODP packet container and a socket handle

Other OFP user application APIs

  • Initiation of Open Fast Path
  • Interface configuration
  • Route and MAC table access
  • Packet Ingress and Egress processing
  • Hooks for IP local, IP forwarding and GRE
  • Timer callbacks
  • Statistics
  • Packet capture

You can find the public API in the ./include/api/ folder at the GitHub project page.

COMMAND LINE INTERFACE

OFP contains a Telnet based CLI for configuration and statistics browsing. It can also read the commands from a file when the application starts. Basically CLI is used for debugging and enabling some routing features that are not available by copying routes from Linux kernel. Basic commands are:

  • Dump traffic to the console or to a PCAP file. Configuration options:
  • What kind of traffic (RX, TX, slowpath)
  • Which ports to print
  • File name to print
  • Show information
  • ARP table
  • Routing tables
  • Interfaces(multiple IP addresses per interface)
  • Statistics
  • Configure
  • Routes (IPv4 with VRF)
  • Interfaces (VRF, VLAN, VxLAN)

LIMITATIONS

IP and UDP implementations has been optimized for performance, TCP implementation is functional but not performance optimized.

FURTHER READING

If you want to read more about OFP, take a look at the project documentation in Github.