* WGs marked with an * asterisk has had at least one new draft made available during the last 5 days

Bmwg Status Pages

Benchmarking Methodology (Concluded WG)
Ops Area: Benoit Claise, Warren Kumari | 1989-Nov-01 — 2015-Feb-07 

2017-05-11 charter

Benchmarking Methodology (bmwg)


 Current Status: Active

     Al Morton <acmorton@att.com>
     Sarah Banks <sbanks@encrypted.net>

 Operations and Management Area Directors:
     Benoit Claise <bclaise@cisco.com>
     Warren Kumari <warren@kumari.net>

 Operations and Management Area Advisor:
     Warren Kumari <warren@kumari.net>

 Mailing Lists:
     General Discussion: bmwg@ietf.org
     To Subscribe:       https://www.ietf.org/mailman/listinfo/bmwg
     Archive:            https://mailarchive.ietf.org/arch/browse/bmwg/

Description of Working Group:

  The Benchmarking Methodology Working Group (BMWG) will continue to
  produce a series of recommendations concerning the key performance
  characteristics of internetworking technologies, or benchmarks for
  network devices, systems, and services. Taking a view of networking
  divided into planes, the scope of work includes benchmarks for the
  management, control, and forwarding planes.

  Each recommendation will describe the class of equipment, system, or
  service being addressed; discuss the performance characteristics that
  are pertinent to that class; clearly identify a set of metrics that aid
  in the description of those characteristics; specify the methodologies
  required to collect said metrics; and lastly, present the requirements
  for the common, unambiguous reporting of benchmarking results.

  The set of relevant benchmarks will be developed with input from the
  community of users (e.g., network operators and testing organizations)
  and from those affected by the benchmarks when they are published
  (networking and test equipment manufacturers). When possible, the
  benchmarks and other terminologies will be developed jointly with
  organizations that are willing to share their expertise. Joint review
  requirements for a specific work area will be included in the detailed
  description of the task, as listed below.

  To better distinguish the BMWG from other measurement initiatives in the
  IETF, the scope of the BMWG is limited to the characterization of
  implementations of various internetworking technologies
  using controlled stimuli in a laboratory environment. Said differently,
  the BMWG does not attempt to produce benchmarks for live, operational
  networks. Moreover, the benchmarks produced by this WG shall strive to
  be vendor independent or otherwise have universal applicability to a
  given technology class.

  Because the demands of a particular technology may vary from deployment
  to deployment, a specific non-goal of the Working Group is to define
  acceptance criteria or performance requirements.

  An ongoing task is to provide a forum for development and
  advancement of measurements which provide insight on the
  capabilities and operation of implementations of inter-networking technology.

  Ideally, BMWG should communicate with the operations community
  through organizations such as NANOG, RIPE, and APRICOT.

  The BMWG is explicitly tasked to develop benchmarks and methodologies
  for the following technologies:

  BGP Control-plane Convergence Methodology (Terminology is complete):
  With relevant performance characteristics identified, BMWG will prepare
  a Benchmarking Methodology Document with review from the Routing Area
  (e.g., the IDR working group and/or the RTG-DIR). The Benchmarking
  Methodology will be Last-Called in all the groups that previously
  provided input, including another round of network operator input during
  the last call.

  SIP Networking Devices: Develop new terminology and methods to
  characterize the key performance aspects of network devices using
  SIP, including the signaling plane scale and service rates while
  considering load conditions on both the signaling and media planes. This
  work will be harmonized with related SIP performance metric definitions
  prepared by the PMOL working group.

  Traffic Management: Develop the methods to characterize the capacity
  of traffic management features in network devices, such as classification,
  policing, shaping, and active queue management. Existing terminology
  will be used where appropriate. Configured operation will be verified
  as a part of the methodology. The goal is a methodology to assess the
  maximum forwarding performance that a network device can sustain without
  dropping or impairing packets, or compromising the accuracy of multiple
  instances of traffic management functions. This is the benchmark for
  comparison between devices. Another goal is to devise methods that
  utilize flows with congestion-aware transport as part of the traffic
  load and still produce repeatable results in the isolated test environment.

  IPv6 Neighbor Discovery: Large address space in IPv6 subnets presents
  several networking challenges, as described in RFC 6583. Indexes to
  describe the performance of network devices, such as the number of
  reachable devices on a sub-network, are useful benchmarks to the
  operations community. The BMWG will develop the necessary
  terminology and methodologies to measure such benchmarks.

  In-Service Software Upgrade: Develop new methods and benchmarks to
  characterize the upgrade of network devices while in-service,
  considering both data and control plane operations and impacts.
  These devices are generally expected to maintain control plane session
  integrity, including routing connections. Quantification of upgrade
  impact will include packet loss measurement, and other forms of recovery
  behavior will be noted accordingly. The work will produce a definition
  of ISSU, which will help refine the scope. Liaisons will be established
  as needed.

  Data Center Benchmarking: This work will define additional terms,
  benchmarks, and methods applicable to data center performance evaluations.
  This includes data center specific congestion scenarios, switch buffer
  analysis, microburst, head of line blocking, while also using a wide mix
  of traffic conditions. Some aspects from BMWG's past work are not
  meaningful when testing switches that implement new IEEE specifications
  in the area of data center bridging. For example, throughput as defined
  in RFC 1242 cannot be measured when testing devices that implement three
  new IEEE specifications: priority-based flow control (802.1Qbb);
  priority groups (802.1Qaz); and congestion notification (802.1Qau).
  This work will update RFC1242, RFC2544, RFC2889 (and other key RFCs),
  and exchange Liaisons with relevant SDOs, especially at WG Last Call.

  VNF and Related Infrastructure Benchmarking: Benchmarking Methodologies
  have reliably characterized many physical devices. This work item extends
  and enhances the methods to virtual network functions (VNF) and their
  unique supporting infrastructure. A first deliverable from this activity
  will be a document that considers the new benchmarking space to ensure
  that common issues are recognized from the start, using background
  materials from industry and SDOs (e.g., IETF, ETSI NFV).
  Benchmarks for platform capacity and performance characteristics of
  virtual routers, switches, and related components will follow, including
  comparisons between physical and virtual network functions. In many cases,
  the traditional benchmarks should be applicable to VNFs, but the lab
  set-ups, configurations, and measurement methods will likely need to
  be revised or enhanced.

Goals and Milestones:
  Done     - Basic BGP Convergence Benchmarking Methodology to IESG Review
  Done     - Terminology for SIP Device Benchmarking to IESG Review
  Done     - Methodology for SIP Device Benchmarking to IESG Review
  Done     - Draft on Traffic Management Benchmarking to IESG Review
  Done     - Draft on In-Service Software Upgrade Benchmarking to IESG Review
  Aug 2016 - Draft on VNF Benchmarking Considerations to IESG Review
  Mar 2017 - Draft on vSwitch Benchmarking in OPNFV to IESG Review
  Jul 2017 - Methodology for IPv6 Transition Technologies to IESG Review
  Jul 2017 - Methodology for SDV Controller Benchmarking to IESG Review
  Jul 2017 - Terminology for SDV Controller Benchmarking to IESG Review
  Dec 2017 - Draft on IPv6 Neighbor Discovery to IESG Review
  Dec 2017 - Drafts on Data Center Benchmarking to IESG Review

All charter page changes, including changes to draft-list, rfc-list and milestones:

Generated from PyHt script /wg/bmwg/charters.pyht Latest update: 24 Oct 2012 16:51 GMT -