返回首页
0
0

网卡中断模式

大发排列3官网 发表于 2014年09月19日 16:05 | Hits: 2415
Tag: all

Table of Contents

1   中断模式

大发排列3网址

为了研究 一个机器每秒能处理多少package.

之前观察的结果是:

redis 服务在单机达到 4w qps时 (redis + twemproxy两层)

  • twemproxy: (4w in + 4w out) * 2

  • redis: 4w in + 4w out

  • 此时看到
    • 系统的包速率: 20w in + 20w out, 中断数没观察, softirq打满单核 (nmon+vmstat)
    • 平时大约是 14w in + 14w out, 中断数每秒3.5w

发现:

  • 单核处理包的速度大约是 40w pps,
  • 并非一个包对应一个中断.

1.2   研究

读了这个文档: Interrupt Moderation Using Intel® GbE Controllers

http://www.intel.com/content/dam/doc/application-note/gbe-controllers-interrupt-moderation-appl-note.pdf

  • 如果每个包来了就对应一个中断的话, 包处理延迟很小, 但是很耗CPU
  • 如果来好多个包才一个中断的话, 延迟可能很大

所以这里需要一个算法, 文章中提到三个timer, 总之就是限制每秒不要产生太多中断, 同时保证及时性(如果包来的慢, 也要及时中断)

1.3   文摘

  • Too few interrupts can lead to latencies and too many can unduly burden the server’s processor

  • Basic Interrupt Processing

    Without any interrupt moderation, the GbE controller interrupts the CPU after every packet event (both packet transmission and reception). This involves:

    1. The GbE controller successfully transmits or receives a packet.
    2. The GbE controller generates an interrupt in response to this event.
    3. The CPU suspends its current activity in order to handle the interrupt. This involves saving state information and executing an interrupt handler for the GbE controller.
    4. Software (device drivers) examine the GbE controller to determine the cause of the interrupt. Software might also take additional action(s) based on the exact nature of the interrupt.
    5. The CPU resumes its previous activity.
  • the GbE controller employs a series of timers for moderating or limiting the number of interrupts it generates.

    Rather than immediately interrupting the transmission or reception of a new packet, software can configure the GbE controller to delay the generation of this interrupt. By postponing this interrupt, the GbE controller can collect any additional interrupt events that arrive within this delay period. When it finally does interrupt, the GbE controller is then able to deliver several events to software all at once.

根据处理的是大数据, 还是小数据包(IO patterns), 可以选择几种模式:

Linux里面可以配的几种模式:

.. image:: imgs/network-interrupt-003.png

1.4   怎么配置

# modinfo igb |grep parm
parm:           debug:Debug level (0=none, ..., 16=all)
parmtype:       debug:int
parm:           IntMode:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default 2
parmtype:       IntMode:array of int

parm:           InterruptThrottleRate:Maximum interrupts per second, per vector, (max 100000), default 3=adaptive
parmtype:       InterruptThrottleRate:array of int

我们 InterruptThrottleRate 默认是3.

2   网卡多队列

修改modprobe.conf, 设置igb模块参数:

alias eth0 igb
alias eth1 igb
options igb RSS=8,8

执行:

rmmod igb
modprobe igb
/etc/init.d/network restart

2.1   小结

  • 1G的网卡(1,000,000,000 bit / second) => 每个bit需要1个ns.

  • Ethernet 包大小在66-1538 bytes 之间

  • 如果都处理 1538 byte的包, 每个包需要1538*8 ns => 12.3 us 每秒能处理1G/(1538*8) = 8w个包

  • 如果处理最小的66byte包(tcp空ack包66 bytes) 大约能到 1.8M pps

    • 上面modprob看到 InterruptThrottleRate 最大是10w, 假设一个中断处理50个包 => 可以处理5M pps
  • 但是我们需要到应用层, 所以到不了1.8M pps, 应用层我们的经验单核40w pps.

原文链接: 大发排列3

0     0

评价列表(0)

返回首页