3

Linux kernel spinlock使用不当的后果

 2 years ago
source link: http://linuxperf.com/?p=138
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.

Linux kernel spinlock使用不当的后果

spinlock(自旋锁)是内核中最常见的锁,它的特点是:等待锁的过程中不休眠,而是占着CPU空转,优点是避免了上下文切换的开销,缺点是该CPU空转属于浪费,spinlock适合用来保护快进快出的临界区。

spinlock有很多限制条件,其中最重要的是,持有spinlock的CPU不能被抢占,持有spinlock的代码不能休眠。如果违反,会发生死锁,后果很严重。持有spinlock的代码不能休眠,这一条是开发者编写内核程序使用spinlock的时候要人工保证的。而持有spinlock的CPU不能被抢占是由spinlock的API本身提供保证,出于效率的考虑,spinlock的API提供了多种选择,对抢占的防止程度也不一样,开发者在选用的时候需要谨慎,下文对此详细展开。

Linux内核提供了多种spinlock的API,其中最常用的是:
spin_lock/spin_unlock — 禁止内核抢占
spin_lock_irq/spin_unlock_irq — 禁止内核抢占并屏蔽中断
spin_lock_irqsave/spin_unlock_irqrestore — 禁止内核抢占并屏蔽中断,事先保存中断屏蔽位并事后恢复原状

spin_lock()禁止了内核抢占,但是没有屏蔽中断,意味着持有该spinlock的CPU有可能被中断抢占。如果你的某段内核代码选用了spin_lock(),就必须保证这段代码不会被任何中断处理程序调用,否则就会发生死锁(参见后文的一个实际发生的案例)。如果某段内核代码有可能被中断处理程序调用,那就只能选择spin_lock_irq或spin_lock_irqsave。

下面是一个刚发生的实际案例,SLES11 SP4的系统失去响应,kdump生成了vmcore,分析过程中发现以下backtraces揭示了原因:

crash64> bt -c 2
PID: 47     TASK: ffff880230c78400  CPU: 2   COMMAND: "kswapd0"
#0 [ffff88023fa46e40] crash_nmi_callback at ffffffff81024bcf
#1 [ffff88023fa46e50] notifier_call_chain at ffffffff8146d6e7
#2 [ffff88023fa46e80] __atomic_notifier_call_chain at ffffffff8146d72d
#3 [ffff88023fa46e90] notify_die at ffffffff8146d77d
#4 [ffff88023fa46ec0] default_do_nmi at ffffffff8146ad13
#5 [ffff88023fa46ee0] do_nmi at ffffffff8146ae08
#6 [ffff88023fa46ef0] restart_nmi at ffffffff8146a295
    [exception RIP: _raw_spin_lock+21]
    RIP: ffffffff81469795  RSP: ffff88023fa43738  RFLAGS: 00000283
    RAX: 0000000000002b41  RBX: ffff88023337fc00  RCX: 00000000000000d0
    RDX: 0000000000002b3f  RSI: ffff88023fa437d4  RDI: ffffffff81a02700
    RBP: 0000000000000001   R8: 0000000000000000   R9: ffff88023fa43740
    R10: ffff88023ffd95b8  R11: ffff88023ffd9520  R12: ffff88023337fc00
    R13: 0000000000000001  R14: ffff88023337fce0  R15: 0000000000000008
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
--- <NMI exception stack> ---
#7 [ffff88023fa43738] _raw_spin_lock at ffffffff81469795
#8 [ffff88023fa43738] __shrink_dcache_sb at ffffffff81176b36
#9 [ffff88023fa437b8] prune_dcache at ffffffff81176d82
#10 [ffff88023fa43808] shrink_dcache_memory at ffffffff81176e88
#11 [ffff88023fa43818] shrink_slab at ffffffff81110874
#12 [ffff88023fa438b8] do_try_to_free_pages at ffffffff81111c43
#13 [ffff88023fa43928] try_to_free_pages at ffffffff81112072
#14 [ffff88023fa439c8] __alloc_pages_slowpath at ffffffff81104a6f
#15 [ffff88023fa43af8] __alloc_pages_nodemask at ffffffff81105079
#16 [ffff88023fa43b98] alloc_pages_current at ffffffff8113da6e
#17 [ffff88023fa43bd8] bnx2x_alloc_rx_sge at ffffffffa055d484 [bnx2x]
#18 [ffff88023fa43c18] bnx2x_fill_frag_skb at ffffffffa055d75e [bnx2x]
#19 [ffff88023fa43cb8] bnx2x_tpa_stop at ffffffffa055da86 [bnx2x]
#20 [ffff88023fa43d18] bnx2x_rx_int at ffffffffa056084b [bnx2x]
#21 [ffff88023fa43e48] bnx2x_poll at ffffffffa05613b4 [bnx2x]
#22 [ffff88023fa43e88] net_rx_action at ffffffff813adada
#23 [ffff88023fa43ed8] __do_softirq at ffffffff8106925f
#24 [ffff88023fa43f48] call_softirq at ffffffff81472a5c
#25 [ffff88023fa43f60] do_softirq at ffffffff81004695
#26 [ffff88023fa43f90] smp_apic_timer_interrupt at ffffffff81026fd8
#27 [ffff88023fa43fb0] apic_timer_interrupt at ffffffff814721f3
--- <IRQ stack> ---
#28 [ffff880230c7bad8] apic_timer_interrupt at ffffffff814721f3
    [exception RIP: _raw_spin_trylock]
    RIP: ffffffff81469750  RSP: ffff880230c7bb88  RFLAGS: 00000246
    RAX: ffff8802c63aac40  RBX: ffffffff81469c0e  RCX: ffff8802c63aad00
    RDX: ffff880230c7bfd8  RSI: ffff880230c78400  RDI: ffff8802c63aac18
    RBP: ffff8802c63aac18   R8: ffff880230c7a000   R9: 0000000000000000
    R10: ffff88023fa509a0  R11: ffffffff81051970  R12: ffffffff814721ee
    R13: ffffffff81051970  R14: ffffffff81469c0e  R15: ffff880230c7bb80
    ORIG_RAX: ffffffffffffff10  CS: 0010  SS: 0018
#29 [ffff880230c7bb88] __shrink_dcache_sb at ffffffff81176bec
#30 [ffff880230c7bc08] prune_dcache at ffffffff81176d82
#31 [ffff880230c7bc58] shrink_dcache_memory at ffffffff81176e88
#32 [ffff880230c7bc68] shrink_slab at ffffffff81110874
#33 [ffff880230c7bd08] kswapd_shrink_zone at ffffffff81111086
#34 [ffff880230c7bd68] balance_pgdat at ffffffff811115de
#35 [ffff880230c7be78] kswapd at ffffffff81111980
#36 [ffff880230c7bee8] kthread at ffffffff81084946
#37 [ffff880230c7bf48] kernel_thread_helper at ffffffff81472964

我来解释一下,上面的backtraces意思是:CPU 2上正在运行的进程是”kswapd0″(kswapd0是负责swapping的内核线程),它正在压缩dcache以便腾出一些空闲内存,当它执行到__shrink_dcache_sb()的时候被一个中断抢占了CPU,(注意被中断抢占的进程不会离开当前CPU,不会有机会到其它CPU上运行,只能等中断处理结束之后把CPU交还给它),中断处理程序是bnx2x驱动模块(注意看[],表示的是内核模块),它发现内存不够,于是自动清理内存,最终也走到了压缩dcache这一步,也去调用__shrink_dcache_sb(),但是__shrink_dcache_sb()的临界区受到spinlock保护,见下面源代码第0823行,这个名为dcache_lru_lock的spinlock刚才已经被”kswapd0″进程持有了,所以中断处理程序不可能抢到,问题是持有dcache_lru_lock的”kswapd0″进程又被中断抢占了CPU,不可能继续运行,也就没机会释放掉dcache_lru_lock,这就陷入了死锁状态。

// SLES11 SP4: kernel 3.0.101-71, fs/dcache.c
0814 static void __shrink_dcache_sb(struct super_block *sb, int *count, int flags)
0815 {
0816         /* called from prune_dcache() and shrink_dcache_parent() */
0817         struct dentry *dentry;
0818         LIST_HEAD(referenced);
0819         LIST_HEAD(tmp);
0820         int cnt = *count;
0822 relock:
0823         spin_lock(&dcache_lru_lock);
0824         while (!list_empty(&sb->s_dentry_lru)) {
0825                 dentry = list_entry(sb->s_dentry_lru.prev,
0826                                 struct dentry, d_lru);
0827                 BUG_ON(dentry->d_sb != sb);
0829                 if (!spin_trylock(&dentry->d_lock)) {
0830                         spin_unlock(&dcache_lru_lock);
0831                         cpu_relax();
0832                         goto relock;
0833                 }

根本原因在于,既然__shrink_dcache_sb()选用了spin_lock(),就意味着设计者认为它不会被中断处理程序调用,因为spin_lock()不屏蔽中断,是不能防止中断抢占的,只要中断处理程序不调用__shrink_dcache_sb(),死锁就不会发生;如果要让__shrink_dcache_sb()可以被中断处理程序调用,那就不能选用spin_lock(),而应该用spin_lock_irq或spin_lock_irqsave。这个案例中的问题出在bnx2x驱动程序中,它在bnx2x_alloc_rx_sge() 中调用alloc_pages()时不恰当地使用了GFP_KERNEL标志,实际上应该使用GFP_ATOMIC标志,这样alloc_pages()就不会试图去主动回收内存、也就不会最终调用__shrink_dcache_sb()了。此bug记载在SUSE的bsc#975358中,在kernel 3.0.101-77中得以修复。


About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK