gen_tcp接受链接时enfile的问题分析及解决
source link: https://blogread.cn/it/article/4851?f=hot1
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
gen_tcp接受链接时enfile的问题分析及解决
最近我们为了安全方面的原因,在RDS服务器上做了个代理程序把普通的MYSQL TCP连接变成了SSL链接,在测试的时候,皓庭同学发现Tsung发起了几千个TCP链接后Erlang做的SSL PROXY老是报告gen_tcp:accept返回{error, enfile}错误。针对这个问题,我展开了如下的调查:
首先man accept手册,确定enfile的原因,因为gen_tcp肯定是调用accept系统调用的:
EMFILE The per-process limit of open file descriptors has been reached.
ENFILE The system limit on the total number of open files has been reached.
从文档来看是由于系统的文件句柄数用完了,我们顺着来调查下:
$
uname
-r
2.6.18-164.el5
$
cat
/proc/sys/fs/file-nr
2040 0 2417338
$
ulimit
-n
65535
由于我们微调了系统的文件句柄,具体参考这里 老生常谈: ulimit问题及其影响, 这些参数看起来非常的正常。
先看下net/socket.c代码:
static
int
sock_alloc_fd(
struct
file **filep)
{
int
fd;
fd = get_unused_fd();
if
(likely(fd >= 0)) {
struct
file *file = get_empty_filp();
*filep = file;
if
(unlikely(!file)) {
put_unused_fd(fd);
return
-ENFILE;
}
}
else
*filep = NULL;
return
fd;
}
static
int
__sock_create(
int
family,
int
type,
int
protocol,
struct
socket **res,
int
kern)
{
...
/*
* Allocate the socket and allow the family to set things up. if
* the protocol is 0, the family is instructed to select an appropriate
* default.
*/
if
(!(sock = sock_alloc())) {
if
(net_ratelimit())
printk(KERN_WARNING
"socket: no more sockets\\n"
);
err = -ENFILE;
/* Not exactly a match, but its the
closest posix thing */
goto
out;
}
...
}
asmlinkage
long
sys_accept(
int
fd,
struct
sockaddr __user *upeer_sockaddr,
int
__user *upeer_addrlen)
{
struct
socket *sock, *newsock;
struct
file *newfile;
int
err, len, newfd, fput_needed;
char
address[MAX_SOCK_ADDR];
sock = sockfd_lookup_light(fd, &err, &fput_needed);
if
(!sock)
goto
out;
err = -ENFILE;
if
(!(newsock = sock_alloc()))
goto
out_put;
...
}
从代码来看,会返回ENFILE都是由于socket句柄分配不出来了,我们还是本着怀疑的态度来写个stap脚本来再次验证下:
$
cat
enfile.stp
probe kernel.
function
(
"kmem_cache_alloc"
).
return
,
kernel.
function
(
"get_empty_filp"
).
return
{
if
($
return
== 0) { print_backtrace();
exit
();}
}
probe kernel.
function
(
"sock_alloc_fd"
).
return
{
if
($
return
< 0) { print_backtrace();
exit
();}
}
probe syscall.accept.
return
{
if
($
return
== -23) {print_backtrace();
exit
();}
}
probe begin {
println(
":~"
);
}
$
sudo
stap enfile.stp
:~
gen_tcp:accept报告{error, enfile}的时候,也没看到stap报异常,基本上可以排除操作系统的原因了,那么我们现在回到gen_tcp的实现来看。
gen_tcp是个port, 具体实现在erts/emulator/drivers/common/inet_drv.c,我们来看下有ENFILE的地方:
/* Copy a descriptor, by creating a new port with same settings
* as the descriptor desc.
* return NULL on error (ENFILE no ports avail)
*/
static
tcp_descriptor* tcp_inet_copy(tcp_descriptor* desc,SOCKET s,
ErlDrvTermData owner,
int
* err)
{
...
/* The new port will be linked and connected to the original caller */
port = driver_create_port(port, owner,
"tcp_inet"
, (ErlDrvData) copy_desc);
if
((
long
)port == -1) {
*err = ENFILE;
FREE(copy_desc);
return
NULL;
}
...
}
当 driver_create_port 失败的时候,gen_tcp返回ENFILE,看起来这次找对地方了。我们继续看下 driver_create_port的实现:
看下erts/emulator/beam/io.c:
/*
* Driver function to create new instances of a driver
* Historical reason: to be used with inet_drv for creating
* accept sockets inorder to avoid a global table.
*/
ErlDrvPort
driver_create_port(ErlDrvPort creator_port_ix,
/* Creating port */
ErlDrvTermData pid,
/* Owner/Caller */
char
* name,
/* Driver name */
ErlDrvData drv_data)
/* Driver data */
{
...
rp = erts_pid2proc(NULL, 0, pid, ERTS_PROC_LOCK_LINK);
if
(!rp) {
erts_smp_mtx_unlock(&erts_driver_list_lock);
return
(ErlDrvTermData) -1;
/* pid does not exist */
}
if
((port_num = get_free_port()) < 0) {
errno
= ENFILE;
erts_smp_proc_unlock(rp, ERTS_PROC_LOCK_LINK);
erts_smp_mtx_unlock(&erts_driver_list_lock);
return
(ErlDrvTermData) -1;
}
port_id = make_internal_port(port_num);
port = &erts_port[port_num & erts_port_tab_index_mask];
...
}
get_free_port() Basic Applications -> erts -> erl -> System Flags:
+K true|false
Enables or disables the kernel poll functionality if the emulator has kernel poll support. By default the kernel poll; functionality is disabled. If the emulator doesn\'t have kernel poll support and the +K flag is passed to the emulator, a warning is issued at startup.
If you meet all requirements, you can enable it in this way:
erl -s ejabberd +K true ...
Mnesia Tables to Disk
By default, ejabberd uses Mnesia as its database. In Mnesia you can configure each table in the database to be stored on RAM, on RAM and on disk, or only on disk. You can configure this in the web interface: Nodes -> \'mynode\' -> DB Management. Modification of this option will consume some memory and CPU time.
Number of Concurrent ETS and Mnesia Tables: ERL_MAX_ETS_TABLES
The number of concurrent ETS and Mnesia tables is limited. When the limit is reached, errors will appear in the logs:
** Too many db tables **
You can safely increase this limit when starting ejabberd. It impacts memory consumption but the difference will be quite small.
erl -s ejabberd -env ERL_MAX_ETS_TABLES 20000 ...
小结:很多问题是很绕的,要多方面考虑验证。
建议继续学习:
扫一扫订阅我的微信号:IT技术博客大学习
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK