本文最后更新于:January 14, 2022 am
本文主要用于介绍CoreDNS用来记录日志的几种方式以及在生产环境中遇到的一些问题和解决方案。
1、log插件 coredns的日志输出并不如nginx那么完善(并不能在配置文件中指定输出的文件目录,但是可以指定日志的格式),默认情况下不论是log插件还是error插件都会把所有的相关日志输出到程序的standard output
中。使用systemd来管理coredns之后,默认情况下基本就是由rsyslog
和systemd-journald
这两个服务来管理日志。
1.1 log插件配置 前面的文章里我们有简单的介绍过coredns的日志处理,对于log插件也是类似的操作。考虑到CentOS8已经结束支持,这里仅介绍CentOS7的配置方法,更高版本的systemd
可以参考之前的文章 。
对于centos7等系统而言,是不支持在高版本的systemd中的unit
文件中使用append
和file
两个参数的,并且对于这种方式输出的日志缺少了前面的时间和主机名等信息,相对而言还是修改rsyslog的方式要更加的可靠。那么在开启了rsyslog.service
服务的情况下,日志就会输出到/var/log/messages
文件中,或者可以使用journalctl -u coredns
命令来查看全部的日志。
如果想要将coredns的日志全部集中到一个文件进行统一管理,我们可以对负责管理systemd
的日志的rsyslog
服务的配置进行修改,然后再重启rsyslog服务即可生效。
$ vim /etc/rsyslog.confif $programname == 'coredns' then /home/coredns/logs/coredns.log & stop $ systemctl restart rsyslog.service
但是有两点需要额外注意:
CoreDNS在开启日志记录之后的性能要差于不开启日志记录 ,这个很容易理解,因为记录日志需要耗费额外的资源,尤其是当QPS增大之后,记录日志组件的压力也会增大,需要注意两个服务对资源的抢占和分配是否合理;如果遇到写入日志占用大量资源的情况,可以考虑将其配置为只写入部分类型日志 (如只写入错误日志)
rsyslog
和systemd-journald
这两个服务默认情况下都会对日志的写入频率进行限制 ,当QPS增大之后,记录的日志会不完整;当然这个问题可以通过调整rsyslog的参数解决
1.2 rsyslog配置 下面具体介绍一下如何配置rsyslog
和systemd-journald
解除相关的日志限制。
默认情况下我们查看systemd-journald.service
的日志或者是查看/var/log/messages
文件,如果看到类似下面的提示信息,就说明日志的输出频率太高被抑制了
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 $ systemctl status systemd-journald.service ● systemd-journald.service - Journal Service Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static; vendor preset: disabled) Active: active (running) since Wed 2021-12-01 09:30:03 CST; 3 weeks 6 days ago Docs: man:systemd-journald.service(8) man:journald.conf(5) Main PID: 709 (systemd-journal) Status: "Processing requests..." CGroup: /system.slice/systemd-journald.service └─709 /usr/lib/systemd/systemd-journald Dec 28 15:36:27 tiny-server systemd-journal[709]: Suppressed 13174 messages from /system.slice/coredns.service Dec 28 15:36:57 tiny-server systemd-journal[709]: Suppressed 13185 messages from /system.slice/coredns.service Dec 28 15:37:27 tiny-server systemd-journal[709]: Suppressed 11600 messages from /system.slice/coredns.service Dec 28 15:37:57 tiny-server systemd-journal[709]: Suppressed 13896 messages from /system.slice/coredns.service Dec 28 15:38:27 tiny-server systemd-journal[709]: Suppressed 13653 messages from /system.slice/coredns.service Dec 28 15:38:57 tiny-server systemd-journal[709]: Suppressed 16503 messages from /system.slice/coredns.service Dec 28 15:39:27 tiny-server systemd-journal[709]: Suppressed 13300 messages from /system.slice/coredns.service Dec 28 15:39:57 tiny-server systemd-journal[709]: Suppressed 10491 messages from /system.slice/coredns.service Dec 28 15:40:27 tiny-server systemd-journal[709]: Suppressed 12079 messages from /system.slice/coredns.service Dec 28 15:40:57 tiny-server systemd-journal[709]: Suppressed 17182 messages from /system.slice/coredns.service $ tail -f /var/log/messages Dec 28 15:43:55 tiny-server rsyslogd: imjournal: begin to drop messages due to rate-limiting
解决方案也比较简单,我们对这两个服务的配置参数进行调整,将限制日志写入频率和写入时间间隔都设为0(即不做限制),然后重启服务即可。
echo "RateLimitInterval=0" >> /etc/systemd/journald.confecho "RateLimitBurst=0" >> /etc/systemd/journald.conf systemctl restart systemd-journald.service echo "\$imjournalRatelimitInterval 0" >> /etc/rsyslog.confecho "\$imjournalRatelimitBurst 0" >> /etc/rsyslog.conf systemctl restart rsyslog.service
需要特别注意的是:上述两个参数对所有使用rsyslog的服务都会生效,如果有其他大量写入日志的服务,建议调整日志文件输出目录,同时关注/var/log
等目录的硬盘空间情况。
2、dnstap插件 CoreDNS的原生日志功能对于一个DNS服务器的绝大部分应用场景来说是足够使用的,如果有更进阶的需求,可以考虑使用dnstap
插件来实现。
dnstap是一个基于谷歌的缓冲区协议(Protocol buffers)实现的用于DNS的灵活的、结构化的、二进制日志格式记录软件,目前已经获得了bind9、unbound、coredns、knot等几乎所有主流dns软件的支持。dnstap目前支持二进制日志格式、text日志格式、yaml日志格式和json日志格式。
CoreDNS官网对于dnstap插件的介绍 比较简单,不过官网页面 有比较详细的介绍。
dnstap
is a flexible, structured binary log format for DNS software. It uses Protocol Buffers to encode events that occur inside DNS software in an implementation-neutral format.
Currently dnstap
can only encode wire-format DNS messages. It is planned to support additional types of DNS log information.
Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.
缓冲区协议(Protocol buffers)是 Google 用于序列化结构数据的语言中立、平台中立、可扩展机制。可以将其和XML相对比,但Protocol buffers更小、更快、更简单。 使用者只需定义一次数据的结构化方式,然后就可以使用特殊生成的源代码轻松地将结构化数据写入和读取各种数据流,并使用各种语言。
这里需要额外注意的是:
dnstap和CoreDNS原生的log等日志记录插件可以同时使用
尽管CoreDNS默认内置dnstap插件,但是要使用dnstap还是需要我们自己手动安装dnstap,再到corefile中配置dnstap的socket路径或者通信端口才能使用
2.1 dnstap安装 dnstap官方在github上面开源了一个golang-dnstap 的项目,可以使用go来快速安装dnstap
wget https://go.dev/dl/go1.17.5.linux-amd64.tar.gz tar -zxvf go1.17.5.linux-amd64.tar.gz -C /usr/localln -s /usr/local/go/bin/go /usr/bin/goexport GOROOT=/usr/local/goexport GOBIN=$GOROOT /binexport PATH=$PATH :$GOBIN export GOPATH=/home/gopath
执行go version
命令确定go安装无误之后就可以直接安装dnstap:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 go get -u github.com/dnstap/golang-dnstap/dnstap go install github.com/dnstap/golang-dnstap/dnstap@latest go env -w GOPROXY=https://goproxy.cn $ dnstap --help Usage: dnstap [OPTION]... -T value write dnstap payloads to tcp/ip address -U value write dnstap payloads to unix socket -a append to the given file, do not overwrite. valid only when outputting a text or YAML file. -j use verbose JSON output -l value read dnstap payloads from tcp/ip -q use quiet text output -r value read dnstap payloads from file -t duration I/O timeout for tcp/ip and unix domain sockets -u value read dnstap payloads from unix socket -w string write output to file -y use verbose YAML output Quiet text output format mnemonics: AQ: AUTH_QUERY AR: AUTH_RESPONSE RQ: RESOLVER_QUERY RR: RESOLVER_RESPONSE CQ: CLIENT_QUERY CR: CLIENT_RESPONSE FQ: FORWARDER_QUERY FR: FORWARDER_RESPONSE SQ: STUB_QUERY SR: STUB_RESPONSE TQ: TOOL_QUERY TR: TOOL_RESPONSE
2.2 coredns配置 coredns中对dnstap的配置非常简单,可以选择配置socket通信 或者是tcp通信 ,full
参数则是可以记录更多的参数信息,如同样的一次查询,红框是开启了full
参数,而绿框则是不开启,两者相差较大。
dnstap tcp://127.0.0.1:6000 full dnstap unix:///tmp/dnstap.sock full
2.3 dnstap配置 dnstap的配置也比较简单,基本可以分为三个部分:
指定通信方式:如-u /tmp/dnstap.sock
或者-l 127.0.0.1:6000
指定日志文件:如-w /home/coredns/logs/dnstap.log
指定日志格式:默认为二进制格式,-q
为text文本
,-y
为yaml格式
,-j
为json格式
其他参数:如-a
追加写入内容到日志文件中而非覆盖原有的文件
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 dnstap -u /tmp/dnstap.sock -w /home/coredns/logs/dnstap.log $ tail -f /home/coredns/logs/dnstap.log"protobuf:dnstap.Dnstap" dnstap -u /tmp/dnstap.sock -w /home/coredns/logs/dnstap.log -q -a $ tail -f /home/coredns/logs/dnstap.log 16:14:39.329430 CQ 10.31.100.100 UDP 53b "tinychen.com." IN A 16:14:39.329546 FQ 127.0.0.1 UDP 53b "tinychen.com." IN A 16:14:39.330241 FR 127.0.0.1 UDP 69b "tinychen.com." IN A 16:14:39.330441 CR 10.31.100.100 UDP 81b "tinychen.com." IN A dnstap -u /tmp/dnstap.sock -w /home/coredns/logs/dnstap.log -y -a $ tail -f /home/coredns/logs/dnstap.logtype : MESSAGE message: type : CLIENT_QUERY query_time: !!timestamp 2022-01-12 08:23:25.211126656 socket_family: INET socket_protocol: UDP query_address: 10.31.100.100 query_port: 58562 query_message: | ;; opcode: QUERY, status: NOERROR, id : 30286 ;; flags: rd ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; QUESTION SECTION: ;tinychen.com. IN A ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ; EDNS: version 0; flags: ; udp: 4096 ; COOKIE: 22c8a1e55b27c1b9 ---type : MESSAGE message: type : FORWARDER_QUERY query_time: !!timestamp 2022-01-12 08:23:25.211208675 socket_family: INET socket_protocol: UDP query_address: 10.31.100.100 response_address: 127.0.0.1 query_port: 58562 response_port: 35353 query_message: | ;; opcode: QUERY, status: NOERROR, id : 30286 ;; flags: rd ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; QUESTION SECTION: ;tinychen.com. IN A ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ; EDNS: version 0; flags: do ; udp: 2048 ; COOKIE: 22c8a1e55b27c1b9 ---type : MESSAGE message: type : FORWARDER_RESPONSE query_time: !!timestamp 2022-01-12 08:23:25.211208675 response_time: !!timestamp 2022-01-12 08:23:25.243335638 socket_family: INET socket_protocol: UDP query_address: 10.31.100.100 response_address: 127.0.0.1 query_port: 58562 response_port: 35353 response_message: | ;; opcode: QUERY, status: NOERROR, id : 30286 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; QUESTION SECTION: ;tinychen.com. IN A ;; ANSWER SECTION: tinychen.com. 589 IN A 1.12.217.55 ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ; EDNS: version 0; flags: do ; udp: 1232 ---type : MESSAGE message: type : CLIENT_RESPONSE query_time: !!timestamp 2022-01-12 08:23:25.211126656 response_time: !!timestamp 2022-01-12 08:23:25.243503207 socket_family: INET socket_protocol: UDP query_address: 10.31.100.100 query_port: 58562 response_message: | ;; opcode: QUERY, status: NOERROR, id : 30286 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; QUESTION SECTION: ;tinychen.com. IN A ;; ANSWER SECTION: tinychen.com. 589 IN A 1.12.217.55 ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ; EDNS: version 0; flags: ; udp: 4096 ; COOKIE: 22c8a1e55b27c1b9 --- dnstap -u /tmp/dnstap.sock -w /home/coredns/logs/dnstap.log -j -a $ tail -f /home/coredns/logs/dnstap.log {"type" :"MESSAGE" ,"message" :{"type" :"CLIENT_QUERY" ,"query_time" :"2022-01-12T08:36:41.745130199Z" ,"socket_family" :"INET" ,"socket_protocol" :"UDP" ,"query_address" :"10.31.100.100" ,"query_port" :58561,"query_message" :";; opcode: QUERY, status: NOERROR, id: 5835\n;; flags: rd ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1\n\n;; QUESTION SECTION:\n;tinychen.com.\tIN\t A\n\n;; ADDITIONAL SECTION:\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 4096\n; COOKIE: fd6716688b43710a\n" }} {"type" :"MESSAGE" ,"message" :{"type" :"FORWARDER_QUERY" ,"query_time" :"2022-01-12T08:36:41.745243809Z" ,"socket_family" :"INET" ,"socket_protocol" :"UDP" ,"query_address" :"10.31.100.100" ,"response_address" :"127.0.0.1" ,"query_port" :58561,"response_port" :35353,"query_message" :";; opcode: QUERY, status: NOERROR, id: 5835\n;; flags: rd ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1\n\n;; QUESTION SECTION:\n;tinychen.com.\tIN\t A\n\n;; ADDITIONAL SECTION:\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: do; udp: 2048\n; COOKIE: fd6716688b43710a\n" }} {"type" :"MESSAGE" ,"message" :{"type" :"FORWARDER_RESPONSE" ,"query_time" :"2022-01-12T08:36:41.745243809Z" ,"response_time" :"2022-01-12T08:36:41.83691354Z" ,"socket_family" :"INET" ,"socket_protocol" :"UDP" ,"query_address" :"10.31.100.100" ,"response_address" :"127.0.0.1" ,"query_port" :58561,"response_port" :35353,"response_message" :";; opcode: QUERY, status: SERVFAIL, id: 5835\n;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1\n\n;; QUESTION SECTION:\n;tinychen.com.\tIN\t A\n\n;; ADDITIONAL SECTION:\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: do; udp: 1232\n" }} {"type" :"MESSAGE" ,"message" :{"type" :"CLIENT_RESPONSE" ,"query_time" :"2022-01-12T08:36:41.745130199Z" ,"response_time" :"2022-01-12T08:36:41.837144129Z" ,"socket_family" :"INET" ,"socket_protocol" :"UDP" ,"query_address" :"10.31.100.100" ,"query_port" :58561,"response_message" :";; opcode: QUERY, status: SERVFAIL, id: 5835\n;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1\n\n;; QUESTION SECTION:\n;tinychen.com.\tIN\t A\n\n;; ADDITIONAL SECTION:\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 4096\n; COOKIE: fd6716688b43710a\n" }}
2.4 其他问题 目前在使用dnstap的过程中发现了两个小问题:
dnstap日志记录时间使用的是UTC时间记录,对于国内的用户,时间上就有八个小时的差距
dnstap限制缓冲区的消息条数为10000条,超出限制的消息不会被写入dnstap中
Jan 13 20:41:56 coredns2 coredns: [WARNING ] plugin/dnstap: Dropped dnstap messages: 196 Jan 13 20:41:56 coredns2 coredns: [WARNING ] plugin/dnstap: Dropped dnstap messages: 680 Jan 13 20:41:58 coredns2 coredns: [WARNING ] plugin/dnstap: Dropped dnstap messages: 1900 Jan 13 20:41:58 coredns2 coredns: [WARNING ] plugin/dnstap: Dropped dnstap messages: 10118 Jan 13 20:41:59 coredns2 coredns: [WARNING ] plugin/dnstap: Dropped dnstap messages: 1472 Jan 13 20:41:59 coredns2 coredns: [WARNING ] plugin/dnstap: Dropped dnstap messages: 9430 Jan 13 20:42:02 coredns2 coredns: [WARNING ] plugin/dnstap: Dropped dnstap messages: 3669 Jan 13 20:42:02 coredns2 coredns: [WARNING ] plugin/dnstap: Dropped dnstap messages: 23351 Jan 13 20:42:06 coredns2 coredns: [WARNING ] plugin/dnstap: Dropped dnstap messages: 5776 Jan 13 20:42:06 coredns2 coredns: [WARNING ] plugin/dnstap: Dropped dnstap messages: 35511 Jan 13 20:42:17 coredns2 coredns: [WARNING ] plugin/dnstap: Dropped dnstap messages: 13161 Jan 13 20:42:17 coredns2 coredns: [WARNING ] plugin/dnstap: Dropped dnstap messages: 81395 Jan 13 20:42:41 coredns2 coredns: [WARNING ] plugin/dnstap: Dropped dnstap messages: 29500 Jan 13 20:42:41 coredns2 coredns: [WARNING ] plugin/dnstap: Dropped dnstap messages: 180877
如果想要突破这个限制,可以考虑在编译的时候修改源码
const ( tcpWriteBufSize = 1024 * 1024 // there is no good explanation for why this number has this value. queueSize = 10000 // idem. tcpTimeout = 4 * time.Second flushTimeout = 1 * time.Second )
3、errors插件 3.1 errors插件简介 CoreDNS官方有对errors插件比较详细的介绍和使用说明 ,简单来说errors插件就是专门用来记录错误信息日志的一个组件。这里需要注意的是:
并不是不使用errors插件,日志中就不会输出[ERROR]
级别的日志;其他的插件和CoreDNS本身也是可以输出[ERROR]
级别的日志
errors插件默认情况下输出的日志级别都是[ERROR]
,但是也可以通过consolidate
命令进行定义
3.2 errors插件配置 errors的配置非常简单,位置和log插件一样
.:53 { forward . 127.0.0.1 :35353 log errors cache { success 10240 600 60 denial 5120 60 5 } }
开启前后的对比如下:
Jan 14 11:50:12 coredns2 coredns: [INFO] 10.31.53.2:36586 - 34882 "A IN tinychen.com. udp 38 false 4096" - - 0 1.001413555s Jan 14 11:50:33 coredns2 coredns: [INFO] 10.31.53.2:33796 - 62387 "A IN tinychen.com. udp 38 false 4096" - - 0 1.001076641s Jan 14 11:50:33 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:57591->127.0.0.1:35353: read : connection refused
当然还可以有类似正则的配置方式可以简化日志输出,同时定义
errors { consolidate 30s ".* connection refused$" warning consolidate 5m ".* i/o timeout$" warning consolidate 30s "^Failed to .+" }
这样子可以把多条错误日志合并输出,结果如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 Jan 14 14:29:31 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:53697->127.0.0.1:35353: read : connection refused Jan 14 14:29:32 coredns2 coredns: [INFO] 10.31.53.2:38422 - 21429 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000386185s Jan 14 14:29:32 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:34774->127.0.0.1:35353: read : connection refused Jan 14 14:29:33 coredns2 coredns: [INFO] 10.31.53.2:44775 - 40543 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000418209s Jan 14 14:29:33 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:51575->127.0.0.1:35353: read : connection refused Jan 14 14:29:33 coredns2 coredns: [INFO] 10.31.53.2:41858 - 30773 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000285244s Jan 14 14:29:33 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:54027->127.0.0.1:35353: read : connection refused Jan 14 14:29:34 coredns2 coredns: [INFO] 10.31.53.2:56657 - 31457 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000280881s Jan 14 14:29:34 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:59189->127.0.0.1:35353: read : connection refused Jan 14 14:29:34 coredns2 coredns: [INFO] 10.31.53.2:48445 - 29130 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000399419s Jan 14 14:29:34 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:35171->127.0.0.1:35353: read : connection refused Jan 14 14:29:35 coredns2 coredns: [INFO] 10.31.53.2:42290 - 310 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000359849s Jan 14 14:29:35 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:40662->127.0.0.1:35353: read : connection refused Jan 14 14:29:36 coredns2 coredns: [INFO] 10.31.53.2:33545 - 28529 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000456204s Jan 14 14:29:36 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:37439->127.0.0.1:35353: read : connection refused Jan 14 14:29:36 coredns2 coredns: [INFO] 10.31.53.2:53167 - 18372 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000426039s Jan 14 14:29:36 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:45881->127.0.0.1:35353: read : connection refused Jan 14 14:29:37 coredns2 coredns: [INFO] 10.31.53.2:44851 - 52388 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000264627s Jan 14 14:29:37 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:58678->127.0.0.1:35353: read : connection refused Jan 14 14:29:37 coredns2 coredns: [INFO] 10.31.53.2:59500 - 19169 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000385832s Jan 14 14:29:37 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:36355->127.0.0.1:35353: read : connection refused Jan 14 14:29:38 coredns2 coredns: [INFO] 10.31.53.2:48272 - 35987 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000359493s Jan 14 14:29:38 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:52055->127.0.0.1:35353: read : connection refused Jan 14 14:29:39 coredns2 coredns: [INFO] 10.31.53.2:55974 - 48529 "A IN tinychen.com. udp 38 false 4096" - - 0 0.00035554s Jan 14 14:29:39 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:51253->127.0.0.1:35353: read : connection refused Jan 14 14:29:39 coredns2 coredns: [INFO] 10.31.53.2:40670 - 43374 "A IN tinychen.com. udp 38 false 4096" - - 0 0.00024835s Jan 14 14:29:39 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:41919->127.0.0.1:35353: read : connection refused Jan 14 14:29:40 coredns2 coredns: [INFO] 10.31.53.2:42067 - 780 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000257447s Jan 14 14:29:40 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:49359->127.0.0.1:35353: read : connection refused Jan 14 14:29:40 coredns2 coredns: [INFO] 10.31.53.2:56758 - 62500 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000338204s Jan 14 14:29:40 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:44658->127.0.0.1:35353: read : connection refused Jan 14 14:29:41 coredns2 coredns: [INFO] 10.31.53.2:36149 - 32955 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000400937s Jan 14 14:29:41 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:33750->127.0.0.1:35353: read : connection refused Jan 14 14:29:41 coredns2 coredns: [INFO] 10.31.53.2:60504 - 49021 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000272807s Jan 14 14:29:41 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:41003->127.0.0.1:35353: read : connection refused Jan 14 14:32:48 coredns2 coredns: [INFO] 10.31.53.2:60095 - 763 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000389099s Jan 14 14:33:10 coredns2 coredns: [INFO] 10.31.53.2:44256 - 21788 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000556023s Jan 14 14:33:11 coredns2 coredns: [INFO] 10.31.53.2:39654 - 18638 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000394553s Jan 14 14:33:11 coredns2 coredns: [INFO] 10.31.53.2:46340 - 23770 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000405404s Jan 14 14:33:12 coredns2 coredns: [INFO] 10.31.53.2:56168 - 59588 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000351365s Jan 14 14:33:13 coredns2 coredns: [INFO] 10.31.53.2:52337 - 28311 "A IN tinychen.com. udp 38 false 4096" - - 0 0.00033022s Jan 14 14:33:14 coredns2 coredns: [INFO] 10.31.53.2:55564 - 52773 "A IN tinychen.com. udp 38 false 4096" - - 0 0.0002756s Jan 14 14:33:14 coredns2 coredns: [INFO] 10.31.53.2:36233 - 21289 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000307114s Jan 14 14:33:15 coredns2 coredns: [INFO] 10.31.53.2:41822 - 41368 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000379377s Jan 14 14:33:18 coredns2 coredns: [WARNING] plugin/errors: 9 errors like '.* connection refused$' occurred in last 30s