In Percona\u2019s managed services, we often receive customer questions on communication failure errors\u2014where customers are faced with intermittent \u201cGot an error reading communication packets\u201d messages. I thought this topic deserved blog post so we can discuss possible reasons for this error, and how to remedy this problem. I hope this will help readers on how to investigate and resolve this problem.<\/p>\n
First of all, whenever a communication error occurs it increments the status counter for either\u00a0Aborted_clients<\/a>\u00a0or\u00a0Aborted_connects<\/a>, which describe the\u00a0number of connections that were aborted because the client died without closing the connection properly and the number of failed attempts to connect to MySQL server (respectively). The possible reasons for both errors are numerous (see the\u00a0Aborted_clients increments<\/strong>\u00a0or\u00a0Aborted_connects increments<\/strong>\u00a0sections in the MySQL\u00a0manual<\/a>).<\/p>\n In the case of\u00a0log_warnings<\/a>>1, MySQL also writes this information to the error log (shown below):<\/p>\n In this case, MySQL increments the status counter for\u00a0 The above is not an all-inclusive list.Now, how to identify what causing this problem and how to remedy this problem.<\/p>\n To be honest, aborted connection errors are not easy to diagnose. But in my experience, it\u2019s related to network\/firewall issues most of the time. We usually investigate those issues with the help of Percona toolkit scripts, i.e.\u00a0pt-summary<\/a>\u00a0\/\u00a0pt-mysql-summary<\/a>\u00a0\/\u00a0pt-stalk<\/a>. The outputs from those scripts can be very helpful.<\/p>\n Some of the reasons can be:<\/p>\n Aborted connections happen because a connection was not closed properly. The server can\u2019t cause aborted connections unless there is a networking problem\u00a0between the server and the client (like the\u00a0server is half duplex, and the client is full duplex)\u2014but that is the network causing the problem, not the server. In any case, such problems should show up as errors on the networking interface. To be extra sure, check the\u00a0 Another way to troubleshoot this problem is via\u00a0 I found this\u00a0blog post<\/a>\u00a0useful in explaining how to use\u00a0 For network issues, use a ping to calculate the round trip time (RTT) between machine where mysqld is located and the machine from where the application makes requests. Send a large file (1GB or more) to and from client and server machines, watch the\u00a0process\u00a0using\u00a0 One other idea I can think of is to capture the\u00a0 Along with that, you should also check the network infrastructure sitting between the client and the server for proxies, load balancers, and\u00a0firewalls that could be causing a problem.<\/p>\n I\u2019ve tried to cover communication failure errors, and how to identify and fix the possible aborted connections. Take into account,\u00a0faulty ethernets, hubs, switches, cables, and so forth can cause this issue as well. You must replace the hardware itself to properly diagnose these issues.<\/p>\n","protected":false},"excerpt":{"rendered":" In Percona\u2019s managed services, we often receive customer questions on communication failure errors\u2014where customers are faced with intermittent \u201cGot an error reading communication packets\u201d messages. I thought this topic deserved blog post so we can discuss possible reasons for this error, and how to remedy this problem. I hope this will help readers on how […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[13],"tags":[],"class_list":["post-25719","post","type-post","status-publish","format-standard","hentry","category-13"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p6cOVM-6GP","_links":{"self":[{"href":"https:\/\/tom.tomwork.net\/index.php?rest_route=\/wp\/v2\/posts\/25719","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tom.tomwork.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tom.tomwork.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tom.tomwork.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tom.tomwork.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=25719"}],"version-history":[{"count":1,"href":"https:\/\/tom.tomwork.net\/index.php?rest_route=\/wp\/v2\/posts\/25719\/revisions"}],"predecessor-version":[{"id":25720,"href":"https:\/\/tom.tomwork.net\/index.php?rest_route=\/wp\/v2\/posts\/25719\/revisions\/25720"}],"wp:attachment":[{"href":"https:\/\/tom.tomwork.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=25719"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tom.tomwork.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=25719"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tom.tomwork.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=25719"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}[Warning<\/span>] Aborted<\/span> connection<\/span> 305628<\/span> to<\/span> db<\/span>: 'db'<\/span> user<\/span>: 'dbuser'<\/span> host<\/span>: 'hostname'<\/span> (Got<\/span> an<\/span> error<\/span> reading<\/span> communication<\/span> packets<\/span>)<\/span><\/pre>\n<\/div>\n[Warning<\/span>] Aborted<\/span> connection<\/span> 305627<\/span> to<\/span> db<\/span>: 'db'<\/span> user<\/span>: 'dbuser'<\/span> host<\/span>: 'hostname'<\/span> (Got<\/span> an<\/span> error<\/span> reading<\/span> communication<\/span> packets<\/span>)<\/span><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\nAborted_clients<\/code>, which could mean:<\/p>\n\n
How Do We Identify What Caused This Problem, and How Do We Fix It?<\/h2>\n
\n
wait_timeout<\/code>\u00a0to close them. I strongly recommend changing the application logic to properly close connections at the end of an operation.<\/li>\nmax_allowed_packet<\/code>\u00a0is high enough, and that your clients are not receiving a \u201cpacket too large\u201d message. This situation aborts the connection without properly closing it.<\/li>\nTIME_WAIT<\/code>. I\u2019ve noticed many\u00a0TIME_WAIT<\/code>\u00a0notifications from the netstat, so I would recommend confirming the connections are well managed to close on the application side.<\/li>\nnet_read_timeout<\/code>\u00a0is rarely the problem unless you have an\u00a0extremely poor network. Try tweaking those values, however, because in most cases a query is generated and sent as a\u00a0single packet to the server, and applications can\u2019t switch to doing something else while leaving the server with a\u00a0partially received query. There is a very detailed\u00a0blog post<\/a>\u00a0on this topic from our CEO, Peter Zaitsev.<\/li>\n<\/ul>\nifconfig -a<\/code>\u00a0output on the MySQL server to check if there are errors.<\/p>\ntcpdump<\/code>. You can refer to this\u00a0blog post<\/a>\u00a0on how to track down the\u00a0source of aborted connections. Look\u00a0for potential network issues, timeouts and resource issues with MySQL.<\/p>\ntcpdump<\/code>\u00a0on busy hosts. It provides help for tracking down the TCP exchange sequence that led to the aborted connection, which can help you figure out why the connection broke.<\/p>\ntcpdump<\/code>, then check if an error occurred during transfer. Repeat this test few times. I also found this from my colleague Marco Tusa useful:\u00a0Effective way to check network connection<\/a>.<\/p>\nnetstat -s<\/code>\u00a0output along with a timestamp after every N seconds (e.g., 10 seconds so you can relate\u00a0\u00a0netstat -s<\/code>\u00a0output of BEFORE and AFTER an aborted connection error from the MySQL error log). With the aborted connection error timestamp, you can co-relate it with the\u00a0netstat<\/code>\u00a0sample captured as per a timestamp of\u00a0netstat<\/code>, and watch which error counters increased under the TcpExt section of\u00a0netstat -s<\/code>.<\/p>\nConclusion<\/strong><\/h2>\n