Entropy drained
Today I stumpled upon a strange problem with my Cyrus Imapd. Suddenly new connections just stalled and timed out. The daemon was still running and listening on the specified ports. I connected with telnet to the imapd (143/tcp) and got the correct banner, but connections made with Thunderbird and other clients timed out while Thunderbird was saying "sending authentication data".
My troubleshooting procedure was as follows:
Restart the daemon: Problem still exists.
Check the logfiles: Nothing interesting in the logs.
Telnet to the Imapd: Everything seams to work.
Check more logfiles and restart the service another time: Just in case I missed something before. After this step I knew that something was a bit more skrewed, than I first thougt.
Trace an Imapd process while connecting to it:
This Step revealed the cause of the problem. Take a look at the end of the strace output:
open("/dev/random", O_RDONLY|O_LARGEFILE) = 11 read(11,To calculate hashes for the secure authentication (I use only CRAM-MD5 and DIGEST-MD5) the imapd opened
/dev/random, which is in contrast to/dev/urandom, a blocking device. That means, if you read from/dev/randomand it's empty the process will wait forever - and at this point the tcp connections timed out.To verify that the entropy was drained i checked
/proc/sys/kernel/random/entropy_availwich showed me that no entropy was available:cat /proc/sys/kernel/random/entropy_avail 0
Now I had to fix this problem, but how can you produce enough entropy on a colocated server without hardware random number generator support. The Linux kernel mainly gets the entropy from keyboard and mouse input, which doesn't happen on a rackmounted server, so i checked the entropy hint from Robert Connolly, which was discussed on the hlfs mailing list a time ago, for other solutions. I needed a quick fix, because all connections to the Imapd were timing out for about 40 minutes. I decided to give the rngd-tools a try.
I downloaded the tools and compiled the rngd daemon. This daemon should use
/dev/hwrandom to feed entropy from a hardware random number generator to `
the kernel (/dev/random), but I don't have compatible hardware, so I
decided to use the daemon to feed some entropy from /dev/urandom (which
is non-blocking) to /dev/random:
rngd -r /dev/urandom -o /dev/random
This may not be the best aproach, but instantly entropy_avail showed
non-zero values and connecting to the Imapd worked again. The problem with
this solution is, that the the entropy from /dev/urandom may not be truly
random (only pseudo random), but I will dive deeper into this topic another
time, for now it is important, that the service is functional again.
I needed aproximately 1 hour to troubleshoot and fix this problem, but most of the time, about 45 minutes, was needed to find the root of the problem, implementing the fix (or maybe it's only work-around) was rather quick.
Och nee, net auf English bloggen. Das macht keinen Spass zu lesen :/ Naja, man muss sich ja an alles gewöhnen ^^
Geschrieben von Tapedeck 1 Tag, 15 Stunden nach Veröffentlichung des Blog-Eintrags am 3. Nov. 2006, 15:16. Antworten
Keine Angst Tapedeck, die meißten Einträge werde ich weiterhin auf deutsch bloggen, aber ab und zu wird halt mal was auf englisch eingestreut ...
Geschrieben von Arne 1 Tag, 15 Stunden nach Veröffentlichung des Blog-Eintrags am 3. Nov. 2006, 15:23. Antworten
Thanks for a great article! This has been bugging me forever. Especially when running a server under vmware. So I put a link into http://www.vmware.com/community/thread.jspa?messageID=530909
So vielen Dank! Und es geht vohl ob etwas in Deutch ist, aber nicht zu viel! :D (Däne bin ich selbst)
Peter
Geschrieben von Peter Mørch 1 Monat, 1 Woche nach Veröffentlichung des Blog-Eintrags am 12. Dez. 2006, 16:07. Antworten
in debian, if you want rngd to start up like this through the /etc/init.d/* scripts, then you can append HRNGDEVICE="/dev/urandom" to your /etc/default/rnd-tools file..
thanks for this though, I think it's going to sort my problems out too :)
Geschrieben von Scott Lees 3 Monate, 3 Wochen nach Veröffentlichung des Blog-Eintrags am 21. Feb. 2007, 15:35. Antworten
Danke, hat mich gerettet (mini system, nämlich das NSLU2 "Slug" mit null entropie ;)
vg
sacha
Geschrieben von Sacha 1 Jahr nach Veröffentlichung des Blog-Eintrags am 8. Nov. 2007, 11:24. Antworten
ich habe das Problem mit
rm /dev/random
ln -s /dev/urandom /dev/random
(also ohne rngd) gelöst.
Spricht da was gegen?
Allerdings muss man die Schritte unter Debian nach einem dist-upgrade wiederholen.
Geschrieben von Theo 1 Jahr nach Veröffentlichung des Blog-Eintrags am 27. Nov. 2007, 15:14. Antworten
Linux gives you two random devices, /dev/random and /dev/urandom. The first one guarantees you a minimal entropy which can result in blocking io. The latter one is the unblocking-random-device which reduces the entropy. From a security perspective, you want to use the high entropy device for the generation of SSL keys.
Geschrieben von Benjamin Schweizer 1 Jahr, 2 Monate nach Veröffentlichung des Blog-Eintrags am 30. Jan. 2008, 16:05. Antworten
Same problem with RH 5.5 under vSphere.
It seems Oracle jdbc driver eating all entropy available.
rngd with urandom seems to alleviate symptoms.
Any further help?
Geschrieben von Mimmus 4 Jahre, 6 Monate nach Veröffentlichung des Blog-Eintrags am 17. Mai 2011, 10:40. Antworten