How I got rid of the dreaded 'Too many open files' ERROR.

One of the most frustrating thing about continuous integration namely Jenkins, is getting to work in the morning with a happy smile expecting a successful Jenkins build and being faced with the dreaded errors/ or system failures. Not forgetting that these builds take about 3-5 hours to run and run on actual hardware and in real-time/

Going through the console log, you find yourself faced with....

Your build failed because of too many opened files, bla bla blah ...

After hours of gruesome Googling, I found a solution...

On Ubuntu at least the open files limit for normal users defaults to 1024.  So, if you are going to have more than ~1000 simultaneous connections you need to bump this up.

Solution: Raise open file descriptor limits

Here's how to do it on Ubuntu

Open and add the following line.
sudo vim /etc/security/limits.conf

* soft nofile 16384
* hard nofile 16384
The first column describes WHO the limit is to apply for. * is a wildcard, meaning all users. To raise the limits for root, you have to explicitly enter 'root' instead of '*'.

You also need to edit sudo nano /etc/pam.d/common-session* and add the following line to the end:

session required pam_limits.so

logout and then back in before you can use the new max limit, test with

ulimit -n

Done, No more yada yada Too many open files.