The behaviour changed in 5.6.12. For more details see this bug
report (which I consider fixed):
http://bugs.mysql.com/bug.php?id=62224
There is also a small patch attached to that bug which changes the behaviour and sets the limit to the hard limit instead of the soft limit if setting the requested limit fails.
And for troubleshooting something like this should be used
instead the more classical ‘switch to the user and do a ulimit
-a’. With cgroups in linux the limit for the mysql user and the
mysql process might be different.
egrep ‘^(Limit|Max open files)’ /proc/`pidof mysqld`/limits
And this one is also a classic:
http://www.mysqlperformanceblog.com/2013/02/04/cant_create_thread_errno_11/