You can use .htaccess or server directives to deny access to favicon.ico, but the server will send an access denied reply to the browser and this still slows page access.
You can stop the browser requesting favicon.ico when a user returns to your site, by getting it to stay in the browser cache.
First, provide a small favicon.ico image, could be blank, but as small as possible. I made a black and white one under 200 bytes. Then, using .htaccess or server directives, set the file Expires header a month or two in the future. When the same user comes back to your site it will be loaded from the browser cache and no request will go to your site. No more 404's in the server logs too.
If you have control over a complete Apache server or maybe a virtual server you can do this:-
If the server document root is say /var/www/html then add this to /etc/httpd/conf/httpd.conf:-
Alias /favicon.ico "/var/www/html/favicon.ico"
ExpiresDefault "access plus 1 month"
Then a single favicon.ico will work for all the virtual hosted sites since you are aliasing it. It will be drawn from the browser cache for a month after the users visit.
For .htaccess this is reported to work (not checked by me):-
AddType image/x-icon .ico
ExpiresByType image/x-icon "access plus 1 month"