There is a bacterium that feeds on the beta-keratin protein- a protein which is found in the feathers of birds and also the fur of mammals. The bacteria 'feeds' by hydrolysing this protein, which makes up for most of the mass of fur/feather structures. I have only ever heard of this bacteria on the feathers of birds, but if it can be found feeding on the beta-keratin protein in the feathers then it's likely to also be found feeding on fur. Research into feathers shows that the bacteria is more prevalent in bright colours and there is a correlation between the bacteria and adverse effects on the health of a bird. So basically bright= bacteria= chance of ill health. If most mammalian colouring is quite dull, then there will be less of this bacteria feeding on the fur structure and possibly affecting the health of the mammal. Perhaps far back down the evolutionary path, when birds became brighter for whatever purposes, selective pressure forced most mammals to take a far less bright aesthetic route due to a beta-keratin protein-eating-bacteria. That's the only explanation I could think of. Or perhaps it's really simple, and that most mammals are found in grassy or wooded areas that require the fur to be brown/yellow for camouflage.