Have a single CPU T440 with 96gb, 6x16gb. Looking to add more memory without adding another processor (so I don't violate server core licensing).

I see there are 10 DIMM slots allocated to Socket 1 which would provide a theoretical maximum of 160gb if all 10 slots are populated. However the service manual states to populate six slots for best performance. If I add two or four more DIMMs to get to 128gb or 160gb will there be a performance penalty for running in those memory configurations, or should I just stay at 96gb?