Try entering the BIOS by pressing the F2 key as soon as you turn on the machine, and navigate the bios to the "advanced" section.

Locate "XMP" and select the correct XMP profile for your memory.

XMP settings , page 96-97.

Note: this is under the assumption you have 4 identical memory sticks and did not mix and match brands or different speed memory sticks.

You don't need a K model to use XMP overclocking, problem is you had 2400mhz ram then added 2 sticks of 2666mhz ram, you mixed and matched ram and they have conflicting XMP profiles 

I think I figured out the reason why. Reading the fine text on the box of the ram, it says that it needs to be overclocked to go to speeds of 2400 or 2666. Unfortunately though, the overclocking is greyed out in the BIOS, which after researching, seems to be because I don't have a "K" processor. Is there any way to get around this? And if not, how much does the 2133 vs 2400 ram speed matter? Would it be worth bringing back to the store and replacing?

Ok I think I kind of understand now, but now the question is, what is the difference between 2400 and 2133? Should I return it and try to find a compatible one, or is the difference small/unnoticeable?

The difference between the two is probably negligible for most intents and purposes so I wouldn't worry about it. 

You can't enable XMP in BIOS because the profiles between the two RAM sticks be different, and only one XMP profile can be active at a time.  So the XMP option will be greyed out or if not, it will not post. 

You might have been able to try setting an OC profile in AWCC that OC'ed the RAM to a canned profile OC level 1 or 2, if you had a K class CPU.  Problem is, with the non-K you do not have the OC tab.

Although you can enable XMP regardless of locked/unlcoked CPU status, Alienware does not provide the OC tab unless you have an unlocked CPU.   The reason it makes a difference is because the changing the OC setting to level 1 or 2 via AWCC does not engage the XMP profile to OC the RAM, it uses pre-defined OC settings.  

My intent is generally just editing videos which is what I got the upgrade for, and some gaming here and there. Do you think it would really effect that?

The difference will be noticeable in games between 2133 and 2666mhz, replace the original sticks with the same sticks as the one you just bought and you'll regain your XMP ability in the bios and have 2666mhz ram like little expected

Also, the sticks have to support XMP, not all sticks do. 

It's not the overclocking feature.  Look for "XMP Memory" under advanced performance options, and there's XMP1 and XMP2.  The overclocking feature is something different which is not available with a locked CPU. 

I just took out my older sticks and I just have the new ones in. I still don't have Overclocking enabled. Are there any sticks I could get that would reach 2400 without overclocking?

In

This is the only thing under advanced performance settings in my bios.

I say you need to reset the BIOS to default settings and properly install the RAM correctly. Make sure you install the original sticks in the correct slots and then reboot. Then add the second sticks and you should be good.