Welcome to the forum.
From the motherboard memory specifications that you linked to:
*Due to the operating system limitation, the actual memory size may be less than 4GB for the reservation for system usage under Windows® 32-bit OS. For Windows® 64-bit OS with 64-bit CPU, there is no such limitation.
**The highest supported DRAM frequency depends on the CPU you’ve installed. For detailed CPU specifications, please refer to Intel® official website.
With a windows 32-bit operating system, less than 4GB of memory will be recognized.
With a windows 64-bit operating system, the H77M motherboard can recognize up to 16GB of installed memory.
To avoid memory compatibility problems, only buy a memory kit from the ASRock memory support list or a memory kit listed on the memory manufacturer's website, where they have tested the specific kit with your motherboard.
Some memory manufacturers make a blanket statement where they say that a memory kit is designed to work with all Sandy Bridge or other cpu systems. Don't believe it!!