Using your DVOM, make sure the test leads are plugged into the right connections (usually volts/ ohms) and put the red to the + terminal of the battery, the black to the negative.
Depending on temperature a good battery will generally be in the range to 12.8 - 13.2 volts. However, just because the voltage looks OK does not mean the battery is good. It could be sulfated, dead cell, etc... and still have OK looking voltage.
The only good way to test is to see how the voltage drops under high current draw conditions. If you do not have a carbon pile tester, the auto parts store will be able to do it for you. They draw a bunch of current through the tester and watch how the voltage drops off. Usually the tester is automated so that the user can't mess it up. Best to have your batteries fully charged before you have them tested.