Infrared spectroscopy (IR) is the most widely used analytical tool to quantify trace water in silicate and silica minerals. A prerequisite for highly accurate IR measurements of trace water is a good understanding of the effect of the second Si–O vibrational overtones/combination bands (2nd Si–O VOCBs) on the water peaks. Silicate and silica minerals can be divided as isolated (Q0), paired (Q1), ring (Q2), chain (Q1 or Q2), sheet (Q3) and framework (Q4) structures according to the polymerization of their SiO4 tetrahedral units, and the 2nd Si–O VOCBs of these different structural types attain different vibrational features which are expected to affect the water peaks to different extents. Here, we selected olivine (Q0) and α-quartz (Q4) as two endmember-like structural examples, performed extensive IR measurements on both pristine and heat-treated thin sections prepared for these two minerals, and explored the vibrational features of the 2nd Si–O VOCBs. We have found that the 2nd Si–O VOCBs are well separated from the water peaks in olivine, but severely overlap with the water peaks in α-quartz, confirming the different roles that the 2nd Si–O VOCBs play in quantifying trace water in silicate and silica minerals with different structural polymerizations. To remove the influence of the 2nd Si–O VOCBs (or any other species rather than water), an experimental protocol has been successfully developed, as approved by some fundamental equations and verified by the data of α-quartz in the literature. This development should lead to significant accuracy improvement in quantifying trace amounts of water in Earth and planetary materials.