Lipomatous tumors are among the most common soft tissue tumors (STTs). Magnetic resonance imaging (MRI) is a state-of-the-art diagnostic tool used to differentiate and characterize STTs. Radiological misjudgment can lead to incorrect treatment. This was a single-center retrospective study. Two hundred and forty lipomatous tumors were included. MRI diagnoses were categorized as benign, intermediate, or malignant and were compared with histological diagnoses. Tumor volumes were measured by MRI and from surgical specimens. The tumor was correctly categorized 73.3% of the time. A total of 21.7% of tumors were categorized as more malignant in MRI reports than they were by histology, and vice versa for 5.0% of tumors. Volume measured by MRI was not different from actual tumor size in pathology. Atypical lipomatous tumors (ALTs) and liposarcomas (LPSs) were larger when compared with lipomata and occurred in older patients. Based on the MRI-suspected tumor entity, surgical treatment can be planned. Large lipomatous tumors in elderly patients are more likely to be ALTs. However, a safe threshold size or volume for ALTs cannot be determined.