Modify the swinging arm ?

Due to unoriginal weels there is almost no clearence to the swinging arm. Can I cut off about 6-7 mm on each side without doing any harm ?