cold enough to freeze the balls off a brass monkey

The story

Unknown. There is a discredited theory that it refers to a brass container for cannon balls on a British man-of-war, and a somewhat more plausible theory that "brass monkey" used to refer to a cannon on a man-of-war.