How To Keep Baby Milk Warm When Traveling. Placing these small baby bottles inside travel mugs and filling them with just the right amount of hot water makes a great and inexpensive solution for keeping milk hot when traveling outside your home. Milk bottles should not be kept warm all day, so store breast milk in an iced bag or at room temperature. Store hot water in thermos. Place milk bottle in thermos when baby is ready to eat.