Milling machines came originally from machine tools called rotary files. They were circular cutters that worked inside of a lathe. This machine was originally developed because hand filing materials was taking too long.
The first of the milling machines was between the years 1814 and 1818. There were two armories that used the first milling machines, and they were called Springfield and Harpers Ferry. Soon after, various private factories began using these machine tools to quickly produce machined products at a rate much faster than what any number of workers using hand files could do on their own.
Many private inventors began creating these machines at the same time. The most famous of these was Eli Whitney, who is generally given the honor of having created the first "true" milling machine. Other inventors have also been credited with contributing to the process, such as Robert Johnson, Captain John H. Hall, Simeon North, Roswell Lee, Thomas Blanchard, and others from Harpers Ferry, Springfield, and other locations running machine tools in the early 19th century.
There is some controversy surrounding either Eli Whitney actually created the first milling machine itself or not. Some schools, such as Peter Banda, claim that the machine that was long credited to Whitney was not actually created until after his death.
The early part of the 19th century, such as the teens, were critical for the development of milling machines. For example, the inventor James Nasymyth made milling machines that were quite advanced because they could mill a hex nut used in indexing fixtures, even though they had six sides.
One problem that Eli Whitney's milling machines had classically, is that it had no room for the knee to move up and down during the milling process. Machine tools used by the Gay & Silver Corporation in the 1830s fixed this oversight, so making it much more comfortable for people to use it, which increased productivity.
A reason for this oversight is probably because Whitney never envisioned just how far milling machines would be used. He thought that the machine tools would be used as a way to allow workers to not hand file everything. He likely envisioned a process where workers would use the machine for the rough parts of the project, and then finish up with hand files for the rest. The idea that hand filing would be eliminated by the machines would've been a surprise.
The Lincoln miller was a very influential machine from the 1840s. It was developed by George S. Lincoln & Company who made one for the Colt armory in 1855. This miller also had the problem of having no way to position it vertically. The development of machines that had better vertical positioning would not come until the problem was noticed later on.