As with the rest of the planets in the Solar System, Mars is believed to have formed out of the primitive solar nebula 4.5 billion years ago.
Instead of condensing directly, from dust particles to planet, Mars and the rest of the terrestrial planets probably started out as a collection of small particles. Dust particles clumped together to form larger and larger objects. Dust became sand, pebbles, rocks, asteroids, and eventually planetoids. The formation of Mars happened when these particles all came together.
The energy from all these collisions heated up planet Mars, giving it a molten core and volcanic activity. We can see evidence of the end of the planetary formation period because of the impact craters strewn across the surface of the planet. This period was called the late heavy bombardment period, and all the planets in the Solar System were devastated too.
Astronomers think that Mars is relatively small because Jupiter finished its own formation a little earlier, and scooped up most of the available material. The gravity from Jupiter also seems to have prevented the formation of another planet in between Mars and Jupiter; instead, we’ve just got the asteroid belt.
Although Mars doesn’t have active plate tectonics, and its volcanism ended millions of years ago, the planet is much more similar to Earth and Venus, and different to the Moon and Mercury. Mars is the only other world in the Solar System that has a transparent atmosphere, and surface conditions that could be considered somewhat habitable.
Additional information about the history and formation of Mars. And even more information here.
Finally, if you’d like to learn more about Mars in general, we have done several podcast episodes about the Red Planet at Astronomy Cast. Episode 52: Mars, and Episode 91: The Search for Water on Mars.