Yamato once belonged to Vergil's father, Sparda, and according to the Order of the Sword, it was used to seal the "true Hell Gate". It was eventually given to Vergil as a memento, and he used it as his primary weapon.
Vergil uses Yamato as his primary weapon while attempting to reactivate the Temen-ni-gru. When his attempt fails, Vergil and his Yamato fall into the Demon World, where he is found by Mundus. Vergil attempts to defeat the demon king like his father, but is slain and reanimated as Nelo Angelo, presumably along with his blade.
At some later point, the Yamato was found, shattered, by the Order of the Sword. It was kept in Agnus's Containment Room within Fortuna Castle, but is restored and taken by Nero when his dormant powers as an heir to Sparda emerge, and thereafter, Nero is able to use the sword while in Devil Trigger. However, it is taken back from him by Sanctus when he is absorbed into the Savior, and is then used by Agnus to reopen the Hell Gate. After slaying Agnus, Dante retrieves the Yamato from the gate seal and uses it to destroy the gate, after which he wields it while in his Dark Slayer Style. While fighting the Savior, Dante uses the blade to free Nero, and sends it back to him so that the youth can finish the fight against Sanctus. After the battle, Dante charges Nero with keeping it.