The probe will be centred on "a former student of the pre-seminary Saint Pius X", who later became a priest, according to a statement.
"Following several reports, anonymous or not, investigations were conducted from 2013 on several occasions, both by the superiors of the pre-seminary and the Bishop of Como (north)", the diocese where the teachers are attached, the statement said.
The allegations concern pupils, "some of whom were no longer at the institution at the time of the investigations", adds the Holy See.
The opening of a new investigation follows the "recent appearance of new elements".
In a book published a few days ago, "The Original Sin", Italian journalist Gianluigi Nuzzi discusses the allegations.
The book alleges that in a Vatican palace, a seminarist sexually abused at least one high school student aged 17 or 18 in 2011-2012.
A witness, Kamil Tadeusz Jarzembowksi from Poland, was living in an institution in the Vatican City, where children and adolescents from around the world plan to become priests.
According to him, a former pupil who was allowed to stay in the palace came very often at night - up to 140 times - to have sex with his roommate.
The boarders attend a private school in the centre of Rome and participate as choir boys at Masses celebrated in St. Peter's Basilica.