After Napoleon's fall in 1815, his uncle, Cardinal Fesch, Archbishop of Lyon, fled from France and sought refuge in Rome. He was replaced by Monseigneur Gaston de Pins who ruled the diocese. A man with strong ties to the Tarn area, being born in Montségou-sur-l'Agout, a village near Brassac. He owned a country house in the nearby village of Ouillats. During this time the order of the Sisters of Saint-Joseph, founded in Puy-en-Velay in the 17th Century, expanded and sent out their Sisters into the communities. Monseigneur de Pins invited the Sisters, 5 of whom arrived at Ouillats on 14th August 1824.With the permission of the Archbishop of Albi they founded the Order as it still exists today.

Throughout the years the Sisters at Couvent d'Oulias gained a reputation as teachers forming a domestic science school and boarding school for girls. Couvent d'Oulias was also greatly respected in the region for its old people's home nursing and visiting the sick of the area. The last Sisters moved to surrounding convents in 1992. you can visit every wednesday morning.