We do not know the date of construction of the primitive hermitage, but due to its dedication to San Sebastián, protector against diseases and epidemics, and its location outside the walls of the town, it is thought that it may date from the Fernandine period or from the last third of the 14th century, shortly after the arrival of the Black Death in the town. It served as the town’s parish church until the mid-17th century, when the current church was renovated and gradually led to the abandonment of this building, which had a cemetery annexed to it. We can highlight its Baroque lintelled façade with a pediment divided by a niche flanked by volutes, with two lateral pinnacles and a central one, and a belfry crowning the whole.

After various historical vicissitudes and being used as a school at the beginning of the 20th century, as a prison after the Civil War, and as a cereal warehouse in later years, in 1973 it underwent its last exterior renovation in which it acquired its current image, and in 1992 the last interior renovation. Nowadays, it shares the name of the church with Our Mrs. Rosary, patron saint of the town, whose image it houses.