Scalable Controllable Accented TTS

Human Language Technology Center of Excellence
Johns Hopkins University
Baltimore, Maryland
xli257@jhu.edu

Abstract

We tackle the challenge of scaling accented TTS systems, expanding their capabilities to include much larger amounts of training data and a wider variety of accent labels, even for accents that are poorly represented or unlabeled in traditional TTS datasets. To achieve this, we employ two strategies: 1. Accent label discovery via a speech geolocation model, which automatically infers accent labels from raw speech data without relying solely on human annotation; 2. Timbre augmentation through kNN voice conversion to increase data diversity and model robustness. These strategies are validated on CommonVoice, where we fine-tune XTTS-v2 for accented TTS with accent labels discovered or enhanced using geolocation. We demonstrate that the resulting accented TTS model not only outperforms XTTS-v2 fine-tuned on self-reported accent labels in CommonVoice, but also existing accented TTS benchmarks.

Accented TTS

Target Sentence Target Accent Ground Truth Baselines Fine-tuned XTTS
They are the most potent inhibitors
of acid secretion available.
US
AccentBox
CosyVoice 2
XTTS-v2
CommonVoice labels filtered using geolocation
CommonVoice labels, unfiltered
Labels discovered using geolocation
Smith was born in Essex and was
educated at Bancroft's School.
England
AccentBox
CosyVoice 2
XTTS-v2
CommonVoice labels filtered using geolocation
CommonVoice labels, unfiltered
Labels discovered using geolocation
The odor of spring makes
young hearts jump.
Scotland
AccentBox
CosyVoice 2
XTTS-v2
CommonVoice labels filtered using geolocation
CommonVoice labels, unfiltered
Labels discovered using geolocation
Warmley is situated in between
Bristol and Bath.
Ireland
AccentBox
CosyVoice 2
XTTS-v2
CommonVoice labels filtered using geolocation
CommonVoice labels, unfiltered
Labels discovered using geolocation
He started boxing
at age fifteen.
Australia
AccentBox
CosyVoice 2
XTTS-v2
CommonVoice labels filtered using geolocation
CommonVoice labels, unfiltered
Labels discovered using geolocation
The site was from the sea
and from London.
India
AccentBox
CosyVoice 2
XTTS-v2 (target language: Hindi)
CommonVoice labels filtered using geolocation
CommonVoice labels, unfiltered
Labels discovered using geolocation
The crabs have also been
known to take up
residence in swimming
pools.
Germany
AccentBox
CosyVoice 2
XTTS-v2 (target language: German)
CommonVoice labels filtered using geolocation
CommonVoice labels, unfiltered
Labels discovered using geolocation
Many Italian nationals
are employed as
civilian workers
at the base.
South Africa
AccentBox
CosyVoice 2
XTTS-v2 (target language: Hindi)
CommonVoice labels filtered using geolocation
CommonVoice labels, unfiltered
Labels discovered using geolocation
It is also funded through
the National Lottery,
Creative Scotland and
Northern Ireland Screen.
Philippines
AccentBox
CosyVoice 2
XTTS-v2 (target language: Hindi)
CommonVoice labels filtered using geolocation
CommonVoice labels, unfiltered
Labels discovered using geolocation
These speculations began
when the British discovered
the full scope of the treaty.
Malaysia
AccentBox
CosyVoice 2
XTTS-v2 (target language: Hindi)
CommonVoice labels filtered using geolocation
CommonVoice labels, unfiltered
Labels discovered using geolocation