Benchmarking Modern Text-To-Speech Models: A Comprehensive Evaluation of Accuracy, Naturalness, and Efficiency

(Paper Currently in Review)

ABSTRACT

This study introduces a robust, quantitative framework for benchmarking Text-to-Speech (TTS) models, systematically evaluating their performance across three critical dimensions: intelligibility, expressiveness, and computational efficiency. Conventional TTS assessments predominantly depend on subjective Mean Opinion Score (MOS) studies, which, despite capturing human perceptual insights, suffer from inconsistent reproducibility and scalability due to listener variability. To overcome these limitations, we evaluate nine state-of-the-art TTS models, including Tacotron2 variants, FastPitch, and VITS, using a comprehensive suite of objective metrics: Word Error Rate (WER), and Character Error Rate (CER) for intelligibility, Signal-to-Noise Ratio (SNR) and pitch variance for expressiveness, and processing latency for efficiency. Leveraging a standardized dataset of 58 carefully curated prompts from the LibriVox corpus, we generate 522 diverse speech samples, which are analyzed through automated transcription with Whisper ASR and advanced spectral processing techniques. Our results expose significant trade-offs among accuracy, naturalness, and computational cost, highlighting that model performance is highly context-dependent rather than universally optimal. For instance, high-efficiency models like FastPitch excel in speed but compromise on expressive nuance, while models like Neural-HMM prioritize naturalness at the expense of processing time. This framework provides a holistic, data-driven comparison of modern TTS systems. It establishes a scalable and reproducible methodology, paving the way for standardized, future advancements in TTS evaluation and development.

  1. INTRODUCTION

The Voder being demonstrated at the New York World's Fair

Text-to-speech (TTS) synthesis is a cornerstone of modern artificial intelligence, bridging the gap between textual information and auditory communication. Since its inception with devices such as the Voder in 1939 [1, 2], TTS has evolved into a critical technology underpinning applications in accessibility, education, entertainment, and human computer interaction [3]. Today, advancements in generative AI have propelled TTS systems into diverse industries, enabling tools such as virtual assistants, audiobooks, and real-time translation services. This domainโ€™s importance is enhancing user experiences by delivering intelligible, expressive, and efficient speech synthesis, fostering seamless technology integration into daily life. As TTS continues to shape how we interact with machines, its development demands rigorous evaluation to ensure it meets the needs of accuracy, naturalness, and computational performance [4, 5].

Despite these advancements, a persistent challenge in TTS research is the lack of standardized, reproducible methods to evaluate model performance across these critical dimensions [6]. Traditional assessments often rely on subjective measures like Mean Opinion Scores (MOS), which, while valuable for capturing human perception, suffer from variability due to listener bias and lack of scalability [7]. Objective metrics such as Word Error Rate (WER) and Character Error Rate (CER) provide insights into accuracy, yet they fail to fully address naturalness and efficiency attributes equally vital for real-world deployment. This problem is compounded by the diversity of modern TTS architectures, from concatenative approaches like unit selection synthesis [8] to statistical parametric synthesis using Hidden Markov Models (HMMs) [9], each exhibiting unique trade-offs. Consequently, the field lacks a cohesive framework to compare state-of-the-art models under uniform conditions systematically.

The importance of addressing this evaluation challenge cannot be overstated, as it directly impacts TTS systemsโ€™ practical utility and scientific advancement. Accurate speech synthesis ensures intelligibility, which is crucial for applications like assistive technologies, where misinterpretations can hinder communication. Naturalness, expressiveness, and tonal richness enhance user engagement, making synthesized speech indistinguishable from human voices, a goal with profound implications for entertainment and education [10]. Efficiency, measured by processing latency, determines feasibility in real-time scenarios such as conversational agents, where delays disrupt interaction flow. Developers and researchers struggle to identify optimal models for specific use cases without a robust benchmarking methodology, slowing progress and adoption across industries.

Existing work in TTS evaluation has made significant strides but remains fragmented. Early systems such as MITalk [11] introduced automated synthesis, while unit selection synthesis improved naturalness by leveraging significant speech corpora [8]. More recent approaches, such as statistical parametric synthesis [12] and neural network-based models like Tacotron2 [7], have pushed the boundaries of quality and flexibility. Studies have employed metrics such as WER, CER, and signal-to-noise ratio to assess accuracy and clarity [13, 14]. At the same time, spectral flatness and pitch variance have been used to gauge audio quality and expressiveness [15, 16]. However, these efforts often focus on isolated aspects of performance, with subjective evaluations dominating naturalness assessments and efficiency rarely integrated into comprehensive analyses.

This fragmentation reveals a critical gap: the absence of a unified, objective framework that simultaneously evaluates accuracy, naturalness, and efficiency across diverse TTS architectures. While individual studies highlight strengths, e.g., FastPitchโ€™s speed [8, 17] or VITSโ€™s end-to-end quality [18], they rarely provide a holistic comparison grounded in standardized datasets and metrics. This leaves researchers unable to discern trade-offs systematically, such as the inverse relationship between processing speed and expressiveness observed in your results, or to replicate findings due to inconsistent evaluation protocols. The opportunity lies in developing a scalable, reproducible methodology that bridges these dimensions, offering a more straightforward path to optimize TTS systems for varied applications.

MITalkโ€™s ATN diagram for noun groups

To address this gap, this study introduces a quantitative benchmarking framework that evaluates nine state-of-the-art TTS models using a curated subset of the LibriVox dataset [19]. By synthesizing 522 speech samples from 58 diverse prompts, we assess performance across intelligibility (WER, CER), expressiveness (pitch variance, Harmonics-to-Noise Ratio), audio quality (SNR, spectral flatness), and efficiency (processing time). Leveraging automated transcription via Whisper ASR and spectral analysis, our approach ensures objectivity and repeatability. This framework reveals model-specific trade-offsโ€”e.g., FastPitchโ€™s speed versus Neural-HMMโ€™s naturalness. It establishes a foundation for future TTS research to build upon, aligning evaluation with real-world demands.

The main contributions of this paper are as follows:

  1. A benchmarking framework evaluating TTS models across accuracy, naturalness, and efficiency using objective metrics.

  2. A comprehensive analysis of nine modern TTS architectures, including Tacotron2 variants, FastPitch, and VITS, based on 522 synthesized samples.

  3. Identification of three performance categories (high-, moderate-, and low-efficiency) emphasizing trade-offs between synthesis speed and speech quality

  4. Quantification of each modelโ€™s five key metrics (WER, CER, SNR, pitch variance, and processing time) and three statistics (amplitude variance, RMS loudness, and spectral flatness).

  5. A standardized dataset curation process using 58 prompts from LibriVox, spanning emotional, technical, and conversational styles.

  6. Reproducible methodology integrating Whisper ASR and spectral processing, enabling future comparative studies.

The remainder of this paper is organized as follows: Section II reviews related work on the evolution of text-to-speech systems and prior evaluation approaches. Section III describes the methodology, including dataset curation, model selection, and performance metrics. Section IV presents experimental results, detailing the benchmarking outcomes across nine TTS models and their performance categories. Section V discusses the findings, analyzing top-performing models and observed trade-offs. Finally, Section VI concludes the paper and discusses future research directions.

2. BACKGROUND AND RELATED WORK

The journey of speech synthesis began with early innovations like the Voder, introduced in 1939 by Bell Telephone Laboratory, which marked a pioneering step in mimicking human speech [19]. The Voder replicated the human vocal tractโ€™s characteristics using a wrist bar to produce a buzzing sound (driven by a relaxation oscillator), ten keys to adjust bandpass filter gains, and a foot pedal to control pitch. Operators played it like an instrument, synchronizing these components to articulate words. However, its dependence on skilled technicians to infuse emotion and tone and a limited sound repertoire restricted its practicality and naturalness [19]. This section traces the evolution of Text-to-Speech technology from such rudimentary systems to modern statistical approaches, providing context for our benchmarking study by reviewing key milestones and methodologies that have shaped the field.

2.1 Early Speech Synthesis Techniques

Advancements beyond the Voderโ€™s manual control emerged with Concatenative Speech Synthesis (CSS), introduced in the 1980s, which automated speech generation using pre-recorded segments [7]. Unlike the Voder, CSS extracted unitsโ€”phonemes, syllables, or wordsโ€”from a speech corpus and stitched them together to form fluid, coherent output, improving efficiency and naturalness. Diphone synthesis, an early CSS technique, focused on diphones, the transitions between adjacent phonemes, to ensure smooth phonetic transitions vital for intelligibility [20]. By joining these pairs, it produced clear speech, though its reliance on limited recorded segments often yielded mechanical, less natural results [21]. A significant milestone in diphone synthesis was MITalk, developed in the late 1970s as the first primary text-to-speech synthesizer of its kind [11]. MITalk automated synthesis through five modules: FORMAT preprocessed text, segmenting it and marking boundaries, stress, and punctuation; DECOMP broke words into morphemes using a recursive algorithm and a 12,000-entry dictionary; PARSER identified grammatical structures and calculated rhythmic features like timing and pauses; SOUND2 applied 400 letter-to-sound rules for unanalyzed words; and PHONET adjusted pitch, loudness, and duration for realistic intonation [10]. Despite its progress, MITalkโ€™s diphone-based approach retained limitations, producing robotic tones due to pre-recorded segment constraints [22, 23].

2.2 Advanced Synthesis Methods

These shortcomings spurred the development of Unit Selection Synthesis, which built on CSS principles but used larger unitsโ€”often entire words or phrasesโ€”for more fluid, natural speech [8]. Drawing from extensive speech databases (sometimes hours of recordings), it combined complete segments into sentences, guided by target cost (matching unit attributes like pitch and timbre to the desired sound) and concatenation cost (ensuring smooth transitions without gaps or pitch shifts) [13, 24, 25]. While offering superior naturalness over diphone synthesis, unit selection faced challenges: its large databases demanded significant storage and processing power, and output quality hinged on recording consistency, faltering with noise or limited data [26โ€“28]. Statistical Parametric Synthesis (SPS) emerged in the early 2000s to address these issues, shifting from extensive recordings to statistical models for greater efficiency and flexibility [12]. SPS analyzed acoustic featuresโ€”pitch, duration, and tonal qualitiesโ€”constructing models like Hidden Markov Models (HMMs) trained on smaller datasets to predict and generate speech [9, 18]. This reduced dependency on curated corpora, enhancing cost-effectiveness [29, 30].

A notable SPS implementation, HMM-based synthesis, mapped text to parameters (frequency, amplitude, duration) using statistical estimates, requiring less storage than unit selection and adapting to varied styles by tweaking model parameters [15, 31]. HMMs modeled speech as state sequences (e.g., phonemes or diphones), estimating transition probabilities and acoustic feature distributions (pitch, energy) for smoother sound transitions. However, their reliance on simplified parameters often missed the subtle complexities of human speech, resulting in less natural flow [32]. These historical and modern approachesโ€”from the Voderโ€™s manual operation to HMMโ€™s statistical efficiencyโ€”highlight the fieldโ€™s progression and the persistent challenge of balancing quality, efficiency, and naturalness, setting the stage for our evaluation of contemporary TTS models.

Tacotron 2 Audio Samples (taken from Tacotron2 GitHub) (Neural Networks)

  • โ€œHe has read the whole thing.โ€

  • โ€œHe reads books.โ€

Tacotron or Human?

  • โ€œShe earned a doctorate in sociology at Columbia University.โ€

3. METHODOLOGY

This section will describe the dataset, models, performance metrics, and evaluation criteria.

3.1 Dataset Description

The dataset used in this study is a publicly available collection of 13,100 short audio clips. These clips feature a single speaker reading passages from seven non-fiction books. Each clip includes a corresponding transcription and varies in length from 1 to 10 seconds, resulting in approximately 24 hours of recorded audio. The texts, originally published between 1884 and 1964, are in the public domain, and the audio recordings, created in 2016โ€“2017 as part of the LibriVox project, are also publicly available. The dataset can be found at LibriVox. A subset of this dataset was curated to evaluate TTS model performance systematically. This subset includes sentences of different lengths and styles designed to address different aspects of speech synthesis. The selection includes short, concise phrases that assess basic articulation and clarity and longer, more complex sentences that test syntactic processing and contextual understanding. The dataset also includes a variety of categories, such as emotional language that calls for expressive delivery, technical content that requires precision and clarity, and conversational tones that emphasize naturalness. Treating all sentences as part of a unified dataset ensures a balanced evaluation, and performance metrics are averaged across all categories.

3.2 Models

Following dataset curation, we comprehensively analyzed nine state-of-the-art Text-to-Speech models to evaluate their architectural characteristics and performance capabilities. The independent variable in this study is the TTS model architecture, defined as the distinct computational framework each model employs to convert text into synthesized speech. These architectures leverage a spectrum of techniquesโ€”ranging from spectral modeling and neural networks to transformer-based methodsโ€”each underpinned by specialized algorithms designed to optimize the generation of naturalsounding speech. This diversity results in varying strengths and weaknesses, such as trade-offs between synthesis speed, phonetic accuracy, and expressive quality, as observed in prior studies [7, 8, 18]. To provide a structured overview, Table 1 details each of the nine modelsโ€”Tacotron2-DDC, Tacotron2-DDC_ph, Tacotron2-DDC_DCA, Speedy-Speech, Glow-TTS, Overflow, Neural-HMM, FastPitch, and VITSโ€”highlighting their names, core architectural features, and primary design objectives.

The selected models represent a broad cross-section of contemporary TTS approaches, enabling a robust comparative analysis. For instance, Tacotron2 variants (e.g., Tacotron2-DDC, Tacotron2-DDC_ph) emphasize spectral modeling with convolutional and LSTM components, augmented by vocoders to enhance output quality, while Tacotron2-DDC_DCA introduces Dynamic Convolutional Attention to address long-term dependencies [7]. Speedy-Speech and FastPitch leverage transformer-based encoders for rapid, non-autoregressive synthesis, prioritizing efficiency without sacrificing naturalness [8]. Glow-TTS adopts a flow-based generative approach with monotonic alignment search, balancing quality and computational simplicity, whereas Overflow utilizes a neural transducer framework for flexible prosody modeling. Neural-HMM integrates statistical Hidden Markov Models with neural networks to improve clarity and naturalness [9], and VITS unifies encoder and vocoder training in an end-to-end system for seamless, expressive output [18]. This architectural diversity, detailed in Table 1, underpins our evaluation across intelligibility, expressiveness, and efficiency, as elaborated in subsequent sections.

3.3 Performance Metrics and Evaluation Criteria

This study evaluates Text-to-Speech models across five essential dimensionsโ€”accuracy, audio quality, clarity, expressiveness, and efficiencyโ€”to capture their multifaceted performance in generating human-like speech. These dimensions address distinct yet interconnected aspects of TTS systems, from ensuring textual fidelity to delivering perceptually pleasing and computationally viable outputs. To achieve a robust and objective comparison, we consistently apply a suite of specific metrics across 522 synthesized samples derived from our curated LibriVox dataset [19]. The following subsections detail each dimensionโ€™s evaluation criteria, outlining the metrics employed, their theoretical underpinnings, and their application to diverse linguistic contexts, thereby establishing a comprehensive framework for benchmarking modern TTS architectures.

3.3.1 Accuracy. Accuracy quantifies the fidelity with which synthesized speech reproduces the intended textual input, reflecting a modelโ€™s ability to produce correct words, phrases, and pronunciations without errors or omissions. We utilize two primary metrics: Word Error Rate and Character Error Rate. WER measures the proportion of substituted, inserted or deleted words in the transcribed output relative to the reference text, serving as a robust indicator of semantic integrity [13]. In contrast, CER assesses errors at the character level, offering finer granularity to detect subtle mispronunciations or structural inaccuracies, particularly in complex sentences. These metrics are applied across the datasetโ€™s varied styles from simple phrases to intricate technical contentโ€”and validate each modelโ€™s precision and coherence, ensuring reliable speech synthesis across diverse scenarios.

3.3.2 Audio Quality. Building on accuracy, audio quality evaluates the perceptual attributes of synthesized speech, focusing on its smoothness, richness, and freedom from artifactsโ€”qualities essential for an authentic and pleasing auditory experience. We employ Spectral Flatness as the primary metric, which analyzes energy distribution across the frequency spectrum to distinguish harmonic, tone-like sounds from noise-like distortions [15]. Lower spectral flatness values signify richer, more natural audio, whereas higher values indicate synthetic degradation or noise. This metric is tested across emotional, technical, and conversational samples, enabling us to assess each modelโ€™s capacity to maintain high-quality output under the diverse synthesis demands presented by our dataset.

3.3.3 Clarity. Clarity extends the evaluation to intelligibility, assessing how easily synthesized speech can be understood, especially in challenging contexts such as technical narrations or complex queries where precise articulation is critical. We measure clarity using the Harmonics-to-Noise Ratio (HNR) and Signal-to-Noise Ratio. HNR quantifies the proportion of harmonic components relative to background noise, reflecting vocal smoothness and quality. At the same time, SNR evaluates the strength of the desired speech signal against unwanted noise, ensuring audibility [9]. These metrics are particularly relevant for prompts requiring technical precision or nuanced articulation, guaranteeing that each model delivers comprehensible speech across a range of practical use cases.

3.3.4 Expressiveness. Expressiveness shifts the focus to TTS systemsโ€™ emotive and prosodic capabilities, gauging their ability to convey intonation, emotion, and naturalness that render speech engaging and human-like. We use pitch variance as the key metric, capturing dynamic shifts in intonation and stress that enhance perceptual naturalness [15]. By analyzing pitch dynamics across the datasetโ€™s emotional exclamations, conversational exchanges, and expressive narratives, we determine how effectively each model adapts its output to contextually appropriate tonal variations. This dimension is crucial for applications where user engagement hinges on the synthesized speechโ€™s lifelike quality.

3.3.5 Efficiency. Finally, efficiency addresses the computational practicality of TTS systems, a vital factor for real-time applications like virtual assistants or large-scale audiobook production. We measure efficiency primarily through Time to Run, the duration required to synthesize audio from text across prompts of varying lengths. This is complemented by Root Mean Square (RMS) analysis of signal energy distribution, which correlates with processing load. Grounded in prior efficiency studies [8], these metrics enable us to assess each modelโ€™s suitability for deployment under time-sensitive constraints, balancing speed with resource demands to meet diverse operational needs.

4. RESULTS AND ANALYSIS

This study rigorously evaluates the performance of nine contemporary Text-to-Speech modelsโ€”Fast_Pitch, Glow_TTS, Speedy-Speech, Neural_HMM, Overflow, Tacotron2-DCA, Tacotron2-DDC, Tacotron2-DDC_ph, and VITSโ€”across a comprehensive set of metrics: processing time, pitch accuracy, amplitude variance, Root Mean Square loudness, spectral flatness, Word Error Rate, Character Error Rate, Signal-to-Noise Ratio, and Harmonics-to-Noise Ratio. These metrics collectively assess computational efficiency, perceptual quality, and linguistic accuracy, benchmarked against a human speech reference to measure naturalness and expressiveness. Conducted under controlled conditions with identical input data, our analysis illuminates distinct performance profiles, categorizing models into high-efficiency (Fast_Pitch, Glow_TTS, Speedy-Speech), moderate-efficiency (Neural_HMM, Overflow, Tacotron2-DCA, Tacotron2-DDC_ph), and low-efficiency (Tacotron2-DDC) groups based on synthesis speed and sound quality. The following subsections present detailed results for each metric, their computation methods, and performance rankings, culminating in an analysis of observed trade-offs.

4.1 Processing Time

Processing time is defined as the duration from text input to audio output, quantifies synthesis efficiency in seconds, calculated as the difference between start and end times. Figure 1 illustrates the distribution across models, with Fast_Pitch, Glow_TTS, and Speedy-Speech exhibiting tight interquartile ranges (IQR) and low medians, averaging 0.28, 0.32, and 0.46 seconds, respectively (Figure 2). These top performers excel in real-time applications where minimal delays are paramount. Moderate-efficiency modelsโ€”Neural_HMM, Overflow, Tacotron2-DCA, and Tacotron2-DDC_phโ€”range from 1.1 to 1.8 seconds, balancing speed and reliability, while Tacotron2-DDC, averaging 4.1 seconds with outliers exceeding 40 seconds (median 2.7 seconds), ranks lowest due to inconsistent performance.

4.2 Pitch Accuracy and Variability

Pitch is the perceived frequency of sound, reflects intonation accuracy and expressiveness, computed via Short-Time Fourier Transform (STFT) on overlapping audio frames (Figure 3). Figure 4 compares average pitch (blue bars) and variability (orange bars) to the human reference (469.481 mean, 279.293 standard deviation). Most models approximate the mean, with Tacotron2-DCA, Tacotron2-DDC, and Speedy-Speech lower (414.889โ€“434.914), indicating reduced fidelity. Variability favors Tacotron2-DDC_ph and Neural_HMM (314.763, 297.920), marking them as top performers in expressiveness, though most models fall below the human standard, suggesting limited dynamic range.

4.3 Amplitude Variance

Amplitude variance, assessing loudness fluctuations, uses frame-adjusted variance (Figure 5). Overflow and Glow_TTS lead with medians of 0.053 and 0.054 (vs. human 0.023), suggesting robust expressiveness, while Fast_Pitch, Neural_HMM, and Speedy-Speech (0.044โ€“0.046) exhibit narrower ranges, leaning toward monotony. All models exceed the human benchmark, implying less natural dynamics overall.

4.4 RMS Loudness

RMS loudness, averaging amplitude over time (Figure 6), shows all models (0.075โ€“0.085) surpassing the human median (0.05), with Neural_HMM closest (Figure 7). Overflow and VITS maintain consistency, while Speedy-Speech and Tacotron2-DCA vary more, indicating uniform loudness elevation across models.

4.5 Spectral Flatness

Spectral flatness, distinguishing harmonic tones from noise (Figure 8), reveals all models exceeding the human reference. Fast_Pitch, Neural_HMM, and VITS exhibit lower values (more harmonic), outperforming Speedy-Speech and Tacotron2- DDC_ph (noisier), though variability and outliers suggest tonal inconsistency.

4.6 Word and Character Error Rates

WER and CER, calculated as Error Rate = (๐‘†+๐ผ+๐ท)/๐‘ (substitutions ๐‘†, insertions ๐ผ, deletions ๐ท, total words/characters ๐‘) via Whisper ASR with normalized transcriptions, assess linguistic accuracy (Figures 9, 10). VITS, Fast_Pitch, and Overflow top WER (0.05โ€“0.10) and CER (<0.05), Glow_TTS and Tacotron2 variants moderate (0.10โ€“0.15, 0.05โ€“0.07), and Neural_HMM, Speedy-Speech lag (0.22โ€“0.27, >0.10), reflecting phonetic precision disparities.

4.6 Word and Character Error Rates

WER and CER, calculated as Error Rate = (๐‘†+๐ผ+๐ท)/๐‘ (substitutions ๐‘†, insertions ๐ผ, deletions ๐ท, total words/characters ๐‘) via Whisper ASR with normalized transcriptions, assess linguistic accuracy (Figures 9, 10). VITS, Fast_Pitch, and Overflow top WER (0.05โ€“0.10) and CER (<0.05), Glow_TTS and Tacotron2 variants moderate (0.10โ€“0.15, 0.05โ€“0.07), and Neural_HMM, Speedy-Speech lag (0.22โ€“0.27, >0.10), reflecting phonetic precision disparities.

SNR, computed via signal power (๐‘›๐‘.๐‘ ๐‘ข๐‘š(๐‘ ๐‘–๐‘”๐‘›๐‘Ž๐‘™2 )) and noise differences after standardization and alignment (Figure 11), and HNR, via STFT and HPSS (Figure 11), assess clarity (Figures 12, 13). VITS, Fast_Pitch, and Overflow excel in both, while Neural_HMM ranks lower, highlighting noise suppression strengths.

4.8 Performance Analysis and Trade-offs

The evaluation of nine TTS models across a diverse set of metrics unveils a landscape of strengths, reflecting inherent trade-offs in design priorities. High-efficiency modelsโ€”Fast_Pitch, Glow_TTS, and Speedy-Speechโ€”consistently deliver rapid synthesis, with processing times as low as 0.28โ€“0.46 seconds, making them prime candidates for real-time applications like virtual assistants. However, this speed comes at a cost: their reduced pitch variability (below the human standard of 279.293) and narrower amplitude range (medians 0.044โ€“0.046 vs. human 0.023) suggest a sacrifice in expressiveness, producing speech that, while clear and audible (RMS 0.075โ€“0.085), lacks the dynamic richness of human intonation. In contrast, the low-efficiency Tacotron2-DDC, despite its sluggish 4.1-second average (median 2.7 seconds), excels at replicating human-like patterns, particularly in pitch accuracy and spectral flatness, though its variability and outliers indicate inconsistent performance.

Moderate-efficiency modelsโ€”Neural_HMM, Overflow, Tacotron2-DCA, and Tacotron2-DDC_phโ€”strike a balance, averaging 1.1โ€“1.8 seconds, offering a middle ground between speed and quality. Neural_HMM stands out here, leading in expressiveness with high pitch variability (297.920), SNR, and HNR, closely mimicking human nuances, yet it falters in linguistic accuracy, with WER (0.22โ€“0.27) and CER (>0.10) trailing behind. Conversely, VITS and Overflow shine in precision, topping WER (0.05โ€“0.10) and CER (<0.05) alongside strong SNR and HNR values, reflecting robust phonetic articulation and clarity. However, their dynamic range (amplitude variance, pitch variability) remains constrained, hinting at a design focus on accuracy over emotive depth. Spectral flatness further complicates the picture: all models exceed the human reference, with Fast_Pitch, Neural_HMM, and VITS leaning toward harmonic richness, while Speedy-Speech and Tacotron2-DDC_ph veer noisier, underscoring a universal challenge in matching natural tonality.

This interplay of metrics highlights a critical trade-off: efficiency-driven models prioritize speed at the expense of naturalness, whereas quality-focused ones incur time penalties. RMS loudness, uniformly elevated across models (0.075โ€“0.085 vs. human 0.05), suggests a tendency toward exaggerated audibility, with Neural_HMMโ€™s proximity to human levels a rare exception. No single model emerges as universally superior; instead, context dictates suitabilityโ€”Fast_Pitch for speed-critical scenarios, VITS for accuracy-dependent tasks, and Neural_HMM where expressiveness trumps precision. These findings invite reflection on TTS design: balancing these dimensions may require hybrid approaches, blending the strengths of high-efficiency architectures with the expressive fidelity of slower systems.

4.7 SNR and HNR

5. CONCLUSION

This study evaluates nine TTS models across five metrics (processing time, pitch, WER, CER, SNR) and three statistics (amplitude variance, RMS, spectral flatness), comparing waveform-based metrics to human speech and ranking performance-based ones. Fast_Pitch excels in speed (0.49 seconds) and ranks high in most metrics except pitch, balancing efficiency, and quality. VITS and Overflow lead in WER (0.29) and CER (0.03), prioritizing accuracy, while Neural_HMM shines in expressiveness (pitch, amplitude, SNR) but falters in accuracy (WER 0.5, CER 0.23). A key trade-off emerges: accuracy-focused models (VITS, Overflow) sacrifice naturalness, while expressive ones (Neural_HMM) compromise precision. Future research into hybrid architectures, self-supervised, and multi-task learning could bridge this gap, enhancing scalability and real-world applicability.

REFERENCES

[1] Masanori Morise, โ€˜ Fumiya Yokomori, and Kenji Ozawa. WORLD: A Vocoder-Based High-Quality Speech Synthesis System for Real-Time Applications. IEICE Transactions on Information and Systems, E99-D(7):1877โ€“1884, 2016.

[2] Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang, Yanqing Liu, Xi Wang, Yichong Leng, Yuanhao Yi, Lei He, et al. Naturalspeech: End-to-end text-to-speech synthesis with human-level quality. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.

[3] Memoona Aziz, Umair Rehman, Muhammad Umair Danish, and Katarina Grolinger. Global-local image perceptual score (glips): Evaluating photorealistic quality of ai-generated images. IEEE Transactions on Human-Machine Systems, 2025.

[4] Xiaoxue Gao, Yiming Chen, Xianghu Yue, Yu Tsao, and Nancy F Chen. Ttslow: Slow down text-to-speech with efficiency robustness evaluations. IEEE Transactions on Audio, Speech and Language Processing, 2025.

[5] Rui Liu, Yifan Hu, Haolin Zuo, Zhaojie Luo, Longbiao Wang, and Guanglai Gao. Text-to-speech for low-resource agglutinative language with morphology-aware language model pre-training. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:1075โ€“1087, 2024.

[6] Wenbin Wang, Yang Song, and Sanjay Jha. Usat: A universal speaker-adaptive text-to-speech approach. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

[7] Rubeena A. Khan. Concatenative speech synthesis: A review. International Journal of Computer Applications, 136(3):1โ€“6, 2016.

[8] Yeon-Jun Kim, Mark Beutnagel, Alistair Conkie, and Ann Syrdal. System and method for automatic detection of abnormal stress patterns in unit selection synthesis. 02 2015.

[9] Hawraz A. Ahmad and Tarik A. Rashid. Planning the development of text-to-speech synthesis models and datasets with dynamic deep learning. Journal of King Saud University - Computer and Information Sciences, 36(7):102131, 2024.

[10] D. Pisoni and S. Hunnicutt. Perceptual evaluation of mitalk: The mit unrestricted text-to-speech system. 5:572โ€“575, 1980.

[11] Sunil Ravindra Shukla. Improving High Quality Concatenative Text-to-Speech Synthesis Using the Circular Linear Prediction Model. PhD thesis, Georgia Institute of Technology, May 2007.

[12] King Simon. An introduction to statistical parametric speech synthesis. Sadhana, 36:837โ€“852, 2011.

[13] Matej Rojc and Izidor Mlakar. A new fuzzy unit selection cost function optimized by relaxed gradient descent algorithm. Expert Systems with Applications, 159:113552, 2020.

[14] Yinghao Aaron Li, Cong Han, and Nima Mesgarani. Styletts: A style-based generative model for natural and diverse text-to-speech synthesis. IEEE Journal of Selected Topics in Signal Processing, 2025.

[15] Keiichi Tokuda, Yoshihiko Nankaku, Tomoki Toda, Heiga Zen, Junichi Yamagishi, and Keiichiro Oura. Speech synthesis based on hidden markov models. Proceedings of the IEEE, 101(5):1234โ€“1252, 2013.

[16] Xupeng Chen, Ran Wang, Amirhossein Khalilian-Gourtani, Leyao Yu, Patricia Dugan, Daniel Friedman, Werner Doyle, Orrin Devinsky, Yao Wang, and Adeen Flinker. A neural speech decoding framework leveraging deep learning and speech synthesis. Nature Machine Intelligence, 6(4):467โ€“480, 2024.

[17] Surendrabikram Thapa, Kritesh Rauniyar, Farhan Ahmad Jafri, Surabhi Adhikari, Kengatharaiyer Sarveswaran, Bal Krishna Bal, Hariram Veeramani, and Usman Naseem. Natural language understanding of devanagari script languages: Language identification, hate speech and its target detection. In Proceedings of the First Workshop on Challenges in Processing South Asian Languages (CHiPSAL 2025), pages 71โ€“82, 2025.

[18] King Simon. An introduction to statistical parametric speech synthesis. Sadhana, 36:837โ€“852, 2011.

[19] James L. Flanagan. The past, present, and future of speech processing. IEEE Signal Processing Magazine, 12(3):24โ€“48, 1995.

[20] Donata Moers, Igor Jauk, Bernd Mรถbius, and Petra Wagner. Synthesizing fast speech by implementing multi-phone units in unit selection speech synthesis. pages 353โ€“358, 2010.

[21] Robert B. Dunn, Agaath M.C. Sluijter, and Steven Greenberg. Diphone synthesis for phonetic vocoding. IEEE Transactions on Speech and Audio Processing, 1(3):273โ€“277, 1993.

[22] David Ferris. Techniques and challenges in speech synthesis. 2017.

[23] Karolina Kuligowska, Paweล‚ Kisielewicz, and Aleksandra Wล‚odarz. Speech synthesis systems: Disadvantages and limitations. International Journal of Engineering and Technology(UAE), 7:234โ€“239, 05 2018.

[24] Xian-Jun Xia, Zhen-Hua Ling, Yuan Jiang, and Li-Rong Dai. Hmm-based unit selection speech synthesis using log likelihood ratios derived from perceptual data. Speech Communication, 63-64:27โ€“37, 2014.

[25] Concatenation cost calculation and optimisation for unit selection in tts. pages 231โ€“234, 2002.

[26] Karolina Kuligowska, Paweล‚ Kisielewicz, and Aleksandra Wล‚odarz. Speech synthesis systems: Disadvantages and limitations. International Journal of Engineering and Technology(UAE), 7:234โ€“239, 05 2018.

[27] Sangramsing N. Kayte, Monica Mal, and Charansing Kayte. A review of unit selection speech synthesis. 5:5, 11 2015.

[28] Sangramsing N. Kayte, Monica Mal, and Charansing Kayte. A review of unit selection speech synthesis. 5:5, 11 2015.

[29] Sivanand Achanta, KNRK Raju Alluri, and Suryakanth V Gangashetty. Statistical parametric speech synthesis using bottleneck representation from sequence auto-encoder. 2016.

[30] Sivanand Achanta, KNRK Raju Alluri, and Suryakanth V Gangashetty. Statistical parametric speech synthesis using bottleneck representation from sequence auto-encoder. 2016.

[31] Kayte Sangramsing, Mundada Monica, and Gujrathi Jayesh. Hidden markov model based speech synthesis: A review. International Journal of Computer Applications, 130(3):35โ€“39, 2015.

[32] Keiichi Tokuda, Yoshihiko Nankaku, Tomoki Toda, Heiga Zen, Junichi Yamagishi, and Keiichiro Oura. Speech synthesis based on hidden markov models. Proceedings of the IEEE, 101(5):1234โ€“1252, 2013.