diff --git a/README.md b/README.md index 560415a..93a3de0 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,19 @@ # Supertonic โ€” Lightning Fast, On-Device TTS -[![Demo](https://img.shields.io/badge/๐Ÿค—%20Hugging%20Face-Demo-yellow)](https://huggingface.co/spaces/Supertone/supertonic#interactive-demo) -[![Models](https://img.shields.io/badge/๐Ÿค—%20Hugging%20Face-Models-blue)](https://huggingface.co/Supertone/supertonic) +[![v2 Demo](https://img.shields.io/badge/๐Ÿค—%20v2-Demo-yellow)](https://huggingface.co/spaces/Supertone/supertonic-2) +[![v2 Models](https://img.shields.io/badge/๐Ÿค—%20v2-Models-blue)](https://huggingface.co/Supertone/supertonic-2) +[![v1 Demo](https://img.shields.io/badge/๐Ÿค—%20v1%20(old)-Demo-lightgrey)](https://huggingface.co/spaces/Supertone/supertonic#interactive-demo) +[![v1 Models](https://img.shields.io/badge/๐Ÿค—%20v1%20(old)-Models-lightgrey)](https://huggingface.co/Supertone/supertonic)

- Supertonic Banner + Supertonic Banner

**Supertonic** is a lightning-fast, on-device text-to-speech system designed for **extreme performance** with minimal computational overhead. Powered by ONNX Runtime, it runs entirely on your deviceโ€”no cloud, no API calls, no privacy concerns. ### ๐Ÿ“ฐ Update News +- **2026.01.06** - ๐ŸŽ‰ **Supertonic 2** released with multilingual support! Now supports English (`en`), Korean (`ko`), Spanish (`es`), Portuguese (`pt`), and French (`fr`). [Demo](https://huggingface.co/spaces/Supertone/supertonic-2) | [Models](https://huggingface.co/Supertone/supertonic-2) - **2025.12.10** - Added `supertonic` PyPI package! Install via `pip install supertonic`. For details, visit [supertonic-py documentation](https://supertone-inc.github.io/supertonic-py) - **2025.12.10** - Added [6 new voice styles](https://huggingface.co/Supertone/supertonic/tree/b10dbaf18b316159be75b34d24f740008fddd381) (M3, M4, M5, F3, F4, F5). See [Voices](https://supertone-inc.github.io/supertonic-py/voices/) for details - **2025.12.08** - Optimized ONNX models via [OnnxSlim](https://github.com/inisis/OnnxSlim) now available on [Hugging Face Models](https://huggingface.co/Supertone/supertonic) @@ -43,7 +46,7 @@ https://github.com/user-attachments/assets/64980e58-ad91-423a-9623-78c2ffc13680 --- -> ๐ŸŽง **Try it now**: Experience Supertonic in your browser with our [**Interactive Demo**](https://huggingface.co/spaces/Supertone/supertonic#interactive-demo), or get started with pre-trained models from [**Hugging Face Hub**](https://huggingface.co/Supertone/supertonic) +> ๐ŸŽง **Try it now**: Experience Supertonic in your browser with our [**Interactive Demo**](https://huggingface.co/spaces/Supertone/supertonic-2), or get started with pre-trained models from [**Hugging Face Hub**](https://huggingface.co/Supertone/supertonic-2) ## Why Supertonic? @@ -92,7 +95,7 @@ Before running the examples, download the ONNX models and preset voices, and pla > - Generic: see `https://git-lfs.com` for installers ```bash -git clone https://huggingface.co/Supertone/supertonic assets +git clone https://huggingface.co/Supertone/supertonic-2 assets ``` ### Quick Start @@ -428,9 +431,9 @@ This paper describes the self-purification technique for training flow matching This project's sample code is released under the MIT License. - see the [LICENSE](https://github.com/supertone-inc/supertonic?tab=MIT-1-ov-file) for details. -The accompanying model is released under the OpenRAIL-M License. - see the [LICENSE](https://huggingface.co/Supertone/supertonic/blob/main/LICENSE) file for details. +The accompanying model is released under the OpenRAIL-M License. - see the [LICENSE](https://huggingface.co/Supertone/supertonic-2/blob/main/LICENSE) file for details. This model was trained using PyTorch, which is licensed under the BSD 3-Clause License but is not redistributed with this project. - see the [LICENSE](https://docs.pytorch.org/FBGEMM/general/License.html) for details. -Copyright (c) 2025 Supertone Inc. +Copyright (c) 2026 Supertone Inc. diff --git a/cpp/README.md b/cpp/README.md index 424429f..7ab8d93 100644 --- a/cpp/README.md +++ b/cpp/README.md @@ -4,6 +4,8 @@ High-performance text-to-speech inference using ONNX Runtime. ## ๐Ÿ“ฐ Update News +**2026.01.06** - ๐ŸŽ‰ **Supertonic 2** released with multilingual support! Now supports English (`en`), Korean (`ko`), Spanish (`es`), Portuguese (`pt`), and French (`fr`). [Demo](https://huggingface.co/spaces/Supertone/supertonic-2) | [Models](https://huggingface.co/Supertone/supertonic-2) + **2025.12.10** - Added [6 new voice styles](https://huggingface.co/Supertone/supertonic/tree/b10dbaf18b316159be75b34d24f740008fddd381) (M3, M4, M5, F3, F4, F5). See [Voices](https://supertone-inc.github.io/supertonic-py/voices/) for details **2025.12.08** - Optimized ONNX models via [OnnxSlim](https://github.com/inisis/OnnxSlim) now available on [Hugging Face Models](https://huggingface.co/Supertone/supertonic) @@ -74,15 +76,16 @@ Process multiple voice styles and texts at once: ```bash ./example_onnx \ --voice-style ../assets/voice_styles/M1.json,../assets/voice_styles/F1.json \ - --text "The sun sets behind the mountains, painting the sky in shades of pink and orange.|The weather is beautiful and sunny outside. A gentle breeze makes the air feel fresh and pleasant." \ + --text "The sun sets behind the mountains, painting the sky in shades of pink and orange.|์˜ค๋Š˜ ์•„์นจ์— ๊ณต์›์„ ์‚ฐ์ฑ…ํ–ˆ๋Š”๋ฐ, ์ƒˆ์†Œ๋ฆฌ์™€ ๋ฐ”๋žŒ ์†Œ๋ฆฌ๊ฐ€ ๋„ˆ๋ฌด ์ข‹์•„์„œ ํ•œ์ฐธ์„ ๋ฉˆ์ถฐ ์„œ์„œ ๋“ค์—ˆ์–ด์š”." \ + --lang en,ko \ --batch ``` This will: - Use `--batch` flag to enable batch processing mode - Generate speech for 2 different voice-text pairs -- Use male voice style (M1.json) for the first text -- Use female voice style (F1.json) for the second text +- Use male voice style (M1.json) for the first English text +- Use female voice style (F1.json) for the second Korean text - Process both samples in a single batch (automatic text chunking disabled) ### Example 3: High Quality Inference @@ -120,14 +123,17 @@ This will: |----------|------|---------|-------------| | `--onnx-dir` | str | `../assets/onnx` | Path to ONNX model directory | | `--total-step` | int | 5 | Number of denoising steps (higher = better quality, slower) | +| `--speed` | float | 1.05 | Speech speed factor (higher = faster, lower = slower) | | `--n-test` | int | 4 | Number of times to generate each sample | | `--voice-style` | str | `../assets/voice_styles/M1.json` | Voice style file path(s) (comma-separated for batch) | | `--text` | str | (long default text) | Text(s) to synthesize (pipe-separated for batch) | +| `--lang` | str | `en` | Language(s) for text(s): `en`, `ko`, `es`, `pt`, `fr` (comma-separated for batch) | | `--save-dir` | str | `results` | Output directory | | `--batch` | flag | False | Enable batch mode (disables automatic text chunking) | ## Notes - **Batch Processing**: The number of `--voice-style` files must match the number of `--text` entries +- **Multilingual Support**: Use `--lang` to specify language(s). Available: `en` (English), `ko` (Korean), `es` (Spanish), `pt` (Portuguese), `fr` (French) - **Long-Form Inference**: Without `--batch` flag, long texts are automatically chunked and combined into a single audio file with natural pauses - **Quality vs Speed**: Higher `--total-step` values produce better quality but take longer diff --git a/cpp/example_onnx.cpp b/cpp/example_onnx.cpp index 2f3aa02..9d7d68b 100644 --- a/cpp/example_onnx.cpp +++ b/cpp/example_onnx.cpp @@ -16,6 +16,7 @@ struct Args { std::vector text = { "This morning, I took a walk in the park, and the sound of the birds and the breeze was so pleasant that I stopped for a long time just to listen." }; + std::vector lang = {"en"}; std::string save_dir = "results"; bool batch = false; }; @@ -41,6 +42,7 @@ Args parseArgs(int argc, char* argv[]) { else if (arg == "--n-test" && i + 1 < argc) args.n_test = std::stoi(argv[++i]); else if (arg == "--voice-style" && i + 1 < argc) args.voice_style = splitString(argv[++i], ','); else if (arg == "--text" && i + 1 < argc) args.text = splitString(argv[++i], '|'); + else if (arg == "--lang" && i + 1 < argc) args.lang = splitString(argv[++i], ','); else if (arg == "--save-dir" && i + 1 < argc) args.save_dir = argv[++i]; else if (arg == "--batch") args.batch = true; } @@ -58,6 +60,7 @@ int main(int argc, char* argv[]) { std::string save_dir = args.save_dir; std::vector voice_style_paths = args.voice_style; std::vector text_list = args.text; + std::vector lang_list = args.lang; bool batch = args.batch; if (voice_style_paths.size() != text_list.size()) { @@ -87,9 +90,9 @@ int main(int argc, char* argv[]) { auto result = timer("Generating speech from text", [&]() { if (batch) { - return text_to_speech->batch(memory_info, text_list, style, total_step, speed); + return text_to_speech->batch(memory_info, text_list, lang_list, style, total_step, speed); } else { - return text_to_speech->call(memory_info, text_list[0], style, total_step, speed); + return text_to_speech->call(memory_info, text_list[0], lang_list[0], style, total_step, speed); } }); diff --git a/cpp/helper.cpp b/cpp/helper.cpp index c02048d..111f2fb 100644 --- a/cpp/helper.cpp +++ b/cpp/helper.cpp @@ -6,10 +6,14 @@ #include #include #include +#include #include using json = nlohmann::json; +// Available languages for multilingual TTS +const std::vector AVAILABLE_LANGS = {"en", "ko", "es", "pt", "fr"}; + // Global tensor buffers for memory management static std::vector> g_tensor_buffers_float; static std::vector> g_tensor_buffers_int64; @@ -45,7 +49,7 @@ UnicodeProcessor::UnicodeProcessor(const std::string& unicode_indexer_json_path) indexer_ = loadJsonInt64(unicode_indexer_json_path); } -std::string UnicodeProcessor::preprocessText(const std::string& text) { +std::string UnicodeProcessor::preprocessText(const std::string& text, const std::string& lang) { // TODO: Need advanced normalizer for better performance // NOTE: C++ doesn't have built-in Unicode normalization like Python's NFKD // For full Unicode normalization, consider using ICU library @@ -53,17 +57,8 @@ std::string UnicodeProcessor::preprocessText(const std::string& text) { std::string result = text; - // FIXME: this should be fixed for non-English languages - - // Remove emojis and various Unicode symbols - // Using regex to remove common emoji ranges and special symbols - // Note: This is a simplified version - full emoji support needs UTF-8 handling - std::regex emoji_pattern( - "[\xF0\x9F][\x80-\xBF]{2}|" // Common emoji pattern in UTF-8 - "[\xE2][\x80-\xBF]{2}|" // Various symbols - "[\xE2][\x98-\x9E][\x80-\xBF]" // More symbols - ); - result = std::regex_replace(result, emoji_pattern, ""); + // IMPORTANT: Do symbol replacements FIRST (before emoji removal) + // to preserve curly quotes and other punctuation that might be matched by emoji patterns // Replace various dashes and symbols struct Replacement { @@ -75,12 +70,11 @@ std::string UnicodeProcessor::preprocessText(const std::string& text) { {"โ€“", "-"}, // en dash {"โ€‘", "-"}, // non-breaking hyphen {"โ€”", "-"}, // em dash - {"ยฏ", " "}, // macron {"_", " "}, // underscore - { u8"\u201C", "\"" }, // left double quote โ€œ - { u8"\u201D", "\"" }, // right double quote โ€ - { u8"\u2018", "'" }, // left single quote โ€˜ - { u8"\u2019", "'" }, // right single quote โ€™ + { u8"\u201C", "\"" }, // left double quote " + { u8"\u201D", "\"" }, // right double quote " + { u8"\u2018", "'" }, // left single quote ' + { u8"\u2019", "'" }, // right single quote ' {"ยด", "'"}, // acute accent {"`", "'"}, // grave accent {"[", " "}, // left bracket @@ -100,12 +94,12 @@ std::string UnicodeProcessor::preprocessText(const std::string& text) { } } - // Remove combining diacritics (common combining marks in UTF-8) - // FIXME: this should be fixed for non-English languages - std::regex diacritics_pattern( - "[\xCC\xCD][\x80-\xBF]" // Combining diacritical marks range + // Remove emojis AFTER symbol replacements + // Only target actual emoji ranges (4-byte UTF-8 sequences starting with F0 9F) + std::regex emoji_pattern( + "[\xF0][\x9F][\x80-\xBF][\x80-\xBF]" // 4-byte emoji (U+1F000-U+1FFFF) ); - result = std::regex_replace(result, diacritics_pattern, ""); + result = std::regex_replace(result, emoji_pattern, ""); // Remove special symbols const char* special_symbols[] = {"โ™ฅ", "โ˜†", "โ™ก", "ยฉ", "\\"}; @@ -187,14 +181,168 @@ std::string UnicodeProcessor::preprocessText(const std::string& text) { } } + // Validate language + bool valid_lang = false; + for (const auto& available_lang : AVAILABLE_LANGS) { + if (lang == available_lang) { + valid_lang = true; + break; + } + } + if (!valid_lang) { + throw std::runtime_error("Invalid language: " + lang + ". Available: en, ko, es, pt, fr"); + } + + // Wrap text with language tags + result = "<" + lang + ">" + result + ""; + return result; } +// Hangul syllable decomposition constants (Unicode Standard Annex #15) +static const uint32_t HANGUL_SBASE = 0xAC00; // Start of Hangul syllables +static const uint32_t HANGUL_LBASE = 0x1100; // Start of Hangul Jamo (leading consonants) +static const uint32_t HANGUL_VBASE = 0x1161; // Start of Hangul Jamo (vowels) +static const uint32_t HANGUL_TBASE = 0x11A7; // Start of Hangul Jamo (trailing consonants) +static const int HANGUL_LCOUNT = 19; // Number of leading consonants +static const int HANGUL_VCOUNT = 21; // Number of vowels +static const int HANGUL_TCOUNT = 28; // Number of trailing consonants (including none) +static const int HANGUL_NCOUNT = HANGUL_VCOUNT * HANGUL_TCOUNT; // 588 +static const int HANGUL_SCOUNT = HANGUL_LCOUNT * HANGUL_NCOUNT; // 11172 + +// Latin character NFKD decompositions for Spanish, Portuguese, French +static const std::unordered_map> LATIN_DECOMPOSITIONS = { + // Acute accent + {0x00C1, {0x0041, 0x0301}}, // ร โ†’ A + ฬ + {0x00C9, {0x0045, 0x0301}}, // ร‰ โ†’ E + ฬ + {0x00CD, {0x0049, 0x0301}}, // ร โ†’ I + ฬ + {0x00D3, {0x004F, 0x0301}}, // ร“ โ†’ O + ฬ + {0x00DA, {0x0055, 0x0301}}, // รš โ†’ U + ฬ + {0x00E1, {0x0061, 0x0301}}, // รก โ†’ a + ฬ + {0x00E9, {0x0065, 0x0301}}, // รฉ โ†’ e + ฬ + {0x00ED, {0x0069, 0x0301}}, // รญ โ†’ i + ฬ + {0x00F3, {0x006F, 0x0301}}, // รณ โ†’ o + ฬ + {0x00FA, {0x0075, 0x0301}}, // รบ โ†’ u + ฬ + // Grave accent + {0x00C0, {0x0041, 0x0300}}, // ร€ โ†’ A + ฬ€ + {0x00C8, {0x0045, 0x0300}}, // รˆ โ†’ E + ฬ€ + {0x00CC, {0x0049, 0x0300}}, // รŒ โ†’ I + ฬ€ + {0x00D2, {0x004F, 0x0300}}, // ร’ โ†’ O + ฬ€ + {0x00D9, {0x0055, 0x0300}}, // ร™ โ†’ U + ฬ€ + {0x00E0, {0x0061, 0x0300}}, // ร  โ†’ a + ฬ€ + {0x00E8, {0x0065, 0x0300}}, // รจ โ†’ e + ฬ€ + {0x00EC, {0x0069, 0x0300}}, // รฌ โ†’ i + ฬ€ + {0x00F2, {0x006F, 0x0300}}, // รฒ โ†’ o + ฬ€ + {0x00F9, {0x0075, 0x0300}}, // รน โ†’ u + ฬ€ + // Circumflex + {0x00C2, {0x0041, 0x0302}}, // ร‚ โ†’ A + ฬ‚ + {0x00CA, {0x0045, 0x0302}}, // รŠ โ†’ E + ฬ‚ + {0x00CE, {0x0049, 0x0302}}, // รŽ โ†’ I + ฬ‚ + {0x00D4, {0x004F, 0x0302}}, // ร” โ†’ O + ฬ‚ + {0x00DB, {0x0055, 0x0302}}, // ร› โ†’ U + ฬ‚ + {0x00E2, {0x0061, 0x0302}}, // รข โ†’ a + ฬ‚ + {0x00EA, {0x0065, 0x0302}}, // รช โ†’ e + ฬ‚ + {0x00EE, {0x0069, 0x0302}}, // รฎ โ†’ i + ฬ‚ + {0x00F4, {0x006F, 0x0302}}, // รด โ†’ o + ฬ‚ + {0x00FB, {0x0075, 0x0302}}, // รป โ†’ u + ฬ‚ + // Tilde + {0x00C3, {0x0041, 0x0303}}, // รƒ โ†’ A + ฬƒ + {0x00D1, {0x004E, 0x0303}}, // ร‘ โ†’ N + ฬƒ + {0x00D5, {0x004F, 0x0303}}, // ร• โ†’ O + ฬƒ + {0x00E3, {0x0061, 0x0303}}, // รฃ โ†’ a + ฬƒ + {0x00F1, {0x006E, 0x0303}}, // รฑ โ†’ n + ฬƒ + {0x00F5, {0x006F, 0x0303}}, // รต โ†’ o + ฬƒ + // Diaeresis + {0x00C4, {0x0041, 0x0308}}, // ร„ โ†’ A + ฬˆ + {0x00CB, {0x0045, 0x0308}}, // ร‹ โ†’ E + ฬˆ + {0x00CF, {0x0049, 0x0308}}, // ร โ†’ I + ฬˆ + {0x00D6, {0x004F, 0x0308}}, // ร– โ†’ O + ฬˆ + {0x00DC, {0x0055, 0x0308}}, // รœ โ†’ U + ฬˆ + {0x00E4, {0x0061, 0x0308}}, // รค โ†’ a + ฬˆ + {0x00EB, {0x0065, 0x0308}}, // รซ โ†’ e + ฬˆ + {0x00EF, {0x0069, 0x0308}}, // รฏ โ†’ i + ฬˆ + {0x00F6, {0x006F, 0x0308}}, // รถ โ†’ o + ฬˆ + {0x00FC, {0x0075, 0x0308}}, // รผ โ†’ u + ฬˆ + // Cedilla + {0x00C7, {0x0043, 0x0327}}, // ร‡ โ†’ C + ฬง + {0x00E7, {0x0063, 0x0327}}, // รง โ†’ c + ฬง +}; + +// Decompose a character using NFKD (Hangul + Latin accented) +static void decomposeCharacter(uint32_t codepoint, std::vector& output) { + // Check Hangul syllables first + if (codepoint >= HANGUL_SBASE && codepoint < HANGUL_SBASE + HANGUL_SCOUNT) { + // Decompose Hangul syllable into Jamo + uint32_t sIndex = codepoint - HANGUL_SBASE; + uint32_t lIndex = sIndex / HANGUL_NCOUNT; + uint32_t vIndex = (sIndex % HANGUL_NCOUNT) / HANGUL_TCOUNT; + uint32_t tIndex = sIndex % HANGUL_TCOUNT; + + output.push_back(static_cast(HANGUL_LBASE + lIndex)); + output.push_back(static_cast(HANGUL_VBASE + vIndex)); + if (tIndex > 0) { + output.push_back(static_cast(HANGUL_TBASE + tIndex)); + } + return; + } + + // Check Latin decompositions + auto it = LATIN_DECOMPOSITIONS.find(codepoint); + if (it != LATIN_DECOMPOSITIONS.end()) { + for (uint16_t cp : it->second) { + output.push_back(cp); + } + return; + } + + // Keep as-is + output.push_back(static_cast(codepoint & 0xFFFF)); +} + std::vector UnicodeProcessor::textToUnicodeValues(const std::string& text) { std::vector unicode_values; - for (char c : text) { - unicode_values.push_back(static_cast(static_cast(c))); + size_t i = 0; + + while (i < text.size()) { + uint32_t codepoint = 0; + unsigned char c = static_cast(text[i]); + + if ((c & 0x80) == 0) { + // 1-byte ASCII (0xxxxxxx) + codepoint = c; + i += 1; + } + else if ((c & 0xE0) == 0xC0 && i + 1 < text.size()) { + // 2-byte UTF-8 (110xxxxx 10xxxxxx) + codepoint = (c & 0x1F) << 6; + codepoint |= (static_cast(text[i + 1]) & 0x3F); + i += 2; + } + else if ((c & 0xF0) == 0xE0 && i + 2 < text.size()) { + // 3-byte UTF-8 (1110xxxx 10xxxxxx 10xxxxxx) - includes Korean + codepoint = (c & 0x0F) << 12; + codepoint |= (static_cast(text[i + 1]) & 0x3F) << 6; + codepoint |= (static_cast(text[i + 2]) & 0x3F); + i += 3; + } + else if ((c & 0xF8) == 0xF0 && i + 3 < text.size()) { + // 4-byte UTF-8 (11110xxx 10xxxxxx 10xxxxxx 10xxxxxx) + codepoint = (c & 0x07) << 18; + codepoint |= (static_cast(text[i + 1]) & 0x3F) << 12; + codepoint |= (static_cast(text[i + 2]) & 0x3F) << 6; + codepoint |= (static_cast(text[i + 3]) & 0x3F); + i += 4; + } + else { + // Invalid UTF-8, skip byte + i += 1; + continue; + } + + // Decompose Hangul syllables and Latin accented characters (NFKD) + decomposeCharacter(codepoint, unicode_values); } + return unicode_values; } @@ -206,25 +354,31 @@ std::vector>> UnicodeProcessor::getTextMask( void UnicodeProcessor::call( const std::vector& text_list, + const std::vector& lang_list, std::vector>& text_ids, std::vector>>& text_mask ) { std::vector processed_texts; - for (const auto& text : text_list) { - processed_texts.push_back(preprocessText(text)); + for (size_t i = 0; i < text_list.size(); i++) { + processed_texts.push_back(preprocessText(text_list[i], lang_list[i])); } + // Convert texts to unicode values first to get correct character counts + std::vector> all_unicode_vals; std::vector text_ids_lengths; for (const auto& text : processed_texts) { - text_ids_lengths.push_back(static_cast(text.length())); + auto unicode_vals = textToUnicodeValues(text); + // Use number of Unicode codepoints, not bytes + text_ids_lengths.push_back(static_cast(unicode_vals.size())); + all_unicode_vals.push_back(std::move(unicode_vals)); } int64_t max_len = *std::max_element(text_ids_lengths.begin(), text_ids_lengths.end()); text_ids.resize(text_list.size()); - for (size_t i = 0; i < processed_texts.size(); i++) { + for (size_t i = 0; i < all_unicode_vals.size(); i++) { text_ids[i].resize(max_len, 0); - auto unicode_vals = textToUnicodeValues(processed_texts[i]); + const auto& unicode_vals = all_unicode_vals[i]; for (size_t j = 0; j < unicode_vals.size(); j++) { if (unicode_vals[j] < indexer_.size()) { text_ids[i][j] = indexer_[unicode_vals[j]]; @@ -315,6 +469,7 @@ void TextToSpeech::sampleNoisyLatent( TextToSpeech::SynthesisResult TextToSpeech::_infer( Ort::MemoryInfo& memory_info, const std::vector& text_list, + const std::vector& lang_list, const Style& style, int total_step, float speed @@ -328,7 +483,7 @@ TextToSpeech::SynthesisResult TextToSpeech::_infer( // Process text std::vector> text_ids; std::vector>> text_mask; - text_processor_->call(text_list, text_ids, text_mask); + text_processor_->call(text_list, lang_list, text_ids, text_mask); std::vector text_ids_shape = {bsz, static_cast(text_ids[0].size())}; std::vector text_mask_shape = {bsz, 1, static_cast(text_mask[0][0].size())}; @@ -530,6 +685,7 @@ TextToSpeech::SynthesisResult TextToSpeech::_infer( TextToSpeech::SynthesisResult TextToSpeech::call( Ort::MemoryInfo& memory_info, const std::string& text, + const std::string& lang, const Style& style, int total_step, float speed, @@ -539,12 +695,13 @@ TextToSpeech::SynthesisResult TextToSpeech::call( throw std::runtime_error("Single speaker text to speech only supports single style"); } - auto text_list = chunkText(text); + int max_len = (lang == "ko") ? 120 : 300; + auto text_list = chunkText(text, max_len); std::vector wav_cat; float dur_cat = 0.0f; for (const auto& chunk : text_list) { - auto result = _infer(memory_info, {chunk}, style, total_step, speed); + auto result = _infer(memory_info, {chunk}, {lang}, style, total_step, speed); if (wav_cat.empty()) { wav_cat = result.wav; @@ -568,11 +725,12 @@ TextToSpeech::SynthesisResult TextToSpeech::call( TextToSpeech::SynthesisResult TextToSpeech::batch( Ort::MemoryInfo& memory_info, const std::vector& text_list, + const std::vector& lang_list, const Style& style, int total_step, float speed ) { - return _infer(memory_info, text_list, style, total_step, speed); + return _infer(memory_info, text_list, lang_list, style, total_step, speed); } // ============================================================================ @@ -911,15 +1069,43 @@ std::vector loadJsonInt64(const std::string& file_path) { std::string sanitizeFilename(const std::string& text, int max_len) { std::string result; - int count = 0; - for (char c : text) { - if (count >= max_len) break; - if (std::isalnum(static_cast(c))) { - result += c; - } else { - result += '_'; + int char_count = 0; + size_t i = 0; + + while (i < text.size() && char_count < max_len) { + unsigned char c = static_cast(text[i]); + + // Check if it's ASCII alphanumeric or underscore + if (std::isalnum(c) || c == '_') { + result += text[i]; + i++; + char_count++; + } + // Check for UTF-8 multi-byte sequences (preserve Unicode letters/numbers) + else if ((c & 0xE0) == 0xC0 && i + 1 < text.size()) { + // 2-byte UTF-8 sequence + result += text.substr(i, 2); + i += 2; + char_count++; + } + else if ((c & 0xF0) == 0xE0 && i + 2 < text.size()) { + // 3-byte UTF-8 sequence (includes Korean, Japanese, Chinese) + result += text.substr(i, 3); + i += 3; + char_count++; + } + else if ((c & 0xF8) == 0xF0 && i + 3 < text.size()) { + // 4-byte UTF-8 sequence + result += text.substr(i, 4); + i += 4; + char_count++; + } + else { + // Replace other characters with underscore + result += '_'; + i++; + char_count++; } - count++; } return result; } diff --git a/cpp/helper.h b/cpp/helper.h index 73ecf5b..34f3941 100644 --- a/cpp/helper.h +++ b/cpp/helper.h @@ -8,6 +8,9 @@ #include #include +// Available languages for multilingual TTS +extern const std::vector AVAILABLE_LANGS; + /** * Configuration structure */ @@ -33,6 +36,7 @@ public: // Process text list to text IDs and mask void call( const std::vector& text_list, + const std::vector& lang_list, std::vector>& text_ids, std::vector>>& text_mask ); @@ -40,7 +44,7 @@ public: private: std::vector indexer_; - std::string preprocessText(const std::string& text); + std::string preprocessText(const std::string& text, const std::string& lang); std::vector textToUnicodeValues(const std::string& text); std::vector>> getTextMask( const std::vector& text_ids_lengths @@ -89,6 +93,7 @@ public: SynthesisResult call( Ort::MemoryInfo& memory_info, const std::string& text, + const std::string& lang, const Style& style, int total_step, float speed = 1.05f, @@ -98,6 +103,7 @@ public: SynthesisResult batch( Ort::MemoryInfo& memory_info, const std::vector& text_list, + const std::vector& lang_list, const Style& style, int total_step, float speed = 1.05f @@ -109,6 +115,7 @@ private: SynthesisResult _infer( Ort::MemoryInfo& memory_info, const std::vector& text_list, + const std::vector& lang_list, const Style& style, int total_step, float speed = 1.05f diff --git a/csharp/ExampleONNX.cs b/csharp/ExampleONNX.cs index 234a51d..a78a876 100644 --- a/csharp/ExampleONNX.cs +++ b/csharp/ExampleONNX.cs @@ -19,6 +19,7 @@ namespace Supertonic { "This morning, I took a walk in the park, and the sound of the birds and the breeze was so pleasant that I stopped for a long time just to listen." }; + public List Lang { get; set; } = new List { "en" }; public string SaveDir { get; set; } = "results"; public bool Batch { get; set; } = false; } @@ -55,6 +56,9 @@ namespace Supertonic case "--text" when i + 1 < args.Length: result.Text = args[++i].Split('|').ToList(); break; + case "--lang" when i + 1 < args.Length: + result.Lang = args[++i].Split(',').ToList(); + break; case "--save-dir" when i + 1 < args.Length: result.SaveDir = args[++i]; break; @@ -76,6 +80,7 @@ namespace Supertonic string saveDir = parsedArgs.SaveDir; var voiceStylePaths = parsedArgs.VoiceStyle; var textList = parsedArgs.Text; + var langList = parsedArgs.Lang; bool batch = parsedArgs.Batch; if (voiceStylePaths.Count != textList.Count) @@ -101,11 +106,11 @@ namespace Supertonic { if (batch) { - return textToSpeech.Batch(textList, style, totalStep, speed); + return textToSpeech.Batch(textList, langList, style, totalStep, speed); } else { - return textToSpeech.Call(textList[0], style, totalStep, speed); + return textToSpeech.Call(textList[0], langList[0], style, totalStep, speed); } }); diff --git a/csharp/Helper.cs b/csharp/Helper.cs index e555630..fa356aa 100644 --- a/csharp/Helper.cs +++ b/csharp/Helper.cs @@ -10,6 +10,12 @@ using Microsoft.ML.OnnxRuntime.Tensors; namespace Supertonic { + // Available languages for multilingual TTS + public static class Languages + { + public static readonly string[] Available = { "en", "ko", "es", "pt", "fr" }; + } + // ============================================================================ // Configuration classes // ============================================================================ @@ -118,13 +124,11 @@ namespace Supertonic return result.ToString(); } - private string PreprocessText(string text) + private string PreprocessText(string text, string lang) { // TODO: Need advanced normalizer for better performance text = text.Normalize(NormalizationForm.FormKD); - // FIXME: this should be fixed for non-English languages - // Remove emojis (wide Unicode range) // C# doesn't support \u{...} syntax in regex, so we use character filtering instead text = RemoveEmojis(text); @@ -135,7 +139,6 @@ namespace Supertonic {"โ€“", "-"}, // en dash {"โ€‘", "-"}, // non-breaking hyphen {"โ€”", "-"}, // em dash - {"ยฏ", " "}, // macron {"_", " "}, // underscore {"\u201C", "\""}, // left double quote {"\u201D", "\""}, // right double quote @@ -157,9 +160,6 @@ namespace Supertonic text = text.Replace(kvp.Key, kvp.Value); } - // Remove combining diacritics // FIXME: this should be fixed for non-English languages - text = Regex.Replace(text, @"[\u0302\u0303\u0304\u0305\u0306\u0307\u0308\u030A\u030B\u030C\u0327\u0328\u0329\u032A\u032B\u032C\u032D\u032E\u032F]", ""); - // Remove special symbols text = Regex.Replace(text, @"[โ™ฅโ˜†โ™กยฉ\\]", ""); @@ -208,6 +208,15 @@ namespace Supertonic text += "."; } + // Validate language + if (!Languages.Available.Contains(lang)) + { + throw new ArgumentException($"Invalid language: {lang}. Available: {string.Join(", ", Languages.Available)}"); + } + + // Wrap text with language tags + text = $"<{lang}>" + text + $""; + return text; } @@ -221,9 +230,9 @@ namespace Supertonic return Helper.LengthToMask(textIdsLengths); } - public (long[][] textIds, float[][][] textMask) Call(List textList) + public (long[][] textIds, float[][][] textMask) Call(List textList, List langList) { - var processedTexts = textList.Select(t => PreprocessText(t)).ToList(); + var processedTexts = textList.Select((t, i) => PreprocessText(t, langList[i])).ToList(); var textIdsLengths = processedTexts.Select(t => (long)t.Length).ToArray(); long maxLen = textIdsLengths.Max(); @@ -328,7 +337,7 @@ namespace Supertonic return (noisyLatent, latentMask); } - private (float[] wav, float[] duration) _Infer(List textList, Style style, int totalStep, float speed = 1.05f) + private (float[] wav, float[] duration) _Infer(List textList, List langList, Style style, int totalStep, float speed = 1.05f) { int bsz = textList.Count; if (bsz != style.TtlShape[0]) @@ -337,7 +346,7 @@ namespace Supertonic } // Process text - var (textIds, textMask) = _textProcessor.Call(textList); + var (textIds, textMask) = _textProcessor.Call(textList, langList); var textIdsShape = new long[] { bsz, textIds[0].Length }; var textMaskShape = new long[] { bsz, 1, textMask[0][0].Length }; @@ -424,20 +433,21 @@ namespace Supertonic return (wavTensor.ToArray(), durOnnx); } - public (float[] wav, float[] duration) Call(string text, Style style, int totalStep, float speed = 1.05f, float silenceDuration = 0.3f) + public (float[] wav, float[] duration) Call(string text, string lang, Style style, int totalStep, float speed = 1.05f, float silenceDuration = 0.3f) { if (style.TtlShape[0] != 1) { throw new ArgumentException("Single speaker text to speech only supports single style"); } - var textList = Helper.ChunkText(text); + int maxLen = lang == "ko" ? 120 : 300; + var textList = Helper.ChunkText(text, maxLen); var wavCat = new List(); float durCat = 0.0f; foreach (var chunk in textList) { - var (wav, duration) = _Infer(new List { chunk }, style, totalStep, speed); + var (wav, duration) = _Infer(new List { chunk }, new List { lang }, style, totalStep, speed); if (wavCat.Count == 0) { @@ -457,9 +467,9 @@ namespace Supertonic return (wavCat.ToArray(), new float[] { durCat }); } - public (float[] wav, float[] duration) Batch(List textList, Style style, int totalStep, float speed = 1.05f) + public (float[] wav, float[] duration) Batch(List textList, List langList, Style style, int totalStep, float speed = 1.05f) { - return _Infer(textList, style, totalStep, speed); + return _Infer(textList, langList, style, totalStep, speed); } } diff --git a/csharp/README.md b/csharp/README.md index 89d97f5..459669a 100644 --- a/csharp/README.md +++ b/csharp/README.md @@ -4,6 +4,8 @@ This guide provides examples for running TTS inference using `ExampleONNX.cs`. ## ๐Ÿ“ฐ Update News +**2026.01.06** - ๐ŸŽ‰ **Supertonic 2** released with multilingual support! Now supports English (`en`), Korean (`ko`), Spanish (`es`), Portuguese (`pt`), and French (`fr`). [Demo](https://huggingface.co/spaces/Supertone/supertonic-2) | [Models](https://huggingface.co/Supertone/supertonic-2) + **2025.12.10** - Added [6 new voice styles](https://huggingface.co/Supertone/supertonic/tree/b10dbaf18b316159be75b34d24f740008fddd381) (M3, M4, M5, F3, F4, F5). See [Voices](https://supertone-inc.github.io/supertonic-py/voices/) for details **2025.12.08** - Optimized ONNX models via [OnnxSlim](https://github.com/inisis/OnnxSlim) now available on [Hugging Face Models](https://huggingface.co/Supertone/supertonic) @@ -45,15 +47,16 @@ Process multiple voice styles and texts at once: ```bash dotnet run -- \ --voice-style assets/voice_styles/M1.json,assets/voice_styles/F1.json \ - --text "The sun sets behind the mountains, painting the sky in shades of pink and orange.|The weather is beautiful and sunny outside. A gentle breeze makes the air feel fresh and pleasant." \ + --text "The sun sets behind the mountains, painting the sky in shades of pink and orange.|์˜ค๋Š˜ ์•„์นจ์— ๊ณต์›์„ ์‚ฐ์ฑ…ํ–ˆ๋Š”๋ฐ, ์ƒˆ์†Œ๋ฆฌ์™€ ๋ฐ”๋žŒ ์†Œ๋ฆฌ๊ฐ€ ๋„ˆ๋ฌด ์ข‹์•„์„œ ํ•œ์ฐธ์„ ๋ฉˆ์ถฐ ์„œ์„œ ๋“ค์—ˆ์–ด์š”." \ + --lang en,ko \ --batch ``` This will: - Use `--batch` flag to enable batch processing mode - Generate speech for 2 different voice-text pairs -- Use male voice style (M1.json) for the first text -- Use female voice style (F1.json) for the second text +- Use male voice style (M1.json) for the first English text +- Use female voice style (F1.json) for the second Korean text - Process both samples in a single batch (automatic text chunking disabled) ### Example 3: High Quality Inference @@ -92,15 +95,18 @@ This will: | `--use-gpu` | flag | False | Use GPU for inference (not supported yet) | | `--onnx-dir` | str | `assets/onnx` | Path to ONNX model directory | | `--total-step` | int | 5 | Number of denoising steps (higher = better quality, slower) | +| `--speed` | float | 1.05 | Speech speed factor (higher = faster, lower = slower) | | `--n-test` | int | 4 | Number of times to generate each sample | | `--voice-style` | str+ | `assets/voice_styles/M1.json` | Voice style file path(s) (comma-separated) | | `--text` | str+ | (long default text) | Text(s) to synthesize (pipe-separated: `|`) | +| `--lang` | str+ | `en` | Language(s) for text(s): `en`, `ko`, `es`, `pt`, `fr` (comma-separated) | | `--save-dir` | str | `results` | Output directory | | `--batch` | flag | False | Enable batch mode (disables automatic text chunking) | ## Notes - **Batch Processing**: The number of `--voice-style` files must match the number of `--text` entries +- **Multilingual Support**: Use `--lang` to specify language(s). Available: `en` (English), `ko` (Korean), `es` (Spanish), `pt` (Portuguese), `fr` (French) - **Long-Form Inference**: Without `--batch` flag, long texts are automatically chunked and combined into a single audio file with natural pauses - **Quality vs Speed**: Higher `--total-step` values produce better quality but take longer - **GPU Support**: GPU mode is not supported yet diff --git a/flutter/README.md b/flutter/README.md index 40dfbb8..ab7a7ff 100644 --- a/flutter/README.md +++ b/flutter/README.md @@ -1,18 +1,29 @@ # Supertonic Flutter Example -This example demonstrates how to use Supertonic in a Flutter application using ONNX Runtime. +This example demonstrates how to use Supertonic 2 in a Flutter application using ONNX Runtime. > **Note:** This project uses the `flutter_onnxruntime` package ([https://pub.dev/packages/flutter_onnxruntime](https://pub.dev/packages/flutter_onnxruntime)). At the moment, only the macOS platform has been tested. Although the flutter_onnxruntime package supports several other platforms, they have not been tested in this project yet and may require additional verification. ## ๐Ÿ“ฐ Update News +**2026.01.06** - ๐ŸŽ‰ **Supertonic 2** released with multilingual support! Now supports English (`en`), Korean (`ko`), Spanish (`es`), Portuguese (`pt`), and French (`fr`). [Demo](https://huggingface.co/spaces/Supertone/supertonic-2) | [Models](https://huggingface.co/Supertone/supertonic-2) + **2025.12.10** - Added [6 new voice styles](https://huggingface.co/Supertone/supertonic/tree/b10dbaf18b316159be75b34d24f740008fddd381) (M3, M4, M5, F3, F4, F5). See [Voices](https://supertone-inc.github.io/supertonic-py/voices/) for details **2025.12.08** - Optimized ONNX models via [OnnxSlim](https://github.com/inisis/OnnxSlim) now available on [Hugging Face Models](https://huggingface.co/Supertone/supertonic) **2025.11.23** - Added and tested macos support. +## Multilingual Support + +Supertonic 2 supports multiple languages. Select the appropriate language from the dropdown: +- **English (en)**: Default language +- **ํ•œ๊ตญ์–ด (ko)**: Korean +- **Espaรฑol (es)**: Spanish +- **Portuguรชs (pt)**: Portuguese +- **Franรงais (fr)**: French + ## Requirements - Flutter SDK version ^3.5.0 diff --git a/flutter/lib/helper.dart b/flutter/lib/helper.dart index 2d66680..b1f45ec 100644 --- a/flutter/lib/helper.dart +++ b/flutter/lib/helper.dart @@ -11,6 +11,205 @@ final logger = Logger( printer: PrettyPrinter(methodCount: 0, errorMethodCount: 5, lineLength: 80), ); +// Available languages for multilingual TTS +const List availableLangs = ['en', 'ko', 'es', 'pt', 'fr']; + +bool isValidLang(String lang) => availableLangs.contains(lang); + +// Hangul Jamo constants for NFKD decomposition +const int _hangulSyllableBase = 0xAC00; +const int _hangulSyllableEnd = 0xD7A3; +const int _leadingJamoBase = 0x1100; +const int _vowelJamoBase = 0x1161; +const int _trailingJamoBase = 0x11A7; +const int _vowelCount = 21; +const int _trailingCount = 28; + +/// Decompose a Hangul syllable into Jamo (NFKD-like decomposition) +List _decomposeHangulSyllable(int codePoint) { + if (codePoint < _hangulSyllableBase || codePoint > _hangulSyllableEnd) { + return [codePoint]; + } + + final syllableIndex = codePoint - _hangulSyllableBase; + final leadingIndex = syllableIndex ~/ (_vowelCount * _trailingCount); + final vowelIndex = + (syllableIndex % (_vowelCount * _trailingCount)) ~/ _trailingCount; + final trailingIndex = syllableIndex % _trailingCount; + + final result = [ + _leadingJamoBase + leadingIndex, + _vowelJamoBase + vowelIndex, + ]; + + if (trailingIndex > 0) { + result.add(_trailingJamoBase + trailingIndex); + } + + return result; +} + +/// Common Latin character decompositions (NFKD) for es, pt, fr +const Map> _latinDecompositions = { + // Uppercase with acute accent + 0x00C1: [0x0041, 0x0301], // ร โ†’ A + ฬ + 0x00C9: [0x0045, 0x0301], // ร‰ โ†’ E + ฬ + 0x00CD: [0x0049, 0x0301], // ร โ†’ I + ฬ + 0x00D3: [0x004F, 0x0301], // ร“ โ†’ O + ฬ + 0x00DA: [0x0055, 0x0301], // รš โ†’ U + ฬ + // Lowercase with acute accent + 0x00E1: [0x0061, 0x0301], // รก โ†’ a + ฬ + 0x00E9: [0x0065, 0x0301], // รฉ โ†’ e + ฬ + 0x00ED: [0x0069, 0x0301], // รญ โ†’ i + ฬ + 0x00F3: [0x006F, 0x0301], // รณ โ†’ o + ฬ + 0x00FA: [0x0075, 0x0301], // รบ โ†’ u + ฬ + // Grave accent + 0x00C0: [0x0041, 0x0300], // ร€ โ†’ A + ฬ€ + 0x00C8: [0x0045, 0x0300], // รˆ โ†’ E + ฬ€ + 0x00CC: [0x0049, 0x0300], // รŒ โ†’ I + ฬ€ + 0x00D2: [0x004F, 0x0300], // ร’ โ†’ O + ฬ€ + 0x00D9: [0x0055, 0x0300], // ร™ โ†’ U + ฬ€ + 0x00E0: [0x0061, 0x0300], // ร  โ†’ a + ฬ€ + 0x00E8: [0x0065, 0x0300], // รจ โ†’ e + ฬ€ + 0x00EC: [0x0069, 0x0300], // รฌ โ†’ i + ฬ€ + 0x00F2: [0x006F, 0x0300], // รฒ โ†’ o + ฬ€ + 0x00F9: [0x0075, 0x0300], // รน โ†’ u + ฬ€ + // Circumflex + 0x00C2: [0x0041, 0x0302], // ร‚ โ†’ A + ฬ‚ + 0x00CA: [0x0045, 0x0302], // รŠ โ†’ E + ฬ‚ + 0x00CE: [0x0049, 0x0302], // รŽ โ†’ I + ฬ‚ + 0x00D4: [0x004F, 0x0302], // ร” โ†’ O + ฬ‚ + 0x00DB: [0x0055, 0x0302], // ร› โ†’ U + ฬ‚ + 0x00E2: [0x0061, 0x0302], // รข โ†’ a + ฬ‚ + 0x00EA: [0x0065, 0x0302], // รช โ†’ e + ฬ‚ + 0x00EE: [0x0069, 0x0302], // รฎ โ†’ i + ฬ‚ + 0x00F4: [0x006F, 0x0302], // รด โ†’ o + ฬ‚ + 0x00FB: [0x0075, 0x0302], // รป โ†’ u + ฬ‚ + // Tilde + 0x00C3: [0x0041, 0x0303], // รƒ โ†’ A + ฬƒ + 0x00D1: [0x004E, 0x0303], // ร‘ โ†’ N + ฬƒ + 0x00D5: [0x004F, 0x0303], // ร• โ†’ O + ฬƒ + 0x00E3: [0x0061, 0x0303], // รฃ โ†’ a + ฬƒ + 0x00F1: [0x006E, 0x0303], // รฑ โ†’ n + ฬƒ + 0x00F5: [0x006F, 0x0303], // รต โ†’ o + ฬƒ + // Diaeresis/Umlaut + 0x00C4: [0x0041, 0x0308], // ร„ โ†’ A + ฬˆ + 0x00CB: [0x0045, 0x0308], // ร‹ โ†’ E + ฬˆ + 0x00CF: [0x0049, 0x0308], // ร โ†’ I + ฬˆ + 0x00D6: [0x004F, 0x0308], // ร– โ†’ O + ฬˆ + 0x00DC: [0x0055, 0x0308], // รœ โ†’ U + ฬˆ + 0x00E4: [0x0061, 0x0308], // รค โ†’ a + ฬˆ + 0x00EB: [0x0065, 0x0308], // รซ โ†’ e + ฬˆ + 0x00EF: [0x0069, 0x0308], // รฏ โ†’ i + ฬˆ + 0x00F6: [0x006F, 0x0308], // รถ โ†’ o + ฬˆ + 0x00FC: [0x0075, 0x0308], // รผ โ†’ u + ฬˆ + // Cedilla + 0x00C7: [0x0043, 0x0327], // ร‡ โ†’ C + ฬง + 0x00E7: [0x0063, 0x0327], // รง โ†’ c + ฬง +}; + +/// Apply NFKD-like decomposition (Hangul + Latin accented characters) +String _applyNfkdDecomposition(String text) { + final result = []; + for (final codePoint in text.runes) { + // Check Hangul first + if (codePoint >= _hangulSyllableBase && codePoint <= _hangulSyllableEnd) { + result.addAll(_decomposeHangulSyllable(codePoint)); + } + // Check Latin decomposition + else if (_latinDecompositions.containsKey(codePoint)) { + result.addAll(_latinDecompositions[codePoint]!); + } + // Keep as-is + else { + result.add(codePoint); + } + } + return String.fromCharCodes(result); +} + +String preprocessText(String text, String lang) { + // Apply NFKD-like decomposition (especially for Hangul syllables โ†’ Jamo) + text = _applyNfkdDecomposition(text); + + // Remove emojis + text = text.replaceAll( + RegExp( + r'[\u{1F600}-\u{1F64F}]|[\u{1F300}-\u{1F5FF}]|[\u{1F680}-\u{1F6FF}]|' + r'[\u{1F700}-\u{1F77F}]|[\u{1F780}-\u{1F7FF}]|[\u{1F800}-\u{1F8FF}]|' + r'[\u{1F900}-\u{1F9FF}]|[\u{1FA00}-\u{1FA6F}]|[\u{1FA70}-\u{1FAFF}]|' + r'[\u{2600}-\u{26FF}]|[\u{2700}-\u{27BF}]|[\u{1F1E6}-\u{1F1FF}]', + unicode: true, + ), + ''); + + // Replace various dashes and symbols + const replacements = { + 'โ€“': '-', + 'โ€‘': '-', + 'โ€”': '-', + '_': ' ', + '\u201C': '"', + '\u201D': '"', + '\u2018': "'", + '\u2019': "'", + 'ยด': "'", + '`': "'", + '[': ' ', + ']': ' ', + '|': ' ', + '/': ' ', + '#': ' ', + 'โ†’': ' ', + 'โ†': ' ', + }; + for (final entry in replacements.entries) { + text = text.replaceAll(entry.key, entry.value); + } + + // Remove special symbols + text = text.replaceAll(RegExp(r'[โ™ฅโ˜†โ™กยฉ\\]'), ''); + + // Replace known expressions + text = text.replaceAll('@', ' at '); + text = text.replaceAll('e.g.,', 'for example, '); + text = text.replaceAll('i.e.,', 'that is, '); + + // Fix spacing around punctuation + text = text.replaceAll(' ,', ','); + text = text.replaceAll(' .', '.'); + text = text.replaceAll(' !', '!'); + text = text.replaceAll(' ?', '?'); + text = text.replaceAll(' ;', ';'); + text = text.replaceAll(' :', ':'); + text = text.replaceAll(" '", "'"); + + // Remove duplicate quotes + while (text.contains('""')) text = text.replaceAll('""', '"'); + while (text.contains("''")) text = text.replaceAll("''", "'"); + while (text.contains('``')) text = text.replaceAll('``', '`'); + + // Remove extra spaces + text = text.replaceAll(RegExp(r'\s+'), ' ').trim(); + + // Add period if needed + if (text.isNotEmpty && + !RegExp(r'[.!?;:,\x27\x22\u2018\u2019)\]}โ€ฆใ€‚ใ€ใ€ใ€‘ใ€‰ใ€‹โ€บยป]$').hasMatch(text)) { + text += '.'; + } + + // Validate language + if (!isValidLang(lang)) { + throw ArgumentError( + 'Invalid language: $lang. Available: ${availableLangs.join(", ")}'); + } + + // Wrap text with language tags + text = '<$lang>$text'; + + return text; +} + class UnicodeProcessor { final Map indexer; @@ -34,11 +233,17 @@ class UnicodeProcessor { return UnicodeProcessor._(indexer); } - Map call(List textList) { - final lengths = textList.map((t) => t.length).toList(); + Map call(List textList, List langList) { + // Preprocess texts with language tags + final processedTexts = []; + for (var i = 0; i < textList.length; i++) { + processedTexts.add(preprocessText(textList[i], langList[i])); + } + + final lengths = processedTexts.map((t) => t.runes.length).toList(); final maxLen = lengths.reduce(math.max); - final textIds = textList.map((text) { + final textIds = processedTexts.map((text) { final row = List.filled(maxLen, 0); final runes = text.runes.toList(); for (var i = 0; i < runes.length; i++) { @@ -77,14 +282,18 @@ class TextToSpeech { chunkCompressFactor = cfgs['ttl']['chunk_compress_factor'], ldim = cfgs['ttl']['latent_dim']; - Future> call(String text, Style style, int totalStep, + Future> call( + String text, String lang, Style style, int totalStep, {double speed = 1.05, double silenceDuration = 0.3}) async { - final chunks = _chunkText(text); + final maxLen = lang == 'ko' ? 120 : 300; + final chunks = _chunkText(text, maxLen: maxLen); + final langList = List.filled(chunks.length, lang); List? wavCat; double durCat = 0; - for (final chunk in chunks) { - final result = await _infer([chunk], style, totalStep, speed: speed); + for (var i = 0; i < chunks.length; i++) { + final result = await _infer([chunks[i]], [langList[i]], style, totalStep, + speed: speed); final wav = _safeCast(result['wav']); final duration = _safeCast(result['duration']); @@ -108,10 +317,10 @@ class TextToSpeech { } Future> _infer( - List textList, Style style, int totalStep, + List textList, List langList, Style style, int totalStep, {double speed = 1.05}) async { final bsz = textList.length; - final result = textProcessor.call(textList); + final result = textProcessor.call(textList, langList); final textIdsRaw = result['textIds']; final textIds = textIdsRaw is List> diff --git a/flutter/lib/main.dart b/flutter/lib/main.dart index f9b8791..5e1deb2 100644 --- a/flutter/lib/main.dart +++ b/flutter/lib/main.dart @@ -14,7 +14,7 @@ class SupertonicApp extends StatelessWidget { @override Widget build(BuildContext context) { return MaterialApp( - title: 'Supertonic', + title: 'Supertonic 2', theme: ThemeData( colorScheme: ColorScheme.fromSeed(seedColor: Colors.deepPurple), useMaterial3: true, @@ -44,6 +44,7 @@ class _TTSPageState extends State { String _status = 'Not initialized'; int _totalSteps = 5; double _speed = 1.05; + String _selectedLang = 'en'; bool _isPlaying = false; String? _lastGeneratedFilePath; @@ -119,6 +120,7 @@ class _TTSPageState extends State { try { final result = await _textToSpeech!.call( _textController.text, + _selectedLang, _style!, _totalSteps, speed: _speed, @@ -208,7 +210,7 @@ class _TTSPageState extends State { return Scaffold( appBar: AppBar( backgroundColor: Theme.of(context).colorScheme.inversePrimary, - title: const Text('Supertonic'), + title: const Text('Supertonic 2'), ), body: Padding( padding: const EdgeInsets.all(16.0), @@ -310,6 +312,31 @@ class _TTSPageState extends State { ), ], ), + const SizedBox(height: 12), + + // Language selector + Row( + children: [ + const Expanded(flex: 2, child: Text('Language:')), + Expanded( + flex: 3, + child: DropdownButton( + value: _selectedLang, + isExpanded: true, + items: const [ + DropdownMenuItem(value: 'en', child: Text('English')), + DropdownMenuItem(value: 'ko', child: Text('ํ•œ๊ตญ์–ด')), + DropdownMenuItem(value: 'es', child: Text('Espaรฑol')), + DropdownMenuItem(value: 'pt', child: Text('Portuguรชs')), + DropdownMenuItem(value: 'fr', child: Text('Franรงais')), + ], + onChanged: _isLoading || _isGenerating + ? null + : (value) => setState(() => _selectedLang = value!), + ), + ), + ], + ), const SizedBox(height: 24), // Generate button diff --git a/flutter/macos/Podfile.lock b/flutter/macos/Podfile.lock index 10327a2..8ac313f 100644 --- a/flutter/macos/Podfile.lock +++ b/flutter/macos/Podfile.lock @@ -41,11 +41,11 @@ EXTERNAL SOURCES: :path: Flutter/ephemeral/.symlinks/plugins/objective_c/macos SPEC CHECKSUMS: - audio_session: eaca2512cf2b39212d724f35d11f46180ad3a33e - flutter_onnxruntime: 546c7c7630c9259f59021319639b30bdaff148c2 + audio_session: 728ae3823d914f809c485d390274861a24b0904e + flutter_onnxruntime: e6887abc1032d3e5c92f84b912ad42c33e9ce1c9 FlutterMacOS: d0db08ddef1a9af05a5ec4b724367152bb0500b1 - just_audio: 4e391f57b79cad2b0674030a00453ca5ce817eed - objective_c: ec13431e45ba099cb734eb2829a5c1cd37986cba + just_audio: a42c63806f16995daf5b219ae1d679deb76e6a79 + objective_c: e5f8194456e8fc943e034d1af00510a1bc29c067 onnxruntime-c: ac65025f01072d25d7d394a2b43ac30d9397b260 onnxruntime-objc: 5fa03134356d47b642ec85b1023d9907a123d201 diff --git a/go/README.md b/go/README.md index e827f6b..f29ac0f 100644 --- a/go/README.md +++ b/go/README.md @@ -4,6 +4,8 @@ This guide provides examples for running TTS inference using `example_onnx.go`. ## ๐Ÿ“ฐ Update News +**2026.01.06** - ๐ŸŽ‰ **Supertonic 2** released with multilingual support! Now supports English (`en`), Korean (`ko`), Spanish (`es`), Portuguese (`pt`), and French (`fr`). [Demo](https://huggingface.co/spaces/Supertone/supertonic-2) | [Models](https://huggingface.co/Supertone/supertonic-2) + **2025.12.10** - Added [6 new voice styles](https://huggingface.co/Supertone/supertonic/tree/b10dbaf18b316159be75b34d24f740008fddd381) (M3, M4, M5, F3, F4, F5). See [Voices](https://supertone-inc.github.io/supertonic-py/voices/) for details **2025.12.08** - Optimized ONNX models via [OnnxSlim](https://github.com/inisis/OnnxSlim) now available on [Hugging Face Models](https://huggingface.co/Supertone/supertonic) @@ -87,13 +89,14 @@ Process multiple voice styles and texts at once: go run example_onnx.go helper.go \ --batch \ -voice-style "assets/voice_styles/M1.json,assets/voice_styles/F1.json" \ - -text "The sun sets behind the mountains, painting the sky in shades of pink and orange.|The weather is beautiful and sunny outside. A gentle breeze makes the air feel fresh and pleasant." + -text "The sun sets behind the mountains, painting the sky in shades of pink and orange.|์˜ค๋Š˜ ์•„์นจ์— ๊ณต์›์„ ์‚ฐ์ฑ…ํ–ˆ๋Š”๋ฐ, ์ƒˆ์†Œ๋ฆฌ์™€ ๋ฐ”๋žŒ ์†Œ๋ฆฌ๊ฐ€ ๋„ˆ๋ฌด ๊ธฐ๋ถ„ ์ข‹์•˜์–ด์š”." \ + -lang "en,ko" ``` This will: -- Generate speech for 2 different voice-text pairs -- Use male voice (M1.json) for the first text -- Use female voice (F1.json) for the second text +- Generate speech for 2 different voice-text-language pairs +- Use male voice (M1.json) for the first text in English +- Use female voice (F1.json) for the second text in Korean - Process both samples in a single batch ### Example 3: High Quality Inference @@ -136,12 +139,14 @@ This will: | `-n-test` | int | 4 | Number of times to generate each sample | | `-voice-style` | str | `assets/voice_styles/M1.json` | Voice style file path(s), comma-separated | | `-text` | str | (long default text) | Text(s) to synthesize, pipe-separated | +| `-lang` | str | `en` | Language(s) for synthesis, comma-separated (en, ko, es, pt, fr) | | `-save-dir` | str | `results` | Output directory | | `--batch` | flag | false | Enable batch mode (multiple text-style pairs, disables automatic chunking) | ## Notes -- **Batch Processing**: When using `--batch`, the number of `-voice-style` files must match the number of `-text` entries +- **Multilingual Support**: Use `-lang` to specify the language for each text. Available: `en` (English), `ko` (Korean), `es` (Spanish), `pt` (Portuguese), `fr` (French) +- **Batch Processing**: When using `--batch`, the number of `-voice-style`, `-text`, and `-lang` entries must match - **Automatic Chunking**: Without `--batch`, long texts are automatically split and concatenated with 0.3s pauses - **Quality vs Speed**: Higher `-total-step` values produce better quality but take longer - **GPU Support**: GPU mode is not supported yet diff --git a/go/example_onnx.go b/go/example_onnx.go index 463b5b7..ae3371c 100644 --- a/go/example_onnx.go +++ b/go/example_onnx.go @@ -19,6 +19,7 @@ type Args struct { nTest int voiceStyle []string text []string + lang []string saveDir string batch bool } @@ -34,9 +35,10 @@ func parseArgs() *Args { flag.StringVar(&args.saveDir, "save-dir", "results", "Output directory") flag.BoolVar(&args.batch, "batch", false, "Enable batch mode (multiple text-style pairs)") - var voiceStyleStr, textStr string + var voiceStyleStr, textStr, langStr string flag.StringVar(&voiceStyleStr, "voice-style", "assets/voice_styles/M1.json", "Voice style file path(s), comma-separated") flag.StringVar(&textStr, "text", "This morning, I took a walk in the park, and the sound of the birds and the breeze was so pleasant that I stopped for a long time just to listen.", "Text(s) to synthesize, pipe-separated") + flag.StringVar(&langStr, "lang", "en", "Language(s) for synthesis, comma-separated (en, ko, es, pt, fr)") flag.Parse() @@ -56,6 +58,14 @@ func parseArgs() *Args { } } + // Parse comma-separated lang + if langStr != "" { + args.lang = strings.Split(langStr, ",") + for i := range args.lang { + args.lang[i] = strings.TrimSpace(args.lang[i]) + } + } + return args } @@ -70,6 +80,7 @@ func main() { saveDir := args.saveDir voiceStylePaths := args.voiceStyle textList := args.text + langList := args.lang batch := args.batch if batch { @@ -78,6 +89,11 @@ func main() { len(voiceStylePaths), len(textList)) os.Exit(1) } + if len(langList) != len(textList) { + fmt.Printf("Error: Number of languages (%d) must match number of texts (%d)\n", + len(langList), len(textList)) + os.Exit(1) + } } bsz := len(voiceStylePaths) @@ -126,7 +142,7 @@ func main() { if batch { Timer("Generating speech from text", func() interface{} { - w, d, err := textToSpeech.Batch(textList, style, totalStep, speed) + w, d, err := textToSpeech.Batch(textList, langList, style, totalStep, speed) if err != nil { fmt.Printf("Error generating speech: %v\n", err) os.Exit(1) @@ -137,7 +153,7 @@ func main() { }) } else { Timer("Generating speech from text", func() interface{} { - w, d, err := textToSpeech.Call(textList[0], style, totalStep, speed, 0.3) + w, d, err := textToSpeech.Call(textList[0], langList[0], style, totalStep, speed, 0.3) if err != nil { fmt.Printf("Error generating speech: %v\n", err) os.Exit(1) diff --git a/go/go.mod b/go/go.mod index d8685ca..65ea362 100644 --- a/go/go.mod +++ b/go/go.mod @@ -7,6 +7,7 @@ require ( github.com/go-audio/wav v1.1.0 github.com/mjibson/go-dsp v0.0.0-20180508042940-11479a337f12 github.com/yalue/onnxruntime_go v1.11.0 + golang.org/x/text v0.14.0 ) require github.com/go-audio/riff v1.0.0 // indirect diff --git a/go/go.sum b/go/go.sum index c456c20..b5a796f 100644 --- a/go/go.sum +++ b/go/go.sum @@ -8,3 +8,5 @@ github.com/mjibson/go-dsp v0.0.0-20180508042940-11479a337f12 h1:dd7vnTDfjtwCETZD github.com/mjibson/go-dsp v0.0.0-20180508042940-11479a337f12/go.mod h1:i/KKcxEWEO8Yyl11DYafRPKOPVYTrhxiTRigjtEEXZU= github.com/yalue/onnxruntime_go v1.11.0 h1:aKH4yPIbqfcB3SfnQWq/WxzLelkyolntHnffL3eMBHY= github.com/yalue/onnxruntime_go v1.11.0/go.mod h1:b4X26A8pekNb1ACJ58wAXgNKeUCGEAQ9dmACut9Sm/4= +golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ= +golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= diff --git a/go/helper.go b/go/helper.go index a116060..5c795f9 100644 --- a/go/helper.go +++ b/go/helper.go @@ -10,12 +10,17 @@ import ( "regexp" "strings" "time" + "unicode" "github.com/go-audio/audio" "github.com/go-audio/wav" ort "github.com/yalue/onnxruntime_go" + "golang.org/x/text/unicode/norm" ) +// Available languages for multilingual TTS +var AvailableLangs = []string{"en", "ko", "es", "pt", "fr"} + // Config structures type SpecProcessorConfig struct { NFFT int `json:"n_fft"` @@ -108,11 +113,11 @@ func NewUnicodeProcessor(unicodeIndexerPath string) (*UnicodeProcessor, error) { } // Call processes text list to text IDs and mask -func (up *UnicodeProcessor) Call(textList []string) ([][]int64, [][][]float64) { +func (up *UnicodeProcessor) Call(textList []string, langList []string) ([][]int64, [][][]float64) { // Preprocess texts processedTexts := make([]string, len(textList)) for i, text := range textList { - processedTexts[i] = preprocessText(text) + processedTexts[i] = preprocessText(text, langList[i]) } // Get text lengths @@ -325,14 +330,21 @@ func splitSentences(text string) []string { return sentences } -// Utility functions -func preprocessText(text string) string { - // TODO: Need advanced normalizer for better performance - // NOTE: Go doesn't have built-in NFKD normalization like Python - // For full Unicode normalization, use golang.org/x/text/unicode/norm - // This implementation handles basic text preprocessing +// isValidLang checks if a language is in the available languages list +func isValidLang(lang string) bool { + for _, l := range AvailableLangs { + if l == lang { + return true + } + } + return false +} - // FIXME: this should be fixed for non-English languages +// Utility functions +func preprocessText(text string, lang string) string { + // TODO: Need advanced normalizer for better performance + // Apply NFKD normalization using golang.org/x/text/unicode/norm + text = norm.NFKD.String(text) // Remove emojis and various Unicode symbols emojiPattern := regexp.MustCompile(`[\x{1F600}-\x{1F64F}\x{1F300}-\x{1F5FF}\x{1F680}-\x{1F6FF}\x{1F700}-\x{1F77F}\x{1F780}-\x{1F7FF}\x{1F800}-\x{1F8FF}\x{1F900}-\x{1F9FF}\x{1FA00}-\x{1FA6F}\x{1FA70}-\x{1FAFF}\x{2600}-\x{26FF}\x{2700}-\x{27BF}\x{1F1E6}-\x{1F1FF}]+`) @@ -343,7 +355,6 @@ func preprocessText(text string) string { "โ€“": "-", // en dash "โ€‘": "-", // non-breaking hyphen "โ€”": "-", // em dash - "ยฏ": " ", // macron "_": " ", // underscore "\u201C": "\"", // left double quote "\u201D": "\"", // right double quote @@ -364,11 +375,6 @@ func preprocessText(text string) string { text = strings.ReplaceAll(text, old, new) } - // Remove combining diacritics (common combining marks) - // FIXME: this should be fixed for non-English languages - diacriticsPattern := regexp.MustCompile(`[\x{0302}\x{0303}\x{0304}\x{0305}\x{0306}\x{0307}\x{0308}\x{030A}\x{030B}\x{030C}\x{0327}\x{0328}\x{0329}\x{032A}\x{032B}\x{032C}\x{032D}\x{032E}\x{032F}]`) - text = diacriticsPattern.ReplaceAllString(text, "") - // Remove special symbols specialSymbols := []string{"โ™ฅ", "โ˜†", "โ™ก", "ยฉ", "\\"} for _, symbol := range specialSymbols { @@ -418,6 +424,14 @@ func preprocessText(text string) string { } } + // Validate language + if !isValidLang(lang) { + panic(fmt.Sprintf("Invalid language: %s. Available: %v", lang, AvailableLangs)) + } + + // Wrap text with language tags + text = fmt.Sprintf("<%s>%s", lang, text, lang) + return text } @@ -661,11 +675,11 @@ func (tts *TextToSpeech) sampleNoisyLatent(durOnnx []float32) ([][][]float64, [] return noisyLatent, latentMask } -func (tts *TextToSpeech) _infer(textList []string, style *Style, totalStep int, speed float32) ([]float32, []float32, error) { +func (tts *TextToSpeech) _infer(textList []string, langList []string, style *Style, totalStep int, speed float32) ([]float32, []float32, error) { bsz := len(textList) // Process text - textIDs, textMask := tts.textProcessor.Call(textList) + textIDs, textMask := tts.textProcessor.Call(textList, langList) textIDsShape := []int64{int64(bsz), int64(len(textIDs[0]))} textMaskShape := []int64{int64(bsz), 1, int64(len(textMask[0][0]))} @@ -785,14 +799,18 @@ func (tts *TextToSpeech) _infer(textList []string, style *Style, totalStep int, } // Call synthesizes speech from a single text with automatic chunking -func (tts *TextToSpeech) Call(text string, style *Style, totalStep int, speed float32, silenceDuration float32) ([]float32, float32, error) { - chunks := chunkText(text, 0) +func (tts *TextToSpeech) Call(text string, lang string, style *Style, totalStep int, speed float32, silenceDuration float32) ([]float32, float32, error) { + maxLen := 300 + if lang == "ko" { + maxLen = 120 + } + chunks := chunkText(text, maxLen) var wavCat []float32 var durCat float32 for i, chunk := range chunks { - wav, duration, err := tts._infer([]string{chunk}, style, totalStep, speed) + wav, duration, err := tts._infer([]string{chunk}, []string{lang}, style, totalStep, speed) if err != nil { return nil, 0, err } @@ -818,8 +836,8 @@ func (tts *TextToSpeech) Call(text string, style *Style, totalStep int, speed fl } // Batch synthesizes speech from multiple texts -func (tts *TextToSpeech) Batch(textList []string, style *Style, totalStep int, speed float32) ([]float32, []float32, error) { - return tts._infer(textList, style, totalStep, speed) +func (tts *TextToSpeech) Batch(textList []string, langList []string, style *Style, totalStep int, speed float32) ([]float32, []float32, error) { + return tts._infer(textList, langList, style, totalStep, speed) } func (tts *TextToSpeech) Destroy() { @@ -917,15 +935,17 @@ func InitializeONNXRuntime() error { return nil } -// sanitizeFilename creates a safe filename from text +// sanitizeFilename creates a safe filename from text (supports Unicode) func sanitizeFilename(text string, maxLen int) string { - if len(text) > maxLen { - text = text[:maxLen] + runes := []rune(text) + if len(runes) > maxLen { + runes = runes[:maxLen] } - result := make([]rune, 0, len(text)) - for _, r := range text { - if (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9') { + result := make([]rune, 0, len(runes)) + for _, r := range runes { + // unicode.IsLetter matches any Unicode letter, unicode.IsDigit matches any Unicode digit + if unicode.IsLetter(r) || unicode.IsDigit(r) { result = append(result, r) } else { result = append(result, '_') diff --git a/img/Supertonic_IMG_v02.svg b/img/Supertonic_IMG_v02.svg deleted file mode 100644 index 9237279..0000000 --- a/img/Supertonic_IMG_v02.svg +++ /dev/null @@ -1,250 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/img/Supertonic_IMG_v02_4x.webp b/img/Supertonic_IMG_v02_4x.webp deleted file mode 100644 index 78b0e03..0000000 Binary files a/img/Supertonic_IMG_v02_4x.webp and /dev/null differ diff --git a/img/supertonic_preview_0.1.jpg b/img/supertonic_preview_0.1.jpg new file mode 100644 index 0000000..be74424 Binary files /dev/null and b/img/supertonic_preview_0.1.jpg differ diff --git a/ios/ExampleiOSApp/ContentView.swift b/ios/ExampleiOSApp/ContentView.swift index ee236b4..b9c6e95 100644 --- a/ios/ExampleiOSApp/ContentView.swift +++ b/ios/ExampleiOSApp/ContentView.swift @@ -12,7 +12,7 @@ struct ContentView: View { Spacer() VStack(spacing: 12) { - Text("SupertonicTTS iOS Demo") + Text("Supertonic 2 iOS Demo") .font(.title2.weight(.semibold)) .foregroundColor(.primary) @@ -44,6 +44,19 @@ struct ContentView: View { } .pickerStyle(SegmentedPickerStyle()) .padding(.horizontal) + + HStack(spacing: 12) { + Text("Language") + .font(.subheadline) + .foregroundColor(.secondary) + Picker("Language", selection: $vm.language) { + ForEach(TTSService.Language.allCases, id: \.self) { lang in + Text(lang.displayName).tag(lang) + } + } + .pickerStyle(MenuPickerStyle()) + } + .padding(.horizontal) } HStack(spacing: 16) { diff --git a/ios/ExampleiOSApp/TTSService.swift b/ios/ExampleiOSApp/TTSService.swift index 4c446e5..a0558da 100644 --- a/ios/ExampleiOSApp/TTSService.swift +++ b/ios/ExampleiOSApp/TTSService.swift @@ -3,6 +3,23 @@ import OnnxRuntimeBindings final class TTSService { enum Voice { case male, female } + enum Language: String, CaseIterable { + case en = "en" + case ko = "ko" + case es = "es" + case pt = "pt" + case fr = "fr" + + var displayName: String { + switch self { + case .en: return "English" + case .ko: return "ํ•œ๊ตญ์–ด" + case .es: return "Espaรฑol" + case .pt: return "Portuguรชs" + case .fr: return "Franรงais" + } + } + } private let env: ORTEnv private let textToSpeech: TextToSpeech @@ -16,13 +33,13 @@ final class TTSService { sampleRate = textToSpeech.sampleRate } - func synthesize(text: String, nfe: Int, voice: Voice) async throws -> URL { + func synthesize(text: String, nfe: Int, voice: Voice, language: Language) async throws -> URL { // Load style for the selected voice let styleURL = try Self.locateVoiceStyleURL(voice: voice) let style = try loadVoiceStyle([styleURL.path], verbose: false) // 2) Synthesize via packed TextToSpeech component - let (wav, duration) = try textToSpeech.call(text, style, nfe) + let (wav, duration) = try textToSpeech.call(text, language.rawValue, style, nfe) let audioSeconds = Double(duration) let wavLenSample = min(Int(Double(sampleRate) * audioSeconds), wav.count) let wavOut = Array(wav[0.. text = Arrays.asList( "This morning, I took a walk in the park, and the sound of the birds and the breeze was so pleasant that I stopped for a long time just to listen." ); + List lang = Arrays.asList("en"); String saveDir = "results"; boolean batch = false; } @@ -58,6 +59,11 @@ public class ExampleONNX { result.text = Arrays.asList(args[++i].split("\\|")); } break; + case "--lang": + if (i + 1 < args.length) { + result.lang = Arrays.asList(args[++i].split(",")); + } + break; case "--save-dir": if (i + 1 < args.length) result.saveDir = args[++i]; break; @@ -85,6 +91,7 @@ public class ExampleONNX { String saveDir = parsedArgs.saveDir; List voiceStylePaths = parsedArgs.voiceStyle; List textList = parsedArgs.text; + List langList = parsedArgs.lang; boolean batch = parsedArgs.batch; if (batch) { @@ -92,6 +99,10 @@ public class ExampleONNX { throw new RuntimeException("Number of voice styles (" + voiceStylePaths.size() + ") must match number of texts (" + textList.size() + ")"); } + if (langList.size() != textList.size()) { + throw new RuntimeException("Number of languages (" + langList.size() + + ") must match number of texts (" + textList.size() + ")"); + } } int bsz = voiceStylePaths.size(); @@ -116,7 +127,7 @@ public class ExampleONNX { if (batch) { ttsResult = Helper.timer("Generating speech from text", () -> { try { - return textToSpeech.batch(textList, style, totalStep, speed, env); + return textToSpeech.batch(textList, langList, style, totalStep, speed, env); } catch (Exception e) { throw new RuntimeException(e); } @@ -124,7 +135,7 @@ public class ExampleONNX { } else { ttsResult = Helper.timer("Generating speech from text", () -> { try { - return textToSpeech.call(textList.get(0), style, totalStep, speed, 0.3f, env); + return textToSpeech.call(textList.get(0), langList.get(0), style, totalStep, speed, 0.3f, env); } catch (Exception e) { throw new RuntimeException(e); } diff --git a/java/Helper.java b/java/Helper.java index a75ade4..dce0770 100644 --- a/java/Helper.java +++ b/java/Helper.java @@ -18,6 +18,17 @@ import java.util.*; import java.util.regex.Pattern; import java.util.regex.Matcher; +/** + * Available languages for multilingual TTS + */ +class Languages { + public static final List AVAILABLE = Arrays.asList("en", "ko", "es", "pt", "fr"); + + public static boolean isValid(String lang) { + return AVAILABLE.contains(lang); + } +} + /** * Configuration classes */ @@ -96,22 +107,28 @@ class UnicodeProcessor { return result.toString(); } - public TextProcessResult call(List textList) { + public TextProcessResult call(List textList, List langList) { List processedTexts = new ArrayList<>(); - for (String text : textList) { - processedTexts.add(preprocessText(text)); + for (int i = 0; i < textList.size(); i++) { + processedTexts.add(preprocessText(textList.get(i), langList.get(i))); + } + + // Convert texts to unicode values first to get correct character counts + List allUnicodeVals = new ArrayList<>(); + for (String text : processedTexts) { + allUnicodeVals.add(textToUnicodeValues(text)); } int[] textIdsLengths = new int[processedTexts.size()]; int maxLen = 0; - for (int i = 0; i < processedTexts.size(); i++) { - textIdsLengths[i] = processedTexts.get(i).length(); + for (int i = 0; i < allUnicodeVals.size(); i++) { + textIdsLengths[i] = allUnicodeVals.get(i).length; // Use code point count, not char count maxLen = Math.max(maxLen, textIdsLengths[i]); } long[][] textIds = new long[processedTexts.size()][maxLen]; - for (int i = 0; i < processedTexts.size(); i++) { - int[] unicodeVals = textToUnicodeValues(processedTexts.get(i)); + for (int i = 0; i < allUnicodeVals.size(); i++) { + int[] unicodeVals = allUnicodeVals.get(i); for (int j = 0; j < unicodeVals.length; j++) { textIds[i][j] = indexer[unicodeVals[j]]; } @@ -121,12 +138,10 @@ class UnicodeProcessor { return new TextProcessResult(textIds, textMask); } - private String preprocessText(String text) { + private String preprocessText(String text, String lang) { // TODO: Need advanced normalizer for better performance text = Normalizer.normalize(text, Normalizer.Form.NFKD); - // FIXME: this should be fixed for non-English languages - // Remove emojis (wide Unicode range) // Java Pattern doesn't support \x{...} syntax for Unicode above \uFFFF // Use character filtering instead @@ -137,7 +152,6 @@ class UnicodeProcessor { replacements.put("โ€“", "-"); // en dash replacements.put("โ€‘", "-"); // non-breaking hyphen replacements.put("โ€”", "-"); // em dash - replacements.put("ยฏ", " "); // macron replacements.put("_", " "); // underscore replacements.put("\u201C", "\""); // left double quote replacements.put("\u201D", "\""); // right double quote @@ -157,9 +171,6 @@ class UnicodeProcessor { text = text.replace(entry.getKey(), entry.getValue()); } - // Remove combining diacritics // FIXME: this should be fixed for non-English languages - text = text.replaceAll("[\\u0302\\u0303\\u0304\\u0305\\u0306\\u0307\\u0308\\u030A\\u030B\\u030C\\u0327\\u0328\\u0329\\u032A\\u032B\\u032C\\u032D\\u032E\\u032F]", ""); - // Remove special symbols text = text.replaceAll("[โ™ฅโ˜†โ™กยฉ\\\\]", ""); @@ -201,15 +212,20 @@ class UnicodeProcessor { text += "."; } + // Validate language + if (!Languages.isValid(lang)) { + throw new IllegalArgumentException("Invalid language: " + lang + ". Available: " + Languages.AVAILABLE); + } + + // Wrap text with language tags + text = "<" + lang + ">" + text + ""; + return text; } private int[] textToUnicodeValues(String text) { - int[] values = new int[text.length()]; - for (int i = 0; i < text.length(); i++) { - values[i] = text.codePointAt(i); - } - return values; + // Use codePoints() stream to correctly handle surrogate pairs + return text.codePoints().toArray(); } private float[][][] getTextMask(int[] lengths) { @@ -269,12 +285,12 @@ class TextToSpeech { this.ldim = config.ttl.latentDim; } - private TTSResult _infer(List textList, Style style, int totalStep, float speed, OrtEnvironment env) + private TTSResult _infer(List textList, List langList, Style style, int totalStep, float speed, OrtEnvironment env) throws OrtException { int bsz = textList.size(); // Process text - UnicodeProcessor.TextProcessResult textResult = textProcessor.call(textList); + UnicodeProcessor.TextProcessResult textResult = textProcessor.call(textList, langList); long[][] textIds = textResult.textIds; float[][][] textMask = textResult.textMask; @@ -360,7 +376,18 @@ class TextToSpeech { OrtSession.Result vocoderResult = vocoderSession.run(vocoderInputs); float[][] wavBatch = (float[][]) vocoderResult.get(0).getValue(); - float[] wav = wavBatch[0]; + + // Flatten all batch audio into a single array for batch processing + int totalSamples = 0; + for (float[] w : wavBatch) { + totalSamples += w.length; + } + float[] wav = new float[totalSamples]; + int offset = 0; + for (float[] w : wavBatch) { + System.arraycopy(w, 0, wav, offset, w.length); + offset += w.length; + } // Clean up textIdsTensor.close(); @@ -421,15 +448,16 @@ class TextToSpeech { /** * Synthesize speech from a single text with automatic chunking */ - public TTSResult call(String text, Style style, int totalStep, float speed, float silenceDuration, OrtEnvironment env) + public TTSResult call(String text, String lang, Style style, int totalStep, float speed, float silenceDuration, OrtEnvironment env) throws OrtException { - List chunks = Helper.chunkText(text, 0); + int maxLen = lang.equals("ko") ? 120 : 300; + List chunks = Helper.chunkText(text, maxLen); List wavCat = new ArrayList<>(); float durCat = 0.0f; for (int i = 0; i < chunks.size(); i++) { - TTSResult result = _infer(Arrays.asList(chunks.get(i)), style, totalStep, speed, env); + TTSResult result = _infer(Arrays.asList(chunks.get(i)), Arrays.asList(lang), style, totalStep, speed, env); float dur = result.duration[0]; int wavLen = (int) (sampleRate * dur); @@ -464,9 +492,9 @@ class TextToSpeech { /** * Batch synthesize speech from multiple texts */ - public TTSResult batch(List textList, Style style, int totalStep, float speed, OrtEnvironment env) + public TTSResult batch(List textList, List langList, Style style, int totalStep, float speed, OrtEnvironment env) throws OrtException { - return _infer(textList, style, totalStep, speed, env); + return _infer(textList, langList, style, totalStep, speed, env); } public void close() throws OrtException { @@ -841,13 +869,20 @@ public class Helper { } /** - * Sanitize filename + * Sanitize filename (supports Unicode characters) */ public static String sanitizeFilename(String text, int maxLen) { - if (text.length() > maxLen) { - text = text.substring(0, maxLen); + // Get first maxLen characters (code points, not chars for surrogate pairs) + int[] codePoints = text.codePoints().limit(maxLen).toArray(); + StringBuilder result = new StringBuilder(); + for (int codePoint : codePoints) { + if (Character.isLetterOrDigit(codePoint)) { + result.appendCodePoint(codePoint); + } else { + result.append('_'); + } } - return text.replaceAll("[^a-zA-Z0-9]", "_"); + return result.toString(); } /** diff --git a/java/README.md b/java/README.md index 5eb45c7..d6bc125 100644 --- a/java/README.md +++ b/java/README.md @@ -4,6 +4,8 @@ This guide provides examples for running TTS inference using `ExampleONNX.java`. ## ๐Ÿ“ฐ Update News +**2026.01.06** - ๐ŸŽ‰ **Supertonic 2** released with multilingual support! Now supports English (`en`), Korean (`ko`), Spanish (`es`), Portuguese (`pt`), and French (`fr`). [Demo](https://huggingface.co/spaces/Supertone/supertonic-2) | [Models](https://huggingface.co/Supertone/supertonic-2) + **2025.12.10** - Added [6 new voice styles](https://huggingface.co/Supertone/supertonic/tree/b10dbaf18b316159be75b34d24f740008fddd381) (M3, M4, M5, F3, F4, F5). See [Voices](https://supertone-inc.github.io/supertonic-py/voices/) for details **2025.12.08** - Optimized ONNX models via [OnnxSlim](https://github.com/inisis/OnnxSlim) now available on [Hugging Face Models](https://huggingface.co/Supertone/supertonic) @@ -47,13 +49,13 @@ This will use: ### Example 2: Batch Inference Process multiple voice styles and texts at once: ```bash -mvn exec:java -Dexec.args="--batch --voice-style assets/voice_styles/M1.json,assets/voice_styles/F1.json --text 'The sun sets behind the mountains, painting the sky in shades of pink and orange.|The weather is beautiful and sunny outside. A gentle breeze makes the air feel fresh and pleasant.'" +mvn exec:java -Dexec.args="--batch --voice-style assets/voice_styles/M1.json,assets/voice_styles/F1.json --text 'The sun sets behind the mountains, painting the sky in shades of pink and orange.|์˜ค๋Š˜ ์•„์นจ์— ๊ณต์›์„ ์‚ฐ์ฑ…ํ–ˆ๋Š”๋ฐ, ์ƒˆ์†Œ๋ฆฌ์™€ ๋ฐ”๋žŒ ์†Œ๋ฆฌ๊ฐ€ ๋„ˆ๋ฌด ๊ธฐ๋ถ„ ์ข‹์•˜์–ด์š”.' --lang en,ko" ``` This will: -- Generate speech for 2 different voice-text pairs -- Use male voice (M1.json) for the first text -- Use female voice (F1.json) for the second text +- Generate speech for 2 different voice-text-language pairs +- Use male voice (M1.json) for the first text in English +- Use female voice (F1.json) for the second text in Korean - Process both samples in a single batch ### Example 3: High Quality Inference @@ -111,14 +113,16 @@ java -jar target/tts-example.jar --total-step 10 --text "Your custom text here" | `--onnx-dir` | str | `assets/onnx` | Path to ONNX model directory | | `--total-step` | int | 5 | Number of denoising steps (higher = better quality, slower) | | `--n-test` | int | 4 | Number of times to generate each sample | -| `--voice-style` | str+ | `assets/voice_styles/M1.json` | Voice style file path(s) | -| `--text` | str+ | (long default text) | Text(s) to synthesize | +| `--voice-style` | str+ | `assets/voice_styles/M1.json` | Voice style file path(s), comma-separated | +| `--text` | str+ | (long default text) | Text(s) to synthesize, pipe-separated | +| `--lang` | str+ | `en` | Language(s) for synthesis, comma-separated (en, ko, es, pt, fr) | | `--save-dir` | str | `results` | Output directory | | `--batch` | flag | False | Enable batch mode (multiple text-style pairs, disables automatic chunking) | ## Notes -- **Batch Processing**: When using `--batch`, the number of `--voice-style` files must match the number of `--text` entries +- **Multilingual Support**: Use `--lang` to specify the language for each text. Available: `en` (English), `ko` (Korean), `es` (Spanish), `pt` (Portuguese), `fr` (French) +- **Batch Processing**: When using `--batch`, the number of `--voice-style`, `--text`, and `--lang` entries must match - **Automatic Chunking**: Without `--batch`, long texts are automatically split and concatenated with 0.3s pauses - **Quality vs Speed**: Higher `--total-step` values produce better quality but take longer - **GPU Support**: GPU mode is not supported yet diff --git a/nodejs/README.md b/nodejs/README.md index 390058f..4629092 100644 --- a/nodejs/README.md +++ b/nodejs/README.md @@ -4,6 +4,8 @@ Node.js implementation for TTS inference. Uses ONNX Runtime to generate speech f ## ๐Ÿ“ฐ Update News +**2026.01.06** - ๐ŸŽ‰ **Supertonic 2** released with multilingual support! Now supports English (`en`), Korean (`ko`), Spanish (`es`), Portuguese (`pt`), and French (`fr`). [Demo](https://huggingface.co/spaces/Supertone/supertonic-2) | [Models](https://huggingface.co/Supertone/supertonic-2) + **2025.12.10** - Added [6 new voice styles](https://huggingface.co/Supertone/supertonic/tree/b10dbaf18b316159be75b34d24f740008fddd381) (M3, M4, M5, F3, F4, F5). See [Voices](https://supertone-inc.github.io/supertonic-py/voices/) for details **2025.12.08** - Optimized ONNX models via [OnnxSlim](https://github.com/inisis/OnnxSlim) now available on [Hugging Face Models](https://huggingface.co/Supertone/supertonic) @@ -51,15 +53,16 @@ Process multiple voice styles and texts at once: ```bash node example_onnx.js \ --voice-style "assets/voice_styles/M1.json,assets/voice_styles/F1.json" \ - --text "The sun sets behind the mountains, painting the sky in shades of pink and orange.|The weather is beautiful and sunny outside. A gentle breeze makes the air feel fresh and pleasant." \ + --text "The sun sets behind the mountains, painting the sky in shades of pink and orange.|์˜ค๋Š˜ ์•„์นจ์— ๊ณต์›์„ ์‚ฐ์ฑ…ํ–ˆ๋Š”๋ฐ, ์ƒˆ์†Œ๋ฆฌ์™€ ๋ฐ”๋žŒ ์†Œ๋ฆฌ๊ฐ€ ๋„ˆ๋ฌด ์ข‹์•„์„œ ํ•œ์ฐธ์„ ๋ฉˆ์ถฐ ์„œ์„œ ๋“ค์—ˆ์–ด์š”." \ + --lang "en,ko" \ --batch ``` This will: - Use `--batch` flag to enable batch processing mode - Generate speech for 2 different voice-text pairs -- Use male voice style (M1.json) for the first text -- Use female voice style (F1.json) for the second text +- Use male voice style (M1.json) for the first English text +- Use female voice style (F1.json) for the second Korean text - Process both samples in a single batch (automatic text chunking disabled) ### Example 3: High Quality Inference @@ -98,15 +101,18 @@ This will: | `--use-gpu` | flag | False | Use GPU for inference (not supported yet) | | `--onnx-dir` | str | `assets/onnx` | Path to ONNX model directory | | `--total-step` | int | 5 | Number of denoising steps (higher = better quality, slower) | +| `--speed` | float | 1.05 | Speech speed factor (higher = faster, lower = slower) | | `--n-test` | int | 4 | Number of times to generate each sample | | `--voice-style` | str+ | `assets/voice_styles/M1.json` | Voice style file path(s). Separate multiple files with commas | | `--text` | str+ | (long default text) | Text(s) to synthesize. Separate multiple texts with pipes | +| `--lang` | str+ | `en` | Language(s) for text(s): `en`, `ko`, `es`, `pt`, `fr`. Separate multiple with commas | | `--save-dir` | str | `results` | Output directory | | `--batch` | flag | False | Enable batch mode (disables automatic text chunking) | ## Notes - **Batch Processing**: The number of voice style files must match the number of texts. Use commas to separate files and pipes to separate texts +- **Multilingual Support**: Use `--lang` to specify language(s). Available: `en` (English), `ko` (Korean), `es` (Spanish), `pt` (Portuguese), `fr` (French) - **Long-Form Inference**: Without `--batch` flag, long texts are automatically chunked and combined into a single audio file with natural pauses - **Quality vs Speed**: Higher `--total-step` values produce better quality but take longer - **GPU Support**: GPU mode is not supported yet diff --git a/nodejs/example_onnx.js b/nodejs/example_onnx.js index 46184c8..eb28a3a 100644 --- a/nodejs/example_onnx.js +++ b/nodejs/example_onnx.js @@ -2,7 +2,7 @@ import fs from 'fs'; import path from 'path'; import { fileURLToPath } from 'url'; -import { loadTextToSpeech, loadVoiceStyle, timer, writeWavFile } from './helper.js'; +import { loadTextToSpeech, loadVoiceStyle, timer, writeWavFile, sanitizeFilename } from './helper.js'; const __filename = fileURLToPath(import.meta.url); const __dirname = path.dirname(__filename); @@ -19,6 +19,7 @@ function parseArgs() { nTest: 4, voiceStyle: ['assets/voice_styles/M1.json'], text: ['This morning, I took a walk in the park, and the sound of the birds and the breeze was so pleasant that I stopped for a long time just to listen.'], + lang: ['en'], saveDir: 'results', batch: false }; @@ -41,6 +42,8 @@ function parseArgs() { args.voiceStyle = process.argv[++i].split(','); } else if (arg === '--text' && i + 1 < process.argv.length) { args.text = process.argv[++i].split('|'); + } else if (arg === '--lang' && i + 1 < process.argv.length) { + args.lang = process.argv[++i].split(','); } else if (arg === '--save-dir' && i + 1 < process.argv.length) { args.saveDir = process.argv[++i]; } @@ -63,6 +66,7 @@ async function main() { const saveDir = args.saveDir; const voiceStylePaths = args.voiceStyle.map(p => path.resolve(__dirname, p)); const textList = args.text; + const langList = args.lang; const batch = args.batch; if (voiceStylePaths.length !== textList.length) { @@ -83,9 +87,9 @@ async function main() { const { wav, duration } = await timer('Generating speech from text', async () => { if (batch) { - return await textToSpeech.batch(textList, style, totalStep, speed); + return await textToSpeech.batch(textList, langList, style, totalStep, speed); } else { - return await textToSpeech.call(textList[0], style, totalStep, speed); + return await textToSpeech.call(textList[0], langList[0], style, totalStep, speed); } }); @@ -95,7 +99,7 @@ async function main() { const wavShape = [bsz, wav.length / bsz]; for (let b = 0; b < bsz; b++) { - const fname = `${textList[b].substring(0, 20).replace(/[^a-zA-Z0-9]/g, '_')}_${n + 1}.wav`; + const fname = `${sanitizeFilename(textList[b], 20)}_${n + 1}.wav`; const wavLen = Math.floor(textToSpeech.sampleRate * duration[b]); const wavOut = wav.slice(b * wavShape[1], b * wavShape[1] + wavLen); diff --git a/nodejs/helper.js b/nodejs/helper.js index 12faf7a..1cde419 100644 --- a/nodejs/helper.js +++ b/nodejs/helper.js @@ -5,6 +5,8 @@ import * as ort from 'onnxruntime-node'; const __filename = fileURLToPath(import.meta.url); +const AVAILABLE_LANGS = ["en", "ko", "es", "pt", "fr"]; + /** * Unicode text processor */ @@ -13,12 +15,10 @@ class UnicodeProcessor { this.indexer = JSON.parse(fs.readFileSync(unicodeIndexerJsonPath, 'utf8')); } - _preprocessText(text) { + _preprocessText(text, lang) { // TODO: Need advanced normalizer for better performance text = text.normalize('NFKD'); - // FIXME: this should be fixed for non-English languages - // Remove emojis (wide Unicode range) const emojiPattern = /[\u{1F600}-\u{1F64F}\u{1F300}-\u{1F5FF}\u{1F680}-\u{1F6FF}\u{1F700}-\u{1F77F}\u{1F780}-\u{1F7FF}\u{1F800}-\u{1F8FF}\u{1F900}-\u{1F9FF}\u{1FA00}-\u{1FA6F}\u{1FA70}-\u{1FAFF}\u{2600}-\u{26FF}\u{2700}-\u{27BF}\u{1F1E6}-\u{1F1FF}]+/gu; text = text.replace(emojiPattern, ''); @@ -28,7 +28,6 @@ class UnicodeProcessor { 'โ€“': '-', 'โ€‘': '-', 'โ€”': '-', - 'ยฏ': ' ', '_': ' ', '\u201C': '"', // left double quote " '\u201D': '"', // right double quote " @@ -48,9 +47,6 @@ class UnicodeProcessor { text = text.replaceAll(k, v); } - // Remove combining diacritics // FIXME: this should be fixed for non-English languages - text = text.replace(/[\u0302\u0303\u0304\u0305\u0306\u0307\u0308\u030A\u030B\u030C\u0327\u0328\u0329\u032A\u032B\u032C\u032D\u032E\u032F]/g, ''); - // Remove special symbols text = text.replace(/[โ™ฅโ˜†โ™กยฉ\\]/g, ''); @@ -92,6 +88,14 @@ class UnicodeProcessor { text += '.'; } + // Validate language + if (!AVAILABLE_LANGS.includes(lang)) { + throw new Error(`Invalid language: ${lang}. Available: ${AVAILABLE_LANGS.join(', ')}`); + } + + // Wrap text with language tags + text = `<${lang}>` + text + ``; + return text; } @@ -103,8 +107,8 @@ class UnicodeProcessor { return lengthToMask(textIdsLengths); } - call(textList) { - const processedTexts = textList.map(t => this._preprocessText(t)); + call(textList, langList) { + const processedTexts = textList.map((t, i) => this._preprocessText(t, langList[i])); const textIdsLengths = processedTexts.map(t => t.length); const maxLen = Math.max(...textIdsLengths); @@ -191,12 +195,12 @@ class TextToSpeech { return { noisyLatent, latentMask }; } - async _infer(textList, style, totalStep, speed = 1.05) { + async _infer(textList, langList, style, totalStep, speed = 1.05) { if (textList.length !== style.ttl.dims[0]) { throw new Error('Number of texts must match number of style vectors'); } const bsz = textList.length; - const { textIds, textMask } = this.textProcessor.call(textList); + const { textIds, textMask } = this.textProcessor.call(textList, langList); const textIdsShape = [bsz, textIds[0].length]; const textMaskShape = [bsz, 1, textMask[0][0].length]; @@ -267,16 +271,17 @@ class TextToSpeech { return { wav, duration: durOnnx }; } - async call(text, style, totalStep, speed = 1.05, silenceDuration = 0.3) { + async call(text, lang, style, totalStep, speed = 1.05, silenceDuration = 0.3) { if (style.ttl.dims[0] !== 1) { throw new Error('Single speaker text to speech only supports single style'); } - const textList = chunkText(text); + const maxLen = lang === 'ko' ? 120 : 300; + const textList = chunkText(text, maxLen); let wavCat = null; let durCat = 0; for (const chunk of textList) { - const { wav, duration } = await this._infer([chunk], style, totalStep, speed); + const { wav, duration } = await this._infer([chunk], [lang], style, totalStep, speed); if (wavCat === null) { wavCat = wav; @@ -292,8 +297,8 @@ class TextToSpeech { return { wav: wavCat, duration: [durCat] }; } - async batch(textList, style, totalStep, speed = 1.05) { - return await this._infer(textList, style, totalStep, speed); + async batch(textList, langList, style, totalStep, speed = 1.05) { + return await this._infer(textList, langList, style, totalStep, speed); } } @@ -502,6 +507,15 @@ export async function timer(name, fn) { return result; } +/** + * Sanitize filename by replacing non-alphanumeric characters with underscores (supports Unicode) + */ +export function sanitizeFilename(text, maxLen) { + const prefix = text.substring(0, maxLen); + // \p{L} matches any Unicode letter, \p{N} matches any Unicode number + return prefix.replace(/[^\p{L}\p{N}_]/gu, '_'); +} + /** * Chunk text into manageable segments */ diff --git a/py/README.md b/py/README.md index 6d94e2c..0df8449 100644 --- a/py/README.md +++ b/py/README.md @@ -4,6 +4,8 @@ This guide provides examples for running TTS inference using `example_onnx.py`. ## ๐Ÿ“ฐ Update News +**2026.01.06** - ๐ŸŽ‰ **Supertonic 2** released with multilingual support! Now supports English (`en`), Korean (`ko`), Spanish (`es`), Portuguese (`pt`), and French (`fr`). [Demo](https://huggingface.co/spaces/Supertone/supertonic-2) | [Models](https://huggingface.co/Supertone/supertonic-2) + **2025.12.10** - Added `supertonic` PyPI package! Install via `pip install supertonic` for a streamlined experience. This is a separate usage method from the ONNX examples in this directory. For more details, visit [supertonic-py documentation](https://supertone-inc.github.io/supertonic-py) and see `example_pypi.py` for usage. **2025.12.10** - Added [6 new voice styles](https://huggingface.co/Supertone/supertonic/tree/b10dbaf18b316159be75b34d24f740008fddd381) (M3, M4, M5, F3, F4, F5). See [Voices](https://supertone-inc.github.io/supertonic-py/voices/) for details @@ -55,15 +57,16 @@ Process multiple voice styles and texts at once: ```bash uv run example_onnx.py \ --voice-style assets/voice_styles/M1.json assets/voice_styles/F1.json \ - --text "The sun sets behind the mountains, painting the sky in shades of pink and orange." "The weather is beautiful and sunny outside. A gentle breeze makes the air feel fresh and pleasant." \ + --text "The sun sets behind the mountains, painting the sky in shades of pink and orange." "์˜ค๋Š˜ ์•„์นจ์— ๊ณต์›์„ ์‚ฐ์ฑ…ํ–ˆ๋Š”๋ฐ, ์ƒˆ์†Œ๋ฆฌ์™€ ๋ฐ”๋žŒ ์†Œ๋ฆฌ๊ฐ€ ๋„ˆ๋ฌด ์ข‹์•„์„œ ํ•œ์ฐธ์„ ๋ฉˆ์ถฐ ์„œ์„œ ๋“ค์—ˆ์–ด์š”." \ + --lang en ko \ --batch ``` This will: - Use `--batch` flag to enable batch processing mode - Generate speech for 2 different voice-text pairs -- Use male voice style (M1.json) for the first text -- Use female voice style (F1.json) for the second text +- Use male voice style (M1.json) for the first English text +- Use female voice style (F1.json) for the second Korean text - Process both samples in a single batch (automatic text chunking disabled) ### Example 3: High Quality Inference @@ -128,12 +131,14 @@ This will: | `--n-test` | int | 4 | Number of times to generate each sample | | `--voice-style` | str+ | `assets/voice_styles/M1.json` | Voice style file path(s) | | `--text` | str+ | (long default text) | Text(s) to synthesize | +| `--lang` | str+ | `en` | Language(s) for text(s): `en`, `ko`, `es`, `pt`, `fr` | | `--save-dir` | str | `results` | Output directory | | `--batch` | flag | False | Enable batch mode (disables automatic text chunking) | ## Notes - **Batch Processing**: The number of `--voice-style` files must match the number of `--text` entries +- **Multilingual Support**: Use `--lang` to specify language(s). Available: `en` (English), `ko` (Korean), `es` (Spanish), `pt` (Portuguese), `fr` (French) - **Long-Form Inference**: Without `--batch` flag, long texts are automatically chunked and combined into a single audio file with natural pauses - **Quality vs Speed**: Higher `--total-step` values produce better quality but take longer - **GPU Support**: GPU mode is not supported yet diff --git a/py/example_onnx.py b/py/example_onnx.py index 02c95d9..3cb96b4 100644 --- a/py/example_onnx.py +++ b/py/example_onnx.py @@ -56,6 +56,13 @@ def parse_args(): ], help="Text(s) to synthesize. Can specify multiple texts for batch processing", ) + parser.add_argument( + "--lang", + type=str, + nargs="+", + default=["en"], + help="Language(s) of the text(s). Can specify multiple languages for batch processing", + ) parser.add_argument( "--save-dir", type=str, default="results", help="Output directory" ) @@ -73,6 +80,7 @@ n_test = args.n_test save_dir = args.save_dir voice_style_paths = args.voice_style text_list = args.text +lang_list = args.lang batch = args.batch assert len(voice_style_paths) == len( @@ -91,9 +99,13 @@ for n in range(n_test): print(f"\n[{n+1}/{n_test}] Starting synthesis...") with timer("Generating speech from text"): if batch: - wav, duration = text_to_speech.batch(text_list, style, total_step, speed) + wav, duration = text_to_speech.batch( + text_list, lang_list, style, total_step, speed + ) else: - wav, duration = text_to_speech(text_list[0], style, total_step, speed) + wav, duration = text_to_speech( + text_list[0], lang_list[0], style, total_step, speed + ) if not os.path.exists(save_dir): os.makedirs(save_dir) for b in range(bsz): diff --git a/py/helper.py b/py/helper.py index cb9f897..182caac 100644 --- a/py/helper.py +++ b/py/helper.py @@ -10,18 +10,18 @@ import onnxruntime as ort import re +AVAILABLE_LANGS = ["en", "ko", "es", "pt", "fr"] + class UnicodeProcessor: def __init__(self, unicode_indexer_path: str): with open(unicode_indexer_path, "r") as f: self.indexer = json.load(f) - def _preprocess_text(self, text: str) -> str: + def _preprocess_text(self, text: str, lang: str) -> str: # TODO: Need advanced normalizer for better performance text = normalize("NFKD", text) - # FIXME: this should be fixed for non-English languages - # Remove emojis (wide Unicode range) emoji_pattern = re.compile( "[\U0001f600-\U0001f64f" # emoticons @@ -45,10 +45,9 @@ class UnicodeProcessor: "โ€“": "-", "โ€‘": "-", "โ€”": "-", - "ยฏ": " ", "_": " ", - "\u201C": '"', # left double quote " - "\u201D": '"', # right double quote " + "\u201c": '"', # left double quote " + "\u201d": '"', # right double quote " "\u2018": "'", # left single quote ' "\u2019": "'", # right single quote ' "ยด": "'", @@ -64,13 +63,6 @@ class UnicodeProcessor: for k, v in replacements.items(): text = text.replace(k, v) - # Remove combining diacritics # FIXME: this should be fixed for non-English languages - text = re.sub( - r"[\u0302\u0303\u0304\u0305\u0306\u0307\u0308\u030A\u030B\u030C\u0327\u0328\u0329\u032A\u032B\u032C\u032D\u032E\u032F]", - "", - text, - ) - # Remove special symbols text = re.sub(r"[โ™ฅโ˜†โ™กยฉ\\]", "", text) @@ -107,6 +99,9 @@ class UnicodeProcessor: if not re.search(r"[.!?;:,'\"')\]}โ€ฆใ€‚ใ€ใ€ใ€‘ใ€‰ใ€‹โ€บยป]$", text): text += "." + if lang not in AVAILABLE_LANGS: + raise ValueError(f"Invalid language: {lang}") + text = f"<{lang}>" + text + f"" return text def _get_text_mask(self, text_ids_lengths: np.ndarray) -> np.ndarray: @@ -119,8 +114,12 @@ class UnicodeProcessor: ) # 2 bytes return unicode_values - def __call__(self, text_list: list[str]) -> tuple[np.ndarray, np.ndarray]: - text_list = [self._preprocess_text(t) for t in text_list] + def __call__( + self, text_list: list[str], lang_list: list[str] + ) -> tuple[np.ndarray, np.ndarray]: + text_list = [ + self._preprocess_text(t, lang) for t, lang in zip(text_list, lang_list) + ] text_ids_lengths = np.array([len(text) for text in text_list], dtype=np.int64) text_ids = np.zeros((len(text_list), text_ids_lengths.max()), dtype=np.int64) for i, text in enumerate(text_list): @@ -176,13 +175,18 @@ class TextToSpeech: return noisy_latent, latent_mask def _infer( - self, text_list: list[str], style: Style, total_step: int, speed: float = 1.05 + self, + text_list: list[str], + lang_list: list[str], + style: Style, + total_step: int, + speed: float = 1.05, ) -> tuple[np.ndarray, np.ndarray]: assert ( len(text_list) == style.ttl.shape[0] ), "Number of texts must match number of style vectors" bsz = len(text_list) - text_ids, text_mask = self.text_processor(text_list) + text_ids, text_mask = self.text_processor(text_list, lang_list) dur_onnx, *_ = self.dp_ort.run( None, {"text_ids": text_ids, "style_dp": style.dp, "text_mask": text_mask} ) @@ -213,6 +217,7 @@ class TextToSpeech: def __call__( self, text: str, + lang: str, style: Style, total_step: int, speed: float = 1.05, @@ -221,11 +226,12 @@ class TextToSpeech: assert ( style.ttl.shape[0] == 1 ), "Single speaker text to speech only supports single style" - text_list = chunk_text(text) + max_len = 120 if lang == "ko" else 300 + text_list = chunk_text(text, max_len=max_len) wav_cat = None dur_cat = None for text in text_list: - wav, dur_onnx = self._infer([text], style, total_step, speed) + wav, dur_onnx = self._infer([text], [lang], style, total_step, speed) if wav_cat is None: wav_cat = wav dur_cat = dur_onnx @@ -238,9 +244,14 @@ class TextToSpeech: return wav_cat, dur_cat def batch( - self, text_list: list[str], style: Style, total_step: int, speed: float = 1.05 + self, + text_list: list[str], + lang_list: list[str], + style: Style, + total_step: int, + speed: float = 1.05, ) -> tuple[np.ndarray, np.ndarray]: - return self._infer(text_list, style, total_step, speed) + return self._infer(text_list, lang_list, style, total_step, speed) def length_to_mask(lengths: np.ndarray, max_len: Optional[int] = None) -> np.ndarray: @@ -365,11 +376,13 @@ def timer(name: str): def sanitize_filename(text: str, max_len: int) -> str: - """Sanitize filename by replacing non-alphanumeric characters with underscores""" + """Sanitize filename by replacing non-alphanumeric characters with underscores (supports Unicode)""" import re prefix = text[:max_len] - return re.sub(r"[^a-zA-Z0-9]", "_", prefix) + # \w matches Unicode word characters (letters, digits, underscore) with re.UNICODE + # We replace non-word characters except keeping existing underscores + return re.sub(r"[^\w]", "_", prefix, flags=re.UNICODE) def chunk_text(text: str, max_len: int = 300) -> list[str]: diff --git a/rust/README.md b/rust/README.md index 89a2683..25cb6ac 100644 --- a/rust/README.md +++ b/rust/README.md @@ -4,6 +4,8 @@ This guide provides examples for running TTS inference using Rust. ## ๐Ÿ“ฐ Update News +**2026.01.06** - ๐ŸŽ‰ **Supertonic 2** released with multilingual support! Now supports English (`en`), Korean (`ko`), Spanish (`es`), Portuguese (`pt`), and French (`fr`). [Demo](https://huggingface.co/spaces/Supertone/supertonic-2) | [Models](https://huggingface.co/Supertone/supertonic-2) + **2025.12.10** - Added [6 new voice styles](https://huggingface.co/Supertone/supertonic/tree/b10dbaf18b316159be75b34d24f740008fddd381) (M3, M4, M5, F3, F4, F5). See [Voices](https://supertone-inc.github.io/supertonic-py/voices/) for details **2025.12.08** - Optimized ONNX models via [OnnxSlim](https://github.com/inisis/OnnxSlim) now available on [Hugging Face Models](https://huggingface.co/Supertone/supertonic) @@ -58,19 +60,21 @@ Process multiple voice styles and texts at once: cargo run --release --bin example_onnx -- \ --batch \ --voice-style assets/voice_styles/M1.json,assets/voice_styles/F1.json \ - --text "The sun sets behind the mountains, painting the sky in shades of pink and orange.|The weather is beautiful and sunny outside. A gentle breeze makes the air feel fresh and pleasant." + --text "The sun sets behind the mountains, painting the sky in shades of pink and orange.|์˜ค๋Š˜ ์•„์นจ์— ๊ณต์›์„ ์‚ฐ์ฑ…ํ–ˆ๋Š”๋ฐ, ์ƒˆ์†Œ๋ฆฌ์™€ ๋ฐ”๋žŒ ์†Œ๋ฆฌ๊ฐ€ ๋„ˆ๋ฌด ๊ธฐ๋ถ„ ์ข‹์•˜์–ด์š”." \ + --lang en,ko # Or using the binary directly ./target/release/example_onnx \ --batch \ --voice-style assets/voice_styles/M1.json,assets/voice_styles/F1.json \ - --text "The sun sets behind the mountains, painting the sky in shades of pink and orange.|The weather is beautiful and sunny outside. A gentle breeze makes the air feel fresh and pleasant." + --text "The sun sets behind the mountains, painting the sky in shades of pink and orange.|์˜ค๋Š˜ ์•„์นจ์— ๊ณต์›์„ ์‚ฐ์ฑ…ํ–ˆ๋Š”๋ฐ, ์ƒˆ์†Œ๋ฆฌ์™€ ๋ฐ”๋žŒ ์†Œ๋ฆฌ๊ฐ€ ๋„ˆ๋ฌด ๊ธฐ๋ถ„ ์ข‹์•˜์–ด์š”." \ + --lang en,ko ``` This will: -- Generate speech for 2 different voice-text pairs -- Use male voice (M1.json) for the first text -- Use female voice (F1.json) for the second text +- Generate speech for 2 different voice-text-language pairs +- Use male voice (M1.json) for the first text in English +- Use female voice (F1.json) for the second text in Korean - Process both samples in a single batch ### Example 3: High Quality Inference @@ -124,14 +128,16 @@ This will: | `--onnx-dir` | str | `assets/onnx` | Path to ONNX model directory | | `--total-step` | int | 5 | Number of denoising steps (higher = better quality, slower) | | `--n-test` | int | 4 | Number of times to generate each sample | -| `--voice-style` | str+ | `assets/voice_styles/M1.json` | Voice style file path(s) | -| `--text` | str+ | (long default text) | Text(s) to synthesize | +| `--voice-style` | str+ | `assets/voice_styles/M1.json` | Voice style file path(s), comma-separated | +| `--text` | str+ | (long default text) | Text(s) to synthesize, pipe-separated | +| `--lang` | str+ | `en` | Language(s) for synthesis, comma-separated (en, ko, es, pt, fr) | | `--save-dir` | str | `results` | Output directory | | `--batch` | flag | False | Enable batch mode (multiple text-style pairs, disables automatic chunking) | ## Notes -- **Batch Processing**: When using `--batch`, the number of `--voice-style` files must match the number of `--text` entries +- **Multilingual Support**: Use `--lang` to specify the language for each text. Available: `en` (English), `ko` (Korean), `es` (Spanish), `pt` (Portuguese), `fr` (French) +- **Batch Processing**: When using `--batch`, the number of `--voice-style`, `--text`, and `--lang` entries must match - **Automatic Chunking**: Without `--batch`, long texts are automatically split and concatenated with 0.3s pauses - **Quality vs Speed**: Higher `--total-step` values produce better quality but take longer - **GPU Support**: GPU mode is not supported yet diff --git a/rust/src/example_onnx.rs b/rust/src/example_onnx.rs index fdb106f..74912f8 100644 --- a/rust/src/example_onnx.rs +++ b/rust/src/example_onnx.rs @@ -42,6 +42,10 @@ struct Args { #[arg(long, value_delimiter = '|', default_values_t = vec!["This morning, I took a walk in the park, and the sound of the birds and the breeze was so pleasant that I stopped for a long time just to listen.".to_string()])] text: Vec, + /// Language(s) for synthesis (en, ko, es, pt, fr) + #[arg(long, value_delimiter = ',', default_values_t = vec!["en".to_string()])] + lang: Vec, + /// Output directory #[arg(long, default_value = "results")] save_dir: String, @@ -61,6 +65,7 @@ fn main() -> Result<()> { let n_test = args.n_test; let voice_style_paths = &args.voice_style; let text_list = &args.text; + let lang_list = &args.lang; let save_dir = &args.save_dir; let batch = args.batch; @@ -72,6 +77,13 @@ fn main() -> Result<()> { text_list.len() ); } + if lang_list.len() != text_list.len() { + anyhow::bail!( + "Number of languages ({}) must match number of texts ({})", + lang_list.len(), + text_list.len() + ); + } } let bsz = voice_style_paths.len(); @@ -90,11 +102,11 @@ fn main() -> Result<()> { let (wav, duration) = if batch { timer("Generating speech from text", || { - text_to_speech.batch(text_list, &style, total_step, speed) + text_to_speech.batch(text_list, lang_list, &style, total_step, speed) })? } else { let (w, d) = timer("Generating speech from text", || { - text_to_speech.call(&text_list[0], &style, total_step, speed, 0.3) + text_to_speech.call(&text_list[0], &lang_list[0], &style, total_step, speed, 0.3) })?; (w, vec![d]) }; diff --git a/rust/src/helper.rs b/rust/src/helper.rs index e0206db..4dc5875 100644 --- a/rust/src/helper.rs +++ b/rust/src/helper.rs @@ -8,12 +8,19 @@ use serde_json; use std::fs::File; use std::io::BufReader; use std::path::Path; -use anyhow::{Result, Context}; +use anyhow::{Result, Context, bail}; use unicode_normalization::UnicodeNormalization; use hound::{WavWriter, WavSpec, SampleFormat}; use rand_distr::{Distribution, Normal}; use regex::Regex; +// Available languages for multilingual TTS +pub const AVAILABLE_LANGS: &[&str] = &["en", "ko", "es", "pt", "fr"]; + +pub fn is_valid_lang(lang: &str) -> bool { + AVAILABLE_LANGS.contains(&lang) +} + // ============================================================================ // Configuration Structures // ============================================================================ @@ -79,11 +86,11 @@ impl UnicodeProcessor { Ok(UnicodeProcessor { indexer }) } - pub fn call(&self, text_list: &[String]) -> (Vec>, Array3) { - let processed_texts: Vec = text_list - .iter() - .map(|t| preprocess_text(t)) - .collect(); + pub fn call(&self, text_list: &[String], lang_list: &[String]) -> Result<(Vec>, Array3)> { + let mut processed_texts: Vec = Vec::new(); + for (text, lang) in text_list.iter().zip(lang_list.iter()) { + processed_texts.push(preprocess_text(text, lang)?); + } let text_ids_lengths: Vec = processed_texts .iter() @@ -108,16 +115,14 @@ impl UnicodeProcessor { let text_mask = get_text_mask(&text_ids_lengths); - (text_ids, text_mask) + Ok((text_ids, text_mask)) } } -pub fn preprocess_text(text: &str) -> String { +pub fn preprocess_text(text: &str, lang: &str) -> Result { // TODO: Need advanced normalizer for better performance let mut text: String = text.nfkd().collect(); - // FIXME: this should be fixed for non-English languages - // Remove emojis (wide Unicode range) let emoji_pattern = Regex::new(r"[\x{1F600}-\x{1F64F}\x{1F300}-\x{1F5FF}\x{1F680}-\x{1F6FF}\x{1F700}-\x{1F77F}\x{1F780}-\x{1F7FF}\x{1F800}-\x{1F8FF}\x{1F900}-\x{1F9FF}\x{1FA00}-\x{1FA6F}\x{1FA70}-\x{1FAFF}\x{2600}-\x{26FF}\x{2700}-\x{27BF}\x{1F1E6}-\x{1F1FF}]+").unwrap(); text = emoji_pattern.replace_all(&text, "").to_string(); @@ -127,7 +132,6 @@ pub fn preprocess_text(text: &str) -> String { ("โ€“", "-"), // en dash ("โ€‘", "-"), // non-breaking hyphen ("โ€”", "-"), // em dash - ("ยฏ", " "), // macron ("_", " "), // underscore ("\u{201C}", "\""), // left double quote ("\u{201D}", "\""), // right double quote @@ -148,10 +152,6 @@ pub fn preprocess_text(text: &str) -> String { text = text.replace(from, to); } - // Remove combining diacritics // FIXME: this should be fixed for non-English languages - let diacritics_pattern = Regex::new(r"[\u{0302}\u{0303}\u{0304}\u{0305}\u{0306}\u{0307}\u{0308}\u{030A}\u{030B}\u{030C}\u{0327}\u{0328}\u{0329}\u{032A}\u{032B}\u{032C}\u{032D}\u{032E}\u{032F}]").unwrap(); - text = diacritics_pattern.replace_all(&text, "").to_string(); - // Remove special symbols let special_symbols = ["โ™ฅ", "โ˜†", "โ™ก", "ยฉ", "\\"]; for symbol in &special_symbols { @@ -201,7 +201,15 @@ pub fn preprocess_text(text: &str) -> String { } } - text + // Validate language + if !is_valid_lang(lang) { + bail!("Invalid language: {}. Available: {:?}", lang, AVAILABLE_LANGS); + } + + // Wrap text with language tags + text = format!("<{}>{}", lang, text, lang); + + Ok(text) } pub fn text_to_unicode_values(text: &str) -> Vec { @@ -505,15 +513,12 @@ where } pub fn sanitize_filename(text: &str, max_len: usize) -> String { - let text = if text.len() > max_len { - &text[..max_len] - } else { - text - }; - + // Take first max_len characters (Unicode code points, not bytes) text.chars() + .take(max_len) .map(|c| { - if c.is_ascii_alphanumeric() { + // is_alphanumeric() works with all Unicode letters and digits + if c.is_alphanumeric() { c } else { '_' @@ -570,6 +575,7 @@ impl TextToSpeech { fn _infer( &mut self, text_list: &[String], + lang_list: &[String], style: &Style, total_step: usize, speed: f32, @@ -577,7 +583,7 @@ impl TextToSpeech { let bsz = text_list.len(); // Process text - let (text_ids, text_mask) = self.text_processor.call(text_list); + let (text_ids, text_mask) = self.text_processor.call(text_list, lang_list)?; let text_ids_array = { let text_ids_shape = (bsz, text_ids[0].len()); @@ -676,18 +682,20 @@ impl TextToSpeech { pub fn call( &mut self, text: &str, + lang: &str, style: &Style, total_step: usize, speed: f32, silence_duration: f32, ) -> Result<(Vec, f32)> { - let chunks = chunk_text(text, None); + let max_len = if lang == "ko" { 120 } else { 300 }; + let chunks = chunk_text(text, Some(max_len)); let mut wav_cat: Vec = Vec::new(); let mut dur_cat: f32 = 0.0; for (i, chunk) in chunks.iter().enumerate() { - let (wav, duration) = self._infer(&[chunk.clone()], style, total_step, speed)?; + let (wav, duration) = self._infer(&[chunk.clone()], &[lang.to_string()], style, total_step, speed)?; let dur = duration[0]; let wav_len = (self.sample_rate as f32 * dur) as usize; @@ -712,11 +720,12 @@ impl TextToSpeech { pub fn batch( &mut self, text_list: &[String], + lang_list: &[String], style: &Style, total_step: usize, speed: f32, ) -> Result<(Vec, Vec)> { - self._infer(text_list, style, total_step, speed) + self._infer(text_list, lang_list, style, total_step, speed) } } diff --git a/swift/README.md b/swift/README.md index 7d273c7..075a917 100644 --- a/swift/README.md +++ b/swift/README.md @@ -4,6 +4,8 @@ This guide provides examples for running TTS inference using `example_onnx`. ## ๐Ÿ“ฐ Update News +**2026.01.06** - ๐ŸŽ‰ **Supertonic 2** released with multilingual support! Now supports English (`en`), Korean (`ko`), Spanish (`es`), Portuguese (`pt`), and French (`fr`). [Demo](https://huggingface.co/spaces/Supertone/supertonic-2) | [Models](https://huggingface.co/Supertone/supertonic-2) + **2025.12.10** - Added [6 new voice styles](https://huggingface.co/Supertone/supertonic/tree/b10dbaf18b316159be75b34d24f740008fddd381) (M3, M4, M5, F3, F4, F5). See [Voices](https://supertone-inc.github.io/supertonic-py/voices/) for details **2025.12.08** - Optimized ONNX models via [OnnxSlim](https://github.com/inisis/OnnxSlim) now available on [Hugging Face Models](https://huggingface.co/Supertone/supertonic) @@ -48,13 +50,14 @@ Process multiple voice styles and texts at once: .build/release/example_onnx \ --batch \ --voice-style assets/voice_styles/M1.json,assets/voice_styles/F1.json \ - --text "The sun sets behind the mountains, painting the sky in shades of pink and orange.|The weather is beautiful and sunny outside. A gentle breeze makes the air feel fresh and pleasant." + --text "The sun sets behind the mountains, painting the sky in shades of pink and orange.|์˜ค๋Š˜ ์•„์นจ์— ๊ณต์›์„ ์‚ฐ์ฑ…ํ–ˆ๋Š”๋ฐ, ์ƒˆ์†Œ๋ฆฌ์™€ ๋ฐ”๋žŒ ์†Œ๋ฆฌ๊ฐ€ ๋„ˆ๋ฌด ๊ธฐ๋ถ„ ์ข‹์•˜์–ด์š”." \ + --lang en,ko ``` This will: -- Generate speech for 2 different voice-text pairs -- Use male voice (M1.json) for the first text -- Use female voice (F1.json) for the second text +- Generate speech for 2 different voice-text-language triplets +- Use male voice (M1.json) for the first English text +- Use female voice (F1.json) for the second Korean text - Process both samples in a single batch ### Example 3: High Quality Inference @@ -97,12 +100,23 @@ This will: | `--n-test` | int | 4 | Number of times to generate each sample | | `--voice-style` | str+ | `assets/voice_styles/M1.json` | Voice style file path(s) | | `--text` | str+ | (long default text) | Text(s) to synthesize | +| `--lang` | str+ | `en` | Language(s) for synthesis (en, ko, es, pt, fr) | | `--save-dir` | str | `results` | Output directory | -| `--batch` | flag | False | Enable batch mode (multiple text-style pairs, disables automatic chunking) | +| `--batch` | flag | False | Enable batch mode (multiple text-style-lang triplets, disables automatic chunking) | + +## Multilingual Support + +Supertonic 2 supports multiple languages. Use the `--lang` argument to specify the language: + +- `en` - English (default) +- `ko` - Korean (ํ•œ๊ตญ์–ด) +- `es` - Spanish (Espaรฑol) +- `pt` - Portuguese (Portuguรชs) +- `fr` - French (Franรงais) ## Notes -- **Batch Processing**: When using `--batch`, the number of `--voice-style` files must match the number of `--text` entries +- **Batch Processing**: When using `--batch`, the number of `--voice-style`, `--text`, and `--lang` entries must match - **Automatic Chunking**: Without `--batch`, long texts are automatically split and concatenated with 0.3s pauses - **Quality vs Speed**: Higher `--total-step` values produce better quality but take longer - **GPU Support**: GPU mode is not supported yet \ No newline at end of file diff --git a/swift/Sources/ExampleONNX.swift b/swift/Sources/ExampleONNX.swift index 7e35fc5..702d562 100644 --- a/swift/Sources/ExampleONNX.swift +++ b/swift/Sources/ExampleONNX.swift @@ -9,6 +9,7 @@ struct Args { var nTest: Int = 4 var voiceStyle: [String] = ["assets/voice_styles/M1.json"] var text: [String] = ["This morning, I took a walk in the park, and the sound of the birds and the breeze was so pleasant that I stopped for a long time just to listen."] + var lang: [String] = ["en"] var saveDir: String = "results" var batch: Bool = false } @@ -54,6 +55,11 @@ func parseArgs() -> Args { args.text = arguments[i + 1].components(separatedBy: "|") i += 1 } + case "--lang": + if i + 1 < arguments.count { + args.lang = arguments[i + 1].components(separatedBy: ",") + i += 1 + } case "--save-dir": if i + 1 < arguments.count { args.saveDir = arguments[i + 1] @@ -84,6 +90,10 @@ struct ExampleONNX { print("Error: Number of voice styles (\(args.voiceStyle.count)) must match number of texts (\(args.text.count))") return } + guard args.lang.count == args.text.count else { + print("Error: Number of languages (\(args.lang.count)) must match number of texts (\(args.text.count))") + return + } } let bsz = args.voiceStyle.count @@ -108,13 +118,13 @@ struct ExampleONNX { if args.batch { let result = try timer("Generating speech from text") { - try textToSpeech.batch(args.text, style, args.totalStep, speed: args.speed) + try textToSpeech.batch(args.text, args.lang, style, args.totalStep, speed: args.speed) } wav = result.wav duration = result.duration } else { let result = try timer("Generating speech from text") { - try textToSpeech.call(args.text[0], style, args.totalStep, speed: args.speed, silenceDuration: 0.3) + try textToSpeech.call(args.text[0], args.lang[0], style, args.totalStep, speed: args.speed, silenceDuration: 0.3) } wav = result.wav duration = [result.duration] diff --git a/swift/Sources/Helper.swift b/swift/Sources/Helper.swift index de0f3bd..1e8e3bd 100644 --- a/swift/Sources/Helper.swift +++ b/swift/Sources/Helper.swift @@ -2,6 +2,14 @@ import Foundation import Accelerate import OnnxRuntimeBindings +// MARK: - Available Languages + +let AVAILABLE_LANGS = ["en", "ko", "es", "pt", "fr"] + +func isValidLang(_ lang: String) -> Bool { + return AVAILABLE_LANGS.contains(lang) +} + // MARK: - Configuration Structures struct Config: Codable { @@ -42,12 +50,16 @@ class UnicodeProcessor { self.indexer = try JSONDecoder().decode([Int64].self, from: data) } - func call(_ textList: [String]) -> (textIds: [[Int64]], textMask: [[[Float]]]) { - let processedTexts = textList.map { preprocessText($0) } + func call(_ textList: [String], _ langList: [String]) -> (textIds: [[Int64]], textMask: [[[Float]]]) { + var processedTexts = [String]() + for (i, text) in textList.enumerated() { + processedTexts.append(preprocessText(text, lang: langList[i])) + } + // Use unicodeScalars.count for correct length after NFKD decomposition var textIdsLengths = [Int]() for text in processedTexts { - textIdsLengths.append(text.count) + textIdsLengths.append(text.unicodeScalars.count) } let maxLen = textIdsLengths.max() ?? 0 @@ -71,11 +83,9 @@ class UnicodeProcessor { } } -func preprocessText(_ text: String) -> String { - // TODO: Need advanced normalizer for better performance - var text = text.precomposedStringWithCompatibilityMapping - - // FIXME: this should be fixed for non-English languages +func preprocessText(_ text: String, lang: String) -> String { + // Use NFKD (decomposed) for proper Hangul Jamo decomposition + var text = text.decomposedStringWithCompatibilityMapping // Remove emojis (wide Unicode range) // Swift NSRegularExpression doesn't support Unicode escapes above \uFFFF @@ -101,7 +111,6 @@ func preprocessText(_ text: String) -> String { "โ€“": "-", // en dash "โ€‘": "-", // non-breaking hyphen "โ€”": "-", // em dash - "ยฏ": " ", // macron "_": " ", // underscore "\u{201C}": "\"", // left double quote "\u{201D}": "\"", // right double quote @@ -122,11 +131,6 @@ func preprocessText(_ text: String) -> String { text = text.replacingOccurrences(of: old, with: new) } - // Remove combining diacritics // FIXME: this should be fixed for non-English languages - let diacriticsPattern = try! NSRegularExpression(pattern: "[\\u0302\\u0303\\u0304\\u0305\\u0306\\u0307\\u0308\\u030A\\u030B\\u030C\\u0327\\u0328\\u0329\\u032A\\u032B\\u032C\\u032D\\u032E\\u032F]") - let diacriticsRange = NSRange(text.startIndex..., in: text) - text = diacriticsPattern.stringByReplacingMatches(in: text, range: diacriticsRange, withTemplate: "") - // Remove special symbols let specialSymbols = ["โ™ฅ", "โ˜†", "โ™ก", "ยฉ", "\\"] for symbol in specialSymbols { @@ -179,6 +183,14 @@ func preprocessText(_ text: String) -> String { } } + // Validate language + guard isValidLang(lang) else { + fatalError("Invalid language: \(lang). Available: \(AVAILABLE_LANGS.joined(separator: ", "))") + } + + // Wrap text with language tags + text = "<\(lang)>\(text)" + return text } @@ -560,11 +572,11 @@ class TextToSpeech { self.sampleRate = cfgs.ae.sample_rate } - private func _infer(_ textList: [String], _ style: Style, _ totalStep: Int, speed: Float = 1.05) throws -> (wav: [Float], duration: [Float]) { + private func _infer(_ textList: [String], _ langList: [String], _ style: Style, _ totalStep: Int, speed: Float = 1.05) throws -> (wav: [Float], duration: [Float]) { let bsz = textList.count // Process text - let (textIds, textMask) = textProcessor.call(textList) + let (textIds, textMask) = textProcessor.call(textList, langList) // Flatten text IDs let textIdsFlat = textIds.flatMap { $0 } @@ -688,14 +700,16 @@ class TextToSpeech { return (wav, duration) } - func call(_ text: String, _ style: Style, _ totalStep: Int, speed: Float = 1.05, silenceDuration: Float = 0.3) throws -> (wav: [Float], duration: Float) { - let chunks = chunkText(text) + func call(_ text: String, _ lang: String, _ style: Style, _ totalStep: Int, speed: Float = 1.05, silenceDuration: Float = 0.3) throws -> (wav: [Float], duration: Float) { + let maxLen = lang == "ko" ? 120 : 300 + let chunks = chunkText(text, maxLen: maxLen) + let langList = Array(repeating: lang, count: chunks.count) var wavCat = [Float]() var durCat: Float = 0.0 for (i, chunk) in chunks.enumerated() { - let result = try _infer([chunk], style, totalStep, speed: speed) + let result = try _infer([chunk], [langList[i]], style, totalStep, speed: speed) let dur = result.duration[0] let wavLen = Int(Float(sampleRate) * dur) @@ -717,8 +731,8 @@ class TextToSpeech { return (wavCat, durCat) } - func batch(_ textList: [String], _ style: Style, _ totalStep: Int, speed: Float = 1.05) throws -> (wav: [Float], duration: [Float]) { - return try _infer(textList, style, totalStep, speed: speed) + func batch(_ textList: [String], _ langList: [String], _ style: Style, _ totalStep: Int, speed: Float = 1.05) throws -> (wav: [Float], duration: [Float]) { + return try _infer(textList, langList, style, totalStep, speed: speed) } } diff --git a/test_all.sh b/test_all.sh index 1ba6dba..0ff292b 100755 --- a/test_all.sh +++ b/test_all.sh @@ -57,11 +57,13 @@ case $test_mode in esac echo "" -# Batch inference test data - base variables +# Batch inference test data - multilingual examples BATCH_VOICE_STYLE_1="assets/voice_styles/M1.json" BATCH_VOICE_STYLE_2="assets/voice_styles/F1.json" BATCH_TEXT_1="The sun sets behind the mountains, painting the sky in shades of pink and orange." -BATCH_TEXT_2="The weather is beautiful and sunny outside. A gentle breeze makes the air feel fresh and pleasant." +BATCH_TEXT_2="์˜ค๋Š˜ ์•„์นจ์— ๊ณต์›์„ ์‚ฐ์ฑ…ํ–ˆ๋Š”๋ฐ, ์ƒˆ์†Œ๋ฆฌ์™€ ๋ฐ”๋žŒ ์†Œ๋ฆฌ๊ฐ€ ๋„ˆ๋ฌด ๊ธฐ๋ถ„ ์ข‹์•˜์–ด์š”." +BATCH_LANG_1="en" +BATCH_LANG_2="ko" # Long-form inference test data LONGFORM_VOICE_STYLE="assets/voice_styles/M1.json" @@ -179,7 +181,7 @@ if [ "$TEST_DEFAULT" = true ]; then run_test "Python (default)" "py" "uv run example_onnx.py" fi if [ "$TEST_BATCH" = true ]; then - run_test "Python (batch)" "py" "uv run example_onnx.py --batch --voice-style $BATCH_VOICE_STYLE_1 $BATCH_VOICE_STYLE_2 --text '$BATCH_TEXT_1' '$BATCH_TEXT_2'" + run_test "Python (batch)" "py" "uv run example_onnx.py --batch --voice-style $BATCH_VOICE_STYLE_1 $BATCH_VOICE_STYLE_2 --text '$BATCH_TEXT_1' '$BATCH_TEXT_2' --lang $BATCH_LANG_1 $BATCH_LANG_2" fi if [ "$TEST_LONGFORM" = true ]; then run_test "Python (long-form)" "py" "uv run example_onnx.py --voice-style $LONGFORM_VOICE_STYLE --text '$LONGFORM_TEXT'" @@ -195,7 +197,7 @@ if [ "$TEST_DEFAULT" = true ]; then run_test "JavaScript (default)" "nodejs" "node example_onnx.js" fi if [ "$TEST_BATCH" = true ]; then - run_test "JavaScript (batch)" "nodejs" "node example_onnx.js --batch --voice-style $BATCH_VOICE_STYLE_1,$BATCH_VOICE_STYLE_2 --text '$BATCH_TEXT_1|$BATCH_TEXT_2'" + run_test "JavaScript (batch)" "nodejs" "node example_onnx.js --batch --voice-style $BATCH_VOICE_STYLE_1,$BATCH_VOICE_STYLE_2 --text '$BATCH_TEXT_1|$BATCH_TEXT_2' --lang $BATCH_LANG_1,$BATCH_LANG_2" fi if [ "$TEST_LONGFORM" = true ]; then run_test "JavaScript (long-form)" "nodejs" "node example_onnx.js --voice-style $LONGFORM_VOICE_STYLE --text '$LONGFORM_TEXT'" @@ -212,7 +214,7 @@ if [ "$TEST_DEFAULT" = true ]; then run_test "Go (default)" "go" "go run example_onnx.go helper.go" fi if [ "$TEST_BATCH" = true ]; then - run_test "Go (batch)" "go" "go run example_onnx.go helper.go --batch -voice-style $BATCH_VOICE_STYLE_1,$BATCH_VOICE_STYLE_2 -text '$BATCH_TEXT_1|$BATCH_TEXT_2'" + run_test "Go (batch)" "go" "go run example_onnx.go helper.go --batch -voice-style $BATCH_VOICE_STYLE_1,$BATCH_VOICE_STYLE_2 -text '$BATCH_TEXT_1|$BATCH_TEXT_2' -lang $BATCH_LANG_1,$BATCH_LANG_2" fi if [ "$TEST_LONGFORM" = true ]; then run_test "Go (long-form)" "go" "go run example_onnx.go helper.go -voice-style $LONGFORM_VOICE_STYLE -text '$LONGFORM_TEXT'" @@ -228,7 +230,7 @@ if [ "$TEST_DEFAULT" = true ]; then run_test "Rust (default)" "rust" "cargo run --release" fi if [ "$TEST_BATCH" = true ]; then - run_test "Rust (batch)" "rust" "cargo run --release -- --batch --voice-style $BATCH_VOICE_STYLE_1,$BATCH_VOICE_STYLE_2 --text '$BATCH_TEXT_1|$BATCH_TEXT_2'" + run_test "Rust (batch)" "rust" "cargo run --release -- --batch --voice-style $BATCH_VOICE_STYLE_1,$BATCH_VOICE_STYLE_2 --text '$BATCH_TEXT_1|$BATCH_TEXT_2' --lang $BATCH_LANG_1,$BATCH_LANG_2" fi if [ "$TEST_LONGFORM" = true ]; then run_test "Rust (long-form)" "rust" "cargo run --release -- --voice-style $LONGFORM_VOICE_STYLE --text '$LONGFORM_TEXT'" @@ -244,7 +246,7 @@ if [ "$TEST_DEFAULT" = true ]; then run_test "C# (default)" "csharp" "dotnet run --configuration Release" fi if [ "$TEST_BATCH" = true ]; then - run_test "C# (batch)" "csharp" "dotnet run --configuration Release -- --batch --voice-style ../$BATCH_VOICE_STYLE_1,../$BATCH_VOICE_STYLE_2 --text '$BATCH_TEXT_1|$BATCH_TEXT_2'" + run_test "C# (batch)" "csharp" "dotnet run --configuration Release -- --batch --voice-style ../$BATCH_VOICE_STYLE_1,../$BATCH_VOICE_STYLE_2 --text '$BATCH_TEXT_1|$BATCH_TEXT_2' --lang $BATCH_LANG_1,$BATCH_LANG_2" fi if [ "$TEST_LONGFORM" = true ]; then run_test "C# (long-form)" "csharp" "dotnet run --configuration Release -- --voice-style ../$LONGFORM_VOICE_STYLE --text '$LONGFORM_TEXT'" @@ -260,7 +262,7 @@ if [ "$TEST_DEFAULT" = true ]; then run_test "Java (default)" "java" "mvn exec:java -q" fi if [ "$TEST_BATCH" = true ]; then - run_test "Java (batch)" "java" "mvn exec:java -q -Dexec.args='--batch --voice-style $BATCH_VOICE_STYLE_1,$BATCH_VOICE_STYLE_2 --text \"$BATCH_TEXT_1|$BATCH_TEXT_2\"'" + run_test "Java (batch)" "java" "mvn exec:java -q -Dexec.args='--batch --voice-style $BATCH_VOICE_STYLE_1,$BATCH_VOICE_STYLE_2 --text \"$BATCH_TEXT_1|$BATCH_TEXT_2\" --lang $BATCH_LANG_1,$BATCH_LANG_2'" fi if [ "$TEST_LONGFORM" = true ]; then run_test "Java (long-form)" "java" "mvn exec:java -q -Dexec.args='--voice-style $LONGFORM_VOICE_STYLE --text \"$LONGFORM_TEXT\"'" @@ -276,7 +278,7 @@ if [ "$TEST_DEFAULT" = true ]; then run_test "Swift (default)" "swift" ".build/release/example_onnx" fi if [ "$TEST_BATCH" = true ]; then - run_test "Swift (batch)" "swift" ".build/release/example_onnx --batch --voice-style $BATCH_VOICE_STYLE_1,$BATCH_VOICE_STYLE_2 --text '$BATCH_TEXT_1|$BATCH_TEXT_2'" + run_test "Swift (batch)" "swift" ".build/release/example_onnx --batch --voice-style $BATCH_VOICE_STYLE_1,$BATCH_VOICE_STYLE_2 --text '$BATCH_TEXT_1|$BATCH_TEXT_2' --lang $BATCH_LANG_1,$BATCH_LANG_2" fi if [ "$TEST_LONGFORM" = true ]; then run_test "Swift (long-form)" "swift" ".build/release/example_onnx --voice-style $LONGFORM_VOICE_STYLE --text '$LONGFORM_TEXT'" @@ -292,7 +294,7 @@ if [ "$TEST_DEFAULT" = true ]; then run_test "C++ (default)" "cpp/build" "./example_onnx" fi if [ "$TEST_BATCH" = true ]; then - run_test "C++ (batch)" "cpp/build" "./example_onnx --batch --voice-style ../$BATCH_VOICE_STYLE_1,../$BATCH_VOICE_STYLE_2 --text '$BATCH_TEXT_1|$BATCH_TEXT_2'" + run_test "C++ (batch)" "cpp/build" "./example_onnx --batch --voice-style ../$BATCH_VOICE_STYLE_1,../$BATCH_VOICE_STYLE_2 --text '$BATCH_TEXT_1|$BATCH_TEXT_2' --lang $BATCH_LANG_1,$BATCH_LANG_2" fi if [ "$TEST_LONGFORM" = true ]; then run_test "C++ (long-form)" "cpp/build" "./example_onnx --voice-style ../$LONGFORM_VOICE_STYLE --text '$LONGFORM_TEXT'" diff --git a/web/README.md b/web/README.md index 9768084..48dbe0e 100644 --- a/web/README.md +++ b/web/README.md @@ -4,6 +4,8 @@ This example demonstrates how to use Supertonic in a web browser using ONNX Runt ## ๐Ÿ“ฐ Update News +**2026.01.06** - ๐ŸŽ‰ **Supertonic 2** released with multilingual support! Now supports English (`en`), Korean (`ko`), Spanish (`es`), Portuguese (`pt`), and French (`fr`). [Demo](https://huggingface.co/spaces/Supertone/supertonic-2) | [Models](https://huggingface.co/Supertone/supertonic-2) + **2025.12.10** - Added [6 new voice styles](https://huggingface.co/Supertone/supertonic/tree/b10dbaf18b316159be75b34d24f740008fddd381) (M3, M4, M5, F3, F4, F5). See [Voices](https://supertone-inc.github.io/supertonic-py/voices/) for details **2025.12.08** - Optimized ONNX models via [OnnxSlim](https://github.com/inisis/OnnxSlim) now available on [Hugging Face Models](https://huggingface.co/Supertone/supertonic) @@ -18,9 +20,10 @@ This example demonstrates how to use Supertonic in a web browser using ONNX Runt - ๐ŸŒ Runs entirely in the browser (no server required for inference) - ๐Ÿš€ WebGPU support with automatic fallback to WebAssembly +- ๐ŸŒ Multilingual support: English (en), Korean (ko), Spanish (es), Portuguese (pt), French (fr) - โšก Pre-extracted voice styles for instant generation - ๐ŸŽจ Modern, responsive UI -- ๐ŸŽญ Multiple voice style presets (2 Male, 2 Female) +- ๐ŸŽญ Multiple voice style presets (5 Male, 5 Female) - ๐Ÿ’พ Download generated audio as WAV files - ๐Ÿ“Š Detailed generation statistics (audio length, generation time) - โฑ๏ธ Real-time progress tracking @@ -52,20 +55,28 @@ This will start a local development server (usually at http://localhost:3000) an 1. **Wait for Models to Load**: The app will automatically load models and the default voice style (M1) 2. **Select Voice Style**: Choose from available voice presets - - **Male 1 (M1)**: Default male voice - - **Male 2 (M2)**: Alternative male voice - - **Female 1 (F1)**: Default female voice - - **Female 2 (F2)**: Alternative female voice -3. **Enter Text**: Type or paste the text you want to convert to speech -4. **Adjust Settings** (optional): + - **Male 1-5 (M1-M5)**: Male voice styles + - **Female 1-5 (F1-F5)**: Female voice styles +3. **Select Language**: Choose the language that matches your input text + - **English (en)**: Default language + - **ํ•œ๊ตญ์–ด (ko)**: Korean + - **Espaรฑol (es)**: Spanish + - **Portuguรชs (pt)**: Portuguese + - **Franรงais (fr)**: French +4. **Enter Text**: Type or paste the text you want to convert to speech +5. **Adjust Settings** (optional): - **Total Steps**: More steps = better quality but slower (default: 5) -5. **Generate Speech**: Click the "Generate Speech" button -6. **View Results**: +6. **Generate Speech**: Click the "Generate Speech" button +7. **View Results**: - See the full input text - View audio length and generation time statistics - Play the generated audio in the browser - Download as WAV file +## Multilingual Support + +Supertonic 2 supports multiple languages. Make sure to select the correct language for your input text to get the best results. The model will automatically handle text preprocessing and pronunciation for the selected language. + ## Technical Details ### Browser Compatibility @@ -80,7 +91,7 @@ This demo uses: - The ONNX models must be accessible at `assets/onnx/` relative to the web root - Voice style JSON files must be accessible at `assets/voice_styles/` relative to the web root - Pre-extracted voice styles enable instant generation without audio processing -- Four voice style presets are provided (M1, M2, F1, F2) +- Ten voice style presets are provided (M1-M5, F1-F5) ## Troubleshooting diff --git a/web/helper.js b/web/helper.js index e62dd6b..931d1d7 100644 --- a/web/helper.js +++ b/web/helper.js @@ -1,5 +1,12 @@ import * as ort from 'onnxruntime-web'; +// Available languages for multilingual TTS +export const AVAILABLE_LANGS = ['en', 'ko', 'es', 'pt', 'fr']; + +export function isValidLang(lang) { + return AVAILABLE_LANGS.includes(lang); +} + /** * Unicode Text Processor */ @@ -8,8 +15,8 @@ export class UnicodeProcessor { this.indexer = indexer; } - call(textList) { - const processedTexts = textList.map(text => this.preprocessText(text)); + call(textList, langList) { + const processedTexts = textList.map((text, i) => this.preprocessText(text, langList[i])); const textIdsLengths = processedTexts.map(text => text.length); const maxLen = Math.max(...textIdsLengths); @@ -27,12 +34,10 @@ export class UnicodeProcessor { return { textIds, textMask }; } - preprocessText(text) { + preprocessText(text, lang) { // TODO: Need advanced normalizer for better performance text = text.normalize('NFKD'); - // FIXME: this should be fixed for non-English languages - // Remove emojis (wide Unicode range) const emojiPattern = /[\u{1F600}-\u{1F64F}\u{1F300}-\u{1F5FF}\u{1F680}-\u{1F6FF}\u{1F700}-\u{1F77F}\u{1F780}-\u{1F7FF}\u{1F800}-\u{1F8FF}\u{1F900}-\u{1F9FF}\u{1FA00}-\u{1FA6F}\u{1FA70}-\u{1FAFF}\u{2600}-\u{26FF}\u{2700}-\u{27BF}\u{1F1E6}-\u{1F1FF}]+/gu; text = text.replace(emojiPattern, ''); @@ -42,7 +47,6 @@ export class UnicodeProcessor { 'โ€“': '-', 'โ€‘': '-', 'โ€”': '-', - 'ยฏ': ' ', '_': ' ', '\u201C': '"', // left double quote " '\u201D': '"', // right double quote " @@ -62,9 +66,6 @@ export class UnicodeProcessor { text = text.replaceAll(k, v); } - // Remove combining diacritics // FIXME: this should be fixed for non-English languages - text = text.replace(/[\u0302\u0303\u0304\u0305\u0306\u0307\u0308\u030A\u030B\u030C\u0327\u0328\u0329\u032A\u032B\u032C\u032D\u032E\u032F]/g, ''); - // Remove special symbols text = text.replace(/[โ™ฅโ˜†โ™กยฉ\\]/g, ''); @@ -106,6 +107,14 @@ export class UnicodeProcessor { text += '.'; } + // Validate language + if (!isValidLang(lang)) { + throw new Error(`Invalid language: ${lang}. Available: ${AVAILABLE_LANGS.join(', ')}`); + } + + // Wrap text with language tags + text = `<${lang}>${text}`; + return text; } @@ -150,11 +159,11 @@ export class TextToSpeech { this.sampleRate = cfgs.ae.sample_rate; } - async _infer(textList, style, totalStep, speed = 1.05, progressCallback = null) { + async _infer(textList, langList, style, totalStep, speed = 1.05, progressCallback = null) { const bsz = textList.length; // Process text - const { textIds, textMask } = this.textProcessor.call(textList); + const { textIds, textMask } = this.textProcessor.call(textList, langList); const textIdsFlat = new BigInt64Array(textIds.flat().map(x => BigInt(x))); const textIdsShape = [bsz, textIds[0].length]; @@ -259,16 +268,18 @@ export class TextToSpeech { return { wav, duration }; } - async call(text, style, totalStep, speed = 1.05, silenceDuration = 0.3, progressCallback = null) { + async call(text, lang, style, totalStep, speed = 1.05, silenceDuration = 0.3, progressCallback = null) { if (style.ttl.dims[0] !== 1) { throw new Error('Single speaker text to speech only supports single style'); } - const textList = chunkText(text); + const maxLen = lang === 'ko' ? 120 : 300; + const textList = chunkText(text, maxLen); + const langList = new Array(textList.length).fill(lang); let wavCat = []; let durCat = 0; - for (const chunk of textList) { - const { wav, duration } = await this._infer([chunk], style, totalStep, speed, progressCallback); + for (let i = 0; i < textList.length; i++) { + const { wav, duration } = await this._infer([textList[i]], [langList[i]], style, totalStep, speed, progressCallback); if (wavCat.length === 0) { wavCat = wav; @@ -284,8 +295,8 @@ export class TextToSpeech { return { wav: wavCat, duration: [durCat] }; } - async batch(textList, style, totalStep, speed = 1.05, progressCallback = null) { - return await this._infer(textList, style, totalStep, speed, progressCallback); + async batch(textList, langList, style, totalStep, speed = 1.05, progressCallback = null) { + return await this._infer(textList, langList, style, totalStep, speed, progressCallback); } sampleNoisyLatent(duration, sampleRate, baseChunkSize, chunkCompress, latentDim) { diff --git a/web/index.html b/web/index.html index 816d5e9..527627e 100644 --- a/web/index.html +++ b/web/index.html @@ -8,8 +8,8 @@
-

๐ŸŽค Supertonic

-

Text-to-Speech with ONNX Runtime Web

+

๐ŸŽค Supertonic 2

+

Multilingual Text-to-Speech with ONNX Runtime Web

@@ -30,8 +30,25 @@ +
+ +
+ +
diff --git a/web/main.js b/web/main.js index 13155c5..ee134c1 100644 --- a/web/main.js +++ b/web/main.js @@ -24,6 +24,7 @@ let currentStylePath = DEFAULT_VOICE_STYLE_PATH; const textInput = document.getElementById('text'); const voiceStyleSelect = document.getElementById('voiceStyleSelect'); const voiceStyleInfo = document.getElementById('voiceStyleInfo'); +const langSelect = document.getElementById('langSelect'); const totalStepInput = document.getElementById('totalStep'); const speedInput = document.getElementById('speed'); const generateBtn = document.getElementById('generateBtn'); @@ -188,12 +189,14 @@ async function generateSpeech() { const totalStep = parseInt(totalStepInput.value); const speed = parseFloat(speedInput.value); + const lang = langSelect.value; showStatus('โ„น๏ธ Generating speech from text...'); const tic = Date.now(); const { wav, duration } = await textToSpeech.call( - text, + text, + lang, currentStyle, totalStep, speed,