[{"content":"X-Codec2 is a single-codebook neural audio codec with a large vocabulary (65536 tokens). It pairs directly with the Llasa TTS model and is the recommended codec for autoregressive speech generation tasks.\nStable version: xcodec2==1.3.0 — newer versions may be unstable.\nHuggingFace: HKUSTAudio/xcodec2\nInstallation conda create -n xcodec2_env python=3.10 conda activate xcodec2_env pip install torch soundfile transformers xcodec2==1.3.0 Usage import torch import soundfile as sf from xcodec2.modeling_xcodec2 import XCodec2Model model = XCodec2Model.from_pretrained(\u0026#34;HKUSTAudio/xcodec2\u0026#34;) model.eval().cuda() wav, sr = sf.read(\u0026#34;test.wav\u0026#34;) wav_tensor = torch.from_numpy(wav).float().unsqueeze(0) with torch.no_grad(): vq_code = model.encode_code(input_waveform=wav_tensor) print(\u0026#34;Code shape:\u0026#34;, vq_code.shape) recon_wav = model.decode_code(vq_code).cpu() sf.write(\u0026#34;reconstructed.wav\u0026#34;, recon_wav[0, 0, :].numpy(), sr) ","permalink":"https://aadonis-ai.github.io/notebook/neural-audio-codecs/xcodec2/","summary":"X-Codec2 setup and usage — single-codebook codec used by Llasa for TTS. Stable version is 1.3.0.","title":"X-Codec2"},{"content":"WavTokenizer (ICLR 2025) compresses 24kHz audio to just 40 or 75 tokens/sec using a single quantizer. Excellent for clean speech tasks like TTS. Avoid for speech enhancement — it was not trained on degraded audio and can introduce additional artifacts when encoding noisy input.\nPaper: WavTokenizer GitHub: jishengpeng/WavTokenizer\nSetup git clone https://github.com/jishengpeng/WavTokenizer.git cd WavTokenizer conda create -n wavtokenizer python=3.9 conda activate wavtokenizer pip install -r requirements.txt # skip fairseq if it fails — inference works without it Select the config matching your model\u0026rsquo;s token rate:\n40 tok/sec: ./configs/wavtokenizer_smalldata_frame40_3s_nq1_code4096_dim512_kmeans200_attn.yaml 75 tok/sec: ./configs/wavtokenizer_smalldata_frame75_3s_nq1_code4096_dim512_kmeans200_attn.yaml Download a checkpoint:\nwget -O wavtokenizer_large_speech_320_v2.ckpt \\ https://huggingface.co/novateur/WavTokenizer-large-speech-75token/resolve/main/wavtokenizer_large_speech_320_v2.ckpt Usage from encoder.utils import convert_audio import torchaudio import torch from decoder.pretrained import WavTokenizer device = torch.device(\u0026#39;cpu\u0026#39;) config_path = \u0026#34;./configs/wavtokenizer_smalldata_frame40_3s_nq1_code4096_dim512_kmeans200_attn.yaml\u0026#34; model_path = \u0026#34;./wavtokenizer_large_unify_600_24k.ckpt\u0026#34; wavtokenizer = WavTokenizer.from_pretrained0802(config_path, model_path).to(device) wav, sr = torchaudio.load(\u0026#34;audio.wav\u0026#34;) wav = convert_audio(wav, sr, 24000, 1).to(device) bandwidth_id = torch.tensor([0]) _, discrete_code = wavtokenizer.encode_infer(wav, bandwidth_id=bandwidth_id) features = wavtokenizer.codes_to_features(discrete_code) audio_out = wavtokenizer.decode(features, bandwidth_id=bandwidth_id).cpu() torchaudio.save(\u0026#34;reconstructed.wav\u0026#34;, audio_out, 24000) ","permalink":"https://aadonis-ai.github.io/notebook/neural-audio-codecs/wavtokenizer/","summary":"WavTokenizer setup — extreme compression (40–75 tok/sec, single codebook). Best for clean TTS; avoid for noisy/degraded speech.","title":"WavTokenizer"},{"content":"This training script is designed for training audio language models from scratch or from a checkpoint. It handles the full setup: loading a custom HuggingFace tokenizer, building a LLaMA model of any size, masking labels up to a delimiter token (useful for conditional generation), and running distributed training with accelerate.\nWhat it does:\nLoads any HuggingFace-compatible tokenizer and adapts the model vocabulary size automatically Loads training/test datasets from HuggingFace Hub (requires a \u0026quot;sequence\u0026quot; column) Masks all labels before (and including) a delimiter token (e.g. \u0026lt;|start_clean|\u0026gt;) with -100 Initializes a LLaMA model (250M–8B) with Flash Attention 2 Runs with accelerate launch --num_processes N --mixed_precision bf16 Supports checkpoint resuming with automatic vocab size adaptation Run with:\naccelerate launch --num_processes 4 --mixed_precision bf16 train.py \\ --model_size 1B \\ --run_name my_run \\ --tokenizer_path YOUR_HF_NAME/snac_tokenizer \\ --train_dataset YOUR_HF_NAME/train_data \\ --test_dataset YOUR_HF_NAME/test_data Full Training Script import torch from torch.utils.data import Dataset from transformers import LlamaConfig, LlamaForCausalLM, Trainer, TrainingArguments, TrainerCallback, AutoTokenizer from datasets import load_dataset import wandb import os, sys, datetime, argparse, gc, random import torch.distributed as dist # =========================== # Argument Parsing # =========================== def parse_args(): parser = argparse.ArgumentParser(description=\u0026#34;Train LLaMA with Custom Tokenizer\u0026#34;) parser.add_argument(\u0026#34;--model_size\u0026#34;, type=str, choices=[\u0026#34;250M\u0026#34;, \u0026#34;1B\u0026#34;, \u0026#34;2B\u0026#34;, \u0026#34;4B\u0026#34;, \u0026#34;7B\u0026#34;, \u0026#34;8B\u0026#34;], default=\u0026#34;1B\u0026#34;) parser.add_argument(\u0026#34;--run_name\u0026#34;, type=str, default=\u0026#34;custom_tokenizer_v1\u0026#34;) parser.add_argument(\u0026#34;--checkpoint\u0026#34;, type=str, default=None) parser.add_argument(\u0026#34;--tokenizer_path\u0026#34;, type=str, required=True) parser.add_argument(\u0026#34;--train_dataset\u0026#34;, type=str, required=True) parser.add_argument(\u0026#34;--test_dataset\u0026#34;, type=str, required=True) parser.add_argument(\u0026#34;--eval_samples\u0026#34;, type=int, default=100) parser.add_argument(\u0026#34;--seed\u0026#34;, type=int, default=42) parser.add_argument(\u0026#34;--delimiter_token\u0026#34;, type=str, default=\u0026#34;\u0026lt;|start_clean|\u0026gt;\u0026#34;) parser.add_argument(\u0026#34;--per_device_batch_size\u0026#34;, type=int, default=1) parser.add_argument(\u0026#34;--gradient_accumulation_steps\u0026#34;, type=int, default=16) parser.add_argument(\u0026#34;--learning_rate\u0026#34;, type=float, default=1e-5) parser.add_argument(\u0026#34;--weight_decay\u0026#34;, type=float, default=0.01) parser.add_argument(\u0026#34;--max_grad_norm\u0026#34;, type=float, default=1.0) parser.add_argument(\u0026#34;--max_steps\u0026#34;, type=int, default=-1) parser.add_argument(\u0026#34;--eval_steps\u0026#34;, type=int, default=200) parser.add_argument(\u0026#34;--save_steps\u0026#34;, type=int, default=200) parser.add_argument(\u0026#34;--disable_audio_callback\u0026#34;, action=\u0026#34;store_true\u0026#34;) return parser.parse_args() args = parse_args() MODEL_SIZE = args.model_size RUN_NAME = args.run_name rank = int(os.environ.get(\u0026#39;RANK\u0026#39;, 0)) world_size = int(os.environ.get(\u0026#39;WORLD_SIZE\u0026#39;, 1)) local_rank = int(os.environ.get(\u0026#39;LOCAL_RANK\u0026#39;, 0)) num_gpus = torch.cuda.device_count() if torch.cuda.is_available() else 1 if not torch.cuda.is_available(): print(\u0026#34;ERROR: No CUDA GPUs available!\u0026#34;) sys.exit(1) # =========================== # Load Tokenizer # =========================== print(f\u0026#34;Loading tokenizer from: {args.tokenizer_path}\u0026#34;) tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_path) vocab_size = tokenizer.vocab_size # Get delimiter token ID if hasattr(tokenizer, \u0026#39;convert_tokens_to_ids\u0026#39;): delimiter_token_id = tokenizer.convert_tokens_to_ids(args.delimiter_token) if delimiter_token_id == tokenizer.unk_token_id: print(f\u0026#34;ERROR: Delimiter token \u0026#39;{args.delimiter_token}\u0026#39; not found in tokenizer!\u0026#34;) sys.exit(1) print(f\u0026#34;Tokenizer vocab size: {vocab_size}, delimiter token ID: {delimiter_token_id}\u0026#34;) # =========================== # Dataset # =========================== class CustomDataset(Dataset): def __init__(self, sequences, tokenizer): self.sequences = sequences self.tokenizer = tokenizer def __len__(self): return len(self.sequences) def __getitem__(self, idx): seq = self.sequences[idx] if isinstance(seq, str): return torch.tensor(tokenizer.encode(seq, add_special_tokens=True), dtype=torch.long) return torch.tensor(seq, dtype=torch.long) def create_labels_with_delimiter(batch, delimiter_token_id, pad_token_id): \u0026#34;\u0026#34;\u0026#34;Mask labels before delimiter token with -100\u0026#34;\u0026#34;\u0026#34; if not batch: return {\u0026#34;input_ids\u0026#34;: torch.tensor([]), \u0026#34;labels\u0026#34;: torch.tensor([]), \u0026#34;attention_mask\u0026#34;: torch.tensor([])} input_sequences, label_sequences = [], [] for sequence in batch: sequence = torch.as_tensor(sequence, dtype=torch.long) delimiter_positions = (sequence == delimiter_token_id).nonzero(as_tuple=True)[0] input_sequences.append(sequence) labels = sequence.clone() if len(delimiter_positions) \u0026gt; 0: labels[:delimiter_positions[0].item() + 1] = -100 label_sequences.append(labels) max_len = max(len(x) for x in input_sequences) batch_size = len(input_sequences) input_ids = torch.full((batch_size, max_len), pad_token_id, dtype=torch.long) labels = torch.full((batch_size, max_len), -100, dtype=torch.long) attention_mask = torch.zeros((batch_size, max_len), dtype=torch.long) for i, (inp, lab) in enumerate(zip(input_sequences, label_sequences)): input_ids[i, :len(inp)] = inp labels[i, :len(lab)] = lab attention_mask[i, :len(inp)] = 1 return {\u0026#34;input_ids\u0026#34;: input_ids, \u0026#34;labels\u0026#34;: labels, \u0026#34;attention_mask\u0026#34;: attention_mask} def collate_fn(batch): return create_labels_with_delimiter(batch, delimiter_token_id, tokenizer.pad_token_id) # =========================== # Load Datasets # =========================== print(\u0026#34;Loading datasets...\u0026#34;) def get_dataset_split(dataset, split_name=None): if isinstance(dataset, dict): if split_name and split_name in dataset: return dataset[split_name] return list(dataset.values())[0] return dataset train_data = get_dataset_split(load_dataset(args.train_dataset), \u0026#39;train\u0026#39;) test_data = get_dataset_split(load_dataset(args.test_dataset), \u0026#39;test\u0026#39; if \u0026#39;test\u0026#39; in load_dataset(args.test_dataset) else None) if \u0026#34;sequence\u0026#34; not in train_data.column_names: print(\u0026#34;ERROR: Dataset must have \u0026#39;sequence\u0026#39; column!\u0026#34;) sys.exit(1) train_sequences = train_data[\u0026#34;sequence\u0026#34;] test_sequences = test_data[\u0026#34;sequence\u0026#34;] random.seed(args.seed) val_sequences = random.sample(test_sequences, min(args.eval_samples, len(test_sequences))) \\ if len(test_sequences) \u0026gt; args.eval_samples else test_sequences print(f\u0026#34;Training: {len(train_sequences):,} sequences, Evaluation: {len(val_sequences):,} sequences\u0026#34;) train_dataset = CustomDataset(train_sequences, tokenizer) val_dataset = CustomDataset(val_sequences, tokenizer) first_10_samples = val_sequences[:10] # =========================== # Model Configuration # =========================== def get_model_config(model_size, vocab_size, tokenizer): configs = { \u0026#34;250M\u0026#34;: {\u0026#34;hidden_size\u0026#34;: 1024, \u0026#34;intermediate_size\u0026#34;: 4096, \u0026#34;num_hidden_layers\u0026#34;: 20, \u0026#34;num_attention_heads\u0026#34;: 16, \u0026#34;num_key_value_heads\u0026#34;: 16}, \u0026#34;1B\u0026#34;: {\u0026#34;hidden_size\u0026#34;: 1536, \u0026#34;intermediate_size\u0026#34;: 6144, \u0026#34;num_hidden_layers\u0026#34;: 24, \u0026#34;num_attention_heads\u0026#34;: 24, \u0026#34;num_key_value_heads\u0026#34;: 24}, \u0026#34;2B\u0026#34;: {\u0026#34;hidden_size\u0026#34;: 2048, \u0026#34;intermediate_size\u0026#34;: 8192, \u0026#34;num_hidden_layers\u0026#34;: 28, \u0026#34;num_attention_heads\u0026#34;: 32, \u0026#34;num_key_value_heads\u0026#34;: 32}, \u0026#34;4B\u0026#34;: {\u0026#34;hidden_size\u0026#34;: 2816, \u0026#34;intermediate_size\u0026#34;: 11264, \u0026#34;num_hidden_layers\u0026#34;: 32, \u0026#34;num_attention_heads\u0026#34;: 44, \u0026#34;num_key_value_heads\u0026#34;: 44}, \u0026#34;7B\u0026#34;: {\u0026#34;hidden_size\u0026#34;: 3456, \u0026#34;intermediate_size\u0026#34;: 13824, \u0026#34;num_hidden_layers\u0026#34;: 36, \u0026#34;num_attention_heads\u0026#34;: 54, \u0026#34;num_key_value_heads\u0026#34;: 54}, \u0026#34;8B\u0026#34;: {\u0026#34;hidden_size\u0026#34;: 3584, \u0026#34;intermediate_size\u0026#34;: 14336, \u0026#34;num_hidden_layers\u0026#34;: 40, \u0026#34;num_attention_heads\u0026#34;: 56, \u0026#34;num_key_value_heads\u0026#34;: 56}, } params = configs[model_size] return LlamaConfig( vocab_size=vocab_size, max_position_embeddings=8192, rms_norm_eps=1e-6, rope_theta=100000.0, attention_bias=False, attention_dropout=0.1, hidden_act=\u0026#34;silu\u0026#34;, hidden_dropout_prob=0.3, initializer_range=0.005, use_cache=True, pad_token_id=tokenizer.pad_token_id, bos_token_id=tokenizer.bos_token_id, eos_token_id=tokenizer.eos_token_id, tie_word_embeddings=True, attn_implementation=\u0026#34;flash_attention_2\u0026#34;, torch_dtype=torch.bfloat16, **params ) # =========================== # Initialize Model # =========================== gc.collect() if torch.cuda.is_available(): torch.cuda.empty_cache() if args.checkpoint: print(f\u0026#34;Loading from checkpoint: {args.checkpoint}\u0026#34;) loaded_config = LlamaConfig.from_pretrained(args.checkpoint, attn_implementation=\u0026#34;flash_attention_2\u0026#34;) checkpoint_vocab_size = loaded_config.vocab_size if checkpoint_vocab_size != vocab_size: print(f\u0026#34;Resizing vocab: {checkpoint_vocab_size} -\u0026gt; {vocab_size}\u0026#34;) loaded_config.vocab_size = vocab_size loaded_config.pad_token_id = tokenizer.pad_token_id loaded_config.bos_token_id = tokenizer.bos_token_id loaded_config.eos_token_id = tokenizer.eos_token_id base_model = LlamaForCausalLM.from_pretrained(args.checkpoint, config=loaded_config, device_map=\u0026#34;cpu\u0026#34;, torch_dtype=torch.bfloat16, low_cpu_mem_usage=True) model = LlamaForCausalLM(loaded_config) model.load_state_dict(base_model.state_dict(), strict=False) if checkpoint_vocab_size != vocab_size: model.resize_token_embeddings(vocab_size) del base_model gc.collect() model = model.to(dtype=torch.bfloat16) else: print(\u0026#34;Initializing new model\u0026#34;) config = get_model_config(MODEL_SIZE, vocab_size, tokenizer) model = LlamaForCausalLM(config) model = model.to(dtype=torch.bfloat16, device=\u0026#39;cpu\u0026#39;) if model.config.vocab_size != vocab_size: model.resize_token_embeddings(vocab_size) # =========================== # Training Setup # =========================== if rank == 0: wandb.init(project=f\u0026#34;Custom-Tokenizer-{RUN_NAME}\u0026#34;, name=f\u0026#34;{RUN_NAME}_{MODEL_SIZE}_{datetime.datetime.now().strftime(\u0026#39;%Y%m%d_%H%M%S\u0026#39;)}\u0026#34;) effective_batch_size = args.per_device_batch_size * args.gradient_accumulation_steps * num_gpus steps_per_epoch = len(train_sequences) // effective_batch_size if len(train_sequences) \u0026gt; 0 else 0 max_steps = max(steps_per_epoch, 100) if args.max_steps == -1 else args.max_steps training_args = TrainingArguments( output_dir=f\u0026#34;./{MODEL_SIZE}_results_{RUN_NAME}\u0026#34;, max_steps=max_steps, per_device_train_batch_size=args.per_device_batch_size, per_device_eval_batch_size=1, gradient_accumulation_steps=args.gradient_accumulation_steps, logging_steps=10, eval_strategy=\u0026#34;steps\u0026#34;, eval_steps=args.eval_steps, save_steps=args.save_steps, save_total_limit=5, learning_rate=args.learning_rate, warmup_steps=int(max_steps * 0.02), weight_decay=args.weight_decay, max_grad_norm=args.max_grad_norm, lr_scheduler_type=\u0026#34;cosine\u0026#34;, optim=\u0026#34;adamw_torch\u0026#34;, report_to=[\u0026#34;wandb\u0026#34;], save_strategy=\u0026#34;steps\u0026#34;, bf16=True, save_safetensors=True, dataloader_pin_memory=True, dataloader_num_workers=0, gradient_checkpointing=True, gradient_checkpointing_kwargs={\u0026#34;use_reentrant\u0026#34;: False}, dataloader_drop_last=True, remove_unused_columns=False, ddp_find_unused_parameters=False, local_rank=local_rank, load_best_model_at_end=True, metric_for_best_model=\u0026#34;eval_loss\u0026#34;, greater_is_better=False, label_names=[\u0026#34;labels\u0026#34;], seed=args.seed, data_seed=args.seed, ) trainer = Trainer( model=model, args=training_args, train_dataset=train_dataset, eval_dataset=val_dataset, tokenizer=tokenizer, data_collator=collate_fn, ) print(\u0026#34;Starting training...\u0026#34;) trainer.train() print(\u0026#34;Training completed!\u0026#34;) # =========================== # Save Final Model # =========================== if rank == 0: final_model_path = f\u0026#34;./final_model_{MODEL_SIZE}_{RUN_NAME}\u0026#34; model.save_pretrained(final_model_path, safe_serialization=True) tokenizer.save_pretrained(final_model_path) print(f\u0026#34;Final model saved to: {final_model_path}\u0026#34;) if world_size \u0026gt; 1 and dist.is_initialized(): dist.barrier() print(\u0026#34;Done!\u0026#34;) ","permalink":"https://aadonis-ai.github.io/notebook/build-and-train/hf-trainer/","summary":"A complete training script for LLaMA (250M–8B) with a custom tokenizer, label masking, gradient checkpointing, and multi-GPU support via Accelerate.","title":"HuggingFace Trainer: Custom LLaMA Training"},{"content":"The Qwen3 ecosystem includes a forced aligner that provides high-resolution word and phoneme timestamps by aligning a known transcript to audio. Unlike MFA, it doesn\u0026rsquo;t require a pronunciation dictionary and handles accented and multilingual speech better.\nASR Model: Qwen/Qwen3-ASR-1.7B Forced Aligner: Qwen/Qwen3-ForcedAligner-0.6B Word Alignment Script import torch from qwen_asr import Qwen3ASRModel def align_audio(audio_path, text=None): device = \u0026#34;cuda\u0026#34; if torch.cuda.is_available() else \u0026#34;cpu\u0026#34; model = Qwen3ASRModel.from_pretrained( \u0026#34;Qwen/Qwen3-ASR-1.7B\u0026#34;, dtype=torch.bfloat16, device_map=device, forced_aligner=\u0026#34;Qwen/Qwen3-ForcedAligner-0.6B\u0026#34; ) # If text is provided → forced alignment. # If text is None → transcribes first, then aligns. results = model.transcribe( audio=audio_path, language=\u0026#34;English\u0026#34;, return_time_stamps=True ) for ts in results.time_stamps: print(f\u0026#34;Word: {ts.text:15s} | Start: {ts.start_time:.3f}s | End: {ts.end_time:.3f}s\u0026#34;) # align_audio(\u0026#34;path/to/audio.wav\u0026#34;) Key Features Sub-word precision — high-resolution timestamps for linguistic analysis Robustness — handles background noise and varied accents Multilingual — supports all languages in Qwen3\u0026rsquo;s training set No dictionary required — unlike MFA, works on any language/accent without a G2P dictionary ","permalink":"https://aadonis-ai.github.io/notebook/speech-alignment/qwen3-aligner/","summary":"Precise word and phoneme-level timestamps using Qwen3-ASR and its paired forced aligner — multilingual, robust to noise.","title":"Qwen3 Forced Aligner"},{"content":"SNAC matches DAC in perceptual reconstruction quality for speech and music while having significantly lower token rate. It uses multi-scale residual vector quantization with downsampled residuals, depthwise convolutions, local attention, and noise blocks. Best general-purpose choice for speech enhancement and editing pipelines.\nPaper: SNAC: Multi-Scale Neural Audio Codec GitHub: hubertsiuzdak/snac\nInstallation pip install snac librosa numpy torch Wrapper Class import numpy as np import torch import librosa from snac import SNAC class SNAC_tools: def __init__(self): self.model = SNAC.from_pretrained(\u0026#34;hubertsiuzdak/snac_24khz\u0026#34;).eval().cuda() self.num_codebooks = 3 self.device = next(self.model.parameters()).device def audio_to_codes(self, audio, sr): if sr is None or sr != 24000: audio = librosa.resample(audio, orig_sr=sr, target_sr=24000) audio = audio.astype(np.float32) audio = torch.from_numpy(audio).unsqueeze(0).unsqueeze(0).cuda() with torch.inference_mode(): codes = self.model.encode(audio) return codes def codes_to_audio(self, codes): with torch.inference_mode(): audio = self.model.decode(codes) return audio ","permalink":"https://aadonis-ai.github.io/notebook/neural-audio-codecs/snac/","summary":"SNAC setup and encode/decode wrapper — matches DAC quality at significantly lower token rate via multi-scale residual vector quantization.","title":"SNAC — Multi-Scale Neural Audio Codec"},{"content":"When working with Neural Audio Codecs (NACs) and Transformer-based models, you need a custom tokenizer that maps discrete codec tokens to integer IDs, handles special tokens (BOS, EOS, PAD), and is fully compatible with the HuggingFace AutoTokenizer API so you can upload it to the Hub and reuse it across projects.\nThe example below is for the 22kHz SNAC codec (3 codebooks, 4096 vocab size per codebook), but the pattern generalizes to any NAC.\nKey design decision: NAC tokens are set as special tokens so the tokenizer treats them atomically. The standard .decode() and .batch_decode() skip special tokens, so we override them with custom functions that preserve NAC tokens.\nBuilding the Tokenizer import os import re import json from tokenizers import Tokenizer, models, normalizers, pre_tokenizers, decoders, processors from huggingface_hub import login, HfApi from transformers import AutoTokenizer # Adapt for your NAC num_codebooks = 3 codebook_size = 4096 special_tokens = [\u0026#34;\u0026lt;|bos|\u0026gt;\u0026#34;, \u0026#34;\u0026lt;|pad|\u0026gt;\u0026#34;, \u0026#34;\u0026lt;|eos|\u0026gt;\u0026#34;, \u0026#34;\u0026lt;|start_clean|\u0026gt;\u0026#34;, \u0026#34;\u0026lt;|unk|\u0026gt;\u0026#34;] codebook_tokens = [f\u0026#34;\u0026lt;|q{i}_t{a}|\u0026gt;\u0026#34; for i in range(num_codebooks) for a in range(codebook_size)] vocab_tokens = special_tokens + codebook_tokens token_to_id = {tok: i for i, tok in enumerate(vocab_tokens)} os.makedirs(\u0026#34;./snac_tokenizer_hf\u0026#34;, exist_ok=True) hf_tokenizer = Tokenizer(models.WordLevel(vocab=token_to_id, unk_token=\u0026#34;\u0026lt;|unk|\u0026gt;\u0026#34;)) hf_tokenizer.normalizer = normalizers.Sequence([normalizers.NFC()]) # Pre-tokenizer: match all tokens all_token_pattern = \u0026#34;|\u0026#34;.join(re.escape(token) for token in vocab_tokens) hf_tokenizer.pre_tokenizer = pre_tokenizers.Split(pattern=all_token_pattern, behavior=\u0026#34;isolated\u0026#34;) # Add BOS/EOS post-processing hf_tokenizer.post_processor = processors.TemplateProcessing( single=\u0026#34;\u0026lt;|bos|\u0026gt; $A \u0026lt;|eos|\u0026gt;\u0026#34;, special_tokens=[(\u0026#34;\u0026lt;|bos|\u0026gt;\u0026#34;, 0), (\u0026#34;\u0026lt;|eos|\u0026gt;\u0026#34;, 2)] ) hf_tokenizer.decoder = decoders.Sequence([decoders.Replace(\u0026#34;▁\u0026#34;, \u0026#34; \u0026#34;)]) hf_tokenizer.save(\u0026#34;./snac_tokenizer_hf/tokenizer.json\u0026#34;) # Create tokenizer config snac_tokens_only = [t for t in vocab_tokens if t not in special_tokens] tokenizer_config = { \u0026#34;tokenizer_class\u0026#34;: \u0026#34;PreTrainedTokenizerFast\u0026#34;, \u0026#34;auto_map\u0026#34;: {\u0026#34;AutoTokenizer\u0026#34;: [\u0026#34;tokenizer.json\u0026#34;, None]}, \u0026#34;model_max_length\u0026#34;: 8192, \u0026#34;padding_side\u0026#34;: \u0026#34;right\u0026#34;, \u0026#34;truncation_side\u0026#34;: \u0026#34;right\u0026#34;, \u0026#34;clean_up_tokenization_spaces\u0026#34;: True, \u0026#34;bos_token\u0026#34;: \u0026#34;\u0026lt;|bos|\u0026gt;\u0026#34;, \u0026#34;eos_token\u0026#34;: \u0026#34;\u0026lt;|eos|\u0026gt;\u0026#34;, \u0026#34;pad_token\u0026#34;: \u0026#34;\u0026lt;|pad|\u0026gt;\u0026#34;, \u0026#34;unk_token\u0026#34;: \u0026#34;\u0026lt;|unk|\u0026gt;\u0026#34;, \u0026#34;additional_special_tokens\u0026#34;: [\u0026#34;\u0026lt;|start_clean|\u0026gt;\u0026#34;] + snac_tokens_only } with open(\u0026#34;./snac_tokenizer_hf/tokenizer_config.json\u0026#34;, \u0026#34;w\u0026#34;) as f: json.dump(tokenizer_config, f, indent=2) Uploading to Hugging Face Hub HF_API_KEY = \u0026#34;hf_YOUR_TOKEN_HERE\u0026#34; login(token=HF_API_KEY) api = HfApi() api.create_repo(\u0026#34;YOUR_HF_NAME/snac_tokenizer\u0026#34;, token=HF_API_KEY, exist_ok=True) api.upload_folder( folder_path=\u0026#34;./snac_tokenizer_hf\u0026#34;, repo_id=\u0026#34;YOUR_HF_NAME/snac_tokenizer\u0026#34;, token=HF_API_KEY ) print(\u0026#34;Tokenizer uploaded successfully!\u0026#34;) Loading and Patching decode() hf_tokenizer = AutoTokenizer.from_pretrained(\u0026#34;YOUR_HF_NAME/snac_tokenizer\u0026#34;) basic_special_tokens = [\u0026#34;\u0026lt;|bos|\u0026gt;\u0026#34;, \u0026#34;\u0026lt;|pad|\u0026gt;\u0026#34;, \u0026#34;\u0026lt;|eos|\u0026gt;\u0026#34;, \u0026#34;\u0026lt;|unk|\u0026gt;\u0026#34;] original_decode = hf_tokenizer.decode original_batch_decode = hf_tokenizer.batch_decode def custom_decode(token_ids, skip_special_tokens=True, **kwargs): if skip_special_tokens: filtered_ids = [ t_id for t_id in token_ids if hf_tokenizer.convert_ids_to_tokens(t_id) not in basic_special_tokens ] return original_decode(filtered_ids, skip_special_tokens=False, **kwargs) return original_decode(token_ids, skip_special_tokens=False, **kwargs) def custom_batch_decode(token_ids_list, skip_special_tokens=True, **kwargs): results = [] for ids in token_ids_list: if hasattr(ids, \u0026#39;tolist\u0026#39;): ids = ids.tolist() results.append(custom_decode(ids, skip_special_tokens=skip_special_tokens, **kwargs)) return results hf_tokenizer.decode = custom_decode hf_tokenizer.batch_decode = custom_batch_decode print(\u0026#34;Tokenizer ready with custom decode functions.\u0026#34;) ","permalink":"https://aadonis-ai.github.io/notebook/build-and-train/custom-tokenizer/","summary":"Building a HuggingFace-compatible tokenizer for neural audio codec tokens — vocabulary design, special tokens, and uploading to the Hub.","title":"Custom Tokenizer for Audio LLMs"},{"content":"DAC encodes audio into 9 codebooks at 89 frames/sec, giving 801 tokens/sec when flattened. The high token rate makes it the highest-quality general-purpose codec, but sequences are long — plan accordingly for Transformer context windows.\nPaper: Descript Audio Codec HuggingFace: descript/dac_44khz GitHub: descriptinc/descript-audio-codec\nInstallation pip install transformers soundfile librosa numpy torch Wrapper Class The class below handles encode/decode and both flattening layouts (time-major and codebook-major) for use in autoregressive Transformer pipelines.\nfrom transformers import DacModel, AutoProcessor import torch import librosa import soundfile as sf import numpy as np class DAC: def __init__(self): self.model = DacModel.from_pretrained(\u0026#34;descript/dac_44khz\u0026#34;) self.processor = AutoProcessor.from_pretrained(\u0026#34;descript/dac_44khz\u0026#34;) self.num_codebooks = 9 self.device = next(self.model.parameters()).device def audio_to_codebook_matrix(self, audio_wav_path): audio, sr = sf.read(audio_wav_path) if sr != self.processor.sampling_rate: audio = librosa.resample(audio, orig_sr=sr, target_sr=self.processor.sampling_rate) inputs = self.processor(raw_audio=audio, sampling_rate=self.processor.sampling_rate, return_tensors=\u0026#34;pt\u0026#34;) encoder_outputs = self.model.encode(inputs[\u0026#34;input_values\u0026#34;].to(self.device)) return encoder_outputs.audio_codes def flatten_matrix_to_vector_time_major(self, codes): # Interleaves codebooks: t0_cb0, t0_cb1, ..., t0_cb8, t1_cb0, ... return codes[0].T.flatten().tolist() def flatten_matrix_to_vector_codebook_major(self, codes): # All of codebook 0, then all of codebook 1, ... return codes[0].flatten().tolist() def vector_time_major_to_matrix(self, tokens): tokens = torch.tensor(tokens, dtype=torch.long, device=self.device) num_steps = len(tokens) // self.num_codebooks return tokens.view(num_steps, self.num_codebooks).T.unsqueeze(0) def vector_codebook_major_to_matrix(self, tokens): tokens = torch.tensor(tokens, dtype=torch.long, device=self.device) num_steps = len(tokens) // self.num_codebooks return tokens.view(self.num_codebooks, num_steps).unsqueeze(0) def codebook_matrix_to_audio(self, audio_codes): audio_values = self.model.decode(audio_codes=audio_codes.to( self.device)).audio_values return audio_values[0].cpu().detach().numpy() def audio_array_to_audio_wav(self, audio_array, output_path): sf.write(output_path, audio_array, self.processor.sampling_rate) return output_path ","permalink":"https://aadonis-ai.github.io/notebook/neural-audio-codecs/dac/","summary":"DAC setup and a wrapper class for encoding/decoding audio as flattened token sequences — time-major and codebook-major layouts.","title":"DAC — Descript Audio Codec"},{"content":"EZ-VC is a simple zero-shot voice conversion model that combines discrete speech representations from XEUS (a self-supervised model trained on 4000 languages) with a flow-matching diffusion decoder based on F5-TTS. The key differentiator is a single encoder architecture — no separate speaker/content encoders needed.\nPaper: EZ-VC (EMNLP 2025 Findings) Codebase: Github HuggingFace: Model\nPerformance vs Seed-VC Model SSIM ↑ NMOS ↑ SMOS ↑ UTMOS ↑ Seed-VC 0.69 3.55 3.78 3.02 kNN-VC 0.59 1.94 2.05 2.42 Vec2Wav2.0 0.61 3.67 3.55 3.55 EZ-VC 0.71 3.91 3.90 3.56 When to use EZ-VC: cross-lingual conversion (especially unseen languages), highest naturalness scores. When to use Seed-VC: real-time conversion, singing, accent/emotion conversion (V2).\nInstallation git clone https://github.com/EZ-VC/EZ-VC cd EZ-VC git submodule update --init --recursive conda create -n ez-vc python=3.10 conda activate ez-vc # NVIDIA GPU pip install torch==2.4.0+cu124 torchaudio==2.4.0+cu124 --extra-index-url https://download.pytorch.org/whl/cu124 # Apple Silicon # pip install torch torchaudio pip install -e . # Install espnet for XEUS (EXACTLY this version) pip install \u0026#39;espnet @ git+https://github.com/wanchichen/espnet.git@ssl\u0026#39; Inference Script #!/usr/bin/env python3 \u0026#34;\u0026#34;\u0026#34; EZ-VC Voice Conversion Inference Usage: python inference.py --ref_audio ref.wav --src_audio source.wav --output output.wav \u0026#34;\u0026#34;\u0026#34; import os, sys, argparse import soundfile as sf import torch from pathlib import Path from cached_path import cached_path from omegaconf import OmegaConf from hydra.utils import get_class src_path = os.path.join(os.getcwd(), \u0026#34;src\u0026#34;) if src_path not in sys.path: sys.path.append(src_path) from f5_tts.infer.utils_infer import infer_process, load_model, load_vocoder, target_rms from f5_tts.infer.utils_xeus import ApplyKmeans, load_xeus_model, extract_units DEFAULT_CFG_STRENGTH = 2.0 DEFAULT_SWAY_COEF = -1.0 def load_all_models(device, vocoder_name, config_path): print(f\u0026#34;Loading models on {device}...\u0026#34;) xeus_model = load_xeus_model(device).eval() apply_kmeans = ApplyKmeans(device) vocoder = load_vocoder(vocoder_name=vocoder_name, device=device) ckpt_file = str(cached_path(\u0026#34;hf://SPRINGLab/EZ-VC/model_2700000.safetensors\u0026#34;)) vocab_file = str(cached_path(\u0026#34;hf://SPRINGLab/EZ-VC/vocab.txt\u0026#34;)) model_cfg = OmegaConf.load(config_path) model_cls = get_class(f\u0026#34;f5_tts.model.{model_cfg.model.backbone}\u0026#34;) model_arch = model_cfg.model.arch ema_model = load_model( model_cls, model_arch, ckpt_file, mel_spec_type=vocoder_name, vocab_file=vocab_file, device=device, ) return xeus_model, apply_kmeans, vocoder, ema_model def run_inference(ref_audio, src_audio, output_path, device=\u0026#34;cuda\u0026#34;, nfe=32, speed=1.0, config_path=\u0026#34;src/f5_tts/configs/F5TTS_Base_EZ-VC.yaml\u0026#34;): xeus_model, apply_kmeans, vocoder, ema_model = load_all_models(device, \u0026#34;bigvgan\u0026#34;, config_path) print(f\u0026#34;Extracting units from Reference: {ref_audio}\u0026#34;) ref_text = extract_units(ref_audio, xeus_model, apply_kmeans, device) print(f\u0026#34;Extracting units from Source: {src_audio}\u0026#34;) src_text = extract_units(src_audio, xeus_model, apply_kmeans, device) print(f\u0026#34;Running Inference (NFE={nfe})...\u0026#34;) audio_segment, final_sample_rate, _ = infer_process( ref_audio, ref_text, src_text, ema_model, vocoder, mel_spec_type=\u0026#34;bigvgan\u0026#34;, target_rms=target_rms, cross_fade_duration=0.15, nfe_step=nfe, cfg_strength=DEFAULT_CFG_STRENGTH, sway_sampling_coef=DEFAULT_SWAY_COEF, speed=speed, fix_duration=None, device=device, ) sf.write(output_path, audio_segment, final_sample_rate) print(f\u0026#34;\\nSaved to: {output_path}\u0026#34;) if __name__ == \u0026#34;__main__\u0026#34;: parser = argparse.ArgumentParser(description=\u0026#34;EZ-VC Inference\u0026#34;) parser.add_argument(\u0026#34;--ref_audio\u0026#34;, type=str, required=True, help=\u0026#34;Reference audio (target voice)\u0026#34;) parser.add_argument(\u0026#34;--src_audio\u0026#34;, type=str, required=True, help=\u0026#34;Source audio (content to convert)\u0026#34;) parser.add_argument(\u0026#34;--output\u0026#34;, type=str, default=\u0026#34;output_ezvc.wav\u0026#34;) parser.add_argument(\u0026#34;--device\u0026#34;, type=str, default=\u0026#34;cuda\u0026#34; if torch.cuda.is_available() else \u0026#34;cpu\u0026#34;) parser.add_argument(\u0026#34;--nfe\u0026#34;, type=int, default=32, help=\u0026#34;Inference steps (higher = better quality)\u0026#34;) parser.add_argument(\u0026#34;--config\u0026#34;, type=str, default=\u0026#34;src/f5_tts/configs/F5TTS_Base_EZ-VC.yaml\u0026#34;) args = parser.parse_args() run_inference( ref_audio=args.ref_audio, src_audio=args.src_audio, output_path=args.output, device=args.device, nfe=args.nfe, config_path=args.config ) Architecture Overview EZ-VC uses a two-stage pipeline:\nSpeech-to-Units (XEUS + K-means)\nXEUS encoder processes speech at 50 embeddings/second (25ms window, 20ms stride) K-means (500 clusters) quantizes features from the 14th layer Results in discrete speech units capturing linguistic content without speaker identity Units-to-Speech (F5-TTS based)\nConditional flow matching diffusion decoder Reconstructs speech from discrete units + speaker reference via in-context learning ","permalink":"https://aadonis-ai.github.io/notebook/voice-conversion/ez-vc/","summary":"Easy Zero-shot Any-to-Any Voice Conversion — single encoder architecture, excellent cross-lingual performance, and a clean inference script.","title":"EZ-VC"},{"content":"The Montreal Forced Aligner (MFA) aligns spoken audio with its transcript at word and phoneme level. It is a prerequisite for speech editing (to locate edit boundaries), dataset preparation, and TTS training.\npyfoal: Github MFA: Github MFA docs: readthedocs\nInstallation conda install -c conda-forge montreal-forced-aligner pip install pyfoal Inference Script import os import torch import librosa import pyfoal os.environ[\u0026#39;MFA_ROOT_DIR\u0026#39;] = \u0026#39;/path/to/mfa_temp\u0026#39; os.makedirs(os.environ[\u0026#39;MFA_ROOT_DIR\u0026#39;], exist_ok=True) def align_audio_text(audio_path, text): \u0026#34;\u0026#34;\u0026#34; Align audio with text using MFA. Returns a list of dicts with word, phonemes, start, end, is_silence. \u0026#34;\u0026#34;\u0026#34; audio_np, sr = librosa.load(audio_path, sr=16000) audio = torch.FloatTensor(audio_np).unsqueeze(0) alignment = pyfoal.from_text_and_audio( text, audio, 16000, aligner=\u0026#39;mfa\u0026#39;, gpu=0 ) results = [] for mfa_word in alignment.words(): word_text = mfa_word.word phonemes = [p.phoneme for p in mfa_word.phonemes] if mfa_word.phonemes: start = mfa_word.phonemes[0]._start end = mfa_word.phonemes[-1]._end else: start = end = 0.0 is_silence = (word_text.strip() == \u0026#39;\u0026#39; or len(phonemes) == 0) results.append({ \u0026#39;word\u0026#39;: word_text, \u0026#39;phonemes\u0026#39;: phonemes, \u0026#39;start\u0026#39;: start, \u0026#39;end\u0026#39;: end, \u0026#39;is_silence\u0026#39;: is_silence }) return results if __name__ == \u0026#34;__main__\u0026#34;: audio_path = \u0026#34;test_audio.wav\u0026#34; text = \u0026#34;This is a transcript of the test audio.\u0026#34; results = align_audio_text(audio_path, text) for i, w in enumerate(results): if w[\u0026#39;is_silence\u0026#39;]: print(f\u0026#34;{i:2d}. [SILENCE] [{w[\u0026#39;start\u0026#39;]:.3f}s - {w[\u0026#39;end\u0026#39;]:.3f}s]\u0026#34;) else: print(f\u0026#34;{i:2d}. \u0026#39;{w[\u0026#39;word\u0026#39;]:10s}\u0026#39; [{w[\u0026#39;start\u0026#39;]:.3f}s - {w[\u0026#39;end\u0026#39;]:.3f}s] {w[\u0026#39;phonemes\u0026#39;]}\u0026#34;) Common Problem: GitHub Rate Limit Problem: pyfoal checks GitHub for MFA model files on every call, hitting the 60 requests/hour rate limit quickly during bulk processing.\nFix: Edit the pyfoal source to check for local files first.\nFile: path/to/conda_env/lib/python3.10/site-packages/pyfoal/baselines/mfa.py\nBackup first:\ncp path/to/pyfoal/baselines/mfa.py path/to/pyfoal/baselines/mfa.py.backup Change lines 64–67 from:\nmanager = mfa.models.ModelManager() manager.download_model(\u0026#39;dictionary\u0026#39;, \u0026#39;english_mfa\u0026#39;) manager.download_model(\u0026#39;acoustic\u0026#39;, \u0026#39;english_mfa\u0026#39;) To:\nimport os manager = mfa.models.ModelManager() mfa_root = os.environ.get(\u0026#39;MFA_ROOT_DIR\u0026#39;, os.path.expanduser(\u0026#39;~/Documents/MFA\u0026#39;)) dict_path = os.path.join(mfa_root, \u0026#39;pretrained_models/dictionary/english_mfa.dict\u0026#39;) acoustic_path = os.path.join(mfa_root, \u0026#39;pretrained_models/acoustic/english_mfa.zip\u0026#39;) if os.path.exists(dict_path): print(f\u0026#34;[MFA] Dictionary found locally\u0026#34;) else: print(f\u0026#34;[MFA] Downloading dictionary from GitHub\u0026#34;) manager.download_model(\u0026#39;dictionary\u0026#39;, \u0026#39;english_mfa\u0026#39;) if os.path.exists(acoustic_path): print(f\u0026#34;[MFA] Acoustic model found locally\u0026#34;) else: print(f\u0026#34;[MFA] Downloading acoustic model from GitHub\u0026#34;) manager.download_model(\u0026#39;acoustic\u0026#39;, \u0026#39;english_mfa\u0026#39;) This has been tested and eliminates the rate limit problem entirely.\n","permalink":"https://aadonis-ai.github.io/notebook/speech-alignment/mfa-pyfoal/","summary":"Word and phoneme-level forced alignment with MFA and pyfoal — installation, inference script, and a fix for GitHub rate limit errors.","title":"Montreal Forced Aligner (MFA)"},{"content":"These datasets are used to synthesize degraded speech by convolving clean recordings with room impulse responses (RIRs) and mixing with environmental noise at varying SNRs.\nESC-50 2,000 environmental audio recordings across 50 classes. Lightweight and commonly used.\nHuggingFace:\nfrom datasets import load_dataset dataset = load_dataset(\u0026#34;ashraq/esc50\u0026#34;) Kaggle:\nimport kagglehub path = kagglehub.dataset_download(\u0026#34;mmoreaux/environmental-sound-classification-50\u0026#34;) print(\u0026#34;Path:\u0026#34;, path) Room Impulse Response and Noise Database (OpenSLR #28) Large collection of real and simulated RIRs and noise recordings.\nPage: OpenSLR RIRs\nwget https://openslr.trmal.net/resources/28/rirs_noises.zip unzip rirs_noises.zip -d rirs_noises DEMAND Diverse Environments Multichannel Acoustic Noise Database — 18 real noise environments.\nTARGET_DIR=\u0026#34;/path/to/your/datasets/demand\u0026#34; mkdir -p \u0026#34;$TARGET_DIR\u0026#34; curl -L -o \u0026#34;$TARGET_DIR/demand.zip\u0026#34; \\ https://www.kaggle.com/api/v1/datasets/download/chrisfilo/demand unzip \u0026#34;$TARGET_DIR/demand.zip\u0026#34; -d \u0026#34;$TARGET_DIR\u0026#34; rm \u0026#34;$TARGET_DIR/demand.zip\u0026#34; VGG Sound 200+ categories of audio-visual events extracted from YouTube. More than 200k 10-second clips.\nKaggle: VGG Sound\nimport kagglehub path = kagglehub.dataset_download(\u0026#34;codebreaker619/vggsound\u0026#34;) print(\u0026#34;Path:\u0026#34;, path) ","permalink":"https://aadonis-ai.github.io/notebook/speech-datasets/noise-rirs/","summary":"Downloading ESC-50, the OpenSLR RIR database, DEMAND, and VGG Sound for noise augmentation and speech enhancement training.","title":"Noise \u0026 RIR Datasets"},{"content":"HiFiTTS-2 High-quality English speech dataset at 44kHz. Ideal for TTS and codec training.\nMore information: HiFiTTS-2 on Hugging Face\nmkdir -p ~/datasets/hifitts2 cd ~/datasets/hifitts2 # Download manifest and chapters (replace 44khz with 22khz for lower SR) wget https://huggingface.co/datasets/nvidia/hifitts-2/resolve/main/44khz/manifest_44khz.json wget https://huggingface.co/datasets/nvidia/hifitts-2/resolve/main/44khz/chapters_44khz.json Then install NeMo Speech Data Processor and download the audio:\npython /home/NeMo-speech-data-processor/main.py \\ --config-path=\u0026#34;/home/NeMo-speech-data-processor/dataset_configs/english/hifitts2\u0026#34; \\ --config-name=\u0026#34;config_44khz.yaml\u0026#34; \\ workspace_dir=\u0026#34;/home/hifitts2\u0026#34; \\ max_workers=8 SparkAudio VoxBox A merged corpus of 60k+ hours of English and Chinese speech from CommonVoice, GigaSpeech, LibriSpeech, and others.\nHuggingFace: SparkAudio/voxbox GitHub: VoxBox\nThe script below downloads a specific subset (e.g. casia, cremad, emns) by name:\n\u0026#34;\u0026#34;\u0026#34; Download a voxbox dataset subset. Usage: python download_voxbox_subset.py --subset casia python download_voxbox_subset.py --subset cremad --download_dir ./downloads \u0026#34;\u0026#34;\u0026#34; import os import argparse from huggingface_hub import login, HfApi, hf_hub_download from tqdm import tqdm def download_voxbox_subset(subset_name, repo_id=\u0026#34;SparkAudio/voxbox\u0026#34;, download_dir=None, hf_api_key=None): if download_dir is None: download_dir = os.path.join(os.environ.get(\u0026#39;TMPDIR\u0026#39;, \u0026#39;./downloads\u0026#39;), \u0026#39;voxbox_downloads\u0026#39;) os.makedirs(download_dir, exist_ok=True) if hf_api_key: login(token=hf_api_key) api = HfApi() dataset_info = api.dataset_info(repo_id=repo_id) all_paths = [s.rfilename for s in dataset_info.siblings] downloaded_files = [] # Download metadata metadata_path = f\u0026#34;metadata/{subset_name}.jsonl\u0026#34; if metadata_path in all_paths: hf_hub_download( repo_id=repo_id, repo_type=\u0026#34;dataset\u0026#34;, filename=metadata_path, local_dir=download_dir, local_dir_use_symlinks=False, token=hf_api_key ) downloaded_files.append(os.path.join(download_dir, metadata_path)) else: print(f\u0026#34;Metadata not found: {metadata_path}\u0026#34;) available = [p for p in all_paths if p.startswith(\u0026#34;metadata/\u0026#34;)] for m in available[:10]: print(f\u0026#34; - {m}\u0026#34;) # Download audio tar.gz files audio_tars = [f for f in all_paths if f.startswith(f\u0026#34;audios/{subset_name}/\u0026#34;) and f.endswith(\u0026#34;.tar.gz\u0026#34;)] if not audio_tars: print(f\u0026#34;No audio files found for subset \u0026#39;{subset_name}\u0026#39;\u0026#34;) return downloaded_files for tar_file in tqdm(audio_tars, desc=\u0026#34;Downloading audio\u0026#34;): hf_hub_download( repo_id=repo_id, repo_type=\u0026#34;dataset\u0026#34;, filename=tar_file, local_dir=download_dir, local_dir_use_symlinks=False, token=hf_api_key ) downloaded_files.append(os.path.join(download_dir, tar_file)) print(f\u0026#34;Download complete: {len(downloaded_files)} files in {download_dir}\u0026#34;) return downloaded_files def main(): parser = argparse.ArgumentParser() parser.add_argument(\u0026#39;--subset\u0026#39;, type=str, required=True) parser.add_argument(\u0026#39;--repo_id\u0026#39;, type=str, default=\u0026#39;SparkAudio/voxbox\u0026#39;) parser.add_argument(\u0026#39;--download_dir\u0026#39;, type=str, default=None) parser.add_argument(\u0026#39;--hf_api_key\u0026#39;, type=str, default=None) args = parser.parse_args() if args.hf_api_key is None: args.hf_api_key = os.environ.get(\u0026#39;HF_TOKEN\u0026#39;) download_voxbox_subset( subset_name=args.subset, repo_id=args.repo_id, download_dir=args.download_dir, hf_api_key=args.hf_api_key ) if __name__ == \u0026#34;__main__\u0026#34;: main() ","permalink":"https://aadonis-ai.github.io/notebook/speech-datasets/clean-speech/","summary":"Downloading HiFiTTS-2 (high-quality 44kHz) and SparkAudio VoxBox (60k+ hours, multi-language merged corpus).","title":"Clean Speech Datasets"},{"content":"Comparison Table Codec Flat Token Rate (tok/sec) Codebooks Framerate (Hz) Vocab Size DAC 801 9 89 1024 SNAC 24kHz ~150 3 varies 4096 WavTokenizer 40 or 75 1 — 4096 X-Codec2 ~50 1 — 65536 Flat Token Rate — codecs produce a (codebooks × timesteps) matrix. Flattened into a 1D sequence for autoregressive modeling, the total tokens/sec is codebooks × framerate.\nDAC produces 9 codebooks at 89 frames/sec → 801 tok/sec. High quality, but sequences are ~10× longer than WavTokenizer for the same audio duration. Memory-intensive for LLM training.\nSNAC matches DAC in perceptual quality with ~3–6× lower token rate. Best general-purpose choice for speech tasks.\nWavTokenizer achieves extreme compression (40–75 tok/sec, single codebook). Excellent for clean TTS tasks. Performs poorly on degraded/noisy speech — it was not trained on such data and can introduce artifacts.\nX-Codec2 — stable at version 1.3.0. Single codebook, low token rate, large vocabulary. Pairs well with Llasa for TTS.\nWhen to Use Each Task Recommended Codec Speech enhancement (noisy input) SNAC or DAC TTS / zero-shot voice cloning X-Codec2 or WavTokenizer Speech editing DAC (fine-grained control) Autoregressive LM on long audio WavTokenizer or X-Codec2 Codec quality research DAC (richest representation) ","permalink":"https://aadonis-ai.github.io/notebook/neural-audio-codecs/codec-comparison/","summary":"Side-by-side comparison of DAC, SNAC, WavTokenizer, and X-Codec2 — token rates, codebook counts, and when to use each.","title":"Codec Comparison \u0026 Overview"},{"content":"espeak-ng is a text-to-phoneme engine required by several audio tools (VoiceCraft, phonemizer, etc.). Installing it is straightforward with sudo, but on shared HPC clusters you often need a local build.\nCase 1: Standard Install (with sudo) sudo apt-get install espeak-ng Case 2: No-sudo Install (HPC / restricted Linux) Step 1 — Clone and build\ncd ~ git clone https://github.com/espeak-ng/espeak-ng.git cd espeak-ng ./autogen.sh ./configure --prefix=$HOME/.local make -j$(nproc) make install Step 2 — Add to PATH and library path\necho \u0026#39;export PATH=$HOME/.local/bin:$PATH\u0026#39; \u0026gt;\u0026gt; ~/.bashrc echo \u0026#39;export LD_LIBRARY_PATH=$HOME/.local/lib:$LD_LIBRARY_PATH\u0026#39; \u0026gt;\u0026gt; ~/.bashrc source ~/.bashrc Step 3 — Verify\nwhich espeak-ng # Expected: /home/\u0026lt;username\u0026gt;/.local/bin/espeak-ng espeak-ng \u0026#34;Hello world\u0026#34; Step 4 — Clean removal (optional)\nrm -rf ~/espeak-ng ~/.local/bin/espeak-ng ~/.local/lib/libespeak-ng* sed -i \u0026#39;/.local\\/bin/d\u0026#39; ~/.bashrc sed -i \u0026#39;/.local\\/lib/d\u0026#39; ~/.bashrc ","permalink":"https://aadonis-ai.github.io/notebook/speech-alignment/espeak-ng/","summary":"Installing espeak-ng with and without sudo — including a no-admin local build for HPC clusters.","title":"espeak-ng"},{"content":"When running large-scale training or inference on shared HPC clusters, your home directory quota fills up quickly from pip wheels, HuggingFace model checkpoints, and PyTorch caches. The fix is to redirect all caches to your scratch (net_scratch) partition before installing anything.\nRedirecting Caches to Scratch # Base path for your scratch project export SCRATCH_DIR=/path/to/net_scratch/my_project # 1. Temporary directory for builds and unpacking wheels export TMPDIR=$SCRATCH_DIR/tmp_build mkdir -p $TMPDIR # 2. PIP cache and build directories export PIP_CACHE_DIR=$SCRATCH_DIR/pip_cache mkdir -p $PIP_CACHE_DIR # 3. Hugging Face cache (for transformers, datasets, etc.) export HF_HOME=$SCRATCH_DIR/hf_home export TRANSFORMERS_CACHE=$HF_HOME/transformers export HF_DATASETS_CACHE=$HF_HOME/datasets mkdir -p $HF_HOME $TRANSFORMERS_CACHE $HF_DATASETS_CACHE # 4. PyTorch and other caches (optional) export TORCH_HOME=$SCRATCH_DIR/torch_cache mkdir -p $TORCH_HOME # 5. Now install packages without caching to $HOME pip install --no-cache-dir -r requirements.txt Add these exports to your job submission script (.sh / .sbatch) so they apply consistently across runs. Alternatively, put them in ~/.bashrc if you want them active in all interactive sessions.\n","permalink":"https://aadonis-ai.github.io/notebook/build-and-train/hpc-clusters/","summary":"Redirecting pip, HuggingFace, and PyTorch caches to scratch storage to avoid filling your home directory on shared clusters.","title":"HPC Clusters"},{"content":"LlaSE is a state-of-the-art speech enhancement model based on a speech language model architecture. It treats enhancement as a conditional generation problem: given discrete tokens of degraded speech, the model generates discrete tokens of clean speech, which are then decoded to a waveform.\nPaper: LlaSE Codebase: Github Repo\nSetup git clone https://github.com/Kevin-naticl/LLaSE.git cd LLaSE conda create -n LLaSE python=3.10 conda activate LLaSE pip install -r requirements.txt cd ckpt bash download.sh Running Inference The main script for inference is inference.py, which is configured via ./config/test.yml.\nStep 1 — Edit ./config/test.yml:\nAdjust chunk and overlap durations for your audio length Set wav_dir to the directory where output files will be saved: wav_dir: /path/to/LLASE_outputs Step 2 — Create filelist.txt in the LLaSE directory. Each line is an absolute path to a degraded audio file:\n/absolute/path/to/noisy_speech_1.wav /absolute/path/to/noisy_speech_2.wav /absolute/path/to/noisy_speech_3.wav Set its location in the config:\nfilename: filelist.txt Step 3 — Run inference:\nbash inference.sh The enhanced .wav files will be written to wav_dir.\n","permalink":"https://aadonis-ai.github.io/notebook/speech-enhancement/llase-g1/","summary":"Setup and inference for LlaSE — a language model-based speech enhancement system that converts degraded audio to high-quality speech.","title":"LlaSE-G1"},{"content":"Seed-VC is a zero-shot voice conversion framework using a diffusion transformer architecture. Key features:\nExternal timbre shifter during training perturbs source speech timbre, preventing leakage Diffusion transformer uses full reference speech context for fine-grained timbre capture Supports real-time voice conversion (~300ms algorithm delay) Supports singing voice conversion (V1) and accent/emotion conversion (V2) Paper: Zero-shot Voice Conversion with Diffusion Transformers Codebase: Github HuggingFace: Model Checkpoints\nAvailable Models Version Name Purpose SR Params v1.0 seed-uvit-tat-xlsr-tiny Voice Conversion 22050 25M v1.0 seed-uvit-whisper-small-wavenet Voice Conversion 22050 98M v1.0 seed-uvit-whisper-base Singing Voice Conversion 44100 200M v2.0 hubert-bsqvae-small Voice \u0026amp; Accent Conversion 22050 67M+90M Installation git clone https://github.com/Plachtaa/seed-vc.git cd seed-vc conda create -n seedvc python=3.10 conda activate seedvc pip install -r requirements.txt # Optional: ~6x speedup on V2 models (Windows) pip install triton-windows==3.2.0.post13 Command Line Inference (V1) python inference.py \\ --source \u0026lt;source-wav\u0026gt; \\ --target \u0026lt;reference-wav\u0026gt; \\ --output \u0026lt;output-dir\u0026gt; \\ --diffusion-steps 25 \\ --length-adjust 1.0 \\ --inference-cfg-rate 0.7 \\ --f0-condition False \\ --auto-f0-adjust False \\ --semi-tone-shift 0 \\ --fp16 True Parameter guide:\ndiffusion-steps: 25 default; 30–50 for best quality; 4–10 for fastest length-adjust: \u0026lt;1.0 speeds up, \u0026gt;1.0 slows down f0-condition: set True for singing voice conversion semi-tone-shift: pitch shift in semitones (SVC only) Command Line Inference (V2 — Accent/Emotion) python inference_v2.py \\ --source \u0026lt;source-wav\u0026gt; \\ --target \u0026lt;reference-wav\u0026gt; \\ --output \u0026lt;output-dir\u0026gt; \\ --diffusion-steps 25 \\ --intelligibility-cfg-rate 0.7 \\ --similarity-cfg-rate 0.7 \\ --convert-style true \\ --anonymization-only false \\ --top-p 0.9 \\ --temperature 1.0 Web UI Options python app_vc.py --fp16 True # V1 Voice Conversion python app_svc.py --fp16 True # Singing Voice Conversion python app_vc_v2.py --compile # V2 Model python app.py --enable-v1 --enable-v2 # Integrated Real-Time Voice Conversion python real-time-gui.py Recommended settings (RTX 3060 Laptop GPU):\nParameter Value Diffusion Steps 10 Inference CFG Rate 0.7 Max Prompt Length 3.0s Block Time 0.18s Crossfade Length 0.04s Extra Context (left) 2.5s Extra Context (right) 0.02s Latency ~430ms Use VB-CABLE to route GUI output to a virtual microphone.\nBatch Voice Conversion Script Converts utterances from an input directory using random reference voices — useful for corpus augmentation and speaker anonymization.\n\u0026#34;\u0026#34;\u0026#34; Batch Seed-VC augmentation script. Usage: python batch_seedvc_augment.py \\ --seedvc-root ../seed-vc \\ --ref-dir /path/to/reference/voices \\ --input-dir /path/to/input/utterances \\ --output-dir /path/to/output \\ --diffusion-steps 30 \\ --recursive --skip-existing \u0026#34;\u0026#34;\u0026#34; from __future__ import annotations import argparse, json, os, random, sys, time from pathlib import Path from typing import Iterable, Sequence def _iter_audio_files(root: Path, recursive: bool, exts: Sequence[str]) -\u0026gt; Iterable[Path]: exts_lc = {e.lower() if e.startswith(\u0026#34;.\u0026#34;) else f\u0026#34;.{e.lower()}\u0026#34; for e in exts} if root.is_file(): if root.suffix.lower() in exts_lc: yield root return if not root.exists(): return if recursive: for p in root.rglob(\u0026#34;*\u0026#34;): if p.is_file() and p.suffix.lower() in exts_lc: yield p else: for p in root.iterdir(): if p.is_file() and p.suffix.lower() in exts_lc: yield p def main() -\u0026gt; int: parser = argparse.ArgumentParser(description=\u0026#34;Batch Seed-VC corpus augmentation\u0026#34;) parser.add_argument(\u0026#34;--seedvc-root\u0026#34;, type=str, default=\u0026#34;../seed-vc\u0026#34;) parser.add_argument(\u0026#34;--ref-dir\u0026#34;, type=str, required=True) parser.add_argument(\u0026#34;--input-dir\u0026#34;, type=str, required=True) parser.add_argument(\u0026#34;--output-dir\u0026#34;, type=str, required=True) parser.add_argument(\u0026#34;--device\u0026#34;, type=str, default=\u0026#34;\u0026#34;) parser.add_argument(\u0026#34;--recursive\u0026#34;, action=\u0026#34;store_true\u0026#34;) parser.add_argument(\u0026#34;--ext\u0026#34;, type=str, default=\u0026#34;wav\u0026#34;) parser.add_argument(\u0026#34;--seed\u0026#34;, type=int, default=0) parser.add_argument(\u0026#34;--limit\u0026#34;, type=int, default=0) parser.add_argument(\u0026#34;--skip-existing\u0026#34;, action=\u0026#34;store_true\u0026#34;) parser.add_argument(\u0026#34;--diffusion-steps\u0026#34;, type=int, default=30) parser.add_argument(\u0026#34;--length-adjust\u0026#34;, type=float, default=1.0) parser.add_argument(\u0026#34;--inference-cfg-rate\u0026#34;, type=float, default=0.7) parser.add_argument(\u0026#34;--f0-condition\u0026#34;, action=\u0026#34;store_true\u0026#34;) parser.add_argument(\u0026#34;--suffix\u0026#34;, type=str, default=\u0026#34;_seedvc\u0026#34;) parser.add_argument(\u0026#34;--manifest\u0026#34;, type=str, default=\u0026#34;manifest_seedvc.jsonl\u0026#34;) args = parser.parse_args() seedvc_root = Path(args.seedvc_root).resolve() sys.path.insert(0, str(seedvc_root)) os.environ.setdefault(\u0026#34;HF_HUB_CACHE\u0026#34;, str(seedvc_root / \u0026#34;checkpoints\u0026#34; / \u0026#34;hf_cache\u0026#34;)) from seed_vc_wrapper import SeedVCWrapper import soundfile as sf ref_files = sorted(_iter_audio_files(Path(args.ref_dir), args.recursive, args.ext.split(\u0026#34;,\u0026#34;))) input_files = sorted(_iter_audio_files(Path(args.input_dir), args.recursive, args.ext.split(\u0026#34;,\u0026#34;))) if args.limit \u0026gt; 0: input_files = input_files[:args.limit] random.seed(args.seed) output_dir = Path(args.output_dir) output_dir.mkdir(parents=True, exist_ok=True) device = None if args.device: import torch device = torch.device(args.device) wrapper = SeedVCWrapper(device=device, load_f0_model=bool(args.f0_condition)) with (output_dir / args.manifest).open(\u0026#34;a\u0026#34;, encoding=\u0026#34;utf-8\u0026#34;) as mf: for idx, inp in enumerate(input_files, start=1): out_path = output_dir / f\u0026#34;{inp.stem}{args.suffix}.wav\u0026#34; if args.skip_existing and out_path.exists(): continue ref = random.choice(ref_files) try: sr, audio = wrapper.convert_voice_npy( source=str(inp), target=str(ref), diffusion_steps=args.diffusion_steps, length_adjust=args.length_adjust, inference_cfg_rate=args.inference_cfg_rate, f0_condition=bool(args.f0_condition), ) sf.write(str(out_path), audio, sr) rec = {\u0026#34;source\u0026#34;: str(inp), \u0026#34;reference\u0026#34;: str(ref), \u0026#34;output\u0026#34;: str(out_path)} mf.write(json.dumps(rec) + \u0026#34;\\n\u0026#34;) mf.flush() if idx % 10 == 0: print(f\u0026#34;[{idx}/{len(input_files)}] Processed\u0026#34;) except Exception as e: print(f\u0026#34;FAILED {inp.name}: {e}\u0026#34;) return 0 if __name__ == \u0026#34;__main__\u0026#34;: raise SystemExit(main()) ","permalink":"https://aadonis-ai.github.io/notebook/voice-conversion/seed-vc/","summary":"Zero-shot voice conversion with Seed-VC — diffusion transformer architecture, real-time inference, batch processing, and a comparison with EZ-VC.","title":"Seed-VC"},{"content":"VoiceCraft edits speech by masking a region of the audio (at the token level), then autoregressively infilling the masked region conditioned on the surrounding context and a new target transcript. It requires forced alignment (MFA) to determine where in the audio the edit should happen.\nCodebase: GitHub\n1. Setup and Dependencies First install espeak-ng (required by the text tokenizer).\nmkdir -p voicecraft_dir cd voicecraft_dir git clone https://github.com/jasonppy/VoiceCraft.git conda create -n voicecraft python==3.9.16 CONDA_ENVIRONMENT=/path/to/conda_envs/voicecraft conda activate ${CONDA_ENVIRONMENT} export TMPDIR=/path/to/voicecraft_dir/tmp export PIP_CACHE_DIR=/path/to/voicecraft_dir/pip_cache export HF_HOME=path/to/voicecraft_dir/huggingface export TRANSFORMERS_CACHE=$HF_HOME/transformers export TORCH_HOME=/path/to/voicecraft_dir/torch mkdir -p $TMPDIR $PIP_CACHE_DIR $HF_HOME $TRANSFORMERS_CACHE $TORCH_HOME # Conda dependencies (MFA + Kaldi) conda install -y -c conda-forge montreal-forced-aligner=2.2.17 openfst=1.8.2 kaldi=5.5.1068 joblib=1.2.0 # Download MFA models/dictionary mfa model download dictionary english_us_arpa mfa model download acoustic english_us_arpa # Pip dependencies pip install -e git+https://github.com/facebookresearch/audiocraft.git@c5157b5bf14bf83449c17ea1eeb66c19fb4bc7f0#egg=audiocraft \\ --no-cache-dir --cache-dir $PIP_CACHE_DIR pip install xformers==0.0.22 torchaudio==2.0.2 torch==2.0.1 tensorboard==2.16.2 \\ phonemizer==3.2.1 datasets==2.16.0 torchmetrics==0.11.1 \\ huggingface_hub==0.22.2 py-espeak-ng soundfile pyflac pyvorbis lxml \\ gradio==3.50.2 nltk\u0026gt;=3.8.1 openai-whisper\u0026gt;=20231117 num2words==0.5.13 \\ --no-cache-dir --cache-dir $PIP_CACHE_DIR || echo \u0026#34;Some optional packages skipped.\u0026#34; 2. Single-File Editing Script Store this as single_voicecraft_edit.py inside the cloned VoiceCraft repo.\n\u0026#34;\u0026#34;\u0026#34; Single Audio Speech Editing with VoiceCraft Edit audio by specifying audio_path, original/target transcripts and edit type. \u0026#34;\u0026#34;\u0026#34; import argparse, logging, os, random, pickle import numpy as np import torch, torchaudio from data.tokenizer import AudioTokenizer, TextTokenizer, tokenize_text, tokenize_audio from models import voicecraft logging.basicConfig(format=\u0026#34;%(asctime)s [%(levelname)s] %(message)s\u0026#34;, level=logging.INFO) def seed_everything(seed): os.environ[\u0026#39;PYTHONHASHSEED\u0026#39;] = str(seed) random.seed(seed) np.random.seed(seed) torch.manual_seed(seed) torch.cuda.manual_seed(seed) torch.backends.cudnn.benchmark = False torch.backends.cudnn.deterministic = True def get_word_spans(orig, target, edit_type): \u0026#34;\u0026#34;\u0026#34;Find word indices that differ between transcripts\u0026#34;\u0026#34;\u0026#34; orig_words, target_words = orig.split(), target.split() if edit_type == \u0026#34;deletion\u0026#34;: diff = len(orig_words) - len(target_words) for i, (o, t) in enumerate(zip(orig_words, target_words)): if o != t: return (i, i + diff - 1) return (len(target_words), len(orig_words) - 1) elif edit_type == \u0026#34;insertion\u0026#34;: diff = len(target_words) - len(orig_words) for i, (o, t) in enumerate(zip(orig_words, target_words)): if o != t: return (max(0, i-1), i) return (len(orig_words) - 1, len(orig_words)) else: # substitution start = next(i for i, (o, t) in enumerate(zip(orig_words, target_words)) if o != t) end = next(i for i in range(len(orig_words)-1, -1, -1) if i \u0026lt; len(target_words) and orig_words[i] != target_words[i]) return (start, end) def get_mask_interval(alignment_csv, word_span, edit_type, left_margin=0.08, right_margin=0.08): with open(alignment_csv) as f: words = [l.strip().split(\u0026#34;,\u0026#34;) for l in f.readlines()[1:] if \u0026#34;words\u0026#34; in l] start_idx, end_idx = word_span start_time = float(words[end_idx][1] if edit_type == \u0026#39;insertion\u0026#39; else words[start_idx][0]) end_time = float(words[end_idx][1]) return (max(start_time - left_margin, 0), end_time + right_margin) def run_mfa(audio_path, transcript, temp_dir, beam=100, retry_beam=400): os.makedirs(temp_dir, exist_ok=True) filename = os.path.splitext(os.path.basename(audio_path))[0] import shutil shutil.copy(audio_path, os.path.join(temp_dir, f\u0026#34;{filename}.wav\u0026#34;)) with open(os.path.join(temp_dir, f\u0026#34;{filename}.txt\u0026#34;), \u0026#34;w\u0026#34;) as f: f.write(transcript) align_out = os.path.join(temp_dir, \u0026#34;mfa_alignments\u0026#34;) csv_path = os.path.join(align_out, f\u0026#34;{filename}.csv\u0026#34;) if not os.path.isfile(csv_path): cmd = f\u0026#34;mfa align -v --clean -j 1 --output_format csv {temp_dir} english_us_arpa english_us_arpa {align_out} --beam {beam} --retry_beam {retry_beam}\u0026#34; os.system(cmd) return csv_path def main(): parser = argparse.ArgumentParser(description=\u0026#34;Edit single audio with VoiceCraft\u0026#34;) parser.add_argument(\u0026#34;--audio_path\u0026#34;, type=str, default=\u0026#34;../original_audio.wav\u0026#34;) parser.add_argument(\u0026#34;--orig_transcript\u0026#34;, type=str, required=True) parser.add_argument(\u0026#34;--target_transcript\u0026#34;, type=str, required=True) parser.add_argument(\u0026#34;--edit_type\u0026#34;, type=str, default=\u0026#34;substitution\u0026#34;, choices=[\u0026#34;insertion\u0026#34;, \u0026#34;deletion\u0026#34;, \u0026#34;substitution\u0026#34;]) parser.add_argument(\u0026#34;--model_name\u0026#34;, type=str, default=\u0026#34;giga330M\u0026#34;, choices=[\u0026#34;giga330M\u0026#34;, \u0026#34;giga830M\u0026#34;]) parser.add_argument(\u0026#34;--exp_dir\u0026#34;, type=str, default=None) parser.add_argument(\u0026#34;--output_dir\u0026#34;, type=str, default=\u0026#34;./outputs\u0026#34;) parser.add_argument(\u0026#34;--temp_dir\u0026#34;, type=str, default=\u0026#34;./temp\u0026#34;) parser.add_argument(\u0026#34;--codec_audio_sr\u0026#34;, type=int, default=16000) parser.add_argument(\u0026#34;--codec_sr\u0026#34;, type=int, default=50) parser.add_argument(\u0026#34;--top_k\u0026#34;, type=int, default=0) parser.add_argument(\u0026#34;--top_p\u0026#34;, type=float, default=0.8) parser.add_argument(\u0026#34;--temperature\u0026#34;, type=float, default=1.0) parser.add_argument(\u0026#34;--stop_repetition\u0026#34;, type=int, default=2) parser.add_argument(\u0026#34;--kvcache\u0026#34;, type=int, default=1) parser.add_argument(\u0026#34;--silence_tokens\u0026#34;, type=str, default=\u0026#34;[1388,1898,131]\u0026#34;) parser.add_argument(\u0026#34;--left_margin\u0026#34;, type=float, default=0.08) parser.add_argument(\u0026#34;--right_margin\u0026#34;, type=float, default=0.08) parser.add_argument(\u0026#34;--seed\u0026#34;, type=int, default=1) parser.add_argument(\u0026#34;--beam_size\u0026#34;, type=int, default=100) parser.add_argument(\u0026#34;--retry_beam_size\u0026#34;, type=int, default=400) parser.add_argument(\u0026#34;--device\u0026#34;, type=str, default=\u0026#34;cuda\u0026#34; if torch.cuda.is_available() else \u0026#34;cpu\u0026#34;) args = parser.parse_args() seed_everything(args.seed) os.makedirs(args.output_dir, exist_ok=True) logging.info(f\u0026#34;Loading model on {args.device}...\u0026#34;) if args.exp_dir: with open(os.path.join(args.exp_dir, \u0026#34;args.pkl\u0026#34;), \u0026#34;rb\u0026#34;) as f: model_args = pickle.load(f) model = voicecraft.VoiceCraft(model_args) ckpt = torch.load(os.path.join(args.exp_dir, \u0026#34;best_bundle.pth\u0026#34;), map_location=\u0026#39;cpu\u0026#39;) phn2num = ckpt[\u0026#39;phn2num\u0026#39;] model.load_state_dict(ckpt[\u0026#39;model\u0026#39;]) else: model = voicecraft.VoiceCraft.from_pretrained(f\u0026#34;pyp1/VoiceCraft_{args.model_name}\u0026#34;) model_args, phn2num = model.args, model_args.phn2num model.to(args.device).eval() encodec_fn = \u0026#34;./pretrained_models/encodec_4cb2048_giga.th\u0026#34; if not os.path.exists(encodec_fn): os.makedirs(\u0026#34;./pretrained_models\u0026#34;, exist_ok=True) os.system(f\u0026#34;wget https://huggingface.co/pyp1/VoiceCraft/resolve/main/encodec_4cb2048_giga.th -O {encodec_fn}\u0026#34;) audio_tokenizer = AudioTokenizer(signature=encodec_fn, device=args.device) text_tokenizer = TextTokenizer(backend=\u0026#34;espeak\u0026#34;) info = torchaudio.info(args.audio_path) audio_dur = info.num_frames / info.sample_rate logging.info(\u0026#34;Running MFA alignment...\u0026#34;) csv = run_mfa(args.audio_path, args.orig_transcript, args.temp_dir, args.beam_size, args.retry_beam_size) if not os.path.exists(csv): logging.error(\u0026#34;Alignment failed!\u0026#34;) return word_span = get_word_spans(args.orig_transcript, args.target_transcript, args.edit_type) mask_interval = get_mask_interval(csv, word_span, args.edit_type, args.left_margin, args.right_margin) mask_interval = (max(mask_interval[0], 1/args.codec_sr), min(mask_interval[1], audio_dur)) mask_frames = torch.LongTensor([[round(mask_interval[0]*args.codec_sr), round(mask_interval[1]*args.codec_sr)]]).unsqueeze(0) logging.info(f\u0026#34;Mask interval: {mask_interval[0]:.3f}s - {mask_interval[1]:.3f}s\u0026#34;) text_tokens = torch.LongTensor([phn2num[p] for p in tokenize_text( text_tokenizer, args.target_transcript.strip()) if p in phn2num]).unsqueeze(0) text_lens = torch.LongTensor([text_tokens.shape[-1]]) orig_audio = tokenize_audio(audio_tokenizer, args.audio_path)[0][0].transpose(2, 1) logging.info(\u0026#34;Running inference...\u0026#34;) silence_toks = eval(args.silence_tokens) if isinstance(args.silence_tokens, str) else args.silence_tokens with torch.no_grad(): gen_audio = model.inference( text_tokens.to(args.device), text_lens.to(args.device), orig_audio[..., :model_args.n_codebooks].to(args.device), mask_interval=mask_frames.to(args.device), top_k=args.top_k, top_p=args.top_p, temperature=args.temperature, stop_repetition=args.stop_repetition, kvcache=args.kvcache, silence_tokens=silence_toks ) gen_sample = audio_tokenizer.decode([(gen_audio, None)])[0].cpu() orig_sample = audio_tokenizer.decode([(orig_audio.transpose(2, 1), None)])[0].cpu() base = os.path.splitext(os.path.basename(args.audio_path))[0] edited_path = os.path.join(args.output_dir, f\u0026#34;{base}_edited_seed{args.seed}.wav\u0026#34;) recon_path = os.path.join(args.output_dir, f\u0026#34;{base}_reconstructed.wav\u0026#34;) torchaudio.save(edited_path, gen_sample, args.codec_audio_sr) torchaudio.save(recon_path, orig_sample, args.codec_audio_sr) print(f\u0026#34;\\n{\u0026#39;=\u0026#39;*60}\\nEDIT SUMMARY\\n{\u0026#39;=\u0026#39;*60}\u0026#34;) print(f\u0026#34;Original: {args.orig_transcript}\\nTarget: {args.target_transcript}\u0026#34;) print(f\u0026#34;Edit type: {args.edit_type} | Word span: {word_span}\u0026#34;) print(f\u0026#34;Time mask: {mask_interval[0]:.3f}s - {mask_interval[1]:.3f}s\u0026#34;) print(f\u0026#34;Edited: {edited_path}\u0026#34;) print(f\u0026#34;Reconstructed: {recon_path}\u0026#34;) if __name__ == \u0026#34;__main__\u0026#34;: main() Usage:\npython single_voicecraft_edit.py \\ --audio_path ../original_audio.wav \\ --orig_transcript \u0026#34;what struck into me the deepest was the look of nearly everyone of the judges\u0026#34; \\ --target_transcript \u0026#34;what struck into me the deepest was the look of nearly all people present\u0026#34; \\ --edit_type substitution \\ --model_name giga330M \\ --seed 1 ","permalink":"https://aadonis-ai.github.io/notebook/speech-editing/voicecraft/","summary":"Token-based speech editing with VoiceCraft — setup, MFA forced alignment, and a self-contained editing script for insertions, deletions, and substitutions.","title":"VoiceCraft"},{"content":" Adonis Asonitis aasonitis@ethz.ch Experience Rime Member of Technical Staff Aug 2026 – Present Working on full-duplex systems, neural codecs, and expressive TTS models.\nETH Zurich — DISCO Lab Student Researcher, Distributed Computing Zurich · Feb 2025 – Present Research student under Prof. Roger Wattenhofer. Speech generation, multilingual audio data, codec language models, and reinforcement learning. First-authored WorldSpeech, the largest publicly available human-transcribed multilingual speech corpus (65k hours across 80+ languages, 150k+ HuggingFace downloads; #1 trending in its first month). Additional work on multilingual speech editing, zero-shot voice conversion, speech enhancement language models, and RL post-training for speech enhancement.\nAGIGO AI Research Engineer (Internship) — Conversational AI, speech processing and post-training Zurich · Jan 2026 – Jul 2026 Designed post-training methods giving fine-grained control over zero-shot pronunciation and cross-lingual phonetic nuances. Built end-to-end data pipelines and evaluation infrastructure, including internal blind listening tests, automated objective eval suites, and dataset curation/QA workflows used across TTS model training. Education ETH Zurich — MSc Computer Science 2025 – Present · Zurich Machine Intelligence · minor in Data Management Systems.\nETH Zurich — BSc Computer Science 2022 – 2025 · Zurich Statistical modeling, ML, computer systems, applied mathematics. Top-graded thesis (6/6), developed into a top-tier ML conference submission.\nCompetitions \u0026 Projects Swiss AI Datathon 2024 1st place (200+ participants). Ensemble models for photovoltaic energy forecasting (one-day-ahead market, \u003e90% accuracy). Greek Urban Planning RAG RAG system for lawyers and engineers navigating Greek urban planning legislation. Gen-AI for DolliBar Generative AI pipeline converting receipt images into structured enterprise expense entries. Arbitrage System High-performance cross-platform arbitrage detection and execution engine in C++. Athletics Swiss Olympic Sailing Team National Team Athlete 2016 – 2020 ILCA · 220k+ boats worldwide 1st European Cup standings (2018) 4× Swiss national medals 8th U17 European Championship (2019) 16th U16 World Championship (2017) Languages English · Fluent\u0026emsp; French · Fluent\u0026emsp; Greek · Fluent\u0026emsp; German · Advanced ","permalink":"https://aadonis-ai.github.io/cv/","summary":"Curriculum Vitae","title":"Curriculum Vitae"},{"content":"","permalink":"https://aadonis-ai.github.io/publications/","summary":"Research publications and datasets","title":"Publications"}]