_{CONTRIBUTES TO  ·  FRONTIER LLM TRAINING & INFERENCE}

         

Research assistant in Prof. Anna Choromanska's lab at NYU (sparse neural routing · self-supervised world models · LiDAR). Alongside the research, I contribute upstream to the LLM training & inference stack — vLLM, NVIDIA NeMo / Megatron, DeepSpeed — chasing correctness bugs in quantized KV caches, weight-tying, checkpointing, and build pipelines.

Open to research internships, PhD positions, and ML-systems / research-engineering roles in efficient training, sparse MoE, and world models.

_{13+ PRs merged into frontier LLM training & inference.}

vllm-project/vllm #49226 Disable cross-layer KV blocks for per-token-head quant — fixes OffloadingConnector KV-cache corruption
NVIDIA/Megatron-LM #5743 Support HSDP deferred DP-outer gradient reduction in experimental Megatron-FSDP
NVIDIA-NeMo/Automodel #2998 Single tie_word_embeddings guard via per-class TieSupport (BOTH / TIED_ONLY / UNTIED_ONLY) + from_pretrained flip check
NVIDIA-NeMo/Automodel #2896 Complete tie_word_embeddings guards for remaining #2512 model families
vllm-project/vllm #47379 Recover raw tail when GPT-OSS Harmony parser ends non-terminal (Responses API)
NVIDIA-NeMo/Megatron-Bridge #4601 Make finetuning batch sampler epoch-aware on checkpoint resume
vllm-project/vllm #47062 Return raw output when GPT-OSS Harmony parser ends in a non-terminal state
NVIDIA-NeMo/Automodel #2805 Reject tie_word_embeddings=True on separate-head model families
deepspeedai/DeepSpeed #8078 Avoid CUDA context initialization during import-time op compatibility checks (fork-safe import)
NVIDIA-NeMo/Automodel #2732 Resolve tie_word_embeddings top-level-first to match HF tying semantics
vllm-project/vllm #44795 Fix nightly Docker ImportError: AnthropicOutputConfig
NVIDIA-NeMo/Automodel #2601 Re-tie lm_head to active embed_tokens on Gemma4 MoE path
NVIDIA-NeMo/Automodel #2709 Cherry-pick #2601 into r0.5.0

Impact

• Inference correctness — fixed KV-cache corruption under per-token-head quant and recovered truncated GPT-OSS output (vLLM).
• Model compatibility — corrected tie_word_embeddings handling across six NeMo Automodel PRs so tied/untied embeddings load correctly.
• Training reliability — epoch-aware checkpoint resume (Megatron-Bridge), fork-safe import (DeepSpeed), HSDP gradient reduction (Megatron-FSDP), and a fixed vLLM nightly build.

Circuit Synchronization Precedes Generalization: A Causal Precursor to Grokking
_{Introduces the Frequency Synchronization Degree (FSD) — a permutation-tested metric that detects Fourier-circuit formation 500–3,000 steps before grokking, with causal weight-decay evidence that the memorization→generalization gap is a regularization phenomenon. Transfers to the non-abelian group S₅. Sole author.
📄 arXiv:2606.12966}

Adaptive Compute in Latent World Models: When Depth Helps, Hurts, or Doesn't Matter
_{Pre-registered study of adaptive-depth latent world models across nine DeepMind Control tasks. Maps when extra depth helps rollouts (ρ up to 4.7×), when shallow beats deep (2/9 tasks), and the routability catch-22 created by early-exit supervision.
📄 arXiv:2607.10203}

Research

• We found the moment grokking actually begins — the FSD preprint in plain English: circuits form 500–3,000 steps before the accuracy jump.
• When depth helps, hurts, or doesn't matter — a taxonomy of adaptive compute in latent world models.

From the trenches

• When resuming training made my model learn the wrong data — the epoch-aware sampler bug behind Megatron-Bridge #4601.
• When Docker cache lies — the BuildKit quirk behind a vLLM nightly build failure.

_{More at blog-blogachyuthan.vercel.app}

Building

• AD-LiST-JEPA — spatiotemporal JEPA world model for autonomous driving; predicts future BEV LiDAR embeddings without labels or contrastive pairs
• KAN-Multi — routing layer that selects among 6 function bases with zero supervision; +6.8% over MLP on CIFAR-100
• MoE-Bench — open diagnostic toolkit for expert collapse & routing entropy in sparse MoE LLMs (OLMoE, JetMoE, Qwen)

Focus  ·  Self-supervised learning · Sparse MoE architectures · Neural routing · World models · LiDAR perception

Stack  ·  Python · PyTorch · C/C++ · Go · HuggingFace · Docker · FastAPI · AWS

→ achyuthan-s.web.app

📫 as21154@nyu.edu  ·  achyuthan.sivasankar@gmail.com  ·  LinkedIn  ·  Portfolio  ·  Blog