Sameer Singh

Appointments

Univ of California
Irvine CA

Professor
2024 - current

Univ of California
Irvine CA

Associate Professor
2021 - 2024

Univ of California
Irvine CA

Assistant Professor
2016 - 2021

Univ of Washington
Seattle WA

Postdoctoral Researcher
2013 - 2016

Industry

Spiffy AI
Seattle WA
Cofounder/CTO
2023 - current

Allen Institute for AI
Seattle WA
Allen Fellow
2021-2023

Microsoft Research
Cambridge UK
Research Intern
Summer 2012

Google Research
Mountain View CA
Research Intern
Summer 2010

Yahoo! Labs
Sunnyvale CA
Research Intern
Summer 2009

Google
Pittsburgh PA
Research Intern
Summer, Fall 2007

Education

PhD (CS)
Univ of Massachusetts

Amherst MA
2014

MS (CS)
Vanderbilt University

Nashville TN
2007

BEng (EE)
University of Delhi

New Delhi
2004

High School
Sardar Patel Vidyalaya

New Delhi
2000

Selected Recent Publications see all...

• Felix Draxler, Justus Will, Farrin Marouf Sofian, Theofanis Karaletsos, Sameer Singh, Stephan Mandt.Parallel Token Prediction for Language Models. International Conference on Learning Representations (ICLR). 2026 Conference
  PDFArXiVOpenReviewCodeAbstractBibTex
  Autoregressive decoding in language models is inherently slow, generating only one token per forward pass. We propose Parallel Token Prediction (PTP), a general-purpose framework for predicting multiple tokens in a single model call. PTP moves the source of randomness from post-hoc sampling to random input variables, making future tokens deterministic functions of those inputs and thus jointly predictable in a single forward pass. We prove that a single PTP call can represent arbitrary dependencies between tokens. PTP is trained by distilling an existing model or through inverse autoregressive training without a teacher. Experimentally, PTP achieves a 2.4x speedup on a diverse-task speculative decoding benchmark. We provide code and checkpoints at https://github.com/mandt-lab/ptp.

  @inproceedings{ptp:iclr26,
    author = {Felix Draxler and Justus Will and Farrin Marouf Sofian and Theofanis Karaletsos and Sameer Singh and Stephan Mandt},
    title = { {Parallel Token Prediction for Language Models} },
    booktitle = {International Conference on Learning Representations (ICLR)},
    year = {2026}
  }

• Yu Fei, Yasaman Razeghi, Sameer Singh.Nudging: Inference-time Alignment of LLMs via Guided Decoding. Association for Computational Linguistics (ACL). 2025 Conference
  PDFACL AnthologyAbstractBibTex
  Large language models (LLMs) require alignment to effectively and safely follow user instructions. This process necessitates training an aligned version for every base model, resulting in significant computational overhead. In this work, we propose NUDGING, a simple, training-free algorithm that aligns any base model at inference time using a small aligned model. NUDGING is motivated by recent findings that alignment primarily alters the model's behavior on a small subset of stylistic tokens (e.g., discourse markers). We find that base models are significantly more uncertain when generating these tokens. Building on this insight, NUDGING employs a small aligned model to generate nudging tokens to guide the base model's output during decoding when the base model's uncertainty is high, with only a minor additional inference overhead. We evaluate NUDGING across 3 model families on a diverse range of open-instruction tasks. Without any training, nudging a large base model with a 7x-14x smaller aligned model achieves zero-shot performance comparable to, and sometimes surpassing, that of large aligned models. By operating at the token level, NUDGING enables off-the-shelf collaboration between model families. For instance, nudging Gemma-2-27b with Llama-27b-chat outperforms Llama-2-70b-chat on various tasks. Overall, our work offers a modular and cost-efficient solution to LLM alignment. Our code and demo are available at: https://fywalter.github.io/nudging/.

  @inproceedings{nudging:acl25,
    author = {Yu Fei and Yasaman Razeghi and Sameer Singh},
    title = { {Nudging: Inference-time Alignment of LLMs via Guided Decoding} },
    booktitle = {Association for Computational Linguistics (ACL)},
    doi = {10.18653/v1/2025.acl-long.623},
    pages = {12702-12739},
    year = {2025}
  }

• Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, Sameer Singh.Calibrate Before Use: Improving Few-shot Performance of Language Models. International Conference on Machine Learning (ICML). 2021 Conference
  PDFArXiVICML PageVideo/SlidesBibTex

  @inproceedings{poisoning:icml21,
    author = {Tony Z. Zhao and Eric Wallace and Shi Feng and Dan Klein and Sameer Singh},
    title = { {Calibrate Before Use: Improving Few-shot Performance of Language Models} },
    booktitle = {International Conference on Machine Learning (ICML)},
    pages = {12697-12706},
    year = {2021}
  }

• Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, Sameer Singh.AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts . Empirical Methods in Natural Language Processing (EMNLP). 2020 Conference
  PDFWebsiteACL AnthologyAbstractBibTex
  The remarkable success of pretrained language models has motivated the study of what kinds of knowledge these models learn during pretraining. Reformulating tasks as fill-in-the-blanks problems (e.g., cloze tests) is a natural approach for gauging such knowledge, however, its usage is limited by the manual effort and guesswork required to write suitable prompts. To address this, we develop AutoPrompt, an automated method to create prompts for a diverse set of tasks, based on a gradient-guided search. Using AutoPrompt, we show that masked language models (MLMs) have an inherent capability to perform sentiment analysis and natural language inference without additional parameters or finetuning, sometimes achieving performance on par with recent state-of-the-art supervised models. We also show that our prompts elicit more accurate factual knowledge from MLMs than the manually created prompts on the LAMA benchmark, and that MLMs can be used as relation extractors more effectively than supervised relation extraction models. These results demonstrate that automatically generated prompts are a viable parameter-free alternative to existing probing methods, and as pretrained LMs become more sophisticated and capable, potentially a replacement for finetuning.

  @inproceedings{autoprompt:emnlp20,
    author = {Taylor Shin and Yasaman Razeghi and Robert L. Logan IV and Eric Wallace and Sameer Singh},
    title = { {AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts } },
    booktitle = {Empirical Methods in Natural Language Processing (EMNLP)},
    pages = {4222–4235},
    year = {2020}
  }

• Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, Sameer Singh.Beyond Accuracy: Behavioral Testing of NLP models with CheckList. Association for Computational Linguistics (ACL). 2020 Conference
  Best Paper Award
  PDFCodeACL AnthologyVideo+SlidesArXiVAbstractBibTex
  Although measuring held-out accuracy has been the primary approach to evaluate generalization, it often overestimates the performance of NLP models, while alternative approaches for evaluating models either focus on individual tasks or on specific behaviors. Inspired by principles of behavioral testing in software engineering, we introduce CheckList, a task-agnostic methodology for testing NLP models. CheckList includes a matrix of general linguistic capabilities and test types that facilitate comprehensive test ideation, as well as a software tool to generate a large and diverse number of test cases quickly. We illustrate the utility of CheckList with tests for three tasks, identifying critical failures in both commercial and state-of-art models. In a user study, a team responsible for a commercial sentiment analysis model found new and actionable bugs in an extensively tested model. In another user study, NLP practitioners with CheckList created twice as many tests, and found almost three times as many bugs as users without it.

  @inproceedings{checklist:acl20,
    author = {Marco Tulio Ribeiro and Tongshuang Wu and Carlos Guestrin and Sameer Singh},
    title = { {Beyond Accuracy: Behavioral Testing of NLP models with CheckList} },
    booktitle = {Association for Computational Linguistics (ACL)},
    pages = {4902-4912},
    year = {2020}
  }

• Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, Sameer Singh.Universal Adversarial Triggers for Attacking and Analyzing NLP. Empirical Methods in Natural Language Processing (EMNLP). 2019 Conference
  PDFarXivBlog postCodeACL AnthologyAbstractBibTex
  Adversarial examples highlight model vulnerabilities and are useful for evaluation and interpretation. We define universal adversarial triggers: input-agnostic sequences of tokens that trigger a model to produce a specific prediction when concatenated to any input from a dataset. We propose a gradient-guided search over tokens which finds short trigger sequences (e.g., one word for classification and four words for language modeling) that successfully trigger the target prediction. For example, triggers cause SNLI entailment accuracy to drop from 89.94% to 0.55%, 72% of “why” questions in SQuAD to be answered “to kill american people”, and the GPT-2 language model to spew racist output even when conditioned on non-racial contexts. Furthermore, although the triggers are optimized using white-box access to a specific model, they transfer to other models for all tasks we consider. Finally, since triggers are input-agnostic, they provide an analysis of global model behavior. For instance, they confirm that SNLI models exploit dataset biases and help to diagnose heuristics learned by reading comprehension models.

  @inproceedings{trigger:emnlp19,
    author = {Eric Wallace and Shi Feng and Nikhil Kandpal and Matt Gardner and Sameer Singh},
    title = { {Universal Adversarial Triggers for Attacking and Analyzing NLP} },
    booktitle = {Empirical Methods in Natural Language Processing (EMNLP)},
    doi = {10.18653/v1/D19-1221},
    pages = {2153-2162},
    year = {2019}
  }

• Eric Wallace, Jens Tuyls, Junlin Wang, Sanjay Subramanian, Matt Gardner, Sameer Singh.AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models. Demo at the Empirical Methods in Natural Language Processing (EMNLP). 2019 Demo
  Best Demonstration Paper Award.
  PDFProject PageACL AnthologyArXivPosterAbstractBibTex
  Neural NLP models are increasingly accurate but are imperfect and opaque---they break in counterintuitive ways and leave end users puzzled at their behavior. Model interpretation methods ameliorate this opacity by providing explanations for specific model predictions. Unfortunately, existing interpretation codebases make it difficult to apply these methods to new models and tasks, which hinders adoption for practitioners and burdens interpretability researchers. We introduce AllenNLP Interpret, a flexible framework for interpreting NLP models. The toolkit provides interpretation primitives (e.g., input gradients) for any AllenNLP model and task, a suite of built-in interpretation methods, and a library of front-end visualization components. We demonstrate the toolkit's flexibility and utility by implementing live demos for five interpretation methods (e.g., saliency maps and adversarial attacks) on a variety of models and tasks (e.g., masked language modeling using BERT and reading comprehension using BiDAF). These demos, alongside our code and tutorials, are available at https://allennlp.org/interpret.

  @inproceedings{interpret:emnlp19,
    author = {Eric Wallace and Jens Tuyls and Junlin Wang and Sanjay Subramanian and Matt Gardner and Sameer Singh},
    title = { {AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models} },
    booktitle = {Demo at the Empirical Methods in Natural Language Processing (EMNLP)},
    doi = {10.18653/v1/D19-3002},
    pages = {7-12},
    year = {2019}
  }

• Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, Matt Gardner.DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs. Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). 2019 Conference
  PDFWebsitearXivDataACL AnthologyLeaderboardDemoAbstractBibTex
  Reading comprehension has recently seen rapid progress, with systems matching humans on the most popular datasets for the task. However, a large body of work has highlighted the brittleness of these systems, showing that there is much work left to be done. We introduce a new reading comprehension benchmark, DROP, which requires Discrete Reasoning Over the content of Paragraphs. In this crowdsourced, adversarially-created, 55k-question benchmark, a system must resolve references in a question, perhaps to multiple input positions, and perform discrete operations over them (such as addition, counting, or sorting). These operations require a much more comprehensive understanding of the content of paragraphs, as they remove the paraphrase-and-entity-typing shortcuts available in prior datasets. We apply state-of-the-art methods from both the reading comprehension and semantic parsing literatures on this dataset and show that the best systems only achieve 38.4% F1 on our generalized accuracy metric, while expert human performance is 96%. We additionally present a new model that combines reading comprehension methods with simple numerical reasoning to achieve 51% F1.

  @inproceedings{drop:naacl19,
    author = {Dheeru Dua and Yizhong Wang and Pradeep Dasigi and Gabriel Stanovsky and Sameer Singh and Matt Gardner},
    title = { {DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs} },
    booktitle = {Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)},
    doi = {10.18653/v1/N19-1246},
    pages = {2368-2378},
    year = {2019}
  }

• Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin."Why Should I Trust You?": Explaining the Predictions of Any Classifier. Knowledge Discovery and Data Mining (KDD). 2016 Conference
  Audience Appreciation Award
  Also presented at the CHI 2016 Workshop on Human-Centred Machine Learning (HCML).
  PDFarXivCodeVideoO'ReillyCode (experiments)ACM PageBibTex

  @inproceedings{lime:kdd16,
    author = {Marco Tulio Ribeiro and Sameer Singh and Carlos Guestrin},
    title = { {"Why Should I Trust You?": Explaining the Predictions of Any Classifier} },
    booktitle = {Knowledge Discovery and Data Mining (KDD)},
    month = {August},
    doi = {10.1145/2939672.2939778},
    pages = {1135-1144},
    year = {2016}
  }