CSE 599 D1: Advanced Natural Language Processing- Spring 2019

Wed, Fri 1:30-2:50 pm in Gates 271


Instructor: Hanna Hajishirzi (Paul Allen Center 470) (hannaneh at washington dot edu)

Office hours: by appointment

TA: Dae Lee (dhlee4 at uw dot edu)

Office hours: by appointment

Course Objectives

This advanced course deeply explores important topics in NLP. The objective of this class is to enhance students' knowledge about current techniques, challenges, and developments in different areas of NLP; to encourage discussions and collaborations among students; to improve students' analytical thinking and creativity; and to improve presentation and writing skills. In this class, students are required to read, think, present, and write intensively.


Theme: Representation and Reasoning in NLP

This course will be concentrated on knowledge representation and reasoning in NLP, including semantic representations, knowledge acquisition, question answering, and reasoning. The goal is to acquire comprehensive understanding and insights into the emerging developments and challenges of these advanced topics, and to develop future research directions and original research ideas.


Class Activities and Evaluation

Students will be evaluated based on these four activities (% of final grade):

(i) leading class discussions (30%),

(ii) participation in the discussions (25%),

(iii) paper review (15%),

(iv) research proposal, in which each student proposes an original research project (30%).

(Optional) students can optionally pursue a research project which will contribute toward additional 25% grade.


Prerequisite and Background Material

This advanced course deeply explores important topics in NLP. It is assumed that participants have taken CSE 517, are familiar with the fundamental ideas in NLP, and pursue research in NLP or related fields. Here are some pointers for background materials.

      UW-NLP CSE517 [here]

      NLP textbooks:

      D. Jurafsky & James H. Martin, Speech and Language Processing: n Introduction to Natural Language Processing, Computational Linguistics and Speech Recognition, Prentice Hall, Second Edition, 2009

      C.D. Manning & H. Schuetze, Foundations of Statistical Natural Language Processing, Cambridge: MIT Press, 1999 (available online, free if accessed from UW computers)

Click here for potential papers for class discussions.










Transformer, GPT, GPT1, BERT, Illustrated Transformer, The Annotated Transformer; led by Antoine and Ari

Contextual text encoding



GCNs, GraphIE, GATs (Optional, will be covered briefly) ,R-GCN (Optional, will be covered briefly); led by James and Colin

Graph encoding



Making the V in VQA Matter, Referring Expression; led by Rowan and Kiana




Knowledgeable Reader, Using Wiki as an external knowledge; led by Samantha and Kaitlyn

QA and Knowledge



Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text, Question Answering on Knowledge Bases and Text using Universal Schema and Memory Networks; led by Shobhit and Sewon

QA and Knowledge



Dynamic Integration of Background Knowledge in Neural NLU Systems, Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehension, Reasoning about Actions and State Changes by Injecting Commonsense Knowledge ;led by Kaidi and Deric

Reasoning in Narratives



CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge, , Ranking and Selecting Multi-Hop Knowledge Paths to Better Predict Human Needs, Modeling Naive Psychology of Characters in Simple Commonsense Stories (Optional), Commonsense Reasoning for Natural Language Understanding: A Survey of Benchmarks, Resources, and Approaches (Optional);led by Maarten and Hannah

Commonsense Reasoning



Guest speaker; ConvE, DisMult(Optional), ComplEx(Optional); led by Tim

Relational Graph Encoding


Learning a Natural Language Interface with Neural Programmer,Neural Semantic Parsing with Type Constraints for Semi-Structured Tables,
Compositional Semantic Parsing on Semi-Structured Tables (Optional), Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision (Optional) ;led by Derek and Nick

Neural Symbolic Methods

Paper review deadline(Before the class)


Rationalizing Neural Predictions ,Training Classifiers with Natural Language Explanations(Optional) ,How Much Attention Do You Need? ,The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation ,Using the Output Embedding to Improve Language Models;led by Aida and Ofir




Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing, Adversarial Decomposition of Text Representation;led by Sean and Phoebe




Unsupervised Data Augmentation , Semi-Supervised Sequence Modeling with Cross-View Training , Data Programming: Creating Large Training Sets, Quickly (Optional)led by Elizabeth and David(W)




Spider , TypeSQL, pair2vec;led by Ethan and David(P)




Graph Convolutional Networks Based Word Embeddings , A Structural Probe for Finding Syntax in Word Representations, Colorless Green Recurrent Networks Dream Hierarchically;led by Naomi and Chih-Chan


First proposal draft submission


Visual Dialog, Wizard of Wikipedia, Conversing by Reading: Contentful Neural Conversation with On-demand Machine Reading;led by Karen and Akshat




Neural Belief Tracker, Interpretation of Natural Language Rules in Conversational Machine Reading , Global-Locally Self-Attentive Dialogue State Tracker (Optional);led by Divye and Victor




Linguistically-Informed Self-Attention for Semantic Role Labeling, GLoMo: Unsupervisedly Learned Relational Graphs as Transferable Representations, THE LOTTERY TICKET HYPOTHESIS: FINDING SPARSE, TRAINABLE NEURAL NETWORKS;led by Zhaofeng and Ellen




Second proposal draft submission(Noon 5-Jun)


A General Framework for Information Extraction using Dynamic Span Graphs, Giving Attention to the Unexpected; led by Vicky and Yi

Review Period Ends(11:59pm 7-Jun)



Last proposal draft submission(Noon 12-Jun)


Paper Review

We will provide the list of papers for the paper report. Check here for instructions for paper review. The paper review should not be longer than 2 pages. Deposit the paper review to the dropbox. Also, to see some more samples for paper reviews you can check the open review website https://openreview.net. For example, these are sample reviews from the most recent ICLR conference: https://openreview.net/group?id=ICLR.cc/2019/Conference


Research Proposal

Write a 4-page research proposal paper describing a line of research in NLP. The research proposal should be about a new project that would extend a clearly identified past research contribution. The research proposal should:

      Build upon or extend what was done in the past work;

      Address challenges or weaknesses in the past research;

      Propose logical extensions or next steps to the focus research; and

      Describe a possible evaluation methodology, experimental design, and required evaluation resources.

You can find some guidelines in the link, or schedule a meeting with the instructor or the TA to learn more.


The 4-page limit is a hard constraint and will be enforced seriously. References don't count toward the page limit. Please use the ACL 2019 style files without modification.

Submit the research proposal to the Dropbox.


Course Administration and Policies

      Assignments should be done individually unless otherwise specified. You are expected to maintain the utmost level of academic integrity in the course.

      Late Policy: Each student has SEVEN PENALTY-FREE DAY FOR THE WHOLE QUARTER(**UPDATED 25 APR 2019**), other than that any late submission will be penalized at a penalty of 10% of the maximum grade per day.


      If you have comments of general interest, please use the discussion forum. Please consider posting your questions there; everyone will benefit. We also encourage you to try to answer questions, which will count as class participation. We will monitor daily and contribute as long as the boards are being used.

      We appreciate feedback throughout the quarter -- you can submit feedback through the Allen School's anonymous feedback tool

      Submit assignments to the dropbox