Middle East Technical University
Computer Engineering Department
CENG7811 Applied Natural Language Processing
FALL 2024-2025
Time:
Thursday, 13:40 - 16:30
Room:
G-102
Course Objectives:
The primary objective of this course is to introduce students how to design, develop, and deploy systems that can extract insights and meaning from human language data. Students will be able to analyze text datasets, work with popular NLP libraries, and apply NLP techniques to solve real-world problems. Through lectures, discussions, and hands-on projects, students will gain a deep understanding of NLP applications, underlying structure of large language models, possible use cases and existing challenges for large language models.
Background Requirements:
Probability and Statistics (required); Python programming (required); Deep learning (strongly recommended).
Course Slides:
Review of NLP Approaches & Text Semantics
Large Language Models (Encoder and Generative Models)
LLM Knowledge, Reasoning, and RAG
LLM Training, Prompting, and Ethical Issues
Grading (Tentative):
Project: 50% (Progress Report 15%, Final Report 15%, Demo %20)
Paper Review and Presentation: 15% (Paper Report and Presentation 10%, Paper Implementation 5%)
Final Exam: 35%
Project Details:
Projects will be done in groups of 2 or 3 students. Project groups will be formed by students. Project topics will be provided by the instructor (student’s project proposal can also be accepted depending on its quality and relevance to the industry applications). Each project will be assigned a mentor from the industry or academia, who will supervise the group together with the instructor. Each project group will submit a Project Progress Report and a Project Final Report (in ACL Conference Style). They will also present a Project Demonstration at the end of the semester.
Paper Review Details:
Paper Review and Presentation will be done individually. Papers will be academic publications (well-known journals or conferences). Papers will be provided by the instructor (student’s paper suggestion can also be accepted depending on its relevance to the project). Each paper will be a related work to the project topic. Students will be expected to understand, review, present, and implement the methods proposed in the papers. The paper implementation (baseline method) will be used in the project’s experiments section to compare with the student’s proposed method.
Final Exam Details:
Final Exam will be a written exam that involves multiple questions types including but not limited to true-false, short answer, and computational questions.
Recommended Reading:
Books:
D. Jurafsky and J. H. Martin. 2023. “Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition” (Third edition).
Papers:
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... and Polosukhin, I. 2017. “Attention is all you need” Advances in neural information processing systems, 30.
Devlin, J., Chang, M. W., Lee, K., and Toutanova, K. 2019. “BERT: Pre-training of deep bidirectional transformers for language understanding” In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 4171-4186).
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. 2019. “Language models are unsupervised multitask learners” OpenAI blog, 1(8), 9.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., ... and Amodei, D. 2020. “Language models are few-shot learners”. Advances in neural information processing systems, 33, 1877-1901.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., ... and Chen, W. 2022. “Lora: Low-rank adaptation of large language models” The Tenth International Conference on Learning Representations (ICLR 2022) Virtual Event.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., ... and Lowe, R. 2022. “Training language models to follow instructions with human feedback” Advances in neural information processing systems, 35, 27730-27744.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M. A., Lacroix, T., ... and Lample, G. 2023. “Llama: Open and efficient foundation language models” arXiv preprint arXiv:2302.13971.