Hi! I’m Jakub, and I lead a team of researchers at Samsung R&D focused on developing AI Agents for devices such as smartphones. Additionally, I’m a third-year Industrial Ph.D. student at Warsaw University of Technology. My research interests include AI Agents, Multimodal LLMs, and AI Safety & Security.

My Research

This year, my primary focus is on enhancing the commercial viability of AI Agents (Next-Generation Voice Assistants for devices like smartphones and TVs). My most significant scientific achievement was the acceptance of the NL-ITI publication to the Interspeech 2024 Conference.

In the ClickAgent report, we introduced a novel architecture that addresses the limitations of MLLMs in accurately locating icons on the screen. By integrating a specialized UI location model, TinyClick, we significantly improved performance, outperforming other popular AI Agents in GUI automation tasks.

Last year, my research had two major focuses. One was on AI safety, where we aimed to align LLMs more closely with human and company values. In the NL-ITI publication, we presented an LLM internal representation engineering method, which delivered a notable performance boost on the TruthfulQA benchmark. Additionally, we investigated fine-tuning LLMs for machine translation tasks, particularly of training data for spoken language understanding (SLU) systems, with the goal of significantly reducing need for manual data annotation. Our work achieved state-of-the-art results on the multiATIS++ - Multilingual SLU benchmark. Furthermore, I identified a vulnerability that LLMs could serve as carriers for hidden messages and invented a method for extracting such covert information.

Awards

Samsung Award for Best Inspirer 2023: Recognized for outstanding research and engineering contributions related to LLMs.

During my Bachelor’s and Master’s studies, I was awarded the Dean’s scholarship twice in recognition of exceptional academic performance.