BERTau: Itau BERT for digital customer service

In the last few years, three major topics received increased interest: deep learning, NLP and conversational agents. Bringing these three topics together to create an amazing digital customer experience and indeed deploy in production and solve real-world problems is something innovative and disruptive. We introduce a new Portuguese financial domain language representation model called BERTau. BERTau is an uncased BERT-base trained from scratch with data from the Itau virtual assistant chatbot solution. The novelty of this contribution lies in that BERTau pretrained language model requires less data, reaches state-of-the-art performance in three NLP tasks, and generates a smaller and lighter model that makes the deployment feasible. We developed three tasks to validate our model: information retrieval with Frequently Asked Questions (FAQ) from Itau bank, sentiment analysis from our virtual assistant data, and a NER solution. All proposed tasks are real-world solutions in production on our environment and the usage of a specialist model proved to be effective when compared to Google BERT multilingual and the Facebook’s DPRQuestionEncoder, available at Hugging Face. BERTau improves the performance in 22% of FAQ Retrieval MRR metric, 2.1% in Sentiment Analysis F1 score, 4.4% in NER F1 score. It can also represent the same sequence in up to 66% fewer tokens when compared to "shelf models".