As artificial intelligence advances, medicine will too. The Healthcare sector is benefiting from new advances in many fields of AI, not only in those applications directly related to the patient, but also in fields such as natural language processing (NLP) and natural language generation (NLG). It’s in that spirit that Narrativa is presenting NarbioBART: a revolutionary model for medical use in Spanish.

The origins of this model

BART (link to paper: https://arxiv.org/abs/1910.13461) is a denoising autoencoder for pretraining sequence-to-sequence models. It is trained by corrupting text with an arbitrary noising function and then learning a model to reconstruct the original text. 

BART uses a standard transformer-based neural machine translation architecture and can be seen as generalizing BERT (due to the bidirectional encoder) and GPT (with the left-to-right decoder), among others. It is effective when fine-tuned for text generation and also works well for comprehension tasks, matching the performance of RoBERTa on certain benchmark datasets and achieving state-of-the-art results on abstractive dialogue, question-answering, and summarization tasks. 

BART also provides improvement over a back-translation system for machine translation, with only target language pretraining. The authors conducted ablation experiments to better understand the factors that influence end-task performance within the BART framework.

Encode and decode: the best of both worlds

The Narrativa innovation team proposed ​​adapting this model to Spanish by combining encoder with decoder—the best of both worlds. NarbioBART is the first initiative toward replicating these Stanford models in Spanish.

Until now there was no generative model with these characteristics. NarbioBART is a mix between RoBERTa and GPT-2; for training we have used the largest corpus for medical tasks, a biomedical domain corpus. It performs just as well and is, of course, a generative model.

Uses of NarbioBART

This model can be used for any downstream like:

• Classification of text
• Recognition of entities 
• Question-answering

But it really shines in text generation tasks such as summarizing or abstractive QA.

Where can I test it?

Check out our Hugging Face page here.