Artigos, Blog

JurisBERT: A New Approach that Converts a Classification Corpus into an STS One

Publicado por

18 ago 2023

Abstract

We propose in this work a new approach that aims to transform a classification corpus into an STS (Semantic Textual Similarity) one. In that sense, we use BERT (Bidirectional Encoder Representations from Transformers) to validate our hypothesis, i.e., a multi-level classification dataset can be converted into an STS dataset which improves the fine-tuning step and evidences the proposed corpus. Also, in our approach, we trained from scratching a BERT model considering the legal texts, called JurisBert which reveals a considered improvement in fastness and precision, and it requires less computational resources than other approaches. JurisBERT uses the concept of sub-language, i.e., a model pre-trained in a language (Brazilian Portuguese) passes through refining (fine-tuning) to better attend to a specific domain, in our case, the legal field. JurisBERT uses 24k pairs of ementas with degrees of similarity varying from 0 to 3. We got this data from search mechanisms available on the court websites to validate the model with real-world data. Our experiments showed JurisBERT is better than other models such as multilingual BERT and BERTimbau with $3.30 %$ better precision ( $F_{1}$ ), 5 times reduced training time, and using accessible hardware, i.e., low-cost GPGPU architecture. The source code is available at https://github.com/alfaneo-ai/brazilian-legal-text-dataset and the model is here: https://huggingface.co/alfaneo.

Keywords

Semantic Textual Similarity
Retrieving Legal Precedents
Sentence Embedding
Bert

Notes

1.

The Conselho Nacional de Justiça is a public institution that aims to help the Brazilian judiciary. It maintains administrative and procedural control and transparency.
2.

STF, STJ, TJRJ, TJMS.
3.

It is an operation that reduces the dimensionality of data by applying an aggregation of type max average.
4.

It is a dense vector of floating points that aims to capture the semantic of the text in the vector space.
5.

The súmulas summarizes the dominant precedent of a given court.
6.

Proposed dataset and web scrappers are available here: https://github.com/alfaneo-ai/brazilian-legal-text-dataset, and the models, here: https://huggingface.co/alfaneo.

References

Agirre, E., Cer, D., Diab, M., Gonzalez-Agirre, A.: SemEval-2012 task 6: A pilot on semantic textual similarity. In: *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pp. 385–393. Association for Computational Linguistics, Montréal, Canada (7–8 Jun 2012), https://aclanthology.org/S12-1051
Banerjee, S., Mishra, B., Jawanpuria, P., Shrivastava, M.: Generalised spherical text embedding (2022)

Google Scholar
Beltagy, I., Lo, K., Cohan, A.: Scibert: A pretrained language model for scientific text. In: EMNLP (2019)

Google Scholar
Cer, D., et al.: Universal sentence encoder for English. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pp. 169–174. Association for Computational Linguistics, Brussels, Belgium (Nov 2018). https://doi.org/10.18653/v1/D18-2029, https://aclanthology.org/D18-2029
Choi, H., Kim, J., Joe, S., Gwon, Y.: Evaluation of bert and albert sentence embedding performance on downstream nlp tasks (2021). https://doi.org/10.48550/ARXIV.2101.10642, https://arxiv.org/abs/2101.10642
CNJ: Justiça em números 2020: ano-base 2019. Tech. rep., Conselho Nacional de Justiça (2020), https://www.cnj.jus.br/pesquisas-judiciarias/justica-em-numeros/
Cunha, A.S.: Introdução ao estudo do direito. Saraiva (2012)

Google Scholar
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018). http://arxiv.org/abs/1810.04805
Fonseca, E.R., Borges dos Santos, L., Criscuolo, M., Aluísio, S.M.: Visão geral da avaliação de similaridade semântica e inferência textual. Linguamática 8(2), 3–13 (Dez 2016). https://linguamatica.com/index.php/linguamatica/article/view/v8n2-1
Hendrycks, D., Gimpel, K.: Gaussian error linear units (gelus) (2016). https://doi.org/10.48550/ARXIV.1606.08415, https://arxiv.org/abs/1606.08415
Jiang, T., et al.: Promptbert: Improving bert sentence embeddings with prompts (2022). https://doi.org/10.48550/ARXIV.2201.04337, https://arxiv.org/abs/2201.04337
Johnson, J., Douze, M., Jégou, H.: Billion-scale similarity search with gpus (2017). https://doi.org/10.48550/ARXIV.1702.08734, https://arxiv.org/abs/1702.08734
Júnior, H.: Curso de direito processual civil, vol. I. Editora Forense (2019)

Google Scholar
Keskar, N.S., McCann, B., Xiong, C., Socher, R.: Unifying question answering, text classification, and regression via span extraction (2019)

Google Scholar
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization (2014). https://doi.org/10.48550/ARXIV.1412.6980, https://arxiv.org/abs/1412.6980
Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R.S., Torralba, A., Urtasun, R., Fidler, S.: Skip-thought vectors (2015). https://doi.org/10.48550/ARXIV.1506.06726, https://arxiv.org/abs/1506.06726
Liu, Y., et al.: Roberta: A robustly optimized bert pretraining approach (2019). https://doi.org/10.48550/ARXIV.1907.11692, https://arxiv.org/abs/1907.11692
Meshram, S., Anand Kumar, M.: Long short-term memory network for learning sentences similarity using deep contextual embeddings. International Journal of Information Technology, pp. 1633–1641 (2021)

Google Scholar
Nguyen, H.T., Nguyen, L.M.: Sublanguage: A Serious Issue Affects Pretrained Models in Legal Domain. arXiv e-prints arXiv:2104.07782 (Apr 2021)
Pennington, J., Socher, R., Manning, C.: GloVe: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543. Association for Computational Linguistics, Doha, Qatar (Oct 2014). https://doi.org/10.3115/v1/D14-1162, https://aclanthology.org/D14-1162
Rasmy, L., Xiang, Y., Xie, Z., Tao, C., Zhi, D.: Med-bert: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction (2020). https://doi.org/10.48550/ARXIV.2005.12833, https://arxiv.org/abs/2005.12833
Real, L., Fonseca, E., Gonçalo Oliveira, H.: The assin 2 shared task: A quick overview. In: Quaresma, P., Vieira, R., Aluísio, S., Moniz, H., Batista, F., Gonçalves, T. (eds.) Computational Processing of the Portuguese Language, pp. 406–412. Springer International Publishing, Cham (2020)

CrossRef Google Scholar
Real, L., et al.: SICK-BR: A Portuguese Corpus for Inference: 13th International Conference, PROPOR 2018, Canela, Brazil, September 24–26, 2018, Proceedings, pp. 303–312. Springer, Cham (01 2018). https://doi.org/10.1007/978-3-319-99722-3_31
Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks (2019). https://doi.org/10.48550/ARXIV.1908.10084, https://arxiv.org/abs/1908.10084
Song, X., Salcianu, A., Song, Y., Dopson, D., Zhou, D.: Fast wordpiece tokenization (2020). https://doi.org/10.48550/ARXIV.2012.15524, https://arxiv.org/abs/2012.15524
Souza, F., Nogueira, R., Lotufo, R.: Bertimbau: Pretrained bert models for brazilian portuguese. In: Intelligent Systems: 9th Brazilian Conference, BRACIS 2020, Rio Grande, Brazil, October 20–23, 2020, Proceedings, Part I. p. 403–417. Springer-Verlag, Berlin, Heidelberg (2020). https://doi.org/10.1007/978-3-030-61377-8_28
Vaswani, A., et al.: Attention is all you need (2017)

Google Scholar
Wagner Filho, J.A., Wilkens, R., Idiart, M., Villavicencio, A.: The brwac corpus: A new open resource for brazilian portuguese. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) (2018)

Google Scholar
Weber, R.: Intelligent jurisprudence research: A new concept. In: Proceedings of the 7th International Conference on Artificial Intelligence and Law. p. 164–172. ICAIL ’99, Association for Computing Machinery, New York, NY, USA (1999). https://doi.org/10.1145/323706.323791, https://doi.org/10.1145/323706.323791
Zhang, G., Lillis, D., Nulty, P.: Can Domain Pre-training Help Interdisciplinary Researchers from Data Annotation Poverty? A Case Study of Legal Argument Mining with BERT-based Transformers. In: Proceedings of the Workshop on Natural Language Processing for Digital Humanities (NLP4DH), pp. 121–130. Association for Computational Linguistics (2021), https://rootroo.com/downloads/nlp4dh_proceedings_draft.pdf

Download references

Acknowledgments

We thank the support of the UFMS (Universidade Federal de Mato Grosso do Sul), FUNDECT, and Finep. We also thank the support of the INCT of the Future Internet for Smart Cities funded by CNPq, proc. 465446/2014-0, Coordenação de Aperfeiçoamento de Pessoal de Nível Superior – Brasil (CAPES) – Finance Code 001, and FAPESP, proc. 2014/50937-1 and 2015/24485-9.

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of FUNDECT, Finep, FAPESP, CAPES, and CNPq.

Author information

Authors and Affiliations

Alfaneo, Goias, 405, 79020-100, Campo Grande, MS, Brazil

Charles F. O. Viegas & Bruno C. Costa
Federal University of Mato Grosso do Sul, Campo Grande, MS, Brazil

Renato P. Ishii