Abstract
We propose in this work a new approach that aims to transform a classification corpus into an STS (Semantic Textual Similarity) one. In that sense, we use BERT (Bidirectional Encoder Representations from Transformers) to validate our hypothesis, i.e., a multi-level classification dataset can be converted into an STS dataset which improves the fine-tuning step and evidences the proposed corpus. Also, in our approach, we trained from scratching a BERT model considering the legal texts, called JurisBert which reveals a considered improvement in fastness and precision, and it requires less computational resources than other approaches. JurisBERT uses the concept of sub-language, i.e., a model pre-trained in a language (Brazilian Portuguese) passes through refining (fine-tuning) to better attend to a specific domain, in our case, the legal field. JurisBERT uses 24k pairs of ementas with degrees of similarity varying from 0 to 3. We got this data from search mechanisms available on the court websites to validate the model with real-world data. Our experiments showed JurisBERT is better than other models such as multilingual BERT and BERTimbau with 3.30%3.30% better precision (F1�1), 5 times reduced training time, and using accessible hardware, i.e., low-cost GPGPU architecture. The source code is available at https://github.com/alfaneo-ai/brazilian-legal-text-dataset and the model is here: https://huggingface.co/alfaneo.
Keywords
- Semantic Textual Similarity
- Retrieving Legal Precedents
- Sentence Embedding
- Bert
Notes
- 1.
The Conselho Nacional de Justiça is a public institution that aims to help the Brazilian judiciary. It maintains administrative and procedural control and transparency.
- 2.
- 3.
It is an operation that reduces the dimensionality of data by applying an aggregation of type max average.
- 4.
It is a dense vector of floating points that aims to capture the semantic of the text in the vector space.
- 5.
The súmulas summarizes the dominant precedent of a given court.
- 6.
Proposed dataset and web scrappers are available here: https://github.com/alfaneo-ai/brazilian-legal-text-dataset, and the models, here: https://huggingface.co/alfaneo.
References
-
Agirre, E., Cer, D., Diab, M., Gonzalez-Agirre, A.: SemEval-2012 task 6: A pilot on semantic textual similarity. In: *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pp. 385–393. Association for Computational Linguistics, Montréal, Canada (7–8 Jun 2012), https://aclanthology.org/S12-1051
-
Banerjee, S., Mishra, B., Jawanpuria, P., Shrivastava, M.: Generalised spherical text embedding (2022)
-
Beltagy, I., Lo, K., Cohan, A.: Scibert: A pretrained language model for scientific text. In: EMNLP (2019)
-
Cer, D., et al.: Universal sentence encoder for English. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pp. 169–174. Association for Computational Linguistics, Brussels, Belgium (Nov 2018). https://doi.org/10.18653/v1/D18-2029, https://aclanthology.org/D18-2029
-
Choi, H., Kim, J., Joe, S., Gwon, Y.: Evaluation of bert and albert sentence embedding performance on downstream nlp tasks (2021). https://doi.org/10.48550/ARXIV.2101.10642, https://arxiv.org/abs/2101.10642
-
CNJ: Justiça em números 2020: ano-base 2019. Tech. rep., Conselho Nacional de Justiça (2020), https://www.cnj.jus.br/pesquisas-judiciarias/justica-em-numeros/
-
Cunha, A.S.: Introdução ao estudo do direito. Saraiva (2012)
-
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018). http://arxiv.org/abs/1810.04805
-
Fonseca, E.R., Borges dos Santos, L., Criscuolo, M., Aluísio, S.M.: Visão geral da avaliação de similaridade semântica e inferência textual. Linguamática 8(2), 3–13 (Dez 2016). https://linguamatica.com/index.php/linguamatica/article/view/v8n2-1
-
Hendrycks, D., Gimpel, K.: Gaussian error linear units (gelus) (2016). https://doi.org/10.48550/ARXIV.1606.08415, https://arxiv.org/abs/1606.08415
-
Jiang, T., et al.: Promptbert: Improving bert sentence embeddings with prompts (2022). https://doi.org/10.48550/ARXIV.2201.04337, https://arxiv.org/abs/2201.04337
-
Johnson, J., Douze, M., Jégou, H.: Billion-scale similarity search with gpus (2017). https://doi.org/10.48550/ARXIV.1702.08734, https://arxiv.org/abs/1702.08734
-
Júnior, H.: Curso de direito processual civil, vol. I. Editora Forense (2019)
-
Keskar, N.S., McCann, B., Xiong, C., Socher, R.: Unifying question answering, text classification, and regression via span extraction (2019)
-
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization (2014). https://doi.org/10.48550/ARXIV.1412.6980, https://arxiv.org/abs/1412.6980
-
Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R.S., Torralba, A., Urtasun, R., Fidler, S.: Skip-thought vectors (2015). https://doi.org/10.48550/ARXIV.1506.06726, https://arxiv.org/abs/1506.06726
-
Liu, Y., et al.: Roberta: A robustly optimized bert pretraining approach (2019). https://doi.org/10.48550/ARXIV.1907.11692, https://arxiv.org/abs/1907.11692
-
Meshram, S., Anand Kumar, M.: Long short-term memory network for learning sentences similarity using deep contextual embeddings. International Journal of Information Technology, pp. 1633–1641 (2021)
-
Nguyen, H.T., Nguyen, L.M.: Sublanguage: A Serious Issue Affects Pretrained Models in Legal Domain. arXiv e-prints arXiv:2104.07782 (Apr 2021)
-
Pennington, J., Socher, R., Manning, C.: GloVe: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543. Association for Computational Linguistics, Doha, Qatar (Oct 2014). https://doi.org/10.3115/v1/D14-1162, https://aclanthology.org/D14-1162
-
Rasmy, L., Xiang, Y., Xie, Z., Tao, C., Zhi, D.: Med-bert: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction (2020). https://doi.org/10.48550/ARXIV.2005.12833, https://arxiv.org/abs/2005.12833
-
Real, L., Fonseca, E., Gonçalo Oliveira, H.: The assin 2 shared task: A quick overview. In: Quaresma, P., Vieira, R., Aluísio, S., Moniz, H., Batista, F., Gonçalves, T. (eds.) Computational Processing of the Portuguese Language, pp. 406–412. Springer International Publishing, Cham (2020)
-
Real, L., et al.: SICK-BR: A Portuguese Corpus for Inference: 13th International Conference, PROPOR 2018, Canela, Brazil, September 24–26, 2018, Proceedings, pp. 303–312. Springer, Cham (01 2018). https://doi.org/10.1007/978-3-319-99722-3_31
-
Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks (2019). https://doi.org/10.48550/ARXIV.1908.10084, https://arxiv.org/abs/1908.10084
-
Song, X., Salcianu, A., Song, Y., Dopson, D., Zhou, D.: Fast wordpiece tokenization (2020). https://doi.org/10.48550/ARXIV.2012.15524, https://arxiv.org/abs/2012.15524
-
Souza, F., Nogueira, R., Lotufo, R.: Bertimbau: Pretrained bert models for brazilian portuguese. In: Intelligent Systems: 9th Brazilian Conference, BRACIS 2020, Rio Grande, Brazil, October 20–23, 2020, Proceedings, Part I. p. 403–417. Springer-Verlag, Berlin, Heidelberg (2020). https://doi.org/10.1007/978-3-030-61377-8_28
-
Vaswani, A., et al.: Attention is all you need (2017)
-
Wagner Filho, J.A., Wilkens, R., Idiart, M., Villavicencio, A.: The brwac corpus: A new open resource for brazilian portuguese. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) (2018)
-
Weber, R.: Intelligent jurisprudence research: A new concept. In: Proceedings of the 7th International Conference on Artificial Intelligence and Law. p. 164–172. ICAIL ’99, Association for Computing Machinery, New York, NY, USA (1999). https://doi.org/10.1145/323706.323791, https://doi.org/10.1145/323706.323791
-
Zhang, G., Lillis, D., Nulty, P.: Can Domain Pre-training Help Interdisciplinary Researchers from Data Annotation Poverty? A Case Study of Legal Argument Mining with BERT-based Transformers. In: Proceedings of the Workshop on Natural Language Processing for Digital Humanities (NLP4DH), pp. 121–130. Association for Computational Linguistics (2021), https://rootroo.com/downloads/nlp4dh_proceedings_draft.pdf
Acknowledgments
We thank the support of the UFMS (Universidade Federal de Mato Grosso do Sul), FUNDECT, and Finep. We also thank the support of the INCT of the Future Internet for Smart Cities funded by CNPq, proc. 465446/2014-0, Coordenação de Aperfeiçoamento de Pessoal de Nível Superior – Brasil (CAPES) – Finance Code 001, and FAPESP, proc. 2014/50937-1 and 2015/24485-9.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of FUNDECT, Finep, FAPESP, CAPES, and CNPq.
Author information
Authors and Affiliations
-
Alfaneo, Goias, 405, 79020-100, Campo Grande, MS, Brazil
Charles F. O. Viegas & Bruno C. Costa
-
Federal University of Mato Grosso do Sul, Campo Grande, MS, Brazil
Renato P. Ishii
Corresponding author
Correspondence to Renato P. Ishii .
Fonte: https://link.springer.com/chapter/10.1007/978-3-031-36805-9_24