Logo

ChronoLingua – Revive Ancient Chinese via Machine Translation

Team Member:

Letian Yu (lyu49), Xiner Zhao (xzhao99), Yuyang Luo (lyuyang), Zhengyang Xu (xzhengy1)

Introduction:

As a representative of traditional Chinese heritage, ancient Chinese classics hold immense historical and literary significance. However, due to challenges such as scarcity of comprehensive sentence-aligned corpora, and disparities in tokenization, word order, and syntax, contemporary machine translation techniques often overlook ancient Chinese translation tasks, consequently falling short of achieving satisfactory outcomes.

The project aims to create a robust solution capable of translating complex ancient Chinese scripts into modern languages. Besides, witnessing the high cost of translation due to the complexity of ancient Chinese grammar and limited sentence-aligned corpora, we further hope to explore the possibility of developing an unsupervised model by leveraging the latest advancements in transformer models and integrating diffusion techniques. We believe that the project will contribute to studies in ancient Chinese classics, and help promote traditional Chinese culture to a broader global audience.

Dataset:

Our project focuses on two tasks: ancient Chinese to English translation and ancient Chinese to mandarin Chinese translation. In terms of datasets, we hope to use the closed-source corpora provided by EvaHan.

The source of the training data includes the Ancient-Chinese-to-Modern-Chinese parallel texts of China Twenty-four Histories (二十四史), the Ancient-Chinese-to-English parallel texts of Pre-Qin classics (先秦经典) and “Zizhi Tongjian” (资治通鉴).

Data Source	Source Data	Target Data
Ancient-Chinese-to-Modern-Chinese parallel texts of China Twenty-four Histories	9,583,749 characters for the original Chinese Classic texts	12,763,534 characters
Ancient-Chinese-to-English parallel texts of Pre-Qin canonical texts and Zizhi Tongjian	618,083 characters for the original Ancient Chinese texts	838,321 words

To enrich our corpora, we might include additional data sources if necessary. Preprocessing of data such as tokenization and removal of uncommon words might be needed.

Related Work:

Researchers have been working on Machine Translation at a feverish pace aroused by Neural Machine Translation (NMT) models like the transformer-based BERT in 2018 which outperformed the former Recurrent Neural Network (RNN) and achieved astonishing success in Natural Language Processing (NLP) applications, including text understanding and thus Machine Translation. Later, Liu et al (2019) released the more advanced derivative RoBERTa, which exceeded BERT thanks to its larger batch size, longer training process, dynamic masking pattern and so on, boosting its abilities in contextualized word processing and offering the potential of model fine-tuning. While optimizing general application of self-attention mechanism and neural network algorithm in Machine Translation (Qin, 2022), researchers also work to introduce models specialized for particular domain, such as the Siku-RoBERTa, which is pre-trained on “Siku Quanshu” for Classical Chinese-related translation. Other efforts were also paid for the Classical Chinese translation quality. For instance, Zhang et al (2019) developed an unsupervised algorithm to overcome the lack of sentence-aligned corpora.

In terms of diffusion models, there have been efforts to explore diffusion models in a cross-lingual setting (Chen, 2023) or extend continuous text diffusion model to seq2seq text generation with encoder-decoder Transformer architecture (Yuan, 2023). However, these studies mainly focus on modern corpora (e.g., FR-EN, EN-GE) instead of ancient Chinese. The cutting-edge approach is what makes us excited and our main target will be to apply similar techniques to ancient Chinese corpora to test its feasibility.

In the preprocessing stage, we may utilize existing tokenization tools such as Jiayan or HanLP for ancient Chinese corpora. However, the effectiveness of such tools remains to be tested, and we may change these in future endeavors.

In case of any pretrained transformer models used, we will only look at the following three to ensure fairness of model comparison.

Pre-Trained Model	Language	Description
SikuRoBERTa	Ancient Chinese	Ancient Chinese RoBERTa pre-trained on high- quality Siku Quanshu(四库全书) full-text corpus.
Chinese-RoBERTa- wwm-ext	Modern Chinese	Modern Chinese pre-trained RoBERTa with Whole Word Masking strategy.
RoBERTa	Modern English	Pre-trained model on English with MLM objective.

[Updating] List of Relevant Implementations: There is no direct implementation of our problem. But we may refer to some similar methods in this field for the diffusion model.

Methodology

Our project includes 3 stages as described in the next section. To ensure feasibility of our problem, we will build RNNs and Transformers in a supervised learning manner for the machine translation task and evaluate the results accordingly. The challenges of this part mainly lie with the bottleneck of ancient Chinese translation (i.e., lack of corpora, character-based language, disparity of syntax). Successful implementation of this could also help us to finalize the evaluation metrics.

In the second stage, we will mainly focus on existing diffusion models on MT tasks and build models on ancient Chinese corpora in a similar manner. The challenges, in addition to the previous mentioned ones, will be model architectures and implementation. Computation resources might also be a concern given the corpora size. In this case, we will parallelly work on these two stages to ensure efficient implementation.

Lastly, with a hope to innovate existing model structure, we may explore the possibility of developing an unsupervised model by leveraging the latest advancements in transformer models and integrating diffusion techniques.

Metrics:

Following the setting of EvaHan 2023, we will evaluate our models using BLEU, chrF and COMET-QE (if possible). Additionally, we might test other scores such as METEOR, TER for performance evaluation.

BLEU measures machine translation quality by word-level n-grams. It is a modified version of the sacreBLEU, which provides hassle-free computation of shareable, comparable, and reproducible BLEU scores. The ChrF metrics evaluates the character-level translation quality and adds a recall metric, thus improving the correlation with human judgment. The COMET-QE is a state-of-the-art metric based on pre-trained models designed to predict human language experts’ judgments of machine translation quality, often with the highest accuracy.

Base goal: Our base goal is to implement different models (RNNs, Transformers) on Ancient Chinese to Modern Chinese and Ancient Chinese to English corpora, and compare the model performance with other baseline models using different metrics.
Target goal: As the core part of our project, we hope to apply the idea of diffusion to ancient Chinese translation tasks, featuring the innovation of our project. We hope the model can outperform certain baseline models, and be able to generate meaningful translations for both modern Chinese and English.
Stretch goal: We further hope to explore the possibility of developing an unsupervised model by leveraging the latest advancements in transformer models and integrating diffusion techniques.

Division of Labor:

We will work on data preprocessing and evaluation together. In terms of methodology, Two people will be in charge of the implementation of RNNs, Transformers and the evaluation of baseline models, and two people will be in charge of the diffusion model as well as potential improvements.

Ethics:

The existing problems in the translation of ancient Chinese classics and reasons why deep learning is a good approach to these problems

Ancient Chinese classics often lack explicit grammatical markers and rely heavily on context, leading to semantic ambiguity. To solve this problem, deep learning models such as machine translation can learn to interpret context and disambiguate meaning from surrounding text. By training on a large corpora of classical Chinese texts and their translations, these models can learn to recognize contextual cues and generate translations that capture the intended meaning more accurately.

Moreover, ancient Chinese classics texts are open to interpretation and can have multiple layers of meaning. Translators must carefully consider the historical, philosophical, and cultural context surrounding the text to provide accurate translations that reflect the author's intended message. However, deep learning models can learn to interpret the meaning of ambiguous or context-dependent phrases according to the implicit historical and cultural context surrounding the word. By training on texts with rich contextual annotations and leveraging techniques such as attention mechanisms, these models can generate translations that take into account the broader context surrounding the text, leading to more accurate and contextually appropriate translations.

Who are the major “stakeholders” in this problem, and what are the consequences of mistakes made by your algorithm?

Stakeholders using deep learning algorithms to translate ancient Chinese classics texts include researchers, scholars, educators, cultural institutions, and the general public. Researchers and scholars rely on accurate translations for academic analysis, while educators and students depend on them for effective teaching and learning. Cultural institutions and heritage organizations utilize translations for exhibitions and educational programs, and the general public benefits from accessible interpretations of classical Chinese literature.

However, mistakes made by the algorithm in translation could have significant consequences. Inaccuracies may lead to misinterpretation of texts, loss of cultural nuance, and hindered educational experiences. Moreover, reputational damage to individuals and institutions, as well as ethical concerns arising from misrepresentations, are potential outcomes of translation errors. To mitigate these risks, developers must prioritize accuracy, transparency, and accountability in their models, involving domain experts and stakeholders throughout the development process. Continuous monitoring and refinement are essential to ensure the production of high-quality translations and to address potential consequences effectively.

Check-in 3 Reflection

Challenges:

What has been the hardest part of the project you’ve encountered so far?

The most significant challenge in our project has been the implementation of the diffusion model. Unlike RNNs and transformers, diffusion models do not have ready-made solutions that can be directly applied to our task of ancient Chinese machine translation. This requires us to not only develop the model from scratch but also ensure that it integrates with our chosen pretrained transformer.

Additionally, we are manually creating unique tokenization separators, a critical factor that could significantly influence the performance of our final model. This task requires meticulous testing and optimization. Furthermore, establishing an efficient pipeline that encompasses preprocessing, training various models, conducting inference, and performing evaluations has been exceptionally time-consuming.

Insights:

Are there any concrete results you can show at this point? How is your model performing compared with expectations?

Over the past two weeks, we dedicated ourselves to implement our base model RNN and data preprocess pipeline. We initially experimented with various tokenizers and found that the HanLP tokenizer significantly outperforms others when it comes to handling ancient Chinese texts.

As of now, we have successfully preprocessed all the necessary data and implemented the transformer model along with the evaluation metrics. These steps are crucial as they form the backbone of our project, allowing us to proceed with the more experimental diffusion model.

Plan:

Are you on track with your project? What do you need to dedicate more time to? What are you thinking of changing, if anything?

Our project remains largely on track. We have successfully completed the data preprocessing, and the transformer model is now fully operational. Our immediate focus is on the crucial task of finalizing the diffusion model, which we aim to complete by the end of this weekend. Following this, we will commence training the RNNs, transformers and diffusion systems together, making adjustments as necessary based on initial results. It may also become necessary to revisit our preprocessing steps to ensure that there is optimal compatibility and performance between the two models. At the same time, given limited data size for classical chinese - english corpora, we will focus on classical chinese - modern chinese task.