Researchers from Beihang University (Beijing) and Wuhan University published a revised version of CangjieBench on August 12, 2026, a benchmark for evaluating Large Language Models in code generation for Cangjie. The programming language is part of the HarmonyOS ecosystem and has limited publicly available code examples and documentation. The study fills a research gap: while LLM coding assistant performance on established languages like Python and Java is well documented, empirical data on behavior with new, resource-scarce languages has been lacking.

Benchmark Design and Evaluation Methodology

CangjieBench consists of 248 manually translated tasks sourced from established benchmarks HumanEval and ClassEval. The dataset is uncontaminated and covers two modalities: Text-to-Code (generating code from natural-language requirements) and Code-to-Code (code translation). The researchers evaluated six LLMs using five different methods:

  • Direct Generation – direct code generation without additional information
  • Syntax-Constrained Prompting – prompting with syntax references
  • Retrieval-Augmented Generation (RAG) via documentation
  • Retrieval via code examples
  • Agent-based workflows with multi-turn repair loops

Evaluation was performed using Pass@1 (successful code generation on first attempt), Compile Rate (compilation rate), and Token Cost (computational costs). Errors were classified into seven categories, validated by two LLM annotators with a Cohen's Kappa of at least 0.98.

Direct Generation Fails in Over 80 Percent of Cases

According to the study, Direct Generation without additional information is impractical on most models. Over four-fifths of failed samples contain foreign syntax rejected by the Cangjie compiler. This result reveals a critical problem with unknown programming languages: LLMs fall back on patterns from training languages that are syntactically invalid in the target language.

A compact syntax reference of 2,146 tokens delivered the best accuracy-to-cost ratio among prompt-based methods. This concise syntax aid was more effective than more comprehensive approaches and significantly improved the compilation rate.

Agents Cost Up to 120 Times More Tokens

Agent-based workflows achieve the highest Pass@1 rates but require 10 to 120 times higher token costs than a single prompt call. The study also shows that weaker agent configurations do not reach the performance of a syntax-constrained request on the same base architecture. This finding challenges the blanket recommendation for agent systems in code generation.

Negative Transfer Phenomenon with Python References

An unexpected finding: in Code-to-Code tasks, Python references can lower compilation rates below the Text-to-Code level. The researchers attribute this to a negative transfer phenomenon where models adapt too strongly to Python source language idioms and transfer them to Cangjie, where they are syntactically invalid. Python references do not always help.

Tiered Assistance Architecture Recommended

For new programming languages, the study recommends a three-tier assistance architecture: a concise syntax cheat sheet as the first tier, curated retrieval methods for API-intensive tasks as the second tier, and agent loops only for tasks requiring multi-turn repair. This approach prioritizes accuracy and cost-efficiency over a universal high-accuracy strategy.

Scientific Contribution

CangjieBench addresses an underexplored area. While existing research on resource-scarce programming languages primarily focuses on Domain-Specific Languages (DSLs), this study is the first to systematically examine a general-purpose programming language with data scarcity. The term "Low-Resource General-Purpose Language" is specific to this category. Code and data are available on GitHub (https://github.com/cjhCoder7/CangjieBench). The paper was submitted on March 15, 2026 and revised on August 12, 2026.