FL-9B-4

FL-9B-4 is a COBOL / mainframe code model fine-tuned from Qwen/Qwen3.5-9B-Base via supervised fine-tuning (SFT) on a curated COBOL instruction dataset. It targets legacy-code understanding, COBOL generation, and COBOL-to-Java translation.

• Base model: Qwen/Qwen3.5-9B-Base (dense 9B, hybrid linear + full attention)
• Method: LoRA SFT (assistant-only masking), ~3 epochs, bf16
• Domain: COBOL, GnuCOBOL, mainframe knowledge, COBOL and Java

Benchmark results

All code benchmarks compile and execute generated programs against reference tests. Evaluated greedy (temperature 0), single sample per task, via vLLM. "Base" = Qwen/Qwen3.5-9B-Base (no fine-tuning), evaluated with the same harness and an injected ChatML template, so the delta reflects the SFT alone.

Benchmark	Metric	Base	FL-9B-4
COBOLEval	pass@1	0.68%	36.99%
	compile rate	8.65%	82.10%
	test pass rate	1.46%	52.98%
COBOL-JavaTrans (C2J)	pass@1	42.66%	80.42%
	compile success rate (CSR)	46.85%	96.50%
MainframeBench	MCQ accuracy	66.23%	71.26%
	QA - Token F1	11.68%	12.75%
	QA - ROUGE-L	9.33%	10.29%
	Summarization - Token F1	23.62%	27.64%
	Summarization - ROUGE-L	16.38%	20.25%
CobolCodeBench	INSTRUCT compile rate	2.17%	47.83%
	COMPLETE compile rate	0.00%	32.61%

The fine-tuning produces very large gains on COBOL generation and understanding: COBOLEval pass@1 rises from ~1% to 37%, COBOL compile rate from 9% to 82%, and CobolCodeBench COMPLETE from 0% to 33%. COBOL-to-Java translation nearly doubles in pass@1 (from 43% to 80%). MainframeBench MCQ moves less (from 66% to 71%), since factual mainframe knowledge is largely already present in the base model.

Notes on evaluation

The MainframeBench MCQ, CobolCodeBench INSTRUCT and COMPLETE numbers were produced after fixing harness-side generation limits (the default 16-token MCQ budget and 2048-token code budget truncated answers, and single-format cobc invocation rejected valid programs written in a different column format). Fixed evaluation uses a larger generation budget and tries variable, free and fixed COBOL formats when compiling. Reported numbers reflect the model's actual capability, not the truncated defaults.

The strongest results - COBOL-to-Java translation (80% pass@1) and COBOLEval (82% compile) - show the model reliably produces valid, working COBOL and translates legacy code into working Java.

Intended use

• Translating legacy COBOL programs to Java
• Completing and generating GnuCOBOL programs
• Answering mainframe / COBOL knowledge questions
• Assisting with legacy-code modernization workflows

Limitations

• Open-ended QA and summarization scores (Token F1 / ROUGE-L) are modest; the model is stronger at code generation and translation than at free-form prose.
• COBOL generation quality varies with column-format conventions; generated code may mix fixed and free formats.
• Not evaluated for safety-critical or production mainframe deployment without human review.

Training

Setting	Value
Base	Qwen/Qwen3.5-9B-Base
Method	LoRA (r=32, alpha=64), assistant-only SFT
Precision	bf16
Epochs	~3
Sequence length	8192 (packed)
Hardware	1x NVIDIA RTX PRO 6000 Blackwell (96 GB)
Frameworks	Unsloth + Transformers

LoRA adapters were applied to attention projections, MLP projections, and the linear-attention (in_proj_*/out_proj) modules of the hybrid Qwen3.5 architecture; the vision tower, MTP head, and router/embedding/LM-head were excluded.

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "FLs-AI/FL-9B-4"  # adjust to your repo
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")

messages = [{"role": "user", "content": "Translate this COBOL program to Java:\n\n<COBOL here>"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=2048, temperature=0.0)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))