BushraAlmod03/ArTrOCR-HTR

Arabic TrOCR for Handwritten Text Recognition

A TrOCR-based model fine-tuned for line-level Arabic handwritten text recognition, adapted from a pretrained English TrOCR checkpoint through a staged curriculum: synthetic structured-font pretraining, synthetic cursive-font pretraining, real KHATT handwriting fine-tuning.

Model Description

• Base model: microsoft/trocr-base-handwritten
• Architecture: Vision Encoder-Decoder Transformer
• Input: Line-level handwritten Arabic text images, 512x102 resolution
• Data:
  • Synthetic pretraining: 18,000 structured-font + 12,000 cursive-font images
  • Main fine-tuning: 11,000 KHATT lines
• CER on primary benchmark test set: 12.61% (95% CI: [11.88%, 13.38%]; mean across 3 seeds: 12.60% ± 0.10 percentage points)

Scope and Limitations

This model targets line-level Arabic handwriting resembling the primary training benchmark's writing style, pen characteristics, aspect ratio, and background conditions. It has not been trained on diacritized text, lined-paper backgrounds, or paragraph-level input. Out-of-distribution testing shows partial robustness to lined paper, noisy backgrounds, colored ink, and slant when combined with preprocessing (cropping, grayscale conversion, binarization, deskewing). Diacritic recognition is not reliable, as diacritics were not represented in training data or vocabulary.

How to Use

! pip install transformers torch Pillow opencv-python numpy scikit-image
from transformers import VisionEncoderDecoderModel, TrOCRProcessor
from PIL import Image
import torch
import cv2
import numpy as np
    
processor = TrOCRProcessor.from_pretrained("BushraAlmod03/ArTrOCR-HTR")
model = VisionEncoderDecoderModel.from_pretrained("BushraAlmod03/ArTrOCR-HTR")
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
# Configure Model 
print(f"Processor configured: {processor.image_processor.size}, do_resize={processor.image_processor.do_resize}")
if not model.config.encoder.interpolate_pos_encoding:
    model.config.encoder.interpolate_pos_encoding = True
print("Position embedding interpolation enabled")
if not hasattr(model.encoder, '_original_forward'):
    model.encoder._original_forward = model.encoder.forward
    def patched_forward(pixel_values, *args, **kwargs):
        kwargs['interpolate_pos_encoding'] = True
        return model.encoder._original_forward(pixel_values, *args, **kwargs)
    model.encoder.forward = patched_forward
    print("Encoder patched")
else:
    print("Already patched, skipping")

path = "path/to/line_image.png"

image = cv2.imread(path)
assert image is not None, f"Cannot read: {path}"

# resize to match model input:
img = Image.fromarray(image).convert("RGB")
target_h = 102
target_w = 512
w, h = img.size
scale = target_h / h
new_w = int(w * scale)
new_h = target_h
img = img.resize((new_w, new_h), Image.Resampling.LANCZOS)
canvas = Image.new("RGB", (target_w, target_h), 255)
if new_w <= target_w:
    canvas.paste(img, (0, 0))
else:
    img = img.resize((target_w, new_h), Image.Resampling.LANCZOS)
    canvas.paste(img, (0, 0))
preprocessed_image = cv2.cvtColor(np.array(canvas), cv2.COLOR_BGR2RGB)

'''
# Optional Preprocessing Step (could enhance feature extraction), Uncomment for use:
from huggingface_hub import hf_hub_download
import importlib.util
file_path = hf_hub_download(repo_id="BushraAlmod03/ArTrOCR-HTR", filename="preprocessing.py")
spec = importlib.util.spec_from_file_location("preprocessing", file_path)
preprocessing = importlib.util.module_from_spec(spec)
spec.loader.exec_module(preprocessing)
preprocessed_image = preprocessing.preprocess_image(image)
'''

pixel_values = processor(preprocessed_image, return_tensors="pt").pixel_values.to(device)
model.eval()
with torch.no_grad():
    generated_ids = model.generate(pixel_values, max_length=140)
predicted_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print('Recognized Text:',predicted_text)

Training Procedure

Full methodology, ablations, and evaluation details are described in the associated paper (link to be added upon publication).