Falcon-H1-Tiny-R-90M — cyber-security QLoRA

A QLoRA fine-tune of tiiuae/Falcon-H1-Tiny-R-90M on public red-team and cyber-security datasets. Weights are merged into the base model, so this repository loads as a normal causal language model.

Training

Setting Value
Method QLoRA (4bit base) + LoRA
LoRA rank / alpha / dropout 8 / 16 / 0.05
LoRA targets q_proj, k_proj, v_proj, o_proj, in_proj, gate_proj, up_proj, down_proj
Learning rate 0.0002
Epochs 1
Micro batch x grad accum 1 x 8
Max sequence length 1024 (longer rows dropped, not truncated)
Optimizer paged_adamw_8bit
Seed 42

Data

reloading0101/threat-intelligence-dataset is CC-BY-NC-SA-4.0 (non-commercial); check every source before commercial use.

Files

  • model.safetensors — merged weights (bf16) with the tokenizer and chat template.
  • training_config.json — resolved configuration used for this run.
  • model-q8_0.gguf

Usage

Transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sourcepirate/falcon-h1-tiny-cyber"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
messages = [{"role": "user", "content": "Write a tcpdump command for DNS only"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=128)[0][inputs["input_ids"].shape[-1]:],
                       skip_special_tokens=True))

llama.cpp (CPU or GPU, no Python):

llama-cli -m model-q8_0.gguf -p "Write a tcpdump command for DNS only" --n-predict 128

Evaluation

Training and per-row evaluation gate on finite loss; no benchmark scores are claimed here. A NaN/inf loss aborts the run instead of publishing an adapter.

Limitations and authorised use

This model is intended for authorised security testing, research and education. It can produce offensive commands and may be wrong; review every output before running it. The base model ships under the Falcon-LLM licence (link), and the merged model inherits it.

Downloads last month
382
Safetensors
Model size
91.1M params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for sourcepirate/falcon-h1-tiny-cyber

Quantized
(6)
this model

Datasets used to train sourcepirate/falcon-h1-tiny-cyber