Enterprise AI Analysis

Exploration of the tokenization process in LLMs, highlighting the limitations of traditional Byte-Pair Encoding (BPE) for morphologically rich languages like Arabic. Introduces the novel Fanar Morphology-based Tokenizer (MorphBPE), designed to align with Arabic's unique linguistic structure while maintaining statistical efficiency. Includes details on tokenizer preprocessing, training, and evaluation metrics such as Fertility, Perplexity, and the proposed Morphological Alignment Score.

Overview of the model architectures for Fanar Star (7.1B) and Fanar Prime (8.78B), both decoder-only Transformer models. Fanar Star is trained from scratch using a two-stage curriculum with dynamic data mixtures, while Fanar Prime is continually pre-trained on Gemma-2-9B. Ablation studies on data filtering and mixture composition are discussed, along with the detailed training recipes and optimization configurations for both models.

Covers the supervised fine-tuning (SFT) and preference learning phases, emphasizing Arabic cultural and safety alignment. Describes data curation from public sources, synthetic data generation, and collection of user feedback. Outlines the multi-stage training workflow and annealing phase. Provides comprehensive evaluation results against Arabic-aware peer models on standard and culturally aware benchmarks, including automatic and human evaluations for both base and instruction-tuned models.

Explains the integration of speech and image modalities into the Fanar platform. Details the inclusive Arabic Speech Recognition (ASR) system supporting multiple Arabic dialects and code-switching, along with Text-to-Speech (TTS) capabilities. Discusses the image generation model, fine-tuned to reflect Arab and Islamic preferences, addressing knowledge and preference biases in existing models through a taxonomy of visual concepts and model averaging.

Description of the various Retrieval Augmented Generation (RAG) systems implemented in Fanar: Islamic RAG, Recency and Biography RAG, and Attribution RAG. These systems enhance the accuracy and factual grounding of generated content across specific domains. Also includes an overview of LLM security and safety, outlining the aiXamine project for evaluating model vulnerabilities and ensuring responsible AI development.