We've developed an AI-powered analytics dashboard for the Prime Minister's Office. The primary goal of this project was to create a tech platform enabling data science teams to assess the effectiveness of their new natural language processing models (LLM) on government data. This dashboard provides an intuitive interface and advanced tools to analyse model performance while ensuring compliance and security of government data.

Challenges in LLM Model Evaluation

The project identified several specific challenges faced by the government's data science teams in evaluating AI models, including:

• Complexity of Public Data: Government data is vast, diverse, and sensitive. It's vital that LLM models can process this data effectively while maintaining confidentiality and security.

• Evaluation and Benchmarking: Evaluating model performance on public data sets is complex, requiring consideration of various criteria (accuracy, processing time, bias, etc.) and comparing results to standard or previously obtained benchmarks.

• Compliance and Transparency: AI solutions must adhere to strict standards of transparency and explainability to ensure decisions based on models are understandable and justifiable, particularly in a governmental context.

Solutions Offered by the AI Analytics Dashboard

Our platform developed for the Prime Minister's Office provides innovative solutions to the mentioned challenges:

• Automated Model Evaluation: The dashboard allows data science teams to test their new LLM models on government data sets, automating evaluation based on key performance indicators (accuracy, recall, F1-score, etc.). Results are presented visually and interactively, easing decision-making.

• Benchmarking and Comparisons: The platform enables comparison of new models with previous ones or existing benchmarks, assisting teams to measure progress and optimise their approaches. Synthetic charts and tables make it easy to visualise each model's gains or limitations.

• Compliance and Auditability: A dedicated section for transparency and auditability has been integrated into the dashboard. It allows teams to trace the origin of the data used, verify the processing steps, and ensure models comply with ethical and legal AI guidelines.

• User-Friendly Interface: The tool features an accessible, customisable interface, facilitating usage by a variety of profiles, whether expert data scientists or decision-makers looking to understand model results.

Impact and Adoption

The AI Analytics dashboard has become a key tool for government data science teams. It accelerates the development and evaluation of LLM models while ensuring decisions are based on reliable data. This solution not only enhances the performance of AI models but also ensures rigorous and transparent governance of public data.

The Prime Minister's Office thus benefits from a platform effectively supporting AI projects at a state level, meeting compliance, ethical, and data security requirements.