IMDA and AI Verify Foundation announce first of its kind Generative AI Evaluation Sandbox for Trusted AI

19 December 2023

On 31 October 2023, the Info-communications Media Development Authority of Singapore (“IMDA”) and the AI Verify Foundation announced the first of its kind Generative AI (“Gen AI”) Evaluation Sandbox (“Sandbox”). Over 10 global players are listed as participants in the Sandbox. The Sandbox brings global ecosystem players together through concrete use cases, to enable the evaluation of trusted artificial intelligence (“AI”) products.

Set out below are some key points to note about the Sandbox:

  • Common language for evaluation of Gen AI: To have a common standard approach to assess Gen AI, the Sandbox will make use of a new draft Evaluation Catalogue (“Catalogue”) which sets out common baseline methods and recommendations for Large Language Models (“LLM”). The Catalogue provides an anchor by (a) compiling the existing commonly used technical testing tools and organising these tests according to what they test for and their methods, and (b) recommending a baseline set of evaluation tests for use in Gen AI products.

The AI Verify Foundation welcomes initial comments and feedback on the draft Catalogue.

  • Build up a body of knowledge on how Gen AI products should be tested: The Sandbox will help build evaluation capabilities beyond what currently resides with model developers. Where possible, each Sandbox use case should involve an upstream Gen AI model developer, a downstream application deployer and a third-party tester to demonstrate how the different players in the ecosystem can work together.
  • Develop new benchmarks and tests: The Sandbox will develop benchmarks for evaluating model performance in specific areas that are important for use cases, and for countries like Singapore because of cultural and language specificities.

AI Verify Foundation and IMDA invite interested model and app developers, and third-party testers to participate in the Sandbox.

More details are available from the press release.

Reference materials

The following materials are available on the IMDA website www.imda.gov.sg and the AI Verify Foundation website www.aiverifyfoundation.sg: