Datasets WG – 01.14.26

Updated Mar 17, 2026 Datasets & Benchmarks

Jan 14, 2026

Attendees

Invited Rosina Haberl John Marconi Pete Bernard Peter Chang raashid.ansari@silabs.com Sherif Eissa Tamas Daranyi Vijay Janapa Reddi Weier Wan EAIF Team Calendar Gianvito Urgese Adam Fuks Emil Jørgensen Njor Eric Smiley Kiruba Subramani Seung Hun SHIN/신승훈 Petrut Bogdan sam@imagimob.com suzen@embedl.com

Summary

Pete Bernard, John Marconi, Eric Smiley, Adam Fuks, and Rosina Haberl shared CES experiences, noting industry maturity and the lack of sickness for John after taking preventative measures. John Marconi is to follow up with Tomer on legal approval for the Sonic Scale and SIBA keyword spotting data sets to meet the March San Diego publication goal, while Pete Bernard will restart discussions with David from ML Commons to clarify input and feedback on their roadmap. The participants agreed on aiming for six data sets for 2024 to establish a publication cadence and discussed defining and scoping an abstracted best practice from the NPU for SLMs for the edge, with Pete Bernard taking the action to invite Brad Scott of Auzone to a future meeting to discuss commercialized solutions benchmarking.

Details

  • CES and Personal Health Pete Bernard shared their experience at CES, mentioning that the whole industry felt more mature and the event was super productive for them compared to the previous year (00:00:54). Pete noted that they felt sick after the event, an experience they had previously when working at Microsoft. John Marconi mentioned taking vitamin C and zinc a week prior to the event and managed not to get sick, while Eric Smiley forgot to take their Airborne supply (00:00:00).
  • Audio Data Set Publication Status The working group is awaiting legal consent to upload the Sonic Scale and a keyword spotting data set from SIBA, which John Marconi clarified are two separate items (00:06:05). Adam Fuks noted that Tomer had indicated approval was expected within a week or two, which was two weeks prior to the meeting. Pete Bernard, having met with SIBA’s new SVP of AI, mentioned they told the SVP that the group was waiting on SIBA to move forward. John Marconi is to follow up with Tomer to check on the approval status (00:05:16). Pete stressed the need to publish these data sets before the San Diego event in March, offering to push for approval by the end of the week if no update is received (00:06:05).
  • 2024 Goals and Data Sets Adam Fuks suggested aiming for a specific number of data sets for the year, while Pete Bernard proposed aiming for six data sets, or one every other month, to establish a publication cadence. Pete agreed that defining the number and broad categorization of data sets would be a good outcome for the year (00:07:58).
  • Collaboration with ML Commons (ML Perf) Pete Bernard noted that discussions with ML Commons have gone silent and suggested pinging David from ML Commons to clarify how the working group can provide input and feedback on their efforts, as the working group does not intend to act as the benchmark police. Pete is taking an action to restart the discussion with David to understand their roadmap for smaller models and SLMs in this space (00:08:44).
  • SLMs for the Edge and Best Practices Adam Fuks raised the potential for work around SLMs for the edge, specifically data sets for RAG, SLMs, and sentence transformers, where the working group could add value (00:09:28). The idea of creating a guide on best practices for SLMs, possibly based on Danilo’s prior suggestion, was discussed (00:10:17). Adam Fuks highlighted the challenge of giving advice publicly, as it might expose an organization’s weaker points (00:12:26). The group agreed that a helpful and doable deliverable could be defining and scoping an abstracted best practice from the NPU itself (00:13:09).
  • White Papers and CANs Data Set Adam Fuks suggested that if the audio data set is obtained, a white paper could be written showcasing what can be done with the data. The group acknowledged the need to decide what to do with the CANs data set, as they have not found anyone working with it (00:14:37). Rosina Haberl confirmed that a lead with a beverage manufacturer in Thailand did not pan out (00:15:35). Pete Bernard proposed checking with Goran for another generative AI-based manufacturing defect detection data set beyond CANs (00:16:25).
  • Benchmarking Commercialized Solutions Pete Bernard brought up Brad Scott of AuZone’s suggestion about a gap in benchmarking actual engineered and developed commercial solutions, rather than just pre-development theoretical performance (00:16:25). Adam Fuks noted that Peter Warden and VJ had discussed creating standards for this at a Tiny ML Summit a few years ago, but the problem persists (00:17:34). Brad is interested in working with the group to provide guidance for evaluating capability and performance for commercialized devices (00:18:31). Pete will invite Brad to a future meeting to present their problem for the brain trust to consider (00:19:16). John Marconi suggested that for commercial benchmarks to be adopted, the pull would need to come from consumers like Starbucks or Walmart (00:19:53).
  • Working Group Attendance and Recruitment Pete Bernard acknowledged that meeting attendance was low, noting that only a few members were present. Rosina Haberl informed the group that Petro is likely on vacation. Pete suggested a recruitment drive targeting new partners like Intel and Analog Devices to boost engagement (00:22:54). The San Diego event is scheduled for March 24 to 26, two weeks after Embedded World (00:23:33). Pete proposed Rosina circulate a blurb about the working group to new members (00:24:19).
  • Working Group Wiki Pete Bernard mentioned a discussion from the steering committee about creating a public-facing wiki for each working group, similar to the Open Compute Foundation, to provide meeting notes, goals, and increase industry engagement (00:25:12). The idea is to stand up a private wiki first, then potentially make it public (00:26:40). The meeting concluded with Adam Fuks, John Marconi, and Kiruba Subramani expressing thanks and looking forward to a successful year .

Suggested next steps

  • uncheckedPete Bernard will send an extra push to ensure the sonic scale data set is done and published before the San Diego event in March if they do not hear by the end of this week.
  • uncheckedPete Bernard will ping David again in the new year to check if the group can help with the ML Perf efforts and understand their roadmap.
  • uncheckedJohn Marconi will reach out to Tomer to see where they are with legal approval for uploading the data set stuff.
  • uncheckedPete Bernard will loop back with Brad Scott and suggest he join the next meeting to discuss his problem regarding benchmarking and evaluating capabilities and performance for commercialized devices.
  • uncheckedRosina Haberl will pitch a blurb about the working group to the new members to recruit more participants.