Cẩm nang công nghệ

How do you interact with AI when your hands and eyes are busy? Explore the technology behind AskAI in this month’s tech highlight.

Table of content:

Introduction

Part 1: Applications

Part 2: User interaction workflow

Part 3: Hardware architecture 

Part 4: Firmware and ai processing flow

Part 5: Current prototype limitations

Part 6: Feature development roadmap

Conclusion

References and abbreviations

INTRODUCTION

1.    Beyond the Touchscreen

While most smart devices today still rely on touchscreens, an important question arises:

Can we interact with AI in a more natural way — without looking, without touching?

In many real-life situations, users need to observe their surroundings or are occupied with manual tasks, making screen-based interaction inconvenient and less efficient. As a result, the need for a more intuitive, seamless, and hands-free communication method has become increasingly clear.

To address this challenge, TMA R&D Center has researched and developed AskAI, a smart headphone prototype integrated with Edge AI. In this technology newsletter, TMA R&D Center  is pleased to introduce the project architecture in detail, including the system operation mechanism, hardware and software design, as well as potential real-world application directions.

2.    What is AskAI?

AskAI is designed as an AI system capable of combining multiple data sources, including voice and images, to understand real-world context and respond more appropriately. The device brings together edge processing capabilities, also known as AI-on-edge, and cloud-based reasoning which enable users to:

• Communicate by voice

• Capture the surrounding environment

• Receive natural audio responses

“From the user’s perspective, AskAI works like a voice assistant inside a headset: turn it on, connect it to Wi-Fi, say the wake word, ask a question, and listen to the answer” 

PART 1: APPLICATIONS

AskAI can be applied in several practical scenarios.

• Support for Visually Impaired Users: AskAI can help users better understand their surroundings by describing objects, reading labels or text, identifying scenes, and providing basic navigation support through voice feedback.

• Personal AI Assistant: Users can ask questions while walking, traveling, working, or performing hands-free activities without needing to open a phone or look at a screen.

• Learning and Education: AskAI can act as an AI learning companion, helping users understand concepts, practice languages, learn vocabulary, or ask questions based on real-world visual context.

• Work and Productivity: AskAI can support engineers, technicians, students, and field workers by providing quick AI assistance in situations where screen-based interaction is inconvenient.

Figure 1: AskAI Application

Overview of the actual AskAI hardware prototype

Figure 2: AskAI hardware prototype PCB

Figure 3: AskAI hardware prototype in frame headset

Figure 4: AskAI hardware prototype (This frame has not been designed with a mounting location for a camera. In the future, camera will be mounted on the sides of the headset frame)

PART 2: USER INTERACTION WORKFLOW

1.      User Setup:

•         Turn on AskAI via Power button
•         AskAI will switch to Access point mode, allowing users to connect to its Wi-Fi (example – via mobile phone)

Figure 5: Connect to Access Point

•         Access default IP of device and input appropriate information to connect AskAI to user Wi-Fi network

Figure 6: Input Wi-Fi information

• After completing all the steps above, AskAI will switch to waiting mode.

2.      Workflow:

•         User wakeup the AskAI using the wake word: “Hey Bubble”
•         AskAI wakes up successfully, then the user can ask a question and device also capture the image data
•         The AI system then processes the context and responds directly through the headset speaker.

Figure 7: AskAI User Manual

PART 3: HARDWARE ARCHITECTURE

This smart device integrates a central processor with a camera, microphone, and specialized audio and power chips to seamlessly capture images and voice commands, handling local AI activation before sending data to the cloud and delivering audio responses straight to your headphones.

Below is the detailed breakdown of each key hardware component. Requirement:

Key hardware component: Camera, microphone, handle AI on edge, PMIC, PCB layout optimal

Key Features: AI on edge, low power consumption, Voice capture, Image capture, Audio codec

• MCU – ESP32-S3:

ESP32 serves as the central processor of the system. It coordinates key components such as the OV5640 camera, ICS-43434 digital microphone, MAX9867 audio codec, Wi-Fi connectivity, LED indicators, power button, volume control, and communication with the backend AI server. ESP32 also runs the wake word AI model directly on the device, enabling local AI-on-edge activation

• Voice Capture – ICS-43434:

The digital microphone captures the user’s question through I2S. The recorded

audio is saved as a WAV file and then sent to the backend for processing. And this microphone also supports Wake word recognition.

• Image Capture – Camera OV5640:

The OV5640 camera captures an image of the surrounding environment. The image is encoded and sent together with the user’s voice input to provide visual context for AI processing.

• Audio Codec - MAX9867:

The MAX9867 is an ultra-low-power stereo audio codec that supports audio input from digital microphones and provides audio output through headphones.

The device includes ADC and DAC functions, allowing conversion between analog audio signals and digital audio data.

• PMIC - TPS65070RSLR:

The power system is managed by the TPS65070RSLR PMIC, which provides stable power rails for the ESP32, camera, microphone, audio codec, LEDs, and charging circuit. The PMIC plays an important role in power distribution, voltage monitoring, charging status management, and reliable system startup.

Figure 8: AskAI Hardware Components

PART 4: FIRMWARE AND AI PROCESSING FLOW

The firmware implements an AI voice assistant workflow running on the embedded device. The device is responsible for listening for user activation, recording the user’s audio question, capturing an image from the camera, sending both inputs to the server, waiting for the server-side AI processing result, and finally playing back the generated audio response through the MAX9867 audio codec.

Figure 9: AskAI Firmware Workflow Overview

Figure 10: AskAI Firmware Workflow Detail

1.  System Initialization

When the device boots, the firmware initializes the required hardware and software components, including the PMIC, Wi-Fi connection, MAX9867 codec, I2S microphone interface, LED status indicator, FFat file system, and RTOS tasks.

The firmware also starts several background tasks:

• Monitors the trigger input.

• Adjusts playback volume.

• Handles power-related control (on/off headphone)

• Controls LED status indication.

2.  Wake Word Detection

At this stage, the microphone should continuously monitor incoming audio and detect a predefined wake word. Once the wake word is detected, the device should transition from idle/listening mode into active recording mode.

The current wake word module is implemented using TensorFlow Lite Micro running directly on the ESP32-S3. Audio is continuously captured from the I2S microphone, buffered in a short rolling window, and converted into FFT-based audio features. These features are passed to the local TensorFlow Lite model to detect the wake word before triggering the main AskAI workflow.

3.  On-Device Processing Flow

After the wake word or trigger event is detected, the device starts the main processing task. The main task performs the following steps:

1.   Mounts and prepares the FFat file system.

2.   Initializes I2C and the MAX9867 codec.

3.   Captures an image from the camera.

4.   Converts the captured image to Base64.

5.   Upload the Base64 image data to the server through the endpoint.

6.   Records the user’s audio question from the I2S microphone.

7.   Saves the recorded audio as a .wav file in FFat.

8.   Uploads the recorded audio file to the server through the endpoint.

9.   Polls the server to check whether the AI response audio is ready.

10.  Downloads and plays the response audio from the server.

4.  Voice Recording Flow

The device records audio through the I2S microphone interface. The recorded data is scaled and written into a file .WAV

The AskAI will play audio “start recording” to inform user about audio recording. Recording automatically stops after 2 seconds of silence. And the audio file is uploaded to the server

5.  Image Capture and Context Collection

The device wakes up the camera, captures an image, converts the image to Base64 format, and sends it to the server.

The image is used as a visual context for multimodal reasoning on the server side. This allows the AI model to answer based on both the user’s spoken question and the captured camera image.

6.  Server-Side AI Processing

On the server side, the system receives both the recorded audio file and the captured image from the device. The audio file is first processed by the speech-to-text model Whisper-1.

After transcription, the server sends the question text together with the Base64 image to the reasoning model GPT-4o (based on user request). This allows the AI to reason based on both the user’s voice question and the camera image, then generate a text response.

Finally, the generated text response is converted into speech using text-to-speech model tts-1 with WAV output format. The server stores the generated audio file, marks the response as ready, and the device downloads this audio file and playback.

7.  Server Response Handling

After uploading the audio and image data, the device waits for the server to finish processing.

The firmware will be in the loop semaphore waiting for the response from the server in 60 seconds. When the server response, the device knows that the response audio is available and starts audio playback.

8.  Audio Playback

The audio response is streamed from the server and written to the MAX9867 codec through I2S. The codec outputs the final speech response to the speaker.

9.  Technology Stack

Figure 11: AskAI Technology Stack

PART 5: CURRENT PROTOTYPE LIMITATIONS

Although AskAI demonstrates a practical prototype for hands-free multimodal AI interaction, the current version still has several limitations that should be considered during product evaluation and future development.

1.  Response Latency

The current AskAI workflow includes several sequential steps, such as wake word detection, image capture, audio recording, server upload, AI processing, audio download, and playback. Since speech-to-text, multimodal reasoning, and text-to-speech are processed on the server side, the total response time depends on network quality and server performance. In the current design, the firmware may wait for the server response for up to 60 seconds. Therefore, the user experience may not always feel fully real-time.

2.  Battery Life and Power Consumption

AskAI uses multiple power-consuming components, including the ESP32-S3 MCU, camera, I2S microphone, MAX9867 audio codec, Wi-Fi module, LED indicators, and PMIC system. Continuous wake word monitoring, camera activation, audio recording, server communication, and playback can increase power consumption. Since the current version is still a prototype, battery life may be limited compared with a fully optimized commercial wearable device. Future optimization is required to improve operating time per charge.

3.  AI Response Accuracy

The accuracy of AskAI’s response depends on the quality of both audio and image inputs. Background noise, low microphone volume, unclear speech, poor lighting, blurry images, or incorrect camera angles may affect the AI’s understanding. If speech-to-text transcribes the question incorrectly or the image does not contain enough visual context, the final response may be incomplete or inaccurate. This limitation is important because AskAI relies on both voice and visual information for reasoning.

4.  Wake Word Reliability

The wake word detection module runs locally on the ESP32-S3 using TensorFlow Lite Micro. However, wake word accuracy can still be affected by background noise, user pronunciation, speaking distance, microphone placement, and limited device resources. In some cases, the system may miss the wake word or be triggered accidentally by similar-sounding audio. Further model tuning and more diverse training data are needed to improve reliability in real-world environments.

5.  Network Dependency

The current AskAI prototype relies on Wi-Fi to upload captured image data and recorded audio to the AI server. This means the main AI workflow depends on a stable network connection. If Wi-Fi is weak, unstable, disconnected, or unavailable, the device may not be able to complete transcription, reasoning, text-to-speech generation, and audio response playback. This can limit usability in outdoor areas or environments with poor network coverage.

6.  Limited Visual Context

The current AskAI workflow uses one captured still image as visual context for AI reasoning. This allows the system to understand what is visible at the moment of capture, but it does not provide continuous video understanding or temporal awareness. As a result, the AI may not fully understand moving objects, changing scenes, or events that happen before or after the captured frame. Future versions may require multi-frame capture or short video input for richer real-world context.

PART 6: FEATURE DEVELOPMENT ROADMAP

Future development of AskAI will focus on improving personalization, edge AI capability, hardware optimization, and user experience.

1.  Personalized AI Profiles

AskAI can support different profiles for visually impaired users, students, technical workers, travelers, or personal assistant use cases. Each profile can have its own prompt strategy, knowledge base, response style, and level of detail.

2.  Edge AI Expansion

Future versions can further optimize wake word detection, reduce false positives, improve recognition in noisy environments, and add lightweight on-device models such as voice activity detection, noise suppression, keyword spotting, or intent classification.

3.  Local AI Deployment

In future versions, the current MCU can be replaced with a more powerful MCU that can support local AI. This upgrade would allow AskAI to run lightweight AI models directly on the device, reducing cloud dependency, improving response latency, increasing privacy.

4.  Hardware Improvement

Future hardware development can focus on reducing PCB size, optimizing power layout, improving high-speed signal routing, enhancing audio quality, stabilizing the camera module, refining LED notifications, and improving battery efficiency.

With these improvements, AskAI can evolve from a working prototype into a practical, compact, and personalized AI wearable device for daily life, accessibility, education, and professional use.

Future development of AskAI will focus on improving personalization, edge AI capability, response latency, network reliability, battery efficiency, AI accuracy, hardware optimization, and overall user experience. These improvements aim to evolve AskAI from a functional prototype into a practical, compact, and reliable AI wearable device.

5.  Network Reliability Improvement

The current prototype depends on Wi-Fi to upload audio/image data and receive AI responses from the server. Future versions should improve reconnection logic, retry handling, network status indication, and fallback behavior.

6.  Battery Life and Power Optimization

Battery life is critical for a wearable device because AskAI uses the microphone, camera, Wi-Fi, audio codec, LEDs, and MCU. Future firmware should optimize low-power listening mode, peripheral sleep/wakeup control, Wi-Fi power saving, and RTOS task scheduling. Hardware improvements can also focus on better battery capacity and power management.

7.  AI Accuracy Improvement

AskAI’s response quality depends on audio clarity, image quality, lighting, camera angle, and AI model performance. Future versions should improve audio preprocessing, noise reduction, automatic gain control, and image quality validation. Better prompt strategy and multimodal reasoning can also improve response reliability.

8.  Wake Word Detection Improvement

The wake word module should be further optimized to work better in noisy environments and with different users, accents, and speaking distances. Future development can include more diverse training data, better noise-robust features, and improved false-positive/false-negative handling. This will make device activation more stable and natural.

9.  Hardware and Wearable Design Improvement

Future hardware development can focus on reducing PCB size, improving power layout, stabilizing the camera module, enhancing audio quality, and improving battery efficiency.

The headphone form factor should also be optimized for comfort, weight balance, durability, and daily wearable use. These improvements will help AskAI become more practical for real-world users.

With these improvements, AskAI can evolve from a working prototype into a compact, personalized, and reliable AI wearable device for daily life, accessibility, education, travel, and professional use.

CONCLUSION

Toward Practical AI Wearables

AskAI represents an important experimental step toward a more natural, seamless, and context-aware AI interaction model. Instead of requiring users to look at a screen or interact through a traditional touch interface, AskAI explores a hands-free experience where users can communicate by voice, capture surrounding visual context, and receive spoken responses directly through a wearable device.

In the future, the product roadmap can move toward more on-device intelligence, including lightweight local AI models, improved noise suppression, better context filtering, and partial offline capability. Reducing cloud dependency will not only improve response latency but also enhance privacy, reliability, and usability in environments with limited network connectivity. At the same time, hardware optimization such as smaller PCB design, improved power management, better camera placement, higher audio quality, and more efficient battery usage will be essential for transforming AskAI into a compact and comfortable wearable device.

Overall, AskAI is not only a prototype of smart headphones integrated with AI, but also a step toward a new generation of multimodal AI wearables. It shows how AI can become more ambient, more accessible, and more naturally embedded into real-world activities. The R&D Department will continue to improve the system, evaluate real-user scenarios, and explore future product directions to bring AskAI closer to a practical, reliable, and personalized AI assistant for everyday use.

REFERENCES AND ABBREVIATIONS

Hardware:

[1]  MCU ESP32 S3: esp32-s3_datasheet_en.pdf

[2]  Camera OV5640: OV5640_datasheet.pdf

[3]  Codec MAX9867: MAX9867 - Ultra-Low Power Stereo Audio Codec

[4]  Micro ICS-43434: DS-000069 ICS-43434 v1.2

[5]  PMIC: TPS65070 data sheet, product information and support | TI.com

Firmware:

[6]  TFLite: tensorflow/tflite-micro: Infrastructure to enable deployment of ML models to low-power resource-con…

[7]  ESPCam: espressif/esp32-camera

[8]  I2S: Inter-IC Sound (I2S) - ESP32-S3 - — ESP-IDF Programming Guide v6.0.1 documentation

[9]  FAT: FAT Filesystem Support - ESP32 - — ESP-IDF Programming Guide v6.0.1 documentation

Abbreviations:

[1]  MCU Microcontroller Unit

[2]  PCB Printed Circuit Board

[3]  PMIC Power Management Integrated Circuit

[4]  IC Integrated Circuit

[5]  I2S Inter-IC Sound

[6]  I2C Inter-Integrated Circuit

[7]  WAV Waveform Audio File Format

[8]  ADC Analog-to-Digital Converter

[9]  DAC Digital-to-Analog Converter

[10]  CAD Computer-Aided Design

[11]  RTOS Real-Time Operating System

[12]  FFat Flash FAT File System