aman-arora.space
⟵ back to mission log

TRANSMISSION · DAY 1241 · May 26, 2022

GSoC 2022 :: The Linux Foundation :: Proposal


Contact Information

  1. Name: Aman Arora
  2. E-Mail: aman.arora9848@gmail.com
  3. Country: India (GMT+5:30)
  4. Address: Panchkula, Haryana, India
  5. GitHub Username: amanarora9848
  6. IRC Nick: amanarora9848
  7. LinkedIn: https://www.linkedin.com/in/aman-arora9848/

Introduction

About the Organization

Organization

The Linux Foundation

Group

Automotive Grade Linux

Mentors

Jan-Simon Möller, Walt Miner

About

Project of Interest <GSoC’22 AGL>

An Offline Voice-agent for Automotive Grade Linux

Deliverables

Project Description

Objective 1: Create a new Yocto Layer meta-agl-devel/meta-voice-agent and integrate it with the Vosk Offline Speech Recognition toolkit to add Speech Recognition capability in AGL.

  1. Automotive Grade Linux is a collaborative, open-source project for the purpose of building open source software based on Linux for automotive applications. Its Unified Code Base (UCB) can power heterogeneous software applications like IVI, Instrument Clusters, and telematics in the vehicle. It is developed using the Yocto Project.

  2. Why Yocto Project?
    Yocto Project is an open-source project that allows people to build custom Linux systems from the ground up, and thus, one can create user application-specific systems based on the Linux kernel. Some of the best reasons to use the Yocto Project are:

    • It provides a wide hardware architecture support (ARM, PPC, MIPS, x86), thanks to the Board Support Package (BSP).
    • Developers can customize their systems in terms of speed, memory footprint, and memory utilization.
    • It provides extensive flexibility, and one can build custom Linux distros on top of existing ones.
    • Active community and a lot of existing layers and recipes.

  3. Yocto combines, maintains, and validates 3 development elements: an OpenEmbedded build system, a reference distribution, and a set of integrated tools that ease and enable working with embedded Linux.

    • The project grew from and is based on the OpenEmbedded Project, which is where the build system and some of the meta-data are derived from. The OpenEmbedded Project co-maintains the OE-core build system and provides a central point for new metadata.

    • Poky ⇒ OpenEmbedded Build system + metadata

      • Metadata: Task definitions / Set of instructions. This contains the files that the OpenEmbedded build system parses when building an image. Includes recipes, configuration files, and instructions on how to build an image.

    • Open Embedded Build system ⇒ BitBake + OE-Core (meta/ directory)

      • BitBake: A task executor and scheduler. It is a build engine that works through recipes written in a specific format in order to perform sets of tasks.
      • OpenEmbedded-Core: OpenEmbedded-Core (OE-Core) is a common layer of metadata (i.e. recipes, classes, and associated files) used by OpenEmbedded-derived systems, which includes the Yocto Project. It consists of foundation recipes, classes, and associated files that are meant to be common among many different OpenEmbedded systems.

  4. Particularly in the Yocto infrastructure, packages are grouped into layers and recipes are part of layers that are nothing but individual pieces of software. Layers are different kinds of repositories or folders and multiple recipes can be present within each layer. Different components of the Yocto project:

    • Configuration files (.conf): Global definition variables [types of machine architecture, global variables, build path, compiler flags]
    • Classes (.bbclass): Used to abstract common functionality and share it amongst multiple recipe (.bb) files [defines how we build Linux kernel, how to generate RPM package, how to create root file system image]
    • Recipes (.bb): Logical units of SW / Images to build [individual piece of SW to be built, what packages get included in the final file system image. Recipes include metadata for the SW such as where we download upstream sources from, build or runtime dependencies, what feature we enable in our application, configuration, compilation options, define what files go into what output package]
    • Layers (bblayers): Repositories that contain related metadata, set of recipes. Different Yocto layers exist in AGL, such as
      • poky
      • meta-agl
      • meta-agl-cluster-demo
      • meta-agl-demo
      • meta-agl-devel
      • meta-agl-extra
      • meta-agl-telematics-demo
      • meta-openembedded
        And so on…
    • Board Support Package (BSP): It is a collection of information that defines how to support different hardware platforms or devices. Essentially, a BSP Layer = base directory + file structure.
    • AGL supports the following hardware platforms. The different supported BSP layers are
      • meta-intel
      • meta-ti
      • meta-raspberrypi4
      • meta-renesas-rcar-gen3
      • meta-sancloud
        And so on…

  5. AGL Unified Code Base (UCB)

    • The AGL UCB is a Linux distribution built by a collaboration of automakers and suppliers to deliver modern in-vehicle infotainment and connected car experience. This provides really good hardware support, as listed here. This enables rapid innovation and development since UCB provides a starting point for the project, upon which an automaker can build customizations specific to their needs.

  6. Vosk: A framework that provides state-of-the-art offline speech recognition capabilities, using advanced deep learning under the hood. The Vosk toolkit is based on the Kaldi-ASR project.

    • Some highlights of Vosk are:
      • Supports 20+ languages and dialects, including English, German, French, Italian, Spanish, Japanese, etc.
      • Works offline and runs comfortably on lightweight devices.
      • Availability of per-language models of less than 50MB each.
      • Provides streaming API for the best user experience.
        And so on…
    • As a demo, the vosk-api repo was cloned and the test_microphone.py file stored in /vosk-api/python/example was run in a virtual environment after importing the English-small model:
        (vosk-test)infitrode@Aman-ThinkPad-E14-Gen3:~/vosk-api/python/example$ python3 test_microphone.py 
    LOG (VoskAPI:ReadDataFiles():model.cc:213) Decoding params beam=10 max-active=3000 lattice-beam=2
    LOG (VoskAPI:ReadDataFiles():model.cc:216) Silence phones 1:2:3:4:5:6:7:8:9:10
    LOG (VoskAPI:RemoveOrphanNodes():nnet-nnet.cc:948) Removed 0 orphan nodes.
    LOG (VoskAPI:RemoveOrphanComponents():nnet-nnet.cc:847) Removing 0 orphan components.
    LOG (VoskAPI:CompileLooped():nnet-compile-looped.cc:345) Spent 0.0268691 seconds in looped compilation.
    LOG (VoskAPI:ReadDataFiles():model.cc:248) Loading i-vector extractor from model/ivector/final.ie
    LOG (VoskAPI:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
    LOG (VoskAPI:ComputeDerivedVars():ivector-extractor.cc:204) Done.
    LOG (VoskAPI:ReadDataFiles():model.cc:281) Loading HCL and G from model/graph/HCLr.fst model/graph/Gr.fst
    LOG (VoskAPI:ReadDataFiles():model.cc:302) Loading winfo model/graph/phones/word_boundary.int
    ################################################################
    Press Ctrl+C to stop the recording
    ################################################################
    {
    "partial" : ""
    }
    {
    "partial" : "offline"
    }
    {
    "partial" : "offline"
    }
    {
    "partial" : "offline voice"
    }
    {
    "partial" : "offline voice agent"
    }
    ^C
    Done

    By default, this example picks the model from the directory model/ stored in /vosk-api/python/examples/. We can define from where to fetch the model using the —model parameter while running the python script.

    • Layers allow us to isolate different customizations from each other and keep our metadata modular and logically separated. Currently, there is no available OpenEmbedded Layer for Vosk and any other libraries that we might have to use for our application. Thus, it would be a good idea to create a new layer that takes care of the speech recognition application, using Vosk.

    • A few dependencies for our application are: | Dependency | Existing Recipes / To be created | | ---------- | -------------------------------- | | PyAudio · PyPI | Existing: python3-pyaudio | | sounddevice | Custom recipe to be created, if used |

    • Keeping the meta-agl-devel layer as the staging layer during development, a new layer, named meta-voice-agent, for example, can be created for the purpose. Since the project is mostly python-based, we need to make sure that the meta-openembedded/meta-python layer is included in our environment.

    • With the introduction of the new Yocto Release 3.5, the python package build process is based on wheels. Thus, respective packaging classes will be used as mentioned in the documentation.

    • Next, create custom recipes for fetching the API and installing required dependencies (for the dependencies for which no recipe exists, or grab recipes from meta-openembedded (if they exist) to build and run Vosk. Since the required models for different languages are needed, we need to retrieve them from the Vosk repository. The respective model has to be stored at a preferred location and needs to be searched, depending on the selected language. A few things to note:

      • The model path can be defined while calling the speech recognition module
      • A config file that defines the required path(s) can be created to select the required model
      • Recipes can be defined that create packages for respective models

    • With the help of example scripts provided by the vosk-api, python programs specific to our application can be written. Thus, different objectives can be achieved, from creating a simple demo for testing purposes to creating an application that uses Vosk for specific tasks. Vosk provides a vosk-server implementation as well. The demo application, if developed, can be based on HTML5/Flutter and will simply display what was detected.

    • AGL uses the pipewire daemon service for audio capture and playback. Pipewire (Gitlab) is a low-level multimedia framework that provides Graph-based processing, flexible media format negotiation, and real-time capable plugins among other things. Then, the session manager (WirePlumber) is a secondary process that also connects to pipewire as a client and can control the whole pipeline. WirePlumber provides an API to write tools that interact with PipeWire. It provides a daemon that is scriptable in Lua. For our application, when the speech recognition demo will be run, the other audio streams would need to go into the background. The corresponding policies will be managed by WirePlumber.

    AGL Sound System Architecture

    Vosk (python) depends on PyAudio, which provides Python bindings for PortAudio. Thus PortAudio modules are needed for a successful install of PyAudio on Linux. As per the PipeWire issue-PortAudio Support, PortAudio works with the JACK, Pulse, and ALSA libraries shipped with PipeWire.

    • This objective will be complete with the creation and testing of a demo that recognizes the user’s audio input in a language of choice and gives a text output.
Objective-2: After completion of Objective-1, develop the voice-agent to execute basic commands and interact with different applications (like the infotainment system, navigation, HVAC) to perform different functions.

  1. A voice agent that is capable of performing tasks will be based upon the following units:

    • Stage - 1 ⇒ Activation: the agent needs to be activated using a prompt, either through an application or Graphical User Interface, or using wake words. Since we are using an offline speech recognition API, we can configure and identify trigger words using that itself.
    • Stage - 2 ⇒ Response to the trigger input: This is the stage that will execute another function. Once the agent is triggered, it will listen for the command that the user wants to execute. After recognizing the command from the given set of commands, the suitable workflow for the function can be chosen and executed.

  2. For our application, we can add two modes to trigger the voice agent:

    • push-to-listen: the user presses a button on the UI to activate the speech recognizer
    • using wake words: the speech recognizer keeps running and listening for wake words. A close implementation for the wake word plugin for Vosk has been done here.

  3. The voice agent has to be integrated with the application for which the command is given. After recognition of the command, different functions can be performed, for example,

The methodology and approach would be finalized after deliberation with mentor (Jan-Simon Möller) at a later stage of GSoC.

Why this project?:

Local System Information

  1. Machine: Lenovo Thinkpad E14 Gen-3

  2. Technical Specifications:

    • Memory: 24GB
    • Processor: 8 core AMD Ryzen 7 5700U
    • Operating Systems:
      • Windows 11
      • Ubuntu 20.04 LTS
      • Fedora 36
    • Storage: 1TB

Timeline

Commitments, Availability, Past Contributions, and Experiences

Commitment and Availability

Past Experience and Contributions

Post-GSoC Period

⟵ back to mission log