Contents

GSoC Weekly Update: Week 1, 2


This post will briefly cover:

  • Learnings
  • Build issue on personal machine, and solving it
  • Identifying dependencies
  • Tasks done, and those in progress
  • Useful resources

For the project proposal, visit here.

Build issue on personal machine:

While building a fresh AGL image on my personal machine, I encountered errors as listed:

1
2
3
4
Summary: 2 tasks failed:
  virtual:native:/home/aman/AGL/marlin/external/poky/meta/recipes-devtools/llvm/llvm_git.bb:do_compile
  /home/aman/AGL/marlin/external/meta-openembedded/meta-oe/dynamic-layers/meta-python/recipes-extended/mozjs/mozjs_60.9.0.bb:do_compile
Summary: There were 3 ERROR messages shown, returning a non-zero exit code.

Separately compiling the packages seemed to have solved the issue in the next run. The reason could be too many parallel processes over-utilizing the available memory.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
$ bitbake mozjs -c compile

...(build output omitted)

NOTE: Tasks Summary: Attempted 747 tasks of which 746 didn\'t need to be rerun and all succeeded.


$ bitbake llvm -c compile

... (build output omitted)

NOTE: Tasks Summary: Attempted 758 tasks of which 753 didn\'t need to be rerun and all succeeded.

# Next build succeeded
$ BB_NUMBER_THREADS=2 bitbake agl-demo-platform
... (build output omitted)

# A NOTE on the above command. BB_NUMBER_THREADS is used to set the number of concurrent bitbake build processes running. We can also set this variable in the conf/local.conf

Ongoing Work:

Checking required libraries and dependencies for the project:

These include (but are not limited to):

  1. Vosk (available as pip package), which further depends on:

On Vosk API github, we get setup.py which lists various dependencies in install_requires.

We can use ensurepip on the target, but it has to be used everytime we build a new image. In order to port dependencies on the image automatically, corresponding recipes are required.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
root@qemux86-64:~# python3 -m ensurepip
Looking in links: /tmp/tmpkzeugzrm
Processing /tmp/tmpkzeugzrm/setuptools-56.0.0-py3-none-any.whl
Processing /tmp/tmpkzeugzrm/pip-22.0.4-py3-none-any.whl
Installing collected packages: setuptools, pip
Successfully installed pip-22.0.4 setuptools-56.0.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

root@qemux86-64:~# python3 -m pip install vosk
Collecting vosk
  Downloading vosk-0.3.42-py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (7.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.2/7.2 MB 4.6 MB/s eta 0:00:00
Collecting tqdm
  Downloading tqdm-4.64.0-py2.py3-none-any.whl (78 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.4/78.4 KB 11.6 MB/s eta 0:00:00
Collecting requests
  Downloading requests-2.28.0-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 KB 10.9 MB/s eta 0:00:00
Collecting cffi>=1.0
  Downloading cffi-1.15.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (446 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 446.7/446.7 KB 6.9 MB/s eta 0:00:00
Collecting srt
  Downloading srt-3.5.2.tar.gz (24 kB)
  Preparing metadata (setup.py) ... done
Collecting pycparser
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 KB 8.5 MB/s eta 0:00:00
Collecting idna<4,>=2.5
  Downloading idna-3.3-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.2/61.2 KB 5.0 MB/s eta 0:00:00
Collecting charset-normalizer~=2.0.0
  Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.0/139.0 KB 10.8 MB/s eta 0:00:00
Collecting certifi>=2017.4.17
  Downloading certifi-2022.6.15-py3-none-any.whl (160 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 160.2/160.2 KB 5.9 MB/s eta 0:00:00
Using legacy 'setup.py install' for srt, since package 'wheel' is not installed.
Installing collected packages: urllib3, tqdm, srt, pycparser, idna, charset-normalizer, certifi, requests, cffi, vosk
  Running setup.py install for srt ... done
Successfully installed certifi-2022.6.15 cffi-1.15.0 charset-normalizer-2.0.12 idna-3.3 pycparser-2.21 requests-2.28.0 srt-3.5.2 tqdm-4.64.0 urllib3-1.26.9 vosk-0.3.42
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

# Similarly, for all other pip based dependencies.

As a working example, the following lines can also be added in conf/local.conf inside our build directory to include packages as required upon image-boot.

1
2
3
IMAGE_INSTALL += " python3-requests"
IMAGE_INSTALL:append = " python3-cffi"
IMAGE_INSTALL:append = " python3-tqdm"

But to avoid repeating the process and changing local.conf everytime, we create the required recipes for the meta-offline-voice-agent layer.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
$ cd $AGL_TOP/marlin/meta-agl-devel/meta-offline-voice-agent/
$ mkdir recipes-vosk
$ cd recipes-vosk/

# python3-vosk will contain recipe for Vosk API and its dependencies.
$ mkdir python3-vosk
$ cd python3-vosk

# Based on the current version of Vosk (v0.3.42), created the recipe file.
$ touch python3-vosk_0.3.42.bb
$ cd ..

# python3-srt will contain recipe for the python srt library.
$ mkdir python3-srt
$ cd python3-srt

# Based on the current version of srt library (v3.5.2), created the recipe file.
$ touch python3-srt_3.5.2.bb

$ cd ../..

# Upon creating the required recipe files (still under development), the layer could look something like:
$ tree -L 3
.
|-- COPYING.MIT
|-- README
|-- conf
|   `-- layer.conf
`-- recipes-vosk
    |-- python3-srt
    |   `-- python3-srt_3.5.2.bb
    `-- python3-vosk
        `-- python3-vosk_0.3.42.bb

The current contents of the recipes python3-srt_3.5.2.bb and python3-vosk_0.3.42.bb (currently working on this recipe) are:

python3-srt_3.5.2.bb which compiles successfully.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# python3-srt_3.5.2.bb

DESCRIPTION = "Recipe for the python 'srt' library"
HOMEPAGE = "https://github.com/cdown/srt"
LICENSE = "CLOSED"

inherit pypi setuptools3

PYPI_PACKAGE="srt"
SRC_URI[sha256sum] = "7aa4ad5ce4126d3f53b3e7bc4edaa86653d0378bf1c0b1ab8c59f5ab41384450"

DEPENDS = "python3-pip-native"

python3-vosk_0.3.42.bb

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
# python3-vosk_0.3.42.bb

DESCRIPTION = "Recipe for Vosk Python API"
HOMEPAGE = "https://github.com/alphacep/vosk-api/tree/master/python"
LICENSE = "CLOSED"

inherit setuptools3

PYTHON_DEPENDS = " \
    python3-requests \
    python3-cffi \
    python3-tqdm \
    python3-srt \
"

PYPI_PACKAGE="vosk"
BB_STRICT_CHECKSUM = "0"

DEPENDS = "python3-pip-native"

RDEPENDS_${PN} += "${PYTHON_DEPENDS}"

On running bitbake, this recipe currently gives the following error:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
ERROR: python3-vosk-0.3.42-r0 do_compile: 'python3 setup.py build ' execution failed.
ERROR: python3-vosk-0.3.42-r0 do_compile: Execution of '/home/aman/AGL/marlin/qemux86-64/tmp/work/corei7-64-agl-linux/python3-vosk/0.3.42-r0/temp/run.do_compile.153930' failed with exit code 1
ERROR: Logfile of failure stored in: /home/aman/AGL/marlin/qemux86-64/tmp/work/corei7-64-agl-linux/python3-vosk/0.3.42-r0/temp/log.do_compile.153930
Log data follows:
| DEBUG: Executing shell function do_compile
| /home/aman/AGL/marlin/qemux86-64/tmp/work/corei7-64-agl-linux/python3-vosk/0.3.42-r0/recipe-sysroot-native/usr/bin/python3-native/python3: can't open file 'setup.py': [Errno 2] No such file or directory
| ERROR: 'python3 setup.py build ' execution failed.
| WARNING: exit code 1 from a shell command.
| ERROR: Execution of '/home/aman/AGL/marlin/qemux86-64/tmp/work/corei7-64-agl-linux/python3-vosk/0.3.42-r0/temp/run.do_compile.153930' failed with exit code 1
ERROR: Task (/home/aman/AGL/marlin/meta-agl-devel/meta-offline-voice-agent/recipes-vosk/python3-vosk/python3-vosk_0.3.42.bb:do_compile) failed with exit code '1'

Week 3 To-Do:

  • Complete the recipes and include vosk and other required packages in the image build.
  • Start looking at how to use PipeWire instead of PyAudio for our application.
  • Start looking at how to work on packages for language models, for Vosk to work.

Helpful Resources:

For the build error:
Development resources (Yocto-Dunfell <==> AGL-marlin):
For issues encountered: