This post will briefly cover:
- Learnings
- Build issue on personal machine, and solving it
- Identifying dependencies
- Tasks done, and those in progress
- Useful resources
For the project proposal, visit here.
Build issue on personal machine:
While building a fresh AGL image on my personal machine, I encountered errors as listed:
1
2
3
4
|
Summary: 2 tasks failed:
virtual:native:/home/aman/AGL/marlin/external/poky/meta/recipes-devtools/llvm/llvm_git.bb:do_compile
/home/aman/AGL/marlin/external/meta-openembedded/meta-oe/dynamic-layers/meta-python/recipes-extended/mozjs/mozjs_60.9.0.bb:do_compile
Summary: There were 3 ERROR messages shown, returning a non-zero exit code.
|
Separately compiling the packages seemed to have solved the issue in the next run. The reason could be too many parallel processes over-utilizing the available memory.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
$ bitbake mozjs -c compile
...(build output omitted)
NOTE: Tasks Summary: Attempted 747 tasks of which 746 didn\'t need to be rerun and all succeeded.
$ bitbake llvm -c compile
... (build output omitted)
NOTE: Tasks Summary: Attempted 758 tasks of which 753 didn\'t need to be rerun and all succeeded.
# Next build succeeded
$ BB_NUMBER_THREADS=2 bitbake agl-demo-platform
... (build output omitted)
# A NOTE on the above command. BB_NUMBER_THREADS is used to set the number of concurrent bitbake build processes running. We can also set this variable in the conf/local.conf
|
Ongoing Work:
Checking required libraries and dependencies for the project:
These include (but are not limited to):
- Vosk (available as pip package), which further depends on:
On Vosk API github, we get setup.py which lists various dependencies in install_requires.
We can use ensurepip on the target, but it has to be used everytime we build a new image. In order to port dependencies on the image automatically, corresponding recipes are required.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
|
root@qemux86-64:~# python3 -m ensurepip
Looking in links: /tmp/tmpkzeugzrm
Processing /tmp/tmpkzeugzrm/setuptools-56.0.0-py3-none-any.whl
Processing /tmp/tmpkzeugzrm/pip-22.0.4-py3-none-any.whl
Installing collected packages: setuptools, pip
Successfully installed pip-22.0.4 setuptools-56.0.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@qemux86-64:~# python3 -m pip install vosk
Collecting vosk
Downloading vosk-0.3.42-py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (7.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.2/7.2 MB 4.6 MB/s eta 0:00:00
Collecting tqdm
Downloading tqdm-4.64.0-py2.py3-none-any.whl (78 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.4/78.4 KB 11.6 MB/s eta 0:00:00
Collecting requests
Downloading requests-2.28.0-py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 KB 10.9 MB/s eta 0:00:00
Collecting cffi>=1.0
Downloading cffi-1.15.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (446 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 446.7/446.7 KB 6.9 MB/s eta 0:00:00
Collecting srt
Downloading srt-3.5.2.tar.gz (24 kB)
Preparing metadata (setup.py) ... done
Collecting pycparser
Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 KB 8.5 MB/s eta 0:00:00
Collecting idna<4,>=2.5
Downloading idna-3.3-py3-none-any.whl (61 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.2/61.2 KB 5.0 MB/s eta 0:00:00
Collecting charset-normalizer~=2.0.0
Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting urllib3<1.27,>=1.21.1
Downloading urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.0/139.0 KB 10.8 MB/s eta 0:00:00
Collecting certifi>=2017.4.17
Downloading certifi-2022.6.15-py3-none-any.whl (160 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 160.2/160.2 KB 5.9 MB/s eta 0:00:00
Using legacy 'setup.py install' for srt, since package 'wheel' is not installed.
Installing collected packages: urllib3, tqdm, srt, pycparser, idna, charset-normalizer, certifi, requests, cffi, vosk
Running setup.py install for srt ... done
Successfully installed certifi-2022.6.15 cffi-1.15.0 charset-normalizer-2.0.12 idna-3.3 pycparser-2.21 requests-2.28.0 srt-3.5.2 tqdm-4.64.0 urllib3-1.26.9 vosk-0.3.42
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
# Similarly, for all other pip based dependencies.
|
As a working example, the following lines can also be added in conf/local.conf inside our build directory to include packages as required upon image-boot.
1
2
3
|
IMAGE_INSTALL += " python3-requests"
IMAGE_INSTALL:append = " python3-cffi"
IMAGE_INSTALL:append = " python3-tqdm"
|
But to avoid repeating the process and changing local.conf everytime, we create the required recipes for the meta-offline-voice-agent layer.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
|
$ cd $AGL_TOP/marlin/meta-agl-devel/meta-offline-voice-agent/
$ mkdir recipes-vosk
$ cd recipes-vosk/
# python3-vosk will contain recipe for Vosk API and its dependencies.
$ mkdir python3-vosk
$ cd python3-vosk
# Based on the current version of Vosk (v0.3.42), created the recipe file.
$ touch python3-vosk_0.3.42.bb
$ cd ..
# python3-srt will contain recipe for the python srt library.
$ mkdir python3-srt
$ cd python3-srt
# Based on the current version of srt library (v3.5.2), created the recipe file.
$ touch python3-srt_3.5.2.bb
$ cd ../..
# Upon creating the required recipe files (still under development), the layer could look something like:
$ tree -L 3
.
|-- COPYING.MIT
|-- README
|-- conf
| `-- layer.conf
`-- recipes-vosk
|-- python3-srt
| `-- python3-srt_3.5.2.bb
`-- python3-vosk
`-- python3-vosk_0.3.42.bb
|
The current contents of the recipes python3-srt_3.5.2.bb and python3-vosk_0.3.42.bb (currently working on this recipe) are:
python3-srt_3.5.2.bb which compiles successfully.
1
2
3
4
5
6
7
8
9
10
11
12
|
# python3-srt_3.5.2.bb
DESCRIPTION = "Recipe for the python 'srt' library"
HOMEPAGE = "https://github.com/cdown/srt"
LICENSE = "CLOSED"
inherit pypi setuptools3
PYPI_PACKAGE="srt"
SRC_URI[sha256sum] = "7aa4ad5ce4126d3f53b3e7bc4edaa86653d0378bf1c0b1ab8c59f5ab41384450"
DEPENDS = "python3-pip-native"
|
python3-vosk_0.3.42.bb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
# python3-vosk_0.3.42.bb
DESCRIPTION = "Recipe for Vosk Python API"
HOMEPAGE = "https://github.com/alphacep/vosk-api/tree/master/python"
LICENSE = "CLOSED"
inherit setuptools3
PYTHON_DEPENDS = " \
python3-requests \
python3-cffi \
python3-tqdm \
python3-srt \
"
PYPI_PACKAGE="vosk"
BB_STRICT_CHECKSUM = "0"
DEPENDS = "python3-pip-native"
RDEPENDS_${PN} += "${PYTHON_DEPENDS}"
|
On running bitbake, this recipe currently gives the following error:
1
2
3
4
5
6
7
8
9
10
|
ERROR: python3-vosk-0.3.42-r0 do_compile: 'python3 setup.py build ' execution failed.
ERROR: python3-vosk-0.3.42-r0 do_compile: Execution of '/home/aman/AGL/marlin/qemux86-64/tmp/work/corei7-64-agl-linux/python3-vosk/0.3.42-r0/temp/run.do_compile.153930' failed with exit code 1
ERROR: Logfile of failure stored in: /home/aman/AGL/marlin/qemux86-64/tmp/work/corei7-64-agl-linux/python3-vosk/0.3.42-r0/temp/log.do_compile.153930
Log data follows:
| DEBUG: Executing shell function do_compile
| /home/aman/AGL/marlin/qemux86-64/tmp/work/corei7-64-agl-linux/python3-vosk/0.3.42-r0/recipe-sysroot-native/usr/bin/python3-native/python3: can't open file 'setup.py': [Errno 2] No such file or directory
| ERROR: 'python3 setup.py build ' execution failed.
| WARNING: exit code 1 from a shell command.
| ERROR: Execution of '/home/aman/AGL/marlin/qemux86-64/tmp/work/corei7-64-agl-linux/python3-vosk/0.3.42-r0/temp/run.do_compile.153930' failed with exit code 1
ERROR: Task (/home/aman/AGL/marlin/meta-agl-devel/meta-offline-voice-agent/recipes-vosk/python3-vosk/python3-vosk_0.3.42.bb:do_compile) failed with exit code '1'
|
Week 3 To-Do:
- Complete the recipes and include vosk and other required packages in the image build.
- Start looking at how to use PipeWire instead of PyAudio for our application.
- Start looking at how to work on packages for language models, for Vosk to work.
Helpful Resources:
For the build error:
Development resources (Yocto-Dunfell <==> AGL-marlin):
For issues encountered: