guid: F2193578-1773-43A9-85CA-79EA8CE48D7B
author: Mihael Pranjić
pubDate: Fri, 03 Mar 2023 08:07:28 +0100
shortDesc: New Backend Logic, Copy-on-Write, FLOSS Course

We are proud to release Elektra 0.9.<<VERSION>>.

What is Elektra?

Elektra serves as a universal and secure framework to access configuration settings in a global, hierarchical key database. For more information, visit https://libelektra.org.

You can also read the news on our website.

You can try out the latest Elektra release using our docker image elektra/elektra. This is the quickest way to get started with Elektra without compiling and other obstacles, simply run:

docker pull elektra/elektra

docker run -it elektra/elektra

Highlights

New Backend
Copy-on-Write
FLOSS

New Backend

The entire logic for backends has been rewritten, to allow for more flexibility und an unlimited number of plugins. Instead of calling plugins directly, libelektra-kdb now only calls so-called backend plugins and special hook plugins. There is a contract between libelektra-kdb and the backend plugins. All backend plugins must adhere to this contract. To achieve this goal, most backend plugins will call other plugins (like libelektra-kdb did previously).

The logic previously implemented in libelektra-kdb was moved to the new default backend plugin backend. It works like the old system, but now also allows an unlimited number of plugins in positions where that makes sense. For example, you can have unlimited postgetstorage plugins, but only a single getresolver.

There have also been slight changes to kdbGet and kdbSet. Please read their API docs to find out, if you rely on any behavior that has been altered. You can also read the new low-level docs to find out all the intricate details.

The structure of system:/elektra/mountpoints changed as well. Take a look at the new docs, if you need to know details.

Updating config

The mountpoint configuration format contains breaking changes and a manual upgrade process is needed. Follow these steps to upgrade the old mountpoint configuration to the new format:

Warning: BACK UP YOUR CONFIG FILES BEFORE UPDATING! We recommend making a backup of the file printed by kdb file system:/elektra/mountpoints before updating your installation. In the unlikely case that the migration script fails, you can still use the information from the backup to manually recreate your mountpoints.

To update your existing system:/elektra/mountpoints data you can use the migration script.

Note: To run the script you must have Elektra, Python (>= 3.7) and the Python binding installed. The script uses the Python binding to manipulate Keys and KeySets, but it does not use the kdb CLI tool, or the KDB API. It is safe to run this script before, or after you update your Elektra installation.

By default, the script loads the file /etc/kdb/elektra.ecf. If you changed where system:/elektra/mountpoints is stored, you can provide an alternative path:

./migrate-mountpoints.py /path/to/your/mountpoints/config.file

Note: Because the script does not use the KDB API it only works, if the mountpoints config file uses the default dump format.

If your mountpoints config file is not using the dump format, you may still be able to use the migration script. However, in that case, you will have to use the script before updating your Elektra installation:

Run kdb export system:/elektra/mountpoints dump to get a copy of your mountpoints config in dump format

Write this data to a file and run the migration script on the file.

To get the data back in your original format you can use

./migrate-mountpoints.py /path/to/file/from/step2 | kdb convert dump your-format > /path/to/converted/file

Run kdb file system:/elektra/mountpoints to find out where your mountpoint config is stored. Make sure to back up this file, before upgrading your installation.

Now upgrade your Elektra installation.

Copy the file /path/to/converted/file from step 3 to the location you got in step 4.

The script will read the old mountpoint configuration from the given file. It will convert the configuration and print the new version to stdout.

You can inspect the output to make sure, everything is in order. When you are ready to commit the changes, you can manually edit the config file, or use:

./migrate-mountpoints.py --output=/etc/kdb/elektra.ecf /etc/kdb/elektra.ecf

Individual changes

Implement hooks. _(Maximilian Irlinger @atmaxinger)_
Removed old global plugins code. _(Maximilian Irlinger @atmaxinger)_
New backend logic, based on PR #2969 by @vLesk. _(@kodebach)_
Add script to migrate system:/elektra/mountpoints to new format. _(@kodebach)_

Copy-on-Write

Thanks to _(Maximilian Irlinger @atmaxinger)_ our Key and KeySet datastructures are now fully copy-on-write! This means noticeably reduced memory usage for cases where keys and keysets are copied and/or duplicated!

We ran some very promising benchmarks, each were performed with 400,000 keys. All benchmarks were executed using valgrind --tool=massif --time-unit=B --max-snapshots=200 --threshold=0.1.

Benchmark	Old Implementation	Copy-on-Write	Size Reduction	Remarks
`createkeys.c`	5.3 MiB	6.5 MiB	-22 %
`deepdup.c`	10.5 MiB	8.2 MiB	22 %
`large.c`	18.9 MiB	15.3 MiB	19 %
`kdb.c`	23.5 MiB	17.8 MiB	24 %
`kdbget.c`	11.0 MiB	8.8 MiB	20 %
`kdbmodify.c`	11.0 MiB	8.8 MiB	20 %	Same results as `kdbget.c`

First, it should be noted that a single key, without counting payload, is about 50% larger with the copy-on-write implementation. This explains why the createkeys.c benchmark yields a negative reduction result. This benchmark only allocates keys, so not much improvement can be expected there. Still, as other stuff also uses heap memory, the overall memory consumption only increased by 22%, which is far less than 50%.

All other benchmarks saw meaningful reductions of heap memory used. One interesting observation is that kdbget.c and kdbmodify.c used exactly the same memory. This can most likely be explained by internal caching within the memory allocator of glibc.

We also performed runtime tests on the same benchmarks using perf stat --repeat 13 to ensure no major performance regressions occur.

Benchmark	Old Implementation	Deviation	Copy-on-Write	Deviation	Runtime Increase
`createkeys.c`	0.209572 s	0.36 %	0.21987 s	0.77 %	4.9 %
`deepdup.c`	0.23025 s	0.47 %	0.231804 s	0.32 %	0.6 %
`large.c`	1.14038 s	0.21 %	1.14837 s	0.21 %	0.7 %
`kdb.c`	1.9270 s	2.63 %	1.93354 s	0.17 %	0.3 %
`kdbget.c`	0.145663 s	0.17 %	0.15763 s	0.70 %	8.2 %
`kdbmodify.c`	0.146506 s	0.19 %	0.156347 s	0.15 %	6.7 %

Overall, the runtime performance hit is less than 10%. The more a program does, the less the additional overhead of the copy-on-write algorithms matter. One interesting detail is that keyCopy and keyDup have become quite a bit faster. This can be seen by comparing the differences between createkeys.c and deepdup.c. The differences are 21 ms for the old implementation and 12 ms for the copy-on-write implementation.

FLOSS

A Free/Libre and Open Source Software (FLOSS) Initiative couldn't survive with many small contributions fixing annoying problems. This release also contains all contributions done via one term of the FLOSS course. The success was tremendous, as shown in the rest of the release notes.

A big thanks to the students for their contributions!

Plugins

The following text lists news about the plugins we updated in this release.

yajl

Fix an issue where trying to set invalid meta-keys won't show an error. _(Juri Schreib @Bujuhu)_

list

Removed the outdated list plugin. _(Maximilian Irlinger @atmaxinger)_ (Was only needed for global plugins, which are now replaced by hooks.)

logchange

Made logchange a notification-send hook plugin. _(Maximilian Irlinger @atmaxinger)_

toml

Fix bug, where meta-keys that cannot be inserted don't report an error. _(@Bujuhu)_

uname

Add error handling if uname call fails. _(Richard Stöckl @Eiskasten)_

quickdump

elektraQuickdumpSet: don't fclose if stdout. _(@hannes99)_

blockresolver

Add encoding test for blockresolver read. _(@dtdirect)_
Refactor and restructure blockresolver. _(@dtdirect)_

desktop

Add a unit test. _(Richard Stöckl @Eiskasten)_

mini

Fix a bug where writing meta-keys will fail silently. _(Juri Schreib @Bujuhu)_

mmapstorage

Remove code duplication in the data block calculation. _(Richard Stöckl @Eiskasten)_

network

Add a retry mechanism. _(Richard Stöckl @Eiskasten)_

xfconf

Add xfconf storage plugin with the ability to read and write to xfconf channels. _(Richard Stöckl @Eiskasten)_
Make xfconf valgrind suppressions more flexible to lib updates. _(Mihael Pranjić @mpranj)_

date

Fix an issue with validationg RFC 822 date-times. _(Juri Schreib @Bujuhu)_ _(Nikola Prvulovic @Dynamichost96)_
Improve Code Coverage. _(Juri Schreib @Bujuhu)_ _(Nikola Prvulovic @Dynamichost96)_

csvstorage

Fix a bug where writing unkown meta-keys will fail silently. _(Juri Schreib @Bujuhu)_

Libraries

The text below summarizes updates to the C (and C++)-based libraries of Elektra.

Compatibility

Global plugins do not work anymore, use hooks instead.

Core

The Key and KeySet datastructures are now fully copy-on-write. _(Maximilian Irlinger @atmaxinger)_
keyCopy now only allocates additional memory if KEY_CP_META or KEY_CP_ALL is used. _(Maximilian Irlinger @atmaxinger)_
Check for circular links (overrides). _(@0x6178656c)_

io

Check file flags for elektraIoFdSetFlags: file flags must be exactly one of: read only, write only or read write. _(Richard Stöckl @Eiskasten)_

Merge

Add methods elektraMergeGetConflictingKeys and elektraMergeIsKeyConflicting to check which keys were causing a merge conflict. _(Maximilian Irlinger @atmaxinger)_

Bindings

Bindings allow you to utilize Elektra using various programming languages. This section keeps you up-to-date with the multi-language support provided by Elektra.

intercept/env

Remove fallback code. _(Markus Raab)_
Command-line functionality is broken due to new-backend differences.

intercept/fs

Use KDB_MAX_PATH_LENGTH for better portability. _(Markus Raab)_

jna

Documentation: Improved build instructions. _(@Bujuhu)_
Add validation on get for whitelist plugin. _(@Bujuhu)_
Upgrade Gradle to 8.0.1. _(Mihael Pranjić @mpranj)_

rust

Fix "feature `resolver` is required" error. _(Markus Raab)_

elixir

Initial release of the Elixir binding. _(@0x6178656c)_
Mark tests as memleak. _(@0x6178656c)_

Tools

kdb

Removed global-mount and global-umount commands. _(Maximilian Irlinger @atmaxinger)_
Fixed SIGSEGV when using kdb find without argument. _(Christian Jonak-Moechel @joni1993)_

elektrad

Removed leftover package-lock.json file. _(stefnotch)_

Scripts

Added automatic spelling corrections for changeset. _(Maximilian Irlinger @atmaxinger)_
Introduce shebang-conventions. _(@0x6178656c)_
Apply auto-fixes from shellcheck. _(@0x6178656c)_
Use prettier 2.8.4. _(Mihael Pranjić @mpranj)_
Add scripts to enable and disable pre-commit hooks. _(Juri Schreib @Bujuhu)_ _(Nikola Prvulovic @Dynamichost96)_
Only let http links pass the check if whitelisted. _(Richard Stöckl @Eiskasten)_
Link Checker: Add documentation for the usage and how it behaves. _(Richard Stöckl @Eiskasten)_
Sed: Add spelling correction for "key-value storage". _(@Bujuhu)_
Fix/extends some shell recorder tests. _(@Joni1993)_
Use clang-format v15. _(Mihael Pranjić @mpranj)_
Fix warning parsing in shell recorder. _(@Joni1993)_
Replaced egrep by grep -E. _(@0x6178656c and @janldeboer)_
Add audit-dependencies script to check for vulnerabilities for npm depndencies. _(Juri Schreib @Bujuhu)_ _(Nikola Prvulovic @Dynamichost96)_

Documentation

Restructured contrib/api. _(Markus Raab)_.
Improve page on compilation. _(@0x6178656c)_
Improve page for bindings. _(@0x6178656c)_
Improve page for getting started. _(@stefnotch)_
Remove version number from docker README and replace it with latest. _(@Joni1993)_
Fix grammar for elektra-granularity.md. _(@dtdirect)_
Rephrase sections in doc/dev/error-\*. _(@dtdirect)_
Improve Git.md. _(Juri Schreib @Bujuhu)_ _(Nikola Prvulovic @Dynamichost96)_
Unify spelling of man pages. _(@stefnotch)_ _(@janldeboer)_
Extend consistency check check_doc.sh to work for contrib, dev and tutorials. _(@Joni1993)_
Fix internal links. _(@0x6178656c)_
Update AUR Link from elektra to libelektra package. _(@Bujuhu)_
Update example Ansible playbook in VISION.md. _(@Bujuhu)_
Harmonize spelling of Git. _(@Joni1993)_
Update packaging instructions for Fedora. _(@0x6178656c)_
Improve use of gender. _(@0x6178656c)_
Fix some minor mistakes in CONTRIBUTING.md. _(@Joni1993)_
Fix various spelling errors. _(@Joni1993)_
Denoted package names & global variable names in INSTALL.md as Code. _(@janldeboer)_
Improve readability of doc/tutorials/highlevel.md. _(@deoknats861)_
Improve reference to Podman documentation. _(@0x6178656c)_
Unify spelling. _(@Joni1993)_
Fix typo in dev/hooks.md. _(@dtdirect)_
Remove unused images from doc/images. _(@dtdirect)_
Fixed Coverage Badge Link. _(@janldeboer)_
Improve CONTRIBUTING doc. _(Juri Schreib @Bujuhu)_ and _(Nikola Prvulovic @Dynamichost96)_
Update Doxyfile with Doxygen 1.9.4. _(@0x6178656c)_
Add project logo to Doxygen in Doxyfile. _(@dtdirect)_
Add mermaid.js to the project using doxygen-mermaid. _(@dtdirect)_
Create diagrams in mermaid.js to use in doxygen. _(@dtdirect)_
Create README for Doxygen and Mermaid JS. _(@dtdirect)_
Tutorial: Add automatic validation to Docker tutorial _(Schreib @Bujuhu)_ _(Nikola Prvulovic @Dynamichost96)_
Add mention of audit-dependencies script in doc/todo/RELEASE.md. _(@Bujuhu)_
Move note in GETSTARTED.md. _(@Joni1993)_
Use code blocks to prevent Markdown from falsy rendering LaTeX. _(@stefnotch)_, _(@janldeboer)_
Fix broken links in use cases for KDB after files were renamed. _(Florian Lindner @flo91)_
Replace http links with https. _(Richard Stöckl @Eiskasten)_
Enhance notifications.md in doc/tutorial. _(@dtdirect)_
Add tutorial how to suppress memleaks in plugins from dependencies. _(Richard Stöckl @Eiskasten)_
Write about history to make plans of Elektra's adoption more clear. _(Markus Raab)_

Use Cases

Improve use cases Template. _(@kodebach and Markus Raab)_
Use cases for KDB. _(@kodebach)_
Use cases for libelektra-core. _(@kodebach)_

Decisions

Decide and implement decision process. _(Markus Raab)_
Decided future library split. _(@kodebach)_
Decided decision process. _(Markus Raab)_
Draft for man pages. _(Markus Raab)_
Add decision for change tracking. _(Maximilian Irlinger @atmaxinger)_
Create decision for allowed and prohibited operation seqences. _(Maximilian Irlinger @atmaxinger)_
Add decisions about location of headers and use of `#include` in the repo. _(@kodebach)_
Add decision about metadata semantics. _(@kodebach)_
Many small fixes to adapt to documentation guidelines and new decision process. _(Markus Raab)_
Add decision for read-only keynames. _(Maximilian Irlinger @atmaxinger)_
Revive keyname decision. _(@kodebach)_
Add decisions for constructor functions and builder functions. _(@kodebach)_
Add decision for copy-on-write and provide implementation suggestions. _(Maximilian Irlinger @atmaxinger)_
Update internal cache. _(Markus Raab)_ _(@kodebach)_
Create transformations. _(Maximilian Irlinger @atmaxinger)_
Replace TOC-style README.md with folders and generate HTML for website. _(@kodebach)_
Restructured decisions directories based on new agreed-upon steps. _(Maximilian Irlinger @atmaxinger)_
Decision for types of `KeySet`s. _(@kodebach)_
Added Documentation Guidelines. _(Markus Raab)_
Add links and formatting to documents affected by PR#4492 (Document Guidelines) and rephrase some parts. _(Florian Lindner @flo91)_
Decisions for changes to keyIsBelow and new keyGetNextPart functions. _(@kodebach)_
Apply fix spelling to more files. _(Markus Raab)_

Tutorials

Add tutorial for manual installation from the AUR on Arch Linux. _(@Bujuhu)_
Add Markdown shell recorder validation to install.webui.md. _(@deoknats861)_
Fix the outdated array tutorial. _(Juri Schreib @Bujuhu)_ _(Nikola Prvulovic @Dynamichost96)_
Reinstate mounting tutorial. _(@Bujuhu)_
Make namespaces tutorial verifiable. _(@0x6178656c)_
Move Podman-related information to a dedicated page. _(@0x6178656c)_

Man Pages

Update man page (patch) as suggested by the CI to fix CI error on master. _(Florian Lindner @flo91)_
Added links to the website & webui after further reading. _(Philipp Nirnberger @nirnberger)_
Upgrade ronn-ng to 0.10.1.pre3. _(Mihael Pranjić @mpranj)_

Tests

Fix an Issue where scripts/dev/fix-spelling does not work, if a resolved path contains whitespaces. _(Juri Schreib @Bujuhu)_ _(Nikola Prvulovic @Dynamichost96)_
Rename scripts/sed to scripts/spelling.sed. _(Juri Schreib @Bujuhu)_ _(Nikola Prvulovic @Dynamichost96)_
Add memleak label to test_getenv. _(@0x6178656c)_
Add test using shellcheck. _(@0x6178656c)_
Remove --rerun-failed from run_* scripts. _(@kodebach)_
Fix paths for icheck test. _(Mihael Pranjić @mpranj)_

Shell Recorder

Add check if file exists. _(@0x6178656c)_

Packaging

Add missing new backend plugin to components of libelektra package. _(Mihael Pranjić @mpranj)_

Build

CMake

Fix warning for CMP0115. _(0x6178656c)_
Change Doxygen configuration for LaTeX. _(0x6178656c)_
Fix developer warning for package DISCOUNT. _(Dennis Toth @dtdirect)_
Pass --stacktrace to gradle for the JNA builds. _(Maximilian Irlinger @atmaxinger)_
Adapt npm build flags to remove reproducability issues. _(Juri Schreib @Bujuhu)_ _(Nikola Prvulovic @Dynamichost96)_
Fix creation of shell recorder tests. _(@0x6178656c)_

Docker

Update packagename libpcrec++-dev to libpcrecpp0v5 in Debian Sid. _(Richard Stöckl @Eiskasten)_
Add shellcheck to Debian containers. _(@0x6178656c)_
Use openjdk-17-jdk in Debian Sid. _(Maximilian Irlinger @atmaxinger)_
Add Fedora 37 images. _(Mihael Pranjić @mpranj)_
Update Debian Sid image to use repository Python modules instead of installing with pip3 due to upstream debian changes. _(Mihael Pranjić @mpranj)_
Debian Bullsye: use clang 13. _(Mihael Pranjić @mpranj)_
Update Alpine Linux to 3.17.2. _(Mihael Pranjić @mpranj)_

Gradle

Use Gradle 7.5.1. _(Mihael Pranjić @mpranj)_
Update java-library.gradle to use archiveClassifier _(Maximilian Irlinger @atmaxinger)_

Infrastructure

Jenkins

Add Fedora 37 builds, drop Fedora 35 builds. _(Mihael Pranjić @mpranj)_
Run more tests also on Master. _(Markus Raab)_
Move doc to main build stage. _(Markus Raab)_
Disable parallel test runs. _(Maximilian Irlinger @atmaxinger)_
Upgrade Jenkins node container to Debian bullseye. _(@0x6178656c)_
Undo previous change that added automatic ctest --rerun-failed to Jenkins CI. _(@kodebach)_

Cirrus

Use Fedora 37. _(Mihael Pranjić @mpranj)_
Fix macos_instance reference, upgrade to macOS Ventura (by default), use Python 3.11 and Ruby 3.x. _(Mihael Pranjić @mpranj)_
Automatically rerun testmod_dbus* tests on macOS. _(@kodebach)_
Fix dbus not starting on macOS. _(Maximilian Irlinger @atmaxinger)_

GitHub Actions

Add auto-cancellation-running action. _(Tomislav Makar @tmakar)_
Automatically rerun testmod_dbus* tests on macOS. _(@kodebach)_
Fix dbus not starting on macOS. _(Maximilian Irlinger @atmaxinger)_
Change stale issue/PR checking to GitHub action. _(@0x6178656c)_
Update configuration of stale issue/PR action. _(@0x6178656c)_
Upgrade actions to recent versions and remove deprecated ruby-setup action. _(Mihael Pranjić @mpranj)_

Website

The website is generated from the repository, so all information about plugins, bindings and tools are always up-to-date. Furthermore, we changed:

Fix broken /pythongen link on homepage. _(@stefnotch)_
Fix redirect logic to not cause loops. _(@stefnotch)_
Remove duplicated link to TESTING.md file. _(@stefnotch)_, _(@janldeboer)_
Restructure parts of the links on the website. _(@stefnotch)_, _(@janldeboer)_
Removed broken links to packages for Linux distributions. _(@Dynamichost96)_
Update npm packages. _(Mihael Pranjić @mpranj)_
Change URLs to say man-page with a dash. _(@stefnotch)_ _(@janldeboer)_

Outlook

We are currently working on following topics:

1.0 API _(Klemens Böswirth @kodebach)_ and _(Stefan Hanreich)_
Session recording and better Ansible integration _(Maximilian Irlinger @atmaxinger)_
Change tracking _(Maximilian Irlinger @atmaxinger)_
Rewriting tools in C _(@hannes99)_
Elektrify KDE and GNOME _(Mihael Pranjić @mpranj)_
Elektrify XFCE _(Richard Stöckl @Eiskasten)_
Mounting SQL databases _(Florian Lindner @flo91)_
Recording Configuration _(Maximilian Irlinger)_
Ansible-Elektra _(Lukas Hartl)_ and _(Maximilian Irlinger)_
Configure Olimex Base Images _(Maximilian Irlinger)_
Improving Build Server Infrastructure _(Lukas Hartl)_ and _(Maximilian Irlinger)_
Improve Java Development Experience _(Michael Tucek)_